There is a long-standing hypothesis in physical anthropology that a sensory trade-off has occurred between smell and vision in primate evolution. This trade-off is suggested to have taken place during the time that anthropoid primates adapted to a diurnal rather than nocturnal activity pattern. Thus, these primates began using visual stimuli more prominently than smell or chemosensation in communication. While this trade-off hypothesis seems supported by reduction in the anatomical structures related to pheromone detection and enhancement of the visual system in some living primates, it has not been tested using the fossil record. Fossils have largely been excluded from answering this question because features related to chemosensation, particularly the vomeronasal organs, which are important in pheromone detection, do not fossilize.
This study identifies bony anatomical features related to chemosensation in primates to understand variation in the expression of these features in living primates and compare the resultant information to fossil primates. We have identified a hard-tissue correlate of primate chemosensation in the form of bilateral bony grooves on the nasal floor, formed by articulation with the cartilage surrounding the vomeronasal organs. The palatal grooves and their relationship to the vomeronasal organs (occurrence, relative size) are analyzed statistically in a range of living primates using histological and CT methods. Skulls of fossil primates that preserve the nasal floor will are also examined using CT to identify the presence, size and morphology of the palatal grooves to understand the timing and context of the reduction of chemosensation in anthropoid primates and test whether this corroborates the sensory trade-off hypothesis.
This research promotes graduate education and will result in a Ph.D. for a female graduate student. The data recovered from this project will increase knowledge on the interface of ecology and adaptation in early primates as well as contributing information to understanding reduction of human chemosensation. This research will be made available to the scientific community and the interested public through journals and scientific magazines.
Mammals have two distinct olfactory (smell) systems, one of which (the main olfactory system) detects a large range of odorants, while the other (the vomeronasal system) detects a narrower range of species- specific, often social, odorants (pheromones). Traditionally, primates are considered visually dominant mammals that rely little on smell. This generalization derives from the observed reduction of anatomical regions related to smell, with an expansion of the visual system. The "Sensory Trade-off" hypothesis has been put forward to explain this, suggesting that smell became reduced in primates over evolutionary time as vision became the dominant sensory modality. While it may be an overgeneralization to state that all primates reduced their reliance on smell, there is a graded reduction of the vomeronasal system leading to functional loss in the Afro-Eurasian "higher" primates (Old World monkeys, apes, and humans). A functional vomeronasal system is present in the primitive lemurs and lorises, while in New World monkeys the vomeronasal system is variably functional and anatomically reorganized. The vomeronasal system is completely lost in Old World monkeys and is nonfunctional, but sometimes present vestigially in apes and humans. Old World monkeys, apes, and humans also have full trichromatic color vision, which many researchers speculate led to this trade-off between vision and olfaction. Data supporting this sensory trade-off hypothesis come from limited information on living primates. The vomeronasal organ consists of small, paired, epithelial tubes at the base of the nasal cavity, and observations of this organ require destructive techniques like histology, limiting the sample size for rare primates. The sensory trade-off hypothesis has not yet been tested using fossil material, because epithelium is a soft tissue and does not fossilize. This project aimed to identify a bony correlate of the vomeronasal organ in order to test the sensory trade-off hypothesis using fossils and also to increase our understanding of the vomeronasal organ in living primates. This project is the first to identify such a bony correlate, which we call the vomeronasal groove, expressed as a trough-like depression on the floor of the nasal cavity. Length of the vomeronasal groove strongly correlates with vomeronasal organ length, making it a useful proxy for the organ in skeletal or fossil specimens. The length of the vomeronasal groove is also strongly associated with the volume of sensory neuroepithelium and the proportion of functional vomeronasal receptor genes. Using microCT scanning, we have observed the vomeronasal groove in extant species that have not been studied histologically. As expected, Old World monkeys and apes lack the groove. The vomeronasal groove is present in some plesiadapiforms, extinct mammals that are either ancestral or closely related to primates, as well as in known primitive primate fossils. The vomeronasal groove also appears to be present (if relatively small) in fossils of the 30 million year old Aegyptopithecus, which is close to the common ancestor of Afro-Eurasian "higher" primates, but it is absent in the slightly younger and more advanced Saadanius. A small groove is similar to the condition in modern New World monkeys, primitive relatives of the Old World forms; the groove (and organ) may have been lost in ancestral Afro-Eurasian "higher" primates by the time of Saadanius. Additionally, this project looked at the relationship between genes encoding receptors for the main olfactory system and the vomeronasal system across mammals and how these relate to their corresponding sensory organs. It is apparent that vomeronasal organ length (adjusted for body size) and complexity scale positively with a larger proportion of functional vomeronasal receptor genes, while absolute numbers of total and functional olfactory receptor genes scale positively with absolute size of the main olfactory system. This study also confirms that the main olfactory system and vomeronasal system are under different selective regimes, and that the vomeronasal system was affected more than the main olfactory system in primate evolution. It is likely that higher primates still maintain very good olfactory sensitivity, acuity, and discrimination. Many studies demonstrate that Old World monkeys and humans, which do not have a functional vomeronasal system, can detect pheromone-like compounds, indicating that the main olfactory system plays a role in the detection of these odorants. This system detects a far greater range of odorants than the vomeronasal system, and many odorous social cues are learned through conditioning. A reasonable evolutionary scenario for vomeronasal system loss in higher primates is that when vision became a dominant sensory strategy, especially as color vision became important in detecting social signals, the role of the vomeronasal system became redundant. If these cues are also detected by the main olfactory system, and the visual system became more energetically expensive, loss of the vomeronasal system may be the result of an energetic rather than sensory trade-off.
