Shinsuke Shimojo

Integration Beyond Sensory Modalities

California Institute of Technology

Although brain localization and computational modularity of sensory modality are often claimed (sometimes even with regard to "innateness" of it), they tend to be overemphasized. Literature on neural plasticity of sensory systems in young animals and deprived humans, for example, is an abundant source for evidence against fixed brain localization and modularity. How much do we know, however, about perceptual integration across modalities in the normal adult humans? Our latest psychophysical findings also dispute localization/modularity in the strict sense. We have found that: (a) sounds can either improve, or degrade temporal order judgment in vision, depending on temporal relationship (1); (b) a sound (or a transient signal in visual or tactile modality) that is synchronized with visual crossing of two objects enhances bounce (collision) and suppresses streaming perception (2); and (c) a single flash can be seen as multiple flashes if it is accompanied with multiple sounds (3). These findings argue against the classical view of the human as a vision-dominated animal, and for crossmodal modulation at early cortical levels and transient signal as the key for perceptual integration. Our developmental study of the stream/bounce perception ((b) above) in human infants further indicates that infants first show behavioral evidence of adult-like categorical perception of a visual motion event modulated by a sound only after the age of 5-6 months, and that it is most likely constrained by development of spatial attention (4, 5). Perhaps the most classical view of crossmodal integration in the human was that it is primarily a vision-dominated animal. Due to a line of evidence indicating auditory modulation of vision in temporal domain, a modern "modality precision (appropriateness)" hypothesis was proposed. Our view yet differs from this; we believe that transient sensory signal is critical in crossmodal integration, regardless of whether the transient signal comes within the modality or across modalities (6).

(1) Scheier. C., Nijhawan, R., and Shimojo, S. (1999) Invest. Ophthal. Vis. Sci., 40(4), 4169.
(2) Watanabe, K. and Shimojo, S. (1998) Perception, 27, 1041-.
(3) Shams, L., Kamitani, Y. and Shimojo, S. (2000) Nature, 408, 788.
(4) Scheier, C., Lewkowicz, D., and Shimojo, S. (2000) Invest. Ophthal. Vis. Sci., 41(4), S327.
(5) Shimojo, S., Watanabe, K., and Scheier, C. In: Visual Attention and Cortical Circuits. J. Braun, C. Koch, and J. Davids (eds.), MIT Press (in press).
(6) Shimojo, S., and Shams, L. Sensory Modalities are Not Separate Modalities: Plasticity and Interaction Beyond Modalities. Current Opinion in Neurobiology, (in press).

Monday, June 5, 2023
About the Swartz Foundation...
The Swartz Foundation was established by Jerry Swartz (bio) in 1994 . . .
Follow us...
The Swartz Foundation is on Twitter: SwartzCompNeuro
2013 Stony Brook Mind/Brain Lecture - Michael Wigler, PhD
2012 Stony Brook Mind/Brain Lecture - John Donoghue
Sloan-Swartz Centers Annual Meeting 2011
2011 Stony Brook Mind/Brain Lecture - Allison J. Doupe
2011 Banbury Workshop
Sloan-Swartz Centers Annual Meeting 2010
2010 Stony Brook Mind/Brain Lecture
Sloan-Swartz Centers Annual Meeting 2009
Conference on Neural Dynamics
2009 Stony Brook Mind/Brain Lecture
Canonical Neural Computation, April 2009
2009 Banbury Workshop
Sloan-Swartz Centers Annual Meeting 2008
Theoretical and Experimental Approaches to Auditory and Visual Attention - Banbury 2008
Stony Brook Mind/Brain 2008: Patricia Smith Churchland, B. Phil. D
Sloan-Swartz Centers Annual Meeting 2007
New Frontiers In Studies Of Nonconscious Processing - Banbury 2007
Stony Brook Mind/Brain 2007: Professor Michael Shadlen, MD, PhD
Multi-level Brain Modeling Workshop 2006
Sloan Swartz Centers Annual Meeting 2006
Banbury 2006: Computational Approaches to Cortical Functions
Stony Brook Mind/Brain 2006: Helen Fisher -- Lecture Videos
Sloan-Swartz Centers for Theoretical Neurobiology
Swartz Center for Computational Neuroscience
Banbury Center Workshop Series
Other Events                           Copyright © The Swartz Foundation 2023