Using music to replace vision

Sensory substitution devices (SSDs) do exactly what their name implies, they substitute one type of sensory input for another. Why would anyone wish to do this? Well, if you were blind and could substitute sound for visual information, you might have an easier time navigating your world.

Researchers from The Hebrew University of Jerusalem developed a new SSD, dubbed EyeMusic, which can translate screen images into musical sounds. The height of a pixel is indicated by pitch (pixels at the top of the image are represented by higher pitched notes) and the horizontal position of the pixel is indicated by the timing of that note. Meanwhile, the color of the pixel is indicated by the type of musical instrument playing the note (for example, piano for white and marimba for blue).

When the system was tested on blindfolded, sighted volunteers, they were able to track the location of squares on a screen just by using EyeMusic. They used the auditory information from EyeMusic to make a visual map.

Of course, the goal isn’t to help blind people play pong. It’s to provide a portable device that blind people can use in place of the visual sense they’re missing.

  

 Image credit: Maxim Dupily, Amir Amedi, Shelly Levy Tzedek.

You can try your hand at EyeMusic here.
 

Levy-Tzedek S, Hanassy S, Abboud S, Maidenbaum S, & Amedi A (2012). Fast, accurate reaching movements with a visual-to-auditory sensory substitution device. Restorative neurology and neuroscience, 30 (4), 313-23 PMID: 22596353.