The Listen Reader

The Listen Reader is a personal interactive reading experience that combines the look and feel of a real book - a beautiful cover, paper pages and printed images and text - ith the rich, evocative quality of a movie soundtrack. It is designed to preserve and even heighten the experience of immersive reading. It explores the idea of multi-modal reading: the use of background sound to provide a sense of place, and to add affect to the experience of reading a book without interrupting the flow of the story.

The Listen Reader also preserves the beauty of the book as token object. The design uses no LCD or pixeled screen, but traditional, richly colored printed pages and a classic immersive reading environment: a comfortable chair, a polished hardwood swing-arm reading stand, a pool of light in an otherwise shadowed corner. The soundtrack is controlled by the motion of the reader's hands above the page; any motion within eight inches of the sensors is read as a volume, pitch, or pan control. Thus the natural gestures of the reader turning the pages provide a simple soundtrack; more complex, deliberate gesturing allows the reader to create richer soundscapes.

The technology is designed to be unobtrusively embedded in the pages and binding of the book. RFID (radio frequency identification) tags embedded in each page sense what page the book is open to; capacitive field sensors measure human proximity to the pages. Proximity measurements (hand gestures) control volume and other expressive parameters of the sounds associated with each page. In this instance, the page-ID data controls which set of sounds is presently being heard. The sound is multi-track and includes music and sound effects (but *not* a spoken version of the words on the page; that would interfere with the reader's own ability to absorb the text.)

You can track our progress as we built the Listen Reader by looking at the papers we wrote along the way. Here's the CHI'99 short paper on the first phases of the project; here'sthe UIST paper about the page-tracking technology; and here is the full paper from CHI 2001, with more detail on the technology and answers to some of the questions we noted in the first, short paper.

Credits

Maribeth Back and Jonathan Cohen

with Rich Gold, Steve Harrison, Dale MacDonald, Scott Minneman, Terry Murphy
Thanks to: Chronicle Books, Atlantic Records, Roy Want, Sharon Masnaghetti, Toby Sticpewich, and most particularly to Neal Stephenson for writing The Diamond Age which features the book we really want to build...

home research projects patents publications CV e-mail me	Listen Reader
	The Listen Reader is a personal interactive reading experience that combines the look and feel of a real book - a beautiful cover, paper pages and printed images and text - ith the rich, evocative quality of a movie soundtrack. It is designed to preserve and even heighten the experience of immersive reading. It explores the idea of multi-modal reading: the use of background sound to provide a sense of place, and to add affect to the experience of reading a book without interrupting the flow of the story. The Listen Reader also preserves the beauty of the book as token object. The design uses no LCD or pixeled screen, but traditional, richly colored printed pages and a classic immersive reading environment: a comfortable chair, a polished hardwood swing-arm reading stand, a pool of light in an otherwise shadowed corner. The soundtrack is controlled by the motion of the reader's hands above the page; any motion within eight inches of the sensors is read as a volume, pitch, or pan control. Thus the natural gestures of the reader turning the pages provide a simple soundtrack; more complex, deliberate gesturing allows the reader to create richer soundscapes. The technology is designed to be unobtrusively embedded in the pages and binding of the book. RFID (radio frequency identification) tags embedded in each page sense what page the book is open to; capacitive field sensors measure human proximity to the pages. Proximity measurements (hand gestures) control volume and other expressive parameters of the sounds associated with each page. In this instance, the page-ID data controls which set of sounds is presently being heard. The sound is multi-track and includes music and sound effects (but not a spoken version of the words on the page; that would interfere with the reader's own ability to absorb the text.) You can track our progress as we built the Listen Reader by looking at the papers we wrote along the way. Here's the CHI'99 short paper on the first phases of the project; here'sthe UIST paper about the page-tracking technology; and here is the full paper from CHI 2001, with more detail on the technology and answers to some of the questions we noted in the first, short paper. Credits Maribeth Back and Jonathan Cohen with Rich Gold, Steve Harrison, Dale MacDonald, Scott Minneman, Terry Murphy Thanks to: Chronicle Books, Atlantic Records, Roy Want, Sharon Masnaghetti, Toby Sticpewich, and most particularly to Neal Stephenson for writing The Diamond Age which features the book we really want to build...