Sonic Mnemonic image/svg+xml Sonic Mnemonic The McGurk effect   Sonic Mnemonic Christian Kroos, Tijs Duel, David Frohlich Centre for Vision, Speech and Signal ProcessingUniversity of Surrey   Interdisciplinary research project - Funded the UK's Engineering and Physical Sciences Research Council (EPSRC)- Hosted by the Centre for Vision, Speech and Signal Processing (CVSSP) of the University of Surrey (UK) and the Acoustics Research Centre at the University of Salford (UK)- Involved areas: Machine learning, signal processing, perceptual and cognitive psychology, design, media arts- Focusses on environmental sounds (not speech, not music)  On the engineering and science side - Detection and classification of sounds- Large data sets of recorded sounds with class labels- Digitised sound pressure measurements converted into spectrotemporal representations- Machine learning methods are used - to learn important features of the given classes from training set - to evaluate on separate test set- At present: Methods almost exclusively so-called Deep Learning: Artificial Neural Networks with multiple layers       On the impact and design side - Demonstrators that implement the new technology (e.g., 'searching for sounds' interface) and explore its potential use (e.g., user studies)- How does technology shape the experience?- How does design shape the experience?- Feedback: How will user expectation and experience steer technology & design - Key demonstrator: Development of a device facilitating acquisition and recall of sonic memories     Applications - Security surveillance- Home monitoring- Robotics- Support in media production- Noise source identification- Wildlife monitoring       Raises questions about: - the nature/properties of sound- the psychology of sound- the experience of sound- the recollection of sound This talk has to be more about the last question than its answer.       Physically, sound is: - Sound pressure variation over time- Waves propagating through suitable medium- Reflection, diffraction, damping, resonance- Time-frequency uncertainty (trade-off relationship)- Time series Hearing and the physiology/psychology of sound - Sound waves are transformed via the ossicles in the middle ear and the cochlear in the inner ear into neuronal activity - Temporal enfolding- Continuous processing and integration- Constantly relative incomplete (in time)- Psychoacoustics effect: A current sound might be obscured by a future sound (backward masking, e.g., Zwicker & Fastl, 1972) - Psycholinguistic effect: Watching a speaker’s face can change the auditory perception of a spoken syllable in the case of auditory-visual incongruence (McGurk & MacDonald, 1976)  Experience of sound - Substantial amount of literature (but not comparable to visual experience)- "Ephemerality dilemma" (Jonathan Sterne)- Most attention for the human voice (e.g., Derrida, Barthes)- Self-presence- Hearing one's own voice arguably more surprising than seeing one's own picture          Try to remember the sound of the aeroplane Or the sound of the black cockatoo Reminiscing using sound - Sentimental audio memory (SAM)- Recorder and player device- Player includes state- of-the-art deep learning-based classifier- User study with three families  Recollection of sound 1. Sound as memory cue2. Sound memory  Sound memory - How does the transient character impact human retrospection? - Auditory memory carries time itself within it ... really?- Does it require temporal expansion in the recollection? - Experiment -->   Did the recollection have an extension in time?Did you have to wait until a certain point?Did you have to 'listen'?What happened to the noises in the room? What did you actually recall? making sense of sounds
1
  1. Title
  2. Logo
  3. Project
  4. Carnaby's Black Cockatoos
  5. Yanchep
  6. Engineering
  7. Applications
  8. Impact and Design
  9. Questions
  10. Physics
  11. Hearing
  12. McGurk Effect
  13. Experiencing Sound
  14. Recollection
  15. Reminiscing
  16. SAM Recorder
  17. SAM Player
  18. User study
  19. Sound memory
  20. Areoplane
  21. Bird
  22. Recall
  23. Final