Interdisciplinary perspectives: Machine Listening & Listening Machines
April 27-28. Sussex Humanities Lab, University of Sussex, Brighton
Archives and Oral History
Listening to the dead
The traces of the past historians normally rely upon are made up of rotting print and fragile manuscript. But a proportion of this material represents speech uttered in known environments – courtrooms, churches and parliamentary chambers. And as these traces of the dead have been digitised and turned in to a new kind of ‘object of study’ (searchable, mash-upable, and macroscopable) we are increasingly challenged to analyse them in light of all the other forms of data that intersect with mere textual recordings This presentation briefly suggest that the addition of a quantifiable understanding of sound (reflecting historical spaces and environments) to a ‘big data’ approach to textual representations of historical speech, allows us to understand the meaning and import of that speech (and inherited text) in a fundamentally new way. It allows to listen to the dead, in hopes of hearing the timbre and rhythms of their words.
Describing and accessing sound recordings
Archivists have long been recipients and keepers of records, from scrolls through to hard copy memos and scribbled letters from the papers of individuals and institutions. Whilst traditional methods of cataloguing have served to describe these ‘traditional’ types of record, a century of technological development has presented archivists with new formats to preserve and describe, including sound recordings. This presentation will concentrate on the challenges and opportunities that are faced by archivists in describing and accessing sound recordings, looking to the opportunities that new technologies and methods may provide to open up sound archives.
Mining oral history collections using audio feature analysis
Recent workshops run by the Sussex Humanities Lab have sought to address an identified gap in the provision and use of audio feature analysis for spoken word collections. Traditionally, oral history methodologies and practices have placed emphasis on working with transcribed textual surrogates of recorded speech rather than the audio files created during the interview process. While practical considerations are responsible for the use of this approach, it inevitably misses a vast amount of potentially meaningful extra-semantic information which is present in the original audio records. Tools developed within the established field of Music Information Retrieval (MIR) inspire a possible means to work directly with the audio. I will outline the motivations for this work and the results of our workshops thus far.
Technical - Machine listening & Learning
Computers listening to birds
The sounds that birds make are a fascinating challenge for algorithmic listening. Often there is evidence of much complexity, yet how little we understand about the content and purpose of each individual sound. (Whereas with human sounds, we fool ourselves into thinking that we do understand the content and purpose.) I will outline the ways in which we have developed machine listening methods adapted to specific aspects of bird vocalisation - its fine details and its temporal structure - and how the challenge relates to the wider field of machine listening.
Ecoacoustics and the need for distant and close listening tools
Within the emerging scientific field of ecoacoustics the acoustic environment is understood as both a resource for the local community of vocalising creatures, and a potential source of information about their interactions and ecological status. This perspective motivates new theoretical frameworks as well as practical approaches to monitoring and conservation; ecologists dream of a meshwork of listening machines hanging in trees to monitor the health of forests globally. I will outline some of the work we have been doing to validate listening algorithms suitable for this task and highlight issues with inferring ecological meaning from statistical modelling. I speculate how other compositional methods might be coopted to help us in this context – humanising algorithmic listening by listening – and ask whether we need to take greater care in the interpretation of machine listening and learning algorithms more generally.
Auditory perception and attention and computational arts
I will present a cursory overview of fMRI and EEG literature relating to auditory perception and attention mechanisms, behavioral science of auditory attention, and detail some computational investigations for understanding audio within a computational arts practice.
Critical - Music & Media
Tuning (with/ to/ by) exosomatic organs
The Old English term hlysnan designates listening, hearing, and paying attention, and musicking history abounds in technical artefacts designed to extend our listening abilities. These exosomatic organs (Robert Innis) augment human expressivity, soliciting their creative users and audiences by employing and deploying algorithmic functions – sets of rules or processes. Drawing on music history, I will argue that cultural tuning is key to humanising emerging algorithmic listening practices.
Listening to Algorhythmics
The aim is to provide a probably diametrically opposed approach to “machine listening“ via a media archaeological inquiry into algorhythmic listening in the era between 1940–1965, where mainframe machine operators and scientists were listening to their computing machinery. I will provide some further implementations for the digital humanities context.
Listening is not listening, content is not content
In spite of continuous efforts to describe the relationship between listening, content, and meaning in music - a stable and widely applicable account remains elusive. Each lacking the integrity or stability to reach much further than a description of the context of the breath which utters them. My interest lies in how interpretations of machine agency / listening interact with, and feed-back into, how we describe and construct meaningful human practices. How metaphors of intentional human behaviour are constructed through the complex network of system outputs, user experiences, and system author descriptions (Jichen Zhu). Leaning on actor-network theory and embodied cognition, this talk questions the usefulness of these metaphors, and examines the heuristic relationships between content, context, and meaning that constitute current machine-listening practices. Envisioning a speculative, explicitly ‘multiscale’ and ecological approach to machine listening - one which questions the ethics of outsourcing the practices of categorisation and group formation to the computational.
Philosophical - Ethical, Analytical & Speculative
Listening with machines that are already ‘listening’? From Augmentation to Automation
What does it mean to listen with or through machines that are already ‘listening’? I engage with this question by examining the anthropomorphism that might be implicit in the notion of ‘listening algorithms’. This will help me to argue for the difference between functions and processes of ‘augmentation’ on the one hand, and of ‘automation’ on the other. I will argue that whilst augmentation implies the extension and exteriorisation of predefined forms or modes of behaviour, contemporary developments in computational automation ask us instead to consider the possibility to move beyond a simulative paradigm or phenomenological analogies.
[removed at request of speaker]
What would it be for a robot to sing?
Drawing on my experiences from investigating this question in the context of a Nao robot, I outline a number of relevant dimensions such as dis/embodiment, playback vs. synthesis, skill, and the potential for lyricism. I focus on the role that a singing robot’s own listening capability can/should play in its performance and/or acquisition of singing skills.
Being heard and remembered: Technical mediation and what it means to be human
The technical mediation approach in the philosophy of technology considers that technology is more than just an external factor that can support and hinder human life. Technology is constitutive for human self-understanding and for what it means to be human. What does this mean for an ethics of technology? Must we learn to take care of our hybrid selves? For this occasion, I will especially refer to the impact of recording and transporting speech.