Humanising Algorithmic Listening

A Sense of Being Listened To

Tue, 06 Mar 2018 00:00:00 +0000

Over the course of the three HAL workshops there has been discussion around the employment of algorithmic listening within a number of contexts. Some of these have focused on functional, utilitarian uses of algorithms, perhaps for the searching and categorising of databases, and others have been looking at more interpretive definitions of algorithmic listening grounded more generally within creative practice. Throughout these workshops discussions have focused on what listening actually is, be it by human or machine.

Following on from these workshops both Tom and I wanted to tease out what we were referring to as “the sense of being listened to”. I saw this as exemplified in the difference between simple voice control and conversation. In interacting with what are now everyday listening algorithms like SIRI we hear natural language give way to “keywordese”. The human speaker modulates their speaking to make it easy for the algorithim to “understand” what they want: “tea, earl grey, hot”. Then, generally after a pause, some audio response sometimes accompanied with screen based feedback, provides an indication of what was ‘understood’. Though this lurch to keyword speak is often a response to the perceived and sometimes real inability of the system to adequately respond to natural language it is also perhaps simply a shift toward increased efficiency. The system has no feelings. So just hit it up with keywords. No need for good mornings or pleasantries. Anyway, SIRI gives up if you don’t speak to it. It doesn’t care if you are listening to it or not.

This interaction is of course very far from that of conversation between humans: human conversation is characterised by a nuanced, on-going and often unconscious modulation by the speaker of their delivery based on the receivers nods, grunts, gaze, posture, or interruption. The sense of being listened to is central to the activity.

In a musical sense, Di Scipio writes about how listening to a concert can alter a sense of place, even after the music has stopped there is an irrevocable change in the environment. He describes how each new performance provides an ‘experiential audible trace of the meeting of human, machine, and environment’ (2002 p 26)¹ or as he puts it later in the article:

Something is welcomed, something is listened to. The one that welcomes is changed a bit after the one that was welcomed has gone. The one that is welcomed is also changed, as it too is one that listens and welcomes.’ (Di Scipio 2002, p.27)

For Di Scipio listening can be thought of as a process that is both reflexive and relational, a process that changes the one that is listened to as well as the one that is listening. We are interested in a similarly reflexive relationship between man and machine. If we can describe our relationships with listening algorithms as social encounters, we can consider them as situations of interaction that change us and the way in which we perceive the world. However, do these interactions also change the algorithmic system? In order for them to be truly relational, the one that is listened to and the one that listens should both be affected by the encounter with the other.

Bringing this in to the context of performance systems, we are interested in thinking about how the system shows it is listening and how that showing changes the human performer’s actions.

Feral Cello Phones Home

In particular Tom has been working on a performance system called the Feral Cello that reconfigures the sound world of an actuated cello live in the performance through a process of machine listening and digital signal processing. Pickups on the cello’s body are fed to a Max patch that analyses the sound for certain pre-recorded sonic gestures. When these gestures are ‘heard’ by the system the Cello switches between different DSP states, effecting the acoustic response of the cello. These gestures are pre-determined by the performer but due to variations in performance and listening errors the system are not 100% predictable. This leads to quite a different performance scenario for the instrumentalist who has to deal with the difficulties of performing with an acoustic system that is constantly being reconfigured live in the moment of performance. Tom would like to extend the listening algorithm in this system such that it has a sense that it is being listened to. In particular he is interested in how can we give our machines a sense of ‘occasion’ that is broader than mere feature extraction.

In performance scenarios, can we develop listening algorithms that are aware of their performance contexts, that respond differently depending on criteria such as the size or the atmosphere of their location, the ‘feel’ of the audience?

Can we give a listening algorithm a sense of being listened to? What would happen if the algorithm were to develop stage fright or performance anxiety?

How does a sense of being ‘listened to’ by algorithms effect the participation of the other performers in this context? How would this affect the human performer who is performing with the system?

These are some of the questions that we would like to explore.

What happens when the listening algorithm’s personality interrupts its ability to pay attention? When the pressures of continuous listening get too much? When anxieties around interaction with strangers give rise to a flight response? Or when the algorithmic agent draws close to the speaker to listen more intently?

Some initial prototyping:

Over two days in November we sought to move from these questions toward some new work. We decided to eschew the digital entirely, at least temporarily, thereby indulging a contrarian desire to park the actual algorithm. In this age of Deep learning, machine learning, AI and the implied digital computational flavour of all things algorithmic we wanted to begin with an all analogue approach for our listening machine.

Pre listening tape making thinking

We’ve started by playing with tape and begun development on our tape based listener, a device that travels forward and back along a strand of magnetic tape alternately listening (recording to tape) and speaking (playing back from tape). To listen or speak it must move. The raising and lowering of one end of the tape is used to accomplish this. We envisage a rake of these listeners installed in a room. This is some form of surveillance. But out in the open. A congregation of daft machines that feel obliged to record your utterings, and their own, and then warble them back, sometimes in reverse. Comical machine listening implemented by a real stupidity. A flock of mad listening blurting machines some of which try to hide away from humans. Running along their tape. Noting the sounds of each other and the humans in the rooms or singing out earlier secrets captured.

Moving speaking listening tapehead prototype #1

In this enquiry we aren’t moralising as to how we should talk nicely to machines. We are more interested in how playing with algorithmic listening might open up fertile avenues of exploration with regard to human relations.

To be continued…

Augustion Di Scipio (2002). Systems of embers, dust, and clouds: Observations after Xenakis and Brün. Computer Music Journal, 26(1), pp.22-32. ↩

Oi! Algorithm, chew on this!

Sun, 04 Feb 2018 00:00:00 +0000

Assaying The Noise Between Human And Algorithm

Over the course of the first two HAL meetings, as we collectively mapped out the territory of the research network, certain flavours of question seemed to crop up that sought to stabilise what might be meant by human or algorithm (or, indeed, listening), and others that wondered at the kind(s) of role that certain disciplinary approaches could play in this combined effort. Our schtick in this project is to investigate what sort of contribution practical arts research can make by approaching these putatively ontological concerns about humans and algorithms as a matters that are inherently unstable.

Ontological Noise

Our premise is that these kinds of ontological question simply don’t admit stable answers because what it is to be human, and what it is to be algorithmic are to a great extent co-indexical: that is, our (historically, culturally located) understandings of what it is to be an algorithm inflect our similarly situated understandings of what it is to be human and vice versa. In really complicated ways. Of course, this stance is hardly novel. It has a great deal in common with the perspectives of much of Science and Technology Studies (e.g. Barad, Latour, Suchman), and with certain philosophies of technology (e.g. Feenberg). Feenberg, for instance, makes precisely this argument:

The stakes in this debate over artificial intelligence are not merely technical. If we understand computers rationalistically, as automata, we prepare a revised self-understanding along the same lines. People become information processors and decision makers, rather than participants in shared communicative activity. ¹

We proposed to map some of this territory through a series of design provocations and a large portfolio of small collaborative makes, which are assembled in performance or installation, and critically reflected upon in the light of the concerns of the Network. Insofar as algorithms are typically conceived as trying to identify order against a background of noise, our attempts to find our way around the territory between human and algorithmic listening destabilise that concern by:

Providing challenging input

oi, you algorithm, with your norms as to what counts as music…chew on this!
Taking noise to be its own kind of information

oi, you algorithm, you think that is signal…chew on this!
Making noise-‐fuelled reductiones ad absurdum of soi-disant high-level algorithms

oi, you algorithm, you think that is emotion…chew on this!
Using algorithms intended to bring a certain kind of order to usher in disorder and discomfort

oi, you algorithm, you want us to index massive arrays of online music… listen to these clicks!).

Methods

We collaborated over the course of three sessions of intensive practice-based work, two located in Huddersfield and one in Newcastle. Our strategy was to follow the character of Bowers, Bowen and Shaw’s (2016) ‘Many Makings’²: a large number of small collaborative makes were created in response to our four noisy destabilisations. A key aspect of this approach is to remain alert to the polysemy of making:

There can be many makings. Of things, problematisations, identities, interests, ecologies, infrastructures, portfolios, federations (Bowers, Bowen & Shaw 2016 p.1255)

Examples of algorithmic images from the series of generated by JB for the project

Makes

We sought to explore how ‘noise’ might provide challenging input to algorithmic listening techniques or make for a desirable, divergent output (in all the varied senses of that word, (see Marie Thompson’s Beyond Unwanted Sound: Noise, Affect and Aesthetic Moralism ³). We sought to misappropriate known techniques to uncover their limitations or the implicit assumptions built into them. Independently, we each brainstormed proposals for makes. Combined we had a long list of 48, some expressed compactly, some at greater length, some in a standard ‘scientific’ language, some deliberately written humorously or facetiously, some with a degree of overlap and convergence with other proposals, some unique, some making reference to existing artworks but bending them to our context of interest, and so forth.

We made work in two concerted sessions of two days duration each, one at each of our host institutions. We worked with a light touch doing just enough to prove the principle of our design ideas before moving on to the next. We were drawn to prioritise proposals that we both shared but we ensured that our individual idiosyncrasies were also represented to maximise the coverage of our work. We conducted a third two day session to combine our makes in a performable installation environment. In total, 18 of our proposals were made to some degree with 14 having a role in the final presentation of the work.

Examples of algorithmic images from the series of generated by JB for the project

AntiGate: Amplitude Version

The input signal is amplitude envelope-followed. When the signal drops below a given threshold, it is let through the gate, thereby performing the opposite action to a classic noise gate. At the moment the signal drops below threshold, it is subject to single frame FFT analysis which is used to create a freeze effect that is held until the next time the gate opens. The sound through the open gate and the frozen spectral texture can be cross-faded. The cross-fade and the threshold are both variable in performance.

AntiGate: Spectral Version

In the Spectral AntiGate (SAG), a carefully engineered multi-resolution spectral gate, made by Harker to showcase his new FrameLib signal processing framework, is hijacked by simply reversing the inequality at its core. Being multi-resolution means that the chirping redolent of crude spectral processing is mitigated somewhat, particularly in higher frequencies that retain a degree of texture. If a feedback loop is set up with an air microphone picking up SAG’s output, a steady cycle is settled into that alternates between more chirpy mid-frequencies and bursts of higher frequency noise, although the inner textures of these components do vary. This behaviour is oddly reminiscent of the change ringing of bells. The rhythmic behaviour changes if a player manipulates the microphone, for instance by shielding the microphone.

Room Tone Shift Register

Bursts of filtered room-tone are fired back into the space with an attack-release energy profile. The timing of these bursts is dictated by a maximum-length pseudo-random sequence (see our discussion of LFSRs below) with four ‘voices’ each with different length sequences and occupying different spectral bands. The room tone is read from a 10-second delay line, so depending on the delay time, there is the possibility of sampling previous output. The overall effect depends to a large degree on how fast the sequencers are driven. High speeds and short bursts produce an impulsive kind of texture, moderate speeds a more rhythmic feel, and low speeds with long bursts can occasionally punctuate whatever else is happening in the space with dramatic impact sounds.

Electrical Field Re-synthesiser

An inductive coil (sometimes known as a phone tap coil) is used to transduce electromagnetic fluctuation into a signal that is presented to the EFR which tries to model its input as coloured noise. This is done using a conventional source-filter technique where noise is filtered in the Fourier domain by a spectral envelope derived by cepstral liftering of the input. This is supplemented with a very simple, single-voice sinusoidal model driven by the sigmund~ external to the Max language. The character of the resynthesis is largely determined by the degree of liftering and, of course, how much sense it makes to model the input with filtered noise in the first place. In variants of the EFR, a microphone has substituted the inductive coil.

Disagreeing Pitch Trackers One

A signal is ‘resynthesised’ by sine oscillators driven by three different pitch trackers in Max (sigmund~, zsa.fund and the built-in Fzero~). One can also mix in another signal of oscillators driven by the difference in frequency between each of the three trackers, ring-modulated with each other. Driven with a pitched signal and a sensible gain structure, the effect is rather like six excited slide whistles. However, introducing feedback and nonlinearity opens up a much wider range of territory. If left in a feedback loop with a suitably large delay (we use 12 seconds here), DPT1 can settle into a quite diverse range of states, especially if there is clipping or distortion somewhere in the loop. Smoothing and delaying the frequency inputs of the oscillators by different amounts can also enrich the emerging dynamics.

Disagreeing Pitch Trackers Two

A similar approach to pitch tracking and making the disagreement in results between algorithms palpable was made using the Pd-vanilla language, The sigmund~ and fiddle~ objects are used to identify pitches in the input and to set the frequencies and amplitude envelopes of two sine waves. These signals are also ring modulated to enhance the perceptibility of their disagreement. The performer can cross-fade between the sine waves and their ring modulation. The identified pitches and their absolute difference as a disagreement measure are made available from DPT2 to other patches (e.g. to parameterise the LFSR, see below).

Eternal Resonance Machine

The ERM is a means for converting any input into a sustained noise texture. On receipt of a button press style event, the momentary spectrum of the sound is subject to an 4096-band FFT and used to synthesise a sustained frozen noise. Successive button presses will add partials to the sustained sound if their FFT bands are louder than in the last analysis. Button presses will also momentarily open a gate to pass the input sound to Pd’s freeverb~ set to a large room size with little damping. When the gate closes, the reverb is frozen to give an infinite reverb effect. This gives an alternative way to synthesise a spectral noise from input sound. The performer can cross-fade between the two methods and reset the analysis (which fades both kinds of noise to silence).

Linear Feedback Shift Register Sequencer-Synthesizer

An 8-bit linear feedback shift register (LFSR) was implemented in Pd-vanilla. A flexible design was adopted where the last bit could feedback to any of the 8 positions in the register for exclusive-OR combination with the position’s contents. This creates an algorithmic system which can generate a variety of behaviour from the digital pseudo-noises of maximal length sequences to varied periodic behaviour. The values in the register were interpreted both as a 8-bit sample values to be read into a wavetable and as 8-bit specifications of frequency with which the wavetable (or a sine or a square wave) would be played. The rate at which the LFSR is clocked and the centre and range values of frequency could be determined manually or received from other processes (e.g. the Disagreeing Pitch Trackers). In this way, pseudo-noises or pitched sequences could be generated which followed identified profiles.

Emotion Recognizer-Generators

We reversed-engineered a music-psychological study that aims to demonstrate a mapping between given musical ‘features’ (timbre, tempo, mode, register, articulation, dynamics) and ‘emotions’ on the basis of rating judgments given by listeners to various transformations of simple melodies. Working in parallel, we each independently came up with ways of trying to estimate these six features from an audio stream. Then, using the paper’s experimentally derived table of correlations between features and emotions, we constructed a mapping function between ‘features’ and the four ‘emotions’ examined in the paper (happy, sad, scary, peaceful). We then set about using this mapping for generative purposes. One of us made a noise/drone generator, which constructed a spectrum based on a shifting histogram of detected pitch classes that was modulated using the detected emotions and features. The other of us made a melody generator which, on the basis of the emotions recognised in the input audio stream, estimated values for the six musical features analysed in the study and played back notes synthesised with enveloped, filtered sawtooth waves.

Random Sample and Holding

The instantaneous digitised value of an input audio stream is sampled at random intervals and read into a wavetable, the insertion point wrapping round when the table is full. Following a fractal expansion technique used previously by JB, the wavetable is read to generate long patterns of nested amplitude modulated sound. The reference rate for reading the wavetable can be set as a linear function of the currently sampled value or from other pitch tracking processes. The range of the random sampling intervals can be set in performance. The output can vary from a noisy reconstruction of the input through a slow pattern which can variably follow the pitch content of the input to a distorted granular-sounding stream.

Arduino Nano Circuit Noise

The analog-in values from an Arduino Nano are read into wavetables and used for direct digital synthesis via nested amplitude modulation as described in the previous section. The analog terminals are left floating so they are sensitive in unpredictable and interactive ways to touch and circuit noise. This creates a lively five oscillator digital synthesizer capable of a range of distorted, bit-reduced and granular-sounding textures which can be steered by touch but not precisely played. Two improvisations were recorded and used by us in performance as a fixed media element.

Schlechtmusik

In recognition of the prominence that Mozart’s music has in the history of algorithmic composition and machine listening, we took a recording his Eine kleine Nachtsmusik and extracted its tonal component using Izotope RX. We followed this with a sinusoidal analysis using Spear and made various resynthesises. For example, we made a version which was reconstructed out of banks of sine waves, another which retained only the transients and yet another in which the tonal analysis was read at a slow rate to generate a 45 minute texture. To explore how machine listening techniques might react to suboptimal renderings, we also degraded the original recording by playing it back in a reverberant space, freely talking over it and recording the result using a gain structure with a tendency to distort. We selected five versions plus the original and mixed them using a good to bad (Nacht- to Schlecht-musik) crossfader. We informally calibrated the crossfader so that at extreme good/Nacht the online music recognition service Shazam would accurately recognise Eine kleine Nachtsmusik while at extreme bad/Schlecht no results were returned, with an approximately 50% hit rate in the middle.

Sincere Resynthesis, Subsequently Violated

Using sigmund~ feeding an oscillator bank with a generous number of partials (100), we found that a reasonable facsimile of even a noisy environment could be rendered, but that it was a simple matter to reduce this to a sludge of artefacts by over-smoothing frequency and / or amplitude tracks. The degree of over-smoothing was made a function of the distribution of averaged spectral centroid in the space by building a histogram (periodically cleared), that was occasionally sampled as if it were a PDF and used to set the amount of smoothing.

I Am Sitting in Skype’s Audio Compression Algorithm

Following the same principle as Alvin Lucier’s I am Sitting in a Room, a prepared text was read by one of us and recirculated through Skype until its original identity had completely dissipated. This was roughly 30 iterations. In contrast to the shifting resonances of Lucier’s acoustic version, the accumulating artefacts included bursts of noise and clicks, and the appearance of a distinctive crescendo of bass-drum-like impact sounds partway through, as well as the chirpy filtering we had expected. Our text was from a deeply critical review of Abraham Moles’ Information Theory and Esthetic Perception, and the results formed a fixed-media component of the final presentation.

Re-De-Reverberation

Using a black-box de-reverberation plugin and a reverberation pedal, we constructed a controllable feedback loop, stimulated with chirps, noise bursts and crackles programmed in Pd-vanilla, and recorded a short, two-person improvisation that was used as a source of fixed material in our presentation. One feature of the plugin is that reverberant components can be boosted as well as suppressed using a ‘focus’ parameter, and that it is easy to mistune the settings to generate plenty of artefacts. The resulting material had a drone-like character but did not tend to collapse into indistinct mush.

Miscellaneous Makes

We made a number of other explorations which we will only briefly relate here. Many of these concern processing fixed media material using offline processes or involve recordings that for reasons of practicality could only be appear in our work as fixed media. For example, one of us created a piece entitled Maximum Zero which takes a recording of David Tudor performing John Cage’s 4’ 33” and subjects it to brickwall limiting to bring out the environmental sounds around the performance at maximum intensity. One of us also made recordings using the aerial array and amplifier designed by NASA’s Radio Jove to bring recordings of the radio transmissions of Jupiter to our project. We also made experiments to see whether we could transmit the results of our machine listening analyses via non-standard means. This included an encoding of identified pitches as audible Morse messages, which we decoded and played back in a feedback loop. In this way, we sought to corrupt conventional understandings of the relationship between representation and the represented and between signal and noise.

Performing our Work

The fruits of our labours were assembled together and explored in the University of Huddersfield’s multichannel Spatialisation and Interactive Research Lab (SPIRAL), which offers 25.4 channels to work in, arranged as three tiered rings of eight, plus a ceiling mounted speaker dubbed the voice of god. A binaural dummy head was used as the input for all listening process, which we dubbed Stookie Helen (people who attended the second HAL meeting in Belfast will have already encountered JB’s partner, Stookie John).

JB and Stookie Helen have some quiet time together

Different processes were placed in different speakers, and kept stationary. In this way, the character of what emerged was driven in part by the interactions of the different processes, affected by the relative gain structure, over which we had control.

This allowed more or less scrutable relationships to emerge between processes as they interfered with each other, and it also encouraged visitors to explore the space, and discover different points of focus. We had some control over each process, in the form of individual gain faders, ‘nudge’ buttons which would push an individual process into a new (possibly random) state, and a combined overall control on a boundless rotary encoder that would affect all processes. This combined control yielded 24 separate control signals internally, based on a set of transfer functions. Processes were free to use whichever of these we fancied, however we wished, the object being to generate variety with coherence. For example, processes like the crossfades on the Anti-Gates or the ERM could set by the values from the transfer functions.

Under the provisional title of All The Noises, our work was first presented on 18th January 2018, and occupied territory between a performance, an installation and a research presentation. We started with a brief, 15 minute, performance whilst an audience composed of colleagues from Huddersfield and members of the public responding to local publicity arrived and explored the space. We then set the system into a lower-key state whilst we explained our project to the room at large. Thereafter, we had a steady trickle of guests passing through and we would alternate between talking, nudging the system, and demonstrating brief performative moves. Finally, we concluded the session with a 10 minute performance crescendo.

Here are edited highlights:

Oi Algorithm Performance, Huddersfield 18 January 2018

OG and JB, tearing it up

Closing Thoughts

Going into this we had a few ambitions: one was to use the many makings approach to try and sketch out an approach for errantly-inclined, artistic algorithmic listening research that complements and challenges to the engineering orthodoxy. Our thoughts on this have been submitted to NIME 2018, so we hope to be pontificating on this topic in public later this year. An other ambition was, straightforwardly, to collaborate, as we hadn’t done so before despite having been in each other’s orbit for a while. We’re encouraged by what we made, and intend to keep refining and gigging it.

Acknowledgement

As well as the support of HAL in making this possible, OG’s time and access to facilities are supported by the ERC through the Fluid Corpus Manipulation project.

Andrew Feenberg (2002). Transforming Technology: A Critical Theory Revisited. Oxford University Press, p. 106 ↩
Bowers, John, Simon Bowen, and Tim Shaw (2016) “Many makings: Entangling publics, participation and things in a complex collaborative context.” Proceedings of the 2016 ACM Conference on Designing Interactive Systems. ACM. ↩
Marie Thompson (2017) Beyond Unwanted Sound: Noise, Affect and Aesthetic Moralism. London: Bloomsbury ↩

Feature Perceptualisation Challenges

Fri, 08 Sep 2017 00:00:00 +0000

Perceptualising audio features for working with big audio data

Across arts, humanities and sciences, researchers are recording and collecting repositories of audio archives well beyond listenable compass. Whilst we can in theory apply machine listening and learning methods to search for salient patterns in, and make sense of this big audio data, much is lost in not being able to engage perceptually with the raw audio. We need new ways to ‘get to know’ audio archives, other than through remote statistical probing for many reasons: to verify data integrity (is it blank, is it distorted etc), to aid cataloging and search (is it music, spoken or environmental recording?) or in scientific domains, to help interpret the models we build from it.

One approach, already being explored in the context of soundscape ecology, is the false-colour spectrogram. Soundscape ecology is a rapidly growing field in which the potential for acoustic analyses to address previously hard-to-reach questions is being explored. Researchers globally are amassing vast audio archives using remote, schedulable devices to record terabytes of environmental soundscapes. Patterns of ecological interest often occur over weeks, months, or even years, involving many terabytes of data, only a fraction of which can ever be listened to.

Michael Towsey’s group in QUT have been experimenting with false-colour spectrograms for visualisation of long-form recordings¹.These are essentially visualisations of audio features, calculated on short audio recordings (typically 1 minute). Three indices known to capture distinct characteristics (ie orthogonal) are mapped to RGB values and displayed in spectrogram-like format, enabling spectro-temporal info over full days to be viewed at a glance (Fig 1)

Fig 1. Example of a false-colour spectrogram from Towsey et al 2014. This is derived from a combination of normalized spectrograms for ACI, 1-H[t] and CVR. The vertical gridlines are at one hour intervals, starting and ending at midnight. The horizontal gridlines are at 1 kHz intervals.

By calculating the indices over full frequency range and representing three indices as RGB pixels values, diurnal patterns of weeks and months, even years can be viewed, in an “Extended Acoustic Summary Image” (Fig 2)

Fig 2. Example of a Extended Acoustic Image from Towsey et al 2014. An Extended Acoustic Summary image for the months March to October, 2013 Each pixel RGB value represents values for three acoustic indices for a 1 minute recording

The false-colour spectrogram will be as informative as the indices it is generated from – and this will vary task to task. This approach is promising for time-series data from a single point, but many other possibilities exist, including interactive resynthesis of audio features, for example.

Challenge

(How) can feature visualisation – or sonification/ resythnesis – provide a meaningful perceptual summary of large audio archives? Can ecologically relevant features be distinguished - such as animal sounds (biophony), weather (geophony) and machinery (technophony)? Can silent and distorted files be made apparent? Might interactive perceptualisation afford deeper insights?

Data

Two 3 hour dawn chorus recordings are available, made in different habitats at the same time. Each is segmented into 180 one minute mono wav files. The recordings start 90mins before sunrise, capturing the onset of the dawn chorus. Beside a bird chorus of increasing density, there are some sheep, various engines (planes, cars) and a thunder storm, followed by rain.
UK dawn chorus recordings on Fig Share

Audio features for gender-based conversational dynamics.

One of the many excruciating features of Trump’s presidential election campaign was his constant interruptions of Hilary Clinton during the presidential debate. And recent research published in the Journal of Social Sciences² reveals similar gender differences in interruptions in academic job talks. Automatic analysis of speaker characteristics, such as gender, would be a powerful tool in analysis of conversational dynamics in oral history, gender studies and numerous other humanities disciplines.

Challenge

Could machine listening in combination with supervised learning, or even unsupervised clustering be used to discriminate between voices in an interview in order to identify conversational dynamics?

Participants are invited to consider which audio features and/or machine learning methods might be best applied.

Trump vs Clinton Presidential Election Campaign 2016.

Data.

For exploration, an audio file of Trump’s Clinton interjections is available here

Resources/ Links

CataRT

Standalone: http://forumnet.ircam.fr/product/catart-standalone-en/

False Colour Spectrogram Sketch

Python implementation of Acoustic Indices for soundscape analysis https://github.com/sandoval31/Acoustic_Indices

Principle Latent Component Relationships

Memory Mosaic

App on Appstore
Video on Vimeo

Daphne Oram Browser

Video on Vimeo

Described in Mital, P. K., & Grierson, M. (2010) Mining Unlabeled Electronic Music Databases through 3D Interactive Visualization of Latent Component Relationships.

Towsey, Michael, Liang Zhang, Mark Cottman-Fields, Jason Wimmer, Jinglan Zhang, and Paul Roe. “Visualization of long-duration acoustic recordings of the environment.” Procedia Computer Science 29 (2014): 703-712. ↩
Blair-Loy, Mary, Laura E. Rogers, Daniela Glaser, Y. L. Wong, Danielle Abraham, and Pamela C. Cosman. “Gender in Engineering Departments: Are There Gender Differences in Interruptions of Academic Job Talks?.” Social Sciences 6, no. 1 (2017): 29. ↩

Improvisation, listening and distributed agency in human-machine musical ecosystems

Mon, 29 May 2017 00:00:00 +0000

I am first and foremost an improvising musician and instrument maker. I have a particular interested in exploring ways of developing and nurturing networks of human and non-human musical interactions, while also exploring how improvisation might be more broadly conceived as a skilled practice that transcends the disciplinary boundaries of music, promoting new approaches to creative decision-making, critical dialogue, risk-taking, and collaboration across diverse domains and levels of expertise.

So what does this have to do with Humanising Algorithmic Listening?

For me, this project is an opportunity to bring different disciplinary perspectives together to discuss how we might imagine the future of machine listening, where our possibly contradictory disciplinary understandings of the act of listening are allowed to occupy a tense but productive coexistence. The aim here is to better understanding the cultural, technical and ethical implications and possibilities of listening with machines in our research practices, as well as in our daily lives.

I would hazard to claim that some musicians might have a different practical understanding of working with technology, one which challenges the commonplace idea that devices (from a smart phone to a bespoke digital musical instrument) are merely transparent tools for achieving the goals of a user/performer. I’m aware that I and some of the other musicians in this network value the resistances of technologies while celebrating instability and chaos as resources that allow for the widening of possible interpretations and the emergence of unexpected behaviour, forcing us to adapt and providing new challenges that sometimes lead to new discoveries or new social relationships.

And while I suspect that researchers in computer science and the digital humanities might be challenged in interesting ways by coming into contact with the views and practices of improvising musicians, I equally suspect we musicians have at least as much to learn from engaging with these topics from the perspectives of other disciplines. My hopes for this network are that we will engage in what Judith Butler has described as a process of “cultural translation” footnote¹, whereby the seeming stability of our own understandings of human and machine listening will be reworked through a translational dialogue that might reveal what we have previously excluded from consideration, and which now might be reshaped through being faced with our own alterity.

For me, the challenge of our network is not only how to listen with machines that have been tuned by our own disciplinary motivations, but to learn how to listen together as human and machines across disciplines, as the challenges and opportunities we will face will continue to become more ubiquitous.

Practicing human-machine artistic collaboration

In a previous blog post, Alice suggests that: “Humanising Algorithmic Listening might mean experientially probing human-machine agency through speculative, experimental and performative investigations.” Likewise, David Kant stressed the need to artistically engage with existing machine perception systems with the aim of expanding rather than reifying our expectations. Together David and Alice’s posts have helped me reflect on my primary motivations for this network:

Humanising Algorithmic Listening might mean offering an alternative to the view that machine listening technologies are merely tools to help us achieve our predefined ends
Humanising Algorithmic Listening might mean speculative exploration of our human-machine relationships, while prioritising emergence and dialogue over control
Humanising Algorithmic Listening might mean learning anew what it means to listen together as humans.

And I believe strongly in the key role that artistic practices can play in our attempts to better understand these possibilities. My own attempts to artistically explore the collaborative potential of computer music technologies includes my ongoing Ambiguous Devices project with Tom Davis. Ambiguous Devices is a distributed musical ecosystem (Bowers ², Waters ³), a network of interconnected music-making machines, people and ideas. The project began in 2011 out of a mutual desire to explore non-linear and resistive forms of networked musical interactions in an attempt to challenge and extend our existing practices as improvisers and instrument makers. As the title suggests, we value “ambiguity as a resource” ⁴ which allows for the possibility to (re)constitute the dynamics of our musical ecosystem, a process I have described elsewhere as co-tuning ⁵ & forthcoming - see paulstapleton.net for updates.

Coda: why do I celebrate instability and adaptability over control?

While writing this blog post I found it interesting to revisit some notes I made in 2012, which were in part generated by reflecting on my early work with Ambiguous Devices.

For myself, falling is an inescapable part of performing improvised music: falling away from my own expectations; falling in love with the unknown; falling towards the other; free falling and then catching myself from falling.

In her performance of Walking & Falling ⁶, Laurie Anderson describes the continuous danger of losing one’s balance while walking, while embodying both the social address of the other and the risk of self-transformation in movement:

“I wanted you. And I was looking for you.
 But I couldn’t find you. 
I wanted you. And I was looking for you.
 But I couldn’t find you. I couldn’t find you.
You’re walking.
And you don’t always realize it, but you’re always falling.
With each step, you fall forward slightly  and then catch yourself from falling.
Over and over, you’re falling.  And then catching yourself from falling. 
And this is how you can be walking and falling at the same time.”

The prosaic fact that falling is a necessary part of bipedal locomotion is hardly what is communicated. As media theorist Sibylle Moser ⁷ has claimed: “By turning language into a sound gesture, the piece explores the interdependence of movement, perception and conceptual interpretation.” Anderson’s patiently timed voice marks the form of her motion across a cyclical electronic soundscape, bringing into sharp relief what musicologist Ainhoa Claver ⁸ has described as ‘the simultaneous presence and absence of ourselves in the course of our events.’

It is through this looking, or listening, while falling that the improvising musician shapes her discipline and her self. It is a response, in gender theorist Judith Butler’s ⁹ words: “to be addressed, claimed, bound to what is not me, but also to be moved, to be prompted to act, to address myself elsewhere, and so to vacate the self-sufficient ‘I’ as a kind of possession.” This is not a poetic metaphor but a real risk that is demanded in performance, be it the performance of gender or music.

In this network, I would like to further explore how improvising with machine perception systems can offer the possibility to transgress established personal and cultural identities; how our stories remain the same and how they change; how we reinvent ourselves in new listening situations, walking and falling at the same time.

Judith Butler, Ernesto Laclau, and Slavoj Žižek, 2000. Contingency, hegemony, universality: Contemporary dialogues on the left. ↩
John Bowers, 2002. Improvising Machines: Ethnographically Informed Design for Improvised Electro-acoustic Music. Masters in Music Dissertation, University of East Anglia, Norwich. ↩
Simon Waters, 2007. “Performance Ecosystems: Ecological approaches to musical interaction.” EMS: Electroacoustic Music Studies Network, pp1-20. ↩
William Gaver, Jacob Beaver & Steve Benford, 2003. Ambiguity as a resource for design. In Proceedings of the SIGCHI conference on Human factors in computing systems, pp233-240. ↩
Paul Stapleton, Simon Waters, Nick Ward, and Owen Green, 2016. “Distributed Agency in Performance” in Proceedings of the International Conference on Live Interfaces, University of Sussex, pp329-330. ↩
Laurie Anderson, United States, 1981 and Big Science, 1982. ↩
Sibylle Moser, 2008. “Walking and Falling” Language as Media Embodied, in Constructivist Foundations 3:3, p262. ↩
Ainhoa Kaiero Claver, 2010. Technological fiction, recorded time and ‘replicants’ in the concerts of Laurie Anderson, in Trans. Revista Transcultural de Música 14, p10. ↩
Judith Butler, 2005. Giving an Acount of Oneself, p.136. ↩

What (experimental media) design could do for humanising algorithmic listening

Thu, 04 May 2017 00:00:00 +0000

Disclaimer: Please read the following keeping in mind that I am not trained in signal processing, data mining nor have an engineering background. The concepts I use need complementation by more knowledgeable persons.

I decided to write up my thoughts right after the great first workshop (which I really enjoyed) and before they fade away. Hopefully I can clarify some of my arguments here; perhaps we can also begin to sketch some common fields of problems for all network participants. As a media archaeologist turned non-affirmative design researcher with a background in improvised laptop music I might care at least for some of the overlapping fields. My remarks resonate mostly with the aspect of „Unpacking black boxes through listening“ Alice sketched out in her introductory blog post.

Standardized Patient Program examining the abdomen, Photo by Steve Perrin CC BY 2.0

Historically considered, listening has been a crucial method of unpacking boxes or enclosed bodies even before literally unpacking it, but by trying to listen to the sounds coming from inside the body both passively with a listening device or even actively by gently knocking it. You can do this with a variety of artefacts, products and found things such as stones, water melons, cheese, earth grounds, human bodies, cars and of course with all sorts of percussive instruments. In all these examples there is a direct epistemic relationship (tight coupling) between the sound created and the characteristics and structure of the body under inquiry. Since the dawn of transducers such as the telephone in the late 1870s, we can listen into electrical signals such as those of muscles or nerves, and at the same time transform electrical signals into sound. And with the dawn of the vacuum tube amplification in the 1910s, we can amplify tiny signals, such as those of brain neurons.

Following the links Alice posted and through some of the more informal discussions I had during the workshop I guess that filtering is a key aspect of machine or algorithmic listening. Filtering here is not only meant in a more metaphorical meaning as a way to structure incoming data and material (thus as a common element to all data analysis) but literally, thus more media archaeologically, as procedures in signal processing. The digitisation and quantification of analog electroacoustic signals into digital data for example is some sort of filtering, also the hand-crafted filtering of essential structures in audio feature extraction, which is, as I learned during the workshop, is often the first step of algorithm-based audio analysis (aka machine listening) before filtering them further with classification, regression and other algorithms as done in data mining and machine learning. This „hand-crafted“ and skilled practice for me is of particular interest: Humanising Algorithmic Listening might mean working through practices of filtering with a combination of ethnographic/STS, historical, critical and practice-based methods (both engineering and design/arts).

Following questions are asked to cause some response: Who decides how, when and why about the value of a certain feature and what kind of media and cultural techniques (visualization, drawing, singing, computing) are involved in the practices of hand-crafted audio feature extraction? Are there aesthetic and designerly decisions? What are the protocols within such a field of practice? Unboxing and unpacking machine or algorithmic listening means for me to critically inquire each step, each of its module in the pipe line of feature extraction, classification, segmentation and visualization. Sonification, audification and visualization of the filtering steps, processes and results are then also helpful in unboxing the more advanced data-driven feature extractors using neural networks as done by Keunwoo Choi currently a PhD Student at Queen Mary University of London.

Is there space for design-based experimentation for example in varying the modes of visualization or for auralisation/ audification? Can we listen to the algorithms filtering the data as sonification, not of the data but of the processes that generate data? Can we design human interfaces for the modules in the data analysis pipeline? Would that be helpful for getting a more comprehensive and critical understanding of machinic listening? Can we diffract algorithmic listening? Who are the data workers and researchers involved in algorithmic listening? What is the „sayable and the visible“ (Foucault) of algorithmic listening and how could we transform it by design? What kind of effects would historical contextualization have on that process? The history of resistor–capacitor circuits and filters is for example strongly connected to telegraphy and might be an early example of mathematical modelling in signal processing. The context of underwater warfare during WW2 and later might be another field of historical contextualization: Operating on different audio filters was a crucial skill for sonar operators.

Hopefully these questions and my short reflections caused some response. Please write some of your thoughts in the comments section.

Making music through machine ears

Sun, 23 Apr 2017 00:00:00 +0000

About two months ago I released the Happy Valley Band’s first album, ORGANVM PERCEPTVS, a project that I like to explain as “pop music heard by a computer algorithm.” The music is jarring and chaotic, out-of-tune and out-of-time, artifact-laden renditions of classic songs many know and love. For some listeners the music is bliss; others find it absolutely maddening and categorically unlistenable. I’ve received responses such as “this is generally distressing to listen to, thanks” and “conceptually fascinating, but I simply cannot bear to listen to it” as well as “completely amazing.” Here is the opening track:

Madonna’s “Like a Prayer” as heard by a computer algorithm and re-performed by humans. From the Happy Valley Band’s ORGANVM PERCEPTVS.

Through the Happy Valley Band I explore the differences between human and machine hearing in the way that I, as a musician, know best: with my ears and my instrument. Starting with original recordings of pop songs, I perform machine listening analysis, render the results in musical notation, and then we, the musicians, play them. My machine listening process first uses source separation algorithms to isolate the individual instruments, then analyzes the separated tracks (source separation artifacts and all) for musical features like pitch, rhythm, dynamics, articulations, and playing techniques, and finally transcribes it all in musical notation for the band to play. The notation is impossibly over-specific, microtonal, and brimming with artifacts of the machine listening process. The liner notes Decomposing Music go into detail about the process of making the music. A typical Happy Valley Band score looks something like this:

Score excerpt of the Happy Valley Band’s (You Make Me Feel Like) A Natural Woman. The small numbers attached to each notehead indicate microtonal tuning deviations in cents.

At this point I have spent so much time fitting my brain into these algorithms — listening to their results, tweaking parameters, anticipating new results, then listening back and comparing — that I am pretty sure I have completely rewired the fundamental structure of my hearing mechanism. This project has put me in intimate contact with the idiosyncrasies of machine listening algorithms in way that I never planned for. It is difficult for me to hear these songs any other way, or, really, any other song any other way. This album should probably come with a disclaimer because my bandmembers, housemates, and friends have had the same experience.

What has driven all of this work — the countless hours spent writing custom code and convincing an ensemble of professional musicians to learn thousands of pages of computer generated notation — is a unwavering conviction that we should use machines to hear differently. The Happy Valley Band is about embracing rather than filtering out these differences, and the machine listening process often leads me to hear things in new ways. It turns the opening guitar riff of Led Zeppelin’s “When the Levee Breaks” into a ringing pile of harmonic series just-intoned 7ths and 3rds, or the simple opening timpani roll of James Brown’s “It’s a Man’s Man’s Man’s World” into a not-so-simple sequence of irregular re-articulations with constant foot-pedaling adjustments. Whether you call these differences “artifacts,” “errors,” or simply “too right,” this music is, to my ears at least, no more or less present in the original than an expert listener’s ground truth.

ORGANVM PERCEPTVS by Happy Valley Band

Happy Valley Band’s It’s a Man’s Man’s Man’s World.

When we listen through machine ears, whose ears are they? I’ll be the first to admit that I am not exactly sure what a machine way of hearing would be, or even what a human way of hearing would be. These are open question for me, and I am trying to find an answers through the Happy Valley Band. I do think machine hearing is a moving target; machines hear in as many ways as we design and build them. Machine hearing is a complex of technical, cultural, political, economic, and biological forces. It is something that changes and can be changed.

When I started this project over six years ago, the terms “machine learning” and “artificial intelligence” did not carry the same cultural connotations that they do today. As we now stare directly into a not-too-distant-future in which our entire experience will be mediated, captured, cataloged, and data-fied through the aperture of machine perception organs — be it Google Glass, Facebook’s augmented reality, or whichever tech giant manages to monopolize our daily experience — I believe more strongly then ever in the importance of machine perception systems that expand rather than reify our expectations — or, more realistically, system designers’ guesses at what our expectations may be. I am not content to wait to find out what unexpected results emerge. For me, Humanising Algorithmic Listening means seeking an understanding now and acting on it.

Of all the responses to the Happy Valley Band that I have received, this one gives me hope: “as you listen over time and the pieces somehow hold together and get tighter and tighter, you realize that they kind of aren’t strictly errors, there’s some kind of turbo-charged high level thinking going on.” I like to think of this “turbo-charged high level thinking” as a utopian future in which technology extends rather than enslaves our minds, in which we are cyborgs and we are better off for it.

For notification of future blog posts, follow @algolistening. For HVB news follow @anindexofmusic.

An interdisciplinary algorithmic listening

Mon, 27 Mar 2017 00:00:00 +0000

This blog is intended as a scratch pad for discussions. In these opening posts Co-I Dr Paul Stapleton and I will share our motivations for instigating this network; network participants are also warmly invited to share relevant research or reflections, so please be warned that we may approach you for a contribution, and feel free to offer some words if there is a pertinent topic you would like to think through.

There is a broad mix of disciplines represented, and it may be that any two participants have very little in common. This is intentional. The network was motivated, in part, by recent personal experiences of working across research communities – computer science, interactive music, soundscape ecology and most recently digital humanities. I have been struck – both confused and inspired – by the differences in attitudes, concerns, approach and ambitions of different disciplines with respect to the use of technology in general, and machine listening in particular, in research and practice. In this first post I will outline some of the personal motivations for the network, and consider some of the ways in which we might fruitfully ‘humanise’ algorithmic listening.

Exosomatic listening organs: The promise of new sonic prostheses

For celebrants of new technology, machine listening is as a key enabler, holding promise of unlocking new ways to understand and interact with the world. Decades of work in speech processing and music information retrieval have produced machine listening algorithms capable of accurately recognising aspects of human speech, or music such as genre, instrument melodic or rhythmic features. Current research is exploring other ways that machine listening could open up new ways of understanding and engaging with our lived environments across many other domains in research and industry: From digital humanities to conservation biology, industrial monitoring to audio archive management we are learning to listen with algorithms.

Rain Forest Connections’ upcycled mobile phones use automatic chain saw detection algorithms and text messaging to alert forest wardens of illegal logging in protected tropical forests

With ecologist and engineering colleagues I am working on the development of acoustic indices as a tool for rapid biodiversity assessment, and the development of low power hardware meshworks to create acoustic biodiversity monitoring systems. This work is driven by an urgent, global need for efficient and effective ways to monitor the planet’s critically endangered species, and to understand and evidence the impact of extractive industries on the fast disappearing pristine rainforests. In the race to develop new acoustic indices, fundamental oversights are being made regarding basic properties of digital audio recording which seriously compromise the effectiveness of this potentially transformative tool. As well as the need for more technical research, ethical considerations around the use of pervasive acoustic monitoring in the wild are yet to make it onto the agenda.

Humanising Algorithmic Listening might mean exploiting (rather than overlooking) the fundamental differences between machine and human listening in order to design meaningful high level algorithms in new application domains.
Humanising Algorithmic Listening might mean considering the rights of humans even where the wellbeing of other species is the primary concern.

A similar enthusiasm for computational methods to afford ‘distant or close listening’ is shared by researchers in the Digital Humanities. The fundamentally aural nature of poetry and oral history have traditionally been overlooked in humanities research, where textual sources are most common. Interviews are transcribed; the semantics of the written word is studied but the expression, intonation and rhythm voice is seldom preserved. Just as musically-meaningful features have been built from low level audio features for application in Music Information Retrieval, there is a huge potential for humanities researchers to engage with large oral and sonic archives in new ways with algorithmic methods. Responses to an exploratory workshop in the application of machine listening in Oral History that we ran at DH2016 and for arts and humanities doctoral students suggest this is a rich future research direction.

Humanising Algorithmic Listening might mean designing new audio features and machine listening methods for use in humanities research.

Pervasive listening machines: Anxiety over always on devices

Enthusiasm over new applications of machine listening in research is balanced by social anxiety over their increasing incorporation into everyday consumer devices. Amazon’s Echo, Google’s Home, Apple’s, Siri, Microsoft’s Cortana, and Matell’s Hello Barbie are all part of an emerging range of voice-activated products that record audio and conversations from phones, wearables, and in-home agents. Recent controversy over access to the recording archive of an Amazon ECHO in a US murder trial raises questions around privacy and the Internet of Things in general, but these ‘always-on’ listening devices are seen to be particularly problematic: They are pervasive, appearing in all aspects of our lives, and able to listen in all directions; they are persistent, with no current legislation over how long records are stored for; and they process the data they collect, seeking to understand what people are saying and acting on what they are able to understand.

This insidious surveillance contributes to widespread social anxieties around automation as it impacts both labour and recreational activities. This automation anxiety is a primary research topic for colleagues in our lab, where doctoral students are also questioning the ethical implications of these always-on listening devices in every-day consumer products through practice based research.

Humanising Algorithmic Listening might mean developing new ethical frameworks to keep corporate and commercial applications of machine listening in check.

Mattell’s controversial Hello Barbie features speech recognition and progressive learning features to create “the first fashion doll that can have two-way conversation with girls”. The girls’ conversations are stored on a remote server. It is not clear if it will also talk with boys.

Unpacking black boxes through listening

Part of this social anxiety stems from the unknown. There is no equivalent of an ingredients list for these algorithmically-enhanced consumer devices; commercial companies are not required to disclose their algorithms. From a public perspective, these are black boxes, the details of their operation is a Great Unknown. In some cases, the workings of bleeding edge machine listening and learning algorithms are equally evasive to the software developers that created them. For example, in the case of multi-layered neural networks used in Deep Learning, the maths is well understood, but in many cases we don’t know why a particular model is successful and we can’t predict it’s response to a particular input without actually trying it.

The extraordinary minds at Deep Mind are no doubt developing analytical understandings and make a concerted effort to communicate the science of deep learning. In a poetic turn, others are attempting to deepen our understanding of how deep learning for machine listening operates by listening to the learned features in a convolution neural network (check out Susurrant too). Through sonification, rather than formal analysis, we can gain an intuitive understanding of how each layer functions, in sonic terms. In a later blog post, network member David Kant will introduce his Happy Valley Band project which unpacks source separation and other machine listening algorithms by deconstructing the Great American Song Book and performing human recompositions of machine decompositions live.

Humanising Algorithmic Listening might mean enriching both our formal and intuitive understandings of how extant and emerging machine listening algorithms work in sonic terms.

Understanding human-machine agency in interactive performance

A belief in the value of practice-based, creative methods in enriching our understanding of technological mediation is common to many network members. And machine listening has been extensively explored and developed in interactive music systems for decades. By imbuing computers with even simple pitch and amplitude tracking abilities we can build software instruments which we can interact with in fundamentally different ways from traditional acoustic, electronic, or even manually controlled digital instruments. This was my first introduction to machine listening. As a cello-playing PhD student I was interested in how to build software performance systems which could convincingly improvise with a human instrumentalist, the litmus test being the ability to make convincing musical responses and more ambitiously, suggestions. When performing with, rather than formally analysing software systems we gain a different, experiential, understanding of machine listening, and its role in melding human-machine agencies. This will the main focus for workshop 2 and the topic of a future blog post by Paul Stapleton.

Humanising Algorithmic Listening might mean experientially probing human-machine agency through speculative, experimental and performative investigations.

Algorithmic listening: Utopia, dystopia or inevitable?

The range of concerns and ambitions present today, resonate with positions held throughout the history of Philosophy of Technology: utopian views of technology post-enlightenment; twentieth century dystopian warnings on the need to limit the rush of technological invention (Habermas, Hans Jonas etc.) to contemporary views that we are natural born cyborgs and inevitably intertwined with technologies of all kinds (Clarke, Latour, Haraway etc.).

Toward a well-rounded design for future hybrid ears

The aim of this network is not to reconcile these different perspectives, but to curate conversations across contemporary research communities in order that technical advances, practical ambition and creative investigation mutually inform and are nourished by critical, philosophical perspectives into how technology mediates our existence. The integration of technical, philosophical, creative and practical perspectives may be messy at first, but is necessary in order to design and manage the applications of effective, ethical and meaningful listening algorithms for the future.