Fusing jazz music with computer science | University of Southampton

Published:: 8 January 2021

Jazz is the ultimate improvisation. It’s complicated, it’s clever, it’s steeped in history and it relies heavily on the musician’s experience and memory. So, could it ever be created well by a machine?

In an unusual research project that fuses jazz music with computer science, Dr Tom Irvine is hoping to get to the bottom of the question.

If the answer is ‘yes’, it could open up a whole world of possibilities for the music industry.

Tom, Associate Professor and Director of Programmes in Music, said: “Jazz is a really historical type of music, it comes from its roots in African-American culture. It’s very much the passing down of knowledge from one generation to the next. Jazz musicians learn history: the tunes and the different versions of the tunes.

“I wanted to know if computers are able to do that yet too – and it turns out they don’t, or at least not so well. An investigation of jazz has turned into an investigation of computers.”

The project, called Jazz as Social Machine, is supported by an Alan Turing Fellowship from the Alan Turing Institute, with which Southampton became affiliated in 2018. Tom also works closely with the University’s Web Science Institute, of which he is a Non-Executive Director, as well as an AI (Artificial Intelligence) and music lab at the Academia Sinica in Taiwan and the Arts Council supported initiative Jazz South, which is based at the Turner Sims Concert Hall.

When AI meets music

The meeting of music and data is a hot topic. “The music industry is really interested in how music works – how to understand it as data,” explained Tom. “If it can do that, it can understand what you like and give you more of that.

The last decade has seen a lot of effort go into how music works as data, which has led to exploring AI composition.

Tom is applying this to the specifics of jazz music.

“This project is about how machines pretend to be creative,” he said. And it’s particularly interesting from the jazz point-of-view, as jazz is so unique.

“People don’t improvise jazz music, or any music, note by note,” added Tom, who has taught jazz history at the University for 14 years. “Jazz musicians improvise in melodies, and a lot of history and memory comes into that. It’s not just the data in the moment, it’s the memory of the history of jazz. And because jazz’s global history is so tied up with ideas of freedom and resistance, it’s often political too.”

To understand the possibilities, Tom is researching music composed by AI. He won funding from the Higher Education Innovation Fund in 2019 to visit Taiwan’s Academia Sinica, where they have a large music and AI lab, to learn more about AI and its possibilities with music.

He then applied for, and won, a Turing AI Fellowship worth £70,000. Through this he has brought on board Dr Valentina Cardo, from Winchester School of Art, and electroacoustic composer Dr Brona Martin.

Their project will see Brona, a Research Fellow in the Department of Music, create an interactive sound art to explore the possibilities of AI-improvised jazz. Alongside this, Tom and Valentina are working on various publications outlining what happens when a computer tries to make jazz and what that means for the AI and music industries, how AI interacts with questions of politics (such as gender and race) and our understanding of jazz itself as a practice.

All that jazz

Thanks to the AI element of the project, it’s not just musical notes – which Tom is pretty comfortable with – that dominate this work. Numbers are an equally important element.

“I am trying to understand machine learning techniques,” said Tom. “The maths are hard, but not impossibly hard for someone like me with an arts and humanities background, I hope! Because my project is about the notes and the numbers, it’s forcing us into conversations that are sometimes not that easy.”

He is working with a team at Academia Sinica on this, and a second visit to Taiwan to develop the project is planned for the autumn.

Ultimately, Tom’s research could have lasting repercussions for the music industry.

He explained: “Big players such as Spotify and Apple Music are waiting for machines to understand music better. Teaching a computer to tell the difference between a samba and a rumba, for example, is trickier than you’d think. But, once computers can do that, the promise is that people will get what they want faster from the service – the computer will be able to help you find what you like really quickly.”

Tom believes the possibilities are almost endless, and impact such areas as the Internet of Things (IoT).

Imagine walking into your house, your house sensing your mood and therefore knowing what lighting and music you’d like at that moment. This could be possible, with a really sophisticated understanding of music as data.

Ironically, despite his expertise and all his research, Tom steers clear of playing jazz music himself. “I’m not good enough!” he laughed. “I play the classical violin and the viola, and I’ll be sticking to those!”

Read more about this research project from the Alan Turing Insitute website.

This article is republished from the University’s Re:action magazine. Read more stories from Re:action here.