By Jason Goodyer

Published: Thursday, 14 October 2021 at 12:00 am

---

When he died in 1827 aged 56, Ludwig van Beethoven left his 10th symphony unfinished. Only a few handwritten notes briefly detailing his plans for the piece have survived, with most just being incomplete ideas or fragments of themes or melodies.

Now, a multidisciplinary team of computer scientists at Rutgers University-based start-up Playform AI have trained an artificial intelligence to mimic the great composer’s style and used it to write a complete symphony based on these initial sketches.

We spoke to the lead researcher on the project, Professor Ahmed Elgammal, to find out more.

How much of Beethoven’s manuscript was available to you to start from?

Beethoven left sketches in different forms, mainly musical sketches, but also some written notes with some ideas in as well. Previously, in 1988 [English musicologist] Barry Cooper used the majority of these sketches, about 250 bars of music, that were meant for a first movement [in his attempt to complete the symphony].

But what was left behind is really very little. So basically, like three bars of music here and four bars of music there and some rough sketches, which sound like basically the starting points of the main themes in the movements that he [Beethoven] wanted to write.

When you look at Beethoven and other classical composers, that’s usually the case. I mean, usually they work with a main theme and develop it into a sequence of a couple of minutes and then another theme comes. That’s the traditional way of composing, and that’s exactly what the AI needed to learn – how Beethoven and other classical composers start with a theme and develop it. Like in the Fifth Symphony – ‘da da da dah’. And then take that and evolve a whole movement around it.

Read more about artificial intelligence and music:

• Will AI replace musicians?
• AI is about to shake up music forever – but not in the way you think
• The hidden genius of Blob Opera – and how it could get even smarter

So, say, you’ve got the theme like Beethoven’s Fifth. How do you train the AI to develop interesting melodies based on just the one motif?

The way AI generates music in general is very similar to the way your email, for example, tries to predict the next word for you. So, when you write an email, you find it jumps into suggesting what you might want to write next.

It’s the same concept, basically – the AI has to learn from a lot of musical data. It asks what would be the next note given what you just wrote? And if you can predict the next note, then you can predict the next note and the next note and so on. That’s the main concept.

But what we soon realise is that if you start picking up the suggestions from the phone for next word and start writing just based on the AI’s suggestions, it doesn’t really hold for a long time. And that’s what happens with music. If you just give it a starting point and leave it to predict, yes, it can predict a couple of notes. But then after that, it becomes nonsense more or less, and is no longer faithful to the main theme.

So that was the main challenge. How can we let the AI stick to the main theme and develop it? So this is where the role of the human expert working with the AI comes in. So we had to work with human experts to annotate and label a lot of music for us to tell the AI what the theme was and where the development of the theme was in a lot of pieces of music. So basically, the AI learnt as a student. That made a big difference because then the AI could really keep sticking to the theme.

Also, the AI had to compose the music in a specific musical form. So if you are composing for a scherzo movement or a trio part of the movement or a fugue etc, each of these musical forms have certain specific structure. The AI also had to learn how to write a fugue, how to write a trio, how to write a fugue, and how to write a scherzo.

Did you train the AI by listening exclusively to Beethoven or did you use other composers as well?

It was very challenging because Beethoven only wrote nine symphonies. That’s a very small dataset compared to the scale of what the AI needed to do. So, the way we approached this was to first imagine ourselves like a young Beethoven learning about music. What he would have listened to?

So, we trained our first version of the AI as if it was somebody living in the 18th Century listening to baroque music like Bach, as well as Hayden and Mozart. And so that was the first version of the AI, which basically would be the kind of music anyone living in that era would study to compose. And then we took that and trained it specifically on Beethoven – on old Beethoven sonatas, concertos, string quartets and the symphonies as well, so not only symphonies.

We first trained the AI to generate the composition as two lines of music, not as a full symphony, which is a typical way of a composer works – by just composing first and then orchestrating. So then, we had another AI that would take that composition and learn how to orchestrate it. I believe this is very similar to the way humans learn – you cannot really master fourth-level college without going through the first and second and third levels first. It’s always incremental.

How did you get the AI to take the melody that you created and then say ‘How would Beethoven harmonise this melody?’

The way we harmonise music is very similar to how we use AI to translate languages. Like when you use Google Translate or another AI to translate a sentence from one language to another. These kind of models used in translation learn a lot of background sentences. So, what is the sentence in German? What is the sentence in English? And from that, they try to learn how to translate them.

So basically, imagine you have these models [for harmonisation]. You put the melody in one side and on the other side you put in how Beethoven would harmonise it so the AI learns how to translate a melody line into harmonised music.

The thing about music is that it’s very structured and follows a lot of rules. But this is very hard for us to capture and write down. You really have to have a PhD in musicology with a speciality in Beethoven to really understand that. But the machine is able to capture that statistically and mathematically in a very implicit way and be able to use that to give us this harmonisation.

I presume the orchestration is just a natural development of that process.

You got it right. That decision is just an extension of the harmonisation. We wanted the machine to translate the composition into multi-track instrumentation, which we also did by training the AI based on how Beethoven and other composers would do so.

What has the response been like from musicians and composers?

Their response is really mixed. There are people who loved this very much, and love the idea of having an AI that understands music and can help you finish your composition or have you explore different musical ideas.

But on the other side of the spectrum, there are people who just reject even the concept of being able to complete a Beethoven symphony using AI. They are afraid of AI taking their jobs and think that it has nothing to do with this kind of thing.

Is it possible that we could get an AI to make a completely original work?

Yeah. I have no doubt about that, we did that in visual art a couple of years ago where we developed an almost autonomous AI artist we had look at, let’s say, the last 500 years of western art. The task was basically to generate new artworks that didn’t follow any existing style.

If the AI generated an impressionist or a Picasso kind of art or a Renaissance-style artwork, it could realise and so it would have to learn how to create something new.

The challenge with this project was actually the constraints – the fact that the AI was not generating music by itself but generating music that is based on Beethoven’s genius and also following the sketches. This makes it even more difficult. The high bar, of course, of expectation was due to the sketches coming from Beethoven. But when it comes to generating music autonomously I think that’s an easier task.

Listen to the symphony below: