We spoke to the lead researcher on the project, Professor Ahmed Elgammal<\/a>, to find out more.<\/p>\n Beethoven left sketches in different forms, mainly musical sketches, but also some written notes with some ideas in as well. Previously, in 1988 [English musicologist] Barry Cooper used the majority of these sketches, about 250 bars of music, that were meant for a first movement [in his attempt to complete the symphony].<\/p>\n But what was left behind is really very little. So basically, like three bars of music here and four bars of music there and some rough sketches, which sound like basically the starting points of the main themes in the movements that he [Beethoven] wanted to write.<\/p>\n When you look at Beethoven and other classical composers, that\u2019s usually the case. I mean, usually they work with a main theme and develop it into a sequence of a couple of minutes and then another theme comes. That\u2019s the traditional way of composing, and that\u2019s exactly what the AI needed to learn \u2013 how Beethoven and other classical composers start with a theme and develop it. Like in the Fifth Symphony \u2013 \u2018da da da dah<\/a>\u2019. And then take that and evolve a whole movement around it.<\/p>\n Read more about artificial intelligence and music:<\/strong><\/p>\n The way AI generates music in general is very similar to the way your email, for example, tries to predict the next word for you. So, when you write an email, you find it jumps into suggesting what you might want to write next.<\/p>\n It\u2019s the same concept, basically \u2013 the AI has to learn from a lot of musical data. It asks what would be the next note given what you just wrote? And if you can predict the next note, then you can predict the next note and the next note and so on. That\u2019s the main concept.<\/p>\n But what we soon realise is that if you start picking up the suggestions from the phone for next word and start writing just based on the AI\u2019s suggestions, it doesn\u2019t really hold for a long time. And that\u2019s what happens with music. If you just give it a starting point and leave it to predict, yes, it can predict a couple of notes. But then after that, it becomes nonsense more or less, and is no longer faithful to the main theme.<\/p>\n So that was the main challenge. How can we let the AI stick to the main theme and develop it? So this is where the role of the human expert working with the AI comes in. So we had to work with human experts to annotate and label a lot of music for us to tell the AI what the theme was and where the development of the theme was in a lot of pieces of music. So basically, the AI learnt as a student. That made a big difference because then the AI could really keep sticking to the theme.<\/p>\n Also, the AI had to compose the music in a specific musical form. So if you are composing for a scherzo movement or a trio part of the movement or a fugue etc, each of these musical forms have certain specific structure. The AI also had to learn how to write a fugue, how to write a trio, how to write a fugue, and how to write a scherzo.<\/p>\n It was very challenging because Beethoven only wrote nine symphonies. That\u2019s a very small dataset compared to the scale of what the AI needed to do. So, the way we approached this was to first imagine ourselves like a young Beethoven learning about music. What he would have listened to?<\/p>\n So, we trained our first version of the AI as if it was somebody living in the 18th Century listening to baroque music like Bach, as well as Hayden and Mozart. And so that was the first version of the AI, which basically would be the kind of music anyone living in that era would study to compose. And then we took that and trained it specifically on Beethoven \u2013 on old Beethoven sonatas, concertos, string quartets and the symphonies as well, so not only symphonies.<\/p>\n We first trained the AI to generate the composition as two lines of music, not as a full symphony, which is a typical way of a composer works \u2013 by just composing first and then orchestrating. So then, we had another AI that would take that composition and learn how to orchestrate it. I believe this is very similar to the way humans learn \u2013 you cannot really master fourth-level college without going through the first and second and third levels first. It\u2019s always incremental.<\/p>\n The way we harmonise music is very similar to how we use AI to translate languages. Like when you use Google Translate or another AI to translate a sentence from one language to another. These kind of models used in translation learn a lot of background sentences. So, what is the sentence in German? What is the sentence in English? And from that, they try to learn how to translate them.<\/p>\n So basically, imagine you have these models [for harmonisation]. You put the melody in one side and on the other side you put in how Beethoven would harmonise it so the AI learns how to translate a melody line into harmonised music.<\/p>\n The thing about music is that it\u2019s very structured and follows a lot of rules. But this is very hard for us to capture and write down. You really have to have a PhD in musicology with a speciality in Beethoven to really understand that. But the machine is able to capture that statistically and mathematically in a very implicit way and be able to use that to give us this harmonisation.<\/p>\n You got it right. That decision is just an extension of the harmonisation. We wanted the machine to translate the composition into multi-track instrumentation, which we also did by training the AI based on how Beethoven and other composers would do so.<\/p>\n Their response is really mixed. There are people who loved this very much, and love the idea of having an AI that understands music and can help you finish your composition or have you explore different musical ideas.<\/p>\n But on the other side of the spectrum, there are people who just reject even the concept of being able to complete a Beethoven symphony using AI. They are afraid of AI taking their jobs and think that it has nothing to do with this kind of thing.<\/p>\n Yeah. I have no doubt about that, we did that in visual art a couple of years ago where we developed an almost autonomous AI artist we had look at, let\u2019s say, the last 500 years of western art. The task was basically to generate new artworks that didn\u2019t follow any existing style.<\/p>\n If the AI generated an impressionist or a Picasso kind of art or a Renaissance-style artwork, it could realise and so it would have to learn how to create something new.<\/p>\n The challenge with this project was actually the constraints \u2013 the fact that the AI was not generating music by itself but generating music that is based on Beethoven\u2019s genius and also following the sketches. This makes it even more difficult. The high bar, of course, of expectation was due to the sketches coming from Beethoven. But when it comes to generating music autonomously I think that\u2019s an easier task.<\/p>\n Listen to the symphony below:<\/strong><\/p>\nHow much of Beethoven\u2019s manuscript was available to you to start from?<\/h4>\n
So, say, you\u2019ve got the theme like Beethoven\u2019s Fifth. How do you train the AI to develop interesting melodies based on just the one motif?<\/h4>\n
Did you train the AI by listening exclusively to Beethoven or did you use other composers as well?<\/h4>\n
![\""The\"](\""https:\/\/images.immediate.co.uk\/production\/volatile\/sites\/4\/2021\/10\/2-crop-f95d9b5.jpg?quality=90&resize=620%2C349"\" "\\"\"The\\"")
<\/source><\/source><\/source><\/source><\/source><\/source><\/source><\/source><\/source><\/source><\/source><\/source><\/source><\/source><\/picture><\/div>How did you get the AI to take the melody that you created and then say \u2018How would Beethoven harmonise this melody?\u2019<\/h4>\n
I presume the orchestration is just a natural development of that process.<\/h4>\n
What has the response been like from musicians and composers?<\/h4>\n
Is it possible that we could get an AI to make a completely original work?<\/h4>\n