How OpenAI’s Jukebox project is deepfaking Sinatra, Elvis, 2Pac and more
OpenAI, which describes itself as a “research and deployment company” and which is the home of, amongst other things, GPT-3, is running a project that uses artificial intelligence to generate music, complete with lyrics, in a variety of genres and artist styles.
Jukebox as it known has used a library of 1.2 million songs with accompanying lyrics and metadata to produce “deepfakes” of a range of artists, including Frank Sinatra, Katy Perry, Elvis Presley, Simon and Garfunkel, 2Pac and Céline Dion. The system can produce raw audio lasting several minutes based on a user’s inputs. Although it is a great technical achievement, creatively the results are more often unsettling than entertaining, especially when the voice being approximated is that of a much-loved and respected and long dead star.
“As a piece of engineering, it’s really impressive. They break down an audio signal into a set of lexemes of music – a dictionary if you like – at three different layers of time, giving you a set of core fragments that is sufficient to reconstruct the music that was fed in. The algorithm can then rearrange these fragments, based on the stimulus you input. So, give it some Ella Fitzgerald for example, and it will find and piece together the relevant bits of the ‘dictionary’ to create something in her musical space.”
Dr Matthew Yee-King, electronic musician, researcher and academic, Goldsmiths
As well as OpenAI’s work in this field, Google’s Magenta project, Amper Music and even Spotify are exploring the possibilities offered. The ethical and legal problems surrounding the production of musical imitations using machines are covered here by The Guardian’s Derek Robertson, but the hope is that the downsides are outweighed by the untapped creative possibilities offered by generating algorithms that make music. After all, the whole industry didn’t implode when musicians discovered sampling, and this is closely related, though a lot weirder at times.