[VIDEO] AI Vocal Clones Can be the Future of the Music Industry, Ask Holly's 'Digital Twin'

Dolly Parton's first hit single, entitled "Jolene," is characterized by a haunting melody and somehow repetitive lyrics. Since its release in 1973, a number of artists have already made covers of the classic, including Olivia Newton-John, Kelly Clarkson, and Miley Cyrus. But this cover, for some reason, is very smooth, flat simple, and enigmatic.

The video shown above was made by an artist named Holly Herndon, using an AI voice clone that she calls Holly+. Yes, after AI programs won art fairs, we now have a machine-learning system that is virtually capable of singing anything.

Meet Holly+

How is this possible? Through the wonders of machine learning and smart computer systems, this artist primed an AI to record and remember her own voice. Holly+ can also sing anything in multiple languages.

In a TedTalk video featuring Holly Herndon's bizarre creation, the artist shared a short part of a song that Holly had sung. It was a traditional song from Catalan, but hey, Herndon does not speak Spanish, nor is she trained to sing that specific song. The artist even admitted that what the AI does will be difficult to imitate, despite Holly+'s voice coming from her.

In the same TedTalk video, Herndon invited PHER, another artist, to perform. He began by singing as himself, passed his voice through a second microphone to Holly+ so she could sing "as" Herndon, and then simultaneously sang into both microphones, essentially duetting live with Holly+. Now, it seems like the word "solo duet" is not an impossibility.

Herndon explains that the AI uses a process called timbre transfer. In this particular process, a certain sound's quality is mapped onto another's performance. She says that the process is done by letting an AI learn a specific sound to make it into a template sound.

Questions on Ownership

Herndon has deep-faked her own voice, so now other musicians may do the same. According to NPR, what makes Herndon's project compelling is the question it brings to the idea of consent and ownership, which she designed in a field where the legality and boundaries of both are still uncomfortably ambiguous. Now, can people use the same technology to do unauthorized collaborations with their favorite artists and get away with it?

"The thorny questions that are being asked right now-it's really important that we get them right," Herndon told Wired in a phone interview. She added that she wants people to understand how powerful these systems are and how having sovereignty over training data is important.

About Herndon

Herndon's exploration of artificial intelligence is similarly remarkable to that of her AI counterpart. The Fast Company reports that the singer-scientist is also a programmer with a Ph.D. from Stanford University's Center for Computer Research in Music and Acoustics.

Herndon gives an example of what artistic control in the context of deepfake technology can look like.