How hologram technology is changing the future of the music industry

Digital Domain analysing
the topology of the human face

“Imagine a world where you can experience iconic events like Woodstock, The Beatles’ debut at the Cavern Club or the last performance of David Bowie’s Ziggy Stardust at the Hammersmith Apollo,” says the British Phonographic Industry (BPI)’s Gennaro Castaldo.

Inside the realm of music entertainment in 2019, anything appears potential. From mind-warping visual effects to superior 3D projections, our live-entertainment universe is expanding at a speedy fee. As Gennaro suggests, there will quickly be no limit on what – and who – we will pay to see, dwelling or lifeless. “It’s so exciting that artificial intelligence (AI), virtual and augmented reality, CGI and holograms all have the potential to keep enhancing and evolving the music experience.”

A star is reborn

In October, music icons Roy Orbison and Buddy Holly can be hitting the street on a joint headline jaunt visiting some of the UK’s most prestigious venues. Their Rock ’N’ Roll Dream Tour follows a collection of 2018 solo ‘live’ dates from Roy, enjoying his hit-laden again catalogue from the type of levels you’d associate with an artist of his magnitude and influence. There’s just one crucial difference to reside outings from comparable legends; Roy has been lifeless for more than 30 years.

These late rock ’n’ roll artists are among an ever-growing number of musical pioneers to be brought back to life, albeit digitally. Hip-hop legend Tupac Shakur was the first to take the public consciousness by storm at California’s Coachella Pageant back in 2012. Michael Jackson quickly adopted and negotiations are in place to reincarnate soul songstress Amy Winehouse.

Hologram technology, Tupac ShakurTupac Shakur performs at Coachella Pageant 2012 as a hologram

The upcoming double headline tour by Roy and Buddy has been put together by BASE Hologram, one of a number of international tech corporations competing for dominance within this new reside music subject. “As a company, BASE has always been at the forefront of what’s new and exciting. To us, holograms are the natural next step in live entertainment,” says Brian Becker, chairman and CEO of BASE Entertainment and BASE Hologram.

“Our focus is to combine the substantial and still-growing demand for live concerts, theatricals and spectacles with the accelerating interest in virtual reality (VR), augmented reality and holographic film technology.”

But regardless of the obvious excitement of witnessing peerless stars ‘perform’ their largest hits stay, these ventures increase new questions which transcend mere entertainment. Is that this new type of musical nostalgia probably going to stop rising expertise from taking centre stage? And is there something that could possibly be construed as disrespectful in treating lifeless artists on this method? It’s arguably a continuing reminder of the talent we’ve misplaced…

Pepper’s ghost

Let’s take a look at the technology in use here. Holograms are 3D pictures created by an interference of mild beams and have existed in numerous types for many years. Whereas new takes on this are out of the blue lighting up modern music levels, these variety of optical illusions have been a go-to for magicians down the ages – and one which is still harnessed by entertainers to this present day.

The ‘Pepper’s Ghost’ method is not technically a hologram; as an alternative, it’s a 2D visible trick collectively invented by the London-based engineer Henry Dircks and scientist John Henry Pepper. First shown in a stage manufacturing in the 1860s, it’s based mostly on an optical illusion utilizing the reflection of an unseen figure to offer the impression they’re floating on stage.

Hologram technology, Tupac ShakurPicture: Marc Broussely / Getty Pictures. Holograms can have other uses, too. For instance, at The ABBAWORLD Experience, the public can rise up and perform with cartoon-like versions of the Swedish foursome

This impact was behind the most jaw-dropping second from Coachella 2012. Out in the Californian desert at one of the pageant world’s largest occasions, Tupac Shakur emerged to share the stage with Snoop Dogg and Dr Dre to carry out two of his legendary cuts. It was a moment that basically rattled the cage of the music industry, and introduced the potential of hologram technology into the minds of many.

All eyes on me

Digital Domain’s Janelle Croshaw was visible effects supervisor on the digital recreation of Tupac, working intently with collaborator Steve Preeg on the challenge. Both had been visual supervisors on the Hollywood hit movie The Curious Case Of Benjamin Button. More lately, Janelle has busied herself working on a film directed by music star Sia in addition to the newest superhero hit Captain Marvel.

“Tupac was one of those projects that terrified most people, because of the difficult nature of recreating human beings digitally. Plus, we only had six weeks in which to do it,” she reveals.

The tight deadline proved to be one of many challenges. This was one of the first attempts to realize such an formidable recreation of a much-loved celebrity beneath the glare of the world’s media. “Often times, when making a digital human, you’ll start with a scan of the actor’s face or a sculpted maquette of the person. But we didn’t have that luxury with Tupac. We used old pictures as reference and had to rely heavily on our eyes alone to achieve his likeness,” she explains.

“It’s a whole different story making the rest of the shapes to make him speak,” she continues. “It’s possible to use already-existing shape libraries from previously built characters as a base to drive the performance, but enough shapes have to match Tupac to really capture his essence. With so little reference, we were struggling to really nail the look.”

An opportunity encounter with a health club consumer in Los Angeles who bore an uncanny resemblance to Tupac in phrases of facial structure and proportions provided the staff a lifeline. “He was really cooperative and let us scan him and take a lot of reference photos of him doing specific Tupac expressions,” says Janelle. “From there, visual-effects company Digital Domain’s super-solid tracking, shading, lighting and computer pipelines were used for the 18,000-frame single shot. It’s a miracle we pushed the project through in the amount of time we had.”

The visual coordinators have been only one half of a huge workforce of collaborators involved in pulling the groundbreaking undertaking together from projections to staging and efficiency. Janelle continues: “AV Ideas from San Diego was in cost of the precise projection and stage setup at Coachella. Director Philip Atwell and producer Dylan Brown oversaw and guided the process. Our image sequence was then projected onto a transparent Mylar foil.

“It’s actually pretty old technology, but it had to be engineered in a way where a curtain could go down and in minutes, it could be set up and solid. There were big issues early on with it swaying in the wind and also vibrating with Dre’s bass cranked up. They remedied that situation, for the most part. The late Demfis Fyssicopulos, lighting designer, played a huge part in orchestrating it.”

New digital domains

Regardless of the challenges, Tupac’s look was an enormous success, having an enormous international influence, amassing tens of millions of YouTube views, Google searches and column inches. Main visual-effects company Digital Domain oversaw the challenge; despite expertise in the world of visible effects in Hollywood blockbusters, it was its first expertise dealing with holograms.

John Canning, government producer, new media and experiential at Digital Area, says: “To bring back Tupac really leveraged our skill and expertise from years in the visual-effects business creating characters such as Benjamin Button, the Beast [for Beauty And The Beast] and Thanos [for Avengers: Infinity War and Endgame]. Over the years, we have created other characters such as [late Taiwanese/Chinese singer] Teresa Teng, where we’ve continued to leverage the process and technology of our core visual-effects work in films.”

Hologram technology, Teresa TengTeresa Teng

So how has technology changed in the final seven years to improve how these dearly departed stars are introduced? “The increase of processing and graphic capabilities of computers has certainly had a huge influence,” explains John. “The rise of refined AI methods akin to machine learning has additionally performed a big issue.

“We take the visual data information from a camera on someone’s face, have the system learn how the face works and then apply that information to driving a character’s face. This led to Josh Brolin’s performance through the face of Thanos in Avengers: Infinity War. We are creating digital humans today with amazing lifelike fidelity.”

Again to black

Seven years after hologram Tupac’s efficiency, industry chatter around holograms, or at the very least visual illusions, is getting louder as growing numbers of deceased performers return to digital life. Whereas BASE will host its Roy and Buddy tour in the autumn, negotiations are also ongoing with Amy Winehouse’s property to convey the Again To Black star to the world once again.

With their roots hardwired into the worlds of VR and particular effects, BASE is one of the leading proponents of this craze. So why did it choose to deliver Roy again to actuality? “Roy Orbison is a very unique and special performer. His music is best experienced in intimate settings, which made him a natural fit for this type of experience,” says Brian.

With the blessing of Roy’s property and family, BASE labored with Eric Schaeffer, director of Broadway musical, Million Greenback Quartet, to assist create the stage show. Brian is excited by the tech’s potential for inspiring youthful generations of music fans.

“Roy influenced so many of the world’s musical heroes from the likes of Bob Dylan to Bruce Springsteen. So we believed it was important to introduce him to a new generation that may have heard his music, but are unaware of his legacy.”

In goals

When this virtual Roy Orbison returns to the UK in the autumn, the tour will function an accompaniment together with a mixture of pre-recorded tracks and stay music from the legendary Philly Pops orchestra. These excursions are created utilising cutting-edge methods to deliver these artists to the stage. While Brian is reluctant to provide an excessive amount of away, there is some important groundwork BASE undertakes to make sure the visual results are as real looking as potential.

“We start with a body double who works closely with our director to choreograph the performances. Then we take the results and go to work on them digitally, along with, in many cases, cleaned up and remastered cuts of the songs,” he states. “The technology has evolved so the team can now for the first time strip out the vocals and separate the tracks from both orchestra and other singers. From there, it’s marrying that audio with digital and laser imaging, CGI techniques and spectacular showmanship to create these projects.”

Hologram technologyDigital Area’s Doug Roble recording his actions in an Xsens MoCap go well with

Together with this brave new world of music leisure, there remain some slightly uncomfortable questions for the tech corporations behind them. Is it morally questionable to deliver again Roy Orbison with out his say? And doesn’t this all really feel just a little bit… creepy? Moreover, the ‘uncanny valley’ concept suggests individuals really feel unsettled by humanoid objects which seem virtually but not fairly precisely like humans. So does this worry BASE?

For Brian, it’s important their work is undertaken in the absolute best taste and with the full permission and help of the artist’s estates. “We are honoured that the families of these beloved performers have entrusted us with their legacies,” he explains.“We wouldn’t want to do this without having them onboard as collaborative partners. These projects are deeply personal to them and that close relationship is invaluable to the show’s authenticity.”

The appropriate cause

For music fans, the concept of hologram excursions is divisive. There are those that react as if this have been an insult to a lifeless legend. Different new listeners are excited by the probability to expertise long-gone icons in the flesh, regardless of the type or medium.

“Typically, artist estates are behind the use of these technology-led events, and while legally and technically, hologram tours may be achievable and without issue, you have to hope they are motivated to do it for the right reasons,” says Gennaro from the BPI.

Hologram technology

As the potential to create characters improves, artists might want to more and more think about permissions round the use of their picture after demise. At the similar time, as the tech behind this visible magic develops, better-quality experiences and more lifelike excursions must be within grasp. Gennaro says: “It means the more the public will respond to them and be prepared to genuinely buy into the idea.”

Jason Edwards, head of music at DICE, the market-leading live-music ticketing app, agrees there can be challenges for the industry to ward towards cynical cash-ins from hard-up artist estates. “Ultimately, fans will vote with their feet. If the ethics are unsound, then diehard fans are going to be really put off and won’t buy tickets.”

Novelty issue

Elsewhere, naysayers have advised the technology is a mere fad and never one thing succesful of rivalling the thrill of new stay music. But when the tech turns into highly effective enough, might it’s the case that in years to return the reside industry is dominated increasingly by hologram excursions, with much less area for dwelling artists to perform?

Controversial Chicago hip-hop star Chief Keef organised The Chief Keef And The Hologram Icons Of American Music Tour with another tech chief, Hologram USA, in a bid to bypass legal bans on him performing. Nonetheless, Jason from DICE is sceptical about the potential for this expertise to enter the mainstream. “For me, the novelty factor remains. The hologram tech looks impressive, but it can’t really compare to a real artist on stage, reacting to a crowd. At the same time, younger audiences do seem more open to new forms of live entertainment. Just look at the impact of the Marshmello Fortnite shows – 10 million viewers watching an artist perform in a virtual world.”

EDM artist Marshmello definitely has music-industry speaking heads staring exhausting into their crystal balls trying to foretell the future. His ‘appearance’ in the wildly common video game noticed consumer weapons disabled so they might take pleasure in the immersive 10-minute live performance in-game. Is this where stay music is heading subsequent?

Dr Timothy Jung is founder and director of the Artistic AR & VR Hub at Manchester Metropolitan College and in addition head of MMU AR & VR Analysis Group. He’s a VR professional and speaks on the subject at a variety of music-industry events. He tells us that: “While there is genuine interest in this technology, it may not recreate the thrill or excitement of an actual live performance. For the time being, this may struggle to replace actual human emotions.”

Leading edge

Nonetheless, Dr Jung is excited by what might happen next within stay music. He and his staff are working at the leading edge of VR, exploring the potential for tasks within entertainment, retail, coaching, schooling and extra.

“The live-music industry will explore a combination of multi-sensory (eg, vision, audition, gustation, olfaction, tactician) immersive experiences using augmented reality, VR as well as holograms, which is one of mixed-reality technology, in order to touch fans’ emotions,” he states.

“It seems holograms will be one of the most powerful pieces of technology for the future of the live-music industry, but the quality of hologram and harmony between physical music performance is essential. It has to go beyond gimmick to survive.”

So with Roy and Buddy anticipated to touch down in the UK this autumn, proof of the lifetime of this new strategy to stay music will come from ticket sales. For BASE, the prospects are countless, stretching past leisure. For the music industry, stay music is the most blatant area to profit from this new technology, nevertheless it doesn’t have to stop there. The BPI’s Gennaro Castaldo believes there are implications for the recorded sector, too.

“Imagine a world where as you stream your favourite song or album, the artist or group performs it to entertain you in the form of a hologram. So streaming continues as a core format, but our experience keeps getting enriched as tech increasingly makes it possible,” he says. “It might sound rather far-fetched and part of the realm of sci-fi fantasy rather than real life. But when you consider the revolutionary changes brought about tech in just a few years, who’s to say what will be possible?”