The Detroit Symphony Orchestra uses Ted, a mannequin head outfitted with binaural microphones and hi-def cameras, which “hears” and then transmits live-sounding immersive audio and video. Ted was developed by composer Anna Clyne and her Grammy-winning engineer husband Jody Elff. Sarah Smarch
I look up and behold the ornately ornamented ceiling of Detroit’s Orchestra Hall, a place I’ve never actually been. I look down and my legs have vanished, replaced by a single black pole terminating into a tripod. I look all around and find every other seat in the hall empty, though I’m told that at least 10 of us are in attendance. And onstage in front of me, I see the 19 socially distanced members of the Detroit Symphony Orchestra tuning their instruments in complete silence – that is, until I pop in my ear buds.
Such are the dissonances that attend settling into one’s figurative seat at Orchestra Hall to experience the DSO through virtual reality.
I’m “here” to virtually attend a rehearsal of “Stride,” a stirring newer work from the British composer Anna Clyne. And Clyne is “here” with me as well, watching along through the eyes and ears of Ted – a standard-issue mannequin head, purchased off the internet and outfitted with a 360-degree camera and an array of microphones by creator, audio engineer and Clyne’s husband, Jody Elff.
Ted offers a live panoramic view of the hall and a crisp stereo audio signal, and, most important, allows Clyne to connect with the musicians playing her music in a way that the pandemic has rendered next to impossible.
Ten years ago, Elff launched a small company to provide underfunded arts organizations with robustly featured and lower-cost livestreaming capabilities. With this infrastructure in place, it wasn’t long before Elff began imagining how to enhance the streaming experience he’d built, such as adding binaural (i.e., left/right) audio or 360-degree video. (Call it a bit of pre-pandemic prescience.)
“There’s always going to be somebody that wants to see a show but can’t get in for some reason,” he says.
The Grammy-winning engineer thinks his homemade prototype could set an example for the role virtual reality could play in expanding the audience for classical music well past the pandemic. Ted takes up only one seat, but could one day serve as a proxy for thousands of listeners.
“I love the idea that for the cost of a $6 cardboard viewer, you could actually jump into another space elsewhere in the world and have something like a presence experience,” Elff says. “You can sell out your 2,000 seats in the hall, but now let’s go sell 10,000 more to virtual viewers.”
Elff’s early vision for Ted was driven by the idea’s big implications, but it was only when the pandemic and its practical demands set in that things came to a head. (Sorry.)
Composer Anna Clyne and her engineer husband, Jody Elff, with Ted. Anna Clyne and Jody Elff
For composer Clyne – who just released the fantastic “Mythologies,” a decade’s worth of her orchestral work – the constrictions of the crisis actually felt strangely compatible with the solitary nature of her practice. But they also severed an essential connection – with the musicians themselves.
The first run of a new work requires an enhanced level of listening. Clyne uses these preliminary rehearsals to make nuanced adjustments in dynamics or tempos, and to do so with a sense of the acoustical space that will shape the sound.
Travel bans and spiking case numbers have disrupted that process indefinitely, and the audio quality of most videoconferencing software proved untenable as a stopgap measure, let alone as a long-term substitute.
Clyne had streaming premieres scheduled through the fall and winter with the Orlando Philharmonic (where she is composer-in-residence), the River Oaks Chamber Orchestra and the DSO, but no way to hear what they truly sounded like.
“Suddenly,” she says of Ted, “this idea that’s been sort of bubbling away is a really viable option to be present, and to nurture this relationship with the musicians and conductors and organizations in a way that wouldn’t be possible without the technology.”
I can confirm that despite Ted’s rough DIY edges (a professional binaural head – something that exists – can go for upward of $8,000), he delivered something closer to an escape from my couch than just about anything else I’ve experienced from my couch – i.e., everything for the past year.
Put on a headset (see below), pop in some ear buds and your faculties quickly surrender to Ted’s. Ever see “Being John Malkovich”? It’s very that.
The sound in Orchestra Hall washes over his mic array and reaches you in a rich, spatial stereo wash. The separation and clarity is convincingly transportive – and richer than most of any livestreaming audio I’ve experienced – just “live” enough to detach you from the living room.
It’s exciting to think what improved video could add to the experience. There are plenty of consumer-level 360-degree cameras on the market, but most can snap only panoramic stills or record video. Cameras designed for HD live-streaming of 360 content can cost tens of thousands of dollars – a smidgen more than Elff or Clyne were seeking to spend for a system that now maxes out at 10 users.
The couple imagine Ted iterations featuring audio tracking, which would allow the sound to change in accordance with the position of the listener’s head.
(And while we’re talking hardware: The wide range of VR headsets available for this kind of experience, such as Facebook’s Oculus line or HTC’s Vive, can run an equally wide price range of $300 to $1,000. I used a $10 DIY cardboard dealie. It was perfectly suited to the intense demands of sitting still in one place for an hour, if a little rough on my nose.)
At one point in the DSO rehearsal, guest conductor John Storgårds paused the orchestra and pivoted on the podium. I shifted uncomfortably in my seat as I realized he was looking directly at me. He hollered a question about the rhythms of a certain measure into the hall, and, after a slight pause, from somewhere behind me, a woman’s voice responded. I turned and saw nothing but vacant rows.
It was Clyne – her voice piped into the hall through a speaker but her attention somewhere inside Ted’s head, swirling around with my own. At this point in the pandemic, I honestly thought we’d exhausted our options for bizarre, technologically enabled social interactions, but sharing a mannequin head with a composer in a faraway concert hall is definitely a new one for me.
Still, it feels appropriate to be trying out this technology mounted in the rows of Orchestra Hall, the longtime home of the self-anointed “most accessible orchestra on the planet.”
In 1922, the DSO became the first orchestra to perform a symphonic concert for a radio audience – a local broadcast on Detroit’s WWJ, featuring pianist Artur Schnabel.
An article in the Detroit News marveled at the potential of open airwaves to bring thousands of listeners “finer music and better played than could ever be heard in the small towns where they make their residence.” “The radio-phone,” it continued, “has opened new worlds of appreciation to music-hungry folk.”
A century later, with concert culture in the United States sidelined by the coronavirus, Americans are more music-hungry than ever, and the pandemic-fueled boom in streaming numbers for classical organizations of all sizes has upturned old assumptions about whether an online audience for classical music actually exists. It does and has.
(The DSO, for its part, has been livestreaming every one of its classical subscription series concerts since 2011. And they’ve kept up through the pandemic with weekly coronavirus testing and an additional 300 square feet of stage space, allowing for up to 45 musicians.)
As we determine which pandemic-inspired measures we want to bring into a post-COVID near-future, Ted could represent an entirely new way for audiences to feel present with orchestras – as well as a vital means for classical organizations to stay present in the minds of the far-flung audiences they’ve spent the past year scrapping to attract.
“It has to be accessible to the widest possible population if it’s going to be of interest at all,” Elff says.
If the last year is any indication, virtual reality and classical music seem awfully curious about each other.
Last May, the New York-based company Bare Opera premiered “Exercises on ‘The Presence of Odradek,” an immersive virtual-reality presentation of excerpts from an opera by David Rosenmeyer and Malena Dayen based on Franz Kafka’s “The Cares of a Family Man.”
And last September, the Binghamton, New York-based Luma Festival premiered a virtual reality adaptation (a “steampunk VR experience”) of Kamala Sankaram’s dystopic opera “Miranda.” (In 2016, Sankaram also created “The Parksville Murders” – “the world’s first virtual reality episodic horror opera.”)
Others have found ways to incorporate VR less as a portal into the music than as an accompaniment. For its recent “Music Illustrated” program, the Houston Symphony Orchestra enlisted artist Topher Sipes to join the orchestra onstage, where he used Google’s Tilt Brush technology to create spontaneous 3-D virtual paintings in response to works by Debussy, Bizet, Saint-Saens and Tchaikovsky.
And the Washington National Opera is in the midst of producing its own VR offering, featuring soprano Tamara Wilson and tenor Russell Thomas performing a pair of arias from Beethoven’s “Fidelio.” It’s due to be released this spring.
It’s an open question if virtual reality will ever be capable of gripping classical listeners the way one Evening Mail critic responded upon hearing the New York Philharmonic’s 1922 debut on the airwaves.
“It may not be beyond the bounds of possibility,” he extolled, “that man is on the way to the development of the senses that will open up boundless fields of adventure and delight.”
But while the great temptation of virtual reality is to go wild, dissolve all the boundaries of old media and treat freedom itself as a medium (and, inevitably, an excuse), the real strength of a head like Ted is its singular focus. He’s there to fill a very specific gap – your seat in front of an orchestra.
And after just a few minutes of “Stride” taking full, soaring flight seemingly right before my ears, the virtual sure felt a lot like the actual. I was no longer negotiating with audio. I was listening to music.
A view of Ted with the Detroit Symphony Orchestra. Grammy-winning engineer Jody Elff, and his wife, composer Anna Clyne, developed the system. “I love the idea that for the cost of a $6 cardboard viewer, you could actually jump into another space elsewhere in the world and have something like a presence experience,” says Elff. Sarah Smarch
Send questions/comments to the editors.
