Mae Jemison, the first woman of color to go into space, stood in the center of the room and prepared to become digital. Around her, 106 cameras captured her image in 3-D, which would later render her as a life-sized hologram when viewed through a HoloLens headset.

Jemison was recording what would become the introduction for a new exhibit at the Intrepid Sea, Air, and Space Museum, which opens tomorrow as part of the Smithsonian’s annual Museum Day. In the exhibit, visitors will wear HoloLens headsets and watch Jemison materialize before their eyes, taking them on a tour of the Space Shuttle Enterprise—and through space history. They’re invited to explore artifacts both physical (like the Enterprise) and digital (like a galaxy of AR stars) while Jemison introduces women throughout history who have made important contributions to space exploration.

Interactive museum exhibits like this are becoming more common as augmented reality tech becomes cheaper, lighter, and easier to create. A few years ago, the gear alone—a dozen HoloLens headsets, which visitors can wear as they file through the exhibit—would have been out of reach. Now, as the technology becomes easier to use and the experiences easier to create, museums are increasingly turning to them as a way to engage visitors—whether that’s fleshing out the skeletons on view at the Smithsonian’s National Museum of Natural History, or taking a tour of Mars with astronaut Buzz Aldrin (as a hologram, naturally).

At the Intrepid, the holographic Jemison isn’t just the docent of the future. She’s also a part of the exhibit, a chance for visitors to come face-to-face with an important figure from space history. “I hope that me taking them on this tour, that it makes it a little bit more real,” she says.


State of the Art

Museums have long relied on technology to give context to their exhibits—whether through informational videos, audio guides, or smartphone apps. Augmented reality, in some ways, is just the next iteration of that. It gives curators a chance to layer more information on top of existing exhibits, and to get visitors more involved with what’s on view.

“Cultural institutions are asking, ‘How do we ensure our relevancy in the future?’” says Chris Barr, the director of arts and technology innovation at the Knight Foundation, which gave over $1 million this year to support museums using new forms of technology. “We’re looking at tech as part of the toolset that they use to do that. There’s a tremendous opportunity, especially around technology like augmented reality, to engage visitors.”

Some museums have experimented with AR to bring damaged or broken artifacts back into their collections, or to remix the collections on view. This year, the San Francisco Museum of Modern Art worked with the design agency frog to create an “augmented reality gallery” to showcase some of René Magritte’s works, currently on view. The Smithsonian National Museum of Natural History put on an exhibit, called Skin and Bones, which lets visitors animate the museum’s collection of skeletons with an AR app on their phones. Even the U.S. Holocaust Memorial Museum has brought one of its exhibits to life, allowing visitors to learn more about the Lithuanian villagers featured in its Tower of Faces display with a companion AR tool.

“Museums are starting to get smarter and smarter about how do we personalize [the experience of visiting a museum], and how do we make those experiences just as magical as the art that you’re seeing,” says Barr.

The Intrepid’s exhibit takes it one step further, using HoloLens headsets to bring Jemison alongside visitors as she guides them through the space shuttle. “We want to make sure that while our artifacts create this exciting and tactile opportunity, we want to make sure we’re capturing our current generation in the language they’re speaking,” says Susan Marenoff-Zausner, President of the Intrepid Sea, Air, and Space Museum.


Behind the Scenes

The Intrepid collaborated with Microsoft, which filmed Jemison at its Mixed Reality Capture Studio in San Francisco. The studio space holds a combination of RGB and infrared cameras that capture scenes in 360 degrees, then render a mesh map in 3-D. “The infrared camera see a very densely speckled version of what’s in that scene, which the computer vision algorithms eat for lunch,” says Steve Sullivan, who heads up Microsoft’s Mixed Reality Capture Studios program.

When Microsoft first started licensing its mixed reality capture tech, it expected most of its business to come from celebrities, sports figures, and the entertainment sector in general. But Sullivan says educational and instructional institutions have become another fast-growing part of what the studio creates. “It’s way richer than video, but not radically more expensive,” he says.

Earlier this year, Microsoft worked with London’s Natural History Museum to create a “behind-the-scenes” museum tour. The experience involves a holographic David Attenborough, who shepherds visitors around the museum and shares stories about some of the artifacts on display—some of which are real, and some of which are digital renderings. Microsoft also worked with the Kyoto National Museum to create an immersive exhibit showcasing the art of Kennin-ji, the oldest Zen temple in Japan. Wearing a HoloLens headset, visitors could see 400-year-old artifacts fill the walls and ceiling of the museum, while a life-sized hologram of a Zen Buddhist monk toured them around.

“It’s getting museums to think outside of their physical confines,” says Sullivan. “They can have hosts and guides showing you more.”

Other tech companies have partnered with museums to bring their products into the gallery space. In 2017, shortly after introducing its AR platform Tango, Google teamed up with the Detroit Institute of Arts to show off what it could do. Museum visitors could borrow a Tango-enabled smartphone to discover hidden features, like an augmented reality skeleton inside the sarcophagi on view. The Perez Art Museum Miami leveraged Apple’s AR Kit to build augmented reality installations in surprising spaces, like the museum’s terrace. (Visitors could see the works through their own iPhones, or could borrow one from the museum.) Earlier this year, Intel worked with the Smithsonian to translate an exhibit in its Renwick Gallery to smartphones everywhere, using Snapchat’s augmented reality tech.

Of course, none of these exhibits rely on augmented reality alone. They still point visitors toward real-world objects and make use of the physical space in museums to create exhibitions. But museum curators hope they can engage visitors on a new level, and bring in new audiences altogether. For Jemison, who discovered her love of science on childhood visitors to Chicago’s Museum of Science and Industry, using HoloLens headsets is just one more way for museums to “engage curiosity and foster it.” If that gets one more kid curious about science and space, then it’s all worth it.

Augmented and Holographic Augmented Reality are technologies that can contribute a lot to the presentation of museum exhibitions. Applications that use such technologies, when used by visitors, enhance the visitors’ museum experience, making it more interesting and fun. With the combination of the partial immersion that they offer, they enhance the feeling of presence (a field studied by the Thematic Area 4 of the ViMM project, alongside the Storytelling and Gamification fields), as the visitors feel that the augmented models that they see are in the same room with them. Technologies like these can be proven very helpful to virtual museums and to the preservation of cultural heritage.