Lessons learned from mixed reality explorations of the USS Intrepid.

Can 3D scanned models help in engaging museum visitors through interactive experiences? How can audiences discover spaces that are inaccessible due to safety reasons or physical limitations?

I have long been thinking about these questions, building interactive apps as part of my architectural and exhibition design practice called Some Place Studio. In Spring 2019, I had a chance to teach a seminar at Columbia GSAPP to explore these questions in further depth with a group of talented students.

The seminar Tools for Show takes these questions as a starting point to create interactive prototypes for storytelling and architectural communication. Discovering multiple identities, qualities and values of an entity through its replication, students were asked to create a series of virtual and analog duplicates throughout the semester.

Using photogrammetry, the class carefully scanned the currently inaccessible Sick Bay of the Intrepid, a former aircraft carrier turned floating museum. Digging into the archives, students excavated stories surrounding the Sick Bay to creative games, quizzes, interactive experiences, and AR apps. The projects are documented online using Tools for Show, creating a virtual archive of the 3D scanned spaces and the stories surrounding them.

Software methodology

We used Torch AR in the prototyping process because it allowed for quick testing or our ideas. Most students had no previous experience with AR, so it was crucial to use something that could be easily picked up. While Torch can be used to build complex apps, in this project the ability to prototype and quickly iterate was more important to the class than high fidelity production.

Following here are two outstanding projects that used AR in an innovative way. They were created by two students teams and were shown on board the Intrepid as part of the final critique of the semester. Invited critics and museum staff were excited to see the projects and ways that AR, VR, and MR can help with making questions of preservation and historical artifacts engaging and relatable.

Pharmacy project: discovering historical prescription mixtures

Project Description: An interactive game putting you in the shoes of a pharmacist

Authors: Eduardo James Meneses, Jake Rosenwald, Zeynep Nazli Ugur

Our guiding concept for the pharmacy was to facilitate a sense of discovery. The pharmacy is a space of clinical efficiency, storage, and objects, each with their own history and stories. We recreated the Intrepid pharmacist’s experience in augmented reality by letting users step into the shoes of the pharmacist and learn from clues and information on virtual objects in this space.

We focussed on the cabinets, allowing users to open the drawers, pick out prescription mixtures, and discover the contents within. This interactive game is won when the user assembles the correct ingredients for the selected prescription. The user then has the option to pick out other ailments to explore.

The ability to interact with these previously off-limits artifacts affords viewers a new perspective into the historical remedies that were used to treat common illnesses onboard the Intrepid. We aim to provide a deeper understanding of the space and its objects by revealing the mystery that has formed over time.

‍

How AR helped us tell the story

AR’s capacity for physical interactions with digital objects in a digital environment became the main generator for our project. After using photogrammetry to map the pharmacy, we also researched the objects that could have been present in the space. The stories behind these objects drove the narrative in the scavenger hunt while the form and materiality of the objects built connection and a feeling of presence.

Consequently, we modeled and texture-mapped some of the original elixirs, tonics, and salves that would have been present on the Intrepid in the mid-century. These objects became the contact points in the space to interact with while providing a specificity and identity that engendered the space with a more personal relatability.

By placing these models in the 3D-scanned space, we came to see the potential of this project to situate itself in the realm of mixed reality. Our digital assets came to exist in this liminal space between virtual reality and an augmented overlay to physical interactivity. The scanned room could appear anywhere while the objects inside this virtual space allowed for educational elements and a sense of discovery. With time, we could see this prototype as an example of a totally curated environment, where stories are embedded in the objects contained.

Exam room project: generic space to discover hidden memories

Authors: Felipe Rocha, Jacky Hoang

Our objective was not to preserve the compartment as it holds no significant historical value. Nor was it to shed light to what the compartment function was, as it was largely overlooked. The space had many uses and names, and throughout its history accrued many objects. A sink, file storage, a pass-through window, hooks and more. These objects all tie back to a period in the ship’s history that although not significant to the space, are linked to deeper memories and events of the ship.

How AR helped us tell the story

AR is still a new frontier for all of us, and we have yet to scratch the surface of its capabilities. For our project, the main driver was the ability to layer or stack information in the physical world. Once the photogrammetry scan of the exam room was complete and rebuilt, we were able to adjust textures, perfect the lighting, and add detailed objects.

At some point during our experiment, the line between the physical space at the USS Intrepid and our Torch model began to blur. The exam room ceased to exist in a 3D physical world for us. We had spent more time inside our digital models in Torch than we had in the physical room. 

This time spent in AR allowed us to create an accurate “digital twin” of the room, that then activates a set of rather mundane objects in the room. Connected to historical images, videos, and GIFs of life on the USS Intrepid, these tools and artifacts teach through the act of discovery while anchoring the story in historical documentation. By creating an interstitial experience between the physical and the digital world we hope to create tools that reach more people and can pass on the liminal stories that are embedded in our world.

Final thoughts

It was exciting to see how quickly the students picked up AR designing with Torch and prototyping AR concepts. Through this work, we see how mobile AR allows for an intimate and portable engagement with these physically inaccessible spaces. Working with difficult content, it shows how AR is evolving beyond a technological gimmick. As more museums are building digital engagement tools, AR is developing into a serious tool with the ability to convey difficult subject matter in accessible ways.

Credits:

Tools for Show (the app)

Concept and Design: Bika Rebek for Some Place Studio (@someplacestudio)

Built by Torin Blankensmith and Peter Whidden, with ongoing support by Blake Rutledge

With the support of NEW INC (@newinc) and the Knight Foundation (@knightfdn)

Thank you to the staff at the Intrepid Museum, especially Jessica Williams and Elaine Charnov

----