Adapting Stagecraft for XR
Until recently, UI design has largely dealt with interactions across all kinds of digital screens. Screens could be big or small. They could be fixed on a wall or go everywhere in a pocket. They could be used to navigate to work, or worn on a wrist to navigate through an evening jog. But in most instances, there was the user, the experience the user was engaged in, and a piece of glass in the middle — flattening users’ interactions and navigation.
In the emerging worlds of MR, VR, AR, and the Metaverse, users no longer perceive a piece of glass between them and the experience. They no longer experience products in two dimensions. And, as designers, we can no longer rely on the confines and conventions of a fixed flat screen. In these rapidly evolving design arenas, users consume content in the first-person perspective. They explore much more freely and intuitively. And location and orientation in these environments alter their perceptions dramatically.
At Punchcut, we work with companies to expand their design techniques and technology toolbox to create immersive experiences in 3D XR environments. As we’ve expanded our spatial design expertise, we’ve combined influences from various disciplines including architecture, environmental design, and more narrative-based meda like filmmaking and theater. In our XR work, we’ve had success creating powerful new experiences by adapting some old ideas from traditional stagecraft.
Globe Theater
While perhaps not as immersive as XR, theater has always been a spatial experience. Some of the earliest forms of theater were performed “in the round” with crowds surrounding raised stages or simple traveling carts. The Elizabethan playhouses of Shakespeare’s day commonly supported 270 degrees of seating with multiple vertically tiered galleries and a milling crowd of standing groundlings that might yell insults or even throw food back at the stage. It was and still is important for a production to read to people in every seat regardless of the angle or distance. This required building strong storytelling fundamentals.
Designing for immersive digital experiences mirrors the fundamentals of traditional stagecraft — scenery, lighting, costumes, props, sound engineering, etc. Like other stages, virtual environments should pull the audience into the experience, and help them forget the world outside, with one distinct difference — in XR environments your user is in the middle of the action. Leveraging stagecraft fundamentals, we’ve identified six key dimensions of XR spatial design, each critical to evolving a team’s design skills for new immersive digital experiences.
1. Narrative
The first thing thing do when developing an XR experience is decide what to build. This sounds simple enough, but not only do you need an idea of the problem you’re trying to solve, you need some initial assumptions about the type of experience you’re hoping to create.
When developing a play, this is typically done through a script with dialogue and stage direction to map out all the interactions between the actors. You need to consider each actor on the stage, and plan out believable dialogue based on the motivation of those characters. The script must also consider what scene setting (location, time, and cultural biases) the actors exist in. Their actions need to give the audience a glimpse of who they are and how they’re progressing in the story.
3D Storyboard
When applying this to XR, your primary actor is actually the user. You’re not telling a story to your audience as much as building one around your audience. For this reason, we’ve found we need a design artifact that’s a bit more visual. We rely on storyboards to plan out what we want our users to do and feel at each stage in the experience. This serves as the basis for what we’re going to build moving forward. With the backbone of our experience in place we create space and scene maps to plan all the items in the scene and determine which points are interacted with and when.
2. Form
With XR, you’re frequently building dimensional objects rather than flat UI elements. One interesting aspect of this is that people have different expectations for objects. We’re all used to touching, moving, lifting, and looking at objects. We know things about an object that we don’t even think about on a conscious level.
In traditional stagecraft, scenery and props are often more about creating the impression of a setting or object rather than complete faithful replication. In fact, objects on a stage would be nearly impossible to decipher without some level of exaggeration. Instead, it’s more important to convey the key attributes of an object.
A prop master might first ask if the object is solid, liquid, or gas, and how might we represent that to the audience given our limitations? For example, water and fire are often represented with fabric because of its ability to flow and reshape. Is it heavy or light? Should people struggle to lift it? Does it obey the laws of physics or does it operate via magical properties? Can you see through it? Does it bend light or emit its own light?
In a digital 3D environment, we deal with hardware and software constraints and might pose some of the same questions to determine stylization considerations. Processing, movement, and sensory constraints, artistic style, and more could all lead you to a similar exercise in XR around representing an object’s attributes. Moreover, one of the key strengths of XR is that it need not be limited by verisimilitude. You can fly, see through walls, breathe underwater, and more. All of this is much like using cloth instead of water on stage. What might suggest water to the audience, but fit my constraints?
IKEA Place
Working around these constraints and building an experience native to XR is when you need to understand modeling for limited polygon count or decide between real-time or pre-baked lighting, for example. It’s crucial to understand the hardware and end-use case you’re designing for. It’s a trade-off between performance and bells and whistles. At Punchcut, we believe designing around constraints with stylistic choices is what makes for a truly impactful experience.
3. Movement
The concept of locomotion is a hot topic in XR design. That’s because it’s so different from designing for flat screens where the term “navigation” is a bit more metaphorical. When we talk about locomotion, we’re speaking about how someone travels through our virtual space. Do they walk? Do they teleport? Do they fly? This receives so much focus because it is an especially tricky problem.
Wander
In stagecraft, movement is all about the layout and space constraints of the stage. And much like form, it’s more about the perception of movement than exact replication of motion and distance. A director will determine stage blocking to establish the positions and choreograph the movements of actors to best communicate the story to the audience. Can actors move the way they need to with the stage layout? If not, how can we adjust this layout during scenes to meet the needs of the actor? If we don’t have enough room on stage to capture the actual distance traveled — an arduous voyage between kingdoms, for example — how else might we communicate this journey?
With XR, we have a lot of the same limitations to solve. In AR, users are in the context of the real world so it makes more sense to adjust the augmented scene around them based on their needs. This is like designing a stage set where the actors only need to move gesturally. In VR, we often need to travel great distances without the physical space to walk those distances. This is where teleportation and joystick locomotion become valuable tools. Ultimately in XR, like in stagecraft, movement is more about perception than reality.
4. Focus
People are naturally curious, especially when introduced to an immersive space. Typically the first thing a person does when putting on a VR headset is look around to explore their environment.
As in a stage performance, drawing attention to specific actors or elements on the virtual stage is crucial. The audience shouldn’t work to understand where to look. In traditional stagecraft, this has taken a multi-modal approach. Directional lighting and sound grab and focus the audience’s attention. The composition of the set is also a strong tool to drive an audience’s eyes to the action of the scene. And the position, motion, gestures, and gaze of actors are pivotal to grabbing and directing the attention of the audience.
Sensory Swimlanes
In XR environments, we take a similar multi-sensory approach to guide user focus to the important aspects of the virtual stage. Sound, haptics, visuals, and especially motion allow you to leave clues about what aspects of the space are meant to be focused on and interacted with. In addition to attracting focus with human senses, it’s important to use a variety of tones of voice. Much like a stage performance, XR needs lead and supporting actors to communicate the message.
5. Immersion
Multi-sensory engagement is also critical for creating and maintaining an audience’s sense of immersion. To make someone feel present in an experience, designers need to activate as many senses as possible. And those senses should feel accurate to how we perceive reality. For example, sounds in the real world always have an origin point. Something is making every sound. It can be off in the distance or right in the palm of your hand, but if we design a space without sound or with sound that feels omnipresent, the silence or constant audio will remind the user that they’re in an artificial experience and undercut their suspension of disbelief.
BOXVR
When designing for a live audience in a theater, sound, lighting, and acting all help to immerse the audience in the narrative, making the experience more believable. Capturing emotion is crucial in telling a memorable story. While sound may not always be realistic in a stage performance, it should always feel like it’s the result of an action on stage or an emotion in the scene. Similarly, lighting choices, though often unrealistic on stage, need to feel like they either progress the narrative or have a natural origin point in the scene.
Immersion is fundamental in a 3D environment as well. When designing for the user’s first-person perspective, you need to engage the senses and imitate the ways they experience the real world for the user to truly feel comfortable and present in our digital space.
6. Production
At Punchcut, testing at every stage of the process is ingrained in our DNA. The rub is that in prototyping 3D experiences, there’s often a greater time investment and fewer mature lightweight (low code/no code) prototyping options. It’s critical to test often because the act of experiencing a 3D environment is more disconnected from the act of designing one (which is frequently on a flat screen in a dimensional design program).
In stagecraft, there’s a similar need for testing the end product at multiple levels. They test the audio and visual equipment, the mechanics of the stage props, and of course the effectiveness of the story itself through a series of rehearsals. Rehearsals typically advance from a basic table read to individual scenes to full run-throughs to technical and dress rehearsals. Testing the production over and over will ensure the audience experiences a flawless event.
XR Prototyping
At Punchcut, we get around the challenges of prototyping for XR by adopting this approach of testing at multiple levels. Similar to stage rehearsals, we often test iteratively, increasing the scope and fidelity of our prototypes over time from simple paper prototypes and physical mockups to immersive digital prototyping tools. Through collaboration with both design technologists and engineering teams, we can test various technical approaches and evaluate asset performance. And we often use validation research and user testing to test interaction mechanics and how effectively our experience is communicating to prospective users.
Stages, Screens & Scenes
While XR and the Metaverse seem to be flooding design discussions these days, we believe all designs should be human-centered and immersive. From Sophocles to Shakespeare to Spielberg, we have centuries of proven techniques to capture audiences’ eyes, ears, and hearts. As we move onto the virtual stage, we can lean on these techniques to immerse people into new worlds and ever more compelling stories. These concepts will lead to impactful XR solutions, but even if you’re not designing for XR just yet, many of these ideas are just as important when designing for more spatial, sensory, and seamless experiences for a wide variety of products and interfaces.
With thousands of hours spent designing immersive experiences, Punchcut has developed a methodology and approach for modern XR design. As people engage in immersive experiences for the personal, professional, and communal aspects of their lives, XR design will move to center stage and influence most aspects of our world. Everyday experiences like driving, shopping, communication, entertainment, education, personal wellness, and more will all be reimagined with XR. Punchcut will be there to ensure these experiences are not just remade, but made better.
A Punchcut Perspective
Contributors: Jay Jansen, Jojo Mendoza, Nick Munro, Nate Cox, Ken Olewiler
© Punchcut LLC, All rights reserved.
