This post orig­i­nally appeared on the Oculus Blog on July 13, 2020. You can view it here.

The team at Schell Games has been providing education and entertainment game development since 2002, and more recently have released titles like I Expect You To Die and Until You Fall. Today we hear from Alexis Miller, Director of Product Management, and Tera Nguyen, Producer, as they provide a number of learnings and insights from their experience playtesting 2D and VR titles.

If you’re interested in learning more on this subject, be sure to check out the recently published Oculus Playtesting Guide, which includes collaboration with the team at Schell.

We’ve done a lot of live moderated playtests over the years, and we recently found success with remote unmoderated playtests and remote live (video) playtests as well. The tips shared here are the result of a lot of shared wisdom from our past and present colleagues at Schell Games. Each method involves different planning and pros and cons, but all methods can provide valuable feedback.

All Methods

All methods of our testing require the following:

• A recruitment plan for finding and scheduling testers.
• A technical plan for how your players will get access to the game.
• A way for players to sign or agree to your company’s non-disclosure agreement.
• A checklist and/or script that includes information about the specific equipment and systems, ways to make your playtesters comfortable with the equipment and with sharing their feedback, and plans for data gathering.
  • A list of questions (prepared in advance) to ask the player about their experience playing your game is really important. While improvising some questions can be good, don’t improvise all the questions.
  • The fewer surprises and frustrations the player has during the setup process, the better the experience and feedback will be.

Moderated In-Person

If players are coming to our office or other location, they’ll need a lot of details that one might take for granted but are important to them and will make it a better experience for them. These details include:

• Where to park and/or what public transportation is nearby.
• What time to arrive and what time they should expect to be done.
• If they need to check in at a front desk or security.
• If they will be paid for their time and how much.
• If you will be providing any food for them (we always ask if they have food allergies).

We’ve recruited testers from our own marketing email lists, friends and family of the studio, and by reaching out to local high schools, colleges, or universities with which our company has relationships.

Remote Unmoderated

For unmoderated, remote playtests, we recently started to use the platform UserTesting. UserTesting records the tester’s screen—either on a PC, tablet, or smartphone. This platform gives us access to nearly 2 million people in the US, Canada, and other countries around the world (they don’t all have VR equipment though). To test on this platform, there’s a lot of upfront, pre-test planning work, but we save time by being able to reuse our test questions and tasks on future tests and not having to spend our time moderating each test individually.

First, we launch a recruitment test to find testers in the UserTesting panel who have the right hardware (like an Oculus Quest) and can demonstrate their ability to cast their Quest on the UserTesting platform—either to their phone or a PC. This 10- to 15-minute test doesn’t require them to download our game yet. It’s a way to make sure the real playtest will go more smoothly from a technical standpoint since we aren’t there in-person, or even virtually, to help them. During recruitment, testers also have the option to provide their email address. This is used for subsequent playtests, as we give them access to our game through our Oculus development release channel.

In order for testers to access our games that are in development, we created a playtest release channel on the Oculus Dashboard and added our testers’ email addresses whenever we launch a remote unmoderated test.

In these unmoderated playtests, we walk testers through each step one at a time via written instructions. There are important, but basic, tips to remind the player of in an unmoderated, remote session. These are a few suggestions:

• Make sure the tester’s headset and controllers are charged before starting the test.
• If they are casting from Quest to their phone for the test, it’s polite to remind them to turn off push notifications for privacy reasons.
• Turn up the volume on their Quest so that the UserTesting screen recording picks up the in-game audio.

We want to make sure that we have a diverse group of playtesters, so one thing that’s really useful is collecting demographics of our testers and asking them a few questions related to accessibility. We recently started asking players which hand is their dominant hand (or if they’re ambidextrous) and if they identify as having any disabilities or accessibility needs.

Most players can cast from their Quest to the Oculus App if given the right instructions and common troubleshooting tips. These are some of the tips that we share with testers that seem to help:

• Make sure Bluetooth is turned on in your headset and phone.
• Make sure you’re signed in with the same account/email address on the Oculus App and your headset.
• Make sure your headset and phone are using the same WiFi network.

With this mode of testing, we learn a great deal from players when they think out loud as they play, so at the beginning of the test, we remind them to talk while they perform the required tasks.

Lastly, because these test instructions are being read on the player’s phone or PC, we try to set them up with all the needed information about the test before they go in their headset to play the game. They only have to answer questions when they’re done playing, so there isn’t a lot of lifting their headset up and down, going between their headset and their phone or PC screen. Besides it being uncomfortable for the player, this behavior can also disconnect casting and cause other technical problems on the UserTesting platform. This also provides a more authentic play experience by not interrupting their play.

Remote Moderated (Live)

For remote moderated (live) tests, we have scheduled several 30-minute to 1-hour long live sessions with our testers over video conference or the UserTesting platform. Our designers act as moderators to give testers instructions on how to set up casting and how to download the game. They also help whenever testers have technical difficulties or get stuck, frustrated, or unsure what to do to move forward in the game. During these live sessions, designers can ask testers open-ended questions like, “What do you think you should do next?” or, “Why did you perform x,y,z interactions?” in order to gather insights on the testers’ expectations.

Remote moderated tests are specifically useful when a product is still in its early development stage and requires additional guidance or prompting. Similarly, it is also useful for testing later levels or parts of the game that may take a lot of time or skill to play through.

In order to ensure that testers have everything they need for the live tests, we follow similar protocols with the unmoderated tests (mentioned in the section above) except that the moderator is reading a script with the instructions out loud instead of the tester reading the instructions on their PC or phone.

It is crucial for us to be able to communicate with testers while hearing and seeing their gameplay. Therefore, we usually spend the first 10 minutes of the live tests helping testers set up their casting method and video/audio settings. The most successful method we have experienced is when the tester casts their VR view to the Oculus App on their smartphone while screen-sharing it to us over a conference video call.

Another easy option is for testers to stream their gameplay privately to a Facebook page; however, if your game has not been announced or released to the public, you may not want to pursue this option. While there is a “private” setting, it could be easy to miss this setting and cast the playtest publicly to the tester’s Facebook page.

Test Facilitation

Here’s what we’ve learned from facilitating VR playtests, including some process and tactics that have helped ensure we gain actionable insights.

Moderated In-Person

• Try to limit the number of people in the room with the playtester and announce when new people enter the room since it can be uncomfortable for a VR tester to not know how many people are near and watching.
• Clean the VR headset materials while the playtester is watching. It’s another way to build rapport and trust.
• Try to limit developer talking during the playtest, especially if it’s not directed to the playtester.
• Create an isolated, large testing space with a soft floor so the playtester knows they’re in a safe space for testing.
• Show the playtester the headset and controllers, and offer to adjust the straps for them.
• When asking the player questions, wait for their answer longer than feels natural before moving on so that they have time to think and respond. It’s amazing how much more talkative players can be when you give them extra silence to fill.
• One good way to get more information is by asking the question, “Can you tell me more about that?”
• If a player becomes stuck to the point where intervention is needed, don’t just give them the answer. Try to replicate the players’ thought process as much as possible so they can find the solution on their own. For example: “Imagine you notice that the second panel on the left has some strange markings on it.”
• It can help to remind playtesters that there are no right or wrong answers to questions and that all of their feedback is helpful, even if they feel frustrated or confused.
• Show playtesters how to calibrate their screen with the Oculus Home button so they can have a centered view of the game.
• Make sure that the tester sets up the appropriate boundary to play the game. In our case, we develop a seated gameplay experience and it is different from a Quest stationary boundary.

Remote Unmoderated

• One of the great things about this method is that the player is in their home with their own headset, so they are playing in their natural environment.
• Because questions have to be typed in advance and then read by the player, be sure to keep the questions as open-ended as possible as there is only one shot. Avoid questions that they could answer easily with “Yes” or “No.” That’s usually not enough information.

Great questions to ask playtesters:

• How often do you play video games? What are your favorite games?
  • This question can help determine their expertise and what kind of player they are.
• Did you feel any discomfort, physical or otherwise?
  • This question can help catch occurrences of discomfort, account for body type, and reveal issues with the content (diversity, representation, etc.)
• What was your favorite part of the experience?
• What was your least favorite part of the experience?
• What was the most frustrating part of the experience?
• When did you feel most clever?
  • This is especially useful for puzzle games.
• What do you think we should change first before we test with others?
• If you had a magic wand and could add, change, or remove anything, what would you wish for?
• How would you describe this game to your friends?
  • This question can be useful for making sure design goals are being met. For marketing, it is helpful to hear the words that players use to describe the game. These statements can be used to hone store descriptions and marketing materials.

Data Analysis / Post Playtest

Next, we’ll look at the processes, tools, and data points we leverage to analyze playtest results.

Remote Unmoderated

The platform we’re using creates a recorded video of each playtest and their written responses to any post-recording questions we send them. It also gives us demographic information about each player, including age, gender, profession (if any), household income, gaming genres they like, and if they have any children. It also allows us to create short video clips and highlight reels with tags, which makes it easier for multiple people on the team to review the relevant parts of the videos. For example, when using a tag like #playerdies, it is easy to search for all the parts in the videos where the player died and analyze any patterns or surprises.

These clips and tags have time stamps too, so we can measure how long it takes players to perform certain actions, like completing the tutorial or successfully completing a level.

All Testing

We’re working on building a pipeline to receive and analyze results using Unity Analytics. This platform promises to provide data on the most common or least common player experiences in the game, such as what the most common reasons were that a player died. We create target behavior ranges in advance and aim for the data to fall within those ranges. Once we have enough data to review, we can make changes to the game and observe the results.

Further Learnings

Don’t be afraid to playtest early, even if the game isn’t totally playable. Most developers find it valuable to gain player insight from prototypes and unfinished levels. It can help to remind players that the game isn’t finished so that their expectations are realistic.

Make early preparations to include playtesting in the process and to allocate time for people on the team to prepare the tests, do the recruiting, moderate the tests, analyze the feedback, and, of course, to make changes in the game. It usually takes longer than expected, but it is completely worth it.

Thanks to our partners at Schell Games for these insights into their playtesting process. For more best practices, recommendations, and example docs to help improve your VR testing process, review our recently published VR Playtesting Guide where we go deep on playtest planning + surveys, as well as facilitation and post-test data analysis.