QA101: How to Create a Plan for Video Game Testing

Are you looking to get started in testing for the gaming industry? Are you a more seasoned quality professional looking to level up your skills? Are you curious to learn more about what testing video games is really like? If you said yes to any the above, then the QA 101 series is for you! We aim to teach the basics of quality assurance by going back to the fundamentals. Each article will contain essential information to explain everything you need to know!

In our last article, we outlined the entire Test Process and gave an overview of the included stages. But you may be wondering what each part represents in more detail. Today we’re covering the integral first step; Planning & Strategy stage. To do so, we’ll also be showcasing examples of the work that you’ll likely be preparing for during this phase. Understanding why testing has been set out the way it has can bring awareness of how to optimize testing. To do this, this article goes into a little more theory of testing types.

Defining Scope

The first stage of the Test Process is to plan out your project’s strategy for how to approach testing. To do this, you need to fully understand the game that your team is making. The creation of the Game Design Document will ideally be complete and already hold all the information you need. If not, it’s time to set up meetings with the leaders on the team to drill down what the game is and all the aspects it has. As a QA specialist, you’ll need to understand how much content there is in the game. What’s the intended length of the story? How many levels there will be? How many weapons, characters, or abilities? How many features and so on. Anything that adds another dimension is something that you’ll need to account for and test. This exercise is part of a process called defining scope.

With this information in hand, it gives a better idea of how much testing will be necessary. This accounts for both how many testers you need to hire and the required time to perform the testing. Who, how and when are very important questions to answer.

Risk Analysis

As mentioned in our 7 Testing Principles article, you’re never going to be able to test everything. This is true within the constraints of a game development timeline and budget. Understanding this allows you to give feedback about the level of quality that can be expected given the allocated resources that the QA team has. There will always an aspect of prioritization because of this. If you can only test 70% of the game, what is the most important 70% to test? QA will tend to put a higher emphasis on things that are most important to the player experience. But as creating video games is a complex process, there are things that are likely to break far more often. The testing plan that you create will have to reflect this dimension also. This process is called risk analysis.

QA should do risk analysis throughout the entire project, so it’s good practice to establish how to do that early. You want to know what the indicators of high risk are and what signs to look out for. Planning a response to high risk is integral towards getting it resolved. Areas where QA are uncovering lots of bugs is a common indicator and a good response may be to schedule more testing in that area. Giving feedback to the development team to shift their focus on to the problematic area will help with getting bugs fixed faster. This results in the lowering of the risk assessment and the quality of the game to rise.

What if a lot of the game is buggy? What do you focus on first? Risk analysis considers what is most important to the experience of the end customer as well as the most egregious bugs. For example, let’s take a look at a crash; one of the worst types of bugs for video games. But if this crash is hidden away on an obscure side-quest and only happens 50% of the time, it may not be considered such a severe bug. Compare this to a bug where the main quest story will not progress unless you talk to an NPC more times that you’re supposed to. Because this is part of the main story quest, it is not skippable, so the severity is very high. Every player will experience it, while the crash is less discoverable. You may weight the risk on a bug in the critical path of the game higher, regardless if it’s not as bad as a crash.

If risk is still high as the release deadline approaches, this can be an indicator that the game needs to be delayed. It can often be also be used as a reason to ask your team to crunch.

Scheduling

Stages of development are often broken down into Pre-Alpha, Alpha, Beta and Final.

Testing during Pre-Alpha tends to be minimal because a lot of coding work is still being done. Once you progress into Alpha, more parts of the game become ready to test. Some of the game will work, some parts won’t. With guidance from the development team and schedule, QA will begin testing things. The Beta stage is often called “feature complete”. This means that the game will be fully functional but there may still be work happening for polish. The designers may still need to add things like visual elements or adjust combat balance. There will be a feature freeze meaning that developers cannot add new systems to the game after Beta has been declared. Beta is one of the busiest times for QA. You’re trying to get the game to Final, which is when the game is ready to finally release.

All these different phases need different combinations of testing approaches. Understanding what to test and when means that you can set out to write test cases and collect them into test suites for easy access.

Test Approach

To an outsider, it seems simple to say “QA just play the game”, but there’s much more to it than that. There are different types of testing that serve different functions. Finding bugs is always the name of the game. But understanding where you are in the development lifecycle and the types of bugs that are most likely to be present can help you to know what testing is going to be most efficient. Let’s look at some examples.

Generally speaking, testing earlier in the development lifecycle will focus on new features as they are implemented. This is often referred to as functional testing. For this QA will need to pay close attention to the development schedule. It’s not good practice and sometimes impossible to test features before they’re ready. Open For Testing (OFT) is a status that describes features that are ready for testing. This doesn’t always mean that they’re finished, but getting QA feedback on them would still be meaningful.

For example, a tester is checking a level and finds that a door does not open. If the level is not yet OFT, then the developers are likely to know that the door doesn’t work yet. A bug to tell them what they already know isn’t the best use of anyone’s time. However, if the level is OFT and the door is supposed to work, then reporting this bug is useful! Your planning stage needs to account for stages of OFT. You need to understand when you’re going to have access to certain parts of the game. Once you know this and the status of the work, you’ll then know the sort of testing that will be appropriate too.

Later in the development process, the implementation of new content will end. It would be less useful for QA to continue doing thorough functional testing as code is no longer being heavily modified. All broken doors should have already been found and fixed. A strategy that is often applied at this point is to use a testing type called smoke tests. QA will do surface-level testing that aims to ensure that the most important components of the game are functional. This testing covers a lot of ground in a short amount of time. It helps to verify each new build of the game before taking a closer look at particular things.

Testing Tools

The planning stage is a good point to decide what kind of tools you’d like to use during your testing process. Maybe you won’t need any at all. Maybe using automated testing is necessary to cover all the use cases. Does your game include big lobbies filled with dozens of players? You might consider getting tooling to emulate those players rather than hiring that many QA. Does your game have a large number of cosmetic skins that need to be visually checked? It may take a long time for a tester to navigate through the game and load in each individual skin. You could create a script that goes through the options, loads each skin and takes a screenshot for each one. The tester could then just open the folder of images and quickly view them all! You can get very creative with tooling and the solutions they produce. This is great to reduce the amount of time needed to run testing or verify assets.

A tool that I would say is a necessity on most large projects is a debug menu. Say you’re working on an RPG and you’re tasked with testing an area of the game that doesn’t appear until 30 hours in. Do you have to play through to that point every time? Do you play through the game and create a save point that you can access every time you do this test? How about having a menu where you pick any area in the game and it will immediately load you there? Undoubtedly the last one is the optimal solution. This debug menu can include anything that will be useful to the QA team: Infinite ammo. God mode. Spawn in enemies. Add items. Unlock levels.

Figuring out the type of tooling that would be suitable is useful to do during planning. There’s a good amount of time to create them before testing begins in earnest. Having all the tooling developed and ready to go means that you can plan your testing around it. However, it’s sometimes not obvious how necessary different types of tools are until you’re in the thick of testing. As always, you may need to return to the plan and add them later.

Defect Management

So you’ve planned what you want to test. You know when and how you’re going to test it. What do you do with the bugs you find? The obvious answer to this is that you report them! Setting up a repository system such as JIRA or Bugzilla is very common. A simple spreadsheet isn’t unheard of for smaller projects. Industry standards are often used to define what details to include in your bug report. Despite this, it’s worthwhile taking a closer look to understand whether that’s enough for your team’s needs. Would videos be better than screenshots? Is there a way to categorize the issues so that they can go to the correct developers more easily? Setting out these standards early on can save a lot of time later on.

There should also be a process defined around how devs should handle the bugs they receive, and QA’s response once they’ve been fixed. Regression testing is common to check that a fix has been successful and hasn’t caused knock-on effects. The lines of communication around this work is critical. This is why tools like JIRA are so popular, as it can send notifications for every change or comment added. After all, putting bugs into a system is not useful if no one looks at them. Having a thousand open bugs in the system is worrying, but if QA is not monitoring the level of issues, then they won’t know to raise the alarm!

Bringing it all Together

Once you’ve looked at all the different areas, you will have a much better idea of what your testing plan looks like. Using this, you will have an outline that you can take into the next phase and begin designing your test cases & suites. You will know the areas to focus during the separate phases of development and how to deal with the bugs that you find.

The point of this article isn’t to go too in-depth about specific strategies that can be used during game development. But I hope that it showcases that testing isn’t the same all the way through. The planning & strategy stage of the Test Process is important to set the QA team up for success. When a QA team is first established, planning should be a huge focus. However, this doesn’t mean that it’s frowned upon to revisit it throughout development. Checking back to see if making adjustments are necessary will depend on how things are going. Sometimes features are cut or design is changed, so you’ll have to adapt the plan to work with that. Understanding which tactics are going to be most effective at any point is imperative to allowing QA to be as useful as they can be.

Other posts in the QA101 series: