QA 101: How to write awesome Test Cases

February 19, 2021 Chloe 2 Comments

In our last article, How to create a Test Suite, we focused on creating test suites for a game project. We put together an outline of features that we should test and thought about different testing scenarios. Through this, we have a good idea of the types of ways in which to test the features of our game. It’s now time to move on to writing test cases!

What does a test case need?

First things first: What do you actually need to include in a test case? While different projects require different amounts of documentation around this, there tends to be a standard set of basics that you should always consider including.

Test Case ID

It’s a good idea to associate a Test Case ID with each of your test cases. If you’re using a test repository program, the ID will likely be added automatically. The reasoning for using an ID is because you’re likely to have many tests that are similar to each other. It’s easier to tell somebody that test 539 failed, rather than “That one test that checks the cooldown animation on the mage NPCs in the woodland forest. No, not the healers, just the black mages…”.

Test Case Title

Test titles need to be descriptive, but not exhaustive. You want a good idea of what the test is supposed to look at, but you also want to make it clear what the pass criteria are. For example, a test with the title “Login to server” is far too broad, and doesn’t suggest what the expectations are. “Server login failure with incorrect credentials” or “Login failure with offline server” are much better titles. Although these 2 tests are looking at similar things, you can tell at a glance how they differ. You do not need to write the exact expected results in the title, as these should be expanded within the test case itself.

Environment

Environment describes the hardware, software or configurations needed to run a test. For game testing, this can be as simple as whether you’re testing on a PlayStation or Xbox. This may not be necessary to include in every single test as you might set up your test suites to cover platforms separately. However, it’s important to ensure that the tester is set up correctly so that any test results are reflective of the intended testing.

Pre-conditions

Similar to environment information, there may be additional set up steps that the tester will need when running this test. Often this will be setting up when and where the test takes place, such as a level or checkpoint. Sometimes it’s essential that another test case has been run first. Whether that’s to get to a certain point in the game or ensuring another system works. For example, it’s pointless to run a test case that ensures multiplayer matchmaking works as intended if you cannot join an online lobby. Having the pre-condition that a lobby join test case passes beforehand is useful to prevent wasted time running tests that you should logically know will fail.

Test steps

These are the individual descriptive steps that a tester will perform to run the test. These need to be in-depth enough that anybody could run through them precisely. Remember that QA members aren’t the only people that look at tests. You’ll need to make sure any developer, artist, or even somebody on their first day on the team will understand the instructions! Some QA teams are close-knit, small and only have members that have been on the team for a long time. Because of this, test steps will often be written vaguely. This can leave you with situations where tests are run slightly differently by different people. This can produce different results and misunderstandings between team members. It’s also harder for new testers to learn the system and know what the correct usage is if the instructions are unclear.

Expected results

Expected results are exactly what they sound like: What should happen once you’ve run the test steps. They are the second most important aspect after the test steps themselves. You can use the expected results to describe the behaviour for each individual step or for the test as a whole. This helps to make sure that the tester knows what is supposed to happen. Sometimes a game will look good on the surface level, but ultimately isn’t fulfilling the expected design criteria.

How do you write a good test case?

Make sure all your steps are clear and concise. Sometimes including screenshots, gifs, or videos is the easiest way to be the most accurate.
Use standards or uniform styling to make the steps easier to read. This can be things like bolding names of files, levels and similar. Doing this makes it easier to scan the test and pick out the important pieces of information.
Do not make any assumptions about the testers knowledge or leave out information. You should always assume that you won’t be the one running the test. It’s good practice to get into the mindset of someone who has never even booted up the game before. This way you can ensure that you do not miss any details.
At the same time, you should try to reduce redundancy. Make sure that you’re not testing the same things over and over again in a test suite. Consider using the pre-conditions to ask a tester to run another test first if that makes more sense. You may also want to consider using tooling for your testing. Having to play through to a certain section of a level each time is redundant testing. Having a debug tool or a save file would allow you to start exactly where you need to be every time.
Having good test data, such as appropriate save files, is really useful. You need to ensure that they can be reverted back to the same state as before you ran the test (ie. don’t lose the checkpoint to a more up to date autosave). This is also true of different character unlocks, such as skins or abilities.
Remember that with testing, you need to have a healthy mix of valid & invalid testing. You want to make sure that all the intended behaviours exist, but you also want to see what happens with non-intentional behaviour. For example, does the game crash when you can’t connect to a server? What happens when you try to select a character that isn’t unlocked yet? With this, you’ll also want to check that the game gives appropriate feedback to the player.
If the test is too long you should try breaking it up into smaller tests. You could also consider adding test data or debug tools that help you to skip steps. For functional testing, it’s ideal to try to limit the number of ways that an individual test can fail. The more steps you have, the more ways that one test can fail and it becomes too broad. You would have to rerun this entire test when a fix has been made, and it can be time-consuming to go through it all to confirm the step now runs correctly.
Revise/Update often! The game will change throughout development. You’ll come across situations that you hadn’t realized before. You might want to change some of the test data that is used for particular tests. It is healthy to keep checking your test cases are good enough for the level of quality you’re looking to assure!

What does a test case look like?

We’ve done a lot of describing test cases so far, but it’s honestly easier to demonstrate these practices with an example. Let’s take another look at Tetris!

Test Case ID: 001
Test Title: Block Movement

Environment: NES Build
Preconditions: Load into the game with the following settings: 1 Player – Level 0 – Handicap 0
Test Steps:

Step	Insructions	Expected Behaviour
1	Observe falling block with no gamepad inputs	The block should move downwards incrementally allowing the player to input button presses
2	Press Left button once on gamepad (short press)	The block should move one unit to the left
3	Press & hold Left button on gamepad (long press)	The block will continue to move unit by unit to the left side of the playing area
4	Press Right button once on gamepad (short press)	The block should move one unit to the right
5	Press & hold Right button on gamepad (long press)	The block will continue to move unit by unit to the right side of the playing area
6	Press Up button on the gamepad once (short press)	The block should immediately drop to the bottom of the playing area in line with its position when the button was pressed
7	Press & hold Up button on the gamepad (long press)	The block should immediately drop to the bottom of the playing area in line with its position when the button was pressed. The newly spawned pieces should also continue to drop to the bottom of the screen as soon as they appear on screen
8	Press Down button on the gamepad (short press & long press)	The block should speed up its descent to the bottom of the playing area only the down button is pressed
9	Press A button once on gamepad (short press)	The block should rotate clockwise 90°
10	Press & hold A button on gamepad (long press)	The block should rotate clockwise 90° once, but should not continue rotating more as the button is held
11	Press B button once on gamepad (short press)	The block should rotate anti-clockwise 90°
12	Press & hold B button on gamepad (long press)	The block should rotate anti-clockwise 90° once, but should not continue rotating more as the button is held

Expected Results: All button presses should result in the expected movements, without any additional unwanted behaviour.

—

Hopefully this article gives you a much clearer idea of what test cases look like, and gives you an understanding of how to create them! They’re the backbone of every testing team, so making sure that you use good practices can really set the QA team up for success.

QA101: How to create a Test Suite

February 3, 2021 Chloe 2 Comments

Put on your lab coats, because today the QA101 series is going to get practical. No more theory, we’re going to look into the process of writing test suites! If you’re still uncertain of the theory, then take a glance back at our piece on designing your test cases and test suites. If you’re already in the know, or once you’re all caught up, you’re ready to learn how to create test suites!

Which came first; the Test Suites or the Test Cases?

It might seem backwards but it’s often easier to think about the test suite before you have the test cases to go in it. There is no right or wrong order to this, but having guiding categories helps to structure your focus when writing the tests. If you’re working on a much larger team, you can even give the different suites to different QA members. They can add the test cases for each specific area, allowing them to focus in or partner with relevant members of the dev team.

The earlier you get working on test suites for the project, the easier it is to create your test cases. If the game ends up being huge, you can slowly add to your test coverage over time. Early in development, features will be incrementally added and improved upon. While this is happening you can summarize the feature with a test suite and write new test cases as they’re added. If you join much later and there are no tests for the many features, you’ve got a lot more ground to cover to catch up!

So where do you actually start? As mentioned before, some games are huge and have an expansive amount of content to test. The best way to explain how to do this is to break down this process by using a real-life but simpler example.

Testing Tetris

Say you are responsible for testing the original Tetris. It is a simple game in terms of its features compared to AAA games today. The first thing that you want to do is break down the features of the game. If your team has a Game Design Document, this information should be pretty easy to extract. If there’s no document, then a brainstorming session can help you to quickly pull out the main points. The idea isn’t to make this perfect on the first try but to create a very high-level skeleton.

Thanks to Tetris’ ubiquity, it is often used to teach game development and design theory. Because of this, I was able to find a Game Design Documentation created by students online. Using the information from their GDD and my own brainstorming I came up with these notes in about 5 minutes:

You may notice that I’ve brainstormed some test cases at the same time as the suites. This is important because they both inform each other. I focused on the gameplay mechanics, the controls, and the visuals/aesthetic, as these were the areas first outlined in the GDD. The gameplay mechanics of Tetris all revolve around the blocks; how they move, stack, and clear. Tetris wouldn’t be Tetris if it didn’t do these things, so they’re the first thing in the brainstorm. You’ll notice how I’ve highlighted some of these lines and suggested that they fall into more than one area.

Testing one thing, many ways

Let’s take the example of “Multiple Shapes”. The reason that I call this out as falling into many areas is because there are different ways to test this core gameplay element. You want to check that these shapes can spawn in and fall down the screen correctly. Each block has a different shape and will display a different pattern/colour. You want to test that the collision for each block is correct and will stack as expected when it hits the bottom. You’ll need to test that for every shape in the game because they’ll each have different collisions. You’ll need to check to see if the stacked blocks at the bottom continue to keep the correct collision. You will need to test that the controls for rotating, dropping and moving the pieces work as expected. And so on. All these tests focus on one thing (the blocks) but they’ll be tested in different ways. They can also be classified in different ways.

Sweeping into Suites

So what is the correct way to group these tests into suites? There really is no right answer! You could categorize them as I extracted from the GDD for the original brainstorm. But as we’ve just found out, you’d have tests for blocks all over the place. This could make it more difficult to construct test runs, especially if you want to focus on a particular feature. I would more likely set up multiple test suites like this for blocks alone:

Blocks – Gameplay
Blocks – Shapes
Blocks – Controls
Blocks – Collision
Blocks – Visuals
Etc.

Then other features outside of that that I could group are:

Field of play
Sidebar
Menus

To my logic, this makes the most sense, but there may be a need to switch it up depending on how development is going. For example, say you’re testing the gameplay of the blocks and it passes all tests except for the dropping feature. It would be useful to pull that out into its own suite or subgroup so you could give it the testing focus it deserves. Remember, as a member of QA you need to focus on the important areas of gameplay, but you also need to focus on risk. If a particular feature is breaking more often than others, that’s a good reason to spend time testing it more thoroughly. I’d be interested to hear from others how they would categorize suites for Tetris!

Early in development, having such specific testing is extremely useful. Active development is the most likely time that individual features will break, so this type of testing is valuable. Yet, when you’re a week away from release, the team should have fixed most of the issues. It no longer makes sense to check the blocks individually for their shapes, patterns and collisions. Hopefully, those features haven’t broken for a long while. This is where you’d start using smoke tests that would be more akin to “Start a game and play a level, ensure blocks rotate and drop”, etc.

Never stop adding new tests

You may notice that I missed out a REALLY important gameplay feature in my brainstorm. Do you know what it is? I’ll tell you the answer at the bottom of the article. Remember that this conceptualization stage is not the finalized testing plan; it’s just a starting place to help spark ideas. It can be helpful to try out the game and see what comes up in a play session. You can then add tests every time you realize that something isn’t covered. Sometimes testing will uncover tangential but unrelated bugs that you realise you hadn’t been looking for before. You’ll find that you’ll need to add tests often through the development of a game. However, remember that testing can also never be completely exhaustive. Blocks falling at level 1 may behave differently at level 100 (especially so with the speed multiplier in play). Would there be value in checking behaviours for every single level 1-100? Maybe not.

Putting it into practice

The original Tetris is a fairly simple game by today’s standards, but even within a 5-minute brainstorm, we’ve already come up with a sizable amount of tests. Upgrading that to Tetris Effect or Tetris 99 would explode the number of tests needed! You’ll need to add testing for online play or the more advanced visual effects. Even still the tests for that would likely be smaller than a 100-hour open-world RPG!

Try it for yourself! Think about another simple old-school game (like Pacman or Snake) and try to see the test cases & suites that you can think of. As a reminder, googling the name of the game and “game design doc” will likely bring up student’s work that can help you! Let me know in the comments or in a tweet what you come up with. I often see people talking about how QA is just “trying to break the game” or “running into walls”, but they don’t analytically think about how you try to do that. This exercise is a REALLY useful one for getting into the mindset of QA.

(* So what obvious thing did I miss out? In my test plan, I didn’t think about the lose condition, eg. Game Over. Pretty integral, huh?)

QA101: How to Design your Test Cases and Test Suites

January 27, 2021 Chloe 2 Comments

So you’re working through the test process. You’ve gotten through the initial planning and information collection. You can now move on to the design stage and begin devising the testing that you’re going to use on your project. Taking the gathered information, you start writing test cases. Once you’ve done that, you should collect them into test suites. But what is a test case? Why are test suites used? For some, these questions may sound like they have obvious answers. However, there are best practices that can help you design great tests. Testing is not exhaustive, after all, so you don’t want to waste time and money on low-value work. Understanding in more depth can help keep you focused and clear about what your tests and test suites could or should contain.

Test Cases

A test case is a piece of actionable documentation that contains instructions of how to test. They generally are targeted on specific areas or features of the game. Sometimes described as “test scripts”, their aim is to confirm that the game is behaving in the intended or expected way. Think of a test script the same as a coded script. If you run through the steps of the test exactly as written, you expect to get the same results every time. Unless there is a bug!

Writing test cases step by step is useful in many ways. A game development team is usually larger than one person, so good communication is key! You want every QA tester to check things in the same way. You want to make sure to test every important feature so you’ve left no holes in coverage. The testers need to know how a feature works (or at least how it’s supposed to work). Knowing this, they can report issues without guesswork. The way to ensure all this is to write a complete test case, with individual steps and expected results. This all helps so that when a QA tester says “there is a bug in X system”, they can efficiently communicate why. They have run the same steps as others and they know how the feature is expected to work. If they’re seeing something different, they can be confident they’ve found a bug.

Depending on your current focus, you can write many test cases for the same features. For example, you want to know that your game does the correct thing AND doesn’t do the incorrect thing. You want to know that your heal spell gives 50hp to the target. You also want to know that the heal spell doesn’t give an extra 50hp if the target is already at max health. There can be a lot of creativity in coming up with design ideas for writing your test cases. Testing can never be exhaustive so it’s impossible to cover everything. While it is good to have a lot of tests, past a certain point you’re unlikely to be able to use all of them, all the time. It’s a good idea to keep your test writing efforts focused rather than writing as many tests as possible. Quality over quantity. To do this, you need to ensure that you have a wide coverage across all areas and features of the game. You should also ensure that you have enough depth to the tests you have as well.

But what should actually go into a test case? I’d like to go into more depth, but the elements are far too many to include in this article! Take a look at the following post outlining more important tips on how to write great test cases! QA 101: How to write awesome Test Cases

Test Suites

A test suite is a collection of test cases. As you’re writing more test cases, they can quickly get out of hand! So it’s important to collect them into groups to help organize them. There are many different ways to do this, but the most common is to group tests that focus on the same area. All your weapons tests. All your character animation tests. All your VFX tests. This makes it easier to construct a testing pass that focuses on those particular areas. It also makes it easier to find a particular test that you’re looking for.

Which tests do you run when you’ve been given a new build of the game that revamps the way that characters move? You’re gonna want to run your character animation suite. This also makes it easier when reporting. Which makes more sense; saying that you have 2 failing tests out of the total 1000 or saying that 2 tests in the animation suite failed? The latter immediately communicates the information more concisely. It’s easier to see where the problem is likely to be, and who to send the bug to.

It’s also not unusual to have test suites created for different types of testing. For thorough testing, your tests will likely touch on specific areas and functions. Later in development, your testing will be more of an overview so you’d want to run higher-level tests. For example, checking each specific animation performs as intended vs a quick check that a character has animations at all. This is an example of the difference between functional and smoke testing. There are a variety of other types of testing that could be defined this way, such as build verification testing, regression testing and so on.

Want to learn more about creating Test Suites? We got you covered: QA101: How to create a Test Suite

Test cases and test suites are fundamental to a lot of the day to day work of a game tester. Because of this, it’s important to have a fuller understanding and appreciation of each. It’s useful to know the essential characteristics when you’re designing testing. This helps you to be more mindful of their functionality. A well-constructed set of test suites can make or break how successful you can be at creating effective test passes.

The purpose of this article is to give you an outline of what test cases and test suites are. In our next pieces, we’ll go into much more depth about the process of collecting your test suites, plus all the things you need to think of when writing test cases!

QA101: How to Create a Plan for Video Game Testing

January 20, 2021 Chloe 2 Comments

In our last article, we outlined the entire Test Process and gave an overview of the included stages. But you may be wondering what each part represents in more detail. Today we’re covering the integral first step; Planning & Strategy stage. To do so, we’ll also be showcasing examples of the work that you’ll likely be preparing for during this phase. Understanding why testing has been set out the way it has can bring awareness of how to optimize testing. To do this, this article goes into a little more theory of testing types.

Defining Scope

The first stage of the Test Process is to plan out your project’s strategy for how to approach testing. To do this, you need to fully understand the game that your team is making. The creation of the Game Design Document will ideally be complete and already hold all the information you need. If not, it’s time to set up meetings with the leaders on the team to drill down what the game is and all the aspects it has. As a QA specialist, you’ll need to understand how much content there is in the game. What’s the intended length of the story? How many levels there will be? How many weapons, characters, or abilities? How many features and so on. Anything that adds another dimension is something that you’ll need to account for and test. This exercise is part of a process called defining scope.

With this information in hand, it gives a better idea of how much testing will be necessary. This accounts for both how many testers you need to hire and the required time to perform the testing. Who, how and when are very important questions to answer.

Risk Analysis

As mentioned in our 7 Testing Principles article, you’re never going to be able to test everything. This is true within the constraints of a game development timeline and budget. Understanding this allows you to give feedback about the level of quality that can be expected given the allocated resources that the QA team has. There will always an aspect of prioritization because of this. If you can only test 70% of the game, what is the most important 70% to test? QA will tend to put a higher emphasis on things that are most important to the player experience. But as creating video games is a complex process, there are things that are likely to break far more often. The testing plan that you create will have to reflect this dimension also. This process is called risk analysis.

QA should do risk analysis throughout the entire project, so it’s good practice to establish how to do that early. You want to know what the indicators of high risk are and what signs to look out for. Planning a response to high risk is integral towards getting it resolved. Areas where QA are uncovering lots of bugs is a common indicator and a good response may be to schedule more testing in that area. Giving feedback to the development team to shift their focus on to the problematic area will help with getting bugs fixed faster. This results in the lowering of the risk assessment and the quality of the game to rise.

What if a lot of the game is buggy? What do you focus on first? Risk analysis considers what is most important to the experience of the end customer as well as the most egregious bugs. For example, let’s take a look at a crash; one of the worst types of bugs for video games. But if this crash is hidden away on an obscure side-quest and only happens 50% of the time, it may not be considered such a severe bug. Compare this to a bug where the main quest story will not progress unless you talk to an NPC more times that you’re supposed to. Because this is part of the main story quest, it is not skippable, so the severity is very high. Every player will experience it, while the crash is less discoverable. You may weight the risk on a bug in the critical path of the game higher, regardless if it’s not as bad as a crash.

If risk is still high as the release deadline approaches, this can be an indicator that the game needs to be delayed. It can often be also be used as a reason to ask your team to crunch.

Scheduling

Stages of development are often broken down into Pre-Alpha, Alpha, Beta and Final.

Testing during Pre-Alpha tends to be minimal because a lot of coding work is still being done. Once you progress into Alpha, more parts of the game become ready to test. Some of the game will work, some parts won’t. With guidance from the development team and schedule, QA will begin testing things. The Beta stage is often called “feature complete”. This means that the game will be fully functional but there may still be work happening for polish. The designers may still need to add things like visual elements or adjust combat balance. There will be a feature freeze meaning that developers cannot add new systems to the game after Beta has been declared. Beta is one of the busiest times for QA. You’re trying to get the game to Final, which is when the game is ready to finally release.

All these different phases need different combinations of testing approaches. Understanding what to test and when means that you can set out to write test cases and collect them into test suites for easy access.

Test Approach

To an outsider, it seems simple to say “QA just play the game”, but there’s much more to it than that. There are different types of testing that serve different functions. Finding bugs is always the name of the game. But understanding where you are in the development lifecycle and the types of bugs that are most likely to be present can help you to know what testing is going to be most efficient. Let’s look at some examples.

Generally speaking, testing earlier in the development lifecycle will focus on new features as they are implemented. This is often referred to as functional testing. For this QA will need to pay close attention to the development schedule. It’s not good practice and sometimes impossible to test features before they’re ready. Open For Testing (OFT) is a status that describes features that are ready for testing. This doesn’t always mean that they’re finished, but getting QA feedback on them would still be meaningful.

For example, a tester is checking a level and finds that a door does not open. If the level is not yet OFT, then the developers are likely to know that the door doesn’t work yet. A bug to tell them what they already know isn’t the best use of anyone’s time. However, if the level is OFT and the door is supposed to work, then reporting this bug is useful! Your planning stage needs to account for stages of OFT. You need to understand when you’re going to have access to certain parts of the game. Once you know this and the status of the work, you’ll then know the sort of testing that will be appropriate too.

Later in the development process, the implementation of new content will end. It would be less useful for QA to continue doing thorough functional testing as code is no longer being heavily modified. All broken doors should have already been found and fixed. A strategy that is often applied at this point is to use a testing type called smoke tests. QA will do surface-level testing that aims to ensure that the most important components of the game are functional. This testing covers a lot of ground in a short amount of time. It helps to verify each new build of the game before taking a closer look at particular things.

Testing Tools

The planning stage is a good point to decide what kind of tools you’d like to use during your testing process. Maybe you won’t need any at all. Maybe using automated testing is necessary to cover all the use cases. Does your game include big lobbies filled with dozens of players? You might consider getting tooling to emulate those players rather than hiring that many QA. Does your game have a large number of cosmetic skins that need to be visually checked? It may take a long time for a tester to navigate through the game and load in each individual skin. You could create a script that goes through the options, loads each skin and takes a screenshot for each one. The tester could then just open the folder of images and quickly view them all! You can get very creative with tooling and the solutions they produce. This is great to reduce the amount of time needed to run testing or verify assets.

A tool that I would say is a necessity on most large projects is a debug menu. Say you’re working on an RPG and you’re tasked with testing an area of the game that doesn’t appear until 30 hours in. Do you have to play through to that point every time? Do you play through the game and create a save point that you can access every time you do this test? How about having a menu where you pick any area in the game and it will immediately load you there? Undoubtedly the last one is the optimal solution. This debug menu can include anything that will be useful to the QA team: Infinite ammo. God mode. Spawn in enemies. Add items. Unlock levels.

Figuring out the type of tooling that would be suitable is useful to do during planning. There’s a good amount of time to create them before testing begins in earnest. Having all the tooling developed and ready to go means that you can plan your testing around it. However, it’s sometimes not obvious how necessary different types of tools are until you’re in the thick of testing. As always, you may need to return to the plan and add them later.

Defect Management

So you’ve planned what you want to test. You know when and how you’re going to test it. What do you do with the bugs you find? The obvious answer to this is that you report them! Setting up a repository system such as JIRA or Bugzilla is very common. A simple spreadsheet isn’t unheard of for smaller projects. Industry standards are often used to define what details to include in your bug report. Despite this, it’s worthwhile taking a closer look to understand whether that’s enough for your team’s needs. Would videos be better than screenshots? Is there a way to categorize the issues so that they can go to the correct developers more easily? Setting out these standards early on can save a lot of time later on.

There should also be a process defined around how devs should handle the bugs they receive, and QA’s response once they’ve been fixed. Regression testing is common to check that a fix has been successful and hasn’t caused knock-on effects. The lines of communication around this work is critical. This is why tools like JIRA are so popular, as it can send notifications for every change or comment added. After all, putting bugs into a system is not useful if no one looks at them. Having a thousand open bugs in the system is worrying, but if QA is not monitoring the level of issues, then they won’t know to raise the alarm!

Bringing it all Together

Once you’ve looked at all the different areas, you will have a much better idea of what your testing plan looks like. Using this, you will have an outline that you can take into the next phase and begin designing your test cases & suites. You will know the areas to focus during the separate phases of development and how to deal with the bugs that you find.

The point of this article isn’t to go too in-depth about specific strategies that can be used during game development. But I hope that it showcases that testing isn’t the same all the way through. The planning & strategy stage of the Test Process is important to set the QA team up for success. When a QA team is first established, planning should be a huge focus. However, this doesn’t mean that it’s frowned upon to revisit it throughout development. Checking back to see if making adjustments are necessary will depend on how things are going. Sometimes features are cut or design is changed, so you’ll have to adapt the plan to work with that. Understanding which tactics are going to be most effective at any point is imperative to allowing QA to be as useful as they can be.

QA101: The 5 Main Stages of Test Process

January 13, 2021 Chloe 3 Comments

Picture the scene. You’re on the QA team of a big new game project. Development is coming along well and they’ve hired you to help manage the quality of the game. The main question is: what do you test, and how do you test it? The answer is obviously “everything”. This can be such a large and overwhelming task that you must break that down into steps.

The testing process and the strategies used tend to be similar for most titles. But specifics may differ depending on the type of game and the way that the development has been set out. This process isn’t strictly linear and you will likely need to move backwards and forwards throughout the cycle as necessary.

The 5 main stages of the testing process are as follows:

Plan

If we’ve said it once, we’ve said it a million times; QA isn’t about just testing the game. You’ve gotta plan the project strategically. When establishing QA for a new game project, you should outline out the approach(es) that you’re going to take. As mentioned before, this is very similar across all games but will change depending on the scheduling and plans of the development team. For example, you shouldn’t plan to do visual checks in early development as the artwork is likely to have placeholder and non-final textures. You can’t plan to test five levels if the team has only implemented three. The idea of the QA planning stage is to coordinate up with other leaders/producers and understand the current road map. With this information, you can interpret how the QA team can provide the most value.

QA can also feedback into the planned schedule for the game. Advising whether there is going to be enough testing or if there has been enough time scheduled to fix bugs or work on polish is important. It can be difficult to predict this at the start of a project, so revisiting the planning stage often to revise initial estimates is essential.

Design

This is where you begin to fill out the flesh of your plan. It’s all very well to say “run functional tests”, but if you haven’t written these, how do you know what to test & how to test it? How can you tell that tester A does the same type of testing as tester B? You need to have test cases written out. You can plan using the game design document to ensure that you write tests that cover all the features. You can then put these test cases into groupings called test suites. For example, you could group together tests that go over all the weapons. Tests may also be grouped together into suites that address a specific testing need for development. For example, creating a smoke test pass can be used as a daily check that the basic functionality of the game is working as expected. It would contain a wide range of different types of tests at a basic level.

However, the design phase isn’t all about writing tests. You should also lay out the specifics of the testing schedule, create tooling to make testing easier, setting up a bug logging system (like JIRA or Bugzilla), writing automation testing, and so on. The design phase can be revisited many times. This especially true for a long project where writing all the tests upfront doesn’t make sense. Game features may change so you’d either have wasted work, or you’d have to write new tests for content that didn’t initially appear in the GDD.

Implement/Execute

Video Game QA definitely puts a considerable amount of emphasis on running tests cases… Over and over and over! This is the phase that most people think of when they think about QA. This is for good cause as it is given the most time, is the most visible, and the most valuable. Implementing & executing the test plan involves not only testing but all the activities that you planned in the design stage. So you will be writing up bugs, checking through your bug logging system to manage incoming fixes, updating documentation, writing feedback. This is basically all the good stuff that helps make sure that the game is as good as it can be.

Report

The reporting phase of the testing process is where you record and report the quality of the game. How many bugs are you finding? How quickly and accurately are the fixes coming in? Do you feel confident that you’re testing enough? Is there enough time to do the testing necessary? Using all this information, is the game going to be a high enough quality to release on time? Reporting might not seem like a very important part of the process, but it is extremely crucial. Remember, QA isn’t about making sure that a game has zero bugs; it’s about having an accurate measure of exactly what the level of quality is. The development team can then use this information to make decisions on if that is high enough. Reporting is essential in that it allows producers, directors & product managers to make informed decisions about the next steps to take.

Closure

Closure, otherwise known as exit criteria, is the last part of the testing process, but that doesn’t mean the end. It’s often more accurately used to define the completion of the cycle. It makes sure that all of the goals have been addressed. This could either be by having successful test passes or ensuring that any failures have been analysed and attended to. If the current cycle’s goal was to release the game, then the exit criteria would have been to ensure the game does all the things that were defined in the game design document to an acceptable quality level. However, if this cycle was just to mark the end of a milestone in development, you would now revisit the planning stage!

The Testing Process doesn’t have to run linearly from start to finish every time. It’s important to continually check that the work you’re doing is achieving the desired results and if not, to adjust as necessary. This may mean going back to the planning and designing phases multiple times. In the next few articles of the QA 101 series, we’re going to be visiting in more detail how to plan your testing and the creations of test suites & test cases.

QA101: How to use the 7 Testing Principles in Video Game QA

January 6, 2021 Chloe 3 Comments

The International Software Testing Qualifications Board (ISTQB) is a certification board for gaining software testing credentials. Since 2004, almost 1 million professionals have taken their exams making it one of the most well-known certifications boards for QA. Though it is not as prevalent within the video game industry, a huge amount of the knowledge is transferable. This includes the 7 Testing Principles outlined in the foundations of software testing. Read on to learn more about principles to keep in mind to become a better QA professional.

Testing shows the presence of defects, not their absence

QA tests to find defects with the intention of getting them fixed.
Despite this, you can only find and fix issues in the areas that your test plan covers. There could always be bugs in any area of the game that you didn’t test enough or at all. The more thorough your testing, the more confident you can be that there are few issues.
So while testing reduces the probability of undiscovered issues, it cannot prove that there are zero bugs because:

Exhaustive testing is not possible

Given all the money and all the time in the world, you could get close to testing everything. But it’s nigh on impossible to be able to do this within the confines of active game development. There will always be restrictions on the budget or a looming deadline that you must meet.
It’s also important to remember that games and their creation are complex. Just changing one line of code could have all kinds of ramifications. Managing risk is all about understanding which areas are most likely to be affected and focusing testing effort there.
Looking at speedrunning glitches or exploits really showcases how complex games are. For example, a chain of seemingly unrelated actions can let you skip parts of the game. Often speedrunners will spend time learning how to perform frame perfect button inputs to get certain results. The average user is unlikely to come across these events in normal play. Yet because of the nature of speedrunning, runners will pursue replicating the same bug over and over, making it seem simple and obvious.
Yes, potentially a tester should have tested this, but what are the odds? There is an infinite number of button inputs. It’s impossible to test them all! Watching speedrunning strategies is actually a good way to learn new testing techniques to consider.
Because of this QA must do testing with prioritization (what is most important) and adequate risk analysis (what is most likely to break).
Similarly:

Absence of Error Fallacy

As mentioned in our previous piece, QA101: Is Quality Assurance & Testing the same thing?, “A bug free game doesn’t mean that it has high quality”.
Even if there are a low amount of bugs, it will not mean that the game is playable, fun, or contain any other of the facets that you hoped the game would have.
A game could follow the game design documentation exactly, but design can be faulty too! Perhaps your menu flow doesn’t make sense, or the game’s battle system is wildly unbalanced. These aren’t bugs, but they both affect the quality of the game.
This is why doing the right kinds of testing is important. Sitting back and looking at the game as a whole is vitally important. Playtesting is a good method for doing this. It can help the team to understand whether the game is actually ready to ship, regardless of the bug numbers.

Early Testing

Test as early as possible. Not only does this give the team more time, but it also allows the development process to run more smoothly. Reporting bugs early and as often as possible means that they are more likely to be fixed before the implementation of the next set of features.
It is faster for a developer to fix a bug in code that they wrote last week than for them to go back to it 3 months later. QA testing early is the best way to uncover bugs fast.
Finding bugs early also gives you better guidance on how likely you are to hit your deadlines. If you don’t test until the last minute, you won’t know how many bugs you might find or how bad they are. If you don’t know how many bugs there are, you won’t know how long it will take to fix them! If you test early and there are few bugs, sweet! If you test early and find lots of bugs, you have lots of time to work on them.

Defect Clustering

It is often said that 80% of defects arise from 20% of modules. What this means is that bugs appear in particular areas more often, rather than evenly spread across all the game code.
This can happen for many reasons, such as those areas having higher complexity or more dependencies than other parts of the code.
You should consider this when writing your test plan. If an area is particularly prone to breakage or has a higher level of risk, then adding more in-depth testing is a good response.
Because defects often show up in clusters, with knock-on effects in related systems, it’s also a common occurrence that a bug fix will create other bugs. This is why regression testing is so important!

Pesticide Paradox

There is a theory in agriculture, that if you use the same pesticide over and over, the insects will slowly adapt and become immune.
In the same way, repeatedly running the same tests will mean that only the focus of those tests are proven to be bug free. You must regularly review and update your test plan and test cases to ensure there aren’t gaps.
For example, devs could implement a brand new game feature but your test plan doesn’t even touch it. The plan was written before the feature existed, so there’d be no coverage. This means that you likely have no way to check its quality!
Of course, you shouldn’t completely change your test plan because that will leave you open to bugs returning in those old areas. But you should approach your auditing with an additive mindset: Where can we add more coverage? Which areas are not yet being thoroughly tested?

Testing is context-dependent

Testing for video games is different from many other types of testing. You’re unlikely to find a web tester with an Xbox controller in their hands! Software development often doesn’t cover as many different disciplines as game development, so software testing can be more focused.
Though the testing approach for different types of products may not look the same, there are lots of practices and principles that are similar… Such as the ones in this article!

You probably won’t need to get certified by the ISTQB and knowing the 7 testing principles isn’t necessary to be a tester. But understanding the practices is a great way to become a better QA professional. Keeping in mind how much quality you can actually assure and how to approach testing a game puts you well ahead of those that don’t understand these concepts.

QA101: Dissecting bugs, The 3 factors of a Video Game Glitch

December 30, 2020 Chloe Leave a comment

There are many different names for problems found in video games. Bugs, glitches, issues, defects, crashes, hangs. Some will be regularly used by players and some only by a development team internally. In this article, we’ll go over a little bit of QA theory to understand the foundation of what makes a bug and how they’re addressed within the development of a game.

A bug is generally made up of 3 parts: An error, a defect and a failure.

Errors

An error is a human mistake during coding and is what causes a bug. It could have incorrect or missing handling, typos in code, or logic implementation problems. A programmer could misunderstand the design documentation and create code that functions correctly but doesn’t do what the design asks for. The reasons that an error exist are endless. It could be anything from coding before that first coffee on a Monday morning to working through extremely complex and dependant systems. An error is what causes a defect in the first place. You may also hear this called the “root cause” of a bug. A good first line of defence to decrease errors is to have developers proofread their own work. Getting code reviews from other programmers is also a commonplace strategy to reduce the number of errors. Generally speaking though, the QA process is set up specifically to catch these problems.

Defects

A defect is the bug itself. Taking the erroneous code and discovering that it makes the game not work as intended. A tester’s main role is to observe the behaviour in the game and make reports for when it is not running as expected. With any bugs that are put into the tracking system, a developer can go back and debug the code to look for the root cause of the defect, and hopefully make a fix!

Failures

A failure is the effect of the defect. Yup, this is what most people will be talking about when they call something a bug or glitch. It could be anything from a menu item not being selectable to the game completely crashing when you talk to a specific NPC. A game crash or freeze is extremely severe and will attract a high priority for getting fixed, whereas a much less problematic issue, such as a typo in dialogue, will be low priority, and depending on how much time there is until game release, may not be fixed at all!

If these definitions all sound extremely similar to you, it’s because they are! The easiest way to differentiate between them is by considering who discovers the issue. If a developer is the first to find it, it’s an error. If the tester finds the developer’s error, it’s a defect. And then a defect is often not considered a failure unless it is experienced by a player.

QA101: What is the Difference Between Testing and Debugging?

December 23, 2020 Chloe 1 Comment

We’ve already covered what testing is and why it’s necessary during game development, but have you ever heard about debugging and wonder how it relates? The phrases are often thrown around in the same context, so it can be easy to confuse the two. Both are activities aimed at improving the quality of a game, but the work is very different.

Testing is the methodical investigation of the game via a variety of measures. The most obvious would be playing the game in the same way that a player would and observing for any bad behaviour. By doing this, a tester can (hopefully) find any bugs and report them to the team. Testing doesn’t care about why a bug exists or how to fix it, so debugging is often necessary afterwards.

Debugging is when a developer dives into the code to figure out the exact cause of a bug. This process can often lead to experimentation that points to the fix as well.

Meticulously testing through the game system helps to holistically check that the product is of good quality. A tester will be trying to discover as many issues as possible and ensure that the game is functioning as intended.

On the other side, debugging is usually only focused on the specific area of a reported issue. It also isn’t always a thorough investigation. Sometimes bugs & their fixes are obvious or there might be an error message that tells the developer explicitly what’s broken. Typos are a lot more common than you’d think. Ideally, code will be reviewed before testing by QA, so often errors can be fixed before anybody else even sees them.

Debugging is a necessary part of the effort towards developing a bug free product, but testing is integral in guiding the direction. Neither one is more important than the other. Debugging is a response to a bug report. Testing is not useful if you are not going to investigate and fix issues. This two-punch combo is the standard path towards creating a bug free and high quality game.

QA101: Are Quality Assurance and Testing the same thing?

December 16, 2020 Chloe 1 Comment

You may have heard different titles for people working within QA. Game Tester, Quality Assurance Engineer, Quality Designer; the list goes on! But what do these titles actually mean? Some of these jobs may focus only on testing while others work in QA in a bigger way; Testing and QA are not the same thing! Read on to learn how a good QA team, with the help of testing, can contribute to the production of a video game.

Let’s set the record straight: Quality Assurance (QA) is not a single activity. It is a group of many different types of work. Testing is the most visible and widely used of that work, but it’s still only one task within the larger sphere of QA.

Here are some examples of QA work:

QUALITY ASSURANCE ACTIVITIES
A non-exhaustive list of quality assurance activities that may be used during game development

QA Work:
Quality Strategy
Defining Standards, Procedures & Processes
Writing & Reviewing Requirements
Risk Analysis & Management
Developing Tools
Data Analytics
Writing & Auditing Documentation

Testing:
Smoke Testing
Functional Testing
Integration Testing
Performance Testing
Playtesting

Testing Adjacent:
Creating Test Plans
Creating Test Suites
Writing Test Cases
Writing & Triaging Bugs
Developing Automated Tests

Simply put, Quality Assurance is about the process that helps to ensure a game has a great level of quality. Testing is just a single tool in the toolbox that helps to look at the game to check what its current state is. QA can tell you when and where to test a game. Testing tells you what bugs or issues there are. But testing is not going to help improve the game if you don’t write up the bugs and try to fix them!

It’s important to note that a bug free game doesn’t mean that it has high quality. What if all the text is lime green comic sans on a hot pink background? Does the game have the exact same experience across PC, PlayStation & Xbox? There are many aspects to creating a high quality game, a lot of which are subjective. Is it ok that a game runs at 60fps only 80% of the time, but dips at certain points? Quality standards like this must be defined and a QA team can work on monitoring and reporting on this.

It’s also useful to apply Quality Assurance methods to the process of developing a video game itself. What do you think would happen if the testing team didn’t get to test the game until the very end of development? I’m willing to bet there would be a massive list of issues that the dev team would potentially have to fix before release. This is extremely poor practice and could lead to a buggy release, a huge day 1 patch or even the need to delay the release date. A much better QA plan would be to regularly playtest the game throughout development.

When you have QA engaged earlier, you have a better idea of whether you’ll hit your milestone targets, but they can also provide insight to support preventing big issues from happening in the first place.

Different QA staff will take on these different activities depending on how much experience they have. An entry-level Tester will likely only write & perform testing while working with developers to ensure fixes for any bugs found. A Quality Designer however would likely be involved with higher-level analysis to understand how code is being implemented. They would then design processes & procedures to support the quality of the project. A Quality Assurance Engineer may even write automated testing scripts to run testing without the need for a manual tester.

Different companies use QA and job titles in different ways, so it’s worth noting that sometimes the job title could mean all or none of these things! Needless to say, working in QA and/or testing is much more than “playing video games for a living”!

What does a test case need?

Test Case ID

Test Case Title

Environment

Pre-conditions

Test steps

Expected results

How do you write a good test case?

What does a test case look like?

Other posts in the QA101 series:

Which came first; the Test Suites or the Test Cases?

Testing Tetris

Testing one thing, many ways

Sweeping into Suites

Never stop adding new tests

Putting it into practice

Other posts in the QA101 series:

Test Cases

Test Suites

Other posts in the QA101 series:

Defining Scope

Risk Analysis

Scheduling

Test Approach

Testing Tools

Defect Management

Bringing it all Together

Other posts in the QA101 series:

Plan

Design

Implement/Execute

Report

Closure

Other posts in the QA101 series:

Testing shows the presence of defects, not their absence

Exhaustive testing is not possible

Absence of Error Fallacy

Early Testing

Defect Clustering

Pesticide Paradox

Testing is context-dependent

Other posts in the QA101 series:

Errors

Defects

Failures

Other posts in the QA101 series:

Other posts in the QA101 series:

Other posts in the QA101 series:

Bugs

Stability/Reliability

Qualitative Feedback

Fulfilling Design Criteria

Developers won’t test as well as QA professionals

Player Testing is not as useful as it seems

Money

Learn more about Video Game QA