Tag Archives: game dev

qa101

QA 101: How to write awesome Test Cases

2 Comments

Are you looking to get started in testing for the gaming industry? Are you a more seasoned quality professional looking to level up your skills? Are you curious to learn more about what testing video games is really like? If you said yes to any the above, then the QA 101 series is for you! We aim to teach the basics of quality assurance by going back to the fundamentals. Each article will contain essential information to explain everything you need to know!

In our last article, How to create a Test Suite, we focused on creating test suites for a game project. We put together an outline of features that we should test and thought about different testing scenarios. Through this, we have a good idea of the types of ways in which to test the features of our game. It’s now time to move on to writing test cases!

What does a test case need?

First things first: What do you actually need to include in a test case? While different projects require different amounts of documentation around this, there tends to be a standard set of basics that you should always consider including.

Test Case ID

It’s a good idea to associate a Test Case ID with each of your test cases. If you’re using a test repository program, the ID will likely be added automatically. The reasoning for using an ID is because you’re likely to have many tests that are similar to each other. It’s easier to tell somebody that test 539 failed, rather than “That one test that checks the cooldown animation on the mage NPCs in the woodland forest. No, not the healers, just the black mages…”.

Test Case Title

Test titles need to be descriptive, but not exhaustive. You want a good idea of what the test is supposed to look at, but you also want to make it clear what the pass criteria are. For example, a test with the title “Login to server” is far too broad, and doesn’t suggest what the expectations are. “Server login failure with incorrect credentials” or “Login failure with offline server” are much better titles. Although these 2 tests are looking at similar things, you can tell at a glance how they differ. You do not need to write the exact expected results in the title, as these should be expanded within the test case itself.

Environment

Environment describes the hardware, software or configurations needed to run a test. For game testing, this can be as simple as whether you’re testing on a PlayStation or Xbox. This may not be necessary to include in every single test as you might set up your test suites to cover platforms separately. However, it’s important to ensure that the tester is set up correctly so that any test results are reflective of the intended testing.

Pre-conditions

Similar to environment information, there may be additional set up steps that the tester will need when running this test. Often this will be setting up when and where the test takes place, such as a level or checkpoint. Sometimes it’s essential that another test case has been run first. Whether that’s to get to a certain point in the game or ensuring another system works. For example, it’s pointless to run a test case that ensures multiplayer matchmaking works as intended if you cannot join an online lobby. Having the pre-condition that a lobby join test case passes beforehand is useful to prevent wasted time running tests that you should logically know will fail.

Test steps

These are the individual descriptive steps that a tester will perform to run the test. These need to be in-depth enough that anybody could run through them precisely. Remember that QA members aren’t the only people that look at tests. You’ll need to make sure any developer, artist, or even somebody on their first day on the team will understand the instructions! Some QA teams are close-knit, small and only have members that have been on the team for a long time. Because of this, test steps will often be written vaguely. This can leave you with situations where tests are run slightly differently by different people. This can produce different results and misunderstandings between team members. It’s also harder for new testers to learn the system and know what the correct usage is if the instructions are unclear.

Expected results

Expected results are exactly what they sound like: What should happen once you’ve run the test steps. They are the second most important aspect after the test steps themselves. You can use the expected results to describe the behaviour for each individual step or for the test as a whole. This helps to make sure that the tester knows what is supposed to happen. Sometimes a game will look good on the surface level, but ultimately isn’t fulfilling the expected design criteria.

How do you write a good test case?

  • Make sure all your steps are clear and concise. Sometimes including screenshots, gifs, or videos is the easiest way to be the most accurate.
  • Use standards or uniform styling to make the steps easier to read. This can be things like bolding names of files, levels and similar. Doing this makes it easier to scan the test and pick out the important pieces of information.
  • Do not make any assumptions about the testers knowledge or leave out information. You should always assume that you won’t be the one running the test. It’s good practice to get into the mindset of someone who has never even booted up the game before. This way you can ensure that you do not miss any details.
  • At the same time, you should try to reduce redundancy. Make sure that you’re not testing the same things over and over again in a test suite. Consider using the pre-conditions to ask a tester to run another test first if that makes more sense. You may also want to consider using tooling for your testing. Having to play through to a certain section of a level each time is redundant testing. Having a debug tool or a save file would allow you to start exactly where you need to be every time.
  • Having good test data, such as appropriate save files, is really useful. You need to ensure that they can be reverted back to the same state as before you ran the test (ie. don’t lose the checkpoint to a more up to date autosave). This is also true of different character unlocks, such as skins or abilities.
  • Remember that with testing, you need to have a healthy mix of valid & invalid testing. You want to make sure that all the intended behaviours exist, but you also want to see what happens with non-intentional behaviour. For example, does the game crash when you can’t connect to a server? What happens when you try to select a character that isn’t unlocked yet? With this, you’ll also want to check that the game gives appropriate feedback to the player.
  • If the test is too long you should try breaking it up into smaller tests. You could also consider adding test data or debug tools that help you to skip steps. For functional testing, it’s ideal to try to limit the number of ways that an individual test can fail. The more steps you have, the more ways that one test can fail and it becomes too broad. You would have to rerun this entire test when a fix has been made, and it can be time-consuming to go through it all to confirm the step now runs correctly.
  • Revise/Update often! The game will change throughout development. You’ll come across situations that you hadn’t realized before. You might want to change some of the test data that is used for particular tests. It is healthy to keep checking your test cases are good enough for the level of quality you’re looking to assure!

What does a test case look like?

We’ve done a lot of describing test cases so far, but it’s honestly easier to demonstrate these practices with an example. Let’s take another look at Tetris!

Test Case ID: 001
Test Title: Block Movement

Environment: NES Build
Preconditions: Load into the game with the following settings: 1 Player – Level 0 – Handicap 0
Test Steps:

StepInsructionsExpected Behaviour
1Observe falling block with no gamepad inputsThe block should move downwards incrementally allowing the player to input button presses
2Press Left button once on gamepad (short press)The block should move one unit to the left
3Press & hold Left button on gamepad (long press)The block will continue to move unit by unit to the left side of the playing area
4Press Right button once on gamepad (short press)The block should move one unit to the right
5Press & hold Right button on gamepad (long press)The block will continue to move unit by unit to the right side of the playing area
6Press Up button on the gamepad once (short press)The block should immediately drop to the bottom of the playing area in line with its position when the button was pressed
7Press & hold Up button on the gamepad (long press)The block should immediately drop to the bottom of the playing area in line with its position when the button was pressed. The newly spawned pieces should also continue to drop to the bottom of the screen as soon as they appear on screen
8Press Down button on the gamepad (short press & long press)The block should speed up its descent to the bottom of the playing area only the down button is pressed
9Press A button once on gamepad (short press)The block should rotate clockwise 90° 
10Press & hold A button on gamepad (long press)The block should rotate clockwise 90° once, but should not continue rotating more as the button is held
11Press B button once on gamepad (short press)The block should rotate anti-clockwise 90° 
12Press & hold B button on gamepad (long press)The block should rotate anti-clockwise 90° once, but should not continue rotating more as the button is held

Expected Results: All button presses should result in the expected movements, without any additional unwanted behaviour.

—

Hopefully this article gives you a much clearer idea of what test cases look like, and gives you an understanding of how to create them! They’re the backbone of every testing team, so making sure that you use good practices can really set the QA team up for success.

Other posts in the QA101 series:

qa101

QA101: How to create a Test Suite

2 Comments

Are you looking to get started in testing for the gaming industry? Are you a more seasoned quality professional looking to level up your skills? Are you curious to learn more about what testing video games is really like? If you said yes to any the above, then the QA 101 series is for you! We aim to teach the basics of quality assurance by going back to the fundamentals. Each article will contain essential information to explain everything you need to know!

Put on your lab coats, because today the QA101 series is going to get practical. No more theory, we’re going to look into the process of writing test suites! If you’re still uncertain of the theory, then take a glance back at our piece on designing your test cases and test suites. If you’re already in the know, or once you’re all caught up, you’re ready to learn how to create test suites!

Which came first; the Test Suites or the Test Cases?

It might seem backwards but it’s often easier to think about the test suite before you have the test cases to go in it. There is no right or wrong order to this, but having guiding categories helps to structure your focus when writing the tests. If you’re working on a much larger team, you can even give the different suites to different QA members. They can add the test cases for each specific area, allowing them to focus in or partner with relevant members of the dev team.

The earlier you get working on test suites for the project, the easier it is to create your test cases. If the game ends up being huge, you can slowly add to your test coverage over time. Early in development, features will be incrementally added and improved upon. While this is happening you can summarize the feature with a test suite and write new test cases as they’re added. If you join much later and there are no tests for the many features, you’ve got a lot more ground to cover to catch up!

So where do you actually start? As mentioned before, some games are huge and have an expansive amount of content to test. The best way to explain how to do this is to break down this process by using a real-life but simpler example.

Testing Tetris

Say you are responsible for testing the original Tetris. It is a simple game in terms of its features compared to AAA games today. The first thing that you want to do is break down the features of the game. If your team has a Game Design Document, this information should be pretty easy to extract. If there’s no document, then a brainstorming session can help you to quickly pull out the main points. The idea isn’t to make this perfect on the first try but to create a very high-level skeleton.

Thanks to Tetris’ ubiquity, it is often used to teach game development and design theory. Because of this, I was able to find a Game Design Documentation created by students online. Using the information from their GDD and my own brainstorming I came up with these notes in about 5 minutes:

You may notice that I’ve brainstormed some test cases at the same time as the suites. This is important because they both inform each other. I focused on the gameplay mechanics, the controls, and the visuals/aesthetic, as these were the areas first outlined in the GDD. The gameplay mechanics of Tetris all revolve around the blocks; how they move, stack, and clear. Tetris wouldn’t be Tetris if it didn’t do these things, so they’re the first thing in the brainstorm. You’ll notice how I’ve highlighted some of these lines and suggested that they fall into more than one area.

Testing one thing, many ways

Let’s take the example of “Multiple Shapes”. The reason that I call this out as falling into many areas is because there are different ways to test this core gameplay element. You want to check that these shapes can spawn in and fall down the screen correctly. Each block has a different shape and will display a different pattern/colour. You want to test that the collision for each block is correct and will stack as expected when it hits the bottom. You’ll need to test that for every shape in the game because they’ll each have different collisions. You’ll need to check to see if the stacked blocks at the bottom continue to keep the correct collision. You will need to test that the controls for rotating, dropping and moving the pieces work as expected. And so on. All these tests focus on one thing (the blocks) but they’ll be tested in different ways. They can also be classified in different ways.

Sweeping into Suites

So what is the correct way to group these tests into suites? There really is no right answer! You could categorize them as I extracted from the GDD for the original brainstorm. But as we’ve just found out, you’d have tests for blocks all over the place. This could make it more difficult to construct test runs, especially if you want to focus on a particular feature. I would more likely set up multiple test suites like this for blocks alone:

  • Blocks – Gameplay
  • Blocks – Shapes
  • Blocks – Controls
  • Blocks – Collision
  • Blocks – Visuals
  • Etc.

Then other features outside of that that I could group are:

  • Field of play
  • Sidebar
  • Menus

To my logic, this makes the most sense, but there may be a need to switch it up depending on how development is going. For example, say you’re testing the gameplay of the blocks and it passes all tests except for the dropping feature. It would be useful to pull that out into its own suite or subgroup so you could give it the testing focus it deserves. Remember, as a member of QA you need to focus on the important areas of gameplay, but you also need to focus on risk. If a particular feature is breaking more often than others, that’s a good reason to spend time testing it more thoroughly. I’d be interested to hear from others how they would categorize suites for Tetris!

Early in development, having such specific testing is extremely useful. Active development is the most likely time that individual features will break, so this type of testing is valuable. Yet, when you’re a week away from release, the team should have fixed most of the issues. It no longer makes sense to check the blocks individually for their shapes, patterns and collisions. Hopefully, those features haven’t broken for a long while. This is where you’d start using smoke tests that would be more akin to “Start a game and play a level, ensure blocks rotate and drop”, etc.

Never stop adding new tests

You may notice that I missed out a REALLY important gameplay feature in my brainstorm. Do you know what it is? I’ll tell you the answer at the bottom of the article. Remember that this conceptualization stage is not the finalized testing plan; it’s just a starting place to help spark ideas. It can be helpful to try out the game and see what comes up in a play session. You can then add tests every time you realize that something isn’t covered. Sometimes testing will uncover tangential but unrelated bugs that you realise you hadn’t been looking for before. You’ll find that you’ll need to add tests often through the development of a game. However, remember that testing can also never be completely exhaustive. Blocks falling at level 1 may behave differently at level 100 (especially so with the speed multiplier in play). Would there be value in checking behaviours for every single level 1-100? Maybe not.

Putting it into practice

The original Tetris is a fairly simple game by today’s standards, but even within a 5-minute brainstorm, we’ve already come up with a sizable amount of tests. Upgrading that to Tetris Effect or Tetris 99 would explode the number of tests needed! You’ll need to add testing for online play or the more advanced visual effects. Even still the tests for that would likely be smaller than a 100-hour open-world RPG!

Try it for yourself! Think about another simple old-school game (like Pacman or Snake) and try to see the test cases & suites that you can think of. As a reminder, googling the name of the game and “game design doc” will likely bring up student’s work that can help you! Let me know in the comments or in a tweet what you come up with. I often see people talking about how QA is just “trying to break the game” or “running into walls”, but they don’t analytically think about how you try to do that. This exercise is a REALLY useful one for getting into the mindset of QA.

(* So what obvious thing did I miss out? In my test plan, I didn’t think about the lose condition, eg. Game Over. Pretty integral, huh?)

Other posts in the QA101 series:
qa101

QA101: How to Design your Test Cases and Test Suites

2 Comments

Are you looking to get started in testing for the gaming industry? Are you a more seasoned quality professional looking to level up your skills? Are you curious to learn more about what testing video games is really like? If you said yes to any the above, then the QA 101 series is for you! We aim to teach the basics of quality assurance by going back to the fundamentals. Each article will contain essential information to explain everything you need to know!

So you’re working through the test process. You’ve gotten through the initial planning and information collection. You can now move on to the design stage and begin devising the testing that you’re going to use on your project. Taking the gathered information, you start writing test cases. Once you’ve done that, you should collect them into test suites. But what is a test case? Why are test suites used? For some, these questions may sound like they have obvious answers. However, there are best practices that can help you design great tests. Testing is not exhaustive, after all, so you don’t want to waste time and money on low-value work. Understanding in more depth can help keep you focused and clear about what your tests and test suites could or should contain. 

Test Cases

A test case is a piece of actionable documentation that contains instructions of how to test. They generally are targeted on specific areas or features of the game. Sometimes described as “test scripts”, their aim is to confirm that the game is behaving in the intended or expected way.  Think of a test script the same as a coded script. If you run through the steps of the test exactly as written, you expect to get the same results every time. Unless there is a bug!

Writing test cases step by step is useful in many ways. A game development team is usually larger than one person, so good communication is key! You want every QA tester to check things in the same way. You want to make sure to test every important feature so you’ve left no holes in coverage. The testers need to know how a feature works (or at least how it’s supposed to work). Knowing this, they can report issues without guesswork. The way to ensure all this is to write a complete test case, with individual steps and expected results.  This all helps so that when a QA tester says “there is a bug in X system”, they can efficiently communicate why. They have run the same steps as others and they know how the feature is expected to work. If they’re seeing something different, they can be confident they’ve found a bug.

Depending on your current focus, you can write many test cases for the same features. For example, you want to know that your game does the correct thing AND doesn’t do the incorrect thing. You want to know that your heal spell gives 50hp to the target. You also want to know that the heal spell doesn’t give an extra 50hp if the target is already at max health. There can be a lot of creativity in coming up with design ideas for writing your test cases. Testing can never be exhaustive so it’s impossible to cover everything. While it is good to have a lot of tests, past a certain point you’re unlikely to be able to use all of them, all the time. It’s a good idea to keep your test writing efforts focused rather than writing as many tests as possible. Quality over quantity. To do this, you need to ensure that you have a wide coverage across all areas and features of the game. You should also ensure that you have enough depth to the tests you have as well.

But what should actually go into a test case? I’d like to go into more depth, but the elements are far too many to include in this article! Take a look at the following post outlining more important tips on how to write great test cases! QA 101: How to write awesome Test Cases

Test Suites

A test suite is a collection of test cases. As you’re writing more test cases, they can quickly get out of hand! So it’s important to collect them into groups to help organize them. There are many different ways to do this, but the most common is to group tests that focus on the same area. All your weapons tests. All your character animation tests. All your VFX tests. This makes it easier to construct a testing pass that focuses on those particular areas. It also makes it easier to find a particular test that you’re looking for. 

Which tests do you run when you’ve been given a new build of the game that revamps the way that characters move? You’re gonna want to run your character animation suite. This also makes it easier when reporting. Which makes more sense; saying that you have 2 failing tests out of the total 1000 or saying that 2 tests in the animation suite failed? The latter immediately communicates the information more concisely. It’s easier to see where the problem is likely to be, and who to send the bug to.

It’s also not unusual to have test suites created for different types of testing. For thorough testing, your tests will likely touch on specific areas and functions. Later in development, your testing will be more of an overview so you’d want to run higher-level tests. For example, checking each specific animation performs as intended vs a quick check that a character has animations at all. This is an example of the difference between functional and smoke testing. There are a variety of other types of testing that could be defined this way, such as build verification testing, regression testing and so on.

Want to learn more about creating Test Suites? We got you covered: QA101: How to create a Test Suite

Test cases and test suites are fundamental to a lot of the day to day work of a game tester. Because of this, it’s important to have a fuller understanding and appreciation of each. It’s useful to know the essential characteristics when you’re designing testing. This helps you to be more mindful of their functionality. A well-constructed set of test suites can make or break how successful you can be at creating effective test passes.

The purpose of this article is to give you an outline of what test cases and test suites are. In our next pieces, we’ll go into much more depth about the process of collecting your test suites, plus all the things you need to think of when writing test cases!

Other posts in the QA101 series:
qa101

QA101: How to Create a Plan for Video Game Testing

2 Comments

Are you looking to get started in testing for the gaming industry? Are you a more seasoned quality professional looking to level up your skills? Are you curious to learn more about what testing video games is really like? If you said yes to any the above, then the QA 101 series is for you! We aim to teach the basics of quality assurance by going back to the fundamentals. Each article will contain essential information to explain everything you need to know!

In our last article, we outlined the entire Test Process and gave an overview of the included stages. But you may be wondering what each part represents in more detail. Today we’re covering the integral first step; Planning & Strategy stage. To do so, we’ll also be showcasing examples of the work that you’ll likely be preparing for during this phase. Understanding why testing has been set out the way it has can bring awareness of how to optimize testing. To do this, this article goes into a little more theory of testing types.

Defining Scope

The first stage of the Test Process is to plan out your project’s strategy for how to approach testing. To do this, you need to fully understand the game that your team is making. The creation of the Game Design Document will ideally be complete and already hold all the information you need. If not, it’s time to set up meetings with the leaders on the team to drill down what the game is and all the aspects it has. As a QA specialist, you’ll need to understand how much content there is in the game. What’s the intended length of the story? How many levels there will be? How many weapons, characters, or abilities? How many features and so on. Anything that adds another dimension is something that you’ll need to account for and test. This exercise is part of a process called defining scope.

With this information in hand, it gives a better idea of how much testing will be necessary. This accounts for both how many testers you need to hire and the required time to perform the testing. Who, how and when are very important questions to answer.

Risk Analysis

As mentioned in our 7 Testing Principles article, you’re never going to be able to test everything. This is true within the constraints of a game development timeline and budget. Understanding this allows you to give feedback about the level of quality that can be expected given the allocated resources that the QA team has. There will always an aspect of prioritization because of this. If you can only test 70% of the game, what is the most important 70% to test? QA will tend to put a higher emphasis on things that are most important to the player experience. But as creating video games is a complex process, there are things that are likely to break far more often. The testing plan that you create will have to reflect this dimension also. This process is called risk analysis.

QA should do risk analysis throughout the entire project, so it’s good practice to establish how to do that early. You want to know what the indicators of high risk are and what signs to look out for. Planning a response to high risk is integral towards getting it resolved. Areas where QA are uncovering lots of bugs is a common indicator and a good response may be to schedule more testing in that area. Giving feedback to the development team to shift their focus on to the problematic area will help with getting bugs fixed faster. This results in the lowering of the risk assessment and the quality of the game to rise.

What if a lot of the game is buggy? What do you focus on first? Risk analysis considers what is most important to the experience of the end customer as well as the most egregious bugs. For example, let’s take a look at a crash; one of the worst types of bugs for video games. But if this crash is hidden away on an obscure side-quest and only happens 50% of the time, it may not be considered such a severe bug. Compare this to a bug where the main quest story will not progress unless you talk to an NPC more times that you’re supposed to. Because this is part of the main story quest, it is not skippable, so the severity is very high. Every player will experience it, while the crash is less discoverable. You may weight the risk on a bug in the critical path of the game higher, regardless if it’s not as bad as a crash.

If risk is still high as the release deadline approaches, this can be an indicator that the game needs to be delayed. It can often be also be used as a reason to ask your team to crunch.

Scheduling

Stages of development are often broken down into Pre-Alpha, Alpha, Beta and Final.

Testing during Pre-Alpha tends to be minimal because a lot of coding work is still being done. Once you progress into Alpha, more parts of the game become ready to test. Some of the game will work, some parts won’t. With guidance from the development team and schedule, QA will begin testing things. The Beta stage is often called “feature complete”. This means that the game will be fully functional but there may still be work happening for polish. The designers may still need to add things like visual elements or adjust combat balance. There will be a feature freeze meaning that developers cannot add new systems to the game after Beta has been declared. Beta is one of the busiest times for QA. You’re trying to get the game to Final, which is when the game is ready to finally release.

All these different phases need different combinations of testing approaches. Understanding what to test and when means that you can set out to write test cases and collect them into test suites for easy access.

Test Approach

To an outsider, it seems simple to say “QA just play the game”, but there’s much more to it than that. There are different types of testing that serve different functions. Finding bugs is always the name of the game. But understanding where you are in the development lifecycle and the types of bugs that are most likely to be present can help you to know what testing is going to be most efficient. Let’s look at some examples.

Generally speaking, testing earlier in the development lifecycle will focus on new features as they are implemented. This is often referred to as functional testing. For this QA will need to pay close attention to the development schedule. It’s not good practice and sometimes impossible to test features before they’re ready. Open For Testing (OFT) is a status that describes features that are ready for testing. This doesn’t always mean that they’re finished, but getting QA feedback on them would still be meaningful.

For example, a tester is checking a level and finds that a door does not open. If the level is not yet OFT, then the developers are likely to know that the door doesn’t work yet. A bug to tell them what they already know isn’t the best use of anyone’s time. However, if the level is OFT and the door is supposed to work, then reporting this bug is useful! Your planning stage needs to account for stages of OFT. You need to understand when you’re going to have access to certain parts of the game. Once you know this and the status of the work, you’ll then know the sort of testing that will be appropriate too.

Later in the development process, the implementation of new content will end. It would be less useful for QA to continue doing thorough functional testing as code is no longer being heavily modified. All broken doors should have already been found and fixed. A strategy that is often applied at this point is to use a testing type called smoke tests. QA will do surface-level testing that aims to ensure that the most important components of the game are functional. This testing covers a lot of ground in a short amount of time. It helps to verify each new build of the game before taking a closer look at particular things.

Testing Tools

The planning stage is a good point to decide what kind of tools you’d like to use during your testing process. Maybe you won’t need any at all. Maybe using automated testing is necessary to cover all the use cases. Does your game include big lobbies filled with dozens of players? You might consider getting tooling to emulate those players rather than hiring that many QA. Does your game have a large number of cosmetic skins that need to be visually checked? It may take a long time for a tester to navigate through the game and load in each individual skin. You could create a script that goes through the options, loads each skin and takes a screenshot for each one. The tester could then just open the folder of images and quickly view them all! You can get very creative with tooling and the solutions they produce. This is great to reduce the amount of time needed to run testing or verify assets.

A tool that I would say is a necessity on most large projects is a debug menu. Say you’re working on an RPG and you’re tasked with testing an area of the game that doesn’t appear until 30 hours in. Do you have to play through to that point every time? Do you play through the game and create a save point that you can access every time you do this test? How about having a menu where you pick any area in the game and it will immediately load you there? Undoubtedly the last one is the optimal solution. This debug menu can include anything that will be useful to the QA team: Infinite ammo. God mode. Spawn in enemies. Add items. Unlock levels.

Figuring out the type of tooling that would be suitable is useful to do during planning. There’s a good amount of time to create them before testing begins in earnest. Having all the tooling developed and ready to go means that you can plan your testing around it. However, it’s sometimes not obvious how necessary different types of tools are until you’re in the thick of testing. As always, you may need to return to the plan and add them later.

Defect Management

So you’ve planned what you want to test. You know when and how you’re going to test it. What do you do with the bugs you find? The obvious answer to this is that you report them! Setting up a repository system such as JIRA or Bugzilla is very common. A simple spreadsheet isn’t unheard of for smaller projects. Industry standards are often used to define what details to include in your bug report. Despite this, it’s worthwhile taking a closer look to understand whether that’s enough for your team’s needs. Would videos be better than screenshots? Is there a way to categorize the issues so that they can go to the correct developers more easily? Setting out these standards early on can save a lot of time later on.

There should also be a process defined around how devs should handle the bugs they receive, and QA’s response once they’ve been fixed. Regression testing is common to check that a fix has been successful and hasn’t caused knock-on effects. The lines of communication around this work is critical. This is why tools like JIRA are so popular, as it can send notifications for every change or comment added. After all, putting bugs into a system is not useful if no one looks at them. Having a thousand open bugs in the system is worrying, but if QA is not monitoring the level of issues, then they won’t know to raise the alarm!

Bringing it all Together

Once you’ve looked at all the different areas, you will have a much better idea of what your testing plan looks like. Using this, you will have an outline that you can take into the next phase and begin designing your test cases & suites. You will know the areas to focus during the separate phases of development and how to deal with the bugs that you find.

The point of this article isn’t to go too in-depth about specific strategies that can be used during game development. But I hope that it showcases that testing isn’t the same all the way through. The planning & strategy stage of the Test Process is important to set the QA team up for success. When a QA team is first established, planning should be a huge focus. However, this doesn’t mean that it’s frowned upon to revisit it throughout development. Checking back to see if making adjustments are necessary will depend on how things are going. Sometimes features are cut or design is changed, so you’ll have to adapt the plan to work with that. Understanding which tactics are going to be most effective at any point is imperative to allowing QA to be as useful as they can be.

Other posts in the QA101 series:
qa101

QA101: The 5 Main Stages of Test Process

3 Comments

Are you looking to get started in testing for the gaming industry? Are you a more seasoned quality professional looking to level up your skills? Are you curious to learn more about what testing video games is really like? If you said yes to any the above, then the QA 101 series is for you! We aim to teach the basics of quality assurance by going back to the fundamentals. Each article will contain essential information to explain everything you need to know!

Picture the scene. You’re on the QA team of a big new game project. Development is coming along well and they’ve hired you to help manage the quality of the game. The main question is: what do you test, and how do you test it? The answer is obviously “everything”. This can be such a large and overwhelming task that you must break that down into steps.

The testing process and the strategies used tend to be similar for most titles. But specifics may differ depending on the type of game and the way that the development has been set out. This process isn’t strictly linear and you will likely need to move backwards and forwards throughout the cycle as necessary.

The 5 main stages of the testing process are as follows:

5 main stages of the Test Process: Plan Design Implement/Execute Report Closure

Plan

If we’ve said it once, we’ve said it a million times; QA isn’t about just testing the game. You’ve gotta plan the project strategically. When establishing QA for a new game project, you should outline out the approach(es) that you’re going to take. As mentioned before, this is very similar across all games but will change depending on the scheduling and plans of the development team. For example, you shouldn’t plan to do visual checks in early development as the artwork is likely to have placeholder and non-final textures. You can’t plan to test five levels if the team has only implemented three. The idea of the QA planning stage is to coordinate up with other leaders/producers and understand the current road map. With this information, you can interpret how the QA team can provide the most value.

QA can also feedback into the planned schedule for the game. Advising whether there is going to be enough testing or if there has been enough time scheduled to fix bugs or work on polish is important. It can be difficult to predict this at the start of a project, so revisiting the planning stage often to revise initial estimates is essential.

Design

This is where you begin to fill out the flesh of your plan. It’s all very well to say “run functional tests”, but if you haven’t written these, how do you know what to test & how to test it? How can you tell that tester A does the same type of testing as tester B? You need to have test cases written out. You can plan using the game design document to ensure that you write tests that cover all the features. You can then put these test cases into groupings called test suites. For example, you could group together tests that go over all the weapons. Tests may also be grouped together into suites that address a specific testing need for development. For example, creating a smoke test pass can be used as a daily check that the basic functionality of the game is working as expected. It would contain a wide range of different types of tests at a basic level.

However, the design phase isn’t all about writing tests. You should also lay out the specifics of the testing schedule, create tooling to make testing easier, setting up a bug logging system (like JIRA or Bugzilla), writing automation testing, and so on. The design phase can be revisited many times. This especially true for a long project where writing all the tests upfront doesn’t make sense. Game features may change so you’d either have wasted work, or you’d have to write new tests for content that didn’t initially appear in the GDD.

Implement/Execute

Video Game QA definitely puts a considerable amount of emphasis on running tests cases… Over and over and over! This is the phase that most people think of when they think about QA. This is for good cause as it is given the most time, is the most visible, and the most valuable. Implementing & executing the test plan involves not only testing but all the activities that you planned in the design stage. So you will be writing up bugs, checking through your bug logging system to manage incoming fixes, updating documentation, writing feedback. This is basically all the good stuff that helps make sure that the game is as good as it can be.

Report

The reporting phase of the testing process is where you record and report the quality of the game. How many bugs are you finding? How quickly and accurately are the fixes coming in? Do you feel confident that you’re testing enough? Is there enough time to do the testing necessary? Using all this information, is the game going to be a high enough quality to release on time? Reporting might not seem like a very important part of the process, but it is extremely crucial. Remember, QA isn’t about making sure that a game has zero bugs; it’s about having an accurate measure of exactly what the level of quality is. The development team can then use this information to make decisions on if that is high enough. Reporting is essential in that it allows producers, directors & product managers to make informed decisions about the next steps to take.

Closure

Closure, otherwise known as exit criteria, is the last part of the testing process, but that doesn’t mean the end. It’s often more accurately used to define the completion of the cycle. It makes sure that all of the goals have been addressed. This could either be by having successful test passes or ensuring that any failures have been analysed and attended to. If the current cycle’s goal was to release the game, then the exit criteria would have been to ensure the game does all the things that were defined in the game design document to an acceptable quality level. However, if this cycle was just to mark the end of a milestone in development, you would now revisit the planning stage!

The Testing Process doesn’t have to run linearly from start to finish every time. It’s important to continually check that the work you’re doing is achieving the desired results and if not, to adjust as necessary. This may mean going back to the planning and designing phases multiple times. In the next few articles of the QA 101 series, we’re going to be visiting in more detail how to plan your testing and the creations of test suites & test cases.

Other posts in the QA101 series: