test suite | QA Quest

Are you looking to get started in testing for the gaming industry? Are you a more seasoned quality professional looking to level up your skills? Are you curious to learn more about what testing video games is really like? If you said yes to any the above, then the QA 101 series is for you! We aim to teach the basics of quality assurance by going back to the fundamentals. Each article will contain essential information to explain everything you need to know!

Put on your lab coats, because today the QA101 series is going to get practical. No more theory, we’re going to look into the process of writing test suites! If you’re still uncertain of the theory, then take a glance back at our piece on designing your test cases and test suites. If you’re already in the know, or once you’re all caught up, you’re ready to learn how to create test suites!

Which came first; the Test Suites or the Test Cases?

It might seem backwards but it’s often easier to think about the test suite before you have the test cases to go in it. There is no right or wrong order to this, but having guiding categories helps to structure your focus when writing the tests. If you’re working on a much larger team, you can even give the different suites to different QA members. They can add the test cases for each specific area, allowing them to focus in or partner with relevant members of the dev team.

The earlier you get working on test suites for the project, the easier it is to create your test cases. If the game ends up being huge, you can slowly add to your test coverage over time. Early in development, features will be incrementally added and improved upon. While this is happening you can summarize the feature with a test suite and write new test cases as they’re added. If you join much later and there are no tests for the many features, you’ve got a lot more ground to cover to catch up!

So where do you actually start? As mentioned before, some games are huge and have an expansive amount of content to test. The best way to explain how to do this is to break down this process by using a real-life but simpler example.

Testing Tetris

Say you are responsible for testing the original Tetris. It is a simple game in terms of its features compared to AAA games today. The first thing that you want to do is break down the features of the game. If your team has a Game Design Document, this information should be pretty easy to extract. If there’s no document, then a brainstorming session can help you to quickly pull out the main points. The idea isn’t to make this perfect on the first try but to create a very high-level skeleton.

Thanks to Tetris’ ubiquity, it is often used to teach game development and design theory. Because of this, I was able to find a Game Design Documentation created by students online. Using the information from their GDD and my own brainstorming I came up with these notes in about 5 minutes:

You may notice that I’ve brainstormed some test cases at the same time as the suites. This is important because they both inform each other. I focused on the gameplay mechanics, the controls, and the visuals/aesthetic, as these were the areas first outlined in the GDD. The gameplay mechanics of Tetris all revolve around the blocks; how they move, stack, and clear. Tetris wouldn’t be Tetris if it didn’t do these things, so they’re the first thing in the brainstorm. You’ll notice how I’ve highlighted some of these lines and suggested that they fall into more than one area.

Testing one thing, many ways

Let’s take the example of “Multiple Shapes”. The reason that I call this out as falling into many areas is because there are different ways to test this core gameplay element. You want to check that these shapes can spawn in and fall down the screen correctly. Each block has a different shape and will display a different pattern/colour. You want to test that the collision for each block is correct and will stack as expected when it hits the bottom. You’ll need to test that for every shape in the game because they’ll each have different collisions. You’ll need to check to see if the stacked blocks at the bottom continue to keep the correct collision. You will need to test that the controls for rotating, dropping and moving the pieces work as expected. And so on. All these tests focus on one thing (the blocks) but they’ll be tested in different ways. They can also be classified in different ways.

Sweeping into Suites

So what is the correct way to group these tests into suites? There really is no right answer! You could categorize them as I extracted from the GDD for the original brainstorm. But as we’ve just found out, you’d have tests for blocks all over the place. This could make it more difficult to construct test runs, especially if you want to focus on a particular feature. I would more likely set up multiple test suites like this for blocks alone:

Blocks – Gameplay
Blocks – Shapes
Blocks – Controls
Blocks – Collision
Blocks – Visuals
Etc.

Then other features outside of that that I could group are:

Field of play
Sidebar
Menus

To my logic, this makes the most sense, but there may be a need to switch it up depending on how development is going. For example, say you’re testing the gameplay of the blocks and it passes all tests except for the dropping feature. It would be useful to pull that out into its own suite or subgroup so you could give it the testing focus it deserves. Remember, as a member of QA you need to focus on the important areas of gameplay, but you also need to focus on risk. If a particular feature is breaking more often than others, that’s a good reason to spend time testing it more thoroughly. I’d be interested to hear from others how they would categorize suites for Tetris!

Early in development, having such specific testing is extremely useful. Active development is the most likely time that individual features will break, so this type of testing is valuable. Yet, when you’re a week away from release, the team should have fixed most of the issues. It no longer makes sense to check the blocks individually for their shapes, patterns and collisions. Hopefully, those features haven’t broken for a long while. This is where you’d start using smoke tests that would be more akin to “Start a game and play a level, ensure blocks rotate and drop”, etc.

Never stop adding new tests

You may notice that I missed out a REALLY important gameplay feature in my brainstorm. Do you know what it is? I’ll tell you the answer at the bottom of the article. Remember that this conceptualization stage is not the finalized testing plan; it’s just a starting place to help spark ideas. It can be helpful to try out the game and see what comes up in a play session. You can then add tests every time you realize that something isn’t covered. Sometimes testing will uncover tangential but unrelated bugs that you realise you hadn’t been looking for before. You’ll find that you’ll need to add tests often through the development of a game. However, remember that testing can also never be completely exhaustive. Blocks falling at level 1 may behave differently at level 100 (especially so with the speed multiplier in play). Would there be value in checking behaviours for every single level 1-100? Maybe not.

Putting it into practice

The original Tetris is a fairly simple game by today’s standards, but even within a 5-minute brainstorm, we’ve already come up with a sizable amount of tests. Upgrading that to Tetris Effect or Tetris 99 would explode the number of tests needed! You’ll need to add testing for online play or the more advanced visual effects. Even still the tests for that would likely be smaller than a 100-hour open-world RPG!

Try it for yourself! Think about another simple old-school game (like Pacman or Snake) and try to see the test cases & suites that you can think of. As a reminder, googling the name of the game and “game design doc” will likely bring up student’s work that can help you! Let me know in the comments or in a tweet what you come up with. I often see people talking about how QA is just “trying to break the game” or “running into walls”, but they don’t analytically think about how you try to do that. This exercise is a REALLY useful one for getting into the mindset of QA.

(* So what obvious thing did I miss out? In my test plan, I didn’t think about the lose condition, eg. Game Over. Pretty integral, huh?)

Test Cases

A test case is a piece of actionable documentation that contains instructions of how to test. They generally are targeted on specific areas or features of the game. Sometimes described as “test scripts”, their aim is to confirm that the game is behaving in the intended or expected way. Think of a test script the same as a coded script. If you run through the steps of the test exactly as written, you expect to get the same results every time. Unless there is a bug!

Writing test cases step by step is useful in many ways. A game development team is usually larger than one person, so good communication is key! You want every QA tester to check things in the same way. You want to make sure to test every important feature so you’ve left no holes in coverage. The testers need to know how a feature works (or at least how it’s supposed to work). Knowing this, they can report issues without guesswork. The way to ensure all this is to write a complete test case, with individual steps and expected results. This all helps so that when a QA tester says “there is a bug in X system”, they can efficiently communicate why. They have run the same steps as others and they know how the feature is expected to work. If they’re seeing something different, they can be confident they’ve found a bug.

Depending on your current focus, you can write many test cases for the same features. For example, you want to know that your game does the correct thing AND doesn’t do the incorrect thing. You want to know that your heal spell gives 50hp to the target. You also want to know that the heal spell doesn’t give an extra 50hp if the target is already at max health. There can be a lot of creativity in coming up with design ideas for writing your test cases. Testing can never be exhaustive so it’s impossible to cover everything. While it is good to have a lot of tests, past a certain point you’re unlikely to be able to use all of them, all the time. It’s a good idea to keep your test writing efforts focused rather than writing as many tests as possible. Quality over quantity. To do this, you need to ensure that you have a wide coverage across all areas and features of the game. You should also ensure that you have enough depth to the tests you have as well.

But what should actually go into a test case? I’d like to go into more depth, but the elements are far too many to include in this article! Take a look at the following post outlining more important tips on how to write great test cases! QA 101: How to write awesome Test Cases

Test Suites

A test suite is a collection of test cases. As you’re writing more test cases, they can quickly get out of hand! So it’s important to collect them into groups to help organize them. There are many different ways to do this, but the most common is to group tests that focus on the same area. All your weapons tests. All your character animation tests. All your VFX tests. This makes it easier to construct a testing pass that focuses on those particular areas. It also makes it easier to find a particular test that you’re looking for.

Which tests do you run when you’ve been given a new build of the game that revamps the way that characters move? You’re gonna want to run your character animation suite. This also makes it easier when reporting. Which makes more sense; saying that you have 2 failing tests out of the total 1000 or saying that 2 tests in the animation suite failed? The latter immediately communicates the information more concisely. It’s easier to see where the problem is likely to be, and who to send the bug to.

It’s also not unusual to have test suites created for different types of testing. For thorough testing, your tests will likely touch on specific areas and functions. Later in development, your testing will be more of an overview so you’d want to run higher-level tests. For example, checking each specific animation performs as intended vs a quick check that a character has animations at all. This is an example of the difference between functional and smoke testing. There are a variety of other types of testing that could be defined this way, such as build verification testing, regression testing and so on.

Want to learn more about creating Test Suites? We got you covered: QA101: How to create a Test Suite

Test cases and test suites are fundamental to a lot of the day to day work of a game tester. Because of this, it’s important to have a fuller understanding and appreciation of each. It’s useful to know the essential characteristics when you’re designing testing. This helps you to be more mindful of their functionality. A well-constructed set of test suites can make or break how successful you can be at creating effective test passes.

The purpose of this article is to give you an outline of what test cases and test suites are. In our next pieces, we’ll go into much more depth about the process of collecting your test suites, plus all the things you need to think of when writing test cases!

QA Quest

Tag Archives: test suite

QA101: How to create a Test Suite