QA101: How to create a Test Suite

Are you looking to get started in testing for the gaming industry? Are you a more seasoned quality professional looking to level up your skills? Are you curious to learn more about what testing video games is really like? If you said yes to any the above, then the QA 101 series is for you! We aim to teach the basics of quality assurance by going back to the fundamentals. Each article will contain essential information to explain everything you need to know!

Put on your lab coats, because today the QA101 series is going to get practical. No more theory, we’re going to look into the process of writing test suites! If you’re still uncertain of the theory, then take a glance back at our piece on designing your test cases and test suites. If you’re already in the know, or once you’re all caught up, you’re ready to learn how to create test suites!

Which came first; the Test Suites or the Test Cases?

It might seem backwards but it’s often easier to think about the test suite before you have the test cases to go in it. There is no right or wrong order to this, but having guiding categories helps to structure your focus when writing the tests. If you’re working on a much larger team, you can even give the different suites to different QA members. They can add the test cases for each specific area, allowing them to focus in or partner with relevant members of the dev team.

The earlier you get working on test suites for the project, the easier it is to create your test cases. If the game ends up being huge, you can slowly add to your test coverage over time. Early in development, features will be incrementally added and improved upon. While this is happening you can summarize the feature with a test suite and write new test cases as they’re added. If you join much later and there are no tests for the many features, you’ve got a lot more ground to cover to catch up!

So where do you actually start? As mentioned before, some games are huge and have an expansive amount of content to test. The best way to explain how to do this is to break down this process by using a real-life but simpler example.

Testing Tetris

Say you are responsible for testing the original Tetris. It is a simple game in terms of its features compared to AAA games today. The first thing that you want to do is break down the features of the game. If your team has a Game Design Document, this information should be pretty easy to extract. If there’s no document, then a brainstorming session can help you to quickly pull out the main points. The idea isn’t to make this perfect on the first try but to create a very high-level skeleton.

Thanks to Tetris’ ubiquity, it is often used to teach game development and design theory. Because of this, I was able to find a Game Design Documentation created by students online. Using the information from their GDD and my own brainstorming I came up with these notes in about 5 minutes:

You may notice that I’ve brainstormed some test cases at the same time as the suites. This is important because they both inform each other. I focused on the gameplay mechanics, the controls, and the visuals/aesthetic, as these were the areas first outlined in the GDD. The gameplay mechanics of Tetris all revolve around the blocks; how they move, stack, and clear. Tetris wouldn’t be Tetris if it didn’t do these things, so they’re the first thing in the brainstorm. You’ll notice how I’ve highlighted some of these lines and suggested that they fall into more than one area.

Testing one thing, many ways

Let’s take the example of “Multiple Shapes”. The reason that I call this out as falling into many areas is because there are different ways to test this core gameplay element. You want to check that these shapes can spawn in and fall down the screen correctly. Each block has a different shape and will display a different pattern/colour. You want to test that the collision for each block is correct and will stack as expected when it hits the bottom. You’ll need to test that for every shape in the game because they’ll each have different collisions. You’ll need to check to see if the stacked blocks at the bottom continue to keep the correct collision. You will need to test that the controls for rotating, dropping and moving the pieces work as expected. And so on. All these tests focus on one thing (the blocks) but they’ll be tested in different ways. They can also be classified in different ways.

Sweeping into Suites

So what is the correct way to group these tests into suites? There really is no right answer! You could categorize them as I extracted from the GDD for the original brainstorm. But as we’ve just found out, you’d have tests for blocks all over the place. This could make it more difficult to construct test runs, especially if you want to focus on a particular feature. I would more likely set up multiple test suites like this for blocks alone:

Blocks – Gameplay
Blocks – Shapes
Blocks – Controls
Blocks – Collision
Blocks – Visuals
Etc.

Then other features outside of that that I could group are:

Field of play
Sidebar
Menus

To my logic, this makes the most sense, but there may be a need to switch it up depending on how development is going. For example, say you’re testing the gameplay of the blocks and it passes all tests except for the dropping feature. It would be useful to pull that out into its own suite or subgroup so you could give it the testing focus it deserves. Remember, as a member of QA you need to focus on the important areas of gameplay, but you also need to focus on risk. If a particular feature is breaking more often than others, that’s a good reason to spend time testing it more thoroughly. I’d be interested to hear from others how they would categorize suites for Tetris!

Early in development, having such specific testing is extremely useful. Active development is the most likely time that individual features will break, so this type of testing is valuable. Yet, when you’re a week away from release, the team should have fixed most of the issues. It no longer makes sense to check the blocks individually for their shapes, patterns and collisions. Hopefully, those features haven’t broken for a long while. This is where you’d start using smoke tests that would be more akin to “Start a game and play a level, ensure blocks rotate and drop”, etc.

Never stop adding new tests

You may notice that I missed out a REALLY important gameplay feature in my brainstorm. Do you know what it is? I’ll tell you the answer at the bottom of the article. Remember that this conceptualization stage is not the finalized testing plan; it’s just a starting place to help spark ideas. It can be helpful to try out the game and see what comes up in a play session. You can then add tests every time you realize that something isn’t covered. Sometimes testing will uncover tangential but unrelated bugs that you realise you hadn’t been looking for before. You’ll find that you’ll need to add tests often through the development of a game. However, remember that testing can also never be completely exhaustive. Blocks falling at level 1 may behave differently at level 100 (especially so with the speed multiplier in play). Would there be value in checking behaviours for every single level 1-100? Maybe not.

Putting it into practice

The original Tetris is a fairly simple game by today’s standards, but even within a 5-minute brainstorm, we’ve already come up with a sizable amount of tests. Upgrading that to Tetris Effect or Tetris 99 would explode the number of tests needed! You’ll need to add testing for online play or the more advanced visual effects. Even still the tests for that would likely be smaller than a 100-hour open-world RPG!

Try it for yourself! Think about another simple old-school game (like Pacman or Snake) and try to see the test cases & suites that you can think of. As a reminder, googling the name of the game and “game design doc” will likely bring up student’s work that can help you! Let me know in the comments or in a tweet what you come up with. I often see people talking about how QA is just “trying to break the game” or “running into walls”, but they don’t analytically think about how you try to do that. This exercise is a REALLY useful one for getting into the mindset of QA.

(* So what obvious thing did I miss out? In my test plan, I didn’t think about the lose condition, eg. Game Over. Pretty integral, huh?)