QA101: How to use the 7 Testing Principles in Video Game QA

Are you looking to get started in testing for the gaming industry? Are you a more seasoned quality professional looking to level up your skills? Are you curious to learn more about what testing video games is really like? If you said yes to any the above, then the QA 101 series is for you! We aim to teach the basics of quality assurance by going back to the fundamentals. Each article will contain essential information to explain everything you need to know!

The International Software Testing Qualifications Board (ISTQB) is a certification board for gaining software testing credentials. Since 2004, almost 1 million professionals have taken their exams making it one of the most well-known certifications boards for QA. Though it is not as prevalent within the video game industry, a huge amount of the knowledge is transferable. This includes the 7 Testing Principles outlined in the foundations of software testing. Read on to learn more about principles to keep in mind to become a better QA professional.

Diagram illustrating the THE 7 TESTING PRINCIPLES
Testing shows the presence of defects, not their absence
Exhaustive testing is not possible
Absence of Error Fallacy
Early Testing
Defect Clusterin
Pesticide Paradox
Testing is context dependent

Testing shows the presence of defects, not their absence

  • QA tests to find defects with the intention of getting them fixed.
  • Despite this, you can only find and fix issues in the areas that your test plan covers. There could always be bugs in any area of the game that you didn’t test enough or at all. The more thorough your testing, the more confident you can be that there are few issues.
  • So while testing reduces the probability of undiscovered issues, it cannot prove that there are zero bugs because:

Exhaustive testing is not possible

  • Given all the money and all the time in the world, you could get close to testing everything. But it’s nigh on impossible to be able to do this within the confines of active game development. There will always be restrictions on the budget or a looming deadline that you must meet.
  • It’s also important to remember that games and their creation are complex. Just changing one line of code could have all kinds of ramifications. Managing risk is all about understanding which areas are most likely to be affected and focusing testing effort there.
  • Looking at speedrunning glitches or exploits really showcases how complex games are. For example, a chain of seemingly unrelated actions can let you skip parts of the game. Often speedrunners will spend time learning how to perform frame perfect button inputs to get certain results. The average user is unlikely to come across these events in normal play. Yet because of the nature of speedrunning, runners will pursue replicating the same bug over and over, making it seem simple and obvious.
  • Yes, potentially a tester should have tested this, but what are the odds? There is an infinite number of button inputs. It’s impossible to test them all! Watching speedrunning strategies is actually a good way to learn new testing techniques to consider.
  • Because of this QA must do testing with prioritization (what is most important) and adequate risk analysis (what is most likely to break).
  • Similarly:

Absence of Error Fallacy

  • As mentioned in our previous piece, QA101: Is Quality Assurance & Testing the same thing?, “A bug free game doesn’t mean that it has high quality”.
  • Even if there are a low amount of bugs, it will not mean that the game is playable, fun, or contain any other of the facets that you hoped the game would have.
  • A game could follow the game design documentation exactly, but design can be faulty too! Perhaps your menu flow doesn’t make sense, or the game’s battle system is wildly unbalanced. These aren’t bugs, but they both affect the quality of the game.
  • This is why doing the right kinds of testing is important. Sitting back and looking at the game as a whole is vitally important. Playtesting is a good method for doing this. It can help the team to understand whether the game is actually ready to ship, regardless of the bug numbers.

Early Testing

  • Test as early as possible. Not only does this give the team more time, but it also allows the development process to run more smoothly. Reporting bugs early and as often as possible means that they are more likely to be fixed before the implementation of the next set of features.
  • It is faster for a developer to fix a bug in code that they wrote last week than for them to go back to it 3 months later. QA testing early is the best way to uncover bugs fast.
  • Finding bugs early also gives you better guidance on how likely you are to hit your deadlines. If you don’t test until the last minute, you won’t know how many bugs you might find or how bad they are. If you don’t know how many bugs there are, you won’t know how long it will take to fix them! If you test early and there are few bugs, sweet! If you test early and find lots of bugs, you have lots of time to work on them.

Defect Clustering

  • It is often said that 80% of defects arise from 20% of modules. What this means is that bugs appear in particular areas more often, rather than evenly spread across all the game code.
  • This can happen for many reasons, such as those areas having higher complexity or more dependencies than other parts of the code.
  • You should consider this when writing your test plan. If an area is particularly prone to breakage or has a higher level of risk, then adding more in-depth testing is a good response.
  • Because defects often show up in clusters, with knock-on effects in related systems, it’s also a common occurrence that a bug fix will create other bugs. This is why regression testing is so important!

Pesticide Paradox

  • There is a theory in agriculture, that if you use the same pesticide over and over, the insects will slowly adapt and become immune.
  • In the same way, repeatedly running the same tests will mean that only the focus of those tests are proven to be bug free. You must regularly review and update your test plan and test cases to ensure there aren’t gaps.
  • For example, devs could implement a brand new game feature but your test plan doesn’t even touch it. The plan was written before the feature existed, so there’d be no coverage. This means that you likely have no way to check its quality!
  • Of course, you shouldn’t completely change your test plan because that will leave you open to bugs returning in those old areas. But you should approach your auditing with an additive mindset: Where can we add more coverage? Which areas are not yet being thoroughly tested?

Testing is context-dependent

  • Testing for video games is different from many other types of testing. You’re unlikely to find a web tester with an Xbox controller in their hands! Software development often doesn’t cover as many different disciplines as game development, so software testing can be more focused.
  • Though the testing approach for different types of products may not look the same, there are lots of practices and principles that are similar… Such as the ones in this article!

You probably won’t need to get certified by the ISTQB and knowing the 7 testing principles isn’t necessary to be a tester. But understanding the practices is a great way to become a better QA professional. Keeping in mind how much quality you can actually assure and how to approach testing a game puts you well ahead of those that don’t understand these concepts.

Other posts in the QA101 series:

3 thoughts on “QA101: How to use the 7 Testing Principles in Video Game QA”

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s