DevBlog: QA Detectives

Hi guys, my name is Dorina and I am in charge of the Anno 1800 QA department. I have been working in games for 13 years by now, starting as a student assistant on Anno 1701. A few years later, I got a permanent development testing position and then worked on Anno 2070.
It is a great honor to lead the QA team for Anno 1800 and I hope that this blog will give you an interesting look on the diversified tasks of our Quality Assurance team.

When it comes to game testing and QA, most gamers out there already seem to have a set opinion about that matter, and often make their voice heard. If you are able to identify bugs in your favorite game, then it surely can’t be that hard for the developers to fix the issue, right?
Today, we want to shine a light on the Anno 1800 QA team and give you a look behind the scenes of our daily work, in order to paint a clearer picture of game testing and the complex nature of game development.

QA tropes – playing games, all day long
There is a common trope that working in QA means playing games all day long, and while getting your hands on the game is surely part of the deal, reality is quite different. In fact, spending the day playing the game to check its overall shape and functionalities happens only occasionally. Our day-to-day work consists of specific tasks, where we have to test a certain feature of the game to evaluate its quality or to investigate reported issues. A complex matter, as reproducing a situation can be a demanding task and the causes and effects might often seem totally unrelated at first glance. We get an extensive features list to test for every new development milestone. To reach our goals up to the next big milestone, cooperation in our team and with the feature stakeholders (such as a game designer responsible for said feature) is crucial.

Our QA team’s desks are located in the middle of the studio, as we are a central focal point of production, being the team with a broad overview on the entire game. That allows us to build a strong connection with the other teams; you can almost say that QA is the watching eye of the production. People also might get a bit nervous when we spawn at their desk, as that usually means that their features are partly broken in some way.

The detective and his handbook
When we start investigating, our analytic skills, experience and knowledge about various development disciplines are our main weapons. While being on a case, the detailed design document (also referred to as DDD) is the QA detective’s handbook. While testing the game or analyzing issues, we always check back with the DDD if the game elements work as intended. As the DDD defines the function of every  element in the game, it allows us to create both kinds of test cases: the ones where we check if the game behaves as intended and those where we try to break the system on purpose. We will then compile all data in feedback reports and deliver those to the responsible developer and to the other stakeholders.

You all remember how we talked about the trade routes feature in last week’s DevBlog? The DDD describes that feature and all underlying rules of the system in detail, such as how the ships are being added to the routes, but also how menus should work and look, related audio elements, text display, button functionality and much more. When testing that feature, we have to work with the feature checklist to ensure that every single point is properly working. After a bug is fixed, we have to perform further checks to see if the fix breaks any other element of the game.

But it is not only about testing if every element works or if there are hidden bugs, we also help to test the balance of the game, from the production chains to the assessment of the difficulty of the battles. We always have to ask ourselves: is it working as intended and is the feature complex enough (or sometimes even too complex)?
A task rarely finished in one day, as it involves feedback reporting and ongoing discussions with the responsible developers. When the feature owner then applies changes based on that feedback, it means back to the testing process again.
But that is not all- another difference between the QA and QC’s (quality control) work is that we also check the projects globally. If we find out that a tool is not working properly or if there might possibly be a blocker in the production process, we will provide a report to the production team.

Bug hunting!
If we find a bug during the testing process, we will track it down in our bug-tracking tool and forward that report to the responsible game designer. As described before, that kicks off the testing and feedback loop until the issue is resolved.

Are you familiar with the saying that goes “No battle plan survives contact with the enemy”?
When testing and resolving bugs, there is always a probability that the fix breaks something else in the game. Most of the time, we do not know where a blocker is rooted so we have to find out what’s actually causing the issue first. Games are incredibly complex and there is one general rule: if we fix one bug, it might cause another.
When explaining that, I like to refer to the “butterfly effect”, where every flap of a wing can cause something unexpected and kick off a whole series of events. Therefore, before we release an update, we have to test the whole game again to verify that the fix is not breaking something different, which might not seem to be related to the initial issue at first.

This is a case-by-case process and the complex nature of a game, where bugs appear often during the development and we have to ensure that this doesn’t block anything in production process, as bug fixing can require hours of work from our coding team.

Make the game better
But there is another layer of our work which is not necessarily related to bug testing and fixing. When going through the DDD and observing the game, we might notice that something is missing or might cause gameplay issues based on our experience. If that occurs, we will provide a report including improvement suggestions of what might be missing or what could be changed, in order to improve or to balance the game’s content. The knowledge in our team is hereby of a deciding factor, as we are all different types of players and experience levels. That exact level of variety is important to be able to observe things from as many perspectives as possible.

How about you?
I hope that this blog gave you a good overview of the complex nature of game testing, as it affects every aspect of a game’s production across every development discipline.
Working in QA requires enthusiasm, passion and a deep understanding of the games in general, as well as the title you work on.

What your takeaway on the role of a professional game tester? Was the topic all too familiar, or did you get new information? I would also love to know if you have ever tested bugs in the past for games or other projects, maybe just as member of a community or even on a semi- or professional level?
It requires a passionate and dedicated team of experts to tackle the daily QA challenges of a complex strategy title like Anno 1800.