DevBlog: The Role of QA

Hey Anno Community,

Today we want to shine some light on the work of the QA teams and give you some insights into our processes: Which roles do each team fulfill? Is there a difference between the work before and after the game release? What’s a Hotfix and when do we make use of it?

For this purpose, we sat down with Dorina, our QA Manager here in Mainz, as well as – for additional insights – Nadiia, Senior Development Tester here in Mainz (previously: QC Project Lead) and Artem, Live QA Specialist.

In two sentences: What does the QA team do?

One is already enough 😉 We’re ensuring the quality of the product as well as the production process as a whole.

So, we know you’re not just looking for potential issues all day. What other roles does your team fulfill?

There are quite a few, actually.

Generally, QA acts as an info-hub, where information from all teams comes together. We keep an overview of the status of issues and in-development features, keep an eye on the project planning and gather information about features and mechanics.

We’re also doing risk management, meaning we’re escalating problems in regard to both project and product to the production team or other departments. Examples could be stressing resource management planning (meaning too many tasks for too few people in too little time) or detecting workflow issues within the team, maybe processes are unnecessarily complicated or inconsistent between teams etc. Of course, risk management also includes prioritization of testing areas and fixing bugs.

Another would be providing feedback on features and systems during development and during the concept phase. The QA team consists of people with deep knowledge of the game and its systems, but also of people with very different play styles and general preferences when it comes to video games as a whole. This means we can provide feedback from different perspectives on such matters as: Do we think the proposed new features will be fun? Are there two systems that collide and don’t fit together? Is a feature redundant since there’s already something very similar in the game?

We’re also the admins for various internal tools, an important one being JIRA (a tool that can be used for various purposes, in this case, we use it to report and track bugs) alongside several others used for testing.

And finally, we’re also organizing the validations for each release, be it DLC or Game Update. More on that later.

For even more details on the work of our QA, have a look at this older blog.

How does your work change from pre-release to post-release?

One big difference for sure is that there is a loot more user feedback coming in now. This means of course more work on these live issues and requires a lot more prioritization work from us: How important, how severe is the feedback or bug report?

Additionally, we’re having significantly more releases, of course, which means new versions have to be finalized and validated every few months.

On the other hand: Much less features are getting cut now in post-launch, so if something is planned to be part of a DLC, we as QA can usually plan with this feature also releasing and a test is not invalidated by a certain mechanic suddenly not being in the game anymore.

You are located directly here in Mainz. Are you working with other QA teams? How is the work split between all of you?

We’re working with two other teams which aren’t located directly here in the studio: Quality Control (QC) and Live QA, each with a different focus.

Let’s get an overview from Live QA first:

They are a team of QA specialists wo provide post-launch support (i.e. they support the game after its release) for Ubisoft’s games and services. All of their tests happen in the Live Environment alongside the users, meaning they have no debug options or cheats to skip content or play with unlimited resources.

“We help the production team to have a more complete picture and to confidently take an informed decision about the aforementioned issues.”

By having – or by trying to have – the exact end user setup they are able to provide live information about the product, for both Production but also Business topics. They are also using JIRA (however, not the same one we use in the studio) for their daily work: For any issue that come in, it’s their job and responsibility to reproduce it and keep the JIRA ticket updated regarding any new information coming in.

Live QA is a highly transversal team, meaning they are collaborating with multiple teams, including of course the Production team on their day-to-day tasks, but there are other topics that require them to reach quite a few different job families.

Those include Customer Support or Community Management to reach out to the player and provide more information if possible, as well as Release Management (checking that all promised content and offers are working correctly, e.g. “Can players activate their pre-order bonus without issues?”), Quality Control, the Ubisoft Connect team, and all kinds of other Ubisoft services like the Store or the Ubisoft Plus teams.

The third team is QC – the Quality Control team – and true to its name it has a plentitude of different responsibilities.

It makes sure that the game meets all needed requirements that apply for the title: quality standards for the current milestone (i.e. upcoming Game Update/DLC), technical Ubisoft requirements and first party (for games which release on other stores than the Ubisoft Store, games releasing on consoles etc.) as well as partnership requirements (this can be related to partnerships with e.g. Amazon).

Therefore, QC consists of a variety of teams: functionality game testers team who test the game’s features and mechanics and a list of teams in the “technical standards” teams, for example PC requirements, Ubisoft online requirements, UX requirements, Online Performance and Networking (i.e. multiplayer related things), Tracking, Localization, etc.

All those teams are working in close collaboration with each other and with the production team in Mainz during the whole main production and post-launch time. Any game update which goes live first goes through validations made by all those teams, who then compile a detailed report of the results.

Overall, it’s worth stressing that all this is a big team effort, with all three teams working together to make sure we can release game updates and DLC on time, learn of potential problems early enough, take informed decisions and address issues when they appear and based on their priority..

I’ve reported a bug on the forums, how long does it take to fix it?

Generally, the time it takes to fix an issue depends on several different factors, from the severity over the complexity to our current project planning.

All issues reported to us are first categorized (e.g. severity and probability) and we’re attempting to reproduce it, which greatly helps the investigation. Looking for the exact cause without knowing where to look is extremely time-consuming. Changing something to fix an issue can additionally create a new issue elsewhere.

“Player reported issues are like a jigsaw puzzle, where we have to figure out the missing pieces and once we have the full picture, we send it to the Production team for a resolution. This process also involves finding viable workarounds for the players where possible.”

That’s why before the release of each Game Update, we have validation period during which each fix as well as the Game Update as a whole are tested. If problems are encountered, the validation fails and we first have to address them.

To give you some detailed insights into which steps are taken from finding an issue to a fix being released, we’re currently working on a separate blog. Stay tuned!

As we just established, fixing an issue and releasing an update does take time. Are there possible exceptions to this process, e.g. the so-called Hotfix?

Hotfixes are possible, but only for single, very specific high priority fixes. These also need to go through validation, but we can speed up the process (e.g. certain validation teams are not needed if nothing is changed, that would be relevant for their specific team) since there’s significantly less to check. In total, assuming the issue can be reproduced and fixed quickly, we could release a Hotfix in 4 to 7 days, if necessary.

We have done various playtests in the past, from Technical Tests to Diary Studies. In which capacity is QA involved in preparing, monitoring, and follow-ups/summaries?

Just like with the versions that need to be prepared for each Game Update and DLC release, we also have to check the versions for all playtests. The requirements are less strict, of course, smaller bugs and placeholder assets are therefore not a deal-breaker. We still perform basic checks to make sure the version is stable and the mechanics work.

We’re also in contact with the other teams involved, e.g. the User Research Lab in Düsseldorf for Diary Studies or the team managing the registration and invite flow for the Technical Tests.

During the Technical Tests, at least some of us are also checking the forums, sometimes just to read, sometimes also to ask you for more details and we create JIRA tickets based on the reported bugs and requested improvements.

We hope this provided you with some useful insights into how our internal processes work and how much time some things can take, which we understand is not always visible from the outside.

If you have further questions for our QA teams that were not covered here or in the older blog, feel free to post them in the comments and we will try to get the answers for you.