Testers: The Unsung Heroes of Games

By Tom Sloper

NOTE: these lessons are primarily aimed at aspiring game designers, but many of the concepts described herein also apply to those who aspire to other types of jobs in the game industry.

You have undoubtedly heard that a recommended way of getting started in the games biz is to get a job as a game tester, if you do not have a programming degree, an art degree, a business degree, etc. And you have undoubtedly also heard a lot of negative reactions to this advice.

I am here to tell you that the negative things you have heard are all WRONG. There is a common perception that testing is a "lowly entry-level job" and that testers are at the bottom of the totem pole. The fact is, testing is extremely important and the test phase is vital in polishing a game into a fun experience for the end user. That's not to say that if you have an art degree or a programming degree, a law degree, or a business degree (or even a "game design" degree, if you find a college that offers such), that you MUST begin in the game industry in Q.A. Obviously, if you can enter the industry in a job closely related to the subject you mastered in at college, that's fine too. For those who have not gotten a degree in one of those areas...

... Testing is an EXCELLENT way to get your foot in the door, for a lot of reasons that will be explored in this article.

Terminology note: The Test department of a game publishing company is called "Quality Assurance," or "QA" for short. The term "QA" is also often used to describe the function or process of testing. In this article, the terms "test" and "QA" are sometimes used interchangeably.

Also note: This article is discussing the full-time internal job of tester (wherein the tester is an employee who comes to work daily at the game company to test games, for wages). Volunteer (unpaid) "beta testing" (wherein someone at home gets a copy of a game and provides feedback via email, usually without pay) is a separate matter entirely. Getting a job as a tester is a good way to get started in the game biz -- volunteering to do some beta testing is more akin to simply being a customer (an end user).

Quality Assurance testing jobs are usually found at game publishing companies. Developers also do some testing (but usually not full Q.A.) - but game development companies probably don't have full-time testing jobs (unless the development company is very large and well-staffed).

Typically, someone who works at a small game development company usually performs multiple job functions. Publisher jobs are usually more specialized. So someone who starts as a tester at a publishing company might eventually move up into producing, while someone who works as a tester at a smaller development company might eventually move up into any of a number of roles.

There are also independent testing labs who hire testers. Jobs at these places are great if you just want to test and you don't have aspirations of moving up in the industry - a tester who works at one of these labs would have to quit in order to move up in the industry. It's recommended that if you want to work as a tester as a steppingstone to other jobs in the industry, that you work for a publisher or a large developer, not an independent test lab.

Testing - The Final Phase

In a large game publishing company such as Activision, the QA phase comes towards the end of the project. And the testers are usually not involved in the project until after most of the work on the game has already been performed. This can have some unfortunate consequences, since testers brought in at the end were not involved in design decisions and don't necessarily know the rationale behind them. In a smaller company, team members who helped create the game may put on their tester hats towards the end -- thus they are already aware of the circumstances that led to project decisions made along the way.

The fact that most testers come in at the end of the project, powerless to have a major impact on the design of the game, is perhaps what leads to some of the negativity about the job. A military analogy can be drawn, putting the testers in the role of foot soldiers and the design/production team as the officers in a battle. This analogy has a limited usefulness, so I think it's worth mentioning, but this analogy falls apart if you try to apply it across the board to the process of making a game.

The foot soldier does not have the general's-eye-view of the battle, and is expected to just do what he's ordered to do. In a battle, there's rarely time to pass the strategic thinking all along the line so every foot soldier understands what the general is doing. In the process of making a game, I like to share my strategic thinking with my testers as much as possible -- not all designer/producers do things the way that I do. It's a hard thing for the testers to have to accept that it's too late to add features, and I'm sympathetic to that.

Testing Is Really Important

As a designer and producer, it's very important to me that the games I make are easy to use, friendly, and fun. I am my game's worst critic. In the QA phase I typically am the most prolific writer of bugs. But I'm also the guy who often has to reject testers' bugs as "Not a bug" or "Works As Designed." Sometimes the tester whose bug is rejected may think I'm not on his side, but there are no sides! QA and I share the same goal -- to make a game that will be a positive experience for the end user.

The way to make sure the game will be a positive experience is through thorough testing. Get multiple pairs of eyes looking at the game, get multiple pairs of hands taking the game through all its paces. I play the game one way, but somebody else plays it another way, trying things I never thought of. So I need the help so all the flaws can be found before my game goes out the door.

Testing Is Not Easy

To some folks, the term "testing" conjures up a mental image of guys (and some gals) sitting and playing games all day. Sounds like fun, easy work.

Far from it. It can be fun in the long run (it's an enjoyable job as jobs go), but it often feels like work. And it definitely isn't easy.

It's fun to play a game, but it's not easy. Ask any gamer. If a game is easy, it's no fun.

If a game isn't fun yet (if the balance isn't there yet, as sometimes happens), then it's definitely not easy to force yourself to continue playing it. If your job is to test, you have to do it even if it's no fun. And THAT ain't easy!

And when you're testing a game repeatedly, playing the same section over and over again, the fun of playing it can sometimes wear thin.

Testers have to be computer literate. You may be called on to test a computer game, so you have to be able to install cards and joysticks and drivers, you have to be able to uninstall and reinstall operating systems. Even if you are testing Playstation 2 games or Game Boy Advance games, you still have to use a computer to write your reports.

Testers have to be good communicators. You can be the most awesome bug-finder in the company, but if you can't explain to the team how you found the bug, what the bug is, what should have happened versus what did happen, and what you think is going wrong down in the code, then it can't be fixed! This part is extremely important, so it bears repetition. A tester must type in complete sentences. A tester must understand, and habitually use, proper punctuation and capitalization. You cannot become a tester at a game company where everybody uses English, if you cannot communicate properly in written English.

A good tester doesn't just find a bug and report "I picked up the green sword and drank the blue potion, and the game crashed." A good tester digs deeper, figures out how to make it happen again, and, knowing how the game works, figures out what's really going wrong. Maybe if he goes back, picks up the green sword and drinks the blue potion again, the game won't crash. Maybe the circumstances that caused the crash are deeper than that. Testing is definitely not a job for someone looking for a fun, easy experience.

Learning The Biz From the Trenches

Working as a tester is an excellent way to learn the game biz. Testers are exposed to (if not directly involved in) several other aspects of the business of games. Through working as a tester on several game projects, a lot can be learned about the business as a whole.

Development aspects

(Terminology note: "development" is used herein to refer to the "pre-production" aspects of a project, thus this refers to the design. "Production" is used to refer to the creation of the code, art, and audio.)

Through testing a game, the tester will have questions like, "why was it designed this way?" There are often several different ways that an interface for a particular feature could work. Sometimes those different ways are enabled as user-selectable features, and sometimes the designer just picks one standard way for the feature to work. The tester is among the earliest users for the game, so is often exposed to design aspects that subsequently get changed (or do not get changed).

Production aspects

The tester finds a bug and reports it. Most of the time it's a simple matter of making a fix. But sometimes it's a complicated matter involving discussions between QA and the Production team. In the meeting at which the discussion takes place, the tester will gain insights into what it's like working on the Production team. After several such meetings, the tester gains a broader perspective on the process as a whole.

Marketing / Sales aspects

Testers don't only test the game, they are also often called on to check the text on the package and in the manual. The hardware specification must be accurate, and so must the product claims (the bullet points describing the game's features). Through analyzing the product claims, the tester can gain an insight into how the product is being presented to the buying public.

Ship dates are hateful deadlines that often preclude the fixing of a pet bug. (More on prioritization of bugs below). Through immersion in a few game projects, the tester comes to learn the importance of prioritizing bugs. The company needs to ship its games in order to make money to pay the testers. Games can't be tested and fixed forever, and as a tester is involved in the process a few times, he learns about the realities of making games for profit -- how to prioritize the bugs he finds.

Customer Support aspects

No matter how thoroughly the testers pound on a game, there are bound to be some problems that only surface when the game is released into the wild. They're animals out there! They do all kinds of things to games that those in Production and QA never thought of, and that results in calls to CS (Customer Support). When those calls start coming in, the first call CS makes is to QA. The QA-CS relationship is therefore important. The tester who thought he was finished with that game is ordered back into service -- "try the game on this hardware configuration, or try doing this and that and see what happens." And the dreaded, "How could you have missed that?" The tester cannot help but learn about the kinds of issues CS faces.

Manufacturing aspects

Even manufacturing is something the tester will learn about through his involvement in making a game.

When the tester is given a box and manual to approve before the game is finished, the tester will learn that it takes longer to print a box and manual than to run off the CDs.
When the game is a console disc (rather than a computer disc), the tester is exposed to the issues involved with platform manufacturer approvals. And to the fact that patches are not possible. It has to be right the first time with a console disc. And on those rare occasions when something goes wrong in the manufacturing process, the tester may be impacted by having to re-test and re-release. And even if it doesn't have to be re-tested, through the day-to-day immersion in the culture, the tester learns all the painful details of what happened with that finished game before the tester gets his own copy (which he may not even want to play any more).
There can even be differences between a manufactured CD and a gold CD burned internally (in a CD-R). On one of my computer games, we had made our music tracks the normal way according to "redbook standard," which worked fine when we tested them. Little did we know that when the game was sent off to be manufactured, that the CD manufacturer would put shorter blank spaces between the music tracks. When the manufactured game was played, the audio would be played from one music track -- but then, before the CD head went back to play the track again from the beginning, it would wander into the beginning of the next track (so there would be a brief false start of another tune before recycling back to the beginning of the proper tune for that level). We had to remaster all the music with blank space at the front of each track. I know I learned something from that -- and I'm sure the testers did too!

See what I mean about how the tester gets to learn a lot about the biz? And here you thought testers just played games all day.

Testing Is a Steppingstone ... for those who are cut out for bigger things

That's right, there was some qualifying fine print in that heading. Not everybody who gets hired as a tester is cut out for bigger things. If you work hard and well, display a good cooperative attitude, communicate well and effectively, it's likely that you will be able to grow into higher positions.

If you shine as a tester on a couple of projects, you will likely get promoted to lead tester.

If you shine as lead tester on several projects, you may get promoted to test manager. Or someone in the production studio may want you to join their team as a production coordinator or assistant producer or even junior designer.

Just showing up and doing your work isn't enough to warrant promotion. You have to be bright, and you have to shine brighter than most. There's nothing unfair about that (despite the grousing you might hear from some who never seem to rise to higher jobs).

I forget where I heard this saying, but it seemed apropos to this topic:

"Smart people learn how to use their skills. Happy people learn how to live with their shortcomings."

If you are good, you will find a testing job to be an excellent entry to the games biz.

A Glossary Of Some Terms Frequently Heard In QA

'A' bug -- The 'A' bug is the very worst kind of bug. This type of bug can be summed up thusly: "It would be an unthinkably bad disaster if the game was released with this problem unfixed." Some examples:

the game crashes;
there is a virus in the game;
there are obvious spelling errors;
there are obvious graphical or audio problems;
a feature (in a menu or accessed by pressing a button) does not function;
there is no copyright language in the game anywhere;
the game is not fun to play.

Releasing a game with this sort of flaw would generate very bad public reaction and bad press, or there could be legal ramifications against the company.

'B' bug -- The 'B' bug is not quite as bad as the 'A' bug. It can be summed up thusly: "It would be unfortunate if the game was released with this problem unfixed, but the game is good regardless." In a pinch, if the company has a need to release the game and stop spending money testing and fixing it, and if Customer Support, Sales, Marketing, QA, and the executive staff all agree, the game may be released with minor flaws. For example:

bugs which do not ruin the experience of playing the game;
noticeable graphical or audio problems (especially if you know where to look for them);
highly desirable features were left out (and are not mentioned anywhere in the game).

'B' bugs will likely show up in press reviews of the game but are things that are probably hard (expensive; time-consuming) to fix. The playing public won't be happy with these problems, but the overall playing experience is not ruined by the existence of these problems.

'C' bug -- The 'C' bug can be summed up, "It would be nice to fix this problem." The tester may feel strongly about this problem s/he has identified, but when weighed against the company's larger need to release the game, the bug isn't that big a problem in the decision-makers' view. When push comes to shove, 'C' bugs may have to fall by the wayside (if they're hard to fix, that is -- a 'C' bug that's easy and quick to fix is likely to simply get fixed, unless the project is coming down to the wire).

'D' bug -- "It would be nice to add this feature." Especially when reported later in the test process, 'D' bugs are likely to remain unfixed.

"All bugs should be fixed." -- Ideally, of course, this is true. But some games are so big and complicated that the fixing would simply never end. And some testers are pickier than other as to what constitutes a bug that needs to be fixed. There have to be checks and balances in a game company (just as there are in a governing body).

"Alpha" -- The terms "Alpha" and "Beta" are defined differently by every company. Especially, developers' definitions of these terms may vary from publishers' definitions of these terms. Some developers may prefer to define Alpha as "code that demonstrates how the game will play." But most publishers (specifically a publisher's QA department) would prefer to define Alpha as "everything has been implemented in the game but there are bugs and the gameplay needs tweaking."

"Beta" -- Some developers may prefer to define Beta as "everything has been implemented in the game but there are bugs and the gameplay needs tweaking." But most publishers (or their QA departments) would prefer to define Beta as "everything has been implemented and as far as the developer knows, there are no bugs and the gameplay has been fully tweaked."

"Beta testing" -- Quality Assurance testing is a different thing from Beta testing. We usually use the term "beta tester" to refer to volunteers who test for free from their homes. Q.A., on the other hand, is a full-time position, a paid job. Beta testing is a good way to break into real testing. Look for opportunities to volunteer when you see that a game company is seeking beta testers (usually in an online gaming bulletin board or something - it's hard to seek out beta testing opportunities, and I can't tell you how to do it, you just have to be active in the gaming community's online forums). Do the beta testing well, and you might get offered a real testing job.

"Can Not Replicate." -- Sometimes a problem will happen to a tester but he can't provide steps to replicate the problem. If the programmer can't cause the problem to occur, with the debugger running to reveal the source of the problem, it may be difficult to fix. A good tester will try to make the problem happen again or figure out why it happened.

"Gold Master" -- The CD or DVD released by QA to manufacturing. This disc has been verified and virus-checked and has gone through an extensive checklist before it's sent out the door.

"It's a feature." -- The corollaries to this one are "Not a bug" and "Works as designed" (below). Sometimes what the tester expects the game to do, and what the game does instead, cause a bug report to be written. The bug report goes to the designer, who says "that's not a problem -- that's the way I designed it to work, and here's why it should remain as is ..." If the testers can present a convincing argument that the "feature" is counterintuitive or unfriendly, then perhaps it needs to be changed.

"Need more info (NMI)." -- This comment is likely scribbled on a bug report that doesn't tell the programmer enough information about how to replicate a bug, or why the tester feels that it is a bug.

"Not a bug (NAB)." -- See "It's a feature" (above).

"Psychotic user behavior." -- Term used to characterize a problem caused by unreasonable user input. For example: "The game crashes if you press F10, then Esc, 30 or 40 times in a row." No reasonable user would do this, and even if someone does do it, it would be unreasonable to fault the game for crashing under these circumstances. If the problem is hard to fix and the project is coming down to the wire, it may be simply written off.

"Release" -- QA signs off on the game and puts their stamp of approval on sending the game off to be manufactured.

"Ship it." -- This phrase is heard at the tail end of the test process, when the test team is starting to see the light at the end of the tunnel.

Tester: "I found a bug."
Lead tester: "What kind of bug?"
Tester: "It's just a 'C' bug, not a biggie. Psychotic user behavior."
Lead tester: "Ship it!"

Finished (tested, quality approved) games are shipped by FedEx or other courier service (or sometimes, if really important and timely, delivered by a member of the team) to the manufacturing facility. Manufactured product is shipped by truck to the stores. "Ship it" is the mantra used to seal the importance of cutting off testing and releasing the game into the wild.

"Tweak" -- Synonymous with "adjust."

"Will not fix (WNF)." -- When time is running short, and minor bugs are reported, the programmer or the designer or the producer may scribble this cryptic note on the bug report. All bugs have to be "closed" (resolved) before the game can be released.

"Works as designed (WAD)." -- See "It's a feature" (above).

Want to Get a Job As a Tester? Here's How to Prepare

I recommend that you have a college degree before applying for a job as a tester, but it's possible to get a testing job without one. But consider for a moment -- what is your ultimate goal? If you eventually want to become a designer or producer, or move up into marketing or become an executive, a college degree is definitely helpful. If you just want to be a tester (and do not have any goals beyond that), then fine, a high school diploma might suffice. But guess what three attributes or skills you need first and foremost to be a tester...?

Communication skills - The tester must be able to communicate in two ways: via the written word and via the spoken word.
- Written communication skills. Bug reports are submitted in writing. They have to be clear and concise. The tester needs to be a good speller and needs to be fluent with punctuation marks and the Shift key. Let me say that again. A tester must type in complete sentences. A tester must understand, and habitually use, proper punctuation and capitalization. You cannot become a tester at a game company where everybody uses English, if you cannot communicate properly in written English. Here's an exercise that will help you...
- Verbal communication skills. The tester must be well-spoken. Words that come out of a tester's mouth must convey his thoughts clearly, giving information to the listener. Imagine these two exercises, which will help the tester in developing verbal communication skills. How a tester performs in these exercises also reveals the level of his existing verbal communication skills. Both of these exercises are best performed in neighboring cubicles -- the two people taking part in the exercise can easily converse but cannot see what the other is doing.
  - The paperclip exercise. In this exercise, the tester must describe a randomly-bent paper clip to another person who has a pencil and paper. The goal is for the tester to get the listener/"customer" to draw a picture of the bent paper clip, without the tester ever saying the words "paper clip" or describing what the object is made of or was originally used for in any way whatsoever. Simply describing how the paper clip looks in its present state, the tester must obtain a correct picture of the paper clip on the second person's piece of paper. It can be enlightening for the tester to see what the drawing looks like, after completing the exercise. This exercise can also be performed using pipecleaners or twist-ties. The clip should be bent in a flat (2D) shape, not a 3D shape, since the listener/"customer" is drawing on 2D paper.
  - The building blocks exercise. This exercise is used at Nintendo of America to train or test their Customer Support representatives, but I think it applies equally well to the communication skills needed for testing. Both parties to the exercise have identical boxes of wooden building blocks (it could also work with Legos, I suppose). The tester builds a structure from his building blocks and describes his structure to the other participant in the exercise. If the tester does it well, the two structures will be identical. If the two structures are not identical, the tester can learn how he ought to improve.
Computer literacy. Testers know how to take computers apart and put them back together. Testers know how to browse the Internet, and they know all about email, instant messaging, and chat room netiquette. Testers know how to troubleshoot installation issues, download drivers, update virus DAT files, and upgrade computers. Testers know how to use word processors, imaging programs, scanners, and modems.
Game literacy. Play as many games as you can. Compare the pros and cons of this game versus that game. Read game magazines. Know the difference between an FRP and an RTS. Online games, console games, handheld games, board games, CCGs.

Snap reading comprehension quiz: What are the three attributes needed for a game tester?

For extra credit: Can you think of any other ways to improve your skills in these three areas?

Location, Location, Location

If you want to get a job as a tester, you must live near a game company that employs testers. You have to be able to commute daily to the game company. You cannot test games for money from home. If there are no game companies near you, move.

About those "game tester" websites

Do NOT send money to sites that promise game testing gigs online. These sites exist to take money from you, not to help you get a full-time testing job. No reputable game testing service should ask YOU for ANY money whatsoever.

In Conclusion , I've said it before, and I'll say it again: If you want to get into the game biz (either to design your own games or just because it is a field that's interesting to you), testing is a great way to get started. Don't listen to those naysayers who say testing is a dead-end job. Most people don't realize how hard testing is. Or how important it is in the process of making games.

Tom Sloper is a designer and producer of video games, best known for his work on the Shanghai series of games for Activision. An engineering designer and modelmaker/draftsman by training, Tom began his video game career in Southern California in the late-1970s.

He designed games for the legendary Vectrex game system (Spike, Bedlam) and other platforms at Western Technologies and Sega Enterprises before joining Atari Corporation, where he was involved in revitalizing the 2600 and 7800 game systems.

Tom is currently a global consultant for game developers, publishers and educational institutions. To learn more, visit Tom's Web site at www.sloperama.com.

Questions?

Resources