Wednesday, July 22, 2015

Your Bug Reports Suck

Introduction

I've been a professional software developer for nearly a decade. In that time, I've read a lot of bug reports, and almost all of them have sucked. A lot. Hardcore suckitude. They really, really suck.

There's something you need to understand about us developers: We typically don't know how to use the software that well. That might come as a shock, but it's true. The people who know the application the best are the people who use it on a regular basis. They have adapted to it, developed their own processes, and repeated them so often that they've become second nature. The support staff and testers (the people typically writing the bug reports) are a close second to the end users in application knowledge, while developers are often a distant third. Developers are, unfortunately, not mind readers, nor do we possess a crystal ball, so our ability diagnose and fix bugs will live and die with the quality of the bug reports we receive.

What to Include in Bug Reports

So how, you may wonder, does one write a good bug report? Well, for one, you need to know what content to include, and, for that, I would start with item #4 on the Joel Test. Here's what Joel has to say:

A minimal useful bug database must include the following data for every bug:

  • complete steps to reproduce the bug
  • expected behavior
  • observed (buggy) behavior
  • who it's assigned to
  • whether it has been fixed or not

Let's look at these a bit further...

The first bullet point is by far the most crucial. Good reproduction steps point the reader to the part of the application that isn't functioning properly and (hopefully) guide us right to the problem. Without complete reproduction steps, a bug report is basically useless.

A common problem that I see is omitting a reproduction step or two. This sometimes occurs due to laziness or by mistake, but it can also happen because the person writing the bug report assumed that the developer knew what to do at a certain point in the user's process. Never assume! As I mentioned in the beginning, developers don't know the software as well as you think we do.

The next two bullet points are obviously pretty important, too. We need to know what the application is supposed to be doing (the expected behavior), as well as what it is doing (the bug). The combination of these two items is what I typically call the problem statement.

The fourth bullet point (the assignee) is pretty self-explanatory. The last bullet point is often referred to as the bug's status, and this is usually pretty straightforward. However, I have seen people invent some rather nebulous terms for issue status. I recommend naming the statuses as clearly as possible (i.e. "Open", "In Progress", "Development Complete") and providing detailed definitions for each status within your documentation.

Joel's list is a pretty good jumping-off point, but I think we can take it a bit further. I also recommend including the following pieces of information whenever available:

  1. Who reported the issue? Some problems are isolated to a specific user, and we need to know what makes them different. What type of user are they? Do they have any specific permissions granted (or missing)? Are they on our network, or are they accessing the software externally? This information can be vital when trying to reproduce and/or diagnose issues. This seems like a lot of information to include in a bug report, and indeed it is, but getting (at a minimum) the user's name, contact info, and login ID is at least a starting point in case more information is needed later.
  2. When did the user first notice the problem? Knowing the date and time that an issue first manifested itself can be of vital importance. It often helps us determine if it was caused by something we did (a software change, hardware change, data modification, etc.) at a certain time.
  3. What is the frequency of occurrence? Does this issue happen every time the user performs a specific action, or does it occur intermittently? Does it only occur at certain points of the day? Are all users experiencing this issue, or are only certain users experiencing it (in which case the information in item #1 can be crucial)?
  4. Which environment was this issue observed in? A bug might be found in production, but it could also first be noticed in a different environment (such as QA). This environment could have a slightly different version of the application. It's important to indicate where the bug is observed so that the developer knows a) where they can reproduce the issue, and b) which version of the source code to look at.
  5. Which web browser was the issue observed in? Much of our software these days is web-based, and, unfortunately, applications that function fine in one web browser can be buggy in others. For web applications, always include the browser name and version number in the bug report.
  6. Is there a workaround? What are the workaround steps? This sort of information can help keep users productive while the issue is being resolved. Occasionally, it can also help developers diagnose and fix the bug being addressed.
  7. What is the severity of the issue? Indicating the severity of an issue is important in helping developers prioritize the issues that need to be addressed (please note that an issue's severity is related to, although not necessarily the same as, its priority). It's highly recommended that some sort of severity scale is developed and strictly adhered to. I recommend something like the following:
    • High = Critical functionality is impacted and no workaround exists.
    • Medium = Critical functionality is impacted, but a workaround exists.
    • Low = Non-critical functionality is impacted.

Obviously, not all of the information above is always necessary (severity and workarounds aren't as important for non-production issues) or even applicable (non-web-based applications don't require a browser). However, the point is that a bug can have a lot of variables, and it's important to know as much about those variables as we can so we can diagnose the issue quickly and efficiently.

Including Quality Content

In addition to knowing what content to provide, you'll want to make sure that content is as clear and as useful as it can be. Here are just a few of the things you can do to improve the content of your bug reports:

  1. Use complete sentences with correct spelling, punctuation, and grammar. This seems like a no-brainer, but you would be amazed at some of the positively dreadful bug reports I've read. Don't assume the person reading your bug report will understand sentence fragments or shorthand. Before submitting the report, reread everything you've written to ensure clarity.
  2. Call things what they are called on the user interface. Users will often use industry jargon or come up with their own terminology for certain screens, UI elements or pieces of data. Developers won't know this terminology; they'll be relying on how things are labeled on the screen. Stay away from cryptic terms whenever possible.
  3. Use ordered (i.e. numbered) lists. This is a rather simple thing to do, but it can help aid in clarity of communication between team members (e.g. "can you elaborate on reproduction step #4?").
  4. Take screenshots. This one is pretty self-explanatory. We've all heard the old saying that a picture is worth a thousand words. While the phrase "refer to screenshot" might not save you from typing a thousand words, it will probably make your life easier.
  5. Use screenshots to include examples of form input data. Here's a common scenario: a user fills out a lengthy form, clicks a submit button, and then receives some nasty error. As much as it might suck, any one of those pieces of data could have caused the issue, and you'll need to include them all. But instead of typing all of those form values into a note in your bug report, you can create a screenshot while you're reproducing the bug. Fill out the form, then take a screenshot before you click the submit button.
  6. Save your reproduction steps! You just typed a TON of words on how to reproduce a problem... do you want to have to retype all that every time this screen breaks? Of course not! Save reproduction steps for common tasks into text files or Word documents, and store them in a folder on your hard drive. Include the name of the UI at the beginning of the file name so you can locate them quickly.
  7. Consider developing (or improving) your own template for bug reports. If your bug reports are hand-written, consider developing a Word template. If you're using a commercial tool for bug tracking, look into customizing it. If you find that the developers are constantly coming back to you asking you for a specific piece of information, that's a strong indication that it should be included in all bug reports.

The Developer's Responsibility

Okay, okay... I've spent a lot of time berating support staff and testers over their bug reports. But, as it turns out, developers aren't perfect either. There are a few things we should always be doing with bugs that get assigned to us:

  1. If you can't find the problem, at least add notes for other developers. Investigating bugs can often feel like "doing science"... no matter how methodical you are, you won't make a discovery until you make a discovery. Diagnosing bugs can be time consuming, and it can be frustrating and costly to retread the same ground that others have already covered. Add as much detail as you can to the bug report about where you looked and what you tried. Try to be as specific as possible; include names of source code files, subroutines, or even line numbers. It sounds like a lot of extra effort, but it may save you or your teammates time later on.
  2. After fixing a bug, add instructions related to how to test changes. Usually the reproduction steps are the testing instructions. But sometimes a change can have a far-reaching impact and require re-testing a variety of screens or functionality that (seemingly) had nothing to do with the original bug. In these instances, add a note to the bug report calling out what areas should be tested.

Conclusion

Bugs suck, but bug reports don't have to suck. By following a few guidelines, we can make bug reports clearer and more informative, which makes them more useful to developers.

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.