The Art of Bug Reporting

I've noticed that some people in the D bug-reporting community haven't quite got the hang of writing good bug reports.  I have therefore prepared this set of tips for filing bugs efficiently on the D Bugzilla.

1. Summary Line

The first thing people will see of a bug report on Bugzilla is the summary line.  It's important to get this right.  The summary line should describe the bug in a few words, while at the same being specific to the bug.

Here are just a few too-vague summary lines I have seen in my time:

wrong error message

What is this wrong error message?  In what circumstances does it happen?

Compiler crashes on invalid code

What kind of invalid code?

Improve error message.

Improve what error message?  How should it be improved?

static assert doesn't fail as expected

This is most likely due to an expression evaluating to the wrong value.  Describe the bug in terms of that expression not being evaluated correctly.  If a given expression normally evaluates correctly, but doesn't seem to when placed in a static assert, then state that in your bug report.

dmd reject valid code

Describe the nature of the valid code.  And describe the nature of the error message, if you can do so in a few words.

Assertion in dmd

Again, describe the circumstance in which it happens.

should this be a bug??

Since you're putting it into Bugzilla, naturally we understand that you think it's a bug.  (True, people do file feature requests there too, but that's another matter.)  Tell us about this behaviour that you think is a bug.

2. Testcase

Post a short, self-contained code sample.  You could write a testcase from scratch.  Or you could start with the program in which you witnessed the bug and simplify it little by little, testing at each step, until you can't cut it down any more without the bug disappearing.

While trimming, be prepared to undo any trimming that causes the bug to disappear.  A good way of doing this is to comment out a piece of code, test, and then delete the commented-out code if the bug still shows without it.

Here are a few more specific tips for simplifying a program to show a bug:

• Use DMD's code coverage analysis to identify bits of code that are untouched while testing for the bug you are reporting.  Remove any untouched code and any conditionals that turn out to be always true or always false (but watch out for side effects).
• Remove protection attributes, if they aren't relevant to the bug being reported.
• If your program comprises more than one module, consolidate them into a single module, or into as few modules as required to show the bug.
• Flatten your class hierarchy, if the bug doesn't rely on it.
• Remove empty functions and any calls to them.
• If a function always returns the same value, remove this return, make the function return void, and set this value at the caller side instead.
• Remove any statements that become pointless as part of the trimming process, e.g. statements that merely set a local variable that then goes out of scope without being used.  Consider compiling with warnings (-w switch in DMD) as it will point out some kinds of dead code (but watch out for bugs in this functionality).
• Remove any comments that don't help to explain the bug.

If done well, your testcase will:

• be small enough that it would be reasonable to post it directly in the bug report, rather than as an attachment
• have no third-party library dependencies (unless the purpose is to report a bug in that library, in which case you wouldn't be using the D Bugzilla) and minimal RTL dependencies

Once you have your perfect testcase, copy and paste it into your bug report.  If you try copying it manually, you're likely to make typos that invalidate the testcase.  Copy the whole of it, so that the user can test it straight off.  Use the Select All command (Ctrl+A in many Windows-based editors) to make sure you don't accidentally drop a bit.

3. Actual Result

State what happens when you try your testcase.  Copy and paste compiler output (if the problem is compile-time errors) or program output (if it's unexpected behaviour at runtime) into your bug report.  Describe as clearly as you can any unfortunate behaviour beyond the wrong output being generated.

"If you want to find out what happens, you can try it out yourself," I hear you say.  This argument often doesn't hold water:

• Some of us might have Internet access on one machine somewhere, and D compiler on another machine somewhere else.
• If I try some code out with my compiler on my computer, it won't tell me whether the result is the same as what happens for you, who might be using a different version of the compiler on a different operating system.  If different people are experiencing a bug in very different ways and don't know it, they can end up talking at cross purposes.  Knowing the various version- and platform-dependent failure modes also helps in determining whether a bug that seems to have gone away really has.
• The Phobos/Tango schism also has some influence on this.  Few Phobos users are going to take the trouble to set up Tango just to compile/run a bug testcase that depends on it.  This works to some degree the other way as well.

4. Expected Result

State what you were expecting to happen, if it isn't obvious.

5. Keywords

Remember to assign keywords as appropriate.  The "Keywords" link on bug pages and the bug-reporting page will also take you to the list of keywords.

Usually, any report of a compiler bug should have at least one of these keywords: accepts-invalid, diagnostic, ice, link-failure, performance, rejects-valid, wrong-code, spec.  (The last of these indicates a problem in the D language specification, but applies to compiler bugs (possibly together with another keyword) if the spec is unclear in how the problem code should be treated.)

Assigning keywords might also help you with step 1, since one warning sign of a useless summary line is the existence of a keyword that says what your summary line says.

D Programming Resources | Programming Resources Home