In general, there’s no fixed rule of thumb that one could use for all situations, especially with debugging. I think one just gets good at recognizing common patterns over time. That said, reading postmortems and debugging stories is quite useful.

Kapil and I were discussing this once when he mentioned that perhaps the most general rule of thumb is to continuously challenge each and every assumption, and I agree. I’d also say that gather as much evidence (or data) as possible and think of the implications. In other words, hypothesize about a possible path that your program might have taken, given the evidence, and compare that with the actual implementation.

Here are some recent interesting benchmarking/profiling/debugging stories I have read.

Most the stuff that Julia writes on her blog is good, especially for when one is starting out. Her approach and exposition very clear; here are some good articles.

Anything from Dan Luu or Brendan Gregg is usually quite good.

Here’s one great piece of advice on benchmarking from Gernot Heiser (of the seL4 fame):

GDB tricks (from Gene):

  • https://course.ccs.neu.edu/cs5650/#debugging