Grabbing cores

I love debuggers. I don’t know what I would do without having a good capable debugger as part of my development toolbox. The ability to see right into the core operation of a piece of software and quickly get and overview of the whole state of the program beats random printf markers anyday.

Oddly enough, my first memory of using a debugger had nothing to do with traditional debugging. I was a co-op student working on some speech processing software using the now defunct Mac CodeWarrior development tools.

I was trying to figure out how to insert timing measurements into a large piece of algorithmic software to determine how to best optimize it.

“Simple” said one of my colleagues at the time. He fired up the debugger and started stepping through the project:

step step, step ………… step

“That’s the function you should look at” indicating the one with the long step time delay.

A true KISS principle that I’ve used over and over to do basic analysis of where a program is spending its time. Simple and effective when you are considering software where delays and latency is measured in seconds.

The cool thing about working with QNX systems is that you can use the same debugger and techniques on just about any piece of system software. Services, drivers and applications are all outside the kernel so there are very few occasions where you can’t attack a problem using a process level debugger.

Neutrino uses gdb as its main debugger (the IDE drives gdb under the covers) and one of the great facilities that gdb enables is the ability to do post-mortem analysis on crashed binaries using core files.

 % ntox86-gdb my-x86-binary my-x86-binary-corefile

or if you are in the IDE then create a post-mortem debug session and select the binary and the core file as part of the launch configuration.

Capturing core files on a target Neutrino system requires that you run the dumper program. If (when?) your program abnormally terminates, assuming dumper is running, a core file will be generated and a diagnostic message logged. You can then load up the debugger with the core file and get right to the root of the problem.

The dumper program has a bunch of usefull options, but one of the most usefull is the -p option that allows you to generate a core file for any running process without terminating that process.

 % dumper -p process_id -d dump_directory

This is a great way to capture program snapshots for those situations where you can’t attach to the running program due to potential latency
or you just want to get an idea of the program state (ie the program seems to be hung up or burning CPU) and don’t have a local debugger connection.

Once you have the core file, you load up into the debugger the same way as if you had received a core file from a crash.

Simple, easy and totally pain free debugging!


2 comments so far

  1. Michael on

    Thats cool, but nearly as nice as Green Hill’s TimeMachine debugger.
    With TimeMachine you don’t do “step, step, step…”. Instead you “run, wait, backstep”.

  2. sendreceivereply on

    For QNX the system instrumentation and profiling behaviour is wrapped up in the System Profiler tooling. TimeMachine is a neat tool though!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: