DbgOut is my own logging and tracing tool, starting out as a simple Windows message logger way back, and evolving in fits and starts ever since. DbgOut in its current form has an extensive feature list:
DbgOut is a logging and tracing API, intended to be compiled into projects. It is not a strap-on tool like valgrind or otrace.
DbgOut made a huge step forward when I moved the logging into a kernel mode driver. This moved the information out of the address space of the target process, making it much more likely to preserve that crucial last bit of text before the process crashed and was terminated, and made it much easier to interleave logs from multiple processes. I created a kernel-mode API that allowed me to use DbgOut from other kernel-mode drivers as well.
DbgOut arrived on Linux/ARM to support a project that involved many small devices running on a network. I ported the DbgOut client API, the kernel Server, and the relay to Linux and cross-compiled for the ARM architecture.
DbgScope evolved from a need to log high frequency events and data that did not fit into text logs well. I found that visualizing software events in an oscilloscope format made it incredibly easy to very quickly identify abnormal behavior. The ability to quickly flip between event traces and text logs was a tremendous help in debugging real-time, mult-threaded, multi-platform timing bugs.
Working on real-time platforms required adapting the DbgOut driver's kernel-mode API to handle logging requests from hardware interrupt handlers and other real-time contexts (IRQL 0 in Windows terms). This allowed me to create a "software oscilloscope" to look at hardware events without requiring any wires or external hardware.
I expanded the DbgScope tracing API to include logging arbitrary data values. This let me build a real-time picture of system performance, memory usage, and any other thing I found interesting. A subset of the arbitrary data was data values that had distinct names. An example would be Windows messages, where the value 1 is the message WM_CREATE. Adding a set of interal lookup tables made it trivial to log every message sent to a window, see exactly how long it took to process, and how the different windows interacted with each other.
