DTrace

DTrace is the new tool on SUN Solaris for debugging and tuning, I am very impressed with it.

DTrace is a comprehensive dynamic tracing framework for the Solaris Operating Environment. DTrace provides a powerful infrastructure to permit administrators, developers, and service personnel to concisely answer arbitrary questions about the behavior of the operating system and user programs.

A few features that really stand out:

1. Non-intrusive nature. Probes are implemented with dynamic instrumentation.
2. Efficiency. It greatly streamlines the tuning process.
3. Cost-effective. It does not cost much performance degradation on production boxes.
4. Reusability. There are a lot scripts already there, don't have to start from scratch.

In hindsight it makes so much sense to trouble shoot and fine tune systems in this way, why didn't it come out earlier? Or why didn't I ever think about this idea :-) ?