a benchmark tells you how slow your code is ("it took 20 seconds to do X Y Z") and a profiler tells you why it's slow ("35% of that time was spent doing compression").

Because rbspy is a sampling profiler (not a tracing profiler), it actually can't tell you how times a function was called -- it just reports "hey, I observed your program 100,000 times, and 98,000 of those times it was in the calculate_thing function". ruby-prof is a tracing profiler for Ruby, which can tell you exactly how many times each function was called at the cost of being higher overhead.

it's useful to understand the difference between "self time" and "total time" spent in a function

Nothing in the profiling guide is Ruby- or rbspy-specific — it all applies to profiling in general.

That is called profiling, not performance testing. Performance testing should ensure that a piece of code runs within a desired amount of time, given a certain context, before the new code goes into production.

optcarrot, is a headless NES emulator that the Ruby core team are using as a CPU-intensive optimization target.

ruby-prof supports excluding specific methods and threads from profiling results. This is useful for reducing connectivity in the call graph, making it easier to identify the source of performance problems when using a graph printer. For example, consider Integer#times: it's hardly ever useful to know how much time is spent in the method itself. We are more interested in how much the passed in block contributes to the time spent in the method which contains the Integer#times call. The effect on collected metrics are identical to eliminating methods from the profiling result in a post process step.