Your benchmark box is lying to you

Quick public service announcement, prompted by a reader mail we get about monthly: the database comparison you ran on a t-class instance is fiction, and it is fiction in the flattering direction.

Burstable instances (AWS t3/t4g, Azure B-series, GCP e2 shared-core) are priced for idling and allowed to sprint on credits. Your benchmark is a sprint. For the first stretch the box performs like a much more expensive machine, because it briefly is one; the credit math is public and nobody reads it before publishing numbers to the team channel.

Then the credits run out. Not during your benchmark, of course. Ten minutes is well inside the budget. The credits run out three weeks later, in production, on the workload you sized using the sprint numbers, and the instance quietly throttles to its baseline, which on the small tiers is a fraction of a vCPU. The graph looks like the database broke. The database is fine. The accounting broke.

The fix costs five minutes. Either bench on a non-burstable instance class so the number you write down is a number the box can hold, or run the bench for forty-five minutes and watch for the cliff. If there is a cliff, you have learned the baseline; that is the number that goes in the comparison, with the burst noted as a bonus.

The deeper habit, which is the actual point: every "as of" we print has a workload and a duration attached, because a benchmark without a duration is a benchmark of the vendor's generosity window. Steady-state or it didn't happen.

Skip the t-class for anything you measure. Idling web servers, fine. Rulers, no.