A game day is a scheduled session where a team deliberately breaks its own systems to see how they — and the people on call — hold up. It's chaos engineering as a team sport: a few hours, a clear hypothesis, and a safe way to fail. If you're still fuzzy on the idea, start with What is chaos engineering? This guide is the how-to.
1. Pick one hypothesis
Resist the urge to break everything. A good game day tests a single, falsifiable claim: "if we lose one web node, the remaining nodes absorb the traffic and no SLA breaches." One hypothesis keeps the session focused and the results readable — you'll know exactly what you learned.
2. Define steady state
Before you touch anything, write down what "healthy" looks like as a number you can watch — error rate near zero, latency under some threshold, morale and open-ticket count stable. If you can't measure steady state, you can't tell whether the experiment broke it, and the game day becomes guesswork.
3. Limit the blast radius
Start with the smallest fault that can test the hypothesis — one node, not the whole tier — and agree on an abort condition up front ("if morale drops below 50, we stop and recover"). The whole point is a failure you control. You widen the blast radius on later runs, once the small one passes.
4. Inject the fault and observe
Now break it, and watch steady state. Does traffic reroute, or does the blast radius spread node to node until the tier is down? Whoever's on call works the incident for real — this is where you find out if the runbook is any good and the alerts actually fire. Resist fixing it instantly; the degradation is the data.
5. Turn findings into fixes
A game day only pays off if each weakness becomes a change: scale the deployment, fix the health check, write the missing runbook, automate the recovery with a playbook. Re-run the same experiment later — if it now passes without drama, the fix worked and that failure mode is boring for good.
Run one safely, today
You don't need a production outage budget to practise. SysAdmin Simulator's Chaos Engine is a ready-made game-day sandbox — a fleet that genuinely breaks, with no customer at risk. Run the loop end to end with chaos engineering practice, then handle the fallout with a full incident response shift.