2025-02-03

Synthetic logs that do not gaslight on-call engineers

By Mateo Silva

monitoringsyntheticson-call
Hero image for Synthetic logs that do not gaslight on-call engineers
Synthetic checks should flirt with failure without disguising success states. We walk through a Prometheus lab where recording rules accidentally double-count requests, then ask learners to decide whether to page. The second paragraph covers ethical constraints: we never mimic personal data fields; instead we generate plausible SKU codes. This keeps exercises procurement-ready without crossing privacy lines. Third, we tie the exercise to incident retrospectives—learners annotate Grafana snapshots with human sentences, not just "CPU high." Mentors model the tone we expect in real postmortems. The fourth paragraph flags a limitation: synthetic workloads cannot reproduce thermal throttling in edge cases; we recommend pairing with hardware labs when budgets allow.