Validate Recovery, Not Just the Happy Path

Most of our downtime is not from code bugs, it is from recovery gaps. FATs usually prove the happy path. At SAT I now push messy scenarios: power blips, network drops, starved infeed, double hits on sensors, changeovers mid-cycle. The goal is simple: can the line get back to a safe known state fast, without a laptop.

A few low-tech tests have paid off. A plug-in interval timer to flicker 24 V to remote I/O or VFDs. Tape over a photoeye to simulate stuck-on. Pull and reinsert Ethernet on a drive. Jog the encoder loose to add jitter. Kill and restore power during a batch. On legacy PLCs like SLC or MicroLogix, I check how comms loss and retentive tags behave after a warm restart.

We log MTTR after a jam clear as a validation metric, and we fail the test if operators need to toggle bits or change setpoints to recover. It has cut our first-month downtime more than any new feature.

What failure and recovery tests do you consider non-negotiable before handover, and what cheap tricks do you use to simulate them?