Incidents: making emergency changes you can unwind

Before anything else, work out which of these you have, because they pull in opposite directions.

Something hostile is arriving. Requests are refused and the refusals are correct. The instinct to widen and tighten is right, and the risk is collateral damage you will not see until later: legitimate visitors caught by a rule written in a hurry for somebody else.

Filtering is refusing your own traffic. Customers or an integration cannot get through, and every entry in the record is a mistake. The instinct to widen is exactly wrong here, and the risk is spending the outage making it worse.

Three questions separate them faster than argument. Did anything change on your side recently, a deployment, a new rule, a new form, since a fault that begins minutes after a change is usually the change. Are the refusals concentrated on one route, since your own collisions concentrate and hostile traffic usually does not. And do the affected requests sit inside otherwise ordinary sequences, which is the pattern described under the activity log and the clearest evidence that the people affected are real.

Two habits, and they cost nothing at the time.

Write down every change as you make it. The time, the URL path or rule you changed, and why, in whatever is already open. Nobody reconstructs this afterwards, and the reconstruction is what the whole review below depends on. A note that says «14:20 blocked the whole subnet, checkout was drowning» is worth more later than a perfect description written from memory a week on.

Prefer changes you can reverse in one action. A broad, single, reversible change is better under pressure than a precise one assembled from several edits, because the precise one cannot be undone quickly and you are the least reliable version of yourself right now. Broad now and narrow later is a legitimate sequence; broad now and forgotten is the failure.

That second habit also decides what to do when a change does not help. Undo it before trying the next thing. Stacked changes that each did nothing are indistinguishable from a change that did something, and by the third one nobody knows what the configuration is.

Turning filtering off entirely is the change most likely to be reached for and least likely to be right.

Before you do it, note what it costs: inspection stops everywhere, including on everything unrelated to your problem, at the exact moment you have the least visibility into what is arriving. If the fault is that legitimate traffic is being refused, the narrower move is returning the affected route to observation, which restores service on that route immediately while keeping the record you will need in an hour.

Off remains the right answer when filtering itself is clearly the cause, the HTTP responses your visitors receive are refusals rather than pages, and you cannot yet tell which rule is responsible. Reaching for it deliberately, with a note of the time, is a different act from reaching for it because it is the biggest available lever.

The part that gets skipped, and the reason this page exists.

Start within a few days, while the note still means something to the person who wrote it. Go through every change and put it in one of three places: it becomes part of the configuration because it was right all along, it becomes an exception with an owner and a date as exceptions describes, or it is removed.

Unwind the broadest changes first. They carry the most collateral, they are the ones least likely to have been targeted at the actual cause, and they are the reason people quietly cannot reach you. A wide block put in during an event is refusing somebody today, and nothing in your systems is going to mention it.

Then unwind one at a time, watching between each. That is the same discipline as removing any exception: put the affected traffic back under observation, confirm what would now be refused, and only then let the change go. Doing this during a quiet period rather than at the end of the same week is not procrastination, it is the difference between a controlled removal and a second incident.

Three things, and none of them is a document nobody reads.

A list of what stayed, with reasons. Anything still in place a week later either has a justification written next to it or should not be there. The list is short if the unwinding was done and unmanageable if it was not, which makes it a useful measure of the review itself.

One change that would have made this cheaper. Not a plan for perfection: a single concrete thing, such as a route that should have been observed before being enforced, or an integration that should have been exempted with a date long ago.

One gap in what you could see. During the event, what did you want to know and could not? That question produces better improvements than any discussion of the attack itself, because it is answerable and because the next incident will be different in every respect except your visibility into it.