0:59 of 1:00 allowed, at 150 words a minute

What a retry storm is

Audience: support and product colleagues

Fictional example {n} of {t} rows ticked by a person
0:00
0:00
Beat 1 · 4.6 s

On 14 March our tracking page

was down for 23 minutes.

On screen23 minutes down

Title card over the tracking page, greyed out.

0:050:10
Beat 2 · 6.3 s

And it started with one slow query.

It took 8 seconds, not 200 milliseconds.

On screen8 seconds, not 200 ms

One query on a stopwatch, running long.

0:15
Beat 3 · 4.9 s

Our app gives up after 2 seconds,

then tries again, up to 3 times.

On screenGive up. Try again.

Four steps in a row: ask, wait, give up, ask again.

Beat 4 · 2.9 s

So one slow request becomes 4 requests.

On screen1 slow request becomes 4

One dot splits into four.

0:20
Beat 5 · 6.0 s

Normal load is 900 a second.

At the peak it hit 3,600.

On screen900 to 3,600 a second

Two bars, normal and peak.

0:250:30
Beat 6 · 5.7 s

The database never catches up,

because retries land faster than it can finish.

On screenIt never catches up

A queue that fills faster than it drains.

Beat 7 · 4.3 s

That is a retry storm.

The trying again is what keeps it down.

On screenThat is a retry storm

The words alone, large.

0:350:40
Beat 8 · 7.1 s

The fix went out on 16 March.

Each retry now waits longer than the last,

plus a random extra wait.

On screenWait longer. Wait unevenly.

Dots spread apart along a line.

0:45
Beat 9 · 4.0 s

And retries are capped

at 10% of normal traffic.

On screenCapped at 10%

A thin slice of a full bar, marked as the cap.

0:50
Beat 10 · 7.1 s

We ran a drill on 21 March.

Same slow query:

40 seconds of slow pages, and no outage.

On screen40 seconds slow, no outage

Before and after: 23 minutes down, 40 seconds slow.

0:55
Beat 11 · 6.0 s

So if a customer says it's slow,

tell them it's slow.

Slow is the safe way to fail now.

On screenSlow is the safe way to fail

The sentence on its own.

Where each beat comes from

Ticks stay on this page, in this browser. A tick is the person's word, not proof.

PrintedSource saysPerson checked
TitleWhat a retry storm isOwn words, no source
Audiencesupport and product colleaguesOwn words, no source
LabelFictional exampleOwn words, no source
Limit60 s60 seconds, for the Thursday all-hands
Beat 1On 14 March our tracking page was down for 23 minutes.
Words: 23 minutes down
On screen: Title card over the tracking page, greyed out.		On 14 March the parcel tracking page was down for 23 minutes.
Beat 2And it started with one slow query. It took 8 seconds, not 200 milliseconds.
Words: 8 seconds, not 200 ms
On screen: One query on a stopwatch, running long.		It started with one slow database query, which took 8 seconds instead of 200 milliseconds.
Beat 3Our app gives up after 2 seconds, then tries again, up to 3 times.
Words: Give up. Try again.
On screen: Four steps in a row: ask, wait, give up, ask again.		The tracking app gives up after 2 seconds and tries again, up to 3 times.
Beat 4So one slow request becomes 4 requests.
Words: 1 slow request becomes 4
On screen: One dot splits into four.		It started with one slow database query
tries again, up to 3 times.
Beat 5Normal load is 900 a second. At the peak it hit 3,600.
Words: 900 to 3,600 a second
On screen: Two bars, normal and peak.		Normal load is 900 requests a second. At the peak it was 3,600 requests a second.
Beat 6The database never catches up, because retries land faster than it can finish.
Words: It never catches up
On screen: A queue that fills faster than it drains.		The database stayed overloaded because retries arrived faster than it could finish them.
Beat 7That is a retry storm. The trying again is what keeps it down.
Words: That is a retry storm
On screen: The words alone, large.		This is called a retry storm.
Beat 8The fix went out on 16 March. Each retry now waits longer than the last, plus a random extra wait.
Words: Wait longer. Wait unevenly.
On screen: Dots spread apart along a line.		The fix shipped on 16 March: each retry now waits longer than the last, plus a random extra wait.
Beat 9And retries are capped at 10% of normal traffic.
Words: Capped at 10%
On screen: A thin slice of a full bar, marked as the cap.		Retries are now capped at 10% of normal traffic.
Beat 10We ran a drill on 21 March. Same slow query: 40 seconds of slow pages, and no outage.
Words: 40 seconds slow, no outage
On screen: Before and after: 23 minutes down, 40 seconds slow.		was down for 23 minutes.
In a drill on 21 March the same slow query caused 40 seconds of slow pages and no outage.
Beat 11So if a customer says it's slow, tell them it's slow. Slow is the safe way to fail now.
Words: Slow is the safe way to fail
On screen: The sentence on its own.		Own words, no source A closing thought for the audience, drawn from the drill. States no new fact.
Worked out41+3

The material, as the writer saved it

The source column and the text below are the writer's copy of the material. Nothing proves the copy matches what you gave. Compare them.

Fictional example. Tollgate Freight is an invented company.
Facts checked by the on-call team:

On 14 March the parcel tracking page was down for 23 minutes.
It started with one slow database query, which took 8 seconds instead of 200 milliseconds.
The tracking app gives up after 2 seconds and tries again, up to 3 times.
Normal load is 900 requests a second. At the peak it was 3,600 requests a second.
The database stayed overloaded because retries arrived faster than it could finish them.
This is called a retry storm.
The fix shipped on 16 March: each retry now waits longer than the last, plus a random extra wait.
Retries are now capped at 10% of normal traffic.
In a drill on 21 March the same slow query caused 40 seconds of slow pages and no outage.
UNSUPPORTED: the most common cause of outages
UNSUPPORTED: 50,000 pounds

Left out, not in the material

  • the most common cause of outagesAsked for in the manager's note. Nothing in the checked facts says so.
  • 50,000 poundsAsked for in the manager's note. The checked facts give no cost.

What the check cannot see

Looked at by machine, against the material: digits and written number words; time and percent units only; every-style frequencies only; from-to order; capitalised names; exact quotations; listed phrases only; left-out claims by key words. Not looked at: whether a sentence means what its source means, which number plays which role, negation, reworded claims, and names without a capital. No script can check meaning, so a person has to read the table.