Spoofed Handler #3 capacity reduced
Available spoofed capacity on Handler #3 dropped by about 40% while stale nodes were replaced. Handlers #1, #2 and #4 stayed at full capacity.
- Affected
- Spoofed Handler #3
- Started
- Sep 8, 2026, 3:05 AM UTC
- Resolved
- Sep 8, 2026, 6:11 AM UTC
- Duration
- 3.1 h
Response time
Peak 96 ms
{"labels":["0m","4m","8m","12m","16m","20m","24m","28m","32m","36m","40m","44m","47m","51m","55m","59m","63m","67m","71m","75m","79m","83m","87m","91m","95m","99m","103m","107m","111m","115m","119m","123m","127m","131m","135m","139m","142m","146m","150m","154m","158m","162m","166m","170m","174m","178m","182m","186m"],"series":[{"name":"Response time","color":"#ff4ecd","values":[40.8,37.9,38.9,39.8,39.4,39.8,37.1,37.4,39.3,38.6,38.6,40.4,39,48.9,58.5,67.5,75.6,82.7,88.4,92.6,95.1,96,95.1,92.6,88.4,82.7,75.6,67.5,58.5,48.9,39,38.5,38.2,39.7,38.4,39,39.9,38.3,39.6,38.9,38.2,39.1,39.6,40,37.5,38.9,39.4,37.4],"unit":"ms"}]}
Error rate
Peak 5.1 %
{"labels":["0m","4m","8m","12m","16m","20m","24m","28m","32m","36m","40m","44m","47m","51m","55m","59m","63m","67m","71m","75m","79m","83m","87m","91m","95m","99m","103m","107m","111m","115m","119m","123m","127m","131m","135m","139m","142m","146m","150m","154m","158m","162m","166m","170m","174m","178m","182m","186m"],"series":[{"name":"Error rate","color":"#3dd6f5","values":[0.6,0.6,0.6,0.6,0.6,0.6,0.6,0.6,0.6,0.6,0.6,0.6,0.6,1.4,2.1,2.8,3.5,4,4.5,4.8,5,5.1,5,4.8,4.5,4,3.5,2.8,2.1,1.4,0.6,0.6,0.6,0.6,0.6,0.6,0.6,0.6,0.6,0.6,0.6,0.6,0.6,0.6,0.6,0.6,0.6,0.6],"unit":"%"}]}
Forwarded capacity
Peak 50 Gbps
{"labels":["0m","4m","8m","12m","16m","20m","24m","28m","32m","36m","40m","44m","47m","51m","55m","59m","63m","67m","71m","75m","79m","83m","87m","91m","95m","99m","103m","107m","111m","115m","119m","123m","127m","131m","135m","139m","142m","146m","150m","154m","158m","162m","166m","170m","174m","178m","182m","186m"],"series":[{"name":"Forwarded capacity","color":"#8b6cff","values":[47.5,47.3,45.8,46.9,48.4,47.6,49.5,49.2,47,49.8,50.2,45.9,48,43.5,39.1,35,31.3,28.1,25.5,23.6,22.4,22,22.4,23.6,25.5,28.1,31.3,35,39.1,43.5,48,46.2,49.8,49.2,48.3,49,49.6,47.8,46.2,48.5,49.8,47.1,46.1,45.8,49.6,48.4,49.6,48.9],"unit":"Gbps"}]}
Timeline
-
Investigating
Capacity telemetry on Handler #3 fell below the expected floor.
-
Identified
Health checks marked several nodes stale after a routing change.
-
Mitigating
Stale nodes were replaced. Jobs were packed onto healthy members.
-
Resolved
Pool rebuilt. Capacity back to the advertised band.