Operations · Lesson 04

How can a network keep working through failure?

Resilient networks limit failure domains, provide alternate paths, and make recovery observable.

The central problem

Contain and recover from failure.

Good network reasoning separates the mechanism from the symptom. Start with the boundary, identify the evidence, and test the explanation that accounts for the observation.

One signal rarely proves the whole path.Compare observations from the relevant layer before changing a configuration.

Prove the behavior

Redundancy is only resilient when failover works.

A second link or component does not guarantee continuity. A failover needs a detectable failure, a decision about the alternate path, a convergence or switching process, and an application that can recover its session or retry safely.

Test these steps before an incident and record the interruption, affected scope, recovery time, and user-visible result. A test that checks only whether the backup exists cannot reveal a shared dependency or a recovery procedure that is too slow.

Test the failure you designed for.Include the trigger, alternate path, dependencies, recovery time, and return to normal operation.

Preserve the essential work

A resilient service can reduce scope before it stops.

Graceful degradation keeps critical operations available while non-essential features, capacity, or freshness are reduced. This can protect the remaining path when a dependency or component is under pressure.

Define the priority before the incident: which requests can be delayed, cached, rejected, or served with fewer features? Then measure whether the degraded mode actually protects the essential user journey.

Continuity is not always full capacity.Make the reduced behavior explicit so operators can recognize a controlled degradation instead of mistaking it for an unexplained partial outage.

Build the model

Four useful signals

01

Failure domain

Defines what can fail together.

02

Alternate path

Provides another route to the destination.

03

Graceful degradation

Keeps essential work available.

04

Recovery test

Confirms that the design works before an incident.

Mini investigation

Inspect the evidence

Select the observation and identify the most direct next check.

NEXT CHECKDirect evidence

This observation narrows the mechanism but does not prove every downstream layer.

Compare it with one nearby-layer check.

Knowledge check

Test your model

Answer eight questions.

0 / 8