- Capability
- End-to-end service resilience
- Assumption
- Platform failover proved application availability
- Decision
- Test resilience from the user-facing service backwards
- Outcome
- Find shared dependencies outside the protected platform
The infrastructure failover completed successfully. Cluster resources moved, protected workloads restarted and the management console reported that the platform was healthy.
The application remained unavailable. Its compute had recovered, but a dependency outside the cluster had not. The service still required name resolution, identity, network policy, storage access and an external connection that the platform-level test did not exercise.
Nothing in the cluster report was false. It proved that the cluster could perform its own recovery sequence. The mistake was treating that result as proof that the complete business service had recovered.
High availability is often purchased and tested one product at a time. The application experiences a chain of dependencies instead: client access, DNS, certificates, authentication, routing, data and third-party services. One unprotected or incorrectly ordered dependency can make several resilient components irrelevant.
The test was redesigned around a usable service outcome. Recovery was not complete when resources were online; it was complete when an authorised user could reach the application, authenticate, perform a representative transaction and receive the expected result.
That approach turned a successful product test into a meaningful service test. It also exposed which dependencies needed protection, which required a documented recovery order and which risks the organisation had consciously chosen to accept.
Engineering lessons
- Component failover does not by itself prove service resilience.
- Recovery tests should begin with the user-facing outcome and include every critical dependency.
- A healthy management console may still coexist with an unavailable application.
- Resilience decisions should make shared and unprotected dependencies explicit.
Read the engineering principles behind this work →
Confidentiality: Engineering Notes are based on real engagements. Client identities, timelines and identifying details may be changed to protect confidentiality. The engineering decisions and lessons remain representative of the work undertaken.