When a production system fails, one of the first questions we usually ask is: What broke? Was it a storage service, a bad deployment, a memory leak, a dependency or a network problem?

That is a useful place to start, but it can also narrow the investigation too early. Some production failures happen even when every component behaves as designed; the problem appears only in their interaction. Systems-safety methods give us a useful way to reason about this class of failure, and I ran into a good example in a large-scale provisioning system.

When Everything Works and the System Still Fails

The system created storage partitions on demand and, as part of that process, applied an access-control policy to every new partition. The policy logic lived in a shared library that already had another user: An operator CLI that engineers used to apply policies manually, one storage instance at a time.

For the CLI, the design worked well. Each invocation read the current policy state of one storage instance into memory, calculated the required changes and applied them. As the CLI operated on one instance at a time, the amount of work and the memory footprint were reasonably bounded.

The automated provisioning service later reused the same library, but with a very different usage pattern. It could create many partitions at once and apply policies concurrently so that a single request could fan out into many simultaneous calls to the policy library. This worked for ordinary requests and had the obvious benefit of improving throughput.

The problem only became visible when a much larger request arrived. The provisioning service accepted it and started processing many policy operations in parallel. Each operation independently loaded the state it needed from the storage service, and the combined memory footprint kept growing until the provisioning service exhausted its memory limit.

At first, this looked like a normal resource problem. Maybe the policy library was inefficient, concurrency was too high or the service simply needed more memory.

But the interesting part was that every component was doing something reasonable. The policy library worked as designed, the provisioning service used concurrency to improve throughput and the requester submitted a valid request. The failure came from how those behaviors combined.

The Hidden Assumption

The policy library had an implicit assumption built into its original design: Its resource cost was safe because callers would use it in a bounded way. That assumption was true for the CLI, but it was no longer true once the same operation was used concurrently by the provisioning service.

The library had also been optimized shortly before the incident, but those improvements had been tested mainly against the original serial usage pattern. That told us little about the aggregate memory cost of many concurrent calls. A locally bounded operation had become an unbounded workload at the system level.

The above figure illustrates how a bounded policy operation becomes an unbounded system workload: Serial use keeps the resource cost predictable, while concurrent fan-out multiplies the same per-call cost across the provisioning service.

The original assumption had never been turned into an explicit system constraint. Nothing limited the size of a provisioning request based on its expected resource cost, and nothing tied the amount of concurrent work to the memory pressure created by those policy operations.

Make the Constraint Explicit

What the design needed was a system-level rule: The provisioning service must not accept or execute more work than it can process within its resource budget.

A fixed concurrency limit can help, but the safe level may change with request size, state size, traffic or available memory. The important part is that the provisioning service can account for resource pressure when deciding how much new work to accept.

Those signals might come from internal counters, runtime metrics or monitoring. What matters is not where the metrics live, but whether the provisioning logic actually uses them. In the original design, engineers could see the pressure after it appeared, but the service did not use that feedback to limit new work.

This is broader than one provisioning service. We often review components independently: Can this function handle errors? Can this service recover from a failed dependency? Can this API retry safely? Those questions are useful, but they miss a system-level one: What happens when several components behave correctly at the same time?

A function that is safe once may not be safe when it runs hundreds of times concurrently. A retry that helps one request may become harmful at scale. Reliability depends on the behavior of the whole system, not only on whether each component works as designed.

Ask About Interactions, not Only Failures

Traditional reliability analysis often starts from component failures: What if this service crashes, this call times out or this storage service becomes unavailable? Systems-safety methods such as STPA take a broader view. STPA describes a system using three main ideas:

• Control actions — what one component tells another to do
• Feedback — how the controller learns what happened
• Constraints — rules that must hold for the system to remain safe

That way of thinking fits this incident well. The provisioning service was the controller accepting and creating work, while the policy library and storage service performed the requested operations.

Memory usage, concurrency, outstanding work and queue depth were the signals the provisioning service needed to consider before accepting more work. The problem was that the provisioning logic did not use resource pressure as feedback.

The above figure explains that the provisioning service ultimately runs out of memory because accepted work is not constrained by its resource budget and resource pressure is missing from the control loop.

You do not need to run a formal STPA exercise every time to benefit from this perspective. During a design review, architecture review or incident follow-up, a few questions are often enough to expose assumptions that would otherwise remain hidden:

• What assumptions does this component make about its callers?
• What happens if this operation runs hundreds of times concurrently?
• Can several individually safe actions combine into an unsafe system state?
• What limits the amount of work the system can accept before it starts doing that work?
• What feedback does the controller need to enforce that limit?

If we had stopped at ‘the service ran out of memory’, we would have described the symptom correctly and still missed the more useful lesson. The system allowed a valid request to create an aggregate workload that none of the individual components understood or controlled.

Sometimes a component really is broken. But some of the more dangerous reliability failures happen when every component is doing exactly what it was designed to do. In those cases, the problem is not in the individual parts, but in the system they form together.