Fail where the fault lives

“Fail fast” starts out as debugging advice. Somewhere along the way it escapes the function body and turns into a management reflex — reject the idea, kill the process, restart the service, pivot the team — as if the shorter the gap between attempt and failure, the more disciplined the work must be.

That isn't what the original technique promises. Jim Shore's 2004 article on fail-fast software says plainly that it may not reduce the number of bugs at first. What it does is make the bugs you have easier to find, because they surface immediately and close to their cause. The same article recommends a global error handler, so the application as a whole can recover or keep going when the failing unit allows it.

Fail fast is a locality rule, not a speed goal. Catch the violated assumption while the program still holds the evidence to explain it, then decide what's actually invalid — one value, one message, the whole process, or something in between. Stopping more than that buys you nothing but collateral damage.

The boundary decides what dies#

The same validation error deserves a different response depending on where it turns up. Missing configuration at startup means the process can't do its job at all. A malformed message in a queue means one work item can't be trusted, while the ten thousand behind it are probably fine. Calling both “fail fast” skips past the only decision worth making here.

const config = Config.parse(process.env);if (!config.ok) throw new Error(config.error.message);for await (const message of messages) {  const order = Order.parse(message);  if (!order.ok) {    await rejected.write({ message, error: order.error });    continue;  }  await dispatch(order.value);}

Both failures surface immediately, but only the first one stops the process. The configuration is a precondition for everything the worker will ever do. The bad message has a natural containment boundary: write it to a rejected queue where someone can inspect it later, and move on to the next one.

Silent defaults are the more familiar sin — they move the error away from its cause, and you find out three services downstream. Crashing the whole process over one bad message makes the opposite mistake, treating local invalidity as global corruption. The narrowest boundary that still preserves a truthful state is where the failure belongs.

A crash needs someone waiting#

Erlang's reputation for “letting it crash” rests on the half of the sentence people leave out: the crash happens under supervision. The official OTP supervision principles describe supervisors whose whole job is to watch child processes and deal with the ones that die. A one_for_one strategy restarts only the child that terminated, and restart intensity limits keep a broken child from dying forever in a tight loop — past a threshold, the failure escalates to a higher-level supervisor instead.

Failure stays ordinary here because recovery lives outside the component whose state went bad. Take away the isolation and the owner for the restart, and “let it crash” is an outage with a slogan attached.

George Candea and Armando Fox tested this boundary directly in an eBay-like auction service. In their study of fine-grained microreboots, restarting a component instead of the server process cut failed user requests by 65 percent and perceived downtime by 78 percent in their experiments. All they changed was the size of the recovery unit.

Fast retries manufacture another failure#

An immediate error helps when the caller knows what to do with it. It turns dangerous when the caller's only policy is to try again right away — the second request consumes the same resources while whatever rejected the first is probably still true. I've written that loop myself more times than I'd like: three attempts, no backoff, no theory of what will be different on the second try.

Google's guidance for handling overload treats retries as extra load, full stop. A three-attempt budget can push traffic toward three times the original request rate before broader retry limits kick in, and when several layers each retry the layer below them, the growth turns combinatorial. Their answer is to confine the retry to the layer immediately above the rejecting dependency, bounded by both per-request and per-client budgets.

Sometimes the correct response is a degraded result, or a quota error for one customer while everyone else carries on. Sometimes the error genuinely has to travel all the way back to the user. A quick rejection protects a system only when the callers around it respect its scope.

Failure is a complete event#

“Just fail” means letting a failure become one complete, useful event. Signal it near the cause. Stop the smallest unit whose state can no longer be trusted, keep enough context around to reproduce the problem, and leave durable state coherent. Retry only when something has actually changed — more time, freed capacity, a different route, different input — and put a budget on the attempts.

The mechanism follows the boundary. An impossible internal state gets an assertion. An expected business refusal gets a typed error, and a poisoned queue item gets a rejection record so a person can look at it. A worker whose state you can no longer trust gets a supervisor; an overloaded dependency gets admission control and a retry budget. Underneath all of them sits the same question: who can act next?

The distinction holds outside software too. An experiment should run long enough to answer the question that justified it. Quitting early because failure has become fashionable produces no evidence, and calling the abandonment “learning” doesn't fill in the missing measurement.

So when the next fault shows up, resist the urge to make everything stop as fast as possible. Find the boundary that knows enough to name the error and can contain it without erasing the evidence, then let that part fail while the rest keeps running.