Partial Failure and the
Fallacies of Distributed Computing

1 · Written problem

Your shortener's link-creation flow now calls a third-party fraud-check service before creating any new short link. That service runs in a different cloud, in a different country, and on a normal day the call takes about forty milliseconds. One afternoon, engineers notice that p99 latency for link creation has jumped to six seconds, and that some requests are timing out altogether. The team that owns the fraud-check service reports nothing wrong on their end.

1. Name two of the eight fallacies this design assumed were true, and say specifically where each assumption is built into the design as described.
2. The link service treats "no response within three hundred milliseconds" as "the fraud-check service is dead," and immediately creates the link locally without waiting further. Using this episode's partial-failure framework, explain how this can cause unintentional duplicate codes for the same URL.
3. Propose one concrete change to the timeout, retry policy, or request flow that would reduce duplicate short codes, without simply "trusting the fraud-check service more."

2 · Coding exercise - slow isn't dead, stop guessing wrong on purpose

~20–30 minutes · JavaScript

The shortener's read replica sits on the other side of a network. replica.get(code) returns a promise. Usually it resolves to the long URL, but sometimes it rejects with TimeoutError, and from where you're standing that could mean the replica is merely slow or that it's genuinely unreachable. primary.get(code) always answers, but you'd rather not lean on it.

Implement a CircuitBreaker class and lookupCode(replica, primary, code, breaker, now) so that the lookup:

1. prefers the replica while it's healthy,
2. stops hammering the replica after a small number of consecutive timeouts,
3. falls back to the primary while the breaker is open,
4. gives the replica exactly one trial call after a cooldown, to see if it has recovered. A failed trial re-opens the breaker.

now() returns the current time in milliseconds (the tests control it, so nothing actually has to wait). Four tests run against whatever you type below. You can't see or edit the replica and primary themselves.

class CircuitBreaker {
  constructor(failThreshold = 2, cooldownMs = 1000) {
    this.failThreshold = failThreshold;
    this.cooldownMs = cooldownMs;
    this.consecutiveFailures = 0;
    this.openedAt = null;
    this.halfOpen = false;
  }

  allowCall(now) {
    if (this.openedAt === null) return true;
    if (now() - this.openedAt >= this.cooldownMs) {
      this.halfOpen = true; // the one trial call
      return true;
    }
    return false;
  }

  recordSuccess() {
    this.consecutiveFailures = 0;
    this.openedAt = null;
    this.halfOpen = false;
  }

  recordFailure(now) {
    this.consecutiveFailures++;
    if (this.halfOpen || this.consecutiveFailures >= this.failThreshold) {
      this.openedAt = now();
      this.halfOpen = false;
    }
  }
}

async function lookupCode(replica, primary, code, breaker, now = () => Date.now()) {
  if (breaker.allowCall(now)) {
    try {
      const result = await replica.get(code);
      breaker.recordSuccess();
      return result;
    } catch (err) {
      if (!(err instanceof TimeoutError)) throw err;
      breaker.recordFailure(now);
    }
  }
  return primary.get(code);
}

3 · Quiz

1. A payment service sends a "charge this card" request to a payment gateway and gets no reply before its timeout. What is the right way to think about the charge?
   1. The charge failed, so it's safe to send it again as a brand new charge
   2. The charge succeeded, because gateways are usually reliable
   3. It's unknown: the charge may or may not have happened, so the retry has to be safe to repeat (for example with an idempotency key)
   4. The gateway is definitely down, so switch to another gateway immediately
   Answer

   C.Silence isn't evidence. The request could have been lost, or applied with only the reply lost. Since the sender can't tell, the retry must be safe to repeat.

2. A mobile app builds its profile screen with twelve API calls made one after another to a server on another continent. Each call takes about 90 ms round trip, and the server itself answers in 2 ms. The screen takes over a second to appear. What helps most?
   1. Cutting the number of round trips: batch the data into fewer calls, and make independent calls at the same time
   2. Buying faster servers
   3. Buying more bandwidth
   4. Compressing the JSON responses
   Answer

   A.The delay is in the distance, multiplied by the number of round trips, not in the servers or the payload size. Fewer round trips, or overlapping independent ones, is what moves the floor.

3. A nightly job copies a 200 GB table between two data centers over a shared link that also carries production traffic. Nothing errors, but every night around 2 a.m. customer requests get noticeably slower. What is the most likely explanation and a sensible fix?
   1. The database is corrupted, so restore it from backup
   2. Customers use too many browsers at night
   3. The link is fine, the customers' laptops are slow
   4. The bulk copy is saturating the shared link and starving other traffic; send only the changes, rate-limit the copy, or give it a separate link
   Answer

   D.Bandwidth is finite, and a large transfer on a shared link takes capacity away from everything else, with no error to point at.

4. An internal admin API has no authentication because it can only be reached from the office VPN. A contractor's laptop on the VPN gets compromised and the API is abused. What was the underlying mistake?
   1. The VPN wasn't fast enough
   2. The API treated "comes from the internal network" as proof of identity; each service should authenticate and authorize every request itself
   3. Contractors shouldn't be allowed to have laptops
   4. The API should have used a longer URL
   Answer

   B.Where a request comes from says nothing about whether it should be trusted. Anything that can reach the service from inside the network inherits all of its access.

5. A service reads the IP addresses of its cache nodes from a config file at startup. An autoscaler later replaces two of the nodes with new machines on new IPs, and the service keeps sending traffic to the old addresses. What should have been done?
   1. Look nodes up dynamically (service discovery, or short-lived DNS the client re-resolves) instead of fixing addresses at startup
   2. Buy larger cache nodes so they never need replacing
   3. Restart the service every hour
   4. Put the IPs in an environment variable instead of a file
   Answer

   A.Machines come and go, so a fixed list of addresses turns into a wrong map. The service has to ask where things are now, not remember where they were.

6. Your checkout depends on a third-party address-validation API. The vendor ships a breaking change at 2 p.m. on a Friday with no notice, and checkout starts failing. What is the best lesson?
   1. Ask the vendor never to make changes again
   2. Move the vendor's code into your own repository
   3. Only use vendors in the same country
   4. You can't control or foresee other people's changes, so design for it: timeouts, fallbacks that degrade gracefully, monitoring, and contract tests against the vendor's API
   Answer

   D.Someone outside your organization decides when their system changes. The protection is in how your system behaves when a dependency changes or fails, not in a promise from the dependency.

7. A team moves a 5 TB dataset to another cloud vendor to use a cheaper compute service. The compute is cheaper, but the total monthly bill goes up. What did the plan miss?
   1. Compute is never cheaper on another vendor
   2. Moving data isn't free: transfer and egress charges (and the time it takes) can outweigh the compute saving, so compute is often cheapest where the data already lives
   3. Datasets over 1 TB can't be moved
   4. The second vendor's servers are slower
   Answer

   B.Data has a cost to move as well as to store and process. Placing compute next to the data, or moving less of it, avoids paying that cost repeatedly.

8. A file-upload feature works flawlessly in staging, where everything runs in one cloud network. In production, users on hotel Wi-Fi and older mobile networks see large uploads fail. What is the best reading of this?
   1. Those users' devices are faulty
   2. Staging's database is faster than production's
   3. Real paths differ: proxies, connection timeouts, packet sizes and bandwidth vary along the way, so uploads should be built to tolerate them (small chunks, resume, retries) and tested on varied networks
   4. The fix is simply to add more servers
   Answer

   C.Staging exercised one very specific network. Production requests cross many different networks, each with its own behaviour, so the software has to tolerate that diversity.

Verify answers