Distributed Systems¶

No — that's the entire content of CAP theorem. During a partition, a node can either answer (possibly with stale data — available) or refuse to answer until it can confirm it has the latest write (consistent), but it cannot do both for the same request. CAP does not say you get all three when the network is healthy — even without a partition you still choose latency vs consistency (PACELC: if Partition, C vs A; Else, Latency vs Consistency). See CAP theorem.

They land in the same "unknown" bucket described above: the request may have been applied on the far side of the partition, or never arrived. A synchronous caller sees a timeout, not a clean error, and cannot distinguish "not yet applied" from "applied but the ack was lost." This is exactly why the idempotency section above exists — the system has to be built to survive not knowing.

Because refusing to answer is a worse user experience than answering with slightly stale data for the vast majority of endpoints — a stale product listing is fine, a rejected checkout page is not. So most systems default to AP and layer strong consistency back in only for the specific fields that need it (payments, inventory decrement), rather than making the whole system CP. Marketing the system as "highly available" is often just a polite way of saying "we chose A."

At-least-once retries until it gets an ack, so a message may be delivered more than once; at-most-once never retries, so a message may be lost. "Exactly-once" is usually at-least-once delivery plus idempotent processing on the receiving end — the transport still redelivers, but duplicate application is suppressed via the idempotency pattern above. True exactly-once delivery, with no dedup logic anywhere, is not achievable across an unreliable network — the guarantee lives at the application layer, not the wire.

It mostly doesn't try to, because physical clocks drift and NTP sync only gets you to millisecond-ish accuracy with unbounded worst case. Instead systems use logical clocks — Lamport timestamps for a "happened-before" ordering, or vector clocks to detect concurrent (unordered) events — which capture causality without needing agreement on wall-clock time. Google's Spanner is the notable exception: it uses TrueTime with bounded uncertainty windows and waits out the uncertainty to get real global ordering, at a real latency cost.

Mutual exclusion — naively, both partitions could end up believing they hold the leader role and both accept writes, so the data diverges and reconciliation becomes lossy. Raft doesn't stop a stale leader from believing it's still in charge — an isolated old leader on the minority side keeps thinking it's leader until it hears from a node with a higher term. What Raft actually guarantees is narrower but sufficient: at most one leader per term, and a leader stuck in the minority partition can never gather the majority acks it needs to commit a new log entry, so its writes never actually take effect even while it believes it's serving them. The real invariant to know is "at most one leader can commit per term," not "only one side ever has a leader" — the majority quorum requirement is what makes the minority side's belief harmless rather than what prevents the belief itself. See Raft.

Coordination overhead grows with node count — quorum writes need acks from a majority, gossip and consensus protocols exchange O(n) or O(n²) messages, and cross-node chatter adds network hops that a single node never paid. Past a certain point the added coordination cost outweighs the added throughput, which is why replication factor and cluster size are tuned, not maximized — "more nodes" is not a free lever.

A majority is a specific quorum — more than half — but "quorum" more generally means any rule for how many nodes must agree before an operation is considered committed, and it doesn't have to be a strict majority. Systems like Dynamo use tunable read/write quorums (R + W > N) where R and W can be set independently to trade off read vs. write latency, as long as their sum exceeds the replica count to guarantee overlap. Majority-quorum (as in Raft) is the special case optimized for leader election and fault tolerance, not the only valid definition.

You mostly can't, with certainty — from the outside, a crashed node and a node stuck in a long GC pause look identical: both stop responding. Practical systems use heartbeats with a timeout threshold and accept the tradeoff explicitly: too short a timeout falsely evicts slow-but-alive nodes, too long a timeout delays real failure detection. Phi-accrual failure detectors (used in Cassandra) improve on a fixed threshold by producing a suspicion level from heartbeat history instead of a binary up/down call, but the fundamental ambiguity — slow vs. dead — never fully goes away.

You've introduced a second copy of the data, which means you've introduced a consistency problem, whether you intended to or not. The cache can now serve stale data after the source of truth changes, and every invalidation strategy (TTL, write-through, write-invalidate) is really just choosing how stale and for how long you're willing to be wrong. A cache is not a free performance win — it's a deliberate trade of some consistency guarantee for latency, and the interview tell is being able to name which guarantee you gave up.