Benchmark Report — Node-Kill Recovery

2026-08-24

Summary

A node is killed while it holds locks under load, and the question is what a surviving node can still do afterwards (scripts/bench/compare/bench-node-kill.sh).

Every backend is killed identically: the victim machine is powered off through sysrq, so nothing runs an exit path, nothing unmounts, nothing releases a lock, and no daemon gets to notice it is dying. Earlier revisions of this scenario used a different fault per backend, which made the comparison meaningless in both directions — that history is at the end of this report.

Three probes run for the whole of each run, and it takes all three to read the result:

• shared probe — the survivor writes a file both nodes were hammering, so the lock on it ping-pongs between them. This measures keeping on doing what it was already doing.
• takeover probe — from the instant of the kill, the survivor starts writing a different file, one only the victim had ever written. This is the only probe that forces the survivor to acquire a lock the dead node held, and it is therefore the one that measures recovery at all.
• death watch — the survivor polls the victim's port at 5 Hz, so the moment the fault actually landed is measured rather than assumed. It is not the moment the kill was issued: a machine powering off through sysrq took 0.08–1.13 s to go quiet here, and one earlier run took 46 s.

Same five isolated 3-node clusters as the other reports.

Results

All times are measured from the victim's actual death, not from the kill command.

Backend	victim down after kill (s)	shared-probe stall (s)	takeover of the dead node's file (s)	survivor still writing at end of run
etcfs	1.098	0.113	2.189	yes
gluster	1.126	77.956	22.640	yes
gfs2	0.079	0.240*	never (>180 s)	no — silent for the last 203 s
nfs	0.631	0.012*	never (>180 s)	no — silent for the last 193 s
juicefs	0.758	0.031*	never (>180 s)	no — silent for the last 202 s

* These stall figures are not what they look like. "Longest gap between two consecutive successful writes" needs a success on both sides of the gap, so a probe that stops writing and never writes again closes no gap and reports a small stall from before the failure. The last column is the one that separates "never stalled" from "stopped answering for good", and it is why the scenario now records it (compare_probe_silence).

Failed operations after the death: zero everywhere except juicefs, whose takeover probe returned 3,356 outright errors rather than hanging. etcfs's takeover probe completed 16,971 successful writes to the dead node's file; gluster's, 20,406; gfs2's and nfs's completed none at all.

Reading these numbers

etcfs recovers a dead node's locks in 2.19 s, with no fencing device and no journal replay. The survivor's own I/O never stopped (0.113 s worst gap, still writing at the end), and its first write to the file the dead node owned landed 2.19 s after that node stopped answering. That is lease-expiry-bound behaviour: the dead node's lock keys are bound to its lock session, a lease with a 2 s TTL (inodeLockTTL), so the takeover is about one session TTL plus acquisition backoff. It does not wait for the fence (the 10 s --lease-ttl is the membership lease, which drives fencing). If the dead node ever comes back, the generation guard makes its writes unacceptable.

GFS2 does not recover here at all, and the reason is structural rather than slow. The survivor's DLM lockspace was captured mid-run in exactly the state the design calls for:

name          vol1
flags         0x00000004 kern_stop
new change    member 2 joined 0 remove 1 failed 1 seq 4,4
new status    wait fencing

kern_stop means the lockspace is stopped: no lock is granted to anyone on that node until the failed member is fenced. The harness configures no STONITH agent, so the fence never completes, so the lockspace never restarts — and note that this did not merely block the takeover. The survivor's other probe, writing a file it had already been writing successfully, went silent too and stayed silent for the remaining 203 s. One node's death stopped the surviving node's filesystem entirely.

GFS2 is not designed to be run without fencing; a production cluster has a fence device, and its recovery time would be fence-device latency + journal replay, which is a real number this harness does not measure. What the run does establish is the shape of the dependency: GFS2's recovery is gated on an external device confirming a kill, and until that happens every surviving node stops. etcfs's recovery is gated on a lease expiring, which requires no device, no operator, and no journal replay — and while it happens, the survivors keep working.

With real fencing configured (2026-08-27)

The objection above is answered by configuring the fence device the harness had been missing, rather than by reasoning about it. COMPARE_BACKEND=gfs2-fenced brings GFS2 up under pacemaker with a fence_aws STONITH device that stops the dead instance through the EC2 API, DLM and the mount as ordered clones, and stonith-action=off so a fenced node cannot return before its journal is replayed. The scenario itself is unchanged — same sysrq power-off, same probes.

	unfenced	fenced
fence confirmed	no device configured	yes, ~10 s after the kill
survivor's own I/O	stopped for the rest of the run	never stopped (0.266 s worst stall, 0 failed ops)
takeover of the dead node's file	never, in 203 s	never, in 180 s

Fencing fixes the part of the failure that made the unfenced run hard to read. The survivors keep their own filesystem: they resume in 0.042 s and lose no operations, where before one node's death stopped every surviving node outright. That is the difference a fence device makes, and it is large.

What it does not do, in this configuration, is hand the dead node's inode to a survivor inside the window. With the fence confirmed at about ten seconds, the file that node had been writing stayed inaccessible for the remaining 180 s — any access to it blocked indefinitely, to the point that an unguarded ls on it hung the harness itself. Whether a further recovery step is missing from this setup or GFS2 takes longer than three minutes to release that lockspace is not established here; what is established is that the take-over did not happen inside the window, with fencing working. Against EtcFS's 2.19 s, GFS2 with real STONITH keeps its survivors alive — the unfenced cluster does not — and still did not recover the dead node's file in three minutes.

Repeated runs on a non-burstable class (2026-10-16)

The 2.19 s above is one run. etcfs was killed eight times and fenced GFS2 three times, each on a fresh m5.large cluster, same scenario and probes:

	etcfs (8 runs)	gfs2-fenced (3 runs)
takeover of the dead node's file	3.00–3.47 s, median 3.11 s	never, in all three
fence confirmed by pacemaker	n/a	yes, all three
survivor's worst stall after the death	0.02–1.47 s	0.12–0.22 s
survivor's shared probe silent at end of run	6–15 s (probe wind-down)	~152 s, all three
failed operations after the death	1 in one run, 0 otherwise	0

GlusterFS and NFS were repeated the next day (2026-10-17) on the same class:

	gluster (3 runs)	nfs (2 runs)
takeover of the dead node's file	22.3–22.7 s	never
survivor's worst stall after the death	76.7–78.1 s	0.02 s, then silent
survivor's shared probe silent at end of run	6–7 s (probe wind-down)	~137 s, both runs

Both reproduce the single t3.medium run: GlusterFS takes over in about 22 s with a stall of over a minute on the shared file, about 7x slower than etcfs, and NFS serves nothing once its server is gone. JuiceFS could not be re-run, because its MinIO binary is no longer downloadable.

The etcfs takeover is narrow rather than spread across a lease period: the dead node's inode lock lives under a 2 s session lease (inodeLockTTL), and the measured 3.0–3.5 s is that lease plus detection and acquisition backoff. The 2.19 s single run sits below this range and should not be quoted on its own.

The fenced GFS2 result needs correcting, not just repeating. In all three runs pacemaker recorded the fence as complete, yet the survivor's DLM lockspace was still in wait fencing with kern_stop two minutes later, and the survivor's own probe, after briefly resuming, went silent for the last ~152 s of every run (a dd blocked in path_openat, in the captured dmesg). So the survivors did not keep their filesystem in this configuration either; the 2026-08-27 "never stopped" row came from a probe window too short to see it. The likely cause is that dlm_controld never learns of the fence pacemaker completed, which would be a harness integration fault rather than GFS2 behaviour. That has not been established, so GFS2's recovery time under working fencing remains unmeasured.

gluster recovers, slowly. The replica set drops to two copies and the client takes 22.6 s to write the dead node's file, with a 78 s worst gap on the shared one. It is the only competitor here that recovers at all inside the window, and it does so about 10x slower than etcfs.

nfs and juicefs are not recovery numbers and should not be read as such. Both lose the single machine that is the filesystem, so the correct result is a total outage, and that is what happened: the NFS client hangs indefinitely on a hard mount (no error, no progress, silent for 193 s), and the JuiceFS client fast-fails every operation once Redis is gone (3,356 errors). They are included because the difference between "one node of several died" and "the filesystem died" is the point of a shared-device design, not because their numbers compete.

What changed since the previous run of this scenario

The 2026-08-16 revision of this report published etcfs as the slowest of the five and speculated that the other backends' faults were milder. Both halves were right, and this run fixes them:

1. The fault was not the same for everyone. killall -9 corosync dlm_controld on GFS2 is an orderly membership change, not a crash. Every backend is now powered off.
2. Two of the faults did not land at all. Blocking NFS's port with iptables -I INPUT --dport 2049 left the server answering on 2049 for the whole of a 180 s run, and systemctl kill nfs-server does not stop the kernel's nfsd threads — so NFS was being credited with surviving a failure it never had. The death watch now proves the fault landed before any number is published.
3. The probe measured the wrong thing. Both nodes hammered one shared file, and a survivor that happens to hold that lock when the other node dies needs nothing from the dead node and sails on. In an earlier run of this revision GFS2 posted a 0.135 s stall while its lockspace was demonstrably stopped waiting for a fence. Recovery is only exercised by asking for a lock the dead node held, which is what the takeover probe does.
4. "No stall" and "stopped forever" looked identical. Hence the silence column.

The etcfs column moved from 0.249 s / 3.501 s (old fault, shared probe only) to 2.189 s takeover under a power-off — a worse-looking number that is the first one on this page that actually measures lock recovery.

Caveats

• The five-way table is one run per backend. The etcfs takeover is bounded by the 2 s inode-lock session lease (inodeLockTTL), not by the 10 s membership lease; eight repeated runs put it at 3.0–3.5 s.
• GFS2's "never" is not a measured GFS2 recovery time. Unfenced, no fence agent exists; fenced, DLM did not leave wait fencing after pacemaker's confirmed fence, and why is not established.
• The victim is never restarted, so nothing here measures rejoin; that is bench-elasticity.sh and bench-join-latency.sh.