Userspace Filesystem Architecture: Production Lessons, Pitfalls, and Modern Approaches¶

A technical research report prepared for the EtcFS project (Raft/etcd-coordinated cluster FS over shared raw block storage). Compiled from production literature, existing FUSE filesystem implementations, kernel documentation, and distributed-systems research.

1. FUSE Daemon Design Patterns¶

1.1 Threading Models¶

FUSE supports two primary threading models, selected at mount time through fuse_loop() vs fuse_loop_mt() in libfuse or equivalent in other language bindings.

Single-threaded mode (fuse_loop):

The daemon processes one FUSE request at a time from the kernel queue.
Simplifies synchronization: no internal locking needed within the filesystem implementation.
Critical weakness: a single slow I/O operation (network call, disk seek, etcd round-trip) blocks all filesystem activity for every process using the mount point. Unacceptable for any multi-client or production workload.
Acceptable only for debugging, trivial read-only filesystems, or mountpoints with a single known consumer.

Multi-threaded mode (fuse_loop_mt):

libfuse spawns threads on demand from a thread pool when multiple requests are available in the kernel queue. Each thread independently reads from /dev/fuse and dispatches to the registered callback handlers (lookup, getattr, read, write, etc.).
Requirement: the filesystem implementation must be thread-safe — every inode table access, every metadata mutation, every cache lookup needs proper synchronization (mutexes, rwlocks, or lock-free structures).
This is the default and the only viable choice for production use.

Advanced: the FUSE_DEV_IOC_CLONE pattern. In standard multi-threaded mode, many threads contend for the lock on a single /dev/fuse file descriptor. On high-core-count machines under heavy load, this becomes a scalable bottleneck. The kernel provides ioctl(worker_fd, FUSE_DEV_IOC_CLONE, &session_fd) to create multiple independent FDs all feeding from the same session queue, eliminating FD-level contention. This is used by ScyllaDB's Seastar-based FUSE experiments and several high-throughput implementations.

1.2 Worker Pool Patterns: Beyond libfuse's Internal Thread Pool¶

When the FUSE daemon performs expensive backend operations (network calls to etcd, object storage, or large disk I/O), relying solely on libfuse's internal thread pool is insufficient. The standard pattern is:

Producer threads: libfuse threads receive FUSE requests from the kernel. They deserialize the request and push a task into a bounded queue.
Consumer pool: a fixed-size pool of worker threads/goroutines dequeues tasks, performs the actual backend work (etcd transactions, block device I/O), and signals completion back to the FUSE request.
Bounded queue = natural backpressure: when the backend is saturated, the queue fills up. The producer threads block on enqueue, which means they stop reading from /dev/fuse, which means the kernel's internal pending queue grows, which eventually blocks the calling application. This is the correct behavior — it throttles the producer rather than dropping work.

Sizing the pool:

CPU-bound tasks: pool size ~= number of cores.
I/O-bound tasks (etcd, disk, network): can over-provision significantly since threads spend most time waiting. Common ratios are 4x–16x core count, tuned empirically against latency targets.

1.3 Backpressure and Queue Depth¶

FUSE maintains multiple internal queues in the kernel:

Queue	Purpose
Pending queue	Requests generated by the kernel, waiting for the daemon to read them from `/dev/fuse`.
Processing queue	Requests read by the daemon, currently being handled.
Background queue	Asynchronous requests (background writes, readahead).

Backpressure mechanics:

When the userspace daemon is slow to pull from /dev/fuse, the pending queue grows. Once limits are reached, the kernel blocks the calling process in uninterruptible sleep — the application stalls until the daemon catches up. FUSE does not drop requests.
The max_background tunable sets the limit on asynchronous (background) requests. When exceeded, the kernel marks the backing device as "congested" via set_bdi_congested(), which signals the kernel's writeback subsystem to throttle dirty page flushing.
Monitor queue depth at /sys/fs/fuse/connections/<conn_id>/waiting. A persistently non-zero value indicates the daemon is falling behind — either it's deadlocked, crashed, or under-provisioned.

Key insight for EtcFS: a FUSE daemon that blocks waiting for an etcd transaction will stall all kernel-to-userspace I/O for its mount point. This makes etcd latency a direct multiplier on filesystem latency unless mitigated via asynchronous dispatch (see worker pool pattern above) or aggressive local caching of metadata read from etcd.

1.4 Blocking in FUSE Handlers Without Stalling the Kernel¶

The FUSE kernel-to-daemon channel is fundamentally synchronous for each individual request — the kernel expects exactly one response per request, in order for that request. However, the daemon can process different requests concurrently. Strategies:

Multi-threaded FUSE + async dispatch: the libfuse thread that receives a request immediately queues it to a worker pool and blocks waiting for the result. Other libfuse threads continue processing other requests. This is the simplest and most common pattern.
Low-level API + fuse_reply_*: the low-level (fuse_lowlevel.h) API is fundamentally asynchronous. A handler receives a fuse_req_t and can reply at any later time — it doesn't need to reply before the handler function returns. This allows fully asynchronous I/O patterns (e.g., submitting to io_uring, then replying from the completion callback).
io_uring FUSE: the kernel has ongoing work to plumb FUSE through io_uring, allowing ring-based submission and completion of FUSE requests with fewer syscalls and explicit async completion notifications. This is available in recent kernels as FUSE_IO_URING.

1.5 Connection Teardown: Graceful Unmount, Forced Unmount, Crashes¶

Graceful unmount: the daemon receives SIGTERM (or equivalent), stops accepting new work, drains in-flight operations, persists any dirty state, closes the /dev/fuse FD, and exits. The kernel detaches the filesystem cleanly.

Forced unmount (umount -f): the kernel forcibly detaches without waiting for userspace. In-flight operations may be dropped. Less safe but necessary when the daemon is unresponsive.

Lazy unmount (umount -l): detaches the mount from the namespace hierarchy immediately but keeps the kernel structures alive until the last reference is dropped. The safest approach when the daemon has crashed — avoids hangs from processes with open FDs inside the mount point.

Daemon crash: the /dev/fuse FD is closed when the daemon process exits. The kernel marks the filesystem as aborted. All in-flight and future I/O operations return EIO (input/output error). The mount point becomes a "stale" entry — ls or cd into it will hang or error out. Processes inside the mount point enter uninterruptible sleep. Recovery requires: umount -l <mountpoint>, then remount with a fresh daemon.

Standard FUSE does not support hot reconnection — once the FD is closed, the mount is dead. Production workarounds include:

Process supervision (systemd): restart the daemon, but the old mount must be unmounted and remounted.
FD persistence (systemd fdstore or equivalent): keep the /dev/fuse FD alive in a supervising process, so a replacement daemon can inherit it.
FUSE_DEV_IOC_REINIT (experimental): a kernel-side proposal to allow reinitializing the FUSE session without losing the mount. Not yet universally available.
Proxy/manager process: keep a thin, stable process that owns the /dev/fuse FD and proxies requests to the actual filesystem logic, which can be restarted independently.

For EtcFS: FUSE daemon restarts on individual nodes are tolerable since the cluster FS remains available through other nodes' mount points. The restarting node reconciles its local state against etcd (via a small local WAL, per init_plan §11). Other nodes continue serving uninterrupted.

2. Metadata Caching in FUSE Filesystems¶

2.1 Kernel-Side Caches: entry_timeout, attr_timeout, negative_timeout¶

The kernel maintains three caches for FUSE filesystems, controlled by timeouts returned in each operation's reply:

Dentry cache (entry_timeout): controls how long the kernel trusts a cached name lookup (dentry). For entry_timeout seconds, the kernel will resolve the name from cache without calling the daemon's lookup. Set to 0 to force every stat()/open() to go through userspace.
Attribute cache (attr_timeout): controls how long stat() results (size, mode, mtime, etc.) are cached in the kernel's inode structure. Set to 0 for always-fresh attributes at the cost of a round-trip per stat().
Negative cache (negative_timeout): controls how long "file not found" results are remembered. Important for workloads that repeatedly check for lock files or temp files.

The zero-timeout trap: for strong consistency, the naive approach sets all timeouts to 0. This forces every metadata operation to cross the user/kernel boundary, including the context-switch cost and the daemon's backend latency (etcd round-trip). In EtcFS, this would mean every stat() call becomes an etcd read. For a local cluster etcd deployment (~1ms RTT), this raises per-operation latency from sub-microsecond (kernel VFS) to 1–2ms (etcd + FUSE round-trip), a 1000x–10000x degradation.

Production approach: set non-zero timeouts (100–1000ms for attr_timeout, 100–500ms for entry_timeout) and couple them with explicit invalidation when remote mutations occur. This gives near-local performance during steady state while maintaining correctness.

2.2 Invalidating Kernel Caches: FUSE_NOTIFY_INVAL_*¶

When another node (or the local daemon itself) modifies metadata that the kernel has cached, the daemon must proactively push invalidation:

FUSE_NOTIFY_INVAL_ENTRY: invalidate a specific dentry (name → inode mapping). The kernel will call lookup next time the name is accessed.
FUSE_NOTIFY_INVAL_INODE: invalidate the attribute cache for a given inode. The kernel will call getattr next time.
FUSE_NOTIFY_DELETE: notify that an inode has been deleted; the kernel unmaps it.
FUSE_NOTIFY_STORE / FUSE_NOTIFY_RETRIEVE: for cache-coherent data page management (rarely used).

How distributed filesystems dispatch invalidations: In a multi-node system, Node A modifies a file, commits to the metadata store (etcd), then the metadata store — via its watch mechanism — notifies other nodes B, C, D about the change. Each node's daemon then issues the appropriate FUSE_NOTIFY_INVAL_* on its local mount point. This is how CephFS, GlusterFS, and JuiceFS maintain cache coherence across nodes without setting timeouts to zero.

For EtcFS: etcd's watch API is the natural invalidation channel. When a node's metadata client sees a change on inode:<ino> (via etcd watch), it issues FUSE_NOTIFY_INVAL_INODE for that inode locally. For dirent changes, it issues FUSE_NOTIFY_INVAL_ENTRY on the parent directory. This gives eventually-consistent cache coherence with latency bounded by etcd's watch propagation delay (typically single-digit milliseconds within a region).

2.3 The `forget` Callback¶

The forget callback is the kernel's mechanism to tell the daemon it's releasing inode references. Key behaviors:

Every lookup, create, mknod, mkdir, symlink, or link increments the kernel's internal lookup count for an inode.
When the kernel decides to evict an inode from its caches (memory pressure, cache expiry, unmount), it sends forget with a nlookup value to decrement the count.
Deferred cleanup rule: the daemon should not immediately destroy an inode on forget. The inode may still be referenced by open file descriptors or as a process's current working directory. Only when the cumulative lookup count reaches zero AND no file handles reference it should the inode be reclaimable.
forget deals with name lookup cache (dentries). release/releasedir deals with open file handles. They are independent — the kernel will not send the final forget until after release, but the ordering isn't strictly guaranteed.
Batching: the kernel may send forget_multi (batch forget) for efficiency.
Unmount hazard: during unmount, individual forget messages are not guaranteed. Do not depend on them for critical cleanup — use explicit shutdown logic.

2.4 Coherence Protocols in Production Distributed FUSE Filesystems¶

CephFS — Capability-based caching: The MDS (Metadata Server) grants clients "capabilities" (caps) — read, write, cache, etc. — for each inode. When another client requests a conflicting operation (e.g., write when another holds a read cache), the MDS issues a capability revocation to the holder. The holder must flush dirty data and release the cap before the new client proceeds. This provides strong POSIX consistency but can suffer from "cache thrashing" under highly concurrent write workloads to the same files. Tuning knobs: client_cache_size, client_caps_release_delay.

GlusterFS — Translator-stack architecture: The FUSE client receives operations from the kernel and passes them through a stack of "translators" — modular components for DHT (hashing), AFR (replication), protocol translation, etc. Cache coherence is not built into the FUSE layer; it relies on application-level coordination or the fact that most GlusterFS deployments use a single-writer-per-file workload model. Coherence across nodes for the same file generally requires external locking or careful workload design.

JuiceFS — Metadata-engine-mediated coherence: All metadata lives in Redis/TiKV/PostgreSQL. The "rich client" (FUSE daemon) caches metadata locally and watches for changes in the metadata engine. When a remote mutation occurs, the watch triggers local cache invalidation. This is architecturally the closest to EtcFS's planned model.

Rclone mount — Timeout-based only (no active invalidation): Rclone has no distributed coherence mechanism. Its cache modes (off, minimal, writes, full) are strictly local. Relying on --dir-cache-time expiry is the only coherence mechanism, making it unsuitable for multi-writer scenarios.

2.5 The Cache Timeout vs. Consistency Tradeoff¶

The fundamental tension:

Approach	Latency	Consistency	Use Case
timeouts = 0	Worst (every op = round-trip)	Strong (always fresh)	Debugging, correctness-critical single-node
long timeouts, no invalidation	Best (kernel-cache hits)	Stale reads, lost writes	Read-only data, WORM workloads
moderate timeouts + watch-based invalidation	Good (cache hits in steady state)	Eventually consistent (bounded by invalidation propagation latency)	General multi-node use
lock-lease-based caching + active invalidation	Good (cache only while holding lease)	Strong (no stale reads during valid lease)	Correctness-critical distributed FS

For EtcFS, the recommended approach is: use moderate attr_timeout/entry_timeout (100–500ms) + etcd watch-triggered FUSE_NOTIFY_INVAL_* for cross-node coherence. For files under exclusive write lock, longer timeouts are safe since only the lock-holder can modify them. For directories, shorter timeouts or explicit invalidation on dirent key changes.

2.6 Split Caches: Kernel Page Cache vs. FUSE Daemon Cache¶

Two distinct caches live at different layers:

Kernel page cache: caches file data at the VFS level. When FUSE is mounted with kernel_cache or auto_cache, the kernel keeps read pages and can serve subsequent reads without calling the daemon. This is fast (no context switch) but must be invalidated when data changes from another node — typically via FUSE_NOTIFY_INVAL_INODE which also drops data cache pages.
FUSE daemon cache: caches metadata (inode records, directory entries, extent maps) in the daemon's own memory. This avoids etcd round-trips. Must be invalidated via etcd watch events.

auto_cache mode: a middle ground where the kernel invalidates its data cache if it detects the file's size or mtime changed since the last open. Provides "close-to-open" consistency (changes visible after the file is closed and reopened) without explicit daemon invalidation. Often sufficient for many distributed workloads.

direct_io mode: bypasses the kernel page cache entirely — every read/write goes directly to the daemon. Avoids "double caching" (data in both kernel cache and daemon cache) at the cost of losing kernel-side read caching. Appropriate for workloads where the daemon manages its own buffer cache and kernel caching would be redundant waste of memory.

3. Userspace I/O Patterns for Block Devices¶

3.1 Opening and Performing I/O on Raw Block Devices¶

Raw block devices (/dev/nvme0n1, /dev/xvdf, etc.) are opened like regular files:

int fd = open("/dev/nvme1n1", O_RDWR | O_DIRECT);

Key differences from file I/O: - No filesystem layer — reads and writes address raw sectors. - No kernel buffering when O_DIRECT is used — the kernel DMA-transfers data directly between the device and the userspace buffer. - pread/pwrite with byte offsets are the standard interface. The offset and length must be multiples of the device's logical block size when using O_DIRECT.

3.2 O_DIRECT: Alignment Requirements¶

O_DIRECT bypasses the kernel's page cache. The kernel performs DMA directly to/from the userspace buffer. This requires three strict alignments:

Buffer address must be a multiple of the logical block size.
File/device offset must be a multiple of the logical block size.
Transfer length (size in bytes) must be a multiple of the logical block size.

Violation of any alignment constraint results in EINVAL (Invalid argument). There is no fallback to buffered I/O.

Memory allocation for O_DIRECT buffers:

void *buf;
size_t alignment = 4096;   // logical block size
size_t size = 1048576;     // e.g., 1 MiB, must be multiple of alignment
posix_memalign(&buf, alignment, size);

posix_memalign guarantees the buffer address is aligned to the specified boundary. Page-aligned allocations (mmap with MAP_ANONYMOUS) also satisfy the requirement since page size (4096) is >= any common block size.

3.3 Discovering Block Device Geometry¶

From /sys/block/<dev>/queue/:

logical_block_size: the smallest addressable unit for I/O. Typically 512 bytes for compatibility, though EBS recommends 4096 for modern OSes.
physical_block_size: the smallest unit the device can physically write atomically. For EBS, this is also typically reported as 512 or 4096 depending on the NVMe driver version.
minimum_io_size: preferred minimum I/O size (often matches physical block size).
optimal_io_size: preferred I/O size for best throughput (often 256K or larger for EBS).

ioctl-based discovery:

BLKSSZGET — get logical block size (the O_DIRECT alignment requirement).
BLKPBSZGET — get physical block size.
BLKGETSIZE64 — get device capacity in bytes.
BLKIOMIN — get minimum I/O size.
BLKIOOPT — get optimal I/O size.

All I/O offsets, lengths, and buffer addresses in EtcFS must be aligned to the logical block size returned by BLKSSZGET on the shared EBS volume.

3.4 EBS Volume Characteristics¶

EBS is a software-defined, distributed network storage system exposed through the AWS Nitro System. Important characteristics:

Logical block size: EBS volumes advertise either 512-byte or 4096-byte logical sectors depending on the OS and NVMe driver version. Modern AMIs (Amazon Linux 2023, Ubuntu 22.04+) typically default to 4096.
No "read-modify-write" penalty: unlike physical spinning disks with 512e/4Kn mismatch, EBS is fully virtualized — there is no RMW performance penalty for sub-4K writes at the hardware level. However, alignment still matters because misaligned writes cross EBS's internal block boundaries and may cause additional internal I/O splitting.
4K alignment best practice: while EBS tolerates 512-byte granularity, 4K alignment avoids EBS-internal overhead and matches the page size of most operating systems. Modern fdisk/gdisk/mkfs auto-align to 1MiB boundaries, which is also 4K-aligned.
Multi-Attach: available only on io2 Block Express volumes. EBS provides block-level shared access but no filesystem-level consistency guarantee. The application (EtcFS) is entirely responsible for I/O fencing and consistency. Without application-level coordination, near-simultaneous writes to the same block range will result in silent corruption — exactly what the init_plan's fencing design prevents.
Performance: EBS throughput and IOPS are tied to provisioned volume type (gp3 vs. io2) and instance-level EBS bandwidth limits, not to sector alignment. Use fio to benchmark your specific workload rather than deriving performance targets from generic EBS documentation.

3.5 io_uring for Block Device I/O¶

io_uring is the modern Linux asynchronous I/O interface, providing significant advantages over the legacy Linux AIO (libaio):

Architecture: - Two shared-memory ring buffers between kernel and userspace: the Submission Queue (SQ) and Completion Queue (CQ). - Userspace writes SQE (submission queue entries) to SQ, the kernel consumes them, performs I/O, writes CQE (completion queue entries) to CQ, userspace reads completions. - With IORING_SETUP_SQPOLL, a kernel thread polls the SQ, eliminating the io_uring_enter syscall entirely — I/O becomes truly asynchronous with zero syscall overhead.

Performance vs. Linux AIO (libaio):

Feature	Linux AIO (libaio)	io_uring
Syscall overhead	High (at least 2 per op: submit + reap)	Low to zero (shared memory rings)
Async reliability	Known to block unexpectedly (e.g., on metadata ops)	Truly asynchronous, no hidden blocking
API scope	Block I/O only (with limitations)	Unified: files, network, block, splice
O_DIRECT requirement	Required (buffered I/O falls back to synchronous)	Works with any I/O mode
Performance	Baseline	Up to 2x throughput in random write benchmarks

io_uring + O_DIRECT: the combination is the current best practice for high-performance block-device I/O. O_DIRECT avoids the kernel page cache (appropriate when the daemon manages its own cache), and io_uring provides efficient async submission/completion.

For EtcFS: io_uring is the recommended data-path I/O mechanism. Each FUSE read/write handler submits an SQE and arranges for the CQE to trigger fuse_reply_* (via the low-level API's asynchronous reply capability). This keeps FUSE handler threads from blocking on disk I/O.

3.6 O_SYNC and O_DIRECT Interaction¶

O_SYNC (or O_DSYNC) requests that every write be committed to stable storage before the syscall returns, equivalent to an implicit fsync after each write. When combined with O_DIRECT:

O_DIRECT already bypasses the kernel page cache, so writes go straight to the device.
Adding O_SYNC on top of O_DIRECT forces the device to confirm the write is on stable media (i.e., flushes the device's write cache) before returning. This is often unnecessary for EtcFS because the design's crash-consistency invariant is "data-then-metadata" — the data must be on stable media before the metadata commit in etcd, but this can be achieved with explicit fdatasync() or sync_file_range() on the block device FD immediately after the data write, without paying the per-write O_SYNC penalty on every I/O.

4. Memory Management for Userspace Filesystems¶

4.1 Buffer Management Strategies for Concurrent I/O¶

Multiple FUSE handler threads performing concurrent reads/writes against the block device need efficient buffer lifecycle management:

Per-request buffers: allocate a buffer for each I/O operation, free after completion. Simple but generates allocation/deallocation churn.
Buffer pooling (slab/slab-like): pre-allocate a pool of aligned buffers, check out for each I/O, return to pool. Eliminates allocation hot-path overhead. Essential for managed languages (Go, Java) where GC pressure from per-request allocation can dominate latency.
Ring buffers for sequential I/O: for streaming reads/writes, a ring buffer with two pointers (producer/consumer) avoids repeated allocation.
Size classes: maintain separate pools for common I/O sizes (4K, 64K, 256K, 1M) to minimize fragmentation.

4.2 Zero-Copy Approaches: splice and vmsplice¶

FUSE supports kernel-to-userspace data transfer via splice to avoid an extra memory copy:

splice (kernel → userspace): the kernel passes page references (pointer + length) rather than copying bytes. The daemon must handle "spliced" buffers specially — they're backed by kernel pages and have different lifetime/reference semantics than userspace-allocated buffers.
FUSE_CAP_SPLICE_WRITE / FUSE_CAP_SPLICE_READ: capability flags negotiated during the FUSE init handshake. The daemon advertises its ability to handle splice.
FUSE_CAP_SPLICE_MOVE: an optimization where the kernel can "move" pages between pipes, avoiding copies entirely.
Limitations: splice is most effective for large I/O (> page size). For small I/O, the overhead of managing splice semantics may exceed the copy cost. Also, some kernel memory regions (particularly under writeback) are "unmovable," forcing fallback to internal copies.

For EtcFS: splice is useful for the FUSE → daemon data path when handling large sequential reads/writes. However, the daemon → block device path (O_DIRECT pread/pwrite) already bypasses the kernel page cache, so the main benefit of splice is eliminating the kernel-to-userspace copy, not the userspace-to-disk copy.

4.3 Huge Pages for I/O Buffers¶

HugeTLB (2MB or 1GB pages) or Transparent Huge Pages (THP) reduce TLB (Translation Lookaside Buffer) pressure when the daemon touches large amounts of memory for I/O buffers:

TLB miss cost: on a system processing millions of I/O operations per second, TLB misses from fragmented 4K page mappings can become a significant portion of CPU time.
When to use: if the daemon's I/O buffer working set exceeds ~100MB and TLB misses appear in perf profiles, enabling huge pages for the buffer pool is a targeted optimization.
NUMA interaction: allocate huge pages per-NUMA-node to avoid cross-socket memory access latency. Use numactl --membind=<node> or mbind() to constrain huge page allocations to the local node.

4.4 Memory Bandwidth Considerations¶

A FUSE daemon handling high-throughput I/O faces memory bandwidth as a potential bottleneck:

Every byte read from the block device and served to FUSE traverses system memory: device → DMA to userspace buffer → kernel reads from userspace buffer (for FUSE reply) → application memory. This is a minimum of two full traversals of the memory bus per byte, plus any extra copies (kernel page cache, splice pipe pages).
The practical ceiling: on a system with 100 GB/s memory bandwidth, sustainable filesystem throughput will be ~30–50 GB/s after accounting for copies, protocol overhead, and metadata work.
For EBS io2 Block Express volumes with a maximum throughput of 4,000 MB/s per volume, memory bandwidth is unlikely to be the bottleneck for EtcFS — the EBS throughput limit will be hit first.

4.5 Lock-Free Data Structures for Inode Caches¶

The inode cache is the classic "read-mostly" data structure in a filesystem — thousands of reads (stat, lookup) for every modification (write, create, delete). Contention on reader-writer locks becomes a bottleneck at high concurrency.

RCU (Read-Copy-Update): the gold standard for kernel-level read-mostly data structures: - Readers enter an RCU critical section (essentially a compiler barrier, not an atomic operation). They can traverse the cache without acquiring any locks. - Writers create a new version of the data structure and atomically swap the pointer. They defer freeing the old version until all pre-existing readers have finished (a "grace period"). - Used extensively in the Linux kernel for the dcache, inode cache, and networking structures.

Hazard Pointers: a userspace alternative to RCU: - Before dereferencing a pointer, a reader publishes the pointer address in a thread-local "hazard pointer" slot. - Before freeing an object, a writer scans all threads' hazard pointer slots. If the object's address appears in any slot, the writer waits (retrying later). - Memory-bounded (unlike RCU, which can accumulate garbage if grace periods are delayed). Suitable for userspace where kernel RCU primitives aren't available.

Concurrent hash tables: for the inode cache, a concurrent hash table (e.g., a sharded hash table with one rwlock per shard, or a lock-free design using compare-and-swap for bucket pointer updates) provides scalable lookup. The et cFS daemon will need this: every lookup and getattr hits the inode cache before (or instead of) consulting etcd.

4.6 NUMA-Aware Memory Allocation¶

On multi-socket machines, memory access latency varies: ~100ns for local memory, ~150–200ns for remote memory. For a daemon processing millions of I/O ops/second, pinning threads and memory to the local NUMA node matters.

Thread affinity: bind FUSE handler threads and I/O worker threads to the same NUMA node as the block device's interrupt affinity.
Memory affinity: numactl --membind=<node> or mbind() for buffer pool allocations. move_pages() to migrate hot pages to the local node.
libnuma: numa_alloc_onnode() for NUMA-aware buffer allocation.
EBS-specific: on AWS Nitro instances, EBS volumes are attached via NVMe controllers on specific PCIe slots; these have a fixed NUMA affinity. Align the daemon's CPU and memory to the same node for best throughput. Use ls /sys/block/nvme*/device/numa_node to determine the NUMA node of each attached EBS volume.

5. Existing Userspace Filesystem Implementations (Case Studies)¶

5.1 s3fs-fuse: S3 as a POSIX Filesystem¶

Architecture: Translates FUSE operations (open, read, write) into S3 API calls (GET, PUT, DELETE). Runs entirely in userspace with no kernel component.

POSIX semantics on eventually-consistent storage: S3 provides strong read-after-write consistency (since 2020), but s3fs operates on top of it with limitations: - No file locking (no flock/fcntl support). - Directories are "simulated" — S3 has no directories, only key prefixes. Renaming a directory requires copy/delete of every object with that prefix, which is slow and non-atomic. - Atomic renames across directories are impossible — they decompose into multiple S3 operations. - chmod/chown are simulated locally via mount options; they don't map to S3 object metadata.

Caching strategy: - Metadata caching in memory (stat cache, directory listing cache). - Optional local disk cache for data (-o use_cache=/path). Files are cached locally until close()/fsync(), then pushed to S3. - Multi-node hazard: s3fs has no cross-node cache invalidation. Two s3fs instances mounting the same bucket will see stale metadata until their local caches expire. No distributed locking. Data corruption is possible if two instances write to the same key.

Production verdict: widely considered not production-ready for multi-writer workloads. Acceptable for single-writer, read-heavy, or WORM (write-once-read-many) use cases. Not a model for EtcFS.

5.2 Rclone Mount: VFS Cache Layers and Writeback¶

Architecture: Rclone's mount command exposes any of its 40+ cloud storage backends as a FUSE filesystem. The VFS layer sits between FUSE and the backend.

VFS cache modes: - off: no local caching. Direct streaming from/to remote. Fragile — many applications break because cloud storage doesn't support partial writes or random I/O. - minimal (default): caches metadata and very small data chunks. Lightweight. - writes: caches writes locally; uploads on close(). More stable than off for write-heavy workloads. - full: full read cache (disk-backed) + write cache. Most "local-like." Suitable for media streaming, random reads, editing.

Writeback architecture: - Writes go to a local temp file in --cache-dir. - On close(), a timer (--vfs-write-back) starts; when it expires, the file is uploaded to the backend. - If rclone crashes, partially written files remain in the cache and resume on restart. - Disk space constraint: the cache directory must have sufficient free space to hold all pending uploads. If the cache fills, writes block.

Dir-cache: a separate, memory-resident cache for directory listings, managed by --dir-cache-time. Avoids LIST/equivalent calls on every ls.

Relevance to EtcFS: Rclone's VFS cache layer architecture — separating metadata cache from data cache, managing writeback asynchronously — is a useful reference for the EtcFS daemon's internal cache design. The key difference: Rclone has no distributed coherence; EtcFS does (via etcd watches).

5.3 JuiceFS: Metadata-Engine-Mediated Distributed Filesystem¶

Architecture: Decoupled data and metadata planes: - Data plane: file content stored as fixed-size chunks (64 MB default) in object storage (S3, MinIO, GCS, etc.). Chunks are subdivided into 4 MB blocks for parallel upload. - Metadata plane: stored in an external database engine — Redis (for low latency, small-to-medium scale), TiKV (for horizontal scalability, billions of files), MySQL, or PostgreSQL. - Rich client: the FUSE daemon handles chunking, local caching, metadata caching, and metadata-engine interaction.

Production lessons: - Metadata engine is the bottleneck: metadata performance directly determines filesystem performance. Redis provides sub-millisecond metadata ops but is limited by single-instance memory. TiKV scales horizontally via Raft replication but has higher per-operation latency. - Always enable automatic metadata backups. JuiceFS includes periodic metadata snapshotting to object storage — this is not optional in production. - Avoid writeback caching in volatile environments (containers, spot instances). Writeback caching trades durability for performance; without proper shutdown hooks, data loss is guaranteed. - For Kubernetes: use the CSI driver with proper preStop hooks to ensure the client flushes gracefully before pod termination. - Client-side caching is essential: JuiceFS achieves near-local-disk performance by caching both metadata and data on the client, with metadata engine watches triggering invalidation.

Relevance to EtcFS: JuiceFS is the closest architectural analogue to EtcFS. The main structural difference: JuiceFS stores data in object storage (S3), while EtcFS stores data on a shared raw block device. The metadata-plane pattern (external database as single source of truth, client-side caching with watch-based invalidation) is identical.

5.4 GeeseFS: Performance-Optimized S3 FUSE in Go¶

Architecture: A Go-based FUSE filesystem for S3, forked from goofys and heavily optimized.

Key performance optimizations: - Asynchronous operations: writes, renames, deletes are performed asynchronously — the FUSE handler returns immediately, and the S3 API call happens in the background. This masks S3's high per-operation latency. - Massive parallelism: parallel uploads (multipart), parallel reads (readahead), parallel metadata fetches. - Heuristic readahead: detects sequential vs. random access patterns and adjusts prefetching behavior. For random reads, proactively downloads smaller blocks to a local cache to satisfy future nearby reads. - Partial updates (--enable-patch): for small modifications to large objects, uploads only the changed bytes rather than the entire object. Critical for append-heavy workloads. - Background directory loading: when a directory is accessed, downloads the entire directory tree in the background, avoiding per-file stat calls on S3 (which are expensive).

Relevance to EtcFS: the async operation pattern (return before backend work completes) maps directly to the low-level FUSE API's fuse_reply_* from completion callbacks. The readahead heuristics and partial-update optimizations apply to any latency-sensitive storage backend, including raw block devices.

5.5 gcsfuse: Partial POSIX Compliance for Google Cloud Storage¶

Architecture: Translates FUSE ops to GCS API calls. Exposes GCS buckets as a local directory tree.

What it does NOT support (and why): - No file locking (GCS has no native locking primitive). - No atomic renames across directories. - No hard links (GCS objects are flat key-value pairs). - No chmod propagation (POSIX permissions are local simulation; real security is IAM-based). - Object versioning and lifecycle policies interact unpredictably with the filesystem abstraction.

When it works: migrating legacy apps that need a filesystem interface to access GCS data, AI/ML training pipelines reading large datasets, static asset serving.

When it fails: any workload requiring POSIX guarantees — file locking, atomic multi-file operations, high-frequency metadata updates, low-latency random I/O.

Relevance to EtcFS: gcsfuse demonstrates how much POSIX semantics you can practically implement on a non-POSIX backend, and where the hard boundaries are. EtcFS has the advantage of a block device and a transactional metadata store, so it can implement full POSIX semantics — the question is which corners are worth cutting for performance, and gcsfuse shows which cuts hurt the most (file locking and atomic renames are the most frequently missed).

5.6 Perkeep (Camlistore): Content-Addressable Storage with FUSE Frontend¶

Architecture: Immutable, content-addressed blobs stored in various backends (local disk, S3, GCS, etc.). The FUSE frontend (pk-mount) provides a POSIX filesystem view over the blob graph.

How the FUSE layer works: - The FUSE daemon dynamically synthesizes directory structures from the metadata index (a searchable graph of "permanodes" and "claims"). - When you open a directory, the daemon queries the index for matching permanodes and presents them as virtual files. - Reads are served from the blob store, with local caching to avoid re-downloading. - The filesystem is read-only for practical purposes — writes change the blob graph, which doesn't map cleanly to in-place file modification.

Relevance to EtcFS: Perkeep demonstrates the architectural pattern of separating storage (blob store) from presentation (FUSE). EtcFS follows the same philosophy: the block device is a dumb byte array, and the FUSE daemon provides all structure. Perkeep's read-only focus limits its applicability, but the dynamic-synthesis approach (directory listing is a query, not a read) maps to EtcFS's etcd prefix-range scans for dirent:<parent_ino>/.

5.7 GlusterFS FUSE Client: Translator-Stack Architecture¶

Architecture: The GlusterFS client process receives FUSE requests and passes them through a stack of dynamically loaded "translators" — modular components that each perform one filesystem function.

Translator stack (client-side): 1. FUSE translator (top): receives requests from /dev/fuse. 2. DHT (Distributed Hash Table) translator: hashes the filename to determine which brick (server-side storage directory) holds the file. 3. AFR (Automatic File Replication) translator: duplicates writes to replica bricks for replicated volumes. 4. Protocol Client translator (bottom): serializes the operation into the GlusterFS RPC protocol and sends it over TCP/IP or RDMA to the storage server.

On the server side: the Protocol Server unpacks the RPC and passes the operation through a server-side translator stack (including a POSIX translator that performs actual filesystem calls against the local XFS/ext4 filesystem).

Relevance to EtcFS: GlusterFS's modular translator architecture is a reference for how to structure EtcFS internally — each subsystem (FUSE frontend, metadata client, data engine, fencing agent) could be a translator-like module with well-defined input/output interfaces rather than a monolithic implementation.

5.8 CephFS FUSE Client: Capability-Based Caching¶

(Detailed in §2.4.) Additional operational notes:

Kernel client vs. FUSE client: the kernel client is ~20–40% faster for large I/O and has ~10–20% lower metadata latency because it avoids the user/kernel context switch. Ceph recommends the kernel client for production; the FUSE client is used primarily for debugging, development, or environments where the kernel version can't match Ceph's required version.
MDS session sensitivity: the FUSE client is sensitive to network instability. mds_session_timeout tuning is critical for WAN deployments.
Memory monitoring: ceph-fuse is a userspace process — its metadata cache grows proportionally to the working set. Memory usage must be monitored.
fuse_big_writes: enabling this allows larger write requests (up to 1MB), significantly improving throughput for sequential writes.

Relevance to EtcFS: the capability-based coherence model (caps granted by the MDS, revoked on conflict) is a more sophisticated alternative to timeout-based or watch-based invalidation. EtcFS could adopt a simplified version: a read lock on an inode grants a "read cache cap" for that inode's attributes; holding an exclusive lock grants a "write cache cap." When the lock is reclaimed (due to fencing), the cap is revoked, and the node must invalidate its local caches.

6. Language Choice Considerations¶

6.1 Summary Comparison¶

Criterion	C (libfuse)	C++ (libfuse)	Rust (fuser/fuse-backend-rs)	Go (go-fuse/bazil/jacobsa)
Maturity of FUSE library	Reference implementation, 20+ years	Same C library, C++ wrappers	Pure Rust rewrite, mature and maintained	Several libraries, hanwen/go-fuse most performant
Performance (raw I/O)	Best (zero overhead, manual memory)	Near-C with virtual dispatch overhead	Near-C (no GC, zero-cost abstractions)	Good for I/O-bound; GC bound for allocation-heavy
P99 latency predictability	Excellent (no GC, no runtime)	Excellent	Excellent (no GC, predictable async)	Variable (GC pauses, improving each Go release)
Memory safety	Manual (buffer overflows, use-after-free, leaks)	Manual (RAII helps, but raw pointers still possible)	Compile-time (ownership/borrowing)	Runtime (GC prevents use-after-free, but leaks possible)
Concurrency safety	Manual (pthreads, mutexes, data races possible)	Manual (std::mutex, std::atomic, races possible)	Compile-time (Send/Sync traits prevent data races)	Runtime (goroutines + channels, races possible but -race detects)
Development velocity	Slowest (manual memory, verbose APIs)	Moderate (RAII, STL, but complex build)	Moderate-to-slow (steep learning curve, fast once proficient)	Fastest (simple concurrency, fast compile, excellent tooling)
Ecosystem	Vast C ecosystem	Full C ecosystem + C++ libraries	Growing; strong systems and async ecosystem	Large standard library; strong networking/cloud ecosystem
Production FUSE filesystems	Most (s3fs, sshfs, unionfs-fuse, CephFS FUSE)	Some (GCSFuse is Go; most are C)	Growing (virtiofs via fuse-backend-rs)	Several (gcsfuse, geesefs, rclone, goofys)

6.2 C: libfuse (Reference Implementation)¶

Strengths: - The reference FUSE implementation. Every kernel FUSE feature appears here first. - Maximum performance: zero runtime overhead, manual memory management, full control over threading. - The low-level API (fuse_lowlevel.h) provides the most control for asynchronous I/O patterns. - Proven in production across dozens of filesystems.

Weaknesses: - Memory safety is the developer's responsibility. Buffer overflows, use-after-free, double-free bugs are common in complex C FUSE daemons. - Concurrency bugs (data races, deadlocks) are easy to introduce and hard to reproduce. - Heavy development burden for complex, multi-threaded systems.

For EtcFS: C gives maximum control over the data path and memory layout, but shifts all safety responsibility to the developer. If chosen, use the low-level API, static analysis (Coverity, Clang analyzer), and rigorous fault-injection testing.

6.3 C++: libfuse with C++ Bindings¶

Approach: libfuse is a C library. C++ integration requires a thin glue layer: store a this pointer in fuse_context->private_data, and write static C callback functions that cast private_data back to the C++ class.

Key rules: - Never let exceptions propagate through C callback boundaries. Wrap every callback in try/catch and translate exceptions to negative errno returns. An uncaught exception across a C -> C++ boundary is undefined behavior. - RAII helps with resource management (file descriptors, aligned buffers, etcd connections) but doesn't prevent use-after-free from concurrent access — data race protection is still manual. - std::shared_ptr/std::weak_ptr for inode reference counting can replace manual refcounting, but atomics on shared_ptr control blocks add cache-line contention under high concurrency.

Production examples: few standalone production C++ FUSE filesystems exist. Most production filesystems using libfuse are either C or have moved to higher-level languages (Go, Rust). C++ is a middle ground: better memory management than C, but still manual concurrency safety.

6.4 Rust: `fuser` Crate and `fuse-backend-rs`¶

fuser crate (cberner/fuser): - A pure Rust rewrite of libfuse, not a binding. - Synchronous blocking I/O model with a thread pool. - Mature, actively maintained, used in production-like settings. - Limitation: the synchronous model means each FUSE operation blocks a thread. For I/O-bound filesystems (etcd calls, block device I/O), threads may sit idle unless the pool is large enough. Over-provisioning threads in Rust is cheaper than in C (Rust threads are still OS threads), but there's a practical limit.

fuse-backend-rs (cloud-hypervisor/fuse-backend-rs): - Designed for high-concurrency, performance-sensitive systems (virtio-fs, container filesystems). - More flexible architecture; used by cloud-hypervisor for virtio-fs daemon. - Better suited for async I/O patterns.

fuse3 / rfuse3: - Native async API support via Tokio or async-std. - Allows the daemon to handle FUSE requests and backend I/O within the same async runtime, avoiding thread-per-request blocking.

Rust advantages for EtcFS: - The Send/Sync trait system prevents data races at compile time — critical for a multi-threaded FUSE daemon where concurrent access to the inode cache, extent lists, and arena free-lists is the norm. - No GC means predictable P99 latency, important for FUSE where kernel-to-userspace timeouts can cause applications to see EIO or hangs. - Box<[u8]> with posix_memalign-like alignment via the alignment crate for O_DIRECT buffers. - The etcd-client crate provides native async Rust bindings for etcd v3 API, with lease management and watch streams — ideal for the metadata client subsystem.

Rust risks for EtcFS: - Learning curve: the team must be proficient in Rust's ownership and async models. - Fewer FUSE-specific examples and community resources compared to C. - fuser's synchronous model may not scale to the concurrency EtcFS needs; fuse-backend-rs or a custom io_uring-based implementation might be required.

6.5 Go: bazil.org/fuse, jacobsa/fuse, hanwen/go-fuse¶

Library landscape: - bazil.org/fuse: widely used, stable, serves as the foundation for several projects. Not the highest-performance option but well-documented. - jacobsa/fuse: cleaner API, but slower (1.8x–3x behind hanwen/go-fuse in throughput benchmarks) and sporadically maintained. Missing support for newer FUSE protocol features. - hanwen/go-fuse: the most performant and actively maintained Go FUSE library. Supports newer protocol versions and has been optimized for throughput. The recommended choice if using Go.

Go performance characteristics for FUSE: - Goroutines are cheap: M:N scheduling means a goroutine-per-request model is practical. The runtime efficiently handles thousands of concurrent FUSE operations. - GC pause concern: Go's GC has improved dramatically (sub-millisecond pauses in Go 1.19+). However, under sustained high allocation rates (millions of small buffers per second), GC can still cause latency spikes. Mitigation: buffer pooling (sync.Pool), pre-allocation, minimizing allocations in the I/O hot path. - Syscall overhead: Go's syscall path has slightly higher overhead than C due to the runtime's stack management and scheduling. For FUSE, where every operation is a syscall, this overhead accumulates. Go FUSE filesystems are typically ~10–30% slower than equivalent C implementations in microbenchmarks, though backend latency (etcd, disk I/O) usually dominates.

Go production FUSE examples: gcsfuse, geesefs, rclone mount, goofys. All are I/O-bound to cloud storage, so Go's overhead is negligible relative to network latency.

Go advantages for EtcFS: - Fastest development velocity. The goroutine model maps naturally to the asynchronous FUSE request pattern. - Excellent etcd client (go.etcd.io/etcd/client/v3), officially maintained. - Strong networking and concurrency primitives.

Go risks for EtcFS: - GC pauses under allocation-heavy workloads could cause kernel-to-daemon latency spikes, which translate to application-visible FUSE operation latency. - O_DIRECT buffer alignment requires syscall.Mmap or CGo (posix_memalign), adding complexity. - Go's io_uring support is nascent; the data path might need to use pread/pwrite with O_DIRECT through the syscall or golang.org/x/sys/unix packages, which is less efficient than io_uring.

6.6 Language Recommendation Summary for EtcFS¶

Given EtcFS's requirements — correctness-critical fencing, concurrent multi-threaded daemon, block device I/O with O_DIRECT, etcd integration, and a build order that prioritizes fencing safety over raw throughput — the tradeoffs are:

Priority	Best Fit
Correctness and data-race safety	Rust (compile-time guarantees)
Development velocity	Go (goroutines, fast compile-test cycle)
Maximum I/O throughput	C (zero overhead)
Balanced safety + performance + velocity	Rust (fuse-backend-rs + async) or Go (go-fuse + sync.Pool)

The init_plan's build order (§15) prioritizes the fault-injection harness and fencing logic before the data path. This makes Rust particularly attractive: the Send/Sync compile-time guarantees prevent entire classes of bugs that would otherwise consume unbounded debugging time in the fault-injection phase. However, if the team's Rust proficiency is low, Go with rigorous use of the race detector (go test -race) and Jepsen-style integration testing is a reasonable alternative.

7. Test Harness and Fault Injection for Filesystems¶

7.1 Filesystem Correctness Testing Tools¶

xfstests: - The industry-standard regression test suite, originally developed for XFS by SGI, now the primary testing toolkit for all Linux filesystems (ext4, btrfs, NFS, CIFS, CephFS). - Contains hundreds of test cases organized into groups: quick (fast sanity checks), auto (full suite for automated runs), dangerous (may crash kernel/cause data loss). - Tests exercise every filesystem operation under stress, concurrency, and edge cases (e.g., fsx random I/O, fsstress randomized operation sequences, ENOSPC handling, crash recovery). - How to run against a FUSE filesystem: mount the FUSE filesystem at a test directory, point xfstests at it with TEST_DEV and TEST_DIR configured, run with ./check -fuse. Many tests assume a real block device and will need adaptation or filtering.

pjdfstest: - A targeted POSIX compliance test suite. Exercises each POSIX filesystem syscall with edge cases (permission checks, timestamp updates, symlink/hardlink semantics, rename atomicity, directory operations). - Smaller and faster than xfstests — useful as a first-pass sanity check during development. - Not a stress or concurrency test; purely about semantic correctness.

fsx (file system exerciser): - Part of xfstests but also available standalone. Generates random sequences of read, write, mmap, truncate, and fsync operations against a test file, comparing the actual content against a golden copy in memory. - Particularly good at finding data corruption bugs from incorrect write ordering or misaligned extent handling.

LTP (Linux Test Project): - A broad kernel test suite that includes filesystem-specific tests. - Less targeted than xfstests for filesystems but covers interactions with other kernel subsystems (memory management, scheduler) under filesystem load.

7.2 Ceph's Testing Approach¶

Ceph has one of the most comprehensive testing infrastructures in open-source distributed systems:

Teuthology: - A Python-based orchestration framework for multi-node integration testing. - Provisions clusters (bare-metal, VMs, or cloud), installs software, executes tests defined in YAML manifests, collects results. - Tests are organized into ceph-qa-suite and run nightly against development branches, with results visible in Pulpito (dashboard) and Paddles (results database). - Can integrate xfstests as "workunits" — xfstests is run against CephFS mount points as part of the standard test suite.

Fault injection in Ceph: - Network faults: dropping packets, injecting latency between OSDs/MDSs to test timeout and retry logic. - I/O faults: simulating disk errors, EIO returns from OSDs, verifying the filesystem handles them without panicking or corrupting data. - Node crash testing: killing OSDs, MDSs, or clients mid-operation and verifying consistency after recovery.

Chaos engineering methodology: - Define a "steady state" (normal throughput, latency, error rate). - Form a hypothesis (e.g., "the system remains available if 20% of OSDs are partitioned"). - Use Teuthology to automate fault injection while monitoring the steady state. - Verify that the system returns to steady state after the fault is removed.

7.3 Jepsen-Style Testing for Distributed Filesystems¶

Jepsen is a Clojure-based framework for verifying the consistency and durability of distributed systems under fault injection. It's famous for finding subtle bugs in etcd, MongoDB, Kafka, and other distributed databases.

How Jepsen works: 1. Setup: deploy a cluster of nodes running the system under test. 2. Workload: run a well-defined set of operations with known expected outcomes (e.g., "append element X to a set; after the test, the set should contain exactly the elements that were acknowledged as appended"). 3. Nemesis: concurrently inject faults — network partitions (iptables rules), node crashes (SIGKILL), clock skews, disk failures, process pauses (SIGSTOP). 4. Checker: after the test, verify whether the system's observed history is consistent with its claimed consistency model (linearizability, serializability, eventual consistency, etc.).

Adapting Jepsen for a FUSE filesystem: - The workload generator writes known patterns to files, fsyncs, and records which writes were acknowledged. - The nemesis injects faults targeting the FUSE daemon, the metadata backend (etcd), the network, and the block device. - The checker verifies that after faults, the filesystem state matches what a linearizable or correctly-ordered history would allow. - Key Jepsen test for EtcFS: partition a node from etcd but keep its connection to the EBS volume alive — the exact scenario the self-fencing watchdog exists to handle. The nemesis must simulate this by iptables-blocking the etcd port while leaving the NVMe/block device path open.

7.4 Deterministic Simulation Testing (FoundationDB Model)¶

FoundationDB's testing approach is widely considered the gold standard for distributed systems correctness:

Core principle: "Simulate everything." - All sources of nondeterminism — networking, file I/O, clocks, random number generation, thread scheduling — are replaced with simulated versions. - The entire distributed system runs inside a single-threaded, discrete-event simulator. - A test seeded with a random value produces the exact same execution path every time. If a bug is found, it's perfectly reproducible.

Key characteristics: - Deterministic execution: the simulator controls all inputs, including pseudo-random ones. - Single-threaded control: eliminates non-deterministic thread interleaving. Concurrency is managed through a custom programming model (Flow) where actors communicate within the virtualized environment. - Aggressive fault injection: the simulator injects network partitions, machine crashes, disk corruption, and bit flips at rates far higher than reality. - Time compression: the simulator can "fast-forward" through time, running the equivalent of trillions of CPU-hours of cluster operation in test suites that complete in minutes.

Results: FoundationDB ran its code exclusively in simulation for 18 months before touching real hardware. When first deployed on real clusters, no data-loss bugs were found.

Projects adopting this approach: TigerBeetle (financial transactions DB), WarpStream (Kafka-compatible streaming), Antithesis (commercial deterministic simulation platform), various internal AWS systems.

For EtcFS: the init_plan (§15) explicitly calls for a "deterministic fault-injection harness, Jepsen-style" before trusting the design with real data. The FoundationDB model — deterministic simulation where every fault is reproducible — is the ideal target. This would require: 1. Abstracting all I/O (etcd calls, block device reads/writes, FUSE protocol messages) behind interfaces that can be replaced with simulated versions. 2. Making the daemon's event loop driven by a simulated clock and simulated I/O completions, not real time and real syscalls. 3. Writing targeted tests against specific invariants (fencing generation CAS, data-then-metadata ordering, extent uniqueness, lock-to-generation binding).

7.5 Building a Reproducible Test Harness¶

Key principles for building a filesystem test harness that can reproduce bugs:

Record-replay: - Record all inputs to the filesystem (FUSE requests, etcd responses, block device read results) and the interleaving of concurrent operations. - Replay the recording against a known state to reproduce the bug. - This requires deterministic FUSE response injection (mock the kernel side) and deterministic etcd responses (mock or record the etcd client).

Minimization: - Once a bug is reproduced, minimize the reproduction case: reduce the number of operations, shrink the file sizes, simplify the concurrency pattern, until you have the smallest possible test that triggers the bug. - Tools like creduce (C-Reduce) or manual delta-debugging can help.

Targeted invariant checking: - Don't just run random operations and check for crashes. Define explicit invariants and check them after every operation: - No extent is referenced by two different inodes. - Every extent in an inode's extent list has a valid disk_off within the owning arena's range. - Every inode:<ino> record's nlink matches the count of dirent:* entries pointing to it. - After a crash + recovery, every file's data matches what was fsynced before the crash. - After a fencing event, no writes occur from the fenced node (check via generation stamps).

7.6 Chaos Engineering for Distributed Storage¶

Beyond specific test suites, chaos engineering provides a methodology for ongoing resilience validation:

Principles: 1. Define steady state: what does "normal" look like? (latency percentiles, throughput, error rate, etcd round-trip time). 2. Hypothesize: "the system remains available and consistent when X fails." 3. Inject real-world faults: run experiments in production (or staging mirroring production). 4. Measure and verify: does the system deviate from steady state? Does it recover? 5. Minimize blast radius: start small (one node, one disk), expand only after the system handles small faults.

Chaos engineering tools for filesystems: - Chaos Mesh / LitmusChaos: Kubernetes-native chaos injection (pod kills, network partitions, I/O faults, CPU/memory pressure). Useful if EtcFS runs in a Kubernetes environment. - ChaosFS: a FUSE-based fault injection layer that can interpose between the filesystem and applications, simulating specific I/O errors. - BPF-based fault injection: eBPF programs can intercept and modify syscall returns (e.g., make pread return partial reads, inject EIO, delay I/O operations). More flexible than kernel-module-based injection. - tc + iptables: network-level chaos: partition the etcd network, inject latency, drop packets, simulate multi-AZ degradation. - systemctl kill / SIGSTOP: node-level chaos: crash the daemon, pause it (simulating GC pause or scheduler freeze), kill the etcd sidecar.

For EtcFS specifically: the most valuable chaos experiments target the fencing boundaries: 1. Partition a node from etcd while the block device path stays alive (self-fencing test). 2. Crash the fencing controller mid-operation (dual-confirmation test). 3. Corrupt a block on the shared device and verify the scrubber detects it. 4. Hold the etcd leader election during an in-flight write transaction. 5. Freeze a node (SIGSTOP) for longer than the etcd lease TTL, then resume it — verify it self-fences and does not resume writing.

References¶

FUSE kernel documentation: Linux kernel source, Documentation/filesystems/fuse.rst
libfuse: https://github.com/libfuse/libfuse
CephFS architecture: https://docs.ceph.com/en/latest/cephfs/
Ceph testing (Teuthology): https://github.com/ceph/teuthology
FoundationDB deterministic simulation: https://www.foundationdb.org/files/fdb-paper.pdf
Jepsen: https://jepsen.io/
JuiceFS architecture: https://juicefs.com/docs/community/architecture
rclone mount VFS: https://rclone.org/commands/rclone_mount/
GlusterFS translator architecture: https://docs.gluster.org/en/latest/Quick-Start-Guide/Architecture/
Perkeep architecture: https://perkeep.org/doc/arch
gcsfuse: https://github.com/GoogleCloudPlatform/gcsfuse
geesefs: https://github.com/yandex-cloud/geesefs
s3fs-fuse: https://github.com/s3fs-fuse/s3fs-fuse
Go FUSE (hanwen/go-fuse): https://github.com/hanwen/go-fuse
Rust fuser: https://github.com/cberner/fuser
Rust fuse-backend-rs: https://github.com/cloud-hypervisor/fuse-backend-rs
io_uring: https://kernel.dk/io_uring.pdf
O_DIRECT alignment: Linux man pages: open(2), posix_memalign(3)
EBS Multi-Attach: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volumes-multi.html
xfstests: https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git
pjdfstest: https://github.com/pjd/pjdfstest