Single-Node Integration

How the four subsystems — FUSE frontend, metadata client, block device I/O, and fencing agent — compose into a single running daemon on one node, and how the daemon survives crashes, recovers state, and resumes operation.

Table of Contents

• Subsystem Architecture
• Startup Sequence
• Membership Heartbeat
• FUSE Session Lifecycle
• IPC Request-Response Loop
• Crash Recovery Protocol
• State Reconstruction
• Write Path Walkthrough
• Interaction with the Self-Fencing Watchdog
• Graceful Shutdown

Subsystem Architecture

Each EtcFS node runs two OS processes that communicate over a Unix stream socket:

┌──────────────────────────────────────────────────────────────────┐
│  etcfuse-meta (Go binary)                                       │
│                                                                  │
│  ┌────────────────┐  ┌────────────────┐  ┌───────────────────┐  │
│  │ Metadata Store │  │ Membership     │  │ Self-Fencing      │  │
│  │ (etcd client)  │  │ Heartbeat      │  │ Watchdog          │  │
│  │                │  │                │  │                   │  │
│  │ inode CRUD     │  │ lease keepalive│  │ lease health poll │  │
│  │ dirent ops     │  │ re-grant on    │  │ 2×TTL margin      │  │
│  │ lock mgmt      │  │ channel close  │  │ exit code 77      │  │
│  │ generation      │  │                │  │                   │  │
│  └───────┬────────┘  └────────────────┘  └───────────────────┘  │
│          │                                                       │
│  ┌───────▼──────────────────────────────────────────────────┐   │
│  │ IPC Service                                              │   │
│  │ dispatches FUSE ops → metadata calls, returns binary     │   │
│  │ responses → writes back to socket                        │   │
│  └───────────────────────┬──────────────────────────────────┘   │
│                          │ Unix socket                          │
└──────────────────────────┼──────────────────────────────────────┘
                           │
┌──────────────────────────┼──────────────────────────────────────┐
│  etcfuse (C daemon)     │                                       │
│                          ▼                                       │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ IPC Worker (single pthread, socket I/O)                 │   │
│  │ processes request queue → send/receive → callback       │   │
│  └──────────┬──────────────────────────────────────────────┘   │
│             │                                                  │
│  ┌──────────▼──────────────────────────────────────────────┐   │
│  │ FUSE Session (multi-threaded, fuse_session_loop_mt)      │   │
│  │ handlers: ec_lookup, ec_getattr, ec_create, ...         │   │
│  │ each handler does synchronous IPC and calls fuse_reply   │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                  │
│  Block Device (O_DIRECT fd, shared EBS Multi-Attach)            │
└──────────────────────────────────────────────────────────────────┘

The Go binary owns all stateful subsystems: the etcd connection, the metadata cache, the membership lease, and the self-fencing watchdog. The C binary owns the FUSE session (the kernel upcall dispatcher) and the block device file descriptor. They meet at the IPC boundary, where a dedicated pthread in C serialises request/response pairs against a single Unix socket connection.

Per-Process Responsibility

Go daemon (etcfuse-meta): - Maintain the connection to the etcd cluster (TLS, auth, connection pool). - Execute metadata operations: inode Lookup/GetAttr/Create/Delete, dirent Create/Remove/Rename, lock Acquire/Release. - Run the membership heartbeat: grant a lease, create membership key, maintain keepalive stream, re-grant on stream failure. - Run the self-fencing watchdog: poll membership health, trigger fence after 2×TTL margin. - Accept Unix socket connections from the C daemon, dispatch FUSE ops to the metadata store, return binary responses. - Handle each connection in its own goroutine; one C daemon opens one connection per FUSE worker thread.

C daemon (etcfuse): - Open the FUSE session: fuse_session_new, register the low-level op table, mount the filesystem, enter fuse_session_loop_mt. - Open the block device: etcfs_block_open, query geometry (BLKSSZGET, BLKGETSIZE64), store the O_DIRECT fd. - Connect to the Go daemon's Unix socket. - Make synchronous IPC calls for each FUSE operation.

Startup Sequence

1. Go Daemon Start

The Go daemon parses CLI flags for etcd endpoints, TLS certificates, node ID, cluster name, and lease TTL. It connects to etcd, creates the metadata Store, initialises the membership heartbeat, and starts the self-fencing watchdog as background goroutines. Then it listens on the Unix socket path (default /run/etcfuse/etcfuse.sock, permissions 0600) and enters the accept loop. The sockets live under /run rather than /tmp because both are removed and re-bound at startup, and only root can write that directory; the mode comes from a umask around the bind rather than a chmod after it, which would leave a window in which the socket carried the process umask.

The daemon does nothing else until the C daemon connects. It does not create filesystem state at startup — the root inode is a synthetic construct in the FUSE layer, not a stored inode in etcd.

2. C Daemon Start

The C daemon reads environment variables for the mountpoint, IPC socket path, volume ID (for the block device), and node identifier. It connects to the Go daemon's Unix socket, optionally opens the block device (read-only fallback if the device is not available), builds the FUSE argument set, creates the session, mounts, and enters the multi-threaded event loop.

3. First FUSE Request

The kernel issues INIT (negotiating FUSE protocol version) and the first LOOKUP (resolving the mount point root directory). The root LOOKUP is handled locally by the C daemon — it returns a synthetic entry with inode FUSE_ROOT_ID, mode S_IFDIR|0755, nlink=2. No IPC call is made for the root.

Subsequent LOOKUPs (path components below the root) are forwarded to the Go daemon, which resolves them via LookupDirent and GetInode in etcd.

4. Metadata is Discovered Lazily

No metadata is loaded at startup. The first access to a file or directory triggers a LOOKUP, which reads the inode from etcd. A file's extents are read on its first read or write (GetExtents), and then kept with the file's lock. The kernel's entry and attribute caches, and the per-lock cache in the Go daemon, fill as paths are used.

Membership Heartbeat

The membership heartbeat is a goroutine (Membership.Run) that maintains the node's lease-backed membership key in etcd:

1. Grant an etcd lease with TTL `--lease-ttl` (default 10 s).
2. Put(membership:<nodeID>, {node_id, cluster, joined_at}) with lease.
3. Start keepalive stream (client.KeepAlive).
4. Loop:
     receive keepalive response → update lastAlive timestamp
     if channel closed:
       grant new lease → re-put membership key → restart keepalive
       if grant fails: set alive=false, return
5. On a clean shutdown: the membership key is deleted in the departure
   transaction (see Graceful Shutdown) → set alive=false

The heartbeat runs independently of request handling. If etcd becomes unreachable, renewals stop, and once the lease has gone unrenewed for twice its TTL, the self-fencing watchdog stops the daemon.

FUSE Session Lifecycle

The C daemon creates a FUSE session with the low-level API (fuse_lowlevel.h) and the multi-threaded event loop (fuse_session_loop_mt). The session is configured with:

• Event loop: fuse_session_loop_mt — multi-threaded, with clone_fd so each worker reads the kernel queue on its own /dev/fuse descriptor.
• Cache timeouts: set per reply by the Go daemon from --entry-timeout and --attr-timeout (60 s by default). Missing names are answered with negative entries, which the kernel keeps for the entry timeout. Watches invalidate all of them when another node changes something (see FUSE Cache Management).

The event loop blocks until the session is unmounted. Each kernel upcall calls the registered handler (e.g., ec_lookup). The handler does synchronous IPC with the Go backend (blocks on socket read/write) over the connection belonging to its own thread, and calls fuse_reply_* on that same thread.

When the session is unmounted (via fusermount -u or process exit), libfuse wakes the event loop with an error, and the daemon performs cleanup: session destroy, socket close.

IPC Request-Response Loop

The Go daemon's IPC handler is a goroutine per connection. It reads binary frames from a net.Conn (the Unix socket) in a loop:

loop:
  read header: [u16:be opcode][u32:be payload_len]
  read payload: payload_len bytes
  dispatch(opcode, payload) → response
  write response: [u32:be response_len] [response bytes]

The dispatch function switches on the opcode (1 = LOOKUP, 2 = GETATTR, 5 = CREATE, 6 = MKDIR, 7 = UNLINK, 8 = RMDIR, 9 = RENAME, 23 = WRITE, 12 = SETATTR, 10 = SYMLINK, 11 = LINK, 25 = MKNOD, etc.) and calls the appropriate handler.

Each handler: 1. Decodes the payload using the readU64/readU32 helper functions. 2. Calls the metadata store method (e.g., store.LookupDirent, store.AtomicCreateFile). 3. Builds the response in a buf (a byte-slice builder with w32/w64/wAttr methods). 4. Returns the response buffer.

The response is written back to the Unix socket, the C daemon's recv_full() completes, and the handler decodes the response and calls fuse_reply_* directly.

Crash Recovery Protocol

The full crash recovery sequence after an unclean shutdown:

1. etcd Side (no recovery needed)

Metadata committed to etcd before the crash is durable. The etcd cluster's Raft log guarantees that any committed transaction survives the crash. No action is required at the metadata layer.

2. Membership Re-registration

On restart, the Go daemon creates a fresh membership key. The previous key expired (lease TTL passed during the downtime). The fencing controller may have bumped the node's generation; if so, the node's generation guard will be updated accordingly when it reads the current generation from etcd.

3. New FUSE Session

The C daemon creates a fresh FUSE session. The kernel assigns a new mount point. Any files or file handles from the previous session are invalid (EIO). Applications must re-open files.

4. Inode Cache Empty

The Go daemon has no in-memory inode cache at startup (the cache was lost in the process death). The first LOOKUP for each path fetches from etcd. The cache warms up over time as the OS workload accesses files.

5. Arena Reconstruction

If a block device is configured, each arena this node owns has its bitmap rebuilt from the live extents in etcd. Blocks reserved by a write that never committed are free again afterwards, which is what makes an unclean shutdown cost nothing but the bytes already on the device.

6. File System Check

No automatic fsck is run on startup. The scrubber runs continuously in the background and detects any inconsistencies that may have been introduced by the crash. The fsck tool can be run manually for a comprehensive offline check.

State Reconstruction

After a crash, the following state is reconstructed from durable storage:

State	Source	Recovery
Inode records	etcd (inode:<ino>)	Read on demand via GetInode
Directory entries	etcd (dirent:<parent>/<name>)	Read on demand via LookupDirent
Extent maps	etcd (extent:<ino>/<chunk>)	Read on demand via GetExtents
Lock state	None (lost)	Locks are released on crash — lease TTL expires
Membership	etcd (membership:<node_id>)	Re-created at startup
Fencing generation	etcd (gen:<node_id>)	Read on demand via GetGeneration
Arena ownership	etcd (arena:<node_id>/<arena_id>)	Read at startup by the allocator
Inode number counter	etcd (inode_alloc_counter)	Advanced by a block on the first allocation after a restart (CAS); numbers left in the previous block are stranded

The key insight is that the etcd cluster holds all durable metadata except the block device content. Crash recovery is primarily about reconnecting to etcd and re-reading the current state, not about replaying a journal or repairing a filesystem.

Write Path Walkthrough

A full write operation from application to disk:

Application:
  fd = open("/mnt/fs/data.txt", O_WRONLY|O_CREAT)
  write(fd, buf, 4096)

Kernel VFS:
  LOOKUP "data.txt" in root → FUSE handler (ec_lookup)
    → IPC: op LOOKUP, parent=1, name="data.txt"
    → Go: LookupDirent(1, "data.txt") → ENOENT
    → C: fuse_reply_err(req, ENOENT)
    → kernel negative dentry (instant, may be cached briefly)

  CREATE "data.txt" with O_WRONLY|O_CREAT → FUSE handler (ec_create)
    → IPC: op CREATE, parent=1, name="data.txt", mode=0644, flags=O_WRONLY
    → Go: allocInode() → next number of this node's reserved block → ino=42
           (an exhausted block CASes inode_alloc_counter forward by 1024)
    → Go: AtomicCreateFile(1, "data.txt", 42, 0644, uid, gid)
           → Txn: check dirent doesn't exist, check inode 42 doesn't exist
                  → put dirent:1/data.txt, put inode:42
    → Go: return entry response with ino=42, mode=0644, size=0
    → C: fuse_reply_entry(req, &entry_param)
    → kernel: dentry created, file opened, returns fd=3

Data Write:
  Application: write(fd, buf, 4096)
  → FUSE handler (ec_write)
    → IPC: op WRITE, ino=42, offset=0, data_len=4096, [data bytes]
    → Go: GetInode(42) → current size=0
    → metadata update: size=4096, Put(inode:42, update_size)
    → data path: reserve arena blocks, pwrite+fsync, etcd commit, reclaim what the write buried
    → Go: return write response with written=4096
    → C: fuse_reply_write(req, 4096)
    → kernel: write acknowledged

Close:
  close(fd)
  → FUSE FLUSH → no-op
  → FUSE RELEASE → no-op

Interaction with the Self-Fencing Watchdog

The watchdog and the FUSE daemon interact through the IsFenced flag on the Go IPC service:

IPC Service:
  handleWrite(ctx, payload):
    if svc.IsFenced():
      return EIO (read-only)
    ... process write ...

  handleCreate(ctx, payload):
    if svc.IsFenced():
      return EROFS
    ... allocate and create ...

When the self-fencing watchdog fires, the Go daemon drops its cached locks, stops its IPC server, releases its arenas and exits with code 77. Its socket connections close. Each of the C daemon's later requests fails, and the handler answers the kernel with EIO. The FUSE session stays mounted, so applications get EIO on every operation until the daemons are restarted (normally by systemd).

Graceful Shutdown

When the Go daemon receives SIGINT or SIGTERM:

1. The IPC server stops accepting and serving requests.
2. The daemon publishes every buffered write and releases every cached lock, then closes its lock session.
3. It releases its arenas to the free pool.
4. If every arena was released, it writes departed:<node_id> and deletes its membership key in one transaction, so the other nodes do not fence it.
5. The self-fencing watchdog stops without fencing, and the daemon exits.

See Elastic Join and Leave § Graceful Leave.

The C daemon must be stopped separately (via SIGINT, SIGTERM, or fusermount -u). When the FUSE session is unmounted:

1. fuse_session_loop_mt returns with an error.
2. The daemon calls fuse_session_unmount and fuse_session_destroy.
3. The sockets are closed — each worker's connection is released when its thread exits.
4. The block device FD is closed.
5. The daemon exits.

If the C daemon exits first, the Go daemon's IPC accept loop detects the socket closure (EOF on the next read) and closes the connection. It continues running — it is an independent process and can accept a new connection from a restarted C daemon without restarting.