Metadata Schema¶

The EtcFS etcd key schema and the data types that power every namespace and structural operation.

Table of Contents¶

Design
Key Layout
Data Types
Key Helpers

Design¶

Every mutable structural datum — inode metadata, directory entries, file locks, arena ownership, membership records, and fencing generation counters — lives in etcd as a discrete key-value pair. The shared block device carries only raw byte extents of file content. There is no on-disk filesystem format beyond those extents.

Keys are short, fixed-prefix strings with no structural hierarchy beyond the / convention in directory entries. Etcd stores all keys in flat lexicographic order; the separators are a naming convention that enables efficient prefix-range scans for directory listings.

All values are encoded as binary blobs. Integer values use big-endian byte order; structured records use a compact fixed-length binary format. The maximum value size is constrained to well under etcd's 1.5 MiB request limit. Extent maps — the largest per-file data — are stored in separate chunked keys per inode to avoid ever approaching that ceiling.

Key Layout¶

Key pattern	Value	Purpose
`inode:<ino>`	`InodeRecord` (84 bytes)	File or directory metadata
`dirent:<parent>/<name>`	`<ino>` (8 bytes)	Directory entry resolving a name to an inode
`xattr:<ino>/<name>`	attribute value (opaque bytes)	One extended attribute of an inode
`quota:<ino>`	`QuotaRecord` (JSON)	Byte and inode limits on a directory that is a quota root
`lock:<ino>/<mode>/<holder>`	holder's node ID	One holder of an inode's lock, written under that node's session lease
`lock_want:<ino>/<node_id>`	requesting node's ID	A peer asking the holder of a cached lock to yield it; outside `lock:` so it cannot block the acquisition it exists to unblock
`arena:<node_id>/<arena_id>`	`<arena_id>` (8 bytes)	One arena a node currently owns
`arena_alloc_log`	counter (8 bytes)	Global arena-ID allocation counter
`membership:<node_id>`	membership metadata	Lease-backed liveness key for cluster membership
`gen:<node_id>`	generation counter	Fencing epoch counter, bumped on confirmed fence
`departed:<node_id>`	RFC3339 timestamp	Written atomically with the membership key's deletion; marks a departure as intentional so peers skip fencing
`inode_alloc_counter`	counter (8 bytes)	Next inode number to hand out; a node CASes it forward by a whole block and allocates from that block in memory
`extent:<ino>/<chunk>`	five comma-separated integers (ASCII)	One extent: a logical byte range of a file mapped onto the shared device

Key semantics¶

Inode keys are the canonical record of a file or directory. The key is derived solely from the inode number, with no parent information — a file can have multiple hard links pointing to the same inode key from different directory-entry keys.

Directory-entry keys encode a parent inode and a child name separated by /. A directory listing is a prefix scan over dirent:<parent>/, which etcd serves in lexicographic order. Each value is simply the target inode number as a big-endian 64-bit integer.

Extended-attribute keys deliberately mirror directory-entry keys, and for the same reason: one key per attribute makes a single attribute readable, writable and removable on its own, and makes "every attribute of this inode" a prefix scan over xattr:<ino>/. A single packed value per inode would turn every setxattr into a read-modify-write of the whole set, which is both slower and a lost-update race between two nodes setting different attributes at once.

Attributes are owned outright by the inode, so removing an inode deletes them in the same transaction, via a range delete over the prefix. That matters beyond leaked keys: inode numbers are reused, and a surviving attribute would be inherited by whatever file next took the number. An inode that still has other hard links keeps its attributes, since they belong to the inode rather than to the name being removed.

Name and value sizes are bounded at the kernel's own XATTR_NAME_MAX (255) and XATTR_SIZE_MAX (65536). The bound is enforced in the metadata store rather than only in the FUSE layer because it is etcd being protected — an attribute is a Raft-replicated value, and an unbounded one is a route into the store quota. Names containing / or NUL are rejected: the first would move the key's split point, the second would end the name early in the NUL-separated listxattr buffer, and either lets a caller address an attribute other than the one it named. The trusted.* namespace is writable only by uid 0, matching the kernel; security.* is left writable so SELinux and IMA labels work, with the contents policed by the LSM hooks that own them.

One name is an action rather than an attribute and is never stored: setting user.etcfs.publish on a file publishes the writing node's buffered data and yields the file's cached lock, handing it to whichever node reads it next. See Explicit Publish.

Quota keys mark a directory as the root of a subtree with byte and inode limits; a zero limit in either dimension means unlimited. Usage is computed by walking the namespace from each root, because an inode records no parent — building the parent index from directory entries is the only way to know which subtree a file is in, the same walk a directory rename already does to check for cycles.

That walk is why accounting is periodic rather than transactional, and the consequence is deliberate: these are soft quotas. Charging a write to its subtree as it happened would need the enclosing root known on the write path, which means either a parent pointer on every inode — a second source of truth to keep consistent — or a counter update inside the transaction that publishes the write. The write path is already bound by the number of Raft round trips it makes, so adding one more to every write is the wrong trade for a limit that is policy rather than a correctness invariant. etcfsctl quota reports usage and flags a root that is over; nothing rejects a write. A file hard-linked into two quota roots is charged to both, which is what every filesystem with subtree quotas does — the alternative, charging whichever root a walk reached first, would make the answer depend on iteration order.

Lock keys are one per holder, not one per inode. The mode is part of the key and the value is just the holder's node ID. Each key carries its holder's own etcd lease, so a holder that stops heartbeating is dropped automatically — and dropping one holder cannot disturb the others, which is what allows a shared lock to have several at once.

Arena keys own a contiguous 1 GiB range on the shared block device. Each node acquires arenas from a global free pool controlled by the arena_alloc_log counter. The key name includes the node ID to guarantee exclusive ownership.

Membership keys are lease-backed liveness records. The presence of membership:<node> signals the node is alive and participating. Expiry of the backing lease triggers the fencing controller.

Departure keys exist because that trigger cannot tell an intentional shutdown from a crash — etcd reports an explicit Revoke and a lease timeout as the same delete event. A node that is shutting down cleanly writes departed:<node> in the same transaction that removes its membership key, and its peers skip the fence. The transaction is conditioned on the membership key still existing, so a node whose lease has already expired cannot write one; it is dropped when the node registers again.

Generation keys are the fencing epoch counter. The fencing controller bumps this value via a CAS transaction after confirming a node has been successfully fenced. Every metadata mutation that modifies extents checks this generation before committing.

Reserved inode numbers¶

Inode 0 is never valid — DecodeUint64 returns 0 for a missing or malformed key, so 0 doubles as an implicit "not found" sentinel and must never be assigned to a real file. Inode 1 is FUSE_ROOT_ID, the root directory: the C daemon answers getattr/lookup for it locally, and seed-etcd writes the root directory record directly to inode:1 before any node starts. The inode allocator (metadata.FirstUsableIno) therefore starts handing out numbers at 2. Handing out 1 to a regular file overwrites the root inode record and makes every subsequent operation on the mount fail — this happened in practice (see the chaos test report) before the allocator's start value was corrected.

Data Types¶

InodeRecord¶

The fixed-length binary record stored at each inode:<ino> key, totalling 84 bytes.

Offset	Size	Field	Description
0	8	`Ino`	Inode number
8	8	`Size`	File size in bytes
16	8	`Blocks`	Number of 512-byte blocks allocated
24	4	`Mode`	File type and permissions (POSIX `st_mode`)
28	4	`Nlink`	Hard link count
32	4	`UID`	Owner user ID
36	4	`GID`	Owner group ID
40	4	`Rdev`	Device ID (for device nodes)
44	4	`Blksize`	Preferred I/O block size
48	8	`Atime`	Last access time (Unix seconds)
56	8	`Mtime`	Last modification time (Unix seconds)
64	8	`Ctime`	Last status-change time (Unix seconds)
72	4	`Atime` nanoseconds	Sub-second part of the access time
76	4	`Mtime` nanoseconds	Sub-second part of the modification time
80	4	`Ctime` nanoseconds	Sub-second part of the status-change time

The three nanosecond fields are appended rather than folded into the timestamps, so a record written before they existed still decodes: it is 72 bytes long and its sub-second parts read as the zero it stored.

The extent list is not embedded in the inode record. Extents are stored in separate keys (extent:<ino>/<chunk>) to keep the inode value small and to allow extent maps to grow beyond the 1.5 MiB etcd value limit without splitting the inode record itself.

The Nlink field tracks the number of directory entries pointing to this inode. When Nlink reaches zero, the inode is eligible for deletion. A directory is the exception: its count is its own . plus the .. of every subdirectory it holds, so it is 2 for an empty directory and rises with each subdirectory. mkdir, rmdir and a directory rename therefore adjust the parent's count in the same transaction that changes the entry, pinned to the revision it was read at.

Every inode is created with the count its first entry implies: 1 for a regular file, symlink, device node or FIFO, and 2 for a directory, which is reached both through its parent's entry and through its own .. metadata.InitialNlink is the single definition of that rule.

Directories keep 2 for their whole life. EtcFS does not model the .. link a subdirectory contributes to its parent, so a directory's count does not vary with its contents. The fsck and scrubber checks assert that fixed value for directories and compare against the real dirent count for everything else.

Extent¶

Unlike the other records, an extent's value is ASCII: five comma-separated decimal integers followed by the writer's node ID, at extent:<ino>/<chunk>.

<logical_off>,<disk_off>,<length>,<generation>,<sequence>,<node>

generation is the writer's fencing generation at commit time and node is the writer. Both are needed together: a generation is per-node, so the scrubber cannot compare a stamp against anything without knowing whose it is. The node ID is the whole remainder of the value, so one containing a comma still round-trips.

sequence orders writes to the same logical bytes. A write is never an in-place update — it allocates fresh blocks and appends an extent — so two extents can cover one range, and the higher sequence is the later write and the one a read resolves to. The chunk number in the key makes the key unique within the inode and nothing more.

Keeping recency in the value rather than the key is what allows an extent to be split. Trimming an overwritten extent down to the pieces still readable can leave two records, and both must remain exactly as old as the extent they were cut from; a second key would instead assert the piece is newer, and it would then win over a genuinely newer extent overlapping it.

All six fields are required. EtcFS is pre-deployment, so the decoder rejects earlier forms outright rather than carrying a compatibility path for records nothing has written.

LockRecord¶

Not a stored value: GetLockInfo assembles it by scanning an inode's holder keys.

Field	Type	Description
`Mode`	`string`	`"exclusive"` if any holder is a writer, otherwise `"shared"`
`Holders`	`[]string`	Node IDs currently holding the lock

When a holder's lease expires (node crash, network partition beyond TTL margin), its key is deleted automatically and it drops out of this list. The inode is unlocked once the last holder is gone.

Arena ownership record¶

Stored at arena:<node_id>/<arena_id>, one key per arena a node owns — not one key per node, since a node acquires a further arena whenever its current ones fill up. The value is the arena ID itself, 8-byte big-endian; DiskStart/DiskEnd are derived (ID * ArenaSizeBytes, (ID+1) * ArenaSizeBytes), not stored.

Each arena is 1 GiB (2^30 bytes) divided into 4 KiB blocks, yielding 262,144 blocks per arena. The allocator tracks free/allocated blocks via an in-memory bitmap local to the node, rebuilt on restart from the live extent: keys — see Arena Allocator § Crash Recovery Integration. Nothing about block-level free/allocated state is persisted; only arena ownership is.

MembershipRecord¶

Stored at membership:<node_id>, a lease-backed liveness beacon.

Field	Type	Description
`NodeID`	`string`	Node identifier
`ClusterName`	`string`	Logical cluster name
`JoinedAt`	`time`	Timestamp when the node joined
`Address`	`string`	Network address for diagnostics

The key is created with an etcd lease of configurable TTL (default 5 seconds). The daemon maintains a keepalive stream on this lease. If the stream breaks and cannot be re-established within the self-fencing margin, the node self-fences.

Key Helpers¶

Each key family has a constructor function that builds the etcd key string from its components. These ensure consistent formatting across all code paths.

InodeKey(ino) — produces "inode:<ino>"
DirentKey(parent, name) — produces "dirent:<parent>/<name>"
DirentPrefix(parent) — produces "dirent:<parent>/" for prefix scans
LockKey(ino, mode, holder) — produces "lock:<ino>/<mode>/<holder>", where the holder token is the node's session lease and a per-acquisition counter; LockPrefix(ino) and LockModePrefix(ino, mode) produce the ranges a transaction compares against
LockWantKey(ino, node) — produces "lock_want:<ino>/<node_id>", the request a blocked node writes to recall a cached lock; ParseLockWantKey reads back the inode and the node that wants it
ArenaOwnerKey(nodeID, arenaID) — produces "arena:<nodeID>/<arenaID>"
ArenaNodePrefix(nodeID) — produces "arena:<nodeID>/" for prefix scans
MembershipKey(nodeID) — produces "membership:<nodeID>"
GenKey(nodeID) — produces "gen:<nodeID>"

All 64-bit integer values stored in etcd use EncodeUint64 / DecodeUint64, which produce and consume 8-byte big-endian byte slices. This is used for inode numbers in dirent values, arena counter values, and allocator counters.