Skip to content

Autoscaling (ASG)

The terraform-asg/ root module in etcfs-terraform-modules (module terraform/modules/etcfs-asg) runs EtcFS nodes in an EC2 Auto Scaling Group, instead of the fixed node_count of Infrastructure (Terraform). Use it when the cluster should grow and shrink by itself rather than through scripts/infra/add-compute-node.sh.

No operator logs in to a node at any point. The ASG launches an instance, the instance configures itself and joins the cluster, and when an instance is terminated a Lambda function removes it from etcd first.

Differences from the fixed-node module

  • Installation from the release. Each node installs etcfuse, etcfuse-meta and etcfsctl from the GitHub release RPMs (see Binaries) in its launch-template user-data script. Nothing connects to the instance over SSH.
  • Self-attachment. Each node attaches the shared io2 Multi-Attach volume to itself (aws ec2 attach-volume with its own instance ID). Multi-Attach allows every node to do this at the same time.
  • One availability zone. A Multi-Attach volume can only be attached within one availability zone, so the ASG is pinned to a single subnet in var.az.

Scale-out: how a new node joins

The whole join procedure runs on the new instance, from its user-data. The user-data has two parts:

  1. a short preamble generated by Terraform that exports the cluster's settings as ETCFS_* environment variables;
  2. terraform/modules/etcfs-asg/scripts/node-bootstrap.sh, included unchanged.

The script is plain bash rather than a Terraform template. That way CI can run shellcheck and bash -n on it, it can be run by hand against a test instance, and the CloudFormation template can reuse it. The cost is that a missing setting is only noticed when the node boots, not at terraform plan. The four settings with no safe default are therefore required, and the script stops if any of them is unset.

The script does the following:

  1. Identify itself. It reads its instance ID and private IP from IMDSv2.
  2. Install. It installs etcd and the three EtcFS packages.
  3. Attach the volume to itself.
  4. Find or elect the seed. The seed is the node that forms the etcd cluster when none exists yet. The election uses one DynamoDB item per cluster (table <cluster_name>-etcfs-seed):
  5. The first node to create the item with a conditional PutItem wins. DynamoDB conditional writes are atomic, so exactly one node wins even when several boot at once.
  6. If the recorded seed has not become reachable within 600 seconds, another node may take over the item. The takeover is a conditional write against the exact item it read, so when several nodes try at once, only one succeeds.

Two simpler mechanisms fail under a concurrent boot: - Choosing a running peer by EC2 tag. An instance is "running" as soon as it boots, long before its etcd listens on port 2379. Nodes booting together all pick a peer with nothing listening yet. - SSM Parameter Store. Creating a parameter without --overwrite is atomic, but taking over a dead seed's parameter can only be done with an unconditional --overwrite. Two nodes taking over at once each form their own single-node cluster.

The DynamoDB item is only a hint about where to join, not a record of who is alive. Nothing updates it after the cluster forms, so in a cluster that has run for a while it usually names a seed that has since been replaced, with a timestamp far past 600 seconds. Taking over on age alone would then start a second etcd cluster next to a healthy one, both using the same block device. Before any takeover, the joining node therefore finds the other nodes by tag and checks each one's :2379/health: - if any peer answers, the cluster is alive; the node joins through that peer and repoints the item at it (a conditional write, which may lose to another node doing the same, and that is fine); - only if the recorded seed is unreachable, the item is older than 600 seconds, and no peer answers does the node take over the item. 5. If it is the seed, it starts a new etcd cluster (--initial-cluster-state new) with itself as the only member. 6. Otherwise it joins the existing cluster: 1. It removes every etcd member whose peer IP belongs to no running or pending instance of the cluster. A node that crashed, rather than leaving through the scale-in path below, is still registered as an etcd member. Adding a new member before removing the dead one raises the quorum (for example to 3 of 4) before the new member is running, and the configuration change can then stall. Removing a member that is already gone does nothing, so every joining node can safely run this step. 2. It runs etcdctl member add, reads the ETCD_INITIAL_CLUSTER string it returns, and starts etcd with --initial-cluster-state existing. If the member was added but the response was lost, it recovers the same way as add-compute-node.sh. 7. Start the daemons. It starts etcd, then etcfuse-meta: - The script runs etcfuse-meta --help to check whether the installed binary supports --config. A binary without it does not reject a config file; it never reads one. Writing settings to a file it ignores would start a daemon that falls back to localhost:2379, opens no block device, and fences with a single signal: the mount appears, and every write returns EIO. - With --config support, the settings every node shares (endpoints, cluster name, lease TTL, device, fencing volume) go in /etc/etcfs/etcfuse-meta.yaml, and the package's ExecStart is left unchanged so an upgrade cannot drop them. Only this node's identity (--node-id, --ec2-instance-id) goes in a systemd drop-in. Command-line flags override the file, so any shared setting also left on the command line would silently win over the file. - Without --config support, every setting is passed as a flag in the drop-in. 8. Wait for the mount. If the FUSE mount does not appear, the script prints the last 60 lines of both daemons' journals and exits non-zero, so the instance fails its EC2 status check instead of running unmounted.

There is no lifecycle hook on launch: an instance that is still configuring itself simply is not healthy yet under the default EC2 health check.

This path was tested on AWS by hard-killing two nodes (a crash, not a graceful termination) while the survivors kept reading and writing, then scaling out from 3 to 5 nodes and back in to 3. It ended with a clean etcd membership and identical file checksums on every node.

Scale-in: removing the etcd member

When the ASG terminates an instance, EtcFS treats it as a crash: self-fencing and the generation guard protect the filesystem straight away. etcd's own membership is separate, though. Without help, the dead node's etcd member stays registered until the next node joins and runs the cleanup above. An ASG that keeps scaling in without scaling out would accumulate dead members and eventually lose quorum.

The module therefore attaches an autoscaling:EC2_INSTANCE_TERMINATING lifecycle hook, which an EventBridge rule on "EC2 Instance-terminate Lifecycle Action" routes to a Lambda function (terraform/modules/etcfs-asg/lambda/graceful_leave.py):

  1. The Lambda looks up the terminating instance's ClusterName tag and private IP.
  2. It finds one surviving node, by the same tag search as the join path. The terminating instance may already be unreachable.
  3. It runs etcd member remove on that survivor through SSM Run Command (AWS-RunShellScript). The same script finds the member's hex ID in member list by the dying node's peer URL, so the ID cannot go stale between two calls.
  4. It always calls autoscaling:CompleteLifecycleAction with CONTINUE, even if the removal failed. A lifecycle action that never completes blocks the termination, which is worse than a leftover etcd member.

The hook's default_result = "CONTINUE" covers the case where the Lambda does not run at all, for example if it is throttled or runs past the 180 s heartbeat: the ASG completes the termination instead of waiting forever.

IAM

Two roles are involved:

  • Node role (etcfs-nodes, created by scripts/infra/fencing-iam.sh create). It holds the fencing permissions (ec2:DetachVolume, ec2:AttachVolume, Describe*) and the managed policy AmazonSSMManagedInstanceCore. That policy registers the node with SSM, so a surviving node can receive the Lambda's Run Command. Without it, scale-in still works but skips the member remove.
  • Lambda role (<cluster_name>-graceful-leave, created by the etcfs-asg module). It allows only what the Lambda calls: autoscaling:CompleteLifecycleAction, ec2:DescribeInstances, ssm:SendCommand, ssm:GetCommandInvocation and writing to CloudWatch Logs.

Run fencing-iam.sh create before the first terraform apply. It is safe to run again.

Usage

./scripts/infra/fencing-iam.sh create   # once per account

terraform -chdir=../etcfs-terraform-modules/terraform-asg init
terraform -chdir=../etcfs-terraform-modules/terraform-asg apply

Check the cluster once the nodes are up:

INSTANCE_IP=$(aws ec2 describe-instances \
  --filters Name=tag:ClusterName,Values=etcfuse Name=tag:Role,Values=etcfs-node Name=instance-state-name,Values=running \
  --query 'Reservations[0].Instances[0].PublicIpAddress' --output text)

etcfsctl --etcd-endpoints=http://$INSTANCE_IP:2379 status

Scale by changing the desired capacity:

aws autoscaling set-desired-capacity --auto-scaling-group-name etcfuse-asg --desired-capacity 4

Teardown:

terraform -chdir=../etcfs-terraform-modules/terraform-asg destroy

CloudFormation

etcfs-cloudformation-template builds the same cluster from a CloudFormation stack. Its user-data exports the stack's parameters as ETCFS_* variables, downloads node-bootstrap.sh from this module and runs it. The seed election, member add and stale-member cleanup therefore have one implementation for both paths. The template does not include the graceful-leave Lambda, because its source is too long to inline; that repository's README.md describes the consequence.

Limitations

  • No cross-AZ resilience. The cluster lives in one availability zone, so an AZ outage takes it down whatever the ASG size.
  • No scaling policies. The module exposes min_size, max_size and desired_capacity. Scaling on CPU or queue depth with an aws_autoscaling_policy is left to the deployer; EtcFS has no metric that should drive it.
  • A crashed member can stay registered. Dead etcd members are removed when the next node joins. A cluster that crashes down to min_size and never scales out again keeps the dead member. etcd computes quorum as (n/2)+1 over all members, including members that are down, so the cluster tolerates one fewer further failure than its live member count suggests (see the etcd FAQ, which also recommends removing a failed member before adding its replacement). Remove it with etcdctl member remove if the cluster will stay that size.