etcd

Set up a High Availability etcd Cluster with Overlord

Etcd is a fast and distributed key-value database very useful for synchronizing and discovering services even if they are not part of the same location.

Most of the applications that are considered distributed have an integration with etcd and Overlord is not an exception. So in this article we are going to create an etcd cluster with a single file and three different servers. Of course you can use more or less nodes but keep in mind that the raft algorithm requires an odd number of nodes in the cluster. An even number of nodes can be used but it will tolerate the same number of failures as the odd counterpart.

1. Create a deployment file.

   etcd-cluster.yml:

   kind: directorProject
   datacenters:
     main:
       entrypoint: 'http://127.0.0.1:8888'
       access_token: '<access token>'
   deployIn:
     labels:
       - desktop
       - c2c
       - centralita
   projectName: etcd
   projectFile: |
     options:
       - alias:
       - ip4_inherit:
     services:
       etcd:
         makejail: 'gh+AppJail-makejails/etcd'
         arguments:
           - etcd_tag: '14.2-34'
         volumes:
           - data: etcd-data
         start-environment:
           - ETCD_NAME: !ENV '${NAME}'
           - ETCD_ADVERTISE_CLIENT_URLS: !ENV 'http://${HOSTIP}:2379'
           - ETCD_LISTEN_CLIENT_URLS: !ENV 'http://${HOSTIP}:2379'
           - ETCD_LISTEN_PEER_URLS: !ENV 'http://${HOSTIP}:2380'
           - ETCD_INITIAL_ADVERTISE_PEER_URLS: !ENV 'http://${HOSTIP}:2380'
           - ETCD_INITIAL_CLUSTER_TOKEN: 'etcd-cluster-1'
           - ETCD_INITIAL_CLUSTER: !ENV '${CLUSTER}'
           - ETCD_INITIAL_CLUSTER_STATE: 'new'
           - ETCD_HEARTBEAT_INTERVAL: '200'
           - ETCD_ELECTION_TIMEOUT: '2000'
           - ETCD_LOG_LEVEL: 'error'
     default_volume_type: '<volumefs>'
     volumes:
       data:
         device: /var/appjail-volumes/etcd/data
   environment:
     CLUSTER: 'etcd0=http://<host-1>:2380,etcd1=http://<host-2>:2380,etcd2=http://<host-3>:2380'
   labelsEnvironment:
     desktop:
       NAME: 'etcd0'
       HOSTIP: '<host-1>'
     c2c:
       NAME: 'etcd1'
       HOSTIP: '<host-2>'
     centralita:
       NAME: 'etcd2'
       HOSTIP: '<host-3>'

2. Deploy.

   overlord apply -f etcd-cluster.yml

3. Check status.

   $ overlord get-info -f etcd-cluster.yml -t projects --filter-per-project
   datacenter: http://127.0.0.1:8888
     entrypoint: main
     chain: None
     labels:
       - all
       - desktop
       - vm-only
     projects:
       etcd:
         state: DONE
         last_log: 2025-03-15_14h37m12s
         locked: False
         services:
           - {'name': 'etcd', 'status': 0, 'jail': 'd7db966df9'}
         up:
           operation: COMPLETED
           output:
            rc: 0
            stdout: {'errlevel': 0, 'message': None, 'failed': []}
           last_update: 34.01 seconds
           job_id: 4
           labels:
            error: False
            message: None
            load-balancer:
              services:
                etcd:
                  error: False
                  message: None
            skydns:
              services:
                etcd:
                  error: False
                  message: None
   datacenter: http://127.0.0.1:8888
     entrypoint: main
     chain: beta
     labels:
       - all
       - centralita
     projects:
       etcd:
         state: DONE
         last_log: 2025-03-15_18h31m04s
         locked: False
         services:
           - {'name': 'etcd', 'status': 0, 'jail': '8e2cdcb3eb'}
         up:
           operation: COMPLETED
           output:
            rc: 0
            stdout: {'errlevel': 0, 'message': None, 'failed': []}
           last_update: 32.38 seconds
           job_id: 1
           labels:
            error: False
            message: None
            load-balancer:
              services:
                etcd:
                  error: False
                  message: None
            skydns:
              services:
                etcd:
                  error: False
                  message: None
   datacenter: http://127.0.0.1:8888
     entrypoint: main
     chain: charlie
     labels:
       - all
       - c2c
     projects:
       etcd:
         state: DONE
         last_log: 2025-03-15_14h37m45s
         locked: False
         services:
           - {'name': 'etcd', 'status': 0, 'jail': 'f98f0da732'}
         up:
           operation: COMPLETED
           output:
            rc: 0
            stdout: {'errlevel': 0, 'message': None, 'failed': []}
           last_update: 16.27 seconds
           job_id: 1
           labels:
            error: False
            message: None
            load-balancer:
              services:
                etcd:
                  error: False
                  message: None
            skydns:
              services:
                etcd:
                  error: False
                  message: None

4. Configure Overlord.

   /usr/local/etc/overlord.yml:

   etcd:
    <host-1>: {}
    <host-2>: {}
    <host-3>: {}

5. Restart Overlord services.

   supervisorctl restart overlord:

As mentioned, there are many distributed projects that can use etcd. In later articles we will use this etcd cluster to demonstrate how Overlord can integrate with other services.