# Module: ETCD > Pigsty has built-in etcd support, which is a reliable distributive consensus storage (DCS), empowering PostgreSQL HA. --- > ETCD is a distributed, reliable key-value store for the most critical data of a distributed system [Configuration](#configuration) | [Administration](#administration) | [Playbook](#playbook) | [Dashboard](#dashboard) | [Parameter](#parameter) Pigsty use [**etcd**](https://etcd.io/) as [**DCS**](https://patroni.readthedocs.io/en/latest/dcs_failsafe_mode.html): Distributed configuration storage (or distributed consensus service). Which is critical to PostgreSQL High-Availability & Auto-Failover. You have to install [`ETCD`](/docs/etcd) module before any [`PGSQL`](/docs/pgsql) modules, since patroni & vip-manager will rely on etcd to work. Unless you are using an external etcd cluster. You don't need [`NODE`](/docs/node) module to install [`ETCD`](/docs/etcd), but it requires a valid `CA` on your local `files/pki/ca`. Check [ETCD Administration SOP](/docs/etcd#administration) for more details. ---------------- ## Configuration You have to define an etcd cluster before deploying it. There some [parameters](#parameter) about etcd. It is recommending to have at least 3 instances for a serious production environment. ### Single Node Define a group `etcd` in the inventory, It will create a singleton etcd instance. ```yaml # etcd cluster for ha postgres etcd: { hosts: { 10.10.10.10: { etcd_seq: 1 } }, vars: { etcd_cluster: etcd } } ``` This is good enough for development, testing & demonstration, but not recommended in serious production environment. ### Three Nodes You can define etcd cluster with multiple nodes. ```yaml etcd: # dcs service for postgres/patroni ha consensus hosts: # 1 node for testing, 3 or 5 for production 10.10.10.10: { etcd_seq: 1 } # etcd_seq required 10.10.10.11: { etcd_seq: 2 } # assign from 1 ~ n 10.10.10.12: { etcd_seq: 3 } # odd number please vars: # cluster level parameter override roles/etcd etcd_cluster: etcd # mark etcd cluster name etcd etcd_safeguard: false # safeguard against purging etcd_clean: true # purge etcd during init process ``` You can use more nodes for production environment, but 3 or 5 nodes are recommended. Remember to use odd number for cluster size. ---------------- ## Administration Here are some useful administration tasks for etcd: - [Create Cluster](#create-cluster) - [Destroy Cluster](#destroy-cluster) - [CLI Environment](#cli-environment) - [Reload Config](#reload-config) - [Add Member](#add-member) - [Remove Member](#remove-member) ---------------- ### Create Cluster If [`etcd_safeguard`](/docs/reference/param/#etcd_safeguard) is `true`, or [`etcd_clean`](/docs/reference/param/#etcd_clean) is `false`, the playbook will abort if any running etcd instance exists to prevent purge etcd by accident. ```yaml etcd: hosts: 10.10.10.10: { etcd_seq: 1 } 10.10.10.11: { etcd_seq: 2 } 10.10.10.12: { etcd_seq: 3 } vars: { etcd_cluster: etcd } ``` ```bash ./etcd.yml # init etcd module on group 'etcd' ``` ---------------- ### Destroy Cluster To destroy an etcd cluster, just use the `etcd_clean` subtask of `etcd.yml`, do think before you type. ```bash ./etcd.yml -t etcd_clean # remove entire cluster, honor the etcd_safeguard ./etcd.yml -t etcd_purge # purge with brutal force, omit the etcd_safeguard ``` ---------------- ### CLI Environment Here's an example of client environment config. Pigsty use etcd v3 API by default. ```bash alias e="etcdctl" alias em="etcdctl member" export ETCDCTL_API=3 export ETCDCTL_ENDPOINTS=https://10.10.10.10:2379 export ETCDCTL_CACERT=/etc/pki/ca.crt export ETCDCTL_CERT=/etc/etcd/server.crt export ETCDCTL_KEY=/etc/etcd/server.key ``` **CRUD** You can do CRUD with following commands. ```bash e put a 10 ; e get a; e del a ; # V3 API ``` ---------------- ### Reload Config If etcd cluster membership changes, we need to refresh etcd endpoints references: * config file of existing etcd members * etcdctl client environment variables * patroni dcs endpoint config * vip-manager dcs endpoint config To refresh etcd config file `/etc/etcd/etcd.conf` on existing members: ```bash ./etcd.yml -t etcd_conf # refresh /etc/etcd/etcd.conf with latest status ansible etcd -f 1 -b -a 'systemctl restart etcd' # optional: restart etcd ``` To refresh `etcdctl` client environment variables ```bash $ ./etcd.yml -t etcd_env # refresh /etc/profile.d/etcdctl.sh ``` To update etcd endpoints reference on `patroni`: ```bash ./pgsql.yml -t pg_conf # regenerate patroni config ansible all -f 1 -b -a 'systemctl reload patroni' # reload patroni config ``` To update etcd endpoints reference on `vip-manager`, (optional, if you are using a L2 vip) ```bash ./pgsql.yml -t pg_vip_config # regenerate vip-manager config ansible all -f 1 -b -a 'systemctl restart vip-manager' # restart vip-manager to use new config ``` ---------------- ### Add Member ETCD Reference: [Add a member](https://etcd.io/docs/v3.5/op-guide/runtime-configuration/#add-a-new-member) You can add new members to existing etcd cluster in 5 steps: 1. issue `etcdctl member add` command to tell existing cluster that a new member is coming (use learner mode) 2. update inventory group `etcd` with new instance 3. init the new member with `etcd_init=existing`, to join the existing cluster rather than create a new one (**VERY IMPORTANT**) 4. promote the new member from leaner to follower 5. update etcd endpoints reference with [reload-config](#reload-config) **Short Version** ```bash etcdctl member add --learner=true --peer-urls=https://:2380 ./etcd.yml -l -e etcd_init=existing etcdctl member promote ```
Detail: Add member to etcd cluster Here's the detail, let's start from one single etcd instance. ```yaml etcd: hosts: 10.10.10.10: { etcd_seq: 1 } # <--- this is the existing instance 10.10.10.11: { etcd_seq: 2 } # <--- add this new member definition to inventory vars: { etcd_cluster: etcd } ``` Add a learner instance `etcd-2` to cluster with `etcd member add`: ```bash # tell the existing cluster that a new member etcd-2 is coming $ etcdctl member add etcd-2 --learner=true --peer-urls=https://10.10.10.11:2380 Member 33631ba6ced84cf8 added to cluster 6646fbcf5debc68f ETCD_NAME="etcd-2" ETCD_INITIAL_CLUSTER="etcd-2=https://10.10.10.11:2380,etcd-1=https://10.10.10.10:2380" ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.10.10.11:2380" ETCD_INITIAL_CLUSTER_STATE="existing" ``` Check the member list with `etcdctl member list` (or `em list`), we can see an `unstarted` member: ``` 33631ba6ced84cf8, unstarted, , https://10.10.10.11:2380, , true 429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false ``` Init the new etcd instance `etcd-2` with `etcd.yml` playbook, we can see the new member is started: ```bash $ ./etcd.yml -l 10.10.10.11 -e etcd_init=existing # etcd_init=existing must be set ... 33631ba6ced84cf8, started, etcd-2, https://10.10.10.11:2380, https://10.10.10.11:2379, true 429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false ``` Promote the new member, from leaner to follower: ```bash $ etcdctl member promote 33631ba6ced84cf8 # promote the new learner Member 33631ba6ced84cf8 promoted in cluster 6646fbcf5debc68f $ em list # check again, the new member is started 33631ba6ced84cf8, started, etcd-2, https://10.10.10.11:2380, https://10.10.10.11:2379, false 429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, fals ``` The new member is added, don't forget to [reload config](#reload-config). Repeat the steps above to add more members. remember to use at least 3 members for production.
---------------- ### Remove Member To remove a member from existing etcd cluster, it usually takes 3 steps: 1. remove/uncomment it from inventory and [reload config](#reload-config) 2. remove it with `etcdctl member remove ` command and kick it out of the cluster 3. temporarily add it back to inventory and purge that instance, then remove it from inventory permanently
Detail: Remove member from etcd cluster Here's the detail, let's start from a 3 instance etcd cluster: ```yaml etcd: hosts: 10.10.10.10: { etcd_seq: 1 } 10.10.10.11: { etcd_seq: 2 } 10.10.10.12: { etcd_seq: 3 } # <---- comment this line, then reload-config vars: { etcd_cluster: etcd } ``` Then, you'll have to actually kick it from cluster with `etcdctl member remove` command: ```bash $ etcdctl member list 429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false 33631ba6ced84cf8, started, etcd-2, https://10.10.10.11:2380, https://10.10.10.11:2379, false 93fcf23b220473fb, started, etcd-3, https://10.10.10.12:2380, https://10.10.10.12:2379, false # <--- remove this $ etcdctl member remove 93fcf23b220473fb # kick it from cluster Member 93fcf23b220473fb removed from cluster 6646fbcf5debc68f ``` Finally, you have to shutdown the instance, and purge it from node, you have to uncomment the member in inventory temporarily, then purge it with `etcd.yml` playbook: ```bash ./etcd.yml -t etcd_purge -l 10.10.10.12 # purge it (the member is in inventory again) ``` After that, remove the member from inventory permanently, all clear!
---------------- ## Playbook There's a built-in playbook: `etcd.yml` for installing etcd cluster. But you have to [define](#configuration) it first. ```bash ./etcd.yml # install etcd cluster on group 'etcd' ``` Here are available sub tasks: - `etcd_assert` : generate etcd identity - `etcd_install` : install etcd rpm packages - `etcd_clean` : cleanup existing etcd - `etcd_check` : check etcd instance is running - `etcd_purge` : remove running etcd instance & data - `etcd_dir` : create etcd data & conf dir - `etcd_config` : generate etcd config - `etcd_conf` : generate etcd main config - `etcd_cert` : generate etcd ssl cert - `etcd_launch` : launch etcd service - `etcd_register` : register etcd to prometheus If [`etcd_safeguard`](/docs/reference/param/#etcd_safeguard) is `true`, or [`etcd_clean`](/docs/reference/param/#etcd_clean) is `false`, the playbook will abort if any running etcd instance exists to prevent purge etcd by accident. [![asciicast](https://asciinema.org/a/566414.svg)](https://asciinema.org/a/566414) ---------------- ## Dashboard There is one dashboard for ETCD module: [ETCD Overview](https://demo.pigsty.cc/d/etcd-overview): Overview of the ETCD cluster
ETCD Overview Dashboard [![etcd-overview.jpg](/img/dashboard/etcd-overview.jpg)](https://demo.pigsty.cc/d/etcd-overview)
---------------- ## Parameter There are 10 parameters about [`ETCD`](/docs/reference/param/#etcd) module. | Parameter | Type | Level | Comment | |------------------------------------------------------------|:------:|:-----:|----------------------------------------------| | [`etcd_seq`](/docs/reference/param/#etcd_seq) | int | I | etcd instance identifier, REQUIRED | | [`etcd_cluster`](/docs/reference/param/#etcd_cluster) | string | C | etcd cluster & group name, etcd by default | | [`etcd_safeguard`](/docs/reference/param/#etcd_safeguard) | bool | G/C/A | prevent purging running etcd instance? | | [`etcd_clean`](/docs/reference/param/#etcd_clean) | bool | G/C/A | purging existing etcd during initialization? | | [`etcd_data`](/docs/reference/param/#etcd_data) | path | C | etcd data directory, /data/etcd by default | | [`etcd_port`](/docs/reference/param/#etcd_port) | port | C | etcd client port, 2379 by default | | [`etcd_peer_port`](/docs/reference/param/#etcd_peer_port) | port | C | etcd peer port, 2380 by default | | [`etcd_init`](/docs/reference/param/#etcd_init) | enum | C | etcd initial cluster state, new or existing | | [`etcd_election_timeout`](/docs/reference/param/#etcd_election_timeout) | int | C | etcd election timeout, 1000ms by default | | [`etcd_heartbeat_interval`](/docs/reference/param/#etcd_heartbeat_interval) | int | C | etcd heartbeat interval, 100ms by default | --- Section pages: - [Metrics](/docs/etcd/metric/): Pigsty ETCD module metric list - [FAQ](/docs/etcd/faq/): Pigsty ETCD dcs module frequently asked questions