PIGSTY

Installation

Mon, 01 Jan 0001 00:00:00 +0000

Install Pigsty with 4 steps: Prepare, Download, Configure and Install.

Also check offline installation if you don’t have the Internet access.

Short Version

Prepare a fresh Linux x86_64 node that runs compatible OS, then run as a sudo-able user:

curl -L https://get.pigsty.cc/install | bash -s v2.7.0

It will download Pigsty source to your home, then perform configure and install to finish the installation.

cd ~/pigsty   # get pigsty source and entering dir
./bootstrap   # download bootstrap pkgs & ansible [optional]
./configure   # pre-check and config templating   [optional]
./install.yml # install pigsty according to pigsty.yml

A pigsty singleton node will be ready with Web Interface on port 80/443 and Postgres service on port 5432. You can add more nodes into Pigsty and deploy modules on them.

Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Modular Architecture and Declarative Interface!

Pigsty deployment is described by config inventory and materialized with ansible playbooks.
Pigsty works on Linux x86_64 common nodes, i.e., bare metals or virtual machines.
Pigsty uses a modular design that can be freely composed for different scenarios.
The config controls where & how to install modules with parameters
The playbooks will adjust nodes into the desired status in an idempotent manner.

Modules

Pigsty uses a modular design, and there are six default modules: PGSQL, INFRA, NODE, ETCD, REDIS, and MINIO.

Features

Mon, 01 Jan 0001 00:00:00 +0000

“PostgreSQL In Great STYle”: Postgres, Infras, Graphics, Service, Toolbox, it’s all Yours.

—— Battery-Included, Local-First PostgreSQL Distribution as an Open-Source RDS Alternative

Values

Extensibility: Powerful Extensions Ready for use: PostGIS, TimescaleDB, Citus, PGVector, Search, HTAP, AIML, and 255 more.
Reliability: Create self-healing High Available PostgreSQL clusters with pre-configured PITR, built-in ACL, CA & SSL in minutes.
Observability: Monitoring best practices build upon the modern Prometheus & Grafana stack: Check our live Demo & Gallery.
Availability: Deliver auto-routed, high-performance, and flexible database Services Access with haproxy, pgbouncer, and VIP.
Maintainability: Easy to use, Infra as Code API, Admin SOP, auto-tuned, local yum/apt repo, Vagrant sandbox, Terraform spec.
Composability: Modular design, Flexible Arch, bonus features: Redis, MinIO, ETCD, FerretDB, DuckDB, Supabase, Docker Apps.

Overview

Battery-Included RDS: Delivers production-ready PostgreSQL services from version 12-16 on Linux x86, spanning kernel to RDS distribution.
Plentiful Extensions: Integrates 255 extensions, providing turnkey capabilities for time-series, geospatial, full-text-search, vector and more!
Flexible Architecture: Compose Redis/Etcd/MinIO/Mongo modules on nodes, monitoring existing cluster and remote RDS, self-hosting Supabase.
Stunning Observability: Leveraging the Prometheus/Grafana modern observability stack, and provides unmatched database insights.
Proven Reliability: Self-healing HA architecture with automatic failover and uninterrupted client access, and pre-configured PITR.
Great Maintainability: Declarative API, GitOps ready, foolproof design, Database/Infra-as-Code, and management SOP seals complexity!
Sound Security: Nothing needs to be worried about database security, as long as your hardware & credentials are safe.
Versatile Application: Lots of applications work well with PostgreSQL. Run them in one command with docker.
Open Source & Free: Pigsty is a free & open source software under AGPLv3. It was built for PostgreSQL with love.

Pigsty is built-upon industry best practices:

Getting Started

Mon, 01 Jan 0001 00:00:00 +0000

About Pigsty

Mon, 01 Jan 0001 00:00:00 +0000

Cluster Model

Mon, 01 Jan 0001 00:00:00 +0000

PGSQL for production environments is organized in clusters, which clusters are logical entities consisting of a set of database instances associated by primary-replica. Each database cluster is an autonomous serving unit consisting of at least one database instance (primary).

ER Diagram

Let’s get started with ER diagram. There are four types of core entities in Pigsty’s PGSQL module:

PGSQL Cluster: An autonomous PostgreSQL business unit, used as the top-level namespace for other entities.
PGSQL Service: A named abstraction of cluster ability, route traffics, and expose postgres services with node ports.
PGSQL Instance: A single postgres server which is a group of running processes & database files on a single node.
PGSQL Node: An abstraction of hardware resources, which can be bare metal, virtual machine, or even k8s pods.

Naming Convention

Modules

Mon, 01 Jan 0001 00:00:00 +0000

Core Modules

Pigsty offers four CORE modules, which are essential for providing PostgreSQL service:

PGSQL : An autonomous, highly available PostgreSQL cluster powered by Patroni, Pgbouncer, HAproxy, PgBackrest, and others.
INFRA : Local software repository, Prometheus, Grafana, Loki, AlertManager, PushGateway, Blackbox Exporter, etc.
NODE : Adjusts the node to the desired state, name, time zone, NTP, ssh, sudo, haproxy, docker, promtail, keepalived.
ETCD : Distributed key-value store, serving as the DCS (Distributed Consensus System) for the highly available Postgres cluster: consensus leadership election, configuration management, service discovery.

Extended Modules

Pigsty offers four OPTIONAL modules,which are not necessary for the core functionality but can enhance the capabilities of PostgreSQL:

Offline Install

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty’s Standard Installation process requires Internet access, but production database servers are often isolated from the Internet.

Therefore, Pigsty offers an offline installation feature, allowing you to complete the installation and deployment in an environment without internet access.

If you have internet access, downloading the pre-made Offline Package in advance can help speed up the installation process and enhance the certainty and reliability of the installation.

Short Version

You have to download the Pigsty source tarball and the Offline Package in addition to the Standard Installation.

Roadmap

Mon, 01 Jan 0001 00:00:00 +0000

New Features on the Radar

Maintaining Deb packages for Pigsty extensions
ARM architecture support for infrastructure components
Adding more extensions to PostgreSQL
A command-line tool that’s actually good
More pre-configured scenario-based templates
Migrating package repositories and download sources entirely to Cloudflare
Deploying and monitoring high-availability Kubernetes clusters with SealOS!
Support for PostgreSQL 17 alpha
Loki and Promtail seem a bit off; could VictoriaLogs and Vector step up?
Swapping Prometheus storage for VictoriaMetrics to handle time-series data
Monitoring deployments of MySQL databases
Monitoring databases within Kubernetes
Offering a richer variety of Docker application templates

Here’s our Issues and Roadmap

Concept

Mon, 01 Jan 0001 00:00:00 +0000

Configuration

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty treats Infra & Database as Code. You can describe the infrastructure & database clusters through a declarative interface. All your essential work is to describe your need in the inventory, then materialize it with a simple idempotent playbook.

Inventory

Each pigsty deployment has a corresponding config inventory. It could be stored in a local git-managed file in YAML format or dynamically generated from CMDB or any ansible compatible format. Pigsty uses a monolith YAML config file as the default config inventory, which is pigsty.yml, located in the pigsty home directory.

History

Mon, 01 Jan 0001 00:00:00 +0000

Origin Story

The Pigsty project kicked off between 2018 and 2019, originating from Tantan, a dating app akin to China’s Tinder, now acquired by Momo. Tantan, a startup with a Nordic vibe, was founded by a team of Swedish engineers. Renowned for their tech sophistication, they chose PostgreSQL and Go as their core tech stack. Tantan’s architecture, inspired by Instagram, revolves around PostgreSQL. They managed to scale to millions of daily active users, millions of TPS, and hundreds of TBs of data using PostgreSQL exclusively. Almost all business logic was implemented using PG stored procedures, including recommendation algorithms with 100ms latency!

Parameters

Mon, 01 Jan 0001 00:00:00 +0000

There are 280+ parameters in Pigsty describing all aspect of the deployment.

ID	Name	Module	Section	Type	Level	Comment
101	`version`	`INFRA`	`META`	string	G	pigsty version string
102	`admin_ip`	`INFRA`	`META`	ip	G	admin node ip address
103	`region`	`INFRA`	`META`	enum	G	upstream mirror region: default,china,europe
104	`proxy_env`	`INFRA`	`META`	dict	G	global proxy env when downloading packages
105	`ca_method`	`INFRA`	`CA`	enum	G	create,recreate,copy, create by default
106	`ca_cn`	`INFRA`	`CA`	string	G	ca common name, fixed as pigsty-ca
107	`cert_validity`	`INFRA`	`CA`	interval	G	cert validity, 20 years by default
108	`infra_seq`	`INFRA`	`INFRA_ID`	int	I	infra node identity, REQUIRED
109	`infra_portal`	`INFRA`	`INFRA_ID`	dict	G	infra services exposed via portal
110	`repo_enabled`	`INFRA`	`REPO`	bool	G/I	create a yum repo on this infra node?
111	`repo_home`	`INFRA`	`REPO`	path	G	repo home dir, `/www` by default
112	`repo_name`	`INFRA`	`REPO`	string	G	repo name, pigsty by default
113	`repo_endpoint`	`INFRA`	`REPO`	url	G	access point to this repo by domain or ip:port
114	`repo_remove`	`INFRA`	`REPO`	bool	G/A	remove existing upstream repo
115	`repo_modules`	`INFRA`	`REPO`	string	G/A	which repo modules are installed in repo_upstream
116	`repo_upstream`	`INFRA`	`REPO`	upstream[]	G	where to download upstream packages
117	`repo_packages`	`INFRA`	`REPO`	string[]	G	which packages to be included
118	`repo_url_packages`	`INFRA`	`REPO`	string[]	G	extra packages from url
120	`infra_packages`	`INFRA`	`INFRA_PACKAGE`	string[]	G	packages to be installed on infra nodes
121	`infra_packages_pip`	`INFRA`	`INFRA_PACKAGE`	string	G	pip installed packages for infra nodes
130	`nginx_enabled`	`INFRA`	`NGINX`	bool	G/I	enable nginx on this infra node?
131	`nginx_exporter_enabled`	`INFRA`	`NGINX`	bool	G/I	enable nginx_exporter on this infra node?
132	`nginx_sslmode`	`INFRA`	`NGINX`	enum	G	nginx ssl mode? disable,enable,enforce
133	`nginx_home`	`INFRA`	`NGINX`	path	G	nginx content dir, `/www` by default
134	`nginx_port`	`INFRA`	`NGINX`	port	G	nginx listen port, 80 by default
135	`nginx_ssl_port`	`INFRA`	`NGINX`	port	G	nginx ssl listen port, 443 by default
136	`nginx_navbar`	`INFRA`	`NGINX`	index[]	G	nginx index page navigation links
140	`dns_enabled`	`INFRA`	`DNS`	bool	G/I	setup dnsmasq on this infra node?
141	`dns_port`	`INFRA`	`DNS`	port	G	dns server listen port, 53 by default
142	`dns_records`	`INFRA`	`DNS`	string[]	G	dynamic dns records resolved by dnsmasq
150	`prometheus_enabled`	`INFRA`	`PROMETHEUS`	bool	G/I	enable prometheus on this infra node?
151	`prometheus_clean`	`INFRA`	`PROMETHEUS`	bool	G/A	clean prometheus data during init?
152	`prometheus_data`	`INFRA`	`PROMETHEUS`	path	G	prometheus data dir, `/data/prometheus` by default
153	`prometheus_sd_dir`	`INFRA`	`PROMETHEUS`	path	G	prometheus file service discovery directory
154	`prometheus_sd_interval`	`INFRA`	`PROMETHEUS`	interval	G	prometheus target refresh interval, 5s by default
155	`prometheus_scrape_interval`	`INFRA`	`PROMETHEUS`	interval	G	prometheus scrape & eval interval, 10s by default
156	`prometheus_scrape_timeout`	`INFRA`	`PROMETHEUS`	interval	G	prometheus global scrape timeout, 8s by default
157	`prometheus_options`	`INFRA`	`PROMETHEUS`	arg	G	prometheus extra server options
158	`pushgateway_enabled`	`INFRA`	`PROMETHEUS`	bool	G/I	setup pushgateway on this infra node?
159	`pushgateway_options`	`INFRA`	`PROMETHEUS`	arg	G	pushgateway extra server options
160	`blackbox_enabled`	`INFRA`	`PROMETHEUS`	bool	G/I	setup blackbox_exporter on this infra node?
161	`blackbox_options`	`INFRA`	`PROMETHEUS`	arg	G	blackbox_exporter extra server options
162	`alertmanager_enabled`	`INFRA`	`PROMETHEUS`	bool	G/I	setup alertmanager on this infra node?
163	`alertmanager_options`	`INFRA`	`PROMETHEUS`	arg	G	alertmanager extra server options
164	`exporter_metrics_path`	`INFRA`	`PROMETHEUS`	path	G	exporter metric path, `/metrics` by default
165	`exporter_install`	`INFRA`	`PROMETHEUS`	enum	G	how to install exporter? none,yum,binary
166	`exporter_repo_url`	`INFRA`	`PROMETHEUS`	url	G	exporter repo file url if install exporter via yum
170	`grafana_enabled`	`INFRA`	`GRAFANA`	bool	G/I	enable grafana on this infra node?
171	`grafana_clean`	`INFRA`	`GRAFANA`	bool	G/A	clean grafana data during init?
172	`grafana_admin_username`	`INFRA`	`GRAFANA`	username	G	grafana admin username, `admin` by default
173	`grafana_admin_password`	`INFRA`	`GRAFANA`	password	G	grafana admin password, `pigsty` by default
174	`grafana_plugin_cache`	`INFRA`	`GRAFANA`	path	G	path to grafana plugins cache tarball
175	`grafana_plugin_list`	`INFRA`	`GRAFANA`	string[]	G	grafana plugins to be downloaded with grafana-cli
176	`loki_enabled`	`INFRA`	`LOKI`	bool	G/I	enable loki on this infra node?
177	`loki_clean`	`INFRA`	`LOKI`	bool	G/A	whether remove existing loki data?
178	`loki_data`	`INFRA`	`LOKI`	path	G	loki data dir, `/data/loki` by default
179	`loki_retention`	`INFRA`	`LOKI`	interval	G	loki log retention period, 15d by default
201	`nodename`	`NODE`	`NODE_ID`	string	I	node instance identity, use hostname if missing, optional
202	`node_cluster`	`NODE`	`NODE_ID`	string	C	node cluster identity, use ’nodes’ if missing, optional
203	`nodename_overwrite`	`NODE`	`NODE_ID`	bool	C	overwrite node’s hostname with nodename?
204	`nodename_exchange`	`NODE`	`NODE_ID`	bool	C	exchange nodename among play hosts?
205	`node_id_from_pg`	`NODE`	`NODE_ID`	bool	C	use postgres identity as node identity if applicable?
210	`node_write_etc_hosts`	`NODE`	`NODE_DNS`	bool	G/C/I	modify `/etc/hosts` on target node?
211	`node_default_etc_hosts`	`NODE`	`NODE_DNS`	string[]	G	static dns records in `/etc/hosts`
212	`node_etc_hosts`	`NODE`	`NODE_DNS`	string[]	C	extra static dns records in `/etc/hosts`
213	`node_dns_method`	`NODE`	`NODE_DNS`	enum	C	how to handle dns servers: add,none,overwrite
214	`node_dns_servers`	`NODE`	`NODE_DNS`	string[]	C	dynamic nameserver in `/etc/resolv.conf`
215	`node_dns_options`	`NODE`	`NODE_DNS`	string[]	C	dns resolv options in `/etc/resolv.conf`
220	`node_repo_modules`	`NODE`	`NODE_PACKAGE`	string	C	upstream repo to be added on node, local by default
221	`node_repo_remove`	`NODE`	`NODE_PACKAGE`	bool	C	remove existing repo on node?
223	`node_packages`	`NODE`	`NODE_PACKAGE`	string[]	C	packages to be installed current nodes
224	`node_default_packages`	`NODE`	`NODE_PACKAGE`	string[]	G	default packages to be installed on all nodes
230	`node_disable_firewall`	`NODE`	`NODE_TUNE`	bool	C	disable node firewall? true by default
231	`node_disable_selinux`	`NODE`	`NODE_TUNE`	bool	C	disable node selinux? true by default
232	`node_disable_numa`	`NODE`	`NODE_TUNE`	bool	C	disable node numa, reboot required
233	`node_disable_swap`	`NODE`	`NODE_TUNE`	bool	C	disable node swap, use with caution
234	`node_static_network`	`NODE`	`NODE_TUNE`	bool	C	preserve dns resolver settings after reboot
235	`node_disk_prefetch`	`NODE`	`NODE_TUNE`	bool	C	setup disk prefetch on HDD to increase performance
236	`node_kernel_modules`	`NODE`	`NODE_TUNE`	string[]	C	kernel modules to be enabled on this node
237	`node_hugepage_count`	`NODE`	`NODE_TUNE`	int	C	number of 2MB hugepage, take precedence over ratio
238	`node_hugepage_ratio`	`NODE`	`NODE_TUNE`	float	C	node mem hugepage ratio, 0 disable it by default
239	`node_overcommit_ratio`	`NODE`	`NODE_TUNE`	float	C	node mem overcommit ratio, 0 disable it by default
240	`node_tune`	`NODE`	`NODE_TUNE`	enum	C	node tuned profile: none,oltp,olap,crit,tiny
241	`node_sysctl_params`	`NODE`	`NODE_TUNE`	dict	C	sysctl parameters in k:v format in addition to tuned
250	`node_data`	`NODE`	`NODE_ADMIN`	path	C	node main data directory, `/data` by default
251	`node_admin_enabled`	`NODE`	`NODE_ADMIN`	bool	C	create a admin user on target node?
252	`node_admin_uid`	`NODE`	`NODE_ADMIN`	int	C	uid and gid for node admin user
253	`node_admin_username`	`NODE`	`NODE_ADMIN`	username	C	name of node admin user, `dba` by default
254	`node_admin_ssh_exchange`	`NODE`	`NODE_ADMIN`	bool	C	exchange admin ssh key among node cluster
255	`node_admin_pk_current`	`NODE`	`NODE_ADMIN`	bool	C	add current user’s ssh pk to admin authorized_keys
256	`node_admin_pk_list`	`NODE`	`NODE_ADMIN`	string[]	C	ssh public keys to be added to admin user
260	`node_timezone`	`NODE`	`NODE_TIME`	string	C	setup node timezone, empty string to skip
261	`node_ntp_enabled`	`NODE`	`NODE_TIME`	bool	C	enable chronyd time sync service?
262	`node_ntp_servers`	`NODE`	`NODE_TIME`	string[]	C	ntp servers in `/etc/chrony.conf`
263	`node_crontab_overwrite`	`NODE`	`NODE_TIME`	bool	C	overwrite or append to `/etc/crontab`?
264	`node_crontab`	`NODE`	`NODE_TIME`	string[]	C	crontab entries in `/etc/crontab`
270	`vip_enabled`	`NODE`	`NODE_VIP`	bool	C	enable vip on this node cluster?
271	`vip_address`	`NODE`	`NODE_VIP`	ip	C	node vip address in ipv4 format, required if vip is enabled
272	`vip_vrid`	`NODE`	`NODE_VIP`	int	C	required, integer, 1-254, should be unique among same VLAN
273	`vip_role`	`NODE`	`NODE_VIP`	enum	I	optional, `master/backup`, backup by default, use as init role
274	`vip_preempt`	`NODE`	`NODE_VIP`	bool	C/I	optional, `true/false`, false by default, enable vip preemption
275	`vip_interface`	`NODE`	`NODE_VIP`	string	C/I	node vip network interface to listen, `eth0` by default
276	`vip_dns_suffix`	`NODE`	`NODE_VIP`	string	C	node vip dns name suffix, empty string by default
277	`vip_exporter_port`	`NODE`	`NODE_VIP`	port	C	keepalived exporter listen port, 9650 by default
280	`haproxy_enabled`	`NODE`	`HAPROXY`	bool	C	enable haproxy on this node?
281	`haproxy_clean`	`NODE`	`HAPROXY`	bool	G/C/A	cleanup all existing haproxy config?
282	`haproxy_reload`	`NODE`	`HAPROXY`	bool	A	reload haproxy after config?
283	`haproxy_auth_enabled`	`NODE`	`HAPROXY`	bool	G	enable authentication for haproxy admin page
284	`haproxy_admin_username`	`NODE`	`HAPROXY`	username	G	haproxy admin username, `admin` by default
285	`haproxy_admin_password`	`NODE`	`HAPROXY`	password	G	haproxy admin password, `pigsty` by default
286	`haproxy_exporter_port`	`NODE`	`HAPROXY`	port	C	haproxy admin/exporter port, 9101 by default
287	`haproxy_client_timeout`	`NODE`	`HAPROXY`	interval	C	client side connection timeout, 24h by default
288	`haproxy_server_timeout`	`NODE`	`HAPROXY`	interval	C	server side connection timeout, 24h by default
289	`haproxy_services`	`NODE`	`HAPROXY`	service[]	C	list of haproxy service to be exposed on node
290	`node_exporter_enabled`	`NODE`	`NODE_EXPORTER`	bool	C	setup node_exporter on this node?
291	`node_exporter_port`	`NODE`	`NODE_EXPORTER`	port	C	node exporter listen port, 9100 by default
292	`node_exporter_options`	`NODE`	`NODE_EXPORTER`	arg	C	extra server options for node_exporter
293	`promtail_enabled`	`NODE`	`PROMTAIL`	bool	C	enable promtail logging collector?
294	`promtail_clean`	`NODE`	`PROMTAIL`	bool	G/A	purge existing promtail status file during init?
295	`promtail_port`	`NODE`	`PROMTAIL`	port	C	promtail listen port, 9080 by default
296	`promtail_positions`	`NODE`	`PROMTAIL`	path	C	promtail position status file path
401	`docker_enabled`	`NODE`	`DOCKER`	bool	C	enable docker on this node?
402	`docker_cgroups_driver`	`NODE`	`DOCKER`	enum	C	docker cgroup fs driver: cgroupfs,systemd
403	`docker_registry_mirrors`	`NODE`	`DOCKER`	string[]	C	docker registry mirror list
404	`docker_image_cache`	`NODE`	`DOCKER`	path	C	docker image cache dir, `/tmp/docker` by default
501	`etcd_seq`	`ETCD`	`ETCD`	int	I	etcd instance identifier, REQUIRED
502	`etcd_cluster`	`ETCD`	`ETCD`	string	C	etcd cluster & group name, etcd by default
503	`etcd_safeguard`	`ETCD`	`ETCD`	bool	G/C/A	prevent purging running etcd instance?
504	`etcd_clean`	`ETCD`	`ETCD`	bool	G/C/A	purging existing etcd during initialization?
505	`etcd_data`	`ETCD`	`ETCD`	path	C	etcd data directory, /data/etcd by default
506	`etcd_port`	`ETCD`	`ETCD`	port	C	etcd client port, 2379 by default
507	`etcd_peer_port`	`ETCD`	`ETCD`	port	C	etcd peer port, 2380 by default
508	`etcd_init`	`ETCD`	`ETCD`	enum	C	etcd initial cluster state, new or existing
509	`etcd_election_timeout`	`ETCD`	`ETCD`	int	C	etcd election timeout, 1000ms by default
510	`etcd_heartbeat_interval`	`ETCD`	`ETCD`	int	C	etcd heartbeat interval, 100ms by default
601	`minio_seq`	`MINIO`	`MINIO`	int	I	minio instance identifier, REQUIRED
602	`minio_cluster`	`MINIO`	`MINIO`	string	C	minio cluster name, minio by default
603	`minio_clean`	`MINIO`	`MINIO`	bool	G/C/A	cleanup minio during init?, false by default
604	`minio_user`	`MINIO`	`MINIO`	username	C	minio os user, `minio` by default
605	`minio_node`	`MINIO`	`MINIO`	string	C	minio node name pattern
606	`minio_data`	`MINIO`	`MINIO`	path	C	minio data dir(s), use {x…y} to specify multi drivers
607	`minio_domain`	`MINIO`	`MINIO`	string	G	minio service domain name, `sss.pigsty` by default
608	`minio_port`	`MINIO`	`MINIO`	port	C	minio service port, 9000 by default
609	`minio_admin_port`	`MINIO`	`MINIO`	port	C	minio console port, 9001 by default
610	`minio_access_key`	`MINIO`	`MINIO`	username	C	root access key, `minioadmin` by default
611	`minio_secret_key`	`MINIO`	`MINIO`	password	C	root secret key, `minioadmin` by default
612	`minio_extra_vars`	`MINIO`	`MINIO`	string	C	extra environment variables for minio server
613	`minio_alias`	`MINIO`	`MINIO`	string	G	alias name for local minio deployment
614	`minio_buckets`	`MINIO`	`MINIO`	bucket[]	C	list of minio bucket to be created
615	`minio_users`	`MINIO`	`MINIO`	user[]	C	list of minio user to be created
701	`redis_cluster`	`REDIS`	`REDIS`	string	C	redis cluster name, required identity parameter
702	`redis_instances`	`REDIS`	`REDIS`	dict	I	redis instances definition on this redis node
703	`redis_node`	`REDIS`	`REDIS`	int	I	redis node sequence number, node int id required
710	`redis_fs_main`	`REDIS`	`REDIS`	path	C	redis main data mountpoint, `/data` by default
711	`redis_exporter_enabled`	`REDIS`	`REDIS`	bool	C	install redis exporter on redis nodes?
712	`redis_exporter_port`	`REDIS`	`REDIS`	port	C	redis exporter listen port, 9121 by default
713	`redis_exporter_options`	`REDIS`	`REDIS`	string	C/I	cli args and extra options for redis exporter
720	`redis_safeguard`	`REDIS`	`REDIS`	bool	G/C/A	prevent purging running redis instance?
721	`redis_clean`	`REDIS`	`REDIS`	bool	G/C/A	purging existing redis during init?
722	`redis_rmdata`	`REDIS`	`REDIS`	bool	G/C/A	remove redis data when purging redis server?
723	`redis_mode`	`REDIS`	`REDIS`	enum	C	redis mode: standalone,cluster,sentinel
724	`redis_conf`	`REDIS`	`REDIS`	string	C	redis config template path, except sentinel
725	`redis_bind_address`	`REDIS`	`REDIS`	ip	C	redis bind address, empty string will use host ip
726	`redis_max_memory`	`REDIS`	`REDIS`	size	C/I	max memory used by each redis instance
727	`redis_mem_policy`	`REDIS`	`REDIS`	enum	C	redis memory eviction policy
728	`redis_password`	`REDIS`	`REDIS`	password	C	redis password, empty string will disable password
729	`redis_rdb_save`	`REDIS`	`REDIS`	string[]	C	redis rdb save directives, disable with empty list
730	`redis_aof_enabled`	`REDIS`	`REDIS`	bool	C	enable redis append only file?
731	`redis_rename_commands`	`REDIS`	`REDIS`	dict	C	rename redis dangerous commands
732	`redis_cluster_replicas`	`REDIS`	`REDIS`	int	C	replica number for one master in redis cluster
733	`redis_sentinel_monitor`	`REDIS`	`REDIS`	master[]	C	sentinel master list, works on sentinel cluster only
801	`pg_mode`	`PGSQL`	`PG_ID`	enum	C	pgsql cluster mode: pgsql,citus,gpsql
802	`pg_cluster`	`PGSQL`	`PG_ID`	string	C	pgsql cluster name, REQUIRED identity parameter
803	`pg_seq`	`PGSQL`	`PG_ID`	int	I	pgsql instance seq number, REQUIRED identity parameter
804	`pg_role`	`PGSQL`	`PG_ID`	enum	I	pgsql role, REQUIRED, could be primary,replica,offline
805	`pg_instances`	`PGSQL`	`PG_ID`	dict	I	define multiple pg instances on node in `{port:ins_vars}` format
806	`pg_upstream`	`PGSQL`	`PG_ID`	ip	I	repl upstream ip addr for standby cluster or cascade replica
807	`pg_shard`	`PGSQL`	`PG_ID`	string	C	pgsql shard name, optional identity for sharding clusters
808	`pg_group`	`PGSQL`	`PG_ID`	int	C	pgsql shard index number, optional identity for sharding clusters
809	`gp_role`	`PGSQL`	`PG_ID`	enum	C	greenplum role of this cluster, could be master or segment
810	`pg_exporters`	`PGSQL`	`PG_ID`	dict	C	additional pg_exporters to monitor remote postgres instances
811	`pg_offline_query`	`PGSQL`	`PG_ID`	bool	I	set to true to enable offline query on this instance
820	`pg_users`	`PGSQL`	`PG_BUSINESS`	user[]	C	postgres business users
821	`pg_databases`	`PGSQL`	`PG_BUSINESS`	database[]	C	postgres business databases
822	`pg_services`	`PGSQL`	`PG_BUSINESS`	service[]	C	postgres business services
823	`pg_hba_rules`	`PGSQL`	`PG_BUSINESS`	hba[]	C	business hba rules for postgres
824	`pgb_hba_rules`	`PGSQL`	`PG_BUSINESS`	hba[]	C	business hba rules for pgbouncer
831	`pg_replication_username`	`PGSQL`	`PG_BUSINESS`	username	G	postgres replication username, `replicator` by default
832	`pg_replication_password`	`PGSQL`	`PG_BUSINESS`	password	G	postgres replication password, `DBUser.Replicator` by default
833	`pg_admin_username`	`PGSQL`	`PG_BUSINESS`	username	G	postgres admin username, `dbuser_dba` by default
834	`pg_admin_password`	`PGSQL`	`PG_BUSINESS`	password	G	postgres admin password in plain text, `DBUser.DBA` by default
835	`pg_monitor_username`	`PGSQL`	`PG_BUSINESS`	username	G	postgres monitor username, `dbuser_monitor` by default
836	`pg_monitor_password`	`PGSQL`	`PG_BUSINESS`	password	G	postgres monitor password, `DBUser.Monitor` by default
837	`pg_dbsu_password`	`PGSQL`	`PG_BUSINESS`	password	G/C	postgres dbsu password, empty string disable it by default
840	`pg_dbsu`	`PGSQL`	`PG_INSTALL`	username	C	os dbsu name, postgres by default, better not change it
841	`pg_dbsu_uid`	`PGSQL`	`PG_INSTALL`	int	C	os dbsu uid and gid, 26 for default postgres users and groups
842	`pg_dbsu_sudo`	`PGSQL`	`PG_INSTALL`	enum	C	dbsu sudo privilege, none,limit,all,nopass. limit by default
843	`pg_dbsu_home`	`PGSQL`	`PG_INSTALL`	path	C	postgresql home directory, `/var/lib/pgsql` by default
844	`pg_dbsu_ssh_exchange`	`PGSQL`	`PG_INSTALL`	bool	C	exchange postgres dbsu ssh key among same pgsql cluster
845	`pg_version`	`PGSQL`	`PG_INSTALL`	enum	C	postgres major version to be installed, 16 by default
846	`pg_bin_dir`	`PGSQL`	`PG_INSTALL`	path	C	postgres binary dir, `/usr/pgsql/bin` by default
847	`pg_log_dir`	`PGSQL`	`PG_INSTALL`	path	C	postgres log dir, `/pg/log/postgres` by default
848	`pg_packages`	`PGSQL`	`PG_INSTALL`	string[]	C	pg packages to be installed, `${pg_version}` will be replaced
849	`pg_extensions`	`PGSQL`	`PG_INSTALL`	string[]	C	pg extensions to be installed, `${pg_version}` will be replaced
850	`pg_safeguard`	`PGSQL`	`PG_BOOTSTRAP`	bool	G/C/A	prevent purging running postgres instance? false by default
851	`pg_clean`	`PGSQL`	`PG_BOOTSTRAP`	bool	G/C/A	purging existing postgres during pgsql init? true by default
852	`pg_data`	`PGSQL`	`PG_BOOTSTRAP`	path	C	postgres data directory, `/pg/data` by default
853	`pg_fs_main`	`PGSQL`	`PG_BOOTSTRAP`	path	C	mountpoint/path for postgres main data, `/data` by default
854	`pg_fs_bkup`	`PGSQL`	`PG_BOOTSTRAP`	path	C	mountpoint/path for pg backup data, `/data/backup` by default
855	`pg_storage_type`	`PGSQL`	`PG_BOOTSTRAP`	enum	C	storage type for pg main data, SSD,HDD, SSD by default
856	`pg_dummy_filesize`	`PGSQL`	`PG_BOOTSTRAP`	size	C	size of `/pg/dummy`, hold 64MB disk space for emergency use
857	`pg_listen`	`PGSQL`	`PG_BOOTSTRAP`	ip(s)	C/I	postgres/pgbouncer listen addresses, comma separated list
858	`pg_port`	`PGSQL`	`PG_BOOTSTRAP`	port	C	postgres listen port, 5432 by default
859	`pg_localhost`	`PGSQL`	`PG_BOOTSTRAP`	path	C	postgres unix socket dir for localhost connection
860	`pg_namespace`	`PGSQL`	`PG_BOOTSTRAP`	path	C	top level key namespace in etcd, used by patroni & vip
861	`patroni_enabled`	`PGSQL`	`PG_BOOTSTRAP`	bool	C	if disabled, no postgres cluster will be created during init
862	`patroni_mode`	`PGSQL`	`PG_BOOTSTRAP`	enum	C	patroni working mode: default,pause,remove
863	`patroni_port`	`PGSQL`	`PG_BOOTSTRAP`	port	C	patroni listen port, 8008 by default
864	`patroni_log_dir`	`PGSQL`	`PG_BOOTSTRAP`	path	C	patroni log dir, `/pg/log/patroni` by default
865	`patroni_ssl_enabled`	`PGSQL`	`PG_BOOTSTRAP`	bool	G	secure patroni RestAPI communications with SSL?
866	`patroni_watchdog_mode`	`PGSQL`	`PG_BOOTSTRAP`	enum	C	patroni watchdog mode: automatic,required,off. off by default
867	`patroni_username`	`PGSQL`	`PG_BOOTSTRAP`	username	C	patroni restapi username, `postgres` by default
868	`patroni_password`	`PGSQL`	`PG_BOOTSTRAP`	password	C	patroni restapi password, `Patroni.API` by default
869	`patroni_citus_db`	`PGSQL`	`PG_BOOTSTRAP`	string	C	citus database managed by patroni, postgres by default
870	`pg_conf`	`PGSQL`	`PG_BOOTSTRAP`	enum	C	config template: oltp,olap,crit,tiny. `oltp.yml` by default
871	`pg_max_conn`	`PGSQL`	`PG_BOOTSTRAP`	int	C	postgres max connections, `auto` will use recommended value
872	`pg_shared_buffer_ratio`	`PGSQL`	`PG_BOOTSTRAP`	float	C	postgres shared buffer memory ratio, 0.25 by default, 0.1~0.4
873	`pg_rto`	`PGSQL`	`PG_BOOTSTRAP`	int	C	recovery time objective in seconds, `30s` by default
874	`pg_rpo`	`PGSQL`	`PG_BOOTSTRAP`	int	C	recovery point objective in bytes, `1MiB` at most by default
875	`pg_libs`	`PGSQL`	`PG_BOOTSTRAP`	string	C	preloaded libraries, `pg_stat_statements,auto_explain` by default
876	`pg_delay`	`PGSQL`	`PG_BOOTSTRAP`	interval	I	replication apply delay for standby cluster leader
877	`pg_checksum`	`PGSQL`	`PG_BOOTSTRAP`	bool	C	enable data checksum for postgres cluster?
878	`pg_pwd_enc`	`PGSQL`	`PG_BOOTSTRAP`	enum	C	passwords encryption algorithm: md5,scram-sha-256
879	`pg_encoding`	`PGSQL`	`PG_BOOTSTRAP`	enum	C	database cluster encoding, `UTF8` by default
880	`pg_locale`	`PGSQL`	`PG_BOOTSTRAP`	enum	C	database cluster local, `C` by default
881	`pg_lc_collate`	`PGSQL`	`PG_BOOTSTRAP`	enum	C	database cluster collate, `C` by default
882	`pg_lc_ctype`	`PGSQL`	`PG_BOOTSTRAP`	enum	C	database character type, `en_US.UTF8` by default
890	`pgbouncer_enabled`	`PGSQL`	`PG_BOOTSTRAP`	bool	C	if disabled, pgbouncer will not be launched on pgsql host
891	`pgbouncer_port`	`PGSQL`	`PG_BOOTSTRAP`	port	C	pgbouncer listen port, 6432 by default
892	`pgbouncer_log_dir`	`PGSQL`	`PG_BOOTSTRAP`	path	C	pgbouncer log dir, `/pg/log/pgbouncer` by default
893	`pgbouncer_auth_query`	`PGSQL`	`PG_BOOTSTRAP`	bool	C	query postgres to retrieve unlisted business users?
894	`pgbouncer_poolmode`	`PGSQL`	`PG_BOOTSTRAP`	enum	C	pooling mode: transaction,session,statement, transaction by default
895	`pgbouncer_sslmode`	`PGSQL`	`PG_BOOTSTRAP`	enum	C	pgbouncer client ssl mode, disable by default
900	`pg_provision`	`PGSQL`	`PG_PROVISION`	bool	C	provision postgres cluster after bootstrap
901	`pg_init`	`PGSQL`	`PG_PROVISION`	string	G/C	provision init script for cluster template, `pg-init` by default
902	`pg_default_roles`	`PGSQL`	`PG_PROVISION`	role[]	G/C	default roles and users in postgres cluster
903	`pg_default_privileges`	`PGSQL`	`PG_PROVISION`	string[]	G/C	default privileges when created by admin user
904	`pg_default_schemas`	`PGSQL`	`PG_PROVISION`	string[]	G/C	default schemas to be created
905	`pg_default_extensions`	`PGSQL`	`PG_PROVISION`	extension[]	G/C	default extensions to be created
906	`pg_reload`	`PGSQL`	`PG_PROVISION`	bool	A	reload postgres after hba changes
907	`pg_default_hba_rules`	`PGSQL`	`PG_PROVISION`	hba[]	G/C	postgres default host-based authentication rules
908	`pgb_default_hba_rules`	`PGSQL`	`PG_PROVISION`	hba[]	G/C	pgbouncer default host-based authentication rules
910	`pgbackrest_enabled`	`PGSQL`	`PG_BACKUP`	bool	C	enable pgbackrest on pgsql host?
911	`pgbackrest_clean`	`PGSQL`	`PG_BACKUP`	bool	C	remove pg backup data during init?
912	`pgbackrest_log_dir`	`PGSQL`	`PG_BACKUP`	path	C	pgbackrest log dir, `/pg/log/pgbackrest` by default
913	`pgbackrest_method`	`PGSQL`	`PG_BACKUP`	enum	C	pgbackrest repo method: local,minio,etc…
914	`pgbackrest_repo`	`PGSQL`	`PG_BACKUP`	dict	G/C	pgbackrest repo: https://pgbackrest.org/configuration.html#section-repository
921	`pg_weight`	`PGSQL`	`PG_SERVICE`	int	I	relative load balance weight in service, 100 by default, 0-255
922	`pg_service_provider`	`PGSQL`	`PG_SERVICE`	string	G/C	dedicate haproxy node group name, or empty string for local nodes by default
923	`pg_default_service_dest`	`PGSQL`	`PG_SERVICE`	enum	G/C	default service destination if svc.dest=‘default’
924	`pg_default_services`	`PGSQL`	`PG_SERVICE`	service[]	G/C	postgres default service definitions
931	`pg_vip_enabled`	`PGSQL`	`PG_SERVICE`	bool	C	enable a l2 vip for pgsql primary? false by default
932	`pg_vip_address`	`PGSQL`	`PG_SERVICE`	cidr4	C	vip address in `<ipv4>/<mask>` format, require if vip is enabled
933	`pg_vip_interface`	`PGSQL`	`PG_SERVICE`	string	C/I	vip network interface to listen, eth0 by default
934	`pg_dns_suffix`	`PGSQL`	`PG_SERVICE`	string	C	pgsql dns suffix, ’’ by default
935	`pg_dns_target`	`PGSQL`	`PG_SERVICE`	enum	C	auto, primary, vip, none, or ad hoc ip
940	`pg_exporter_enabled`	`PGSQL`	`PG_EXPORTER`	bool	C	enable pg_exporter on pgsql hosts?
941	`pg_exporter_config`	`PGSQL`	`PG_EXPORTER`	string	C	pg_exporter configuration file name
942	`pg_exporter_cache_ttls`	`PGSQL`	`PG_EXPORTER`	string	C	pg_exporter collector ttl stage in seconds, ‘1,10,60,300’ by default
943	`pg_exporter_port`	`PGSQL`	`PG_EXPORTER`	port	C	pg_exporter listen port, 9630 by default
944	`pg_exporter_params`	`PGSQL`	`PG_EXPORTER`	string	C	extra url parameters for pg_exporter dsn
945	`pg_exporter_url`	`PGSQL`	`PG_EXPORTER`	pgurl	C	overwrite auto-generate pg dsn if specified
946	`pg_exporter_auto_discovery`	`PGSQL`	`PG_EXPORTER`	bool	C	enable auto database discovery? enabled by default
947	`pg_exporter_exclude_database`	`PGSQL`	`PG_EXPORTER`	string	C	csv of database that WILL NOT be monitored during auto-discovery
948	`pg_exporter_include_database`	`PGSQL`	`PG_EXPORTER`	string	C	csv of database that WILL BE monitored during auto-discovery
949	`pg_exporter_connect_timeout`	`PGSQL`	`PG_EXPORTER`	int	C	pg_exporter connect timeout in ms, 200 by default
950	`pg_exporter_options`	`PGSQL`	`PG_EXPORTER`	arg	C	overwrite extra options for pg_exporter
951	`pgbouncer_exporter_enabled`	`PGSQL`	`PG_EXPORTER`	bool	C	enable pgbouncer_exporter on pgsql hosts?
952	`pgbouncer_exporter_port`	`PGSQL`	`PG_EXPORTER`	port	C	pgbouncer_exporter listen port, 9631 by default
953	`pgbouncer_exporter_url`	`PGSQL`	`PG_EXPORTER`	pgurl	C	overwrite auto-generate pgbouncer dsn if specified
954	`pgbouncer_exporter_options`	`PGSQL`	`PG_EXPORTER`	arg	C	overwrite extra options for pgbouncer_exporter

`INFRA`

Parameters about pigsty infrastructure components: local yum repo, nginx, dnsmasq, prometheus, grafana, loki, alertmanager, pushgateway, blackbox_exporter, etc…

Preparation

Mon, 01 Jan 0001 00:00:00 +0000

Node

Pigsty supports the Linux kernel and x86_64/amd64 arch, applicable to any node.

A “node” refers to a resource that is SSH accessible and offers a bare OS environment, such as a physical machine, a virtual machine, or an OS container equipped with systemd and sshd.

Deploying Pigsty requires at least 1 node. The minimum spec requirement is 1C1G, but it is recommended to use at least 2C4G, with no upper limit: parameters will automatically optimize and adapt.

Event & News

Mon, 01 Jan 0001 00:00:00 +0000

Latest Event

PGCon.Dev 2024
Pigsty v2.7 Released!
- PostgreSQL Website: Pigsty v2.7 Released, free RDS PG with 255 extensions available
- Postgres Weekly: [https://postgresweekly.com/issues/556)
- Pigsty Blog: Pigsty v2.7: Extension Overwhelming
Pigsty v2.6 Released!
- PostgreSQL Website: Pigsty, Battery-included PostgreSQL Distro & Free RDS Alternative, v2.6 released!
- Planet Postgres (Twitter): https://twitter.com/PostgreSQL/status/1765323952669290515
- Postgres Weekly: https://postgresweekly.com/issues/545
- Pigsty Blog: Pigsty v2.6：the new OLAP challenger – PostgreSQL

The name of this project always makes me grin: PIGSTY is actually an acronym, standing for Postgres In Great STYle! It’s a Postgres distribution that includes lots of components and tools out of the box in areas like availability, deployment, and observability. The latest release pushes everything up to Postgres 16.2 standards and introduces new ParadeDB and DuckDB FDW extensions.

Planning

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty v2.7 planning starts by selecting a deployment topology and preparing the matching hosts, operating systems, network, and administrator access.

Planning Checklist

Choose a single-node, monitoring-only, or highly available topology from the provisioning guide and configuration examples.
Verify CPU architecture, operating-system release, storage, and network requirements in Prepare and the compatibility matrix.
Reserve stable node addresses, hostnames, required ports, and passwordless administrator SSH/sudo access before installation.
Decide between online and offline installation and prepare the corresponding software source or offline package.
Review the inventory in Configuration and the security checklist before running playbooks.

The exact cluster, user, database, service, and backup settings remain in the linked v2.7 pages; this page is only the historical planning entry point.

References

Mon, 01 Jan 0001 00:00:00 +0000

Community

Mon, 01 Jan 0001 00:00:00 +0000

The Pigsty community already offers free WeChat/Discord/Telegram Q&A Office Hours, and we are also happy to provide more free value-added services to our supporters.

GitHub

Our GitHub Repo: https://github.com/Vonng/pigsty , welcome to watch and star us.

Everyone is very welcome to submit new Issue or create Pull Request, propose feature suggestions and participate in Pigsty contribution.

Beware that for Pigsty documentation issues, please submit Issue in the github.com/Vonng/pigsty.cc repository.

Infra as Code

Mon, 01 Jan 0001 00:00:00 +0000

Infra as Code, Database as Code, Declarative API & Idempotent Playbooks, GitOPS works like a charm.

Pigsty provides a declarative interface: Describe everything in a config file, and Pigsty operates it to the desired state with idempotent playbooks. It works like Kubernetes CRDs & Operators but for databases and infrastructures on any nodes: bare metal or virtual machines.

Declare Module

Take the default config snippet as an example, which describes a node 10.10.10.10 with modules INFRA, NODE, ETCD, and PGSQL installed.

File Hierarchy

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty FHS

#------------------------------------------------------------------------------
# pigsty
#  ^-----@app                    # extra demo application resources
#  ^-----@bin                    # bin scripts
#  ^-----@docs                   # document (can be docsified)
#  ^-----@files                  # ansible file resources
#            ^-----@pigsty       # pigsty config template files
#            ^-----@prometheus   # prometheus rules definition
#            ^-----@grafana      # grafana dashboards
#            ^-----@postgres     # /pg/bin/ scripts
#            ^-----@migration    # pgsql migration task definition
#            ^-----@pki          # self-signed CA & certs

#  ^-----@roles                  # ansible business logic
#  ^-----@templates              # ansible templates
#  ^-----@vagrant                # vagrant local VM template
#  ^-----@terraform              # terraform cloud VM template
#  ^-----configure               # configure wizard script
#  ^-----ansible.cfg             # default ansible config file
#  ^-----pigsty.yml              # default config file
#  ^-----*.yml                   # ansible playbooks

#------------------------------------------------------------------------------
# /etc/pigsty/
#  ^-----@targets                # file based service discovery targets definition
#  ^-----@dashboards             # static grafana dashboards
#  ^-----@datasources            # static grafana datasource
#  ^-----@playbooks              # extra ansible playbooks
#------------------------------------------------------------------------------

CA FHS

Pigsty’s self-signed CA is located on files/pki/ directory under pigsty home.

High Availability

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Primary Failure RTO ≈ 30s, RPO < 1MB, Replica Failure RTO≈0 (reset current conn)

Pigsty’s PostgreSQL cluster has battery-included high-availability powered by Patroni, Etcd, and HAProxy

When your have two or more instances in the PostgreSQL cluster, you have the ability to self-healing from hardware failures without any further configuration — as long as any instance within the cluster survives, the cluster can serve its services. Clients simply need to connect to any node in the cluster to obtain full services without worrying about replication topology changes.

Playbooks

Mon, 01 Jan 0001 00:00:00 +0000

Playbooks are used in Pigsty to install modules on nodes.

To run playbooks, just treat them as executables. e.g. run with ./install.yml.

Playbooks

Here are default playbooks included in Pigsty.

Playbook	Function
`install.yml`	Install Pigsty on current node in one-pass
`infra.yml`	Init pigsty infrastructure on infra nodes
`infra-rm.yml`	Remove infrastructure components from infra nodes
`node.yml`	Init node for pigsty, tune node into desired status
`node-rm.yml`	Remove node from pigsty
`pgsql.yml`	Init HA PostgreSQL clusters, or adding new replicas
`pgsql-rm.yml`	Remove PostgreSQL cluster, or remove replicas
`pgsql-user.yml`	Add new business user to existing PostgreSQL cluster
`pgsql-db.yml`	Add new business database to existing PostgreSQL cluster
`pgsql-monitor.yml`	Monitor remote postgres instance with local exporters
`pgsql-migration.yml`	Generate Migration manual & scripts for existing PostgreSQL
`redis.yml`	Init redis cluster/node/instance
`redis-rm.yml`	Remove redis cluster/node/instance
`etcd.yml`	Init etcd cluster (required for patroni HA DCS)
`minio.yml`	Init minio cluster (optional for pgbackrest repo)
`cert.yml`	Issue cert with pigsty self-signed CA (e.g. for pg clients)
`docker.yml`	Install docker on nodes
`mongo.yml`	在节点上安装 Mongo/FerretDB

One-Pass Install

License

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty is open sourced under the AGPLv3 license, which is a copyleft license.

Summary

Pigsty use the AGPLv3 license, which is a strong copyleft license that requires you to also distribute the source code of your derivative works under the same license. If you distribute Pigsty, you must make the source code available under the same license, and you must make it clear that the source code is available.

Provisioning

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty runs on nodes, which are Bare Metals or Virtual Machines. You can prepare them manually, or using terraform & vagrant for provisioning.

Sandbox

Pigsty has a sandbox, which is a 4-node deployment with fixed IP addresses and other identifiers. Check demo.yml for details.

The sandbox consists of 4 nodes with fixed IP addresses: 10.10.10.10, 10.10.10.11, 10.10.10.12, 10.10.10.13.

There’s a primary singleton PostgreSQL cluster: pg-meta on the meta node, which can be used alone if you don’t care about PostgreSQL high availability.

Sponsorship

Mon, 01 Jan 0001 00:00:00 +0000

Investor

Pigsty is funded by MiraclePlus (formal YC China), S22 Batch.

Thanks to MiraclePlus and Dr.Qi’s support.

Sponsorship

Pigsty is a free & open-source software nurtured by the passion of PostgreSQL community members.

If our work has helped you, please consider sponsoring or supporting our project. Every penny counts, and advertisements are also a form of support:

Make Donation & Sponsor us.
Share your experiences and use cases of Pigsty through articles, lectures, and videos.
Allow us to mention your organization in “These users who use Pigsty”
Nominate/Recommend our project and services to your friends, colleagues, and clients in need.
Follow our WeChat Column and share technical articles with your friends.

Cost Analysis

Mon, 01 Jan 0001 00:00:00 +0000

Cost Reference

EC2	vCPU-Month	RDS	vCPU-Month
DHH’s self-hosted core-month price (192C 384G)	25.32	Junior open-source DBA reference salary	15K/person-month
IDC self-hosted data center (exclusive physical machine: 64C384G)	19.53	Intermediate open-source DBA reference salary	30K/person-month
IDC self-hosted data center (container, oversold 500%)	7	Senior open-source DBA reference salary	60K/person-month
UCloud Elastic Virtual Machine (8C16G, oversold)	25	ORACLE database license	10000
Alibaba Cloud Elastic Server 2x memory (exclusive without overselling)	107	Alibaba Cloud RDS PG 2x memory (exclusive)	260
Alibaba Cloud Elastic Server 4x memory (exclusive without overselling)	138	Alibaba Cloud RDS PG 4x memory (exclusive)	320
Alibaba Cloud Elastic Server 8x memory (exclusive without overselling)	180	Alibaba Cloud RDS PG 8x memory (exclusive)	410
AWS C5D.METAL 96C 200G (monthly without upfront)	100	AWS RDS PostgreSQL db.T2 (2x)	440

For instance, using RDS for PostgreSQL on AWS, the price for a 64C / 256GB db.m5.16xlarge RDS for one month is $25,817, which is equivalent to about 180,000 yuan per month. The monthly rent is enough for you to buy two servers with even better performance and set them up on your own. The rent-to-buy ratio doesn’t even last a month; renting for just over ten days is enough to buy the whole server for yourself.

Privacy Policy

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty Software

When you install the Pigsty software, if you use offline packages in a network-isolated environment, we will not receive any data about you.

If you choose to install online, then when downloading relevant software packages, our server or the servers of our cloud providers will automatically log the visiting machine’s IP address and/or hostname, as well as the name of the software package you downloaded, in the logs.

Security

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty already provides a secure-by-default authentication and access control model, which is sufficient for most scenarios.

But if you want to further strengthen the security of the system, the following suggestions are for your reference:

Confidentiality

Important Files

Secure your pigsty config inventory

pigsty.yml has highly sensitive information, including passwords, certificates, and keys.
You should limit access to admin/infra nodes, only accessible by the admin/dba users
Limit access to the git repo, if you are using git to manage your pigsty source.

Secure your CA private key and other certs

Access Control

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty has a battery-included access control model based on Role System and Privileges.

Role System

Pigsty has a default role system consist of four default roles and four default users：

Role name	Attributes	Member of	Description
`dbrole_readonly`	`NOLOGIN`		role for global read-only access
`dbrole_readwrite`	`NOLOGIN`	dbrole_readonly	role for global read-write access
`dbrole_admin`	`NOLOGIN`	pg_monitor,dbrole_readwrite	role for object creation
`dbrole_offline`	`NOLOGIN`		role for restricted read-only access
`postgres`	`SUPERUSER`		system superuser
`replicator`	`REPLICATION`	pg_monitor,dbrole_readonly	system replicator
`dbuser_dba`	`SUPERUSER`	dbrole_admin	pgsql admin user
`dbuser_monitor`		pg_monitor	pgsql monitor user

pg_default_roles:                 # default roles and users in postgres cluster
  - { name: dbrole_readonly  ,login: false ,comment: role for global read-only access     }
  - { name: dbrole_offline   ,login: false ,comment: role for restricted read-only access }
  - { name: dbrole_readwrite ,login: false ,roles: [dbrole_readonly] ,comment: role for global read-write access }
  - { name: dbrole_admin     ,login: false ,roles: [pg_monitor, dbrole_readwrite] ,comment: role for object creation }
  - { name: postgres     ,superuser: true  ,comment: system superuser }
  - { name: replicator ,replication: true  ,roles: [pg_monitor, dbrole_readonly] ,comment: system replicator }
  - { name: dbuser_dba   ,superuser: true  ,roles: [dbrole_admin]  ,pgbouncer: true ,pool_mode: session, pool_connlimit: 16 ,comment: pgsql admin user }
  - { name: dbuser_monitor ,roles: [pg_monitor] ,pgbouncer: true ,parameters: {log_min_duration_statement: 1000 } ,pool_mode: session ,pool_connlimit: 8 ,comment: pgsql monitor user }

Default Roles

There are four default roles in pigsty:

FAQ

Mon, 01 Jan 0001 00:00:00 +0000

How to install Docker ？

Install with the docker.yml playbook, targeting at any node managed by Pigsty.

./docker.yml -l <selector>

FAQ

Mon, 01 Jan 0001 00:00:00 +0000

What is the impact of ETCD failure?

ETCD availability is critical for the PGSQL cluster’s HA, which is guaranteed by using multiple nodes. With a 3-node ETCD cluster, if one node is down, the other two nodes can still function normally; and with a 5-node ETCD cluster, two-node failure can still be tolerated. If more than half of the ETCD nodes are down, the ETCD cluster and its service will be unavailable. Before Patroni 3.0, this could lead to a global PGSQL outage; all primary will be demoted and reject write requests.

FAQ

Mon, 01 Jan 0001 00:00:00 +0000

Which components are included in INFRA

Ansible for automation, deployment, and administration;
Nginx for exposing any WebUI service and serving the yum/apt repo;
Self-Signed CA for SSL/TLS certificates;
Prometheus for monitoring metrics
Grafana for monitoring/visualization
Loki for logging collection
AlertManager for alerts aggregation
Chronyd for NTP time sync on the admin node.
DNSMasq for DNS registration and resolution.
ETCD as DCS for PGSQL HA; (dedicated module)
PostgreSQL on meta nodes as CMDB; (optional)
Docker for stateless applications & tools (optional)

How to restore Prometheus targets

If you accidentally deleted the Prometheus targets dir, you can register monitoring targets to Prometheus again with the:

FAQ

Mon, 01 Jan 0001 00:00:00 +0000

Fail to launch multi-node / multi-driver MinIO cluster.

In Multi-Driver or Multi-Node mode, MinIO will refuse to start if the data dir is not a valid mount point.

Use mounted disks for MinIO data dir rather than some regular directory. You can use the regular directory only in the single node, single drive mode.

How to deploy a multi-node multi-drive MinIO cluster?

Check Create Multi-Node Multi-Driver MinIO Cluster

FAQ

Mon, 01 Jan 0001 00:00:00 +0000

Install mongosh

cat > /etc/yum.repos.d/mongo.repo <<EOF
[mongodb-org-7.0]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/7.0/x86_64/
gpgcheck=0
enabled=1
EOF

yum install -y mongodb-mongosh

# or just install via rpm & links
rpm -ivh https://mirrors.tuna.tsinghua.edu.cn/mongodb/yum/el7/RPMS/mongodb-mongosh-1.9.1.x86_64.rpm

FAQ

Mon, 01 Jan 0001 00:00:00 +0000

How to configure NTP service?

If NTP is not configured, use a public NTP service or sync time with the admin node.

If your nodes already have NTP configured, you can leave it there by setting node_ntp_enabled to false.

Otherwise, if you have Internet access, you can use public NTP services such as pool.ntp.org.

If you don’t have Internet access, at least you can sync time with the admin node with the following:

FAQ

Mon, 01 Jan 0001 00:00:00 +0000

ABORT due to existing redis instance

use redis_clean = true and redis_safeguard = false to force clean redis data

This happens when you run redis.yml to init a redis instance that is already running, and redis_clean is set to false.

If redis_clean is set to true (and the redis_safeguard is set to false, too), the redis.yml playbook will remove the existing redis instance and re-init it as a new one, which makes the redis.yml playbook fully idempotent.

FAQ

Mon, 01 Jan 0001 00:00:00 +0000

If you have any unlisted questions or suggestions, please create an Issue or ask the community for help.

How to Get the Pigsty Source Package?

Use the following command to install Pigsty with one click: bash -c "$(curl -fsSL https://get.pigsty.cc/install)" -- v2.7.0

This command will automatically download the latest stable version pigsty.tgz and extract it to the ~/pigsty directory. You can also manually download a specific version of the Pigsty source code from the following locations.

Glossary

Mon, 01 Jan 0001 00:00:00 +0000

Service

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty is a battery-included use PostgreSQL database distribution, a local-first alternative to RDS/cloud database services, allowing users to run a fully-featured local RDS service at a hardware cost of just a few core months. The software itself is completely open-source and free. If it has helped you, please consider sponsoring us.

Although Pigsty is designed to replace manual database operations with database autopilot software, even the best software can only solve some problems. There will always be some infrequent issues and various problems not limited to technology that require expert intervention. Therefore, we also offer professional service subscriptions.

Task & Tutorial

Mon, 18 Apr 2022 00:00:00 +0000

PGSQL Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Overview of the PGSQL module and key concepts

ER Diagram

Let’s get started with ER diagram. There are four types of core entities in Pigsty’s PGSQL module:

RELEASENOTE

Mon, 01 Jan 0001 00:00:00 +0000

Version	Time	Description	Release
v2.7.0	2024-05-20	Extension Overwhelming, new docker apps	v2.7.0
v2.6.0	2024-02-28	PG 16 as default version, ParadeDB & DuckDB	v2.6.0
v2.5.1	2023-12-01	Routine update, pg16 major extensions	v2.5.1
v2.5.0	2023-10-24	Ubuntu/Debian Support: bullseye, bookworm, jammy, focal	v2.5.0
v2.4.1	2023-09-24	Supabase/PostgresML support, graphql, jwt, pg_net, vault	v2.4.1
v2.4.0	2023-09-14	PG16, RDS Monitor, New Extensions	v2.4.0
v2.3.1	2023-09-01	PGVector with HNSW, PG16 RC1, Chinese Docs, Bug Fix	v2.3.1
v2.3.0	2023-08-20	PGSQL/REDIS Update, NODE VIP, Mongo/FerretDB, MYSQL Stub	v2.3.0
v2.2.0	2023-08-04	Dashboard & Provision overhaul, UOS compatibility	v2.2.0
v2.1.0	2023-06-10	PostgreSQL 12 ~ 16beta support	v2.1.0
v2.0.2	2023-03-31	Add pgvector support and fix MinIO CVE	v2.0.2
v2.0.1	2023-03-21	v2 Bug Fix, security enhance and bump grafana version	v2.0.1
v2.0.0	2023-02-28	Compatibility Security Maintainability Enhancement	v2.0.0
v1.5.1	2022-06-18	Grafana Security Hotfix	v1.5.1
v1.5.0	2022-05-31	Docker Applications	v1.5.0
v1.4.1	2022-04-20	Bug fix & Full translation of English documents.	v1.4.1
v1.4.0	2022-03-31	MatrixDB Support, Separated INFRA, NODES, PGSQL, REDIS	v1.4.0
v1.3.0	2021-11-30	PGCAT Overhaul & PGSQL Enhancement & Redis Support Beta	v1.3.0
v1.2.0	2021-11-03	Upgrade default Postgres to 14, monitoring existing pg	v1.2.0
v1.1.0	2021-10-12	HomePage, JupyterLab, PGWEB, Pev2 & Pgbadger	v1.1.0
v1.0.0	2021-07-26	v1 GA, Monitoring System Overhaul	v1.0.0
v0.9.0	2021-04-04	Pigsty GUI, CLI, Logging Integration	v0.9.0
v0.8.0	2021-03-28	Service Provision	v0.8.0
v0.7.0	2021-03-01	Monitor only deployment	v0.7.0
v0.6.0	2021-02-19	Architecture Enhancement	v0.6.0
v0.5.0	2021-01-07	Database Customize Template	v0.5.0
v0.4.0	2020-12-14	PostgreSQL 13 Support, Official Documentation	v0.4.0
v0.3.0	2020-10-22	Provisioning Solution GA	v0.3.0
v0.2.0	2020-07-10	PGSQL Monitoring v6 GA	v0.2.0
v0.1.0	2020-06-20	Validation on Testing Environment	v0.1.0
v0.0.5	2020-08-19	Offline Installation Mode	v0.0.5
v0.0.4	2020-07-27	Refactor playbooks into ansible roles	v0.0.4
v0.0.3	2020-06-22	Interface enhancement	v0.0.3
v0.0.2	2020-04-30	First Commit	v0.0.2
v0.0.1	2019-05-15	POC	v0.0.1

v2.7.0

Highlight