Admin on PIGSTY

Nginx: Expose Web Service

Mon, 18 Apr 2022 00:00:00 +0000

Pigsty will install Nginx on INFRA Node, as a Web service proxy.

Nginx is the access entry for all WebUI services of Pigsty, and it defaults to the use the 80/443 port on INFRA nodes.

Pigsty provides a global parameter infra_portal to configure Nginx proxy rules and corresponding upstream services.

infra_portal:  # domain names and upstream servers
  home         : { domain: h.pigsty }
  grafana      : { domain: g.pigsty ,endpoint: "${admin_ip}:3000" , websocket: true }
  prometheus   : { domain: p.pigsty ,endpoint: "${admin_ip}:9090" }
  alertmanager : { domain: a.pigsty ,endpoint: "${admin_ip}:9093" }
  blackbox     : { endpoint: "${admin_ip}:9115" }
  loki         : { endpoint: "${admin_ip}:3100" }
  #minio        : { domain: sss.pigsty  ,endpoint: "${admin_ip}:9001" ,scheme: https ,websocket: true }

If you access Nginx directly through the ip:port, it will route to h.pigsty, which is the Pigsty homepage (/www/ directory, served as software repo).

Compatibility

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Pigsty recommends using Linux kernel, amd64 arch, and RockyLinux 8.9, Debian 12 or Ubuntu 22.04 as base OS.

Kernel Architecture Compatibility: Linux kernel, amd64 architecture (x86_64)

EL Distribution Support: EL7, EL8, EL9; (RHEL, Rocky, CentOS, Alma, Oracle, Anolis,…)

Debian Distribution Support: Ubuntu 22.04 jammy, 20.04 focal; Debian 12 bookworm and 11 bullseye.

Pigsty does not use any virtualization or containerization technologies, running directly on the bare OS. Supported operating systems include EL 7/8/9 (RHEL, Rocky, CentOS, Alma, Oracle, Anolis,…), Ubuntu 20.04 / 22.04 & Debian 11/12. EL is our long-term supported OS, while support for Ubuntu/Debian systems was introduced in the recent v2.5 version. The main difference between EL and Debian distributions is the significant variation in package names, as well as the default availability of PostgreSQL extensions.

Getting Started

Mon, 01 Jan 0001 00:00:00 +0000

Docker: Container Support & Proxy

Mon, 18 Apr 2022 00:00:00 +0000

Pigsty has a DOCKER module, which provides a set of playbooks to install and manage Docker on the target nodes.

This document will guide you through how to enable Docker support in Pigsty, and how to configure a proxy server for DockerHub.

Install Docker

To install docker on specified nodes, you can use the docker.yml playbook:

./docker.yml -l <ip|group|cls>

That’s it.

Preparation

Mon, 01 Jan 0001 00:00:00 +0000

Node

Pigsty supports the Linux kernel and x86_64/amd64 arch, applicable to any node.

A “node” refers to a resource that is SSH accessible and offers a bare OS environment, such as a physical machine, a virtual machine, or an OS container equipped with systemd and sshd.

Deploying Pigsty requires at least 1 node. The minimum spec requirement is 1C1G, but it is recommended to use at least 2C4G, with no upper limit: parameters will automatically optimize and adapt.

Use PostgreSQL as Grafana Backend

Tue, 18 May 2021 00:00:00 +0000

Grafana uses SQLite for its metadata by default. Pigsty v2.7 can provision a PostgreSQL user and database for Grafana, after which Grafana can be pointed at that database manually.

v2.7 scope

The old grafana_database and grafana_pgurl Pigsty parameters were already obsolete in v2.7.0. The release’s grafana.ini.j2 does not manage a PostgreSQL backend URL, so re-running the grafana_config task can overwrite a manual edit. Keep the change under your own configuration management.

Provision the database

Add the user and database to a PGSQL cluster definition:

Use PostgreSQL as Ansible Inventory CMDB

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty v2.7 normally reads its Ansible inventory from pigsty.yml. The release also ships three scripts that can load that inventory into the pigsty schema of the meta database and switch Ansible to a generated dynamic inventory.

The implementation in this page is tied to the v2.7.0 objects bin/inventory_load, bin/inventory_cmdb, and bin/inventory_conf.

Prerequisites

The CMDB schema is created from files/cmdb.sql when the default pg-meta cluster is provisioned. Complete infra.yml and the pg-meta database initialization first, then verify that the meta service is reachable.

Planning

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty v2.7 planning starts by selecting a deployment topology and preparing the matching hosts, operating systems, network, and administrator access.

Planning Checklist

Choose a single-node, monitoring-only, or highly available topology from the provisioning guide and configuration examples.
Verify CPU architecture, operating-system release, storage, and network requirements in Prepare and the compatibility matrix.
Reserve stable node addresses, hostnames, required ports, and passwordless administrator SSH/sudo access before installation.
Decide between online and offline installation and prepare the corresponding software source or offline package.
Review the inventory in Configuration and the security checklist before running playbooks.

The exact cluster, user, database, service, and backup settings remain in the linked v2.7 pages; this page is only the historical planning entry point.

Bind a L2 VIP to Node Cluster with Keepalived

Mon, 01 Jan 0001 00:00:00 +0000

You can bind an optional L2 VIP on a node cluster with vip_enabled.

proxy:
  hosts:
    10.10.10.29: { nodename: proxy-1 }
    10.10.10.30: { nodename: proxy-2 } # , vip_role: master }
  vars:
    node_cluster: proxy
    vip_enabled: true
    vip_vrid: 128
    vip_address: 10.10.10.99
    vip_interface: eth1

./node.yml -l proxy -t node_vip     # enable for the first time
./node.yml -l proxy -t vip_refresh  # refresh vip config (e.g. designated master)

Administration

Mon, 01 Jan 0001 00:00:00 +0000

How to maintain an existing PostgreSQL cluster with Pigsty?

Here are some SOP for common pgsql admin tasks

Case 1: Create Cluster
Case 2: Create User
Case 3: Create Database
Case 4: Reload Service
Case 5: Reload HBARule
Case 6: Config Cluster
Case 7: Append Replica
Case 8: Remove Replica
Case 9: Remove Cluster
Case 10: Switchover
Case 11: Backup Cluster
Case 12: Restore Cluster
Case 13: Adding Packages
Case 14: Install Extension
Case 15: Minor Upgrade
Case 16: Major Upgrade

Cheatsheet

PGSQL playbooks and shortcuts:

Bind a L2 VIP to PostgreSQL Primary with VIP-Manager

Mon, 01 Jan 0001 00:00:00 +0000

You can define an OPTIONAL L2 VIP on a PostgreSQL cluster, provided that all nodes in the cluster are in the same L2 network.

This VIP works on Master-Backup mode and always points to the node where the primary instance of the database cluster is located.

This VIP is managed by the VIP-Manager, which reads the Leader Key written by Patroni from DCS (etcd) to determine whether it is the master.

Backup & PITR

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty uses pgBackRest for PITR backup & restore.

In the case of a hardware failure, a physical replica failover could be the best choice. Whereas for data corruption scenarios (whether machine or human errors), Point-in-Time Recovery (PITR) is often more appropriate.

Backup

Use the following command to perform the backup:

# stanza name = {{ pg_cluster }} by default
pgbackrest --stanza=${stanza} --type=full|diff|incr backup

# you can also use the following command in pigsty (/pg/bin/pg-backup)
pg-backup       # make a backup, incr, or full backup if necessary
pg-backup full  # make a full backup
pg-backup diff  # make a differential backup
pg-backup incr  # make a incremental backup

Use the following command to print backup info:

Migration

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty has a built-in playbook pgsql-migration.yml to perform online database migration based on logical replication.

With proper automation, the downtime could be minimized to several seconds. But beware that logical replication requires PostgreSQL 10+ to work. You can still use the facility here and use a pg_dump | psql instead of logical replication.

Define a Migration Task

You have to create a migration task definition file to use this playbook.

Point-in-Time Recovery

Mon, 01 Jan 0001 00:00:00 +0000

Overview

You can roll back your cluster to any point in time, avoiding data loss caused by software defects and human errors.

Pigsty’s PostgreSQL clusters come with an automatically configured Point in Time Recovery (PITR) solution, based on the backup component pgBackRest and the optional object storage repository MinIO.

High Availability solutions can address hardware failures, but they are powerless against data deletions/overwrites/deletions caused by software defects and human errors. For such scenarios, Pigsty offers an out-of-the-box Point in Time Recovery (PITR) capability, enabled by default without any additional configuration.

Provisioning

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty runs on nodes, which are Bare Metals or Virtual Machines. You can prepare them manually, or using terraform & vagrant for provisioning.

Sandbox

Pigsty has a sandbox, which is a 4-node deployment with fixed IP addresses and other identifiers. Check demo.yml for details.

The sandbox consists of 4 nodes with fixed IP addresses: 10.10.10.10, 10.10.10.11, 10.10.10.12, 10.10.10.13.

There’s a primary singleton PostgreSQL cluster: pg-meta on the meta node, which can be used alone if you don’t care about PostgreSQL high availability.

Services & Access

Mon, 01 Jan 0001 00:00:00 +0000

Personal User

Service is meaningless to personal users. You can access the database with raw IP address or whatever method you like.

psql postgres://dbuser_dba:DBUser.DBA@10.10.10.10/meta     # dbsu direct connect
psql postgres://dbuser_meta:DBUser.Meta@10.10.10.10/meta   # default business admin user
psql postgres://dbuser_view:DBUser.View@pg-meta/meta       # default read-only user

Service Overview

We utilize a PostgreSQL database cluster based on replication in real-world production environments. Within the cluster, only one instance is the leader (primary) that can accept writes. Other instances (replicas) continuously fetch WAL from the leader to stay synchronized. Additionally, replicas can handle read-only queries and offload the primary in read-heavy, write-light scenarios. Thus, distinguishing between write and read-only requests is a common practice.

ByteBase: PG Schema Migration

Fri, 20 May 2022 00:00:00 +0000

ByteBase

ByteBase is a database schema change management tool, which is a tool for database schema changes. The following command will start a ByteBase on the meta node 8887 port by default.

mkdir -p /data/bytebase/data;
docker run --init --name bytebase --restart always --detach --publish 8887:8887 --volume /data/bytebase/data:/var/opt/bytebase \
    bytebase/bytebase:latest --data /var/opt/bytebase --host http://ddl.pigsty --port 8887

访问 http://10.10.10.10:8887/ 或 http://ddl.pigsty 即可使用 ByteBase，您需要依次创建项目、环境、实例、数据库，即可开始进行模式变更。公开Demo地址： http://ddl.pigsty.cc

Task & Tutorial

Mon, 18 Apr 2022 00:00:00 +0000

PGAdmin4: PG Admin GUI Tool

Mon, 25 Apr 2022 00:00:00 +0000

pgAdmin4 is a useful PostgreSQL management tool. Execute the following command to launch the pgadmin service on the admin node:

cd ~/pigsty/app/pgadmin ; docker compose up -d

The default port for pgadmin is 8885, and you can access it through the following address:

http://adm.pigsty

Demo

Public Demo：http://adm.pigsty.cc。

Credentials: admin@pigsty.cc / pigsty。

TL; DR

cd ~/pigsty/app/pgadmin   # enter docker compose dir
make up                   # launch pgadmin container
make conf view            # load pigsty server list

Shortcuts:

SchemaSPY: Dump & Visualize PG Schema

Sat, 21 May 2022 00:00:00 +0000

Use the following docker command to generate a database schema report, using CMDB as an example:

docker run -v /www/schema/pg-meta/meta/pigsty:/output andrewjones/schemaspy-postgres:latest -host 10.10.10.10 -port 5432 -u dbuser_dba -p DBUser.DBA -db meta -s pigsty

Then visit http://h.pigsty/schema/pg-meta/meta/pigsty to access the schema report.

PGWeb: Browser-based PG Client

Fri, 18 Mar 2022 00:00:00 +0000

PGWEB: https://github.com/sosedoff/pgweb

Simple web-based and cross-platform PostgreSQL database explorer.

Public Demo: http://cli.pigsty.cc

TL; DR

cd ~/pigsty/app/pgweb ; make up

Visit http://cli.pigsty or http://10.10.10.10:8886

Try connecting with example URLs:

postgres://dbuser_meta:DBUser.Meta@10.10.10.10:5432/meta?sslmode=disable
postgres://test:test@10.10.10.11:5432/test?sslmode=disable

make up         # pull up pgweb with docker compose
make run        # launch pgweb with docker
make view       # print pgweb access point
make log        # tail -f pgweb logs
make info       # introspect pgweb with jq
make stop       # stop pgweb container
make clean      # remove pgweb container
make pull       # pull latest pgweb image
make rmi        # remove pgweb image
make save       # save pgweb image to /tmp/docker/pgweb.tgz
make load       # load pgweb image from /tmp/docker/pgweb.tgz

快捷方式

make up         # pull up pgweb with docker-compose
make run        # launch pgweb with docker
make view       # print pgweb access point
make log        # tail -f pgweb logs
make info       # introspect pgweb with jq
make stop       # stop pgweb container
make clean      # remove pgweb container
make pull       # pull latest pgweb image
make rmi        # remove pgweb image
make save       # save pgweb image to /tmp/pgweb.tgz
make load       # load pgweb image from /tmp