Skip to content

Version v2.7.0 of the documentation is no longer actively maintained. The site that you are currently viewing is an archived snapshot. For up-to-date documentation, see the latest version.

Module: INFRA

Independent modules which providers infrastructure services: NTP, DNS, and the modern observability stack ——Grafana & Prometheus

Pigsty has a battery-included, production-ready INFRA module, to provide ultimate observability.

Configuration | Administration | Playbook | Dashboard | Parameter

Overview

Each Pigsty deployment requires a set of infrastructure components to work properly. which including:

Component Port Domain Description
Nginx 80 h.pigsty Web Service Portal (YUM/APT Repo)
AlertManager 9093 a.pigsty Alert Aggregation and delivery
Prometheus 9090 p.pigsty Monitoring Time Series Database
Grafana 3000 g.pigsty Visualization Platform
Loki 3100 - Logging Collection Server
PushGateway 9091 - Collect One-Time Job Metrics
BlackboxExporter 9115 - Blackbox Probing
Dnsmasq 53 - DNS Server
Chronyd 123 - NTP Time Server
PostgreSQL 5432 - Pigsty CMDB & default database
Ansible - - Run playbooks

Pigsty will set up these components for you on infra nodes. You can expose them to the outside world by configuring the infra_portal parameter.

infra_portal:  # domain names and upstream servers
  home         : { domain: h.pigsty }
  grafana      : { domain: g.pigsty ,endpoint: "${admin_ip}:3000" , websocket: true }
  prometheus   : { domain: p.pigsty ,endpoint: "${admin_ip}:9090" }
  alertmanager : { domain: a.pigsty ,endpoint: "${admin_ip}:9093" }
  blackbox     : { endpoint: "${admin_ip}:9115" }
  loki         : { endpoint: "${admin_ip}:3100" }
  #minio        : { domain: sss.pigsty  ,endpoint: "${admin_ip}:9001" ,scheme: https ,websocket: true }

pigsty-arch.jpg

Configuration

To define an infra cluster, use the hard-coded group name infra in your inventory file.

You can use multiple nodes to deploy INFRA module, but at least one is required. You have to assign a unique infra_seq to each node.

# Single infra node
infra: { hosts: { 10.10.10.10: { infra_seq: 1 } }}

# Two INFRA node
infra:
  hosts:
    10.10.10.10: { infra_seq: 1 }
    10.10.10.11: { infra_seq: 2 }

Then you can init INFRA module with infra.yml playbook.

Administration

Here are some administration tasks related to INFRA module:

Install/Remove Infra Module 

./infra.yml     # install infra/node module on `infra` group
./infra-rm.yml  # remove infra module from `infra` group

Manage Local Software Repo 

./infra.yml -t repo             # setup local yum/apt repo

./infra.yml -t repo_dir         # create repo directory
./infra.yml -t repo_check       # check repo exists
./infra.yml -t repo_prepare     # use existing repo if exists
./infra.yml -t repo_build       # build repo from upstream if not exists
./infra.yml   -t repo_upstream  # handle upstream repo files in /etc/yum.repos.d or /etc/apt/sources.list.d
./infra.yml   -t repo_url_pkg   # download packages from internet defined by repo_url_packages
./infra.yml   -t repo_cache     # make upstream yum/apt cache
./infra.yml   -t repo_boot_pkg  # install bootstrap pkg such as createrepo_c,yum-utils,... (or dpkg-dev in debian/ubuntu)
./infra.yml   -t repo_pkg       # download packages & dependencies from upstream repo
./infra.yml   -t repo_create    # create a local yum repo with createrepo_c & modifyrepo_c
./infra.yml   -t repo_use       # add newly built repo
./infra.yml -t repo_nginx       # launch a nginx for repo if no nginx is serving

Manage Infra Component

您可以使用以下剧本子任务，管理 Infra节点 上的各个基础设施组件

./infra.yml -t infra_env      : env_dir, env_pg, env_var
./infra.yml -t infra_pkg      : infra_pkg, infra_pkg_pip
./infra.yml -t infra_user     : setup infra os user group
./infra.yml -t infra_cert     : issue cert for infra components
./infra.yml -t dns            : dns_config, dns_record, dns_launch
./infra.yml -t nginx          : nginx_config, nginx_cert, nginx_static, nginx_launch, nginx_exporter
./infra.yml -t prometheus     : prometheus_clean, prometheus_dir, prometheus_config, prometheus_launch, prometheus_reload
./infra.yml -t alertmanager   : alertmanager_config, alertmanager_launch
./infra.yml -t pushgateway    : pushgateway_config, pushgateway_launch
./infra.yml -t blackbox       : blackbox_config, blackbox_launch
./infra.yml -t grafana        : grafana_clean, grafana_config, grafana_plugin, grafana_launch, grafana_provision
./infra.yml -t loki           : loki clean, loki_dir, loki_config, loki_launch
./infra.yml -t infra_register : register infra components to prometheus
./infra.yml -t nginx_index                        # render Nginx homepage
./infra.yml -t nginx_config,nginx_launch          # render Nginx upstream server config
./infra.yml -t prometheus_conf,prometheus_reload  # render Prometheus main config and reload
./infra.yml -t prometheus_rule,prometheus_reload  # copy Prometheus rules & alert definition and reload
./infra.yml -t grafana_plugin                     # download Grafana plugins from the Internet

Playbook

  • install.yml : Install Pigsty on all nodes in one-pass
  • infra.yml : Init pigsty infrastructure on infra nodes
  • infra-rm.yml : Remove infrastructure components from infra nodes

asciicast

infra.yml

The playbook infra.yml will init pigsty infrastructure on infra nodes.

It will also install NODE module on infra nodes too.

Here are available subtasks:

# ca            : create self-signed CA on localhost files/pki
#   - ca_dir        : create CA directory
#   - ca_private    : generate ca private key: files/pki/ca/ca.key
#   - ca_cert       : signing ca cert: files/pki/ca/ca.crt
#
# id            : generate node identity
#
# repo          : bootstrap a local yum repo from internet or offline packages
#   - repo_dir      : create repo directory
#   - repo_check    : check repo exists
#   - repo_prepare  : use existing repo if exists
#   - repo_build    : build repo from upstream if not exists
#     - repo_upstream    : handle upstream repo files in /etc/yum.repos.d
#       - repo_remove    : remove existing repo file if repo_remove == true
#       - repo_add       : add upstream repo files to /etc/yum.repos.d
#     - repo_url_pkg     : download packages from internet defined by repo_url_packages
#     - repo_cache       : make upstream yum cache with yum makecache
#     - repo_boot_pkg    : install bootstrap pkg such as createrepo_c,yum-utils,...
#     - repo_pkg         : download packages & dependencies from upstream repo
#     - repo_create      : create a local yum repo with createrepo_c & modifyrepo_c
#     - repo_use         : add newly built repo into /etc/yum.repos.d
#   - repo_nginx    : launch a nginx for repo if no nginx is serving
#
# node/haproxy/docker/monitor : setup infra node as a common node (check node.yml)
#   - node_name, node_hosts, node_resolv, node_firewall, node_ca, node_repo, node_pkg
#   - node_feature, node_kernel, node_tune, node_sysctl, node_profile, node_ulimit
#   - node_data, node_admin, node_timezone, node_ntp, node_crontab, node_vip
#   - haproxy_install, haproxy_config, haproxy_launch, haproxy_reload
#   - docker_install, docker_admin, docker_config, docker_launch, docker_image
#   - haproxy_register, node_exporter, node_register, promtail
#
# infra         : setup infra components
#   - infra_env      : env_dir, env_pg, env_var
#   - infra_pkg      : infra_pkg, infra_pkg_pip
#   - infra_user     : setup infra os user group
#   - infra_cert     : issue cert for infra components
#   - dns            : dns_config, dns_record, dns_launch
#   - nginx          : nginx_config, nginx_cert, nginx_static, nginx_launch, nginx_exporter
#   - prometheus     : prometheus_clean, prometheus_dir, prometheus_config, prometheus_launch, prometheus_reload
#   - alertmanager   : alertmanager_config, alertmanager_launch
#   - pushgateway    : pushgateway_config, pushgateway_launch
#   - blackbox       : blackbox_config, blackbox_launch
#   - grafana        : grafana_clean, grafana_config, grafana_plugin, grafana_launch, grafana_provision
#   - loki           : loki clean, loki_dir, loki_config, loki_launch
#   - infra_register : register infra components to prometheus

asciicast

infra-rm.yml

The playbook infra-rm.yml will remove infrastructure components from infra nodes

./infra-rm.yml               # remove INFRA module
./infra-rm.yml -t service    # stop INFRA services
./infra-rm.yml -t data       # remove INFRA data
./infra-rm.yml -t package    # uninstall INFRA packages

install.yml

The playbook install.yml will install Pigsty on all node in one-pass.

Check Playbook: One-Pass Install for details.

Dashboard

Pigsty Home : Home dashboard for pigsty’s grafana

Pigsty Home Dashboard

pigsty.jpg

INFRA Overview : Overview of all infra components

INFRA Overview Dashboard

infra-overview.jpg

Nginx Overview : Nginx metrics & logs

Nginx Overview Dashboard

nginx-overview.jpg

Grafana Overview: Grafana metrics & logs

Grafana Overview Dashboard

grafana-overview.jpg

Prometheus Overview: Prometheus metrics & logs

Prometheus Overview Dashboard

prometheus-overview.jpg

Loki Overview: Loki metrics & logs

Loki Overview Dashboard

loki-overview.jpg

Logs Instance: Logs for a single instance

Logs Instance Dashboard

logs-instance.jpg

Logs Overview: Overview of all logs

Logs Overview Dashboard

logs-overview.jpg

CMDB Overview: CMDB visualization

CMDB Overview Dashboard

cmdb-overview.jpg

ETCD Overview: etcd metrics & logs

ETCD Overview Dashboard

etcd-overview.jpg

Parameter

API Reference for INFRA module:

  • META: infra meta data
  • CA: self-signed CA
  • INFRA_ID : Portals and identity
  • REPO: local yum/atp repo
  • INFRA_PACKAGE : packages to be installed
  • NGINX : nginx web server
  • DNS: dnsmasq nameserver
  • PROMETHEUS : prometheus, alertmanager, pushgateway & blackbox_exporter
  • GRAFANA : Grafana, the visualization platform
  • LOKI : Loki, the logging server
Parameters
Parameter Section Type Level Comment
version META string G pigsty version string
admin_ip META ip G admin node ip address
region META enum G upstream mirror region: default,china,europe
proxy_env META dict G global proxy env when downloading packages
ca_method CA enum G create,recreate,copy, create by default
ca_cn CA string G ca common name, fixed as pigsty-ca
cert_validity CA interval G cert validity, 20 years by default
infra_seq INFRA_ID int I infra node identity, REQUIRED
infra_portal INFRA_ID dict G infra services exposed via portal
repo_enabled REPO bool G/I create a yum/apt repo on this infra node?
repo_home REPO path G repo home dir, /www by default
repo_name REPO string G repo name, pigsty by default
repo_endpoint REPO url G access point to this repo by domain or ip:port
repo_remove REPO bool G/A remove existing upstream repo
repo_modules REPO string G/A which repo modules are installed in repo_upstream
repo_upstream REPO upstream[] G where to download upstream packages
repo_packages REPO string[] G which packages to be included
repo_url_packages REPO string[] G extra packages from url
infra_packages INFRA_PACKAGE string[] G packages to be installed on infra nodes
infra_packages_pip INFRA_PACKAGE string G pip installed packages for infra nodes
nginx_enabled NGINX bool G/I enable nginx on this infra node?
nginx_exporter_enabled NGINX bool G/I enable nginx_exporter on this infra node?
nginx_sslmode NGINX enum G nginx ssl mode? disable,enable,enforce
nginx_home NGINX path G nginx content dir, /www by default
nginx_port NGINX port G nginx listen port, 80 by default
nginx_ssl_port NGINX port G nginx ssl listen port, 443 by default
nginx_navbar NGINX index[] G nginx index page navigation links
dns_enabled DNS bool G/I setup dnsmasq on this infra node?
dns_port DNS port G dns server listen port, 53 by default
dns_records DNS string[] G dynamic dns records resolved by dnsmasq
prometheus_enabled PROMETHEUS bool G/I enable prometheus on this infra node?
prometheus_clean PROMETHEUS bool G/A clean prometheus data during init?
prometheus_data PROMETHEUS path G prometheus data dir, /data/prometheus by default
prometheus_sd_dir PROMETHEUS path G prometheus file service discovery directory
prometheus_sd_interval PROMETHEUS interval G prometheus target refresh interval, 5s by default
prometheus_scrape_interval PROMETHEUS interval G prometheus scrape & eval interval, 10s by default
prometheus_scrape_timeout PROMETHEUS interval G prometheus global scrape timeout, 8s by default
prometheus_options PROMETHEUS arg G prometheus extra server options
pushgateway_enabled PROMETHEUS bool G/I setup pushgateway on this infra node?
pushgateway_options PROMETHEUS arg G pushgateway extra server options
blackbox_enabled PROMETHEUS bool G/I setup blackbox_exporter on this infra node?
blackbox_options PROMETHEUS arg G blackbox_exporter extra server options
alertmanager_enabled PROMETHEUS bool G/I setup alertmanager on this infra node?
alertmanager_options PROMETHEUS arg G alertmanager extra server options
exporter_metrics_path PROMETHEUS path G exporter metric path, /metrics by default
exporter_install PROMETHEUS enum G how to install exporter? none,yum,binary
exporter_repo_url PROMETHEUS url G exporter repo file url if install exporter via yum
grafana_enabled GRAFANA bool G/I enable grafana on this infra node?
grafana_clean GRAFANA bool G/A clean grafana data during init?
grafana_admin_username GRAFANA username G grafana admin username, admin by default
grafana_admin_password GRAFANA password G grafana admin password, pigsty by default
grafana_plugin_cache GRAFANA path G path to grafana plugins cache tarball
grafana_plugin_list GRAFANA string[] G grafana plugins to be downloaded with grafana-cli
loki_enabled LOKI bool G/I enable loki on this infra node?
loki_clean LOKI bool G/A whether remove existing loki data?
loki_data LOKI path G loki data dir, /data/loki by default
loki_retention LOKI interval G loki log retention period, 15d by default
Metrics

Pigsty INFRA module metric list

FAQ

Pigsty INFRA module frequently asked questions