Prometheus Installation and Management

YMatrix also provides its own exporter, which can be seamlessly integrated into the Prometheus monitoring ecosystem.

It should be noted that YMatrix's exporter and corresponding Dashboard only include monitoring of the database itself, and do not include monitoring of operating system metrics. Monitoring of operating system-related metrics requires additional exporter support, such as node_exporter. The module layout is as follows: Module Layout Diagram

Deployment

The YMatrix exporter is included in the YMatrix installation package and can be activated for use. Then install and deploy node_exporter, and Prometheus.

1. YMatrix exporter activation

Create the matrixmgr database

createdb matrixmgr;

Connect to the matrixmgr database, create the matrixts and matrixmgr extensions, and activate the exporter

psql -d matrixmgr

matrixmgr=# CREATE EXTENSION matrixts;
matrixmgr=# CREATE EXTENSION matrixmgr;

matrixmgr=# SELECT mxmgr_init_exporter();

After successful activation, you will see a new schema named “exporter” under the matrixmgr database. The tables and views under this schema contain cluster monitoring and configuration information. Do not modify the definitions or contents of these tables and views.

This command will start matrixdb_exporter on all machines in the cluster.

Notes!
If the cluster has already started the old monitoring system, you need to shut down the old monitoring system first, otherwise it will fail. Shutdown method: SELECT mxmgr_remove_all(‘local’);

2.node_exporter Installation

node_exporter is used to monitor operating system-related metrics. Download the latest version of node_exporter from the official website. Here, we use 1.3.1 as an example (operated by the root user).

# Download node_exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz

# Extract the installation package
tar -xvf node_exporter-1.3.1.linux-amd64.tar.gz -C /usr/local

# Create a symbolic link
ln -s /usr/local/node_exporter-1.3.1.linux-amd64/ /usr/local/node_exporter

# Generate the systemctl configuration file
cat << EOF > /usr/lib/systemd/system/node_exporter.service
[Service]
User=root
Group=root
ExecStart=/usr/local/node_exporter/node_exporter

[Install]
WantedBy=multi-user.target

[Unit]
Description=node_exporter
After=network.target

EOF

# Start node_exporter
systemctl start node_exporter
systemctl status node_exporter
systemctl enable node_exporter

Notes!
node_exporter needs to be deployed on all hosts in the cluster, so the above steps must be performed on all hosts in the cluster.

3.Prometheus Installation

Prepare a host that can access the exporter port of all hosts in the cluster. It can be a Master or Standby Master node, or a separate machine (Linux, MacOS, Windows, etc.).

Install the latest version of Prometheus. The official download link is https://prometheus.io/download/.

The following commands describe the operation method using CentOS 7 as an example. For commands on other operating systems, please refer to the corresponding operating system user guide.

Download and install Prometheus (root privileges required)

Notes!
During the installation of Prometheus, you can choose whether to install the mxgate monitoring interface. The following example contains the mxgate monitoring installation code. If you do not want to install it, simply delete this section of code. See the example code comments for details.

# Download and install the package
wget https://github.com/prometheus/prometheus/releases/download/v2.36.1/prometheus-2.36.1.linux-amd64.tar.gz
# Extract to /usr/local and create a symbolic link
tar -xf ./prometheus-2.36.1.linux-amd64.tar.gz -C /usr/local
ln -s /usr/local/prometheus-2.36.1.linux-amd64/ /usr/local/prometheus
# Create the Prometheus user
useradd -s /sbin/nologin -M prometheus
# Create the data directory
mkdir /data/prometheus -p
# Change the directory owner and group
chown -R prometheus:prometheus /usr/local/prometheus/ 
chown -R prometheus:prometheus /data/prometheus/
# Configure systemctl
cat << EOF > /etc/systemd/system/prometheus.service 
 [Unit] 
 Description=Prometheus 
 Documentation=https://prometheus.io/ 
 After=network.target
 [Service] 
 Type=simple 
 User=prometheus 
 ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml --storage.tsdb.path=/data/prometheus
 Restart=on-failure 
 [Install] 
 WantedBy=multi-user.target
EOF
# Modify the Prometheus configuration file.
# Add the IP and port of matrixdb and node exporter to scrape_configs.
# (The default port for matrixdb_exporter is 9273, and the default port for node_exporter is 9100):
# It should also be noted that the monitoring graphs of matrixdb and mxgate are filtered by the matrixdb_cluster variable.
# so you need to add relabel_configs configuration to replace the job label with matrixdb_cluster,
# and fill in the cluster name in replacement
# You can refer to the following configuration:
scrape_configs:
  - job_name: “matrixdb_exporter”
    relabel_configs:
      - source_labels: [‘job’]
        regex: .*
        target_label: matrixdb_cluster
        replacement: cluster1
        action : replace
    static_configs:
      - targets: [“localhost:9273”]
  - job_name: “node_exporter”
    static_configs:
      - targets: [“localhost:9100”]
# When the cluster contains multiple hosts, you need to add the IP port of each host to the targets array.
# For example:
scrape_configs:
  - job_name: “matrixdb_exporter”
    relabel_configs:
      - source_labels: [‘job’]
        regex: .*
        target_label: matrixdb_cluster
        replacement: cluster1
        action : replace
    static_configs:
    - targets: [“192.168.0.1:9273”, “192.168.0.2:9273”, “192.168.0.3:9273”]
  - job_name: “node_exporter”
    static_configs:
    - targets: [“192.168.0.1:9100”, “192.168.0.2:9100”, “192.168.0.3:9100”]
# If you want to deploy mxgate monitoring, you need to add the following segment here and then restart Prometheus. This segment of code is not mandatory.
  - job_name: “gate_exporter”
    relabel_configs:
      - source_labels: [‘job’]
        regex: .*
        target_label: matrixdb_cluster
        replacement: cluster1
        action : replace
    static_configs:
      - targets: [“192.168.0.1:9275”]
# Notes: Pay attention to indentation when editing the yaml file, otherwise a syntax error will occur, preventing startup.
# Start Prometheus.
systemctl start prometheus
systemctl status prometheus
systemctl enable prometheus

After startup, you can access the WebUI through the UI interface provided by Prometheus and view the running status at http://IP:9090/

In the Prometheus main panel, select Status->Targets View Data Source 1

You will see that the matrixdb_exporter and node_exporter you just configured, as well as Prometheus's own exporter, are all in the UP state, indicating that the monitoring deployment was successful. View Data Source 2

Management

After enabling cluster status collection, each host will run the collection service, and the relevant logs will be stored in the /var/log/matrixdb directory.

If you restart YMatrix, or restart the server and then start YMatrix, the YMatrix exporter will also start automatically without manual intervention.

To disable the matrixdb exporter service, connect to the matrixmgr database and execute the mxmgr_remove_exporter command:

psql -d matrixmgr

matrixmgr=# SELECT mxmgr_remove_exporter();

To re-enable data collection, connect to the matrixmgr database and execute the mxmgr_deploy_exporter command:

matrixmgr=# SELECT mxmgr_deploy_exporter();

Notes!
mxmgr_remove_exporter will only shut down matrixdb exporter. node_exporter, Grafana, and Prometheus need to be shut down separately.

Upgrade

Upgrade the old monitoring system to the new monitoring system that connects to the Prometheus ecosystem.

First, complete the installation and deployment of Prometheus, and upgrade Grafana to the latest version:

# First, shut down the old monitoring system
psql -d matrixmgr
matrixmgr=# SELECT mxmgr_remove_all(‘local’);

# Initialize and start the new monitoring system
matrixmgr=# SELECT mxmgr_init_exporter();

Notes!
The old and new monitoring systems can also be deployed at the same time, that is, the new monitoring system can be started without shutting down the old monitoring system, but there will be some redundancy.