Grafana's alerting feature detects whether monitored metrics meet defined threshold conditions, based on configured monitoring rules. It helps notify users and quickly identify or resolve issues, ensuring your MatrixDB cluster remains secure and stable.
This section describes how to configure alert components and manage their usage. MatrixDB supports alerting on the following metrics:
Four notification methods are available for each metric. You may choose one or combine multiple methods:
To use alerting, you must first deploy and enable monitoring. For detailed steps, see: Monitor Deployment
After deployment, you will see an interface similar to the following:
Before importing predefined alert dashboards, you must copy the alert.json file from the server to your local machine. Similar to importing dashboard.json and database.json in Grafana, the process involves locating the file on the server, copying it, and uploading it locally. Follow these steps:
Log in to the server and switch to the mxadmin user. Locate the alert.json file at the path shown in the image. You can use the cd command, find command, or any other method you prefer.
[mxadmin@mdw ~]$ cd /usr/local/matrixdb/share/doc/postgresql/extension
##or
[mxadmin@mdw ~]$ find /usr/local/matrixdb/share/doc/postgresql/extension -name alert.jsonNext, use the scp command to copy the file to your local machine. Permission issues may arise, so consider first copying the file to the shared /tmp/ directory, then transferring it from /tmp/ to your local system.
Note!
When copying from/tmp/, ensure you switch users appropriately to avoid permission errors.
[mxadmin@mdw]$ scp mxadmin@<server IP>:/usr/local/matrixdb/share/doc/postgresql/extension/"alert.json" mxadmin@<server IP>:/tmp/
~ scp waq@<server IP>:/tmp/"alert.json" /Users/akkepler/workplace/GrafanaFinally, open your local folder or use the command line to verify the file was copied successfully. After confirmation, modify all instances of the variable $host in the local alert.json file to include all actual hostnames in your cluster. For example, if your cluster consists of one master and two segments named mdw, sdw1, and sdw2, replace every occurrence of $host with 'mdw','sdw1','sdw2'. Once modified, import the file into Grafana.
Note!
Thedashboard.jsonanddatabase.jsonfiles uploaded to your Grafana interface must also have their variables modified, and they must be imported beforealert.json. Indashboard.json, replace all${cluster}withlocaland$hostwith the actual hostnames. Indatabase.json, only replace${cluster}.
The alert dashboard functions the same as the monitoring dashboard and can be switched between using the dropdown menu located in the upper-right corner of each panel.
First, set up notification channels (Notification Channels). The entry point is shown below:
MatrixDB includes built-in SMS alerting via Alibaba Cloud SMS service, which is recommended. When an alert is triggered, a message is sent to your configured phone number(s). Bulk sending is supported. Before using this feature, you must apply for and configure Alibaba Cloud SMS service. For details, refer to:
Alibaba Cloud Documentation: Quick Start for Domestic SMS
Alternatively, you can implement custom SMS alerts using webhooks by programming your own webhook logic. See Grafana Alert Documentation: Webhook Notifications for more information.
Create an alert.yaml file under /etc/matrixdb/:
# aliyun service config
aliyun_service:
access_key_id: "your access_key_id"
access_key_secret: " access_key_secret"
signature: "signature"
sms_template_code: "SMS_123445678"Here:
access_key_id and access_key_secret are issued after activating Alibaba Cloud services.signature is the approved signature text registered in the Alibaba Cloud SMS console.sms_template_code is the template ID starting with "SMS_", approved in the Alibaba Cloud SMS console. The template content can be viewed on the Alibaba Cloud platform. Use ${name} as the variable placeholder for the alert message.Example template:
尊敬的管理员您好，贵公司的数据库系统发生${name}报警，请登陆网站查看详情，并及时处理。As shown:
We recommend naming the channel using the following format:
Aliyun Batch Short Message - For SuperAdministrators
"SuperAdministrators" represents a group of recipients. To define another group, create a separate rule. Note that the phoneNumbers parameter is case-sensitive. Multiple numbers can be specified, separated by commas:
http://
: /api/alert/batch-sms?phoneNumbers=18311111111,13811111111
In addition to SMS, MatrixDB supports voice call alerts via Alibaba Cloud’s voice service. When triggered, a call is made to your configured number(s). Bulk calling is supported. Before use, apply for and configure Alibaba Cloud voice service. For details, see: Alibaba Cloud Documentation: Quick Start for Domestic Voice Service
Custom voice alerts via webhook are possible but require custom programming. For details, see Grafana Alert Documentation: Webhook Notifications.
Add tts_template_code and region_id to the alert.yaml file under /etc/matrixdb/:
# aliyun service config
aliyun_service:
access_key_id: "your access_key_id"
access_key_secret: " access_key_secret"
signature: "signature"
sms_template_code: "SMS_123445678"
tts_template_code: "TTS_123456788"
region_id: "cn-hangzhou"access_key_id and access_key_secret are shared with SMS service.tts_template_code starts with "TTS_" and refers to the approved TTS voice template in the Alibaba Cloud console. Use ${name} as the variable for the alert message.Example template:
尊敬的管理员您好，贵公司的数据库系统发生${name}报警，请登陆网站查看详情，并及时处理。
Name the channel according to this format:
Aliyun Voice Message - For SuperAdministrators
"SuperAdministrators" denotes a recipient group. Define additional groups as needed. The phoneNumbers parameter is case-sensitive and supports multiple comma-separated numbers:
http://
: /api/alert/vms?phoneNumbers=18311111111,13811111111
Grafana provides built-in email alerting, which is simple to configure and use.
Since email alerting is a native Grafana feature, SMTP settings are configured in Grafana’s configuration file. The location may vary depending on deployment. On CentOS 7, the default path is:
/usr/local/etc/grafana/grafana.ini
For more information, see: Grafana Official Documentation: Configuration
Below is a sample configuration:
#################################### SMTP / Emailing #####################
[smtp]
enabled = true
host = <your smtp host>
user = <your user>
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
password = <your password>
;cert_file =
;key_file =
skip_verify = true
from_address = <your email address>
from_name = Grafana
;ehlo_identity =
;startTLS_policy =
[emails]
welcome_email_on_sign_up = false
templates_pattern = emails/*.htmlAs shown:
Name the channel using this format:
Email - For SuperAdministrators
"SuperAdministrators" refers to a group of recipients. Define additional groups as needed.
MatrixDB supports DingTalk-based alerts. When triggered, a pre-configured DingTalk robot sends a message to the designated group. To fully utilize this feature, configure both DingTalk and Grafana.
As shown:
Create a DingTalk group and add one or more alert robots.
Click "Settings" in the "Add Robot" interface, select "Custom", customize the robot name, add alert keywords, and save the webhook URL.
First, create a notification rule in Grafana and fill in the required fields.
After configuration, click "Test" to verify connectivity between Grafana and DingTalk. If successful, clicking the alert message in the DingTalk group should redirect to the Grafana dashboard. If you see an error like "localhost refused the connection", you likely forgot to replace localhost with your actual IP address.
After setting up notification channels, configure alert rules for each monitoring panel.
For more details on alert configuration, see: Grafana Official Documentation: Alert
Hover over the panel title to reveal a dropdown arrow on the right. Click "Edit" to enter edit mode.
Switch from "Query" to the "Alert" tab under "Edit Panel". If a popup appears saying "Notifier with invalid ID is detected", do not click "Delete", as this may remove the entire panel. Click "Cancel" and close the window instead.
Scroll down to the "Notifications" section. Enter a concise alert message (4–6 characters recommended). This message appears in emails and replaces the ${name} variable in SMS and voice templates. Keep it short due to message length limits. Select one or more notification channels (configured earlier) under "Send to".
Time-series databases aggregate data over time intervals. "No Data" may occur due to timing issues (e.g., Grafana time is 5 minutes ahead of system time while querying the last minute). Data collection failures can also result in missing data. The "No Data" condition is handled by the "Alerting" mechanism.
As shown:
The red threshold line in the "Used Disk Space" alert indicates the trigger level. Areas shaded in red exceed this threshold and represent critical conditions. The default alert threshold is 85%, but you can customize the "IS ABOVE" value as needed.
Disk I/O read and write alerts (Disk IO Reading, Disk IO Writing) are disabled by default, allowing you to define custom thresholds.
alert.yaml configuration file, run the following commands to apply the changes:source /usr/local/matrixdb/greenplum_path.sh
supervisorctl stop mxui
supervisorctl start mxui