This document describes the cluster parallel backup utility mxbackup.
mxbackup performs parallel backups of a YMatrix cluster. The backed-up data must be restored using mxrestore.mxbackup supports the S3 object storage plugin, allowing data to be streamed directly to S3 without additional I/O overhead. To use S3 for backup and restore, you must prepare the required account credentials and bucket permissions, and create a properly configured YAML file. See below for configuration parameters and usage instructions.| Parameter | Description |
|---|---|
| --backup-dir directory | Absolute path where backup files are written |
| --compression-level level | Compression level from 1 to 9; default is 1 |
| --data-only | Back up only table data, not schema (metadata) |
| --dbname db | Database to back up |
| --debug | Output debug-level log messages |
| --exclude-schema schema | Schema to exclude from backup; can be specified multiple times for multiple schemas |
| --exclude-schema-file file | File containing a list of schemas to exclude |
| --exclude-table table | Table to exclude from backup; can be specified multiple times for multiple tables |
| --exclude-table-file file | File containing a list of tables to exclude |
| --from-timestamp timestamp | Start timestamp for incremental backup; must be used with --incremental |
| --help | Display help information |
| --history | Show historical timestamps in the current backup directory |
| --include-schema schema | Schema to include in backup; can be specified multiple times for multiple schemas |
| --include-schema-file file | File listing schemas to include in backup |
| --include-table table | Table to include in backup; can be specified multiple times for multiple tables |
| --include-table-file file | File listing tables to include in backup |
| --incremental | Perform incremental backup (for AO tables only); requires --from-timestamp |
| --jobs num | Number of concurrent connections during backup; default is 1 |
| --leaf-partition-data | For partitioned tables, create separate data files for each leaf partition |
| --metadata-only | Back up only metadata, not table data |
| --no-compression | Do not compress table data |
| --plugin-config file | Specify location of plugin configuration file |
| --quiet | Suppress non-error and non-warning log messages |
| --single-data-file | Write all data into a single file instead of one file per table |
| --verbose | Print detailed log messages |
| --version | Print tool version and exit |
| --with-stats | Include statistics in backup |
| --without-globals | Exclude global metadata from backup |
| Parameter | Description | Required |
|---|---|---|
| executablepath | Absolute path to the S3 storage plugin | Yes |
| region | Cloud platform region; ignored if endpoint is set | Yes |
| aws_access_key_id | S3 access key ID for connecting to the bucket | Yes |
| aws_secret_access_key | Secret key for the S3 access key ID | Yes |
| bucket | S3 bucket used to store mxbackup data files | Yes |
| endpoint | S3 endpoint URL | No |
| encryption | Enable SSL encryption for S3; valid values: on, off; default: on | No |
| http_proxy | HTTP proxy server URL for connecting to S3 | No |
| backup_max_concurrent_requests | Maximum concurrent requests during backup; default: 6 | No |
| backup_multipart_chunksize | Maximum buffer/chunk size for backup; default: 500MB | No |
| restore_max_concurrent_requests | Maximum concurrent requests during restore; default: 6 | No |
| restore_multipart_chunksize | Maximum buffer/chunk size for restore; default: 500MB | No |
Below is a sample configuration file template. Select the required lines and replace content within "<>" or "[]" (including the "<>" and "[]" symbols) with actual values.
$ executablepath: <absolute-path-to-gpbackup_s3_plugin>
options:
region: <cloud-region>
endpoint: <S3-endpoint>
aws_access_key_id: <user-id>
aws_secret_access_key: <user-key>
bucket: <s3-bucket-name>
folder: <folder-on-s3-for-backup-data>
encryption: [on|off]
backup_max_concurrent_requests: [int]
backup_multipart_chunksize: [string]
restore_max_concurrent_requests: [int]
restore_multipart_chunksize: [string]
http_proxy:
<http://<username>:<password>@proxy.<domain>.com:port>In the following examples, the database name is demo, and the schema name is twitter.
Back up the entire database:
$ mxbackup --dbname demoBack up the demo database excluding the twitter schema:
$ mxbackup --dbname demo --exclude-schema twitterBack up only the twitter schema in the demo database:
$ mxbackup --dbname demo --include-schema twitterBack up the demo database and store backup files in the /home/mxadmin/backup directory:
$ mxbackup --dbname demo --backup-dir /home/mxadmin/backupBefore using S3 for backup and restore, ensure you have proper cloud account credentials and bucket (Bucket) permissions, including but not limited to:
First, prepare the S3 plugin configuration file s3-config-file.yaml. This example includes common settings. For full parameter descriptions, refer to section 2.2 above.
$ executablepath: $GPHOME/bin/mxbackup_s3_plugin # Absolute path to S3 plugin
options:
region: us-west-2 # Cloud region
aws_access_key_id: test-s3-user # S3 login ID
aws_secret_access_key: asdf1234asdf # S3 login key
bucket: matrixdb-backup # S3 bucket name
folder: backup3 # Directory name in S3 object storageThen, use mxbackup to perform a parallel backup of the demo database:
$ mxbackup --dbname demo --plugin-config /tmp/s3-config-file.yamlAfter a successful backup, mxbackup creates a timestamped directory in the S3 object storage. You can later use the mxrestore utility to restore data from S3.
$ backup3/backups/20221208/20221208185654Note!
The log file directory formxbackupis<gpadmin_home>/gpAdminLogs/gpbackup_s3_plugin_timestamp.log, with timestamps in formatYYYYMMDDHHMMSS.
Note!
For more information about backup and restore operations in YMatrix, see Backup and Restore. For details about the restore tool, see mxrestore.