YMatrix

Version Release History
Platform Requirements

YMatrix

Quick Start

Standard Cluster Deployment

Data Modeling

Connecting to The database

Data Writing

Data Migration

Data Query

Maintenance and Monitoring

Performance Tuning

Troubleshooting

Reference Guide

SQL Reference

FAQ

scenario

Version Release History

6.4.0 Release Notes

Release date: 2025-06-26

New version features

  • License management function is completely upgraded
  • Added support for Centos9 operating system
  • Added support for Cgroup V2
  • Resource Group has added support for IO_LIMIT function
  • MatrixUI has added disk space monitoring, including dashboard heatmap, 4 disk alarms, topology page display Util
  • MatrixUI supports displaying query blocking relationships on QE (#ICF0TL)
  • By default, use Domino v2 stream calculation to create streams (document)
  • Incremental online backup supports normal recovery when failover/recover occurs in the cluster (document)
  • Added mx_eager_memory_release switch to actively reduce the memory usage of idle processes (default off) (#ICH4Y9)
  • Support migration of mars3 table data through ALTER TABLE ... SET TABLESPACE statement
  • Supports parallel scanning of replicate tables (default off) (#IC5J6V)
  • The auto-vacuum function for system and non-system tables is turned on by default
  • Golang and toolchain upgrade to version 1.24

Enhancements and Improvements

  • Determine the TLog format of Domino v2, implement buffered read and write, significantly improving disk IO performance
  • Reform mxstop to perform streaming and output progress information.
  • Incremental online backup supports some tablespace functions (document)
  • mars3 supports roi indexing
  • mars3 supports automatic conversion of btree/brin to mars3_btree/mars3_brin index
  • The default relfile version of mars3 is adjusted to v1
  • Mars3 partition table supports drop index (#ICGEOH)
  • mars3 supports shared zstd context for further compression efficiency (#ICFRIE)
  • Dynamic bitmap scan on mars3 table is disabled in orca
  • Supports the case where the index type is mars3_btree when modifying the primary key
  • mxrule supports the ability to judge whether to project based on the number of distincts

Defect repair

  • Fixed the issue where the except statement cannot be accurately effective when the query field value is null under ORCA optimizer (#ICC3P7)
  • Fixed an issue where the explain analyze vectorized subplanning error occurred when opening auto_explain (#ICAUAG)
  • Fixed the issue of querying errors after partition table roi index migrating to new tablespace (#ICCMG7)
  • Solved the problem of analyze too slow after the full table of Delete
  • Fixed an issue where rowstore desired level get error when compact
  • Fixed the issue of session state pin count maintenance
  • Fixed the issue of the constraint check error when attach mars3 (#ICG260)
  • Fixed the issue of duplicating create index when attached mars3 index
  • Fixed freeze issue with non-unique mode tables
  • Solved the problem that mars3.force_allocate cannot be assigned to run in extreme cases

6.3.0 Release Notes

Release date: 2025-04-18

New features released

  • Support mxaddmirrors unbalanced deployment scheme (document)
  • Supports the addmirror function of graphic pages (document)
  • Support incremental cold backup and recovery functions (document)
  • mxstate series enhancement, adding parameters such as -s, -p``-H` (#IBDK0D)
  • mxstart adds progress visualization function, adding -C and -P parameters (#IBGYMU)
  • Upgrade the library intrastream to v2 version (as of the failover part, the default is still v1)
  • MatrixGate list in mxui integrates the function of deleting tasks
  • Matrixshift Migration DDL Added Support for USING, WITH, ECODING Attributes
  • Support recursive creation of extended statistics for partition tables
  • Support the ability to apply ndistinct to extend statistics to estimate the amount of result set data for composite conditions/index (#IBORMW)

Enhancements and Improvements

  • The replication service does not first crash OOM when system OOM (#IBLJ14)
  • Upgrade ETCD to 3.5.x and pass fatigue test verification
  • Golang toolchain upgrade to 1.23.7
  • Upgrade go-common-libs & matrixapis, clean up CVE
  • mxbox refactoring database connection layer
  • pgvector upgrade to v0.8.0 (#IBY4RH)

Defect repair

  • Fixed the issue where mars3 runpick lost run under concurrent
  • Fixed the issue of visibility related to distributed transactions when singlesegment in mars3
  • Fixed the issue where level2 stacking caused by mars3's compactjob (#IBNDHS)
  • Fixed the issue of data loss caused by metadata cache error in mars3btree
  • Fixed the problem that ending the Kafka task through the mxctl gate job --remove interface may cause mxgate force quit (#IBWTUQ)
  • Fixed the issue where the address and hostname in the gp_segment_configuration of the added mirror node in gp_segment_configuration is inconsistent with the hostname (#IBZOEL) in the custom mode of addmirror (#IBZOEL)
  • Fixed the problem that QE kept restarting when mxrecover -r reblance is reported when replication was read pid file failed, err pid file has 7 lines, expect 8, which caused QE to keep restarting (#IC0SVB)
  • Fixed the issue where mx_default_segment_set is triggered by opening mx_get_common_segmentids_with_default when setting mx_default_segment_set to a non-existent segment set (#IBPYOI)
  • Fixed the issue of using mx_digest_ws to optimize crash in vectorization (#IBQZOC)
  • Fixed an issue where duplicate filtering conditions could result in inaccurate estimates (#IBPYA4)
  • Fixed the problem of vectorized VAgg Sum(int64) error result (#IBTZUY)
  • Optimize revert shallow-copy to avoid partial copying of Node (#IBN258)
  • Fixed the issue where the error of generating execution plan ERROR: could not find pathkey item to sort (#IBSE48)
  • Fixed the issue where the Motion compression function was accessed by null pointer in OOM (#IBTESJ)
  • Fixed the issue of MxRule duplicate condition derivation (#IBNWJ0)

6.2.4 Release Notes

Release date: 2025-03-03

***Enhancements and Improvements

  1. Optimizer
    • Optimized ROI logic, no dummy plan is generated when the corresponding partition cannot be found from the ROI (#IBOER)
  2. Tools
    • Optimized the success rate of probing all segments before mxstart starts the node (#IBOPCG)Bug Fix
  3. Storage Engine
    • Fixed the issue of low probability errors in MARS3 fixed-length optimization (#IBMR40)
    • Fixed iterator issue with MARS3 bitmapscan (#IBOE1Q)
  4. Execution Engine
    • Fixed an issue where Table out-of-bounds access may occur in VHashJoin Spill scenario (#IBNSF8)
    • Fixed the issue where VWindow triggers crash or reports an error when reading data from batchstore (#IBONDT)
    • Fixed an issue that caused mirror crash when streaming snapbuild files were not found (#IBONUY)
  5. Service Management
    • Fixed a leak in cmux connections for SOA service (#IBOOX0)
    • Fixed an issue where addmirror blank machine caused plan error (#IBOYWS)

6.2.3 Release Notes

Release date: 2025-02-24

Enhancements and Improvements

  1. mxbox
    • Optimized redundant information about cluster state in etcd, reduces the pressure to write and update etcd services, and enhances cluster stability (#IBFELE)
    • Extended the function of the mxbox client deployer get-cluster-service command line tool. In addition to display the currently active cluster/shard service address, it can also display information about inactive highly available services in standby. At the same time, add a required parameter --service-name, with the value cluster or shard (#IBDQ6W)
    • Caches the information discovered by the SOA service in the deployer service. Other services obtain service discovery results by accessing the deployer API interface, reducing the pressure on direct access to etcd and improving operational stability under large-scale clusters (#IBGJJ8)
    • High Availability Service Process cluster/shard/replication/deployer will more actively release physical memory to the operating system and reduce memory usage (#6e9e4636)
    • When mxstart/mxstop is operated, if the cluster size is large, the shard/replication will be started in batches to avoid the instantaneous ultra-high load on etcd by concurrent startups of all services, which enhances operational stability (#IBI16W)
    • When primary/mirror under a shard is down, the node in gp_segment_configuration that retains the state of the last primary role is UP to ensure that the database can still be restarted successfully (#IBJ58D)
    • The mxrecover tool has added the -s option to specify the shard to be restored (#IBJLRN)
    • The mxstate tool has added the -e option to output segment information in an absolute state (#IBK5EM)
  2. Disaster recovery
    • The restore_point_creation_interval_minute was deleted in the configuration file of the subscriber process, and the restore_point_creation_interval (required) parameter was added. Users can configure the synchronization interval by themselves (#2cd254aa)
    • mxaddmirrors plan now supports the generation of mirror node distributions for clusters for unbalanced deployments. It is recommended that when using custom distributions for unbalanced clusters, they should be used after manual review and adjustment to the most reasonable mirror layout (#IBFUEX)
    • Changed how mxaddmirror custom configuration is used. Users need to first mxaddmirrors init, and then mxaddmirrors config --init-file <path_to_file> to generate config templates. Now, even for unbalanced clusters, a reasonable configuration scheme can be generated according to the new usage method (#IBFUDC)
    • For unbalanced deployment clusters, mxinitstandby allows to add standby to a blank physical machine without deployed segment nodes (#IBIU96)
    • For unbalanced deployment clusters, mxaddmirror allows adding mirror to a blank physical machine without deployed segment nodes in custom configuration mode (#IBKSYA)Bug Fix
  3. MatrixShift
    • Fixed the issue where PRIMARY KEY in the source database was turned into UNIQUE INDEX + NOT NULL constraint after mxshift's DDL migration function (#IBFVS3)
    • Fixed the problem of mismatching parameter positions in the generated DDL statement when migrating the definition of UDF (#IBIN2R) if the function has both in-parameter and out-parameters (#IBIN2R)
  4. MatrixGate
    • Fixed an issue where concurrent slots that were stopped by auto-tune could not be restored to work again when using mxgate set --auto-tune = false when turning off the auto-tune function (#IBKGPH)
  5. mxbox
    • Fixed an issue where the replication service could potentially retain defunct zombie process when starting/stopping the postgres instance (#IBICKJ)
  6. Disaster recovery
    • Fixed an issue where DR synchronization may be stuck when failover occurs on the source database (#IBKM33)

6.2.2 Release Notes

Release date: 2025-01-10

Enhancements and Improvements

  1. MatrixShift
    • If the configuration file does not specify any migration operation, and only when the verification function is turned on, an error message is added to a database that is migrated from non-mxshift (#IBFVMM)
    • When there are duplicate tables or parent tables and some child tables appear in table-list in the scope paragraph of the configuration file, the ability to automatically remove duplicate tables and report warnings (#72ffcea)
  2. MatrixGate
    • Data transfer between mxgate and segment nodes is adjusted to not enable compression by default to save more memory resources. The original default value is Enable compression (#IBD0PW)
    • In transfer mode, allow the migration of data when the mirror node is in Down state (#IBCQPB)
  3. mxbox
    • Added the skip reason output when mxrecover skips certain shards (#IB4NJD)
    • When using mxrecover to perform rebalance operations on a cluster, you can shorten the database downtime caused by the operation by adding CHECKPOINT, and you can skip this behavior by adding --skip-checkpoint parameter (#IBCVOL)
    • Improved output content format of this command in the console when rebalanced to the cluster using mxrecover (#IBCVOS)
    • mxrecover Adds the --prepare-timeout parameter. When mxrecover starts, the timeout of cluster status information can be collected through etcd, default is 1 minute (#IBCVOS)
  4. mxctl
    • When installing and deploying, the mxctl setup collect command will check whether localhost can be resolved to a local loopback address, and if it cannot, an error will be reported (#IBDM77) Error fix
  5. MatrixShift
    • Fixed an issue where errors would be reported when migrating tables without any fields (#IBFVQP)
    • Fixed an issue where the referenced table or other database object in the created function no longer exists when migrating DDL (#d51821f)
  6. MatrixUI
    • Fixed the issue of the self-service inspection function reporting incorrect version number (#IBD0YZ)
  7. mxbox
    • Fixed an issue where DR might get stuck when reading WAL from segment (#216abca3)
    • Fixed logic issue with vmem protect limit calculation after joining unbalanced deployment (#1b30eb0c)
    • Fixed the issue where the monitoring function of mxmgr is enabled in the system, and the monitoring-related service process was not deleted when mxdeletesysystem (#IBB9SM)
    • Fixed an issue where some services (shard and cluster) remained on mxdeletes if there is no segment on some hosts in the physical machine cluster (#63c34070)
    • Fixed the issue where there is a low probability of timeout when using mxrecover -r to restore master and data shard at the same time (#IBD71J)
    • Fixed an issue where there is a chance of NotFound error when the replication service stops the postgres instance (#IBD8H7)
  8. High availability
    • Fixed an issue where the original Primary was mistakenly retained when the network partitioning issue occurred on the primary node (Primary) and Mirror was promoted to Primary and brought double Primary after the original Primary was promoted to Primary (the original Primary should be killed to retain the promoted Mirror) (#IBE2HF)
    • Fixed an issue where if the primary is already running normally when mxstart starts a primary, causing incorrectly interrupting the startup process (#986b5588)
    • Fixed an issue where if the primary has stopped when mxstop stops a primary, causing the stop process to be incorrectly interrupted (#986b5588)
    • Fixed the problem of low probability of automatic partitioning stuck and not working when the database is unexpectedly shut down and restored (#IBDSBY)

6.2.1 Release Notes

Release date: 2024-12-20

Enhancements and Improvements

  1. mxbox
    • The default timeout for mxstart increases from 1 minute to 5 minutes. It still supports users to customize the timeout with the -t parameter. (#9ab2957c)
    • Supports user-defined environment variables when the postmaster process is started. (#39575ad3)
    • When a double failure occurs in the cluster, it allows the nearest primary node to be started by restarting the database instead of failing to start. (#ab0b920a)
    • Improved the number of max open files for supervisor service and etcd process, solving the problem of frequent too many open files errors for etcd services under hyperscale clusters. (#6ca5293dcd0)
    • When a new cluster is deployed, the --enable-pprof parameter of the etcd service was deleted in the service configuration of the supervisor, making the etcd service more stable and the CPU proportion is lower. (#ae00e5a)

Bug Fix

  1. mxbox
    • Fixed an issue where the default value of gp_vmem_protect_limit that is automatically configured is too large when a machine has multiple data nodes during cluster deployment. (#5253cbbb)
    • Fixed an error in which the postmaster crashed unexpectedly when the replication service probed the postmaster, and there was a chance that the replication service would live. (#ab77de4e)

6.2.0 Release Notes

Release date: 2024-12-09

Enhancements and Improvements

  1. MatrixUI
    • When installing or expanding a new machine, it will actively check whether /etc/ssh/sshd_config contains AllowUsers/AllowGroups/DenyUsers/DenyGroups and other configuration items that may cause the ssh mutual trust operation of the host in the cluster during the installation process, and give specific error information and modification prompts (#IB80HF)
  2. MatrixShift
    • Introduce stricter checks on configuration files, incorrect configuration items will be reported and prompted when mxshift starts. For example, disable-connector = true should be written in the [scope] paragraph, but the error is written in other paragraphs. mxshift will report an error and remind you when starting (#IB5002)
  3. mxbox
    • moveseg / initstandby tool can detect whether the host added by the --add subcommand is already in the physical machine cluster and give an error message (#5e39499c)
    • When the host restarts or runs the YMatrix cluster for the first time, it can automatically create the relevant paths that the resource group cgroup depends on (#IB5G9O)

Bug Fix

  1. MatrixUI
    • Fixed the issue where the calls_end value sum after 0 in the query load page causing the query error (#IAMZI0)
    • Fixed an issue where MXUI failed to redeploy the mxui collector metric acquisition process when the cluster topology changes (#IB8BTF)
  2. MatrixGate
    • Fixed an issue where mxgate could not continue to consume data from kafka after the YMatrix cluster was restarted in some environments (#IB6HEL)
  3. MatrixShift
    • Fixed ALTER MATERIALIZED VIEW syntax error issue when DDL migration, supporting MATERIALIZED VIEW syntax migration for GP6 (#3ba9664)
  4. mxbox
    • Fixed an issue where the cache caused the shard status to not be updated (#d5844a19)
    • Fixed an issue where the current version of meta information was eliminated when etcd compaction (#6c5c7c5e)
    • Fixed the issue that the primary node in an abnormal state cannot be stopped in time when the network partition is abnormally restored (#695d45fe)

6.1.1 Release Notes

Release date: 2024-11-19

Enhancements and Improvements

  1. MatrixGate
    • mxgate optimization: Supports segments connection timeout as configurable GUC (#IB25VC)
  2. MatrixShift
    • mxshift optimization: /tmp use data dir when there is no permission (#IB0UY1)
  3. mxbox
    • Supports custom unbalanced deployments (#IAVBH5, #IAVCE2, #IAVFBE, #IAVDNP)
  4. MatrixUI (Document)
    • Supports custom unbalanced deployments UI (#IAVXM5)

6.1.0 Release Notes

Release date: 2024-10-31

Enhancements and Improvements

  1. MatrixShift

    • mxshift optimization: migration from gp6 data can be skipped external table (#IB0IUE)
    • mxshift optimization: compresslevel is forced to 1 when compresstype is automatically mapped from zlib to zstd (#IB0GA9)

  2. Disaster recovery

    • Removed useless configuration item slot in subscriber configuration file
    • Adjusted content in the log

Bug Fix

  1. MatrixShift
    • Fixed an issue where an error occurred when creating a "COLLATABLE" field type

6.0.0 Release Notes

Release date: 2024-10-11

New Feature Release

  1. Added stream computing function (Document)
    • Supports expanded dimensional computing, aggregation computing, multi-level flow computing, dual-stream correlation computing, etc. (Document)
    • Supports multiple configuration and management capabilities (document)

2.** Integrated flow batch architecture

  • The database fusion stream processing and batch processing advantages, supporting the parallel processing of real-time data and historical data.
  1. Added Disaster Recovery Cluster (Document)
    • Supports data transmission and synchronization across data centers
    • Supports the operational status monitoring of disaster recovery clusters.
    • Supports manual switching of backup clusters to service clusters, and can switch all applications to this cluster for normal business operation.
    • The disaster recovery metrics for this disaster recovery cluster are as follows:
      • RPO (Recovery Point Objective, recovery point target) The user can specify a relatively appropriate value to set this target to the minute level without affecting the performance of the source cluster.
      • RTO (Recovery Time Objective, recovery time target) Supports database hot standby implementation, and RTO can reach seconds to minutes.

Performance enhancement

  1. TP performance enhancement

    • Added support for Blocksize 8K, and the configurable data block (block) is 8KB, which greatly improves the transaction processing performance of small data volumes.

  2. MARS3 Storage Engine Enhancement

    • Added support for unique key, upsert, row-level lock capabilities.

  3. Vectorization engine enhancement

    • Added support for Window Function vectorization.

Know issues

  1. When there are many partitions, the query will trigger the database OOM.

  2. In the KILL QUERY scenario, the recovery process may experience crash problems.