YMatrix

Quick Start

Cluster Deployment

Data Model

Data Writing

Data Query

SQL Reference

Maintenance and Monitoring

Tool Guide

Troubleshooting

FAQ

Scenario
Version Release History

Version Release

Notes!
Experimental features are features that exist in a specific version but have not been officially released. They are subject to change or removal without prior notice in terms of syntax or implementation. Please use them with caution.

4.8.12 Release Notes

Bug fixes

  1. Storage engine:
    • Fixed an issue where MARS2 table age could not be frozen (#IAI8EF).

4.8.11 Release Notes

Bug Fixes

  1. Storage Engine:
    • Fixed a crash when there were multiple arraykeys with the same attno in a mars2_btree index. (#7306)
    • Fixed a collation error when pushing down the like operator's brinkey in MARS2. (#7252)
    • Fixed an issue where MARS2 returned incorrect results after pushing down the BRINKEY operator in UniqueMode. (#7225)
  2. Optimizer:
    • Fixed an issue where the optimizer reported an error when converting a subquery to a JOIN operator for execution. (#6524)

4.8.10 Release Notes

Bug Fixes

  1. Storage Engine:
    • Fixed an issue where the mars2_btree index would return data that did not meet the query conditions when the query conditions were repeated. (#6632)
    • Fixed an issue where the mars2_btree index would return incorrect results for specific queries due to incomplete logic for searching duplicate sort keys. (#7041)

4.8.9 Release Notes

增强与改进

Enhancements and Improvements

  1. MatrixGate:
    • Supports enabling or disabling automatic slot number adjustment for specified tasks, and supports querying the adjustment status of all tasks or specified tasks. (Documentation)
    • Introduces the --abort-by-pause-timeout parameter under the [writer.stream] category. Supports abandoning data accumulated in memory when a task (Job) enters the pause state, based on the timeout specified by this parameter. (Documentation)
    • Supports batch writing of JSON-type data via HTTP. (Documentation)

Bug Fixes

  1. MatrixGate:
    • Fixed an issue where data migration would fail and terminate when the number of data channel connections exceeded the number of segments in the source cluster in migration mode. (#6800)
    • Fixed an issue where MatrixGate (mxgate) failed to respond to Data Definition Language (DDL) changes and automatically refresh tasks after the master node (Master) failed and automatically switched to the standby node (Standby) during a failover. (#6850)
    • Fixed an issue where the mxgate service configured via the Kafka data stream feature in MatrixUI would be repeatedly restarted by the supervisor process when the target table did not exist. (#6928-1)
  • Fixed an issue where starting multiple Kafka data streams simultaneously across multiple browser windows could cause conflicts with the mxgate port in the background. (#6928-2)
  • Fixed an issue where, when using the consumer-group mode with the Kafka data source, MatrixGate might fail to continue extracting data from the consumer-group after a database restart/crash. (#6848)
  • Improved JSON mapping semantics: inputting { “key”: null } now maps the key to NULL in the database. (#6536)
  • Fixed an issue where, after pausing a task using the mxgate pause command (without the -X parameter), mxgate could not be stopped using mxgate stop or Ctrl+C. (#6519)
  • Fixed an issue where conflicts occurred between the behavior of the data table DDL being modified, the automatic slot adjustment feature attempting to change the task status, and the task being in a paused state. (#6436)
  • When mapping data rows to JSON, if no corresponding key is found for each column that needs to be mapped in the current data row, the row is reported as invalid, and an empty row is not inserted into the database. (#6488)
  • Improved the performance and memory usage of the mxgate logging component. (#6242)
  • Fixed an issue where mxgate failed to automatically create the matrixts extension when the database's default search path was not set to “public.” (#6430)
  • Fixed an issue where metric reports from mxgate watch became confused when multiple mxgate processes were used on the same database. (#5848)
  • Optimized the timing of memory pool release in the plain transform component, reducing memory usage by the mxgated process. (#6071)
  • Optimized the memory release cycle for zstd compression handles, reducing memory usage by the mxgated process. (#6299)
  • Fixed an issue where database sessions established by MatrixGate could not be refreshed periodically when the cluster had a Standby node. (#6308)
  1. Graphical interface:
    • Fixed an issue where the page could not be displayed due to a JavaScript error when searching on the data flow page. Affected versions are 4.8.7 and 4.8.8. (#6977-1)
  • Fixed an interaction issue where the page automatically refreshed and jumped to the top of the page after manual scrolling, instead of staying at the original scroll position. (#6977-2)

4.8.8 Release Notes

Bug fixes

  1. Query engine:
    • Fixed an issue where executing a WITH statement in certain scenarios would cause the cluster to crash.

4.8.7 Release Notes

Bug fixes

  1. Interconnect:
    • Fixed a crash caused by incorrect release of TCP connections under certain conditions.
  2. Tools:
    • Fixed an insufficient permissions issue with the mxpacklogs tool on certain systems.

4.8.6 Release Notes

Bug Fixes

  1. MatrixGate:
    • Fixed an issue where MatrixGate failed to resume tasks due to internal timeouts during automatic partitioning of data tables, resulting in data entry failures. (#6165)
  2. YMatrix Database:
    • Fixed an issue where releasing connections caused database memory overuse. (#5826)

4.8.5 Release Notes

Bug Fixes

  1. MatrixGate:
    • Fixed an issue where all tables became unwritable after MatrixGate triggered automatic partition management.
  2. Execution Engine:
    • Fixed query errors caused by incorrect operator type settings.

4.8.4 Release Notes

Bug Fixes

  1. Storage Engines:
    • Fixed a crash issue caused by the AOCS storage engine during index scans.
    • Fixed a crash issue caused by defining more than 32 minmax indexes in the MARS2 storage engine.
  2. MatrixGate:
    • Fixed a memory overuse issue caused by MatrixGate when starting compression.
    • Fixed an internal logic error caused by connection timeouts in MatrixGate.
  3. MatrixManager:
    • Fixed memory overuse caused by MatrixMgr not releasing connections for an extended period of time.
  4. Tools:
    • Fixed an error that occurred when executing DDL export operations in mxshift.

4.8.3 Release Notes

Feature Release

  1. mxshift functionality enhancements (documentation):
    • The install-dir configuration item has been added to the [database.source] category in the configuration file. This item replaces the original gphome parameter in the [scope] category and is used to configure the database installation path.
    • The [database.source] category in the configuration file now includes a new hostname-to-ip configuration option, which supports specifying the IP address corresponding to the hostname.

Bug Fixes

  1. Execution Engine:
    • Fixed a crash caused by the sliding window function handling grouping logic.
  2. Tools:
    • Fixed a parsing error caused by special characters in pgbench.
  3. MatrixGate:
    • Fixed an issue where MatrixGate blocked other tables from inserting data when locking a table.

4.8.2 Release Notes

Feature Release

  1. MatrixGate Upgrade:
    • Supports full-process observations for data writing, and is used to debug the transmission performance of MatrixGate to Segment. (Document)
    • Supports hot updates to MatrixGate log levels. (Document)
    • [Experimental Features] Provides the parsing and mapping support for JSON format data under all data sources (http/stdin/kafka/transfer/grpc) types in the --source parameter. (Document)
    • [Experimental Features]--dml-template parameter supports docking data in TEXT/CSV format. (Document)
    • After Master switches to Standby manually, the metric information written to MatrixGate will be switched to Standby.
    • The speed increases dramatically when a transaction that does not perform a write data operation.
  2. mxshift feature enhancement (document):
    • In the unit of the database, implementation migration of data table DDL, and you can choose whether to migrate only DDL.
    • [Experimental Features] Support exporting data table DDL to files.
    • [Experimental Features] Supports data table DDL files to be imported directly or modified to import them into the database concurrently.
  3. Graphical interface upgrade:
    • The deployment page has been completely upgraded, and the steps are simpler. (Document)
    • The expansion page has been completely upgraded, and the steps are more lightweight. (Document)
    • Added Segment high availability status detection function. (Document)
    • Health monitoring page upgrade to optimize page effect. (Document)
    • Easily get started with document upgrades, add write tests and query test pages. (Document)
  4. Monitoring enhancement:
    • Allow custom monitoring scripts: Support users to modify the monitoring scripts when deploying Grafana/Prometheus monitoring through mxmgr_init_local() or mxmgr_init_exporter() to develop monitoring projects by themselves. (Document)

Bug Fix

  1. Storage engine:
    • Fixed an issue where the MARS2 storage engine was incorrect when dealing with text type sort keys.
    • Fixed the issue where AOCO storage engine compression caused memory errors.
    • Fixed an issue where the storage diagnostic tool datainspect error crashed.
    • Fixed the issue of MARS2 storage engine xid overflow.
  2. Cylinder:
    • Fixed an error that the sliding window could not trigger.
    • Fixed an error that the APM/TBS/SV instruction cannot be triggered after expansion and redistribution of data under a certain probability.
  3. MatrixGate:
    • Fixed an issue where some HTTP requests inserted into the database in MatrixGate were not responding when YMatrix restarts or resumes.
    • Fixed an issue where the MatrixGate write queue blocking when the user DROP or ALERT the target table, causing the incoming number of other transactions to be stuck.
    • Fixed the issue where MatrixGate did not report an error when parsing "1,2,3" similar error JSON data.
  4. MatrixManager:
    • Fixed an issue where mxstat restart failed.
    • Fixed the issue where mxstat occasionally crashes when the process exited.
  5. Query engine:
    • Fixed an issue where client encoding settings were incorrect in direct dispatch mode.
    • Fixed an issue where COPY statements handled illegal options incorrectly.

4.8.1 Release Notes

Feature Release

  1. Added a new storage diagnostic tool Datainspect. This tool is only available for MARS2 tables. (Document)
  2. MatrixGate JAVA SDK features enhanced, and write performance is greatly improved. (Document)
  3. Graphical interface upgrade:
    • The query monitoring page has added blocking session monitoring function. (Document)
    • The cluster management page has been fully upgraded, divided into three sections: indicator view, instance view, and list view. (Document)

Bug Fix

  1. Storage engine:
    • When the MARS2 table participates in the nestloop join, the result is incorrect due to incomplete transmission of index scan parameters.
  2. Resource Group:
    • Fixed an issue where resource groups had an operation crash when handling SIGTERM.
  3. Optimizer:
    • Fixed the issue of CTE crashing due to the lack of parallel execution.
    • Fixed an error in incorrectly handling sub-plan calculations when starting ORCA.
  4. Execution Engine:
    • Fixed the issue where Share CTE caused the operation to be stuck due to incorrect handling of Direct Dispatch.

4.8.0 Release Notes

Feature Release

  1. MatrixGate Upgrade:
    • Supports ZSTD compressed data transmission to Segment. (Document)
    • Supports simplified data writing code through integrated JAVA SDK. (Document)
  2. New graphic interface:
    • Health monitoring interface, you can receive alarm emails in a timely manner by configuring your email address. (Document)
    • Thermal upgrade function, supports individual upgrade of graphic interfaces. (Document)
  3. mxbackup supports the S3 object storage plug-in to upload data to the S3 object storage. (Document)
  4. mxshift feature enhancement. (Document)
  5. Added log collection and packaging tools mxpacklogs. (Document)
  6. Ubuntu20 platform support. (Document)
  7. Grafana monitoring item enhancement. (Document)

Bug Fix

  1. Optimizer:
    • Fixed errors caused by handling duplicate scans in nestloop join parameterized path plan.

4.7.5 Release Notes

Bug Fix

  1. MatrixManager:
    • Fixed the issue of inaccurate actual execution time of SQL when getting slow query.

4.7.4 Release Notes

Bug Fix

  1. Optimizer:
    • Fixed an error when the Redistration Motion distribution key contains Param in the query plan.

4.7.3 Release Notes

Bug Fix

  1. Storage engine:
    • Fixed the issue where the MARS2 table in the mirror node did not completely delete the required files when processing vacuum.

4.7.2 Release Notes

Bug Fix

  1. Optimizer:
    • Fixed an error problem involving constant values ​​​​​when executing Redistration Motion.

4.7.1 Release Notes

Bug Fix

  1. MatrixManager:
    • Fixed an issue where the mxstat error caused an operation crash when retaining snapshots.

4.7.0 Release Notes

Feature Release

  1. Support the CREATE INDEX CONCURRENTLY syntax. (Document)
  2. Added a new data migration tool mxshift. (Document)

Bug Fix1. Execution Engine:

  • Fixed the issue where Copy did not properly handle Interrupt caused the operation to be stuck.
  • Fixed memory overflow issues when creating large numbers of partition tables.
  1. Storage engine:
    • Fixed the issue of inaccurate relsize estimates corresponding to the Master Only AOCS table.
  2. Tools:
    • Fixed the issue where Analyze did not handle special characters correctly.
  3. Optimizer:
    • Fixed the issue where Recursive CTE generated an incorrect query plan under Merge Join.

4.6.1 Release Notes

  1. MatrixGate Upgrade:
    • For JSON data connected to Kafka, it supports customizing the mapping relationship between JSON to data columns using the dml-template parameter.
    • Added Kafka identity authentication mechanism. Using the SASL framework, SCRAM verification is currently supported.
  2. Fix known errors.

4.6.0 Release Notes

  1. The MARS2 storage engine aggregate query performance is greatly optimized.
  2. Fully upgraded graphical interface:
    • Upgraded appearance, ultimate visual experience.
    • Added easy-to-operate function to experience timing business scenarios with one click. (Document)
    • Kafka data flow writing supports automatic creation of MARS2 tables. (Document)
    • Added graphical interface expansion. (Document)
  3. MatrixGate upgrade:
    • Added the function of writing related parameters in parallel without shutting down. (Document)
    • Added the function of updating the table structure without shutting down. (Document)
    • Migration mode no longer requires a large number of continuous ports.
    • Supports calling within UDF.
  4. Grafana monitoring page upgrade, supporting Prometheus data monitoring. (Document)
  5. Domestic platform support: Galaxy Kirin V10 operating system + ARM platform support.
  6. Fix known errors.

4.5.0 Release Notes

  1. The MARS2 storage engine is grandly released, the UPSERT scene is optimized significantly, and the background is automatically compressed, eliminating the table rotation process. (Document)
  2. Added a query statistics component, which can be used to count query usage time and resource consumption. (Document)
  3. MatrixGate upgrade (document):
    • Supports arbitrary column timestamp conversion.
    • Added a watch function to view real-time and historical metric information of data written by MatrixGate.
    • UPSERT supports boolean types.
  4. mxbench upgrade, users can flexibly configure indicator types, acquisition frequency, null rate, randomness, etc. (Document)
  5. Added sliding window flow calculation, real-time analysis and monitoring of streaming data, and supports subscription and publishing. (Document)
  6. Kubernetes support, MatrixDB clusters can be deployed on k8s. (Document)

4.4.0 Release Notes

  1. MatrixGate upgrade (document)
    • UPSERT supports deduplication.
    • Added --insert-timeout and --request-timeout parameters to control timeouts to prevent avalanches caused by resource exhaustion in high-load scenarios.
  2. The Kafka data access process is redesigned, supporting two data formats and mapping relationship settings, CSV and JSON. (Document)
  3. Automatic partition management page upgrade. (Document)
  4. The INSERT INTO SELECT statement supports parallel scanning.
  5. Grafana monitoring page has been upgraded, the monitoring panel is split into system panel and database panel, and a Chinese description has been added. (Document)
  6. PXF is upgraded to 6.2.1, and the throughput rate is 2 to 5 times higher in a large number of small file scenarios.

4.3.10 Release Notes

4.3.10 Fixed the following error:

  1. Problem that mirror cannot start after Segment crash.
  2. UDF access to catalog in specific cases causes QD crash problems.
  3. PXF access Hive ORC issue adding filter crash.

4.3.0 Release Notes

  1. Continue to gather and go online
    • Continuous aggregation synchronizes the source table data in real time through materialized views and sorts and merges, making aggregation query more efficient. (Refer to Document)
  2. Spatial data type support
    • An enhanced version of PostGIS component has been developed, which can perfectly support the storage and computing of spatial data types. (Refer to Document)
  3. MatrixGate upgrade
    • Enhanced data access fault tolerance mechanism, and a single data format error will not affect other data in the batch. (Refer to Document)
    • MatrixGate log management, supports automatic archiving and cleaning.
    • Added a migration mode to efficiently synchronize data tables in other Greenplum5, Greenplum6, and YMatrix Database clusters to this cluster. (Refer to Document)
  4. Visual data table is online
    • A visual data table section has been added to the management page, which can easily view the status of the cluster database and tables. (Refer to Document)
  5. Automatic partition graphical management
    • The partition table can be set through a graphical interface and an automated partition management strategy. (Refer to Document)
  6. MARS storage engine upgrade
    • tag_id supports data types. Text, varchar, name, and numeric are added on the basis of integers; and supports multiple grouping keys, that is, multiple columns are used as tag_id.
    • The MARS storage engine supports merge scan operation, and can update existing data through merge.

4.2.0 Release Notes

  1. MARS engine optimization
    • The intelligent compression algorithm is online, and it is determined whether to enable compression based on the data type and characteristics. After using the new encoding compression, the storage footprint dropped by 70%.
    • Query performance improvement, specific queries sorted based on timestamps, the time delay is reduced by a thousand times, from seconds to "millisecond level".
    • Query memory usage has been greatly reduced, and the memory usage rate of SELECT statement has decreased by 90%.
  2. MatrixGate supports UPSERT and is suitable for the following scenarios (see [Document] (/doc/4.2/datainput/matrixgate#upsert) for usage):
    • The device may repeatedly send data or correct error timing data.
    • Different categories of indicator data are sent in batches to realize automatic merging.
  3. Hot and cold hierarchical conversion supports merging and compression: In order to facilitate users to convert hot to cold data in the partition table, multiple partition tables can be merged together. In this way, it can reduce the pressure of partition management and also facilitate maintenance by DBA. Please refer to Document
  4. The "tool" of parallel backup and recovery has been publicly unveiled for the first time: YMatrix Database grandly launched two tools, mxbackup and mxrestore. Compared with pg_dump and pg_restore inherited from PostgreSQL, it supports parallel processing and has higher backup and recovery efficiency. Please refer to Document

Release Notes

  1. Added mxkv user-defined data type for kv storage, which facilitates the expansion of additional fields and has better performance than JSON type.
  2. Support Kafka data access and graphical operation interface.
  3. Provide partition automation management functions. Users can customize partition policies, and the background process automatically completes partition management.

4.0.3 Release Notes

  1. Add visual deployment of multi-machine installation.
  2. Fixed the MARS error in increasing partitions.

4.0.2 Release Notes

  1. MARS compression is enabled by default.
  2. After upgrading, supervisor.conf smoothly transition.
  3. Gapfill parallel optimization.

4.0.1 Release Notes

  1. Fixed the error that the column names were not escaped when building_timeseries_table.
  2. MARS engine optimization.

4.0.0 Release Notes

YMatrix Database 4.0.0 has been officially released, and the new version includes the following new features:

  1. MARS engine The MARS engine is a self-developed storage engine. The engine contains the following features:
    • Column storage
    • Data encoding and compression
    • Gathering push-down optimization
    • Optimization of indexless scanning

In timing scenarios, it is suitable for storing historical cold data, and the aggregation query performance based on timestamps is extremely high.

MARS currently only supports insertion, not updates and deletion.

  1. Multi-node automation deployment
    Based on the single-node automatic deployment of YMatrix Database 3 system, multi-node automatic deployment is realized. include:
    • Add cluster nodes
    • Configure the number of segments per node
    • Configure whether to enable automatic data mirroring

It takes less than 5 minutes to deploy a cluster.

For specific usage methods, please refer to [YMatrix Database 4 cluster deployment] (/doc/install/mx4_cluster). 3. Hot and cold storage UDF In order to facilitate users to establish partitions for timing tables, maintain partition tables, and convert hot and cold data, a complete set of management interfaces are implemented for creating partition templates, establishing partitions, and replacing hot and cold partitions.

For specific usage methods, please refer to [Hot and Cold Graded Storage] (/doc/4.0/datamodel/partition).

  1. time_bucket_gapfill
    In the timing component MatrixTS, the time_bucket_gapfill function is added based on the time_bucket function. Can be filled according to the rules for missing time intervals.

Currently two fill rules are supported:

  • locf: Fill with the values ​​​​​that appeared before in the aggregate group.
  • interpolate: Perform linear interpolation padding on missing values.