English

YMatrix vs ClickHouse SSB Benchmark Comparison Report

Test Overview

This benchmark test compares the performance of YMatrix and ClickHouse (referred to as CK) in single-table query and analytical scenarios using the Star Schema Benchmark (SSB). The results show that YMatrix outperforms CK by 24% and 37% under two different data scales.

SSB is a widely adopted star schema benchmark in the industry. It evolved from TPC-H and serves as a cost-efficiency evaluation standard for analytical workloads. The SSB benchmark defines 13 query scenarios that assess an analytical database’s overall performance across various dimensions.

As the analytics landscape has evolved, the original SSB design no longer fully aligns with modern data warehouse usage patterns. Apache Druid recently proposed a variant of SSB in their benchmark: aggregating multiple source tables into a single wide table before executing the SSB queries. This approach has also been adopted in ClickHouse's latest SSB benchmark results.

Test Environment

Hardware Configuration

Machine	vCPU	RAM	Bandwidth	EBS
AWS EC2, m5.8xlarge	64	128GB	10Gbps	gp3, iops = 3000, throughput = 125MB/s

Software Configuration

To closely replicate ClickHouse's latest benchmark results, YMatrix uses the same test environment and configuration as described in ClickHouse's official benchmark setup. The deployment is single-node, using YMatrix’s default cluster configuration on the above hardware—specifically, 6 MXSegment nodes.

Kernel version: 3.10.0-1127.19.1.el7.x86_64
OS version: CentOS Linux release 7.8.2003
YMatrix: Enterprise edition matrixdb-5.0.0.beta.2+enterprise-1. For cluster deployment details, refer to the YMatrix official documentation and the YMatrix SSB benchmarking tool.
ClickHouse: 22.7.2.15

Test Methodology

YMatrix adopted the same SSB benchmark model as ClickHouse, testing at two data scales: scale factor 100 and scale factor 1000 (data scale derived from TPC-H terminology).

The wide table contains approximately 600 million rows at scale factor 100, and 6 billion rows at scale factor 1000.
For the scale factor 100 test, ClickHouse’s result was taken from official published data. Since ClickHouse did not publish results for scale factor 1000, YMatrix engineers conducted the test independently using identical hardware specifications.

Test Procedure

Prepare the test environment
As described in the "Hardware Environment" section, initialize the virtual machine on AWS and deploy the YMatrix test cluster following the YMatrix official installation guide.

Download the SSB benchmark tool

git clone https://github.com/ymatrix-data/ssb.git

Note!
The YMatrix SSB benchmark tool is open-source, easy to use, and available for community testing.

Environment validation
```
./validate_environment.sh
```
Generate test data
Use -s 100 to generate data at scale factor 100.
```
./generate_data.sh -s 100
```
Import test data
The tool supports importing datasets of various scales. Use -s 100 to import the generated scale factor 100 dataset into YMatrix. The default import method is MatrixGate, which supports high-concurrency, distributed, streaming, and bulk loading. Both MatrixGate and COPY import modes are supported; use the -t option to switch between them.
```
./import_data.sh -s 100
```
Generate the wide table
YMatrix made necessary adjustments to the SSB schema and queries, then executed:
```
./generate_flat_table.sh -s 100
```
Run the SSB benchmark
```
./ssb.sh -s 100
```