Logs

Viewing system operation logs is a fundamental method for diagnosing cluster issues.

1 YMatrix Logs

YMatrix generates two types of logs:

  • Cluster management logs
  • Query operation logs

1.1 Cluster Management Logs

By default, cluster management logs are stored in the $HOME/gpAdminLogs directory. Different tools generate separate log files within this directory, such as mxstart_20210701.log, mxstop_20210701.log, etc.

The naming convention for log files is: {tool_name}_{date}.log. Example: gpconfig_20210723.log mxstart_20210701.log mxstart_20210708.log mxstop_20210701.log mxstop_20210723.log ......

Each log entry follows this format:
Date + Time + Tool Name + Hostname + Username + Log Level + Message. Example:

20230308:17:29:32:001644 mxstart:mdw:mxadmin-[INFO]:-Greenplum Catalog Version: '302012031'  
20230308:17:29:32:001644 mxstart:mdw:mxadmin-[WARNING]:-postmaster.pid file exists on Master, checking if recovery startup required  
......

Log levels include INFO, WARNING, and FATAL:

  • INFO: Normal informational messages.
  • WARNING: An unexpected condition occurred, but execution continues.
  • FATAL: A critical error has occurred, causing the process to stop.

When a tool fails, check the corresponding log file for WARNING and FATAL entries to diagnose the issue.

1.2 Query Operation Logs

Query operation logs are located under the log directory in the data directories of the master and segment nodes. The file name format is gpdb-{date}_{time}.csv. Example:

gpdb-2021-07-01_164653.csv  
gpdb-2021-07-01_164655.csv  
gpdb-2021-07-02_000000.csv  
......

These logs follow PostgreSQL logging format and help identify:

  • Query execution errors
  • Node failures

Each record includes: timestamp, username, process ID, connection ID, log level, and detailed error message.

Below is an example log entry when a user attempts to connect to a non-existent database:

2021-07-26 10:21:34.692106 CST,"mxadmin","postgggg",p10188,th-1455884160,"[local]",,2021-07-26 10:21:34 CST,0,con14,,seg-1,,,,sx1,"FATAL","3D000","database ""postgggg"" does not exist",,,,,,,0,,"postinit.c",1024,

Meaning of each field:

Log Content Description
2021-07-26 10:21:34.692106 CST Timestamp
mxadmin Username
postgggg Database name
p10188 Process ID
th-1455884160 Thread ID
[local] Connection method (local)
con14 Connection ID
seg-1 Segment ID
FATAL Log level
database "postgggg" does not exist Error message

2 MatrixGate Logs

MatrixGate logs are also stored in the gpAdminLogs directory. The file name format is matrixgate.{date}_{time}.log. Example:

matrixgate.2023-04-10_141001-2369.log

The log format is similar to other management tools:

20210726:13:47:17 matrixgate:mxadmin:mdw:010889-[INFO]:-Connected to Database, version PostgreSQL 12 (MatrixDB 5.0.0+enterprise) (Greenplum Database 7.0.0+dev.20438.g08c3e56457 build commit:08c3e5645733a58360aa681b717293ab9a916968) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9), 64-bit compiled on Mar 23 2023 11:45:21  
20210726:13:47:17 matrixgate:mxadmin:mdw:010889-[INFO]:-[Writer.Stream] Initialized with 100ms load interval and 10 prepared insert  
20210726:13:47:17 matrixgate:mxadmin:mdw:010889-[INFO]:-[Writer.Stream] Started, listening on :12120  
20210726:13:47:17 matrixgate:mxadmin:mdw:010889-[INFO]:-[Trans.Plain] Started, parallel level 8  
20210726:13:47:17 matrixgate:mxadmin:mdw:010889-[INFO]:-[Source.HTTP] http listening on :8086  

You can use the mxgate log command to view MatrixGate logs:

[mxadmin@mdw gpAdminLogs]$  mxgate log  
******************************************************
 __  __       _        _       ____       _  
|  \/  | __ _| |_ _ __(_)_  __/ ___| __ _| |_ ___  
| |\/| |/ _ \ '_ \| | '_ \ / _ \___ \ / _ \ '__/ __|  
| |  | | (_| | |_| |  | |>  <| |_| | (_| | ||  __/  
|_|  |_|\__,_|\__|_|  |_/_/\_\\____|\__,_|\__\___|  
  Version: v5.0.0 (git: HEAD 53e21dfa)  
  Your Copy is Licensed to: yMatrix.cn; 2023-06-23; any  
******************************************************
==> [pid:436] matrixgate.2023-04-10_141001-2369.log <==  
github.com/ymatrix-data/matrixgate/src/job.startUpPrepare  
    /home/runner/work/matrixdb-ci/matrixdb-ci/src/job/job.go:55  
github.com/ymatrix-data/matrixgate/src/server.(*Server).Start  
    /home/runner/work/matrixdb-ci/matrixdb-ci/src/server/server.go:84  
main.main  
    /home/runner/work/matrixdb-ci/matrixdb-ci/src/bin/mxgated/main.go:111  
runtime.main  
    /opt/hostedtoolcache/go/1.19.7/x64/src/runtime/proc.go:250  
runtime.goexit  
    /opt/hostedtoolcache/go/1.19.7/x64/src/runtime/asm_amd64.s:1594  
==> [pid:2369] matrixgate.2023-04-10_141001-2369.log <==  
streaming.go:741:2023-04-10:14:33:46.710 matrixgate:mxadmin:mdw:010889-[INFO]:-[Writer.Stream] Initialized with 100ms load interval and 10 prepared insert  
streaming.go:741:2023-04-10:14:33:57.230 matrixgate:mxadmin:mdw:010889-[INFO]:-[Writer.Stream] Started, listening on :12120  
streaming.go:741:2023-04-10:14:34:07.729 matrixgate:mxadmin:mdw:010889-[INFO]:-[Trans.Plain] Started, parallel level 8  
streaming.go:741:2023-04-10:14:34:18.239 matrixgate:mxadmin:mdw:010889-[INFO]:-[Source.HTTP] http listening on :8086  

Note!
Using the mxgate log command to view logs requires that the mxgate process is currently running. The displayed log content comes from the active process.