The goal of Redis Data Integration (RDI) Benchmark Tool is to produce INSERT statements on the source database and calculate the lag between the time a key is inserted into the source database and the time at which the same key is stored in the target Redis database.

Change record calculated latency

The databases that are supported by this tool are:

  • MySQL
  • Oracle
  • PostgreSQL
  • SQLServer

Prerequisites

  • An existing Redis Enterprise cluster version >= 6.2.
  • RedisGears >= 1.2.5 installed on the cluster. Note: RedisGears versions 2.0+ are not supported.
  • A Source database server with Bin Log enabled. The database servers that are supported by this tool are : MySQL, Oracle, PostgreSQL, and SQLServer.
  • A target Redis database.
  • A RDI database that is connected to the target Redis database.
  • Debezium Server up and running and connected to the source database and to the RDI database.
  • Oracle Instant Client installed and configured for Oracle source databases. In case it’s missing, follow this guide to install it.

RDI Benchmark requires connectivity to the following endpoints:

  • Redis Enterprise cluster admin API (port 9443).
  • RDI database (default port 12001).
  • The target Redis database (to which RDI is writing).
  • The source database to which data will be written.

Installation

Download

Ubuntu 20.04

wget https://qa-onprem.s3.amazonaws.com/redis-di/latest/redis-di-benchmark-ubuntu20.04-latest.tar.gz -O /tmp/redis-di-benchmark.tar.gz

Ubuntu 18.04

wget https://qa-onprem.s3.amazonaws.com/redis-di/latest/redis-di-benchmark-ubuntu18.04-latest.tar.gz -O /tmp/redis-di-benchmark.tar.gz

RHEL 8

wget https://qa-onprem.s3.amazonaws.com/redis-di/latest/redis-di-benchmark-rhel8-latest.tar.gz -O /tmp/redis-di-benchmark.tar.gz

RHEL 7

wget https://qa-onprem.s3.amazonaws.com/redis-di/latest/redis-di-benchmark-rhel7-latest.tar.gz -O /tmp/redis-di-benchmark.tar.gz

Install the RDI benchmark tool

Unpack the downloaded redis-di-benchmark.tar.gz into /tmp directory:

cd /tmp
sudo tar xvf /tmp/redis-di-benchmark.tar.gz -C /usr/local/bin/

Note: Non-root users should unpack to a directory with write permissions and run redis-di-benchmark directly from it.

Validate the install

Verify that redis-di-benchmark is installed by running:

redis-di-benchmark --help

Usage: redis_di_benchmark [OPTIONS] COMMAND [ARGS]...

Options:
--help Show this message and exit.

Commands:
bench

Running the RDI benchmark tool

Arguments

To get the list of arguments, run the following command on your terminal:

redis_di_benchmark.py bench --help
ArgumentDescriptionDefaultComments
rc-hostHost/IP of RDI databaselocalhost
rc-portPort of the RDI database12001
rc-passwordPassword for RDI database
cluster-api-portAPI port of Redis Enterprise cluster9443
cluster-hostHost/IP of Redis Enterprise clusterlocalhost
cluster-userRedis Enterprise cluster username
cluster-passwordPassword for the Redis Enterprise cluster username
source-db-typeType of Database (mysql|oracle|postgresql|sqlserver|)
source-db-hostHost/IP of the source databaselocalhost
source-db-portPort of the source database
source-db-nameThe name of the database from which to stream the changes.Do not pass for Oracle DB
source-db-schemaName of the schema from which to the stream the changes-Do not pass for MySQL DB
source-db-userUsername to use when connecting to the source database
source-db-passwordPassword to use when connecting to the source database
target-db-hostHost/IP of Redis target DB
target-db-portPort of Redis target DB
number-cdc-eventsNumber of INSERT statements the tool generates per each table50000
number-of-fieldsNumber of fields per each table5
no-bulkSingle INSERT into the source DB in each transactionTrue
stream-name-prefixThe prefix with which the name of the stream(s) will be createddata:<serverName>The value of <serverName> should be taken from the property debezium.source.database.server.name from Debezium’s application.properties file

Run the redis-di-benchmark tool

From your current directory, run:

redis-di-benchmark bench

Note: Non-root users should run the tool from the directory where the file redis-di-benchmark.tar.gz was extracted.

Output

For each key, RDI benchmark tool calculates the difference between the arrival time of the key to Redis target database and the time it was inserted to the source database. When all the keys arrive to the Redis target database, the tool displays the following statistics:

Statistics with input argument no-bulk = True

End-to-end latency:
	Minimum latency: 0.885s
	Maximum latency: 6.343s
	Mean latency: 3.822s
	Median latency:  3.809s
	Standard deviation:  1.046s

Statistics with input argument no-bulk = False:

End-to-end latency:
	Minimum latency: 1.670s
	Maximum latency: 3.875s
	Mean latency: 2.743s
	Median latency:  2.739s
	Standard deviation:  0.877s
Throughput:
	Number of tables: 1
	Number of fields per each table: 5
	Size of field 10B
	Number of row in each table: 5000
	Number of events per second: 1233 eps

Notes

  • When no-bulk = True, the tool inserts one record at a time.
  • When no-bulk = False, the tool inserts 5000 records at a time.

Install Oracle Instant Client

  • Create a dedicated Oracle folder:

    sudo mkdir /opt/oracle

  • Download and unzip Oracle Instant Client:

    cd opt/oracle
sudo wget https://download.oracle.com/otn_software/linux/instantclient/214000/instantclient-basic-linux.x64-21.4.0.0.0dbru.zip
sudo unzip instantclient-basic-linux.x64-21.4.0.0.0dbru

  • Install libaio:

    • Ubuntu 18.04, Ubuntu 20.04:

      sudo apt update
sudo apt install libaio1

    • RHEL 7, RHEL 8:

      sudo yum update
sudo yum install libaio

  • Install unixODBC:

    • Ubuntu 18.04, Ubuntu 20.04:

      sudo apt install unixodbc

    • RHEL 7, RHEL 8:

      sudo yum install unixODBC

  • Add and link system configuration for Oracle Instant Client path:

    sudo sh -c "echo /opt/oracle/instantclient_21_4 > /etc/ld.so.conf.d/oracle-instantclient.conf"
sudo ldconfig