TAGGED POSTS / OpenLogReplicator

OpenLogReplicator – chained row support

The OpenLogReplicator with the new release 0.2.0 has been extended to support chained rows.

A chained row can appear when there is a bigger DML operation or the table uses more than 254 columns.

Example showing that OpenLogReplicator can support tables containing more than 254 columns

This is an example of a table which uses 1000 columns. The table definition is:

CREATE TABLE SYSTEM.ADAM1(
  a000 numeric,
  a001 numeric,
  a002 numeric,
...
  a999 numeric
);

CFG definition file of OpenLogReplicator contains information to read DML changes for this table:

  "tables": [
     {"table": "SYSTEM.ADAM%"}]

Let’s test with the following DML:

insert into system.adam1(a000, a100, a999) values(0, 1, 2);
commit;

The following JSON message is sent to Kafka:

{"scn": "17817076", "xid": "0x0002.006.00002524", dml: [{"operation":"insert", "table": "SYSTEM.ADAM1", "rowid": "AAAJ49AABAAANH5AAD", "after": {"A000": "0", "A100": "1", "A999": "2"}}]}

You can also test the tool using table definition with big columns (many char(2000) for example).

OpenLogReplicator – Insert multiple rows (OP 11.11)

The OpenLogReplicator has been extended to support also OP code 11.11 – Insert Multiple Rows.

OpenLogReplicator supports OP code 11.11 - insert multiple rows

This type of redo record is used when an INSERT is used with a set of rows to be inserted. For example with INSERT … SELECT … .

Here is an example of usage. Let’s create a table:

CREATE TABLE SYSTEM.ADAM5(
A NUMERIC,
B NUMBER(10)
C NUMBER(10, 2),
D CHAR(10),
E VARCHAR2(10),
F TIMESTAMP,
G DATE
);

Next add to the CFG file in the tables section:

    {"table": "SYSTEM.ADAM5"}

Next run OpenLogReplicator and execute some SQL:

INSERT INTO SYSTEM.ADAM5 
SELECT OBJECT_ID, DATA_OBJECT_ID, 1.0, STATUS, STATUS, CREATED, CREATED FROM ALL_OBJECTS WHERE ROWNUM <= 2;
COMMIT;

The following JSON message is sent to Kafka:

{"scn": "8568574", dml: [
{"operation":"insert", "table": "SYSTEM.ADAM5", "after": {"A": "20","B": "2","C": "1","D": "VALID ","E": "VALID","F": "2018-07-29T21:16:55","G": "2018-07-29T21:16:55"}},
{"operation":"insert", "table": "SYSTEM.ADAM5", "after": {"A": "46","B": "46","C": "1","D": "VALID ","E": "VALID","F": "2018-07-29T21:16:55","G": "2018-07-29T21:16:55"}}
]

I have just adjusted the formatting to make the JSON more readable. This is exactly what is expected. This functionality is available with the latest build (0.0.5). Stay tuned for next OP codes to be implemented too.

OpenLogReplicator – CDC replication from Oracle to Redis

OpenLogReplicator – the first open-source transaction replicator (CDC) to Kafka now supports also Redis as a target.

Check the new released version 0.0.3 (compiled for x86_64). It requires additionally the hiredis library

To run:

export LD_LIBRARY_PATH=/opt/oracle/instantclient_11_2:/usr/local/lib:/opt/hiredis
Debug/OpenLogReplicator

Testing with Redis:

create table adam5(a numeric, b number(10), c number(10, 2), d char(10), e varchar2(10), f timestamp, g date);
alter table adam5 add constraint idx1 primary key(a);

insert into adam5 values(100, 999, 10.22, 'xxx', 'yyy', sysdate, null);
commit;
insert into adam5 values(101, 999, 10.22, 'xxx', 'yyy', sysdate, null);
commit;
insert into adam5 values(102, 999, 10.22, 'xxx', 'yyy', sysdate, null);
insert into adam5 values(103, 999, 10.22, 'xxx', 'yyy', sysdate, null);
commit;

Redis:

127.0.0.1:6379> KEYS *
1) "SYSTEM.ADAM5.\"100\""
2) "SYSTEM.ADAM5.\"103\""
3) "SYSTEM.ADAM5.\"101\""
4) "SYSTEM.ADAM5.\"102\""
127.0.0.1:6379> GET "SYSTEM.ADAM5.\"100\""
"\"100\",\"999\",\"10.22\",\"xxx       \",\"yyy\",\"2019-01-05T11:21:54\",NULL"

All transactions are replicated with consistency (MULTI/EXEC).

Enjoy!

Demo of OpenLogReplicator

loading
×