TAGGED POSTS / cdc

OpenLogReplicator – CDC replication from Oracle to Redis

OpenLogReplicator – the first open-source transaction replicator (CDC) to Kafka now supports also Redis as a target.

Check the new released version 0.0.3 (compiled for x86_64). It requires additionally the hiredis library

To run:

export LD_LIBRARY_PATH=/opt/oracle/instantclient_11_2:/usr/local/lib:/opt/hiredis
Debug/OpenLogReplicator

Testing with Redis:

create table adam5(a numeric, b number(10), c number(10, 2), d char(10), e varchar2(10), f timestamp, g date);
alter table adam5 add constraint idx1 primary key(a);

insert into adam5 values(100, 999, 10.22, 'xxx', 'yyy', sysdate, null);
commit;
insert into adam5 values(101, 999, 10.22, 'xxx', 'yyy', sysdate, null);
commit;
insert into adam5 values(102, 999, 10.22, 'xxx', 'yyy', sysdate, null);
insert into adam5 values(103, 999, 10.22, 'xxx', 'yyy', sysdate, null);
commit;

Redis:

127.0.0.1:6379> KEYS *
1) "SYSTEM.ADAM5.\"100\""
2) "SYSTEM.ADAM5.\"103\""
3) "SYSTEM.ADAM5.\"101\""
4) "SYSTEM.ADAM5.\"102\""
127.0.0.1:6379> GET "SYSTEM.ADAM5.\"100\""
"\"100\",\"999\",\"10.22\",\"xxx       \",\"yyy\",\"2019-01-05T11:21:54\",NULL"

All transactions are replicated with consistency (MULTI/EXEC).

Enjoy!

Demo of OpenLogReplicator

OpenLogReplicator – first log-based open source Oracle to Kafka CDC replication

I have figured out that writing about somebody else’s software is boring. Why not create your own.

So here it is:

  • All code GPL v.3
  • Just C++ code – absolutely no Java/Python or any other interpreted language
  • Purely reading Oracle Redo Log from disk – zero additional load to the database instance
  • High performance architecture from the beginning – no lazy slow code
  • Minimum latency approac
  • Memory-based approach – no storing of intermediate files on disk

Currently is implemented:

  • Possible to compile for Linux x/64 only
  • Supported only Oracle 11.2.0.4
  • Replicated Redo Codes: just single row Insert (OpCode 11.2)
  • Supported full transactionality: begin, commit, rollback, savepoint
  • Supported types: numeric, char, varchar2, timestamp, date

If I have more time, more documentation will appear here.

How to run?

git clone https://github.com/bersler/OpenLogReplicator
vi Debug/makefile
make
cp OpenLogReplicator.json.example OpenLogReplicator.json
vi OpenLogReplicator.json
./OpenLogReplicator

Make sure that you set the proper paths for needed dependencies for libraries:

  • Oracle client 11.2.0.4
  • RapidJson library
  • The Apache Kafka C/C++ library

Make sure the JSON file contains proper information. Everything is very easy and logical.

For who are inpatient and don’t want to compile – here is release 0.0.3 compiled for Linux x64. To execute it run:

export LD_LIBRARY_PATH=/opt/oracle/instantclient_11_2:/usr/local/lib
Debug/OpenLogReplicator

Have fun, but please do not send me any complains about not working code. I will maybe write here some help&docs when I have time. I could of course add more functionality, but I didn’t have time. You have the code – you can do it by yourself!

Sample input is:

create table adam3(a numeric, b number(10), c number(10, 2), d char(10), e varchar2(10), f timestamp, g date);
insert into adam3 values(100, 999, 10.22, 'xxx', 'yyy', sysdate, null);
commit;

In Kafka you should have:

{"scn": "4856388", dml: [{"operation":"insert", "table": "SYSTEM.ADAM3", "after": {"A": "100", "B": "999", "C": "10.22", "D": "xxx       ", "E": "yyy", "F": "2018-12-10T21:27:42"}}]}

Please have in mind that only single row insert operations are now supported. insert .. select, insert append, etc. would not work. Just 1-row INSERT operations.

loading
×