TAGGED POSTS / redo log

OpenLogReplicator – CDC replication from Oracle to Redis

OpenLogReplicator – the first open-source transaction replicator (CDC) to Kafka now supports also Redis as a target.

Check the new released version 0.0.3 (compiled for x86_64). It requires additionally the hiredis library

To run:

export LD_LIBRARY_PATH=/opt/oracle/instantclient_11_2:/usr/local/lib:/opt/hiredis

Testing with Redis:

create table adam5(a numeric, b number(10), c number(10, 2), d char(10), e varchar2(10), f timestamp, g date);
alter table adam5 add constraint idx1 primary key(a);

insert into adam5 values(100, 999, 10.22, 'xxx', 'yyy', sysdate, null);
insert into adam5 values(101, 999, 10.22, 'xxx', 'yyy', sysdate, null);
insert into adam5 values(102, 999, 10.22, 'xxx', 'yyy', sysdate, null);
insert into adam5 values(103, 999, 10.22, 'xxx', 'yyy', sysdate, null);

Redis:> KEYS *
1) "SYSTEM.ADAM5.\"100\""
2) "SYSTEM.ADAM5.\"103\""
3) "SYSTEM.ADAM5.\"101\""
4) "SYSTEM.ADAM5.\"102\""> GET "SYSTEM.ADAM5.\"100\""
"\"100\",\"999\",\"10.22\",\"xxx       \",\"yyy\",\"2019-01-05T11:21:54\",NULL"

All transactions are replicated with consistency (MULTI/EXEC).


OpenLogReplicator – first log-based open source Oracle to Kafka CDC replication

I have figured out that writing about somebody else’s software is boring. Why not create your own.

So here it is:

  • All code GPL v.3
  • Just C++ code – absolutely no Java/Python or any other interpreted language
  • Purely reading Oracle Redo Log from disk – zero additional load to the database instance
  • High performance architecture from the beginning – no lazy slow code
  • Minimum latency approac
  • Memory-based approach – no storing of intermediate files on disk

Currently is implemented:

  • Possible to compile for Linux x/64 only
  • Supported only Oracle
  • Replicated Redo Codes: just single row Insert (OpCode 11.2)
  • Supported full transactionality: begin, commit, rollback, savepoint
  • Supported types: numeric, char, varchar2, timestamp, date

If I have more time, more documentation will appear here.

How to run?

git clone https://github.com/bersler/OpenLogReplicator
vi Debug/makefile
cp OpenLogReplicator.json.example OpenLogReplicator.json
vi OpenLogReplicator.json

Make sure that you set the proper paths for needed dependencies for libraries:

  • Oracle client
  • RapidJson library
  • The Apache Kafka C/C++ library

Make sure the JSON file contains proper information. Everything is very easy and logical.

For who are inpatient and don’t want to compile – here is release 0.0.3 compiled for Linux x64. To execute it run:

export LD_LIBRARY_PATH=/opt/oracle/instantclient_11_2:/usr/local/lib

Have fun, but please do not send me any complains about not working code. I will maybe write here some help&docs when I have time. I could of course add more functionality, but I didn’t have time. You have the code – you can do it by yourself!

Sample input is:

create table adam3(a numeric, b number(10), c number(10, 2), d char(10), e varchar2(10), f timestamp, g date);
insert into adam3 values(100, 999, 10.22, 'xxx', 'yyy', sysdate, null);

In Kafka you should have:

{"scn": "4856388", dml: [{"operation":"insert", "table": "SYSTEM.ADAM3", "after": {"A": "100", "B": "999", "C": "10.22", "D": "xxx       ", "E": "yyy", "F": "2018-12-10T21:27:42"}}]}

Please have in mind that only single row insert operations are now supported. insert .. select, insert append, etc. would not work. Just 1-row INSERT operations.

Preventing archive log deletion before being processed by Oracle GoldenGate

Oracle GoldenGate just like Oracle Streams has a mechanism of preventing archive logs from being deleted before they are processed.

This mechanism is supported both in Classic and Integrated Extract mode. Although the latter one offers more functionality and control. This article brings in dept analysis what is happening in the database. (more…)

Oracle GoldenGate: What is ADD TRANDATA really doing?

In the process of data replication using Oracle GoldenGate the first step is assuring that all the needed information has been written to the database redo log. By default the database does not write all the information that might be needed by the replication process. That’s why some additional supplemental logging is needed.

According to the documentation supplemental logging is being added by the ADD TRANDATA command. A very good research about this subject has been made by Julian Dyke in the article about adding supplemental logging. But what does the  GoldenGate command actually do and which options should be used? (more…)