OpenLogReplicator – Delete row piece (OP 11.3) and Truncate table (OP 24.1)

The OpenLogReplicator has been a new release 0.0.8 with new functionality:

  • Added rowid field to JSON (Kafka target)
  • Support for OP code 11.3 – Delete Row Piece
  • Support for OP code 24.1 – DDL: Truncate table (only Kafka target)

Here is an example of usage. Let’s create a table:

CREATE TABLE SYSTEM.ADAM3(
  A NUMERIC
);

Next add to the CFG file in the tables section:

{"table": "SYSTEM.ADAM3"}

Next run OpenLogReplicator and execute some SQL:

INSERT INTO SYSTEM.ADAM3 VALUES(100);
COMMIT;
TRUNCATE TABLE SYSTEM.ADAM3;
INSERT INTO SYSTEM.ADAM3 VALUES(200);
COMMIT;
DELETE FROM SYSTEM.ADAM3;
COMMIT;

The following JSON message is sent to Kafka:

{"scn": "8849551", dml: [{"operation":"insert", "table": "SYSTEM.ADAM3", "rowid": "AAAIDRAAFAAAACFAAA", "after": {"A": "100"}}]}
 {"scn": "8849568", dml: [{"operation":"truncate", "table": "SYSTEM.ADAM3"}]}
 {"scn": "8849573", dml: [{"operation":"insert", "table": "SYSTEM.ADAM3", "rowid": "AAAIDSAAFAAAACFAAA", "after": {"A": "200"}}]}
 {"scn": "8849580", dml: [{"operation":"delete", "table": "SYSTEM.ADAM3", "rowid": "AAAIDSAAFAAAACFAAA", "before": {"A": "200"}}]}

I have just adjusted the formatting to make the JSON more readable. This is exactly what is expected. This functionality is available with the latest build (0.0.8). Stay tuned for next OP codes to be implemented too.

PS. I know, that the JSON attribute is named dml even though truncate is actually a ddl operation in Oracle, but it’s actually a matter of naming fields. Maybe it should be changed to something more generic…

PS2. You can notice that after running truncate table newly inserted rows contain a different Data Object ID in the rowid.

OpenLogReplicator – Insert multiple rows (OP 11.11)

The OpenLogReplicator has been extended to support also OP code 11.11 – Insert Multiple Rows.

OpenLogReplicator supports OP code 11.11 - insert multiple rows

This type of redo record is used when an INSERT is used with a set of rows to be inserted. For example with INSERT … SELECT … .

Here is an example of usage. Let’s create a table:

CREATE TABLE SYSTEM.ADAM5(
A NUMERIC,
B NUMBER(10)
C NUMBER(10, 2),
D CHAR(10),
E VARCHAR2(10),
F TIMESTAMP,
G DATE
);

Next add to the CFG file in the tables section:

    {"table": "SYSTEM.ADAM5"}

Next run OpenLogReplicator and execute some SQL:

INSERT INTO SYSTEM.ADAM5 
SELECT OBJECT_ID, DATA_OBJECT_ID, 1.0, STATUS, STATUS, CREATED, CREATED FROM ALL_OBJECTS WHERE ROWNUM <= 2;
COMMIT;

The following JSON message is sent to Kafka:

{"scn": "8568574", dml: [
{"operation":"insert", "table": "SYSTEM.ADAM5", "after": {"A": "20","B": "2","C": "1","D": "VALID ","E": "VALID","F": "2018-07-29T21:16:55","G": "2018-07-29T21:16:55"}},
{"operation":"insert", "table": "SYSTEM.ADAM5", "after": {"A": "46","B": "46","C": "1","D": "VALID ","E": "VALID","F": "2018-07-29T21:16:55","G": "2018-07-29T21:16:55"}}
]

I have just adjusted the formatting to make the JSON more readable. This is exactly what is expected. This functionality is available with the latest build (0.0.5). Stay tuned for next OP codes to be implemented too.

OpenLogReplicator – CDC replication from Oracle to Redis

OpenLogReplicator – the first open-source transaction replicator (CDC) to Kafka now supports also Redis as a target.

Check the new released version 0.0.3 (compiled for x86_64). It requires additionally the hiredis library

To run:

export LD_LIBRARY_PATH=/opt/oracle/instantclient_11_2:/usr/local/lib:/opt/hiredis
Debug/OpenLogReplicator

Testing with Redis:

create table adam5(a numeric, b number(10), c number(10, 2), d char(10), e varchar2(10), f timestamp, g date);
alter table adam5 add constraint idx1 primary key(a);

insert into adam5 values(100, 999, 10.22, 'xxx', 'yyy', sysdate, null);
commit;
insert into adam5 values(101, 999, 10.22, 'xxx', 'yyy', sysdate, null);
commit;
insert into adam5 values(102, 999, 10.22, 'xxx', 'yyy', sysdate, null);
insert into adam5 values(103, 999, 10.22, 'xxx', 'yyy', sysdate, null);
commit;

Redis:

127.0.0.1:6379> KEYS *
1) "SYSTEM.ADAM5.\"100\""
2) "SYSTEM.ADAM5.\"103\""
3) "SYSTEM.ADAM5.\"101\""
4) "SYSTEM.ADAM5.\"102\""
127.0.0.1:6379> GET "SYSTEM.ADAM5.\"100\""
"\"100\",\"999\",\"10.22\",\"xxx       \",\"yyy\",\"2019-01-05T11:21:54\",NULL"

All transactions are replicated with consistency (MULTI/EXEC).

Enjoy!

Demo of OpenLogReplicator

OpenLogReplicator – first log-based open source Oracle to Kafka CDC replication

I have figured out that writing about somebody else’s software is boring. Why not create your own.

So here it is:

  • All code GPL v.3
  • Just C++ code – absolutely no Java/Python or any other interpreted language
  • Purely reading Oracle Redo Log from disk – zero additional load to the database instance
  • High performance architecture from the beginning – no lazy slow code
  • Minimum latency approac
  • Memory-based approach – no storing of intermediate files on disk

Currently is implemented:

  • Possible to compile for Linux x/64 only
  • Supported only Oracle 11.2.0.4
  • Replicated Redo Codes: just single row Insert (OpCode 11.2)
  • Supported full transactionality: begin, commit, rollback, savepoint
  • Supported types: numeric, char, varchar2, timestamp, date

If I have more time, more documentation will appear here.

How to run?

git clone https://github.com/bersler/OpenLogReplicator
vi Debug/makefile
make
cp OpenLogReplicator.json.example OpenLogReplicator.json
vi OpenLogReplicator.json
./OpenLogReplicator

Make sure that you set the proper paths for needed dependencies for libraries:

  • Oracle client 11.2.0.4
  • RapidJson library
  • The Apache Kafka C/C++ library

Make sure the JSON file contains proper information. Everything is very easy and logical.

For who are inpatient and don’t want to compile – here is release 0.0.3 compiled for Linux x64. To execute it run:

export LD_LIBRARY_PATH=/opt/oracle/instantclient_11_2:/usr/local/lib
Debug/OpenLogReplicator

Have fun, but please do not send me any complains about not working code. I will maybe write here some help&docs when I have time. I could of course add more functionality, but I didn’t have time. You have the code – you can do it by yourself!

Sample input is:

create table adam3(a numeric, b number(10), c number(10, 2), d char(10), e varchar2(10), f timestamp, g date);
insert into adam3 values(100, 999, 10.22, 'xxx', 'yyy', sysdate, null);
commit;

In Kafka you should have:

{"scn": "4856388", dml: [{"operation":"insert", "table": "SYSTEM.ADAM3", "after": {"A": "100", "B": "999", "C": "10.22", "D": "xxx       ", "E": "yyy", "F": "2018-12-10T21:27:42"}}]}

Please have in mind that only single row insert operations are now supported. insert .. select, insert append, etc. would not work. Just 1-row INSERT operations.

Oracle GoldenGate Parallel Internals – my presentation from POUG 2018

Oracle GoldenGate Parallel Integrated Replicat – DML rearrangement in transactions

While preparing my presentation for POUG 2018 conference I have found some different behavior in Parallel Intgrated Replicat. This is a new Replicat type available from version 12.3. According to Oracle internal tests Parallel Replicat can be up to 5 times faster than Integrated Replicat in the same test case. I have done a number of tests with various options but one thing struck me. Every time the Replicat rearranges the order of DML operations. And there is no way to turn it off.

(more…)

Oracle GoldenGate data filtering with WHERE and FILTER parameters the right way

One of the most trivial tasks for data replication is to define a replica of the table with just a subset of the rows from the source table. For example we would like just to replicate the ACCOUNTS table with some type (one column has some defined value). If you want to replicate that using Oracle GoldenGate and the account type may be modified then this is one of the most difficult tasks to achieve… Let’s find out why.

(more…)

Oracle GoldenGate point in time source database recovery

Even though you would like to secure the database from any disaster with HA, still it might happen that the database gets corrupted and a recovery from backup is needed to recover the database. In a critical scenario you might need to use Point In Time Recovery (PITR).

Let’s focus on a scenario when the source database needs to be recovered from backup to an earlier point in time. How to resync a replication to the target  database  without the need of a  full initial load?

(more…)

Oracle GoldenGate Coordinated Replicat unsynchronized mode troubleshooting

According to the Oracle GoldenGate Coordinated Replicat documentation if the Replicat is stopped in an unclean manner the threads may be unsynchronized. Different threads may reach different checkpoint positions.

Let’s find out what what might be the consequences of this situation.

(more…)


- PAGE 1 OF 4 -

Next Page  

loading
×