CDC(Change Data Capture) using SSIS.

Now CDC helps in tracking the insert update and delete, but has its own way of denoting the insert update and delete.

1 - It indicates Delete

2 It indicates Insert

3 It indicates values before the Update

4 It indicates values after the Update

Does CDC do Incremental or Full Load?

CDC always does an Incremental load because it will track the Changes happening in the Database or the table and then load only those changes in the destination which is also called as Incremental load. This incremental load is very useful when we have large number of records and it is very difficult do the Full Load every time there is a Change in Data. Therefore CDC does Incremental Load.

What are the 2 different phases in the CDC load?

CDC load has two very important phases known as Phase 1 & Phase 2. Phase 1: Phase 1 is where we will load the data from the Source to the Destination one time only and these data will be the new loaded data and also the Changed data. This is also known as Snapshot loading because the data directly goes from the source to the destination. Phase 2: Phase 2 is where now the data will not be loaded from the source but now the data will be loaded from the CDC file which will have all the changes that are tracked like the inserts, updates, deletes. And then every time the changes will happen from that CDC file. Only for the First time the data will load from the source to the destination.

What is the importance of the cdc_states table?

Now a cdc_states table is created to make sure it keeps track of when our Phase 1 and Phase 2 has started and ended. It will keep a track of when the CDC process has started and when it has ended.

How is CDC different from SCD?

Now CDC and SCD both do almost the same function but there are some pros and cons which have to be considered when we are applying it in our project. Here is a small difference between the CDC and SCD.

If the source is SQL server then both CDC and SCD can be used. But if the source is non SQL server then CDC cannot be used because it is a part of SQL Server. If we want to keep History of the Changes with the Data the CDC cannot be used. For that we have to use the Type 1 & Type 2 of SCD. If we want to do the Incremental load the CDC can be used because it loads only those records that have been changed. If there are large records the CDC must be used because it does incremental load, whereas SCD cannot be used it does Full Load and for larger records Full Load will be Time Consuming.

Refer to below SSIS video which demonstrate start-up of the project: -