Faster Way to Resync Log Shipped SQL Server Database After Restore Failure

By: Mohammed Moinudheen || Related Tips:More >Log Shipping

Problem

I was woken up at 3:00 AM due to a pager alert on my phone regarding a SQL Server log shipping failure of a very large critical production database. On investigation, I noticed that the log shipping chain was broken and the restore job was failing as it was unable to find a missing transaction log backup. Is there any quick way by which I can re-sync log shipping again without having to reinitialize log shipping using a full SQL Server database backup ?

Solution

This tip assumes that you already have a log shipping configuration in your environment. If you are new to log shipping, you may refer thistip which has the detailed steps to set up log shipping.

In this tip we will see how to fix a log shipping failure by performing a restore using a differential backup instead of performing a full database backup and then restoring it on the secondary. This method saves us considerable time especially if the database is very big. We will also see in this demo where this method does not work, that is, scenarios where you may not be able to just restore using a differential backup.

We will be using two SQL Server instances for this demo. The database is called VLDB with log shipping set up successfully as shown.

Primary SQL instance: BOULCOTT\INST1 Secondary SQL instance: BOULCOTT\INST2

These are three jobs used for log shipping:

LSBackup_VLDB - creates the log backups on the primary LSCopy_BOULCOTT\INST1_VLDB - copies the log backups to the secondary LSRestore_BOULCOTT\INST1_VLDB - restores the log backups on the secondary

From the screenshot above, we can see that log shipping is working fine for the ‘VLDB’ database and all log shipping jobs Log Backup, Copy and Restore are running as expected.

SQL Server Database Backup Schedule

When dealing with very large databases, full database backups are usually performed on the weekends to avoid peak database usage during the week and differential backups are done every other day of the week.

Let's assume that in our scenario, we have the backup schedule as below for our VLDB database with log shipping.

Backup Job Schedule Full database backup Sunday 11:00 pm Differential backup Monday-Saturday 11:00 pm Transaction log backups Using Log Shipping job every 15 minutes Break SQL Server Log Shipping for Testing

Refer to these steps on how you can break log shipping. We will use these steps to break log shipping, so that you can simulate this tip using certain assumptions based on the time the log shipping breaks.

Perform a transaction log backup of the primary database as shown and then delete it from the backup folder.

BACKUP LOG VLDB
TO DISK = 'C:\Temp\LogShip\VLDB_1.trn'
WITH INIT, STATS, COMPRESSION

Once done, run the following jobs in sequence.

Job Name Server LSBackup_VLDB Primary LSCopy_BOULCOTT\INST1_VLDB Secondary LSRestore_BOULCOTT\INST1_VLDB Secondary

When you run the restore job you will get this error message, because of the deleted log backup file.

When you right click on the SQL instance and view the log shipping status report, you will see that the log shipping report status is still looking good. This is because the restore threshold is set to 45 minutes as shown below.

You can modify this threshold value to a lower or higher value. In this demo, we will reduce it to 5 minutes, so the log shipping failures alerts occur quickly. You can use this stored procedure. It needs to be run on the master database on the secondary server.

EXEC master.dbo.sp_change_log_shipping_secondary_database
@secondary_database = 'VLDB',
@restore_threshold = 5

Once this is run, the restore threshold gets reduced to 5 minutes as shown below.

Using these steps we can break a log shipping configuration, so we can work on fixing it.

Scenario 1: Log shipping breaks Tuesday at 10PM

You are alerted that there is a log shipping failure on a production database. From our earlier assumption, the following jobs should have completed successfully on the primary server.

Backup Job Schedule Status Full database backup Sunday 11:00 pm Completed Successfully Differential backup Monday 11:00 pm Completed Successfully Differential backup Tuesday 11:00 pm Completed Successfully

On investigation, you find that the job shipping failure appears to have occurred on Tuesday around 10 PM.

Let's simulate this scenario

First, perform a full backup of the VLDB database on the primary server using this script.

BACKUP DATABASE VLDB
TO DISK ='C:\Temp\LogShip\VLDB_1.BAK'
WITH INIT, STATS, COMPRESSION

Run the following jobs in sequence.

Job Name Server LSBackup_VLDB Primary LSCopy_BOULCOTT\INST1_VLDB Secondary LSRestore_BOULCOTT\INST1_VLDB Secondary

Next, perform a differential backup of the VLDB database on the primary server using this script.

BACKUP DATABASE VLDB
TO DISK ='C:\Temp\LogShip\VLDB_1.DIFF'
WITH DIFFERENTIAL

Run the following jobs in sequence.

Job Name Server LSBackup_VLDB Primary LSCopy_BOULCOTT\INST1_VLDB Secondary LSRestore_BOULCOTT\INST1_VLDB Secondary

Ensure that the log shipping jobs between the primary and secondary server are working as normal by referring to the log shipping status report as shown below.

Break the log shipping configuration by referring to the steps described earlier in the tip. Check the log shipping status report again to ensure that the log shipping configuration has alerted a failure. The alerting will display based on the restore threshold that is set. In our case, the restore threshold is set to 5 minutes.