Quantcast
Channel: CodeSection,代码区,SQL Server(mssql)数据库 技术分享 - CodeSec
Viewing all 3160 articles
Browse latest View live

Manipulate JSON in SQL Server

$
0
0

Sql Server2016 starts to support JSON data operation. But you still need to make sure the COMPATIBILITY_LEVEL >= 130 in case your database is restored/migrated from an older version.

Please check this link for the mapping of Sql Server version and Compatibility Level.

Check the compatilibity level:

select * from sys.databases

Update the value if your Sql Server supports:

ALTER DATABASE TestDb SET COMPATIBILITY_LEVEL = 130 -- if less than 130. Examples

Following is the sample data:

DECLARE @jsonVariable NVARCHAR(MAX) SET @jsonVariable = N'[ { "Order": { "Number":"SO43659", "Date":"2011-05-31T00:00:00" }, "AccountNumber":"AW29825", "Item": { "Price":2024.9940, "Quantity":1 } }, { "Order": { "Number":"SO43661", "Date":"2011-06-01T00:00:00" }, "AccountNumber":"AW73565", "Item": { "Price":2024.9940, "Quantity":3 } } ]' select @jsonVariable as Orders into SalesReportJson Functions OPENJSON

You can also trasform the JSON Data, and save it to table

SELECT SalesOrderJsonData.* into SalesReport FROM OPENJSON (@jsonVariable) WITH ( Number varchar(200) N'$.Order.Number', Date datetime N'$.Order.Date', Customer varchar(200) N'$.AccountNumber', Quantity int N'$.Item.Quantity' ) AS SalesOrderJsonData;

Or read from SalesReportJson table directly:

select Orders.* into SalesReport from SalesReportJson cross apply openjson(Orders) WITH ( Number varchar(200) N'$.Order.Number', Date datetime N'$.Order.Date', Customer varchar(200) N'$.AccountNumber', Quantity int N'$.Item.Quantity' ) as Orders

The SalesReport table will look like:

Number Date Customer Quantity SO43659 2011-05-31 00:00:00.000 AW29825 1 SO43661 2011-06-01 00:00:00.000 AW73565 3 ISJSON

It is used to check if the data is in a valid json format.

The following sql will return 1 :

select ISJSON(Orders) from SalesReportJson JSON_VALUE

JSON_VALUEis used to extract a scalar value from a JSON string. If the value is not a scalar value, the result will be NULL . In that case, you should use JSON_QUERY instead.

For example, if we want to get the first order’s order number from the Orders column of SalesReportJson , run the following sql:

select JSON_VALUE(Orders, '$[0].Order.Number') as OrderNumber, JSON_VALUE(Orders, '$[0].Item.Quantity') as ItemQuantity, JSON_VALUE(Orders, '$[0].Order') as FirstOrder -- this will be NULL from SalesReportJson

The result will be:

OrderNumber ItemQuantity FirstOrder SO43659 1 NULL JSON_QUERY

JSON_QUERY is used to extract object or list from a JSON string. For the previous example, we can use the JSON_QUERY to get the FirstOrder object:

select JSON_QUERY(Orders, '$[0].Order') as FirstOrder -- this will be an object from SalesReportJson

The result will be :

FirstOrder { “Number”:”SO43659”, “Date”:”2011-05-31T00:00:00” } JSON_MODIFY

It updates the value of a property in a JSON string and returns the updated JSON string.

For example, we update the item price of the first order:

update SalesReportJson set Orders=JSON_MODIFY(Orders, '$[0].Item.Price', 100.0)

The result will be:

[ { "Order": { "Number": "SO43659", "Date": "2011-05-31T00:00:00" }, "AccountNumber": "AW29825", "Item": { "Price": 100.0, "Quantity": 1 } }, { "Order": { "Number": "SO43661", "Date": "2011-06-01T00:00:00" }, "AccountNumber": "AW73565", "Item": { "Price": 2024.9940, "Quantity": 3 } } ] FOR JSON

with FOR JSON , You can also format query results as JSON.

select Number as [Order.Number], Date as [Order.Date], Customer as [Account], Quantity as [Item.Quantity] from SalesReport FOR JSON PATH, ROOT('Orders') -- use PATH here to make the result nested according to the dot syntax

The result json is :

{ "Orders": [ { "Order": { "Number": "SO436592", "Date": "2011-05-31T00:00:00" }, "Account": "AW298252", "Item": { "Quantity": 12 } }, { "Order": { "Number": "SO43661", "Date": "2011-06-01T00:00:00" }, "Account": "AW73565", "Item": { "Quantity": 3 } } ] }

The same sql but with AUTO mode:

select Number as [Order.Number], Date as [Order.Date], Customer as [Account], Quantity as [Item.Quantity] from SalesReport FOR JSON AUTO, ROOT('Orders') -- auto mode

The result will be:

{ "Orders": [ { "Order.Number": "SO436592", "Order.Date": "2011-05-31T00:00:00", "Account": "AW298252", "Item.Quantity": 12 }, { "Order.Number": "SO43661", "Order.Date": "2011-06-01T00:00:00", "Account": "AW73565", "Item.Quantity": 3 } ] }

When you join tables, columns in the first table are generated as properties of the root object. Columns in the second table are generated as properties of a nested object. The table name or alias of the second table is used as the name of the nested array. For example:

select 123 as Id, 'Jason' as FirstName, 'json@gmail.com' as Email into Person select 123 as PersonId, '1/10 High street' as HomeAddress into Address select * from Person join Address on Id = PersonId FOR JSON AUTO, Root('Users')

The result will be:

{ "Users": [ { "Id": 123, "FirstName": "Jason", "Email": "json@gmail.com", "Address": [ { "PersonId": 123, "HomeAddress": "1\/10 High street" } ] } ] } More on Arrays

We can use index to locate the item in an array. Actually when we use OPEN_JSON to read the array in a JSON string, it will output some other informations:

select Order2s.* from SalesReportJson cross apply openjson(Orders) as Order2s

The result will be:

key value type 0 { “Order”: { “Number”:”SO43659”, “Date”:”2011-05-31T00:00:00” }, “AccountNumber”:”AW29825”, “Item”: { “Price”:100.0, “Quantity”:1 } } 5 1 { “Order”: { “Number”:”SO43661”, “Date”:”2011-06-01T00:00:00” }, “AccountNumber”:”AW73565”, “Item”: { “Price”:2024.9940, “Quantity”:3 } } 5

The first column is key , which is the index of each item in the array, the column type indicate it’s an object value. with the key , we can do some conditional modification to some items in an array.


Polling in SQL Agent

$
0
0
Polling in SQL Agent

A fun question over on StackOverflow asked about using SQL Agent with SSIS to poll for a file's existence. As the comments indicate, there's a non-zero startup time associated with SSIS (it must validate the metadata associated to the sources and destinations), but there is a faster, lighter weight alternative. Putting together a host of TSQL ingredients, including undocumented extended stored procedures, the following recipe could be used as a SQL Agent job step.

If you copy and paste the following query into your favorite instance of SQL Server, it will execute for one minute and it will complete by printing the words "Naughty, naughty".

SET NOCOUNT ON; -- http://www.patrickkeisler.com/2012/11/how-to-use-xpdirtree-to-list-all-files.html DECLARE -- Don't do stupid things like adding spaces into folder names @sourceFolder varchar(260) = 'C:\ssisdata\Input' -- Have to use SQL matching rules, not DOS/SSIS , @fileMask sysname = 'SourceData%.txt' -- how long to wait between polling , @SleepInSeconds int = 5 -- Don't exceed 24 hours aka 86400 seconds , @MaxTimerDurationInSeconds int = (3600 * 0) + (60 * 1) + 0 -- parameter for xp_dirtree 0 => top folder only; 1 => subfolders , @depth int = 1 -- parameter for xp_dirtree 0 => directory only; 1 => directory and files , @collectFile int = 1 , @RC bigint = 0; -- Create a table variable to capture the results of our directory command DECLARE @DirectoryTree table ( id int IDENTITY(1, 1) , subdirectory nvarchar(512) , depth int , isFile bit ); -- Use our sleep in seconds time to generate a delay time string DECLARE @delayTime char(10) = CONVERT(char(10), TIMEFROMPARTS(@SleepInSeconds/60 /60, @SleepInSeconds/60, @SleepInSeconds%60, 0, 0), 108) , @stopDateTime datetime2(0) = DATEADD(SECOND, @MaxTimerDurationInSeconds, CURRENT_TIMESTAMP); -- Force creation of the folder EXECUTE dbo.xp_create_subdir @sourceFolder; -- Load the results of our directory INSERT INTO @DirectoryTree ( subdirectory , depth , isFile ) EXECUTE dbo.xp_dirtree @sourceFolder , @depth , @collectFile; -- Prime the pump SELECT @RC = COUNT_BIG(1) FROM @DirectoryTree AS DT WHERE DT.isFile = 1 AND DT.subdirectory LIKE @fileMask; WHILE @rc = 0 AND @stopDateTime > CURRENT_TIMESTAMP BEGIN -- Load the results of our directory INSERT INTO @DirectoryTree ( subdirectory , depth , isFile ) EXECUTE dbo.xp_dirtree @sourceFolder , @depth , @collectFile; -- Test for file existence SELECT @RC = COUNT_BIG(1) FROM @DirectoryTree AS DT WHERE DT.isFile = 1 AND DT.subdirectory LIKE @fileMask; IF @RC = 0 BEGIN -- Put our process to sleep for a period of time WAITFOR DELAY @delayTime; END END -- at this point, we have either exited due to file found or time expired IF @RC > 0 BEGIN -- Take action when file was found PRINT 'Go run SSIS or something'; END ELSE BEGIN -- Take action for file not delivered in expected timeframe PRINT 'Naughty, naughty'; END

If you rerun the above query, in a separate window, assuming you have xp_cmdshell enabled, firing the following query will create a file with the expected pattern. Instead, it'll print out "Go run SSIS or something"

DECLARE @sourceFolder varchar(260) = 'C:\ssisdata\Input' , @fileMask sysname = REPLACE('SourceData%.txt', '%', CONVERT(char(10), CURRENT_TIMESTAMP, 120)) DECLARE @command varchar(1000) = 'echo > ' + @sourceFolder + '\' + @fileMask; -- If you get this error --Msg 15281, Level 16, State 1, Procedure sys.xp_cmdshell, Line 1 [Batch Start Line 0] --SQL Server blocked access to procedure 'sys.xp_cmdshell' of component 'xp_cmdshell' because this component is turned off as part of the security configuration for this server. A system administrator can enable the use of 'xp_cmdshell' by using sp_configure. For more information about enabling 'xp_cmdshell', search for 'xp_cmdshell' in SQL Server Books Online. -- -- Run this --EXECUTE sys.sp_configure'xp_cmdshell', 1; --GO --RECONFIGURE; --GO EXECUTE sys.xp_cmdshell @command;

Once you're satisfied with how that works, now what? I'd likely set up a step 2 which is the actual running of the SSIS package (instead of printing a message). What about the condition that a file wasn't found? I'd likely use throw/raiserrror or just old fashioned divide by zero to force the first job step to fail. And then specify a reasonable number of @retry_attempts and @retry_interval.

SSIS source and destination components

$
0
0
Can SQL Server Compact be used as both Source and Destination in SSIS?

I'm wondering if SQL Server Compact Edition can be used as both a Source and Destination in an SSIS dataflow. I know I can setup a SQLMOBILE connection manager, and I've found some information that mentions using it as a Destination, but nothing on u

MongoDB spatial query on source and destination

I have a collection, name Events: Each document in Events collection has source and destination in lat-long. I would like to make a query on the Events collection and get only those events that are within some distance from source and within some dis

Using robocopies with source and destination as variables

Completely new to scripting, tried to find a solution through searching the web but I'm stumped, so asking for help please! I'm trying to use robocopy with a variable as both the source and destination, different one for each, but I just can't get th

Source and destination query with geospatial indexing mongodb mongomapper

I have source and destination fields in my mongodb collection named Flight. Both the fields are geospatially indexed and written in lat-long format. I am using mongomapper to query it from rails controller. I want to write a query like following. Res

What are the `source 'and' destination 'parameters in MPI_Cart_shift?

Here it is written that the output parameters of MPI_Cart_shift are ranks of source and destination processes. However, in this tutorial (code below) what is returned as the source process is later used in MPI_Isend to send messages. Anyone can clear

Automapper - Can it match only the existing properties in the source and destination objects?

I have a simple update function: public void Update(Users user) { tblUserData userData = _context.tblUserDatas.Where(u => u.IDUSER == user.IDUSER).FirstOrDefault(); if (userData != null) { Mapper.CreateMap<Users, tblUserData>(); userData = Mapper

How to get source and destination widgets with Eclipse Drag and Drop

In my DragSourceListener I receive events of type DragSourceEvent, which has the source widget, but not the destination widget. In my ViewerDropAdapter I receive events of type DropTargetEvent, which has the destination widget, but not the source wid

Why Android Gradle preDexDebug & ldquo; Source and destination must be different & rdquo; Construction has failed?

I have an Android application built with Android Studio 0.8.1 and facing the issue: Error:Execution failed for task ':app:preDexDebug'. > java.lang.IllegalArgumentException: Source C:\Users\mfedorov\AndroidStudioProjects\EPOS2\app\build\intermediates

How to backup source and destination zip files?

All sources are on windows OS, and destination backup is on Unix system (we are using Samba). My source repository is similar to : -Repository --Folder1 ---Files --Folder2 ---Files etc... I would like to get a destination similar to : -Repository --F

The documentation 'git diff' refers to 'source' and 'destination' but does not say which is the index

The documentation for 'git diff' refers to 'source' files (default output prefix is 'a/') and 'destination' files (default output prefix is 'b/'). My question is, does 'source' refer to the file(s) in the index (staging area) and 'destination' the fi

Should I put a slash after the source and destination when copying folders

I want to copy this folder:ajax(/home/thej/public_html/JC/ajax) into this folder: /home/thej/public_html/demo/conf/ the final result will be /home/thej/public_html/demo/conf/ajax, I know the cp command should be something like: cp -r /home/thej/publi

Does Windows.CopyFile create a temporary local file while the source and destination are network shares?

I have a D2007 application that uses Windows.CopyFile to copy MS Word and PowerPoint files from one network folder to another network folder. Our organization is migrating to Windows 7 from Vista. One of my migrated users got an error message that di

Same source and destination register in ldr Is it legal to do ldr r0,[r0] in ARM assembly?When in doubt, always refer to the ARM Architecture Reference Manual, which can be found on arm.com. It says: The destination register. The SP can be used. The PC can be used, provided the instruction is

How do I copy multiple files at once in linux? The source and destination locations of these files being the same directory

I have some files located in one directory /home/john I want to copy all the files with *.text extension from this directory and save them as *.text.bkup, again in the same directory, i.e. /home/john Is there a single command with which I can do that

Dynamic Snapshots in SQL Server Merge Replication

$
0
0
Problem

We have a very large SQL Server database and we need to configure SQL Server merge replication on the database. Subscribers are located at remote locations and connectivity is poor. Creating a snapshot and applying the whole snapshot is taking a lot of time.

Solution

Applying the snapshot from the default location was taking too much time, so we zipped the snapshot and moved it to the remote subscribers and applied the snapshot locally by specifying the alternate snapshot folder in the subscription properties. But, we want to limit data at the snapshot level itself, how can this be done?

In our case, the publisher holds all of the data and the subscriber holds specific data for that location. So, we used filters on the articles and generated dynamic snapshots for each subscriber.

SQL Server Dynamic Snapshots

Dynamic snapshots are created based on filters. Each subscriber will have its own snapshot and data specific to it.

Before generating dynamic snapshots, the below steps need to be performed:

Create the merge publication with at least one or more parameterized row filters. Generate a schema snapshot for the publication.

Now we can generate dynamic snapshots manually or automatically.

Setup SQL Server Merge Replication in Management Studio

For this tip I created a sample database on the same server and simulated as if it was a remote server. Replication was also enabled for this server.

Login to SQL Server and create databases PUB, SUB1 and SUB2. Create table TEST in the PUB database and insert the below sample data. CREATE TABLE TEST (ID INT NOT NULL, NAME VARCHAR(MAX), ORGID VARCHAR (50))
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (1, N'Adam', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (2, N'Anil', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (3, N'Bob', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (4, N'Mary', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (5, N'Paul', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (6, N'Scott', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (7, N'Atul', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (8, N'Uday', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (9, N'Anush', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (10, N'Amir', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (11, N'Ben', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (12, N'Brown', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (13, N'Robin', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (14, N'Rehman', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (15, N'Ravi', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (17, N'Anil', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (18, N'Bob', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (19, N'Mary', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (20, N'Paul', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (21, N'Scott', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (22, N'Atul', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (23, N'Uday', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (24, N'Anush', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (25, N'Amir', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (26, N'Ben', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (27, N'Brown', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (28, N'Robin', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (29, N'Rehman', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (30, N'Ravi', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (16, N'Adam', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (31, N'Adam', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (32, N'Anil', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (33, N'Bob', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (34, N'Mary', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (35, N'Paul', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (36, N'Scott', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (37, N'Atul', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (38, N'Uday', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (39, N'Anush', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (40, N'Amir', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (41, N'Ben', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (42, N'Brown', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (43, N'Robin', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (44, N'Rehman', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (45, N'Ravi', N'ORG1')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (46, N'Adam', N'ORG2')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (47, N'Anil', N'ORG2')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (48, N'Bob', N'ORG2')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (49, N'Mary', N'ORG2')
GO
INSERT [dbo].[TEST] ([ID], [NAME], [ORGID]) VALUES (50, N'Paul', N'ORG2')
GO
ALTER TABLE [dbo].[TEST] ADD CONSTRAINT [PK_ID] PRIMARY KEY CLUSTERED ([ID] ASC ) ON [PRIMARY]
GO

Navigate to Replication > Local Publications and right click New Publication and the wizard will open as shown below.


Dynamic Snapshots in SQL Server Merge Replication

Click Next. Then select the PUB database and click Next. Select Merge publication and click Next. Check SQL Server 2008 or later and click Next.

Expand Tables and select the TEST table as shown below and click Next.


Dynamic Snapshots in SQL Server Merge Replication

A message will show that a unique identifier will be added to the table. Click Next on this screen.

On the Filter Table Rows screen, click the Add button and select Add Filter.


Dynamic Snapshots in SQL Server Merge Replication

Add the below filter in the filter statement. Also, select the A row from this table will go to only one subscription and then click OK.

SELECT <published_columns> FROM [dbo].[TEST] WHERE ORGID=HOST_NAME()

The screen will then show the filter that was just created and then click Next.

Select Create a snapshot immediately and click Next.

Specify the Snapshot Agent security settings and click Next.

Click on Create the publication and click Next.

Enter Publication name as DYNAMICSNAP or any name you want and select Finish.

We created the publication and generated the schema snapshot. Now we need to generate dynamic snapshots.

Creating Dynamic Snapshots in SQL Server Management Studio

In our case we are copying snapshots to a remote subscriber and applying it locally using the alternate snapshot folder. So, I am generating dynamic snapshots manually.

Go to Replication > Local Publications and right click on the publication we just created and selec

How to Perform an Online Page Level Restore in SQL Server

$
0
0
Problem

I have a SQL Server database inFULL recovery model. Our nightly database integrity checks were successful, afull database backup was completed and then an outage occurred which corrupted a couple of data pages in the database. We want to perform a page level restore on our live database from the full database backup. Are all objects still accessible when the page restore operation takes place in SQL Server?

Solution

An online page level restore is only available in SQL Server Enterprise and equivalent, such as Developer and Evaluation Edition. During the online page restore operation, users can still access all other objects in the database except the corrupted pages being restored. In the situation when you only have a latest known good full backup prior to the corruption, this backup is sufficient to be used for page level restore operations.

Even when your database is not on Enterprise\Developer\Evaluation Edition, the same steps can be used with the exception that the database needs to be OFFLINE during the page restore operation.

Create a Test SQL Server Database

We will create a test database using the script below and at the end of the script we will perform a full database backup.

The SQL Server version used in this tip is SQL Server 2016 Developer Edition, which has all the features and functions of SQL Server Enterprise Edition.

-- Update and specify a valid backup path at the end of the script
USE master
GO
DROP DATABASE IF EXISTS [CorruptionTest]
GO
CREATE DATABASE [CorruptionTest]
GO
ALTER DATABASE [CorruptionTest] SET RECOVERY FULL;
GO
ALTER DATABASE [CorruptionTest] SET PAGE_VERIFY CHECKSUM
GO
CREATE TABLE [CorruptionTest].[dbo].[mssqltips_online]
(increment INT, randomGUID uniqueidentifier, randomValue INT, BigCol CHAR(2000) DEFAULT 'a',
INDEX CIX_MSSQLTips_increment1 UNIQUE CLUSTERED (increment))
GO
CREATE TABLE [CorruptionTest].[dbo].[mssqltips_corrupt]
(increment INT, randomGUID uniqueidentifier, randomValue INT, BigCol CHAR(2000) DEFAULT 'a',
INDEX CIX_MSSQLTips_increment1 UNIQUE CLUSTERED (increment))
GO
SET NOCOUNT ON;
DECLARE @counter INT = 1;
BEGIN TRAN
WHILE @counter <= 250000
BEGIN
INSERT INTO CorruptionTest.dbo.mssqltips_online (increment, randomGUID, randomValue)
VALUES (@counter, NEWID(), ABS(CHECKSUM(NewId())) % 140000000)
INSERT INTO CorruptionTest.dbo.mssqltips_corrupt (increment, randomGUID, randomValue)
VALUES (@counter, NEWID(), ABS(CHECKSUM(NewId())) % 140000000)
SET @counter += 1
END;
COMMIT TRAN;
GO
DBCC CHECKDB('CorruptionTest') WITH NO_INFOMSGS
GO
BACKUP DATABASE [CorruptionTest] TO DISK = 'D:\BACKUP\CorruptionTest_Full.BAK' WITH COMPRESSION
GO

The query execution below outputs the top 10 rows with the page id and its slot number. We will (randomly) choose to corrupt page id 282 and 283 by overwriting the 2 highlighted values with 0x0.

SELECT TOP 10
sys.fn_PhysLocFormatter(%%physloc%%) PageId,
*
FROM [CorruptionTest].[dbo].[mssqltips_corrupt]
GO
How to Perform an Online Page Level Restore in SQL Server

Using the undocumented DBCC WRITEPAGE below, we will corrupt the value in the 2 mentioned columns to all zeros. If you wish to learn how the input parameters are calculated, refer to my previous tip on using undocumented DBCC WRITEPAGE to instigate SQL Server database corruption to understand how to use the command. In this demonstration, you can try to corrupt any data page id and we will recover it from the full database backup.

USE master;
GO
ALTER DATABASE [CorruptionTest] SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
GO
DBCC WRITEPAGE ('CorruptionTest', 1, 282, 4182, 4, 0x00000000, 1)
GO
DBCC WRITEPAGE ('CorruptionTest', 1, 283, 4166, 16, 0x00000000000000000000000000000000, 1)
GO
ALTER DATABASE [CorruptionTest] SET MULTI_USER;
GO Start a Control Query The preparation work is now done. We will execute the query below in new Query windows and leave the query running. It will continuously scan all the rows in table [dbo].[mssqltips_online]. This only serves as a control query to make sure the connectivity at the database level is always ONLINE when we perform the page level restore operation. SELECT @@SPID SESSION_ID
GO
DECLARE @COUNT INT
WHILE 1 = 1
BEGIN
SELECT @COUNT = COUNT(*)
FROM [CorruptionTest].[dbo].[mssqltips_online]
END
GO
How to Perform an Online Page Level Restore in SQL Server
We have corrupted 2 data pages, 282 and 283 in table [dbo].[mssqltips_corrupt]. When we execute the query below against [dbo].[mssqltips_corrupt], it will fail with an error message as expected. SELECT COUNT(*) FROM [CorruptionTest].[dbo].[mssqltips_corrupt]
GO
How to Perform an Online Page Level Restore in SQL Server
SQL Server Page Level Restore

We will now execute the steps to perform an online page level restore from our good known full database backup.

Right-click on database [CorruptionTest] > Tasks > Restore > Page…
How to Perform an Online Page Level Restore in SQL Server
In the Restore Page, click the [Check Database Pages] button. This will perform database integrity checks in the background and automatically populate the corrupted Page IDs that needs to be restored. In our demonstration, the list matches the 2 Page IDs that we have corrupted.
How to Perform an Online Page Level Restore in SQL Server
Click on the [Verify] button, this will verify the backup media to make sure the backup set is complete and readable. We can click [OK] to start the page restore from the [Restore Page] GUI here, but let’s investigate further and script out the command.
How to Perform an Online Page Level Restore in SQL Server
The scripted-out command from the [Restore Page] form contains the three commands as below: USE [master]
RESTORE DATABASE [CorruptionTest] PAGE='1:282, 1:283' FROM DISK = N'D:\Backup\CorruptionTest_Full.BAK' WITH FILE = 1, NORECOVERY, NOUNLOAD, STATS = 5
BACKUP LOG [CorruptionTest] TO DISK = N'D:\Backup\CorruptionTest_LogBackup_2018-09-03_20-25-42.bak' WITH NOFORMAT, NOINIT, NAME = N'CorruptionTest_LogBackup_2018-09-03_20-25-42', NOSKIP, NOREWIND, NOUNLOAD, STATS = 5
RESTORE LOG [CorruptionTest] FROM DISK = N'D:\Backup\CorruptionTest_LogBackup_2018-09-03_20-25-42.bak' WITH NOUNLOAD, STATS = 5
GO

The 3 commands can be described as performing the steps below:

Step 1: Perform restore of pages from full database backup file with NORECOVERY Step 2: Perform a backup of (tail) transaction log Step 3: Perform a restore from the (tail) transaction log backup

We will go through Step 1 and Step 2 - 3 separately to understand its impact individually.

Step 1: Restore page from database with NORECOVERY

This restore command only restores specific pages from the full database backup. It does not disrupt any queries executing against the database. As we would expect, the control query (session 59) is still executing.


How to Perform an Online Page Level Restore in SQL Server
The pages restore operation completed. When we execute the query below, it prompts a different error message stating page id 282 is inaccessible because it is in RestorePending. Reason it didn’t prompt page id 283 is inaccessible because the query terminates as it accesses page id 282 first due to the logical ordering of the table enforced by the clustered index on column [increment]. SELECT COUNT(*) FROM [CorruptionTest].[dbo].[mssqltips_corrupt]
GO
How to Perform an Online Page Level Restore in SQL Server

To also prove the point all objects in the database are accessible except the corrupted pages being restored, we will execute the query below which does returns the row count without error. Note that increment 3 and 10 is part of the non-corrupted page. The in-depth explanation is not covered in this tip, but it has to do with the way the SQL Server storage engine accesses data in different situations.

SELECT COUNT(*)
FROM [CorruptionTest].[dbo].[mssqltips_corrupt]
WHERE increment NOT BETWEEN 3 AND 10
GO
How to Perform an Online Page Level Restore in SQL Server
Step 2 & 3: Perform a tail-log backup and Restore of the tail log

Step 2 is an important and a required step. If you have taken any other transaction log backups after the full backup, you will need to restore them in sequence prior to this this step. Then finally the last step would always be to restore the tail log backup.

We only have the full database backup prior. Hence, we will execute Step 2 and 3 in a single batch. Note that the restore tail log does not specify NORECOVERY. So the restored pages will be brought ONLINE when the restore log command completes.


How to Perform an Online Page Level Restore in SQL Server

During this log backup and restore process, the control query (session 59) is still executing. This ensures the database connectivity was ONLINE and available throughout the page restore process.

We now execute the query below which will scan all the rows in table [dbo].[mssqltips_corrupt] and this query executes successfully. We have now completed our page restore operation and fixed the page corruption.
How to Perform an Online Page Level Restore in SQL Server

To doubly make sure the corruption is resolved, we perform a database integrity checks. The command executed successfully without error.


How to Perform an Online Page Level Restore in SQL Server
Summary

SQL Server has made it quite easy to perform page level restore using the SSMS GUI. This feature is especially useful in the situation where few data pages are corrupted in a very large database. By performing page level restores, this can significantly reduce the recovery time objective for the database.

But there is a caveat. In this tip, we have performed the full database backup on the same SQL Server instance. Hence SQL Server was able to automatically map the full database backup to the GUI. Otherwise, you will need to rely on some toolset or script to generate the same commands for the restores. Nevertheless, knowing the required sequence and with practice, you can still achieve the same outcome with accuracy and ease before real disaster strikes.

Next Steps Restore Pages (SQL Server) Online Restore (SQL Server) Checking to make sure a SQL Server backup is useable

Last Update: 2018-10-19


How to Perform an Online Page Level Restore in SQL Server
How to Perform an Online Page Level Restore in SQL Server
About the author
How to Perform an Online Page Level Restore in SQL Server
Simon Liew is an independent SQL Server Consultant in Sydney, Australia. He is a Microsoft Certified Master for SQL Server 2008 and holds a Masters Degree in Distributed Computing. View all my tips

Related Resources

More SQL Server DBA Tips...

SQL Server Reading BACPAC Files

$
0
0

If you have been reading my blog for a while now you would know that a common technique to move to Azure SQL DB is to use BACPAC files. Just a reminder, see the below image.


SQL Server   Reading BACPAC Files

I had a bacpac file but I forgot about the contents, partly because of the bad naming convention, test.bacpac isnt great right? However for whatever reason you may want to know what tables are within the bacpac, this being the main point of the post!

If you do a little “editing” with file extensions you will be able to see what the bacpac file contains. Let’s get to it. As you can see below I have a test.bacpac file.

Step 1 Take a copy of the file and change the extension to .ZIP.


SQL Server   Reading BACPAC Files

Step 2 Extract the ZIP file to whatever location you desire.


SQL Server   Reading BACPAC Files

Step 3 View the folder structure and navigate to the Data folder.


SQL Server   Reading BACPAC Files

Step 4 The table list is here.


SQL Server   Reading BACPAC Files

Very cool right? Just to prove it is the correct list this is my view of the tables via SSMS.


SQL Server   Reading BACPAC Files

Internal usage of TEMPDB by SQL Server DB engine

$
0
0

This post is actually a continuation post for TEMPDB the most important system database in SQL Server which trying to cover one of the important usage of TEMPDB in SQL Server. Some operations in SQL Server uses TEMPDB internally to improve the performance of the operations. It may not be fully aware for the users, but we can understand the usage of tempdb using the below query.

<strong>--Query to understand the internal usage of TEMPDB</strong>
select
reserved_MB=(unallocated_extent_page_count+
version_store_reserved_page_count+
user_object_reserved_page_count+
internal_object_reserved_page_count+
mixed_extent_page_count)*8/1024. ,
unallocated_extent_MB =unallocated_extent_page_count*8/1024.,
internal_object_reserved_page_count,
internal_object_reserved_MB =internal_object_reserved_page_count*8/1024.
from sys.dm_db_file_space_usage

Few of operations as below:

1. DBCC CHECKDB/CHECKALLOC

DBCC CHECKDB/CHECKALLOC are using TEMPDB space for creating internal database snapshot to perform the operation. This is to efficiently avoid the locking behavior on database to obtain the consistency check. DBCC CHECKDB and CHECKALLOC has an option to estimate the usage of TEMPDB by providing ESTIMATEONLY option as below. Please note, this is an estimated value, may not be the correct one, however, this can be a good indication to estimate the space required for the operation. This can be used to make sure the TEMPDB drive has enough space to run on DBCC operations for large databases.


Internal usage of TEMPDB by SQL Server DB engine
2. SORT operations can spill over to TEMPDB

When a query is executed, SQL optimizer will choose the plan already created if prsent or will create a new plan based on cost based algorithms. When optimizer creates a plan, SQL optimizer/relational engine will identify the right operator for the plan to execute the queries and it estimates the memory required to execute the query. This plan will be used for further execution. If the estimation is not happening correctly because of wrong statistics or parameters used while creating the plan is returning less number of records, the estimated number of records will be deviating from the actual number of records while executing the query. In such scenario, the memory granted for the execution may not be sufficient for the execution.

If SORT operator needs more memory to sort the data , other words, the memory granted is not sufficient to do the sorting, it will spill over to TEMPDB. This will have a performance impact for the query execution. The spill over to TEMPDB can be observed in the profiler as SORT warnings(below).


Internal usage of TEMPDB by SQL Server DB engine

In the above snapshot, we can see Sort warnings in profiler with Event Subclass as single or multiple. Whenever the sort operation spills to tempdb, SQL Server raises the ‘Sort Warnings’ event and it takes single or multiple passes to tempdb.

As mentioned, Sort warnings or spilling to TempDB will have some detrimental impact on the query performance, we need to identify those queries and avoid if possible. Through profiler, we may not understand the query is being caused the Sort Warnings, but, we need to identify from the cached plan or by setting up Extended events to capture the Sort warnings.

Once we identified the query caused Sort Warning, the easiest solution would be re-write the query in a manner to avoid the sorting. I have seen queries with CTE using the ORDER BY even though the order by is not required specifically. Try to use ORDER BY genuinely to avoid performance issues. At the same time, it may not be easy for all cases to avoid the ORDER BY completely. Similar scenario, we may need to evaluate adding a supporting index or modify the existing indexes to avoid, but again, this needs more careful and clear understanding of code and index usage. If you are well aware of the reason for Sort warnings are due to incorrect statistics, you can update the stats or use the solutions to avoid the parameter sniffing(As the objective of this post is not going to explain the parameter sniffing, this post does not cover the topic now.).

3. Worktables/intermediate temp objects due to spooling/hash joins/aggregate Operations

Spools are special operators created by SQL optimizer to improve the performance of a query. A spool operator is not an independent operator,but a supporting operator for another operator like clustered index scan, Table scan or even Constant scan. A spool operator reads and stores intermediate “operated” data into TEMPDB from another operator, there by, increasing the performance of the query.

In the below snapshot, we can observe that the spool operator stores the data from the input, here its nothing but constant scan into a temporary tables to avoid multiple rewinds.


Internal usage of TEMPDB by SQL Server DB engine

There are totally five types of spool operators Eager, Lazy, Table, RowCount and Non-Clustered Index spools. All spool operators will store the data into TEMPDB, however, it may be different the way it behaves. Please explore further on the topic to know more details.

Hope you enjoyed this post, please post your feedback/thoughts in the comments.

SQL Server on Amazon Web Services (AWS)

$
0
0
Introduction

In this article we will discuss options to make Microsoft SQL Server (MS SQL Server) available on Amazon Web Service (AWS). AWS is the largest cloud service provider and lots of organizations have been leveraging AWS as their cloud service. Organizations are migrating on premise legacy applications to the cloud as well as building new business critical applications, using the latest technology suites in the cloud to support their businesses. In such cases, it is required for organizations to explore the options of availability of Microsoft SQL Server on AWS for the applications that have been using Microsoft SQL Server as relational database.

Microsoft SQL Server Availability on AWS

There are multiple ways to make Microsoft SQL Server available on AWS. Amazon provides Relational Database Service (RDS) and Amazon Elastic Compute Cloud (EC2) to run Microsoft SQL Server Databases on AWS. These two approaches have their own advantages to maintaining MS SQL Server databases on AWS.

Relational Database Service (RDS), is a manage service that is easy to set up, maintain and scale a Microsoft SQL Server Database on AWS. AWS RDS takes care of the end to end management of MS SQL Server from installation, disk provisioning, version upgrades, security patching to back up and recovery of MS SQL Server databases. Also, AWS RDS supports high availability of databases and scalability of the environment automatically.

Amazon Elastic Compute Cloud (EC2), is a service to provide computing capacity on AWS cloud. Amazon provides the capability to run MS SQL Server on Amazon EC2 like running an MS SQL Server on premise data center. In this scenario, organizations will have maximum control and flexibility to configure MS SQL Server with additional responsibilities to manage the deployed MS SQL Server and databases.

Microsoft SQL Server on Amazon RDS

Amazon RDS is a managed service and supports MS SQL Server availability on Amazon cloud. Amazon RDS supports certain features and options of MS SQL Server like Core database engine features, MS SQL Server Development Tools, MS SQL Server Management Tools, Change Tracking, Service Broker, Spatial and location features. Also, there are certain security features available with RDS like Transparent Data Encryption (TDE), Secure Socket Layer (SSL) connection, Encryption of storage at rest using Amazon Key Management Service to secure SQL Server databases and data.

Amazon RDS manages high availability of MS SQL Server and databases using multi-availability zone capability (Multi-AZ). This reduces the organization’s overhead to manually set up and maintain database mirroring, failover clusters, or Always On Availability Groups to achieve high availability.

Amazon RDS manages provisioning of the database, management of security patches, version upgrades of MS SQL Server and disk storage management.

Amazon RDS gives an opportunity to organizations to focus on high-level tasks, such as performance tuning and schema optimization and RDS manages backups and point-in-time recovery of the database in the event of a crash and failure.

There are certain limitations with Amazon RDS, which are important to explore before organizations choose Amazon RDS as an option to deploy MS SQL Server. Some of the key limitations are as follows:

Availability of up to 30 databases on each of MS SQL Server database instance Amazon RDS doesn't support other MS SQL Server services like SQL Server Analysis Services (SSAS), SQL Server Integration Services (SSIS), SQL Server Reporting Services (SSRS), Data Quality Services (DQS), or Master Data Services (MDS) on the same server as Amazon RDS MS SQL Server DB instance Limitation on use of certain ports, which are reserved for Amazon RDS Maximum storage size for MS SQL Server database instance is 16 TB for General Purpose SSD storage Many server-level roles are not available like sysadmin, serveradmin, bulkadmin, dbcreator, securityadmin and diskadmin Some of the following key features of MS SQL Server are not available with Amazon RDS: Always On Backing up to Microsoft Azure Blob Storage BULK INSERT and OPENROWSET(BULK...) features Database Log Shipping Distributed Queries (i.e., Linked Servers) Distribution Transaction Coordinator (MSDTC) File tables Maintenance Plans PolyBase Replication Server-level triggers Service Broker endpoints Stretch database

You can refer to the AWS portal to get the comprehensive list of MS SQL Server and database limitations on Amazon RDS.

Microsoft SQL Server on Amazon EC2

Availability of MS SQL Server on Amazon EC2 gives full control over the environment. Organizations can have full control over the operating system on Amazon Machine Image, including MS SQL Server installation and configuration, which was not the case with Amazon RDS.

With Amazon EC2, organizations can achieve environment scalability and high availability by quickly provisioning and configuring database instances and storage, and scale database instances by changing the EC2 instances size or storage capacity. High performance can be achieved by provision MS SQL Server in AWS Regions across the world to provide low latency to geographical distributed end users. Multi-Availability Zone deployment will help in configuring the high availability.

Organizations are responsible for data replication and recovery across database instances in the all regions in the event of failure or database crash.

The biggest pro with an Amazon EC2 approach over Amazon RDS is organizations can use SQL Server services and features that are not available in Amazon RDS. On the other hand, the biggest con with using the Amazon EC2 approach is maintenance overhead. Organizations will be responsible for administering the MS SQL Server and databases, including backups and recovery, version upgrade, operating system patching, security management, and configuring high availability or replication.

Summary

Amazon Web Service provides multiple ways to deploy MS SQL Server on AWS. For MS SQL Server, both Amazon RDS and Amazon EC2 have advantages and limitations. Amazon RDS is easy to set up, manage and maintain. Using RDS, organizations have the opportunity to focus on more important tasks, rather than the day-to-day administration of MS SQL Server and the underlying databases. Alternatively, running MS SQL Server in Amazon EC2 gives more control, flexibility, and features to use with additional maintenance overhead.

See all articles by Anoop Kumar


SQL Server 2012 - Handling String Data for the Stored Procedure

$
0
0

Can you give me some pointers (or point in the right direction on what search terms for google)? In a stored procedure I have a parameter @TAG (string). I receive '(14038314,14040071)' (for example) from another application that cannot be altered. In the stored procedure, I need to split apart '(14038314,14040071)' to put quotes around each string value, rebuild it, strip out the outer quotes,strip out the parens and pass it to @TAG in the query below so that it looks like the line commented out below?

SELECT V.NAME AS VARIETY, TAGID FROM mfinv.dbo.onhand h INNER JOIN mfinv.dbo.onhand_tags t on h.onhand_id = t.onhand_id INNER JOIN mfinv.dbo.onhand_tag_details d on t.onhand_tag_id = d.onhand_tag_id INNER JOIN mfinv.dbo.FM_IC_PS_VARIETY V ON V.VARIETYIDX = d.VARIETYIDX LEFT JOIN mfinv.dbo.FM_IC_TAG TG ON TG.TAGIDX = t.TAGIDX WHERE h.onhand_id = (SELECT onhand_id FROM mfinv.dbo.onhand WHERE onhand_id = IDENT_CURRENT('mfinv.dbo.onhand')) AND TG.ID IN (@TAG) --AND TG.ID IN ('14038314','14040071')

You can Use Dynamic SQL Like This

DECLARE @TAG Nvarchar(MAX)='14038314,14040071' set @TAG=''''+REPLACE(@TAG,',',''',''')+'''' --select @TAG DECLARE @SQL NVARCHAR(MAX)=N' Select V.NAME AS VARIETY, TAGID FROM mfinv.dbo.onhand h INNER JOIN mfinv.dbo.onhand_tags t on h.onhand_id = t.onhand_id INNER JOIN mfinv.dbo.onhand_tag_details d on t.onhand_tag_id = d.onhand_tag_id INNER JOIN mfinv.dbo.FM_IC_PS_VARIETY V ON V.VARIETYIDX = d.VARIETYIDX LEFT JOIN mfinv.dbo.FM_IC_TAG TG ON TG.TAGIDX = t.TAGIDX WHERE h.onhand_id = (SELECT onhand_id FROM mfinv.dbo.onhand WHERE onhand_id = IDENT_CURRENT (''mfinv.dbo.onhand'')) AND TG.ID IN ('<a href="/cdn-cgi/l/email-protection" data-cfemail="0922495d484e">[email protected]</a>+')' PRINT @SQL EXEC (@SQL)

Baslining Modern Versions Of SQL Server

$
0
0

Last week I got an email from a community member who had read this older article of mine on baselining , and asked if there were any updates related to SQL Server 2016, SQL Server 2017, or vNext ( SQL Server 2019 ). It was a really good question. I haven’t visited that article in a while and so I took the time to re-read it. I’m rather proud to say that what I said then still holds up today.

The fundamentals of baselining are the same as they were back in 2012 when that article was first published. What is different about today? First, there are a lot more metrics in the current release of SQL Server that you can baseline (e.g. more events in Extended Events, new DMVs, new PerfMon counters, sp_server_diagnostics_component_results). Second, options for capturing baselines have changed. In the article I mostly talked about rolling your own scripts for baselining. If you’re looking to establish baselines for your servers you still have the option to develop your own scripts, but you also can use a third-party tool, and if you’re running SQL Server 2016+ or Azure SQL Database, you can use Query Store.

SQL Server 2019 New DMF sys.dm_db_page_info

$
0
0

Microsoft released preview of SQL Server 2019 recently in Ignite 2018. With every release of SQL Server is enriched with new dynamic management view and functions along with enhancements to existing features.

In this article, we will view the newly introduced dynamic management function (DMF) sys.dm_db_page_info and explore the different scenarios around it.

Overview of sys.dm_db_page_info

SQL Server 2019 provides new database management function ‘sys. dm_db_page_info’ to get information about the pages in SQL Server. Prior to SQL Server 2019, we use undocumented dynamic management function sys.dm_db_database_page_allocations and DBCC Page to get details information about the page in SQL Server for the tables, indexes.

Dynamic Management function sys.dm_db_page_info DMF is introduced to replace DBCC Page function but this DMF returns only page header information while DBCC page shows complete details of the page. We will see this later in the demo section.

Before we move further, let us take a look of new dynamic management function (DMF) syntax and parameters.

Syntax for sys.dm_db_page_info

sys.dm_db_page_info ( DatabaseId, FileId, PageId, Mode )

Below are the parameters to pass into sys.dm_db_page_info.

Argument Description Null allowed Allowed Values

DatabaseId

It is unique ID of the database

No

DBID, or we can use DB_ID() for the current database

FileId

It is ID of the database file

No

DB file id

PageId

It is ID of the page we want to examine

No

PageID

Mode

It shows the level of detail in the output of the function

No

LIMITED No information about description columns

DETAILED it gives information about detailed columns as well

We require the VIEW DATABASE STATE permission in the database. We can provide permissions to run with below command.

GRANT VIEW DATABASE STATE TO [login]

To move further, let us create the sample database and data.

Create Database SQL2019 Go Use SQL2019 Go Create table DemoSQL2019 ( ID int identity(1,1), Name varchar(10) ) Go Insert into DemoSQL2019 Values ('SqlShack')
SQL Server 2019   New DMF sys.dm_db_page_info

In this article, we will take an overview of new DMF sys.dm_db_page_info along and compare it with the previously used DMF sys.dm_db_database_page_allocations.

Now let us view the information about all the pages and extents allocated for the table using the DMF sys.dm_db_database_page_allocations. This is undocumented DMF introduced in SQL Server 2012.

Syntax for sys.dm_db_database_page_allocations:

sys.dm_db_database_page_allocations

(@DatabaseId , @TableId , @IndexId , @PartionID , @Mode)

Argument Description Null allowed Allowed Values

DatabaseId

It is unique ID of the database.

No

DB ID

TableID

It is ID of the database file.

Yes

Table ID or Null

IndexID

It is ID of the page we want to examine

Yes

Index ID or Null

PartionID

We can pass PatitionID if we require.

Yes

partitionID or Null

Mode

It shows the level of detail in the output of the function

No

LIMITED If we use this parameter, it does not show any information about description columns.

DETAILED it gives information about detailed columns as well.

We cannot provide Null value in this parameter.

Now run the below query to get information about allocated page id and page type description.

Select database_id,DB_name(database_id) as [Database], allocated_page_page_id , page_type_desc from sys.dm_db_database_page_allocations(DB_ID(),null,null,null,'Detailed')

In the above query, we can directly pass the DB ID as well. We can get the DB ID from the sp_helpdb or sys.sysdatabases.


SQL Server 2019   New DMF sys.dm_db_page_info

Now we can view the information about any particular page from DBCC Page.

Below is the syntax for DBCC Page:

dbcc page ( {‘dbname’ | dbid}, filenum, pagenum [, printopt={0|1|2|3} ])

Printout values can be as:

0 Prints only page header related information 1 Prints page header and page slot array dump with hex dump for each row 2 Prints page header and whole page hex dump 3 Prints detailed information of per row along with page header

Note: We need to turn on trace flag 3604 for DBCC Page to show output in SQL Server Management Studio.

DBCC TRACEON(3604) DBCC page (6,1,157,3) WITH TABLERESULTS
SQL Server 2019   New DMF sys.dm_db_page_info

We can see that DBCC Page gives detailed information about the page.

Now if we view the same thing from the newly introduced sys.dm_db_page_info. We will use the same page id as used before for a comparison purpose.

We can see here that sys.dm_db_page_info gives one row with all the page information for a given parameter in the detailed mode as specified by us.

Note here, we are using the mode as limited so we can see there are some columns showing null values (highlighted by blue arrow)

Select * from sys.dm_db_page_info(6,1,157,’limited’)
SQL Server 2019   New DMF sys.dm_db_page_info

Important columns in Sys.dm_db_page_info are:

Column Description

database_id

Database ID

page_type_desc

Page type description such as data page, index page, IAM page

page_level

It shows the Level of the page in the index. For a leaf level, its value is 0

slot_count

It shows the total slot counts. It is number of rows for a data page,

ghost_rec_count

A number of ghost records that are marked for deletion.

is_iam_pg

If that particular page is IAM page, its value

Why Did a Plan Get Removed From Cache?

$
0
0
Why Did a Plan Get Removed From Cache? Grant Fritchey Posted on 22 October 2018 Comments I was recently asked if we could tell why a plan was removed from cache. If you read this blog, you know what I’m going to say next. I checked the extended events and there are actually two different events that will tell us information about a plan removed from cache; sp_cache_remove and query_cache_removal_statistics. Let’s […]

The post Why Did a Plan Get Removed From Cache? appeared first on Grant Fritchey .

The Scary DBA

I have twenty+ years experience in IT. That time was spent in technical support, development and database administration. I work forRed Gate Software as a Product Evangelist. I write articles for publication at SQL Server Central, Simple-Talk, PASS Book Reviews and SQL Server Standard. I have published two books, ”Understanding SQL Server Execution Plans” and “SQL Server 2008 Query Performance Tuning Distilled.” I’m one of the founding officers of the Southern New England SQL Server Users Group and its current president. I also work on part-time, short-term, off-site consulting contracts. In 2009 and 2010 I was awarded as a Microsoft SQL Server MVP. In the past I’ve been called rough, intimidating and scary. To which I usually reply, “Good.” You can contact me through grant -at- scarydba dot kom (unobfuscate as necessary).

SQL Server 2019 Big Data Clusters

$
0
0

At the Microsoft Ignite conference, Microsoft announced that SQL Server 2019 is now in preview and that SQL Server 2019 will include Apache Spark and Hadoop Distributed File System (HDFS) for scalable compute and storage. This new architecture that combines together the SQL Server database engine, Spark, and HDFS into a unified data platform is called a “big data cluster”, deployed as containers on Kubernetes.Big data clusters can be deployed in any cloud where there is a managed Kubernetes service, such as Azure Kubernetes Service (AKS), or in on-premises Kubernetes clusters, such as AKS on Azure Stack.The SQL Server 2019 relational database engine in a big data cluster leverages an elastically scalable storage layer that integrates SQL Server and HDFS to scale to petabytes of data storage. The Spark engine is now part of SQL Server:


SQL Server 2019 Big Data Clusters

While extract, transform, load (ETL) has its use cases, an alternative to ETL is data virtualization, which integrates data from disparate sources, locations, and formats, without replicating or moving the data, to create a single “virtual” data layer. The virtual data layer allows users to query data from many sources through a single, unified interface. Access to sensitive data sets can be controlled from a single location. The delays inherent to ETL need not apply; data can always be up to date. Storage costs and data governance complexity are minimized. See the pro’s and con’s of data virtualization via Data Virtualization vs Data Warehouse and Data Virtualization vs. Data Movement .

SQL Server 2019 big data clusters with enhancements to PolyBase act as a virtual data layer to integrate structured and unstructured data from across the entire data estate (SQL Server, Azure SQL Database, Azure SQL Data Warehouse, Azure Cosmos DB, mysql, PostgreSQL, MongoDB, Oracle, Teradata, HDFS, Blob Storage, Azure Data Lake Store) using familiar programming frameworks and data analysis tools:


SQL Server 2019 Big Data Clusters

In SQL Server 2019 big data clusters, the SQL Server engine has gained the ability to natively read HDFS files, such as CSV and parquet files, by using SQL Server instances collocated on each of the HDFS data nodes that can filter and aggregate data locally in parallel across all of the HDFS data nodes.

Performance of PolyBase queries in SQL Server 2019 big data clusters can be boosted further by distributing the cross-partition aggregation and shuffling of the filtered query results to “compute pools” comprised of multiple SQL Server instances that work together (this is similar to a PolyBase scale-out group ).

When you combine the enhanced PolyBase connectors with SQL Server 2019 big data clusters data pools, data from external data sources can be partitioned and cached across all the SQL Server instances in a data pool, creating a “scale-out data mart”. There can be more than one scale-out data mart in a given data pool, and a data mart can combine data from multiple external data sources and tables, making it easy to integrate and cache combined data sets from multiple external sources. This will also be a great solution for importing IoT data.


SQL Server 2019 Big Data Clusters

SQL Server 2019 big data clusters make it easier for big data sets to be joined to the data stored in the enterprise relational database, enabling people and apps that use SQL Server to query big data more easily. The value of the big data greatly increases when it is not just in the hands of the data scientists and big data engineers but is also included in reports, dashboards, and applications used by regular end users. At the same time, the data scientists can continue to use big data ecosystem tools against HDFS while also utilizing easy, real-time access to the high-value data in SQL Server because it is all part of one integrated, complete system.


SQL Server 2019 Big Data Clusters

Azure Data Studio (previously released under the name of SQL Operations Studio) is an open-source, multi-purpose data management and analytics tool for DBAs, data scientists, and data engineers. New extensions for Azure Data Studio integrate the user experience for working with relational data in SQL Server with big data. The new HDFS browser lets analysts, data scientists, and data engineers easily view the HDFS files and directories in the big data cluster, upload/download files, open them, and delete them if needed. The new built-in notebooks in Azure Data Studio are built on Jupyter, enabling data scientists and engineers to write python, R, or Scala code with Intellisense and syntax highlighting before submitting the code as Spark jobs and viewing the results inline. Notebooks facilitate collaboration between teammates working on a data analysis project together. Lastly, the External Table Wizard , which uses PolyBase connectors, simplifies the process of creating external data sources and tables, including column mappings (it’s much easier than the current way of creating external tables).

There will also be a management service that will provision a bunch of agents on each pod that will collect monitoring data and the logs that can be seen via a browser-based cluster admin portal, which will also provide managed services for HA, backup/recovery, security, and provisioning.

In summary,SQL Server 2019 Big Data Clusters improves the 4 V’s of Big Data with these features:


SQL Server 2019 Big Data Clusters

And condition in IN CLAUSE in Sql Server 2008

$
0
0
The equivalent of the mysql LIMIT clause for SQL Server 2008 is so slow

The LIMIT clause can be used to constrain the number of rows returned by the SELECT statement. I looked for an equivalent of the LIMIT clause for SQL server(2008) and found this one : SELECT * FROM (SELECT TOP 30 field1, field2 FROM (SELECT TOP 10 fi

CASE statement for the date comparison in the WHERE clause in SQL Server 2008

I am working with a query which contains CASE statement within WHERE clause inside which I have to compare date column with 2 variables fromDate and toDate if it is not null. But SQL Server 2008 is giving some errors like "Incorrect syntax near '>

How to create and define fields to insert the date and date of last modification in SQL Server 2008?

I'd like to record the insert date and an update date on a table. What is the best way to do this in SQL Server 2008?For Insert Date you can use the following trigger: CREATE TRIGGER INSERT_DATE ON TABLE1 FOR INSERT AS BEGIN SET NOCOUNT ON UPDATE TAB

Division of the full name and write to another table in SQL Server 2008

I have a table, say A, in which there is a column FULLNAME. Values stored under this column are in the format of "surname name middle_name" (with one space between each). And I have another table B, in which I have columns SURNAME, NAME and MIDD

Using if-else in the where clause in SQL Server 2008

I need to implement in the where clause. I tried in case statement but between cannot be used in case statement where if start_mth < end_mth mth_no between start_mth and end_mth else start_mth > end_mth mth_no between start_mth and 12 or mth_no betw

Output value and whether it exists in the SQL Server 2008 database I am trying to create a query that outputs (1) The value (2) Whether the value exists or not. I tried SELECT CASE WHEN EXISTS ( SELECT [IP_ADDR1_TEXT] FROM [dbo].[V_SEM_COMPUTER] WHERE [IP_ADDR1_TEXT] = '10.10.10.10' ) , [IP_ADDR1_TEXT] THEN CAST(1 A What is the performance of & ldquo; Merge & rdquo; clause in sql server 2008?

Merge can performs insert, update, or delete operations on a target table based on the results of a join with a source table. For example, you can synchronize two tables by inserting, updating, or deleting rows in one table based on differences found

Best method to create and use client account numbers in SQL Server 2008 SR2

I am not a DBA so I am scratching my head a bit. I am trying to put together an application for the company I work for. I need to provide a unique customer account number for each office location my company conducts business with. In the past, I have

Median and 95th percentile in SQL Server 2008? - Requirement of NHS reports

I have tried to find out 95th percentile and median build in function in SQL server 2008 but I do not know why MS does not give support for them, really annoying... Our work place reports are very complex and wanted a straight forward function or may

The best way to store date / time data in SQL Server 2008 R2 and SQL Server Compact 4

We are developing an application in C# 4 that uses SQL Server 2008 R2 as backend. SQL Server Compact 4 is also used for disconnected clients in a very few rare scenarios. We are wondering what's the best way to store date/time data into these databas

Read and increment the int value in SQL Server

I need to read and increment a value atomically in SQL Server 2008, using c#. For example, I have to insert items of a "lot", for this i need the number of the last lot and be sure, that no one else get this number. So i have a table only with t

SQL Server 2008: Rotate Multiple Columns

Need help with the pivot clause in SQL Server 2008. I have a table with this info: I have a table with 9 columns: ID, Period_1 to Period_4 as dates (i.e. 2013-04, 2013-07, etc.) and Amount_1 to Amount_4 (i.e. 30, 40, etc.). I want all the distinct da

Detect whether SQL Server 2008 is installed

I am using dotNetInstaller as a bootstrapper and I need to detect if SQL Server 2008 or above is installed as a prerequisite. Currently I am using this registry to detect the installation: HKLM\SOFTWARE\Microsoft\Microsoft SQL Server 2008 Redist\Shar

SQL Server 2008 Requirements

I've read a few questions at stackoverflow and some msdn articles too about SQL Server 2008 system requirements but there's something that i'vent understand yet. I've installed the SQL Server 2008 Express. I'm not using the Management Studio, just ru

Using Script Component to specify input value to OLE DB Source Component in SSIS

$
0
0

Recently we had a requirement to write an SSIS package that will fetch the data from SQL Server and will insert it (or create records) into Dynamics 365 CE.

For connecting and fetching data from SQL Server, we were using OLE DB Source Component with data access mode as SQL Command which had one parameter, whose value we had to fetch from Dynamics 365 CE.


Using Script Component to specify input value to OLE DB Source Component in SSIS

For passing the value to this parameter we had defined a variable within our Package named LastUpdateFromCRM.


Using Script Component to specify input value to OLE DB Source Component in SSIS

Now as we wanted to set the variable value from one of the records in Dynamics 365 CE, we used KingswaySoft’s CDS/CRM Source Component Editor, that will return us one row using fetch xml + max rows returned property.

Then we added a script component to the output of the CDS/CRM Source Component.


Using Script Component to specify input value to OLE DB Source Component in SSIS

Here double-clicking the Script Component and selecting Input Columns inside the editor, we can select the input columns that we want to use within the Script Editor.


Using Script Component to specify input value to OLE DB Source Component in SSIS

Next, we selected Script section inside the editor, there we selected the user variable as Read Write variable (as we are setting its value inside our script) and clicked on Edit Script button to open the VSTA Project.


Using Script Component to specify input value to OLE DB Source Component in SSIS

Within VSTA project, inside main.cs we defined a date time variable to hold value coming from the CRM Source Component i.e. the value of last update field.


Using Script Component to specify input value to OLE DB Source Component in SSIS

Inside ProcessInputRow method, we then set the value of that variable and finally within Post Execute method we use it to set the value of our user-defined variable.


Using Script Component to specify input value to OLE DB Source Component in SSIS

Now as we have specified the value to our LastUpdateFromCRM variable, its value was passed to the SQL Command of OLE DB Source component as intended.

Hope it helps..


How to Pass Data Between SQL Server Agent Job Steps

$
0
0

By: Jeffrey Yao || Related Tips:More >SQL Server Agent

Problem

From time to time, there may arise a business requirement that in a SQL Server Agent Job , we want to pass a parameter value or multiple values from step 1 to step 2. For example, in job step 1, I insert a record to a table with an identity column, and I want to pass the newly inserted identity value, i.e. scope_identity(), to the next job step.

There are many different types of SQL Server Agent Job steps, such as ActiveX Script, PowerShell script, SSIS packages, T-SQL script, etc., out of these job steps, the most difficult one is probably the T-SQL Script type as T-SQL usually does not have an easy way to write data outside of the scope for the next step to consume, while ActiveX Script, PowerShell and SSIS package can easily generate an external file or even access to the OS environment variables for data exchange purposes.

So what solutions do we have to address this data exchange issue for T-SQL type job steps?

Solution

Actually, there is a common way to do this work, i.e. create a permanent table and then use this table as the "liaison" or "messenger" to exchange data between job steps. There is nothing wrong with this approach, however, there may be other ways that are more flexible and robust with less footprint or better alignment with business requirements.

We will explore these potential solutions other than using a permanent table.

We assume that we want to pass two parameters, one is an integer parameter, @parmInt and another a string parameter, @parmStr.

For demo purposes, we assume we have a job with two steps, and we will pass values from step 1 to step 2.

Method 1: Pass data via an Extended Property between SQL Server Agent Job Steps

An extended property is a good place for us to store temporary values as its initial purpose is for "documentation" of database objects.

To make things simpler, we will add extended properties to TempDB to pass the data between job steps.

So, in job step 1, we have the following:

if exists (select * from tempdb.sys.extended_properties where class=0 and name='parmInt')
exec tempdb.sys.sp_dropextendedproperty @name='parmInt';
if exists (select * from tempdb.sys.extended_properties where class=0 and name='parmStr')
exec tempdb.sys.sp_dropextendedproperty @name='parmStr';
EXEC tempdb.sys.sp_addextendedproperty
@name = N'parmStr',
@value = 'HelloWorld'; -- assuming 'HelloWorld' is what we want to pass
EXEC tempdb.sys.sp_addextendedproperty
@name = N'parmInt',
@value = '123'; -- assuming '123' is what we want to pass

In job step 2, we can use the following script to retrieve the values:

declare @parmInt int, @parmStr varchar(128);
select @parmInt = cast([value] as int)
from tempdb.sys.fn_listextendedproperty('parmInt', null, null, null, null, null, null);
select @parmStr = cast([value] as varchar(128))
from tempdb.sys.fn_listextendedproperty('parmStr', null, null, null, null, null, null);
/*
-- other regular business processing with @parmInt and @parmStr;
-- you can even drop these Eps after the work is done using sp_dropextendedproperty
*/ Method 2: Pass value via Error Log between SQL Server Agent Job Steps

In this method, we will write the parameter value into the error log and then in the next job step, read from the SQL Server error log.

So in step 1, we will write to the log using raiserror … with log . The 'with log' will write the message into the SQL Server error log.

For the raiserror statement, if we set the severity level to be 0 (or 10), the job step will not fail, and in such cases, raiserror itself is only to write some information to the error log.

So, in job step 1, we have the following code

-- other business logic block
-- now we want to pass values to next step
declare @parmInt int = 1234, @parmStr varchar(128) = 'hello world'
raiserror ('@parmInt = %d;; @parmStr = %s', 10, 1, @parmInt, @parmStr) with log;

And in the SQL Server error log, we can see the following:


How to Pass Data Between SQL Server Agent Job Steps

In the next step we can run the following code to retrieve the data based on the data format from step 1:

set nocount on;
declare @t table (LogDate datetime, ProcessInfo varchar(100), [text] varchar(300));
insert into @t (LogDate, ProcessInfo, [Text])
exec master.sys.sp_readerrorlog 0, 1, 'parmInt';
declare @parmInt int, @parmStr varchar(128);
select top 1 @parmInt = cast(substring([text], len('@parmInt = ')+1, charindex(';;', [text])-len('@parmInt = ')-1) as int)
, @parmStr = substring([text], charindex('@parmstr = ', [text])+len('@parmStr = ')+1, 128)
from @t
order by LogDate desc; -- find the most recent data
raiserror('@parmInt = %d', 0, 1, @parmInt) with nowait; -- for debug purpose
raiserror('@parmStr = %s', 0, 1, @parmStr) with nowait; -- for debug purpose
/*
-- the result is exactly as expected
@parmInt = 1234
@parmStr = hello world
*/ Method 3: Pass value via Job Description Property between SQL Server Agent Job Steps

We know each SQL Server Agent Job has a description section as highlighted below.


How to Pass Data Between SQL Server Agent Job Steps

We can actually use this section to pass the data. The only thing we need to be aware of is that many jobs may already use this section, so we should append the data to the existing information and later clean it up after parameter values are retrieved.

In step 1, we have the following code, in which, we use the job token '$(ESCAPE_SQUOTE(JOBNAME))'.

/*
The regular business codes
-- the following is to put parameter values into job description section
*/
declare @parmInt int, @parmStr varchar(128);
-- assume we need to pass the following data to next step
select @parmInt = 123, @parmStr = 'hello world';
declare @desc nvarchar(512);
declare @crlf char(2) = char(0x0d) + char(0x0a);
select @desc = description
from msdb.dbo.sysjobs
where name = '$(ESCAPE_SQUOTE(JOBNAME))';
print @desc;
-- append the info
set @desc = @desc + @crlf + '@parmInt = ' + cast(@parmInt as varchar(20)) + ';' + @crlf;
set @desc = @desc + '@parmStr = ' + @parmStr + ';' + @crlf;
-- update the job description with newly appended data
exec msdb.dbo.sp_update_job @job_name = '$(ESCAPE_SQUOTE(JOBNAME))'
, @description = @desc;

If my initial job description is like the following:


How to Pass Data Between SQL Server Agent Job Steps

After job step 1 is run, we will see a modified description section as follows:


How to Pass Data Between SQL Server Agent Job Steps

In job step 2, we can retrieve the appended info with the following code:

declare @parmInt int, @parmStr varchar(128);
declare @desc nvarchar(512);
select @desc = description
from msdb.dbo.sysjobs
where name = '$(ESCAPE_SQUOTE(JOBNAME))';
select @parmInt=cast(substring(@desc,

Developing and Querying a Graph with SQL Server 2017 and R - Part 2

$
0
0

By: Siddharth Mehta || Related Tips:More >SQL Server 2017

Problem

Graph analysis can be divided into two parts Graph Rendering and Graph Querying. In thefirst part of this tip we saw how to create a graph with different aesthetic customizations to represent the graphical nature of the data in a visually interpretable manner. Generating a graph is just the first part of the graph analysis process. The other part of the analysis is querying the graph and rendering the results of the query in the graph visualization. In this tip we will learn how to query graph data and render each step of the analysis in the graph visualization.

Solution

DiagrammeR package has a variety of functions to query and customize different graph attributes, which can be used to display the query results on a graph visualization.

1) As this tip is part 2 of theprevious tip on the same subject, its assumed that the required R packages explained in the previous tip are installed correctly.

2) To demonstrate the query process, first we need to create a sample dataset that can be used for querying. If you already have a sample graph data, you can use it and fit it in the way explained below. Alternatively, you can use a sample dataset that is explained in the DiagrammeR package documentation, which is also explained in the steps below.

3) Let’s say that we have a scenario where people are consuming food items like vegetables, fruits and nuts. So, we have four diverse types of entities here person, fruits, vegetables and nuts. A node data frame can be created using the create_node_df function from DiagrammeR package as shown below. Here we are specifying the type as the type of entity, label as the name of some actual entities like person name, fruit name, etc.

nodes <- create_node_df(14, nodes = 1:14,
type = c("person", "person", "person", "person", "person",
"fruit", "fruit", "fruit",
"veg", "veg", "veg",
"nut", "nut", "nut"),
label = c("Annie", "Donna", "Justine", "Ed", "Graham",
"pineapple", "apple", "apricot",
"cucumber", "celery", "endive",
"hazelnut", "almond", "chestnut")
)

4) People may like as well as dislike some of these items, and some may even be allergic to some of these items. These creates our edges where the relation is likes, dislikes and allergic. An edge data frame can be created using the create_edge_df function from the DiagrammeR package as shown below. In the below code, in the from section we are creating a series of 1 to 5 for 5 times. The intention is to create edges from nodes 1 to 5 which are all the person nodes that can be seen in the above code. In the To section we are creating random 5 edges for 5 times, to associate 5 food items with each person. Finally, in the label section we are giving a random type to each edge. You can also use rel keyword instead of label to add it as a relation attribute of the edge instead of a label attribute.

edges <- create_edge_df(
from = sort(as.vector(replicate(5, 1:5))),
to = as.vector(replicate(5, sample(6:14, 5))),
label = as.vector(replicate(5, sample(c("likes", "dislikes","allergic_to"), 5,TRUE,c(0.5, 0.25, 0.25)))))

5) Now that we understand how nodes, edges and relations are created, it’s time to create the actual graph. Execute the below code to create the graph. We have used few formatting options while creating the nodes and edges, so that the graph looks intuitive. After creating the graph object with the create_graph function, we are exporting it to a png image using the export graph function.

EXECUTE sp_execute_external_script @language = N'R', @script = N'
library(DiagrammeR)
library(magrittr)
library(DiagrammeRsvg)
set.seed(20)
nodes <- create_node_df(14, nodes = 1:14,
type = c("person", "person", "person", "person", "person",
"fruit", "fruit", "fruit",
"veg", "veg", "veg",
"nut", "nut", "nut"),
label = c("Annie", "Donna", "Justine", "Ed", "Graham",
"pineapple", "apple", "apricot",
"cucumber", "celery", "endive",
"hazelnut", "almond", "chestnut"),
fontname = "Helvetica",
fontsize = "6",
fillcolor = "yellowgreen",
color = "red"
)
edges <- create_edge_df(
from = sort(as.vector(replicate(5, 1:5))),
to = as.vector(replicate(5, sample(6:14, 5))),
label = as.vector(replicate(5, sample(c("likes", "dislikes","allergic_to"), 5,TRUE,c(0.5, 0.25, 0.25)))),
fontname = "Helvetica",
fontsize = "7",
fontcolor = "blue",
color = "grey"
)
graph <- create_graph(nodes_df = nodes, edges_df = edges, directed="true")
export_graph(graph, file_name = "C:\\temp\\Graph1.png", file_type = "png", width=1800, height=800)
'

6) Once the above code executes successfully, the file should get created in the provided location. Open the file and the visualization would look as shown below.


Developing and Querying a Graph with SQL Server 2017 and R - Part 2

7) Now that the graph has been created, we are able to first visually understand how the entities are related, and then form a query criterion to query the graph. Lets start the query criteria with a simple condition which can be matched with just one step. For example, lets say we want to find all the nodes of type person in a graph. The above graph is a small dataset, but in real life scenarios, there can be millions of nodes and edges even in modest size graph datasets. Although we can visually identify all the person nodes in the above graph, the programmatic way to identify is by querying the attributes of the graph. You can find the entire language and function reference of DiagrammeR package here .

8) The below code can be used to query the graph and highlight the nodes of type person. After creating the graph, we are creating a new graph object and selecting the nodes of type person by using the select_nodes function. In the next step we are extracting the nodes that we selected in the previous step and changing the fillcolor attribute value to pink to highlight the results in the graph. After completing this step, we are finally exporting the data to a png file.

EXECUTE sp_execute_external_script @language = N'R', @script = N'
library(DiagrammeR)
library(magrittr)
library(DiagrammeRsvg)
set.seed(20)
nodes <- create_node_df(14, nodes = 1:14,
type = c("person", "person", "person", "person", "person",
"fruit", "fruit", "fruit",
"veg", "veg", "veg",
"nut", "nut", "nut"),
label = c("Annie", "Donna", "Justine", "Ed", "Graham",
"pineapple", "apple", "apricot",
"cucumber", "celery", "endive",
"hazelnut", "almond", "chestnut"),
fontname = "Helvetica",
fontsize = "6",
fillcolor = "yellowgreen",
color = "red"
)
edges <- create_edge_df(
from = sort(as.vector(replicate(5, 1:5))),
to = as.vector(replicate(5, sample(6:14, 5))),
label = as.vector(replicate(5, sample(c("likes", "dislikes","allergic_to"), 5,TRUE,c(0.5, 0.25, 0.25)))),
fontname = "Helvetica",
fontsize = "7",
fontcolor = "blue",
color = "grey"
)
graph <- create_graph(nodes_df = nodes, edges_df = edges, directed="true")
querynodes <- graph %>% select_nodes(conditions = "type == \"person\"")
graph

SQL Interview Questions on SQL Server Backup and Restore Part 1

$
0
0

The following is the list of SQL interview questions covering backup and restore, backup encryption and more.

SQL interview questions 1:Can we be able to take the backup for “RESOURCEDB”?

No, we can’t perform backup for resource DB. But we can take the physical file backup for “RESOURCEDB” MDF and LDF files.

SQL interview questions 2:What are the new features added for Backup and Restore on SQL Server 2014? Backup Encryption:

On SQL Server 2014 we can enable encryption while creating native backup either using T-SQL or SSMS.

Backup encryption supported in SQL Server 2014 Standard, Business Intelligence and Enterprise editions.

Encryption supports algorithms: AES 128, AES 192, AES 256 and Triple DES.

Backup to URL Azure Storage:

As expected SQL Server 2014 backup is cloud enabled. Now we can directly perform a backup to Microsoft Azure Storage using the destination as URL.

It supports both T-SQL and SSMS UI. To use SQL backup to URL we need:

An Azure subscription Access to Azure using the Azure Portal or PowerShell An Azure Storage Account A container in the Azure Storage account Managed Backups:

This is primarily designed for Small to Medium Business (SMB). This feature “Managed Backups” automates the backup process and directs the backup to windows Azure storage.

It works for both when the SQL Server is in our internal premises or in Windows Azure VM.

Managed Backups can be setup at database level or at instance level. Since it directs backup to windows azure, it requires an Azure Subscription and storage account.

SQL interview questions 3:Can we control the backup schedules in managed backups on 2014?

No! It depends on log usage and other conditions 16. Can we include databases with “simple recovery” on managed backups on 2014?

No, not possible for system databases and for databases which are in simple recovery mode.

SQL interview questions 4:Can you quickly tell me the process for creating an encrypted backup on SQL Server 2014? Create a database master key on master database Create a Certificate on master database Create a Backup and mention all required parameters for encryption Keyword ENCRYPTION Algorithm example: AES_256 Certificate to be used; the one that we created on master SQL interview questions 5:We get an encrypted backup from Server A and the requirement is this backup has to be restored at Server B. Can you quickly demonstrate the steps? Server A: Backup the Master Key to a file (.key) using the same password that we used while creating the Master Key Backup the certificate to a file (.cer) with a private key. The private key is created using the same password that we used in above step Backup the database using encryption Server B: Restore the Master Key First decrypt using the password used on Server A Then create a new Master key Open master key: Important step Create certificate from the certificate backup created on Server A Now we can restore database without specifying any special options as we are restoring from an encrypted backup SQL interview questions 6:Is there any impact if we enable the backup encryption?

Yes! Mainly strain on CPU.

AES-128 is the best algorithm still it uses 20% to 25% extra CPU for encryption. TRIPLE_DES_3KEY algorithm takes 40% to 45% extra CPU.

We should be very careful on backup timings if backup compression is also enabled along with backup encryption. Because in one of the environment we have seen CPU takes 55% extra while running backup with compression and encryption with TRIPLE_DES_3KEY.

SQL interview questions 7:We are storing backup files on Windows Azure. Can we restore the backup from Windows Azure to our local machine?

Yes! We can do that we just need to mention:

RESTORE <DB NAME> FROM URL = <> WITH CREDENTIALS = <>; SQL interview questions 8:What are the new backup related enhancements in SQL Server 2016? Managed Backup Enhancements: This feature is introduced in SQL Server 2014 System databases can be backed up Backup for databases in simple recovery model is possible A backup schedule can be customized based on business requirement rather than log usage Backup to Azure Storage Enhancement: SQL Server 2014 supports backup on Page blobs whereas SQL Server 2016 supports backup on block blobs on Windows Azure When we check monthly storage price block blobs are cheaper than page Blobs Page Blog has limit to 1 TB and block blob limit is 200 GB But we can take a stripe backup maximum up to 64 block blobs which is equal to 12.8 TB File Snapshot Backups on Azure: SQL Server 2016 is having a feature File-Snapshot backup for the database files stored on Azure Blob store This will be helpful for SQL Server running on Azure Virtual Machine It throws an error message if we try to perform a File Snapshot backup for the databases which are stored on local drives SQL interview questions 9:What are the phases of SQL Server database restore process?

Copy Data: Copies all data, log and index pages from backup file to database MDF, NDF and LDF files

REDO: Roll forward all committed transactions to database and if it finds any uncommitted transactions it goes to the final phase UNDO

UNDO: Rollback any uncommitted transactions and make database available to users

SQL interview questions 10:Is it possible to restore a Database backup of SQL Server 2012 to SQL Server 2008/2008 R2?

No, it’s not possible to restore the upper version database backup to lower version.

SQL interview questions 11:What is Stable media, how to use it?

Stable media is usually a directly attached disk drive, but it can be any device that guarantees that, on power loss, no data will be lost.

Even on direct attached systems, this can be a challenge. Just as disk drives implement write caches, RAID controllers (even at the simplest level) also implement caches, which either must be write-disabled or battery-backed. Any external storage system such as a Storage Area Network (SAN) system must also be checked to confirm that the cache is battery-backed and will guarantee the consistency of any written log records during a power failure.

There is a new trend for using solid-state storage devices, which can take many forms. If you leverage these devices, you must ensure that they either deliver guarantees around writes if a power failure occurs, or that they are used in places where write cache performance is not an issue. An increasingly common trend on high-performance systems that need the highest levels of transaction-log write performance is to place the transaction log on solid-state storage.

Although this is great from a performance perspective, you must also guarantee that the log records can survive a power outage. The SQL Server Database Engine expects the transaction log to be consistent on restart. If it is not, it will identify the database as corrupted because the data consistency of the database cannot be determined.

Summary

That was it for this part. You can find more SQL Interview questions on Backup and Restore inPart 2 article of this series.

SQL Interview Questions on SQL Server Backup and Restore Part 2

$
0
0

The following is the list of SQL interview questions covering backup and restore, accidental deletes and updates and more.

SQL interview questions 1:Have you ever encounter the issue “media family on device is incorrectly formed”? If yes what is the reason and how you resolved it?

Yes! I have seen the error “The media family on device is incorrectly formed. SQL Server cannot process this media family” many times. There are a few common cases that raise this error: Trying to restore backup from higher version to lower version.

Example:

Backup taken from 2014 and trying to restore to 2008 R2. Trying to restore a backup using SSMS or T-SQL in a native method when the backup is taken using a third-party tool.

Example:Lite-speed

Trying to restore a backup file where the backup copy is still in progress. Backup file might be corrupted.

SQL interview questions 2:We have been using SQL Server 2014 and we need to rebuild the system databases due to a corruption in the master database. Do we need to get the installation DVD or Media to rebuild the system databases?

No! Not required. SQL Server can get these original files from a cache location called “Template” folder. These (System Databases) are cached into this folder at the time of installation.

SQL interview questions 3:Ok, so my instance system databases MDF and LDF files are initially cached to Template folder. My question is if we can directly copy and use these MDF and LDF files from Template location instead of running REBUILD when we see a corruption in master database and we need to rebuild the system databases?

No! We are not supposed to directly copy files from Template location to DATA and LOG folders. If we manually copied these files and tried to restart the SQL Server, it might fail with the errors as it cannot identify the location of MDF and LDF files for other system databases. The reason is that simple Master database is a copy of the initial DB when it got installed. It contains the default path for MSDB and MODEL database file locations. But, in reality, we usually don’t keep system databases on C drive.

If we rebuilt the system databases, setup would move the files from Template location to the proper location and it would modify the path of other databases which are stored in the master database.

SQL interview questions 4:We have an 800GB database. We are taking a full backup on daily basis, which takes 3 to 4 hours. You are asked to propose a solution for speeding up the backup process. What would you suggest?

Full and Differential: We can plan for a weekly full backup and daily differential backup and transactional backup based on SLA for data loss acceptance. Differential backup doesn’t take much time when compared to full backup. Make sure the latest differential backup is in safe place as it requires restoring the database with the minimum data loss.

File/File Group backup: Categorize the tables into “Master,” “Lookup” and “Transactional” and keep these files into different file groups. Perform backup only on the tables where there are daily data changes. This is one way but we should do a lot of research before applying this method. This is really useful when databases are huge.

If a full backup, however, is required on daily basis, then your response to SQL interview questions 4 should be something of the below.

Backup database to Multiple Files: If your business demands a full backup on daily basis then you can choose this method. Split backup into 3 or 4 files on different drives. Let’s say I have a database with 1 TB in size and it is taking 4 hours to perform a full backup. If we split this backup into 4 files on 4 different disks, then the full backup time comes to 1 hr. Logic is we have 4 files writing on 4 different physical drives, each drive has a separate thread and all these threads execute in parallel. Remember we should make sure all these files are available for a successful restore when we do split backup.

Backup during off hours/less traffic: This is one of the best practices in scheduling backup.

Backup to local disk and then archive: Avoid using a network path in the backup and make it be like a three-tier architecture:

Perform backup to local disk Schedule a job to copy/move from local to network Then you can continue with moving to the tape

Backup Compression: This is one technique that is mostly used when there is a space issue. But sometimes compression can reduce backup time when we have a lot of CPU cycles free and enough IO bandwidth.

SQL interview questions 5:You are leading a DBA team and one of your team members came to you and reported that he had executed an UPDATE command without WHERE CLAUSE on a production database. Now that entire table has gotten updated with the wrong data, what would you do?

That’s a really embarrassing situation. I actually faced it earlier in my previous role. The command was provided by release team and one of our DBAs has executed it, but he immediately identified the problem and reported it to us. The solution that worked out:

Made database to single user mode Identified the “Transaction ID” and “Begin Time” by reading log file using sys.fn_dblog() Performed a log backup Created a dummy database on the same instance and restored it with this database’ latest full backup Restored the latest differential backup followed by all transaction log backups except the last one that we have taken Restored the latest transactional backup that we performed in the step 3 with the point in time recovery. The time we gave was just before the transaction begin time that we identified in the step 2, i.e.: Transaction Begin Time: “2016-07-26 15:26:39” then recover data till “2016-07-26 15:26:38” We got our original table in dummy database and applied an update statement by joining the same table from the original database and the dummy database where we had the correct data and did the required changes Made the original database to multi user mode Dropped the dummy database

SQL interview questions 6:For the accidental situations like “DELETE WITHOUT WHERE,” “UPDATE WITHOUT WHERE” or “DROPPING A TABLE” do we have any other option except restoring full/diff/log file?

I don’t think we have a native support tool/method to roll back the committed operations. But there are a few third-party tools which can be really helpful in these situations.

ApexSQL: ApexSQL Log

Redgate: SQL Log Rescue

I hope you have found this set of SQL interview questions helpful. Feel free to post any feedback or ask any questions in the comments below.

How SQL Server DBA’s can use Power BI Report Server

$
0
0

Reading Time: 3 minutes

In a previous post I went through a quick introduction of Power BI Report Server . I have decided to do a follow up on how SQL Server DBA’s can use Power BI Report Server.

As I mentioned in my previous post Power BI report Server is where you can install Power BI on your own servers. This now opens the floodgates for you to publish things you weren’t allowed to publish in Power BI before. Possibly due to security concerns.

Security

For example, your boss might have prevented you from publishing a dashboard into a PowerBI in the past. Maybe because they contained server names and configuration details and they were worried about them being in the cloud.

Now there is a version of Power BI that you can install on your own servers you’re free to look to upload the dashboards onto there. After you have done this you’ll have other decisions to make like how to secure your dashboards locally.

With that in mind lets look at some dashboards you might want to Publish onto Power BI Report Server if you are a DBA.

Database Migration Assistant

One thing DBA’s can definitely take advantage of are dashboards showing the results of Database Migration Assistant checks. Database Migration Assistant is a very powerful tool that advises you what you need to do if you are migrating a database from one version of SQL to another. It prevents a lot of migration issues.

I strongly recommend using this if you are doing migrations. In fact I cover it in my session that I recently talked about in a previouspost. I will talk about two versions here.

Microsoft version

Until recently I mostly looked to use the solution that has been developed by Microsoft to view the Database Migration Assistant results. It uses the command line version ‘dmacmd.exe’.

First you export the results in to json files, one file

for each instance containing databases you want to analyze . Then you import the json files into a SQL Server database. After that you use the database as a source for a Power BI dashboard. You can read more information about it here .

Alternative version

However, last week I was made aware of an alternative solution to view the results of these checks in Power BI. It’s done by a guy called Dustin Ryan and is based on using csv files as the source instead. You can read more about this here .

DMA Dashboard Considerations

Whichever of these solutions you use to check databases before migration you might have another big decision to make. If you’re supporting a large estate how do you present this data to application teams so that they can review and make the required changes? Two potential options are as below:

Create one dashboard for each application team so that they can view only their servers. Create one dashboard for everyone and change the security so that application teams view their own data.

If you don’t know Power BI that well the first might be a better option. However if you use one dashboard with the right security settings in place it makes the dashboard easier to maintain and update. An expert called Reza Rad has a webinar about Power BI Security Patterns here .

DBAChecks

Another thing you will be able to advantage of more in Power BI Report Server is DBAChecks dashboards. Which is a new offering from the dbatools team. It allows you to check the configuration of multiple servers using Powershell.

It is supported by various members of the community and improving over time. A lot of places are using this due to the fact it’s quick to implement and easy to customise. They have also designed a Power BI dashboard as well which you can look to publish into Power BI.

It was actually this project that introduced me to the Papercut mail application. You can find out more about DBAChecks here .

Final word

What about yourself. Are there any dashboards you want to use or have already used? Feel free to add with a comment.

Viewing all 3160 articles
Browse latest View live