Parsing SSIS Catalog Messages for Lookup Performance, v2

A couple years ago I wrote Administering SSIS: Parsing SSIS Catalog Messages for Lookups . I’ve updated the script to return all Lookup Transformation performance messages captured by the SSIS Catalog with Basic logging enabled (the SSIS Catalog’s default Logging Level). Some smart data integration people recommend setting the SSIS Catalog Logging Level to Verbose and querying catalog schema views to retrieve row counts and performance metadata. I’m not a fan of running a Verbose logging level by default due to the overhead. Granted, the overhead is relatively light but it’s not nothing. Sometimes I need all the cycles!

I prefer to passively gather performance metadata even if it’s a little convoluted and more work (for me). I don’t often need this metadata immediately available, so I execute queries similar to the query below and store the metadata in a table. I can (and do) build dashboards to track SSIS performance (perhaps I should blog about them…) using this passively-collected performance metadata.

Without further ado, the updated query:

Use SSISDB
go declare @LookupIdString varchar(100) = '% rows in the cache. The processing time was %'
declare @LookupNameStartSearchString varchar(100) = '%:Information: The %'
declare @LookupNameStartLen int = Len(@LookupNameStartSearchString) - 2
declare @LookupNameEndSearchString varchar(100) = '% processed %'
declare @LookupNameEndLen int = Len(@LookupNameEndSearchString) - 2
declare @ProcessingTimeString varchar(100) = 'The processing time was '
declare @ProcessingTimeSearchString varchar(100) = '%' + @ProcessingTimeString + '%'
declare @CacheUsedString varchar(100) = 'The cache used '
declare @CacheUsedSearchString varchar(100) = '%' + @CacheUsedString + '%' Select
SubString(om.[message]
, (PatIndex(@LookupNameStartSearchString, om.[message]) + @LookupNameStartLen)
, (PatIndex(@LookupNameEndSearchString, om.[message]) - (PatIndex(@LookupNameStartSearchString, om.[message]) + @LookupNameStartLen))
) As LookUpName
, Convert(bigint, Substring(om.[message]
, (PatIndex(@LookupNameEndSearchString, om.[message]) + @LookupNameEndLen)
, ((CharIndex(' ', om.[message], PatIndex(@LookupNameEndSearchString, om.[message]) + @LookupNameEndLen))
-
(PatIndex(@LookupNameEndSearchString, om.[message]) + @LookupNameEndLen)
)
)
) As LookupRowsCount
, Convert(decimal(16, 3), Substring(om.[message]
, (PatIndex(@ProcessingTimeSearchString, om.[message]) + Len(@ProcessingTimeString) + 1)
, ((CharIndex(' ', om.[message], PatIndex(@ProcessingTimeSearchString, om.[message]) + Len(@ProcessingTimeString) + 1))
-
(PatIndex(@ProcessingTimeSearchString, om.[message])+ Len(@ProcessingTimeString) + 1)
)
)
) As LookupProcessingSeconds
, Convert(bigint, Substring(om.[message]
, (Patindex(@CacheUsedSearchString, om.[message]) + Len(@CacheUsedString) + 1)
, ((Charindex(' ', om.[message], Patindex(@CacheUsedSearchString, om.[message]) + Len(@CacheUsedString) + 1))
-
(Patindex(@CacheUsedSearchString, om.[message]) + Len(@CacheUsedString) + 1))
)
) As LookupBytesUsed
, Convert(decimal(16, 3), (Convert(bigint, Substring(om.[message]
, (PatIndex(@LookupNameEndSearchString, om.[message]) + @LookupNameEndLen)
, ((CharIndex(' ', om.[message], PatIndex(@LookupNameEndSearchString, om.[message]) + @LookupNameEndLen))
-
(PatIndex(@LookupNameEndSearchString, om.[message]) + @LookupNameEndLen))
)
)
/
(Convert(decimal(16, 3), Substring(om.[message]
, (PatIndex(@ProcessingTimeSearchString, om.[message]) + Len(@ProcessingTimeString) + 1)
, ((CharIndex(' ', om.[message], PatIndex(@ProcessingTimeSearchString, om.[message]) + Len(@ProcessingTimeString) + 1))
-
(PatIndex(@ProcessingTimeSearchString, om.[message])+ Len(@ProcessingTimeString) + 1 ))
)
)
)
)
) As LookupRowsThroughput
, ex.start_time As ExecutionStartTime
, ex.folder_name As FolderName
, ex.project_name As ProjectName
, ex.package_name As PackageName
--Into rpt.LookupMetrics -- drop table rpt.LookupMetrics
From [catalog].operation_messages om
Join [catalog].executions ex
On ex.execution_id = om.operation_id
Where om.message_source_type = 60 -- Data Flow Task
And om.[message] Like @LookupIdString
-- data type-checking
And IsNumeric(Substring(om.[message], (PatIndex(@LookupNameEndSearchString, om.[message]) + @LookupNameEndLen ), ((CharIndex(' ', om.[message], PatIndex(@LookupNameEndSearchString, om.[message]) + @LookupNameEndLen)) - (PatIndex(@LookupNameEndSearchString, om.[message]) + @LookupNameEndLen)))) = 1 -- Check LookupRowsCount
And IsNumeric(Substring(om.[message], (PatIndex(@ProcessingTimeSearchString, om.[message]) + Len(@ProcessingTimeString) + 1 ), ((CharIndex(' ', om.[message], PatIndex(@ProcessingTimeSearchString, om.[message]) + Len(@ProcessingTimeString) + 1)) - (PatIndex(@ProcessingTimeSearchString, om.[message])+ Len(@ProcessingTimeString) + 1 )))) = 1 -- Check LookupProcessingSeconds
And IsNumeric(Substring(om.[message], (Patindex(@CacheUsedSearchString, om.[message]) + Len(@CacheUsedString) + 1 ), ((Charindex(' ', om.[message], Patindex(@CacheUsedSearchString, om.[message]) + Len(@CacheUsedString) + 1)) - (Patindex(@CacheUsedSearchString, om.[message]) + Len(@CacheUsedString) + 1)))) = 1 -- Check LookupBytesUsed
-- catch 0-valued denominator rows
And Convert(bigint, Substring(om.[message], (PatIndex(@LookupNameEndSearchString, om.[message]) + @LookupNameEndLen ), ((CharIndex(' ', om.[message], PatIndex(@LookupNameEndSearchString, om.[message]) + @LookupNameEndLen)) - (PatIndex(@LookupNameEndSearchString, om.[message]) + @LookupNameEndLen)))) <> 0 -- Check LookupRowsCount
Order By operation_id DESC

:{>

Related Training:

IESSIS1: Immersion Event on Learning SQL Server Integration Services April 2017, Chicago

From Zero to Biml - 19-22 Jun 2017, London

IESSIS2: Immersion Event on Advanced Integration Services Oct 2017, Chicago