File Maintenance Cleaning Up Old Files

Using SSIS to Maintain the File System We have all run into a need or a desire to clean up old stale files from the file system, whether it be to remove old backup files or flat files that are created from one process or

another. And based on this need/desire, we have all come up with a method to help with achieve that goal.

Some of the methods might be to include a flag in a maintenance plan that may be used. Other methods may be to use a SQL script employing xp_cmdshell and delete statements. Yet another may utilize the sp_oa stored procs and DMO. And still others may have ventured into powershell to accomplish the same task. The point is, there are many methods.

I am adding yet another method to the mix. Why? I didn’t much like the option of using the sp_oa method or the xp_cmdshell route. I am very novice with powershell and it would take a lot more tinkering to get the script working properly. Also, I felt pretty comfortable with SSIS and had approval to try and get this done using that method. And just because I am a novice with powershell, does not mean that I will not circle back around to try and accomplish this task via that means.

Note: This article was originally written in 2011 and got stuck in an unpublished state. Things have changed since then so i will definitely be circling back around for a powershell version.

Requirements

The method employed needs to be able to do the following:

Remove multiple file types Be configurable Clean out files from numerous directories Remove files older than a specified number of days. Setup

The solution I chose utilizes SSIS. It also requires that there be a table in a database that helps to drive the package.

The table looks like the following.

USE [AdminDB_Test] GO /****** Object:Table [dbo].[FilePaths]Script Date: 12/05/2011 23:36:26 ******/ IFEXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[dbo].[FilePaths]') AND type in (N'U')) DROP TABLE [dbo].[FilePaths] GO USE [AdminDB_Test] GO /****** Object:Table [dbo].[FilePaths]Script Date: 12/05/2011 23:36:26 ******/ SET ANSI_NULLS ON GO SET QUOTED_IDENTIFIER ON GO SET ANSI_PADDING ON GO CREATE TABLE [dbo].[FilePaths]( [PathID] [int] IDENTITY(1,1) NOT NULL, [Process] [varchar](32) NULL, [FilePath] [varchar](256) NULL, CONSTRAINT [PK_FilePaths] PRIMARY KEY CLUSTERED ( [PathID] ASC )WITH (PAD_INDEX= OFF, STATISTICS_NORECOMPUTE= OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS= ON, ALLOW_PAGE_LOCKS= ON) ON [PRIMARY] ) ON [PRIMARY] GO SET ANSI_PADDING OFF GO

The filepath column holds the FileSystem Path for each directory that needs to be cleaned. Paths that are supported are local (e.g. C:\temp ) and unc paths (\\machine\c$\temp). I set this attribute to a length of 256, but if you have a longer path, you will want to adjust the length.

The Process column will hold a value describing what that path relates to, such asMaintainDirectory. In my example, I am usingMaintainDirectory to control which directories hold files that potentially need to be deleted.

Here is an example of the contents of that table I am using currently.

The last piece of the setup before we start working on the SSIS package is the need for a string splitting function. Pick the string splitter of your liking. I have one that I like and am sure you have one that you prefer. The SSIS package relies on the return field from the splitter being named “Item.” If it is named something else, please make the adjustments in the package as necessary.

The Package

The package I created has been created in SSIS 2008. To meet the requirements already set forth, I utilized the following objects: ADO.Net Data Source, 2 Execute SQL Tasks, 2 ForEach Loop Containers, a Script Task, and 8 variables. Let’s take a look at these starting with the variables.

Variables SQLServerName The value held here is used in an Expression for the Data Source. This will overwrite the ServerName value in the Data Source. DatabaseName Used alongside the SQLServerName variable in an Expression for the Data Source. This value will overwrite the InitialCatalog value in the Data Source. This should be the name of the database where the FilePaths table and String Split function exist. DaysToKeep This value is the cutoff point for which files to keep and which files will be deleted. This variable is used as a ReadOnly variable in the Script Task. obj_FileExtension This object variable is used to store the result set from one of the Execute SQL tasks and the results of the string split function from the FileExtensionList variable. FileExtensionList This is a delimited list of file extensions that need to be evaluated for deletion. It is important to note that the file extensions that are to be processed are case sensitive. The extension must appear in this list as it appears in the file system. FileExtension to be used in one of the ForEach loops. This variable will receive the FileExtension from the obj_FileExtension variable one at a time. obj_ListOfDirectories This variable will receive the result set of an Execute SQL Task to be later consumed by one of the ForEach loops. DirectoryToMaintain receives one at a time the Directory to process for file deletion. The ForEach loop stores a value from obj_ListOfDirectories in this variable for processing. Execute SQL Tasks

The two Execute SQL Tasks are simple in function. One is to get the list of directories to maintain from the FilePaths table. The other is strictly to split the string for the FileExtensionList variable.

The first is named “Get Directory List” and should receive the Full Result Set from the following query.

select fp.PathID,fp.Process,fp.FilePath from dbo.FilePaths FP where fp.Process = 'MaintainDirectory' order by fp.PathID

The Result Set tab of this task also needs to be modified. it should look like this.

From this task, we flow to the next Execute SQL Task named “Split FileList.” The setup of this task is very much like the previous task. We want to receive the full result set. We have a configuration to make on the result set tab. We also need to map a parameter. Let’s take a look at those real quick.

Parameter Mapping

Result Set

And this is the query that we will be executing.

Select Item From stringsplitter(@FileExtensionList,',')

Notice that the Parameter we named in the Parameter Mapping tab is being used in the function call. I chose this method because I could see and understand how it works better.

ForEach Loops

The next stop in the flow is the ForEach Loop Directory object. As the name implies, this ForEach Loop is designed to work with the obj_ListOfDirectories variable/array.

With this first Loop container, we have two tabs that need to be configured in the properties. Both Loop containers are similar in that they need the same tabs to be configured. First, let’s talk about the Collection tab.