Downloading multiple files from internet with SSIS and C#

Introduction

Sometimes we need to download the files from the FTP, a Web Site or another resource as the source data for an ETL process.In this article, we will show how to download multiple files from internet using SQL Server Integration Services (SSIS) and the script task. We will learn 4 things:

We will first show how to download a single file using the Script task. Secondly, we will show how to download multiple files using the foreach loop>item enumerator and the script task. We will also show how to send all the files to download using a C# array. Finally, we will use the ADO object enumerator using the foreach loop with the script task. Requirements SQL Server installed. SSIS and SQL Server Data Tools (SSDT). Excel installed (for the example 3 only) How to Download a Single File Using the SSIS Script Task

In the first example, we will download a single file from internet. You can use any website of your preference. In this example, we are using a website named www.textfiles.com/adventure . This URL contains several files to download:

We will open SSDT and create a new SSIS project. In the SSIS Project, go to the menu and click SSIS>Variables:

We will create a string variable named filename and specify the name of the internet text to download. In this example, 221baker.txt is the name of one of the files on the www.textfiles.com\adventure web site:

In SSIS Toolbox in the SSIS project, drag and drop the Script Task:

You can program in C# (by default) or VB. We will work in C# this time. Also in ReadWriteVariables, add the filename variable created before and press Edit Script:

In the C# code expand region and add the System.Net namespace. System.Net will be used to download files in internet:

In the public void main function, add this code:

try
{
string myURI = "http://www.textfiles.com/adventure/";
string fileName = Dts.Variables["User::filename"].Value.ToString(), myWebString = null, myLocalPath = null;
myWebString = myURI + fileName;
myLocalPath = "c:\\sql\\" + fileName;
WebClient myWebClient = new WebClient();
myWebClient.DownloadFile(myWebString, myLocalPath);
Dts.TaskResult = (int)ScriptResults.Success;
}
catch (Exception ex)
{
Dts.Events.FireError(18, "The process failed", ex.ToString(), "", 0);
Dts.TaskResult = (int)ScriptResults.Failure;
}

The code will download a file named 221baker.txt. We first use the try and catch code.Try and catch are used to handle errors. If the code in the try fails, catch will show the error message.

We have in the code the variable definition. myURI stores the website URL with the files. The filename is a string that will receive the SSIS variable named filename created previously. The SSIS variable contains the filename. myWebString is the web concatenated with the filename. myLocalPath is the local drive where we will store the file created (c:\sql).

string myURI = "http://www.textfiles.com/adventure/";
string fileName = Dts.Variables["User::filename"].Value.ToString(), myWebString = null, myLocalPath = null;
myWebString = myURI + fileName;
myLocalPath = "c:\\sql\\" + fileName;

The next section is used to download the file. We create a webclient object and then we use the download function to download the file using the variables created before:

WebClient myWebClient = new WebClient();
myWebClient.DownloadFile(myWebString, myLocalPath);

Once the code is done, save it and close it. Right click the script task and select the Execute Task option:

If everything is OK, you will be able to find the 221baker.txt file in your local c:\sql folder:

How to Download Multiple files Using the Foreach Loop

In the first example, we learned how to download a single file. In this new example, we will download several files from a list using the Foreach Loop Container. This Container is used to use a Loop a task. The Item enumerator allows creating a list manually. In this example, we will create a list of files and the script task will copy the files using that list.

In SSIS, double click the Foreach Loop Container:

In the design pane, move the script task created in the example one inside the Container:

In the Foreach Loop Container, select Foreach Item Enumerator and press the Column button:

In the column created, add the list of files names in the web:

In Variable Mapping, select the filename variable:

If everything is OK, you will have all the files of the list created when you run the package.

How to Send All the Files to Download Using a C# Array

The example 2 is the best option if you have few files. However, in our web page, there more than 400 files in the website. It would take a long time to copy and paste the names using the item enumerator. The new example will show how to create an array with the list of files.

We will first go to www.textfiles.com/adventure and copy the filenames:

Copy and paste the results into Excel:

We will pivot the table to create a list of files in the format requested by C#. Select the Filename column and go to INSERT>Pivot Table:

In Create PivotTable, press OK:

Drag Filename to COLUMNS:

Copy the file names to a new Excel file:

Save the file as a CSV file:

You will receive a message about some features that will be lost. Press Yes:

Open the csv file with a text editor: