Implement SFTP File Transfer with SQL Server Integration Services and PSFTP

By: Koen Verbeeck || Related Tips:More > Integration Services Development

Problem

I want to transfer files from an FTP server to my local server. The server is an SFTP server however, so I can’t use the Integration Services FTP task. How can I resolve this?

Solution

In this tip, we’ll use the free tool PSFTP (from the PuTTy software family) in combination with SQL Server Integration Services (SSIS) to download a file from an SFTP server.

Some remarks:

This tip is an update from the tip Using SFTP with SQL Server Integration Services . There were a couple of questions about the use of PSFTP in the comments and this tip will try to answer those. There are plenty of other tools which also can handle SFTP (or FTPS) file transfers, such as WinSCP . This tip focuses solely on PSFTP, but the method used is very much similar when you would use other tools. The tip uses basic authentication with username and password. Complex set-up with public/private keys is out of scope. You don’t really need SSIS to transfer files with SFTP, you could use PSFTP (or other tools) from batch scripts just as easily. But typically, such a file transfer is part of a larger ETL process orchestrated by SSIS. For example, after downloading the file it could be ingested and written to a database, after which the file is archived. All of this can also be done in the same SSIS work flow. The example package in this tip is created with SSDT 15.7.3 for Visual Studio 2017. The project settings are configured to target SQL Server 2017, but the provided solution is valid for at least SSIS 2012 and up (probably SSIS 2005 and 2008 as well, but this hasn’t been tested). SFTP File Transfer with SSIS Test Set-up

When you want to transfer a file with SFTP, you need an SFTP server of course. The SolarWinds SFTP server is free to use and easy to set up. In the settings, we configure the root directory of the SFTP server, which is a folder on your hard drive:

In the TCP/IP settings, you can optionally change the port:

In the Users tab, we add a user called mssqlTIPS and we specify a password:

Click OK to exit the settings. Don’t forget to actually start the SFTP server. You might need administrative permissions to do this. You can also start the service from the windows Services Manager (services.msc).

As the last step of the test set-up, we place a random file in the root directory. This file will be “downloaded” by SSIS and PSFTP:

Creating the SSIS Package

Add a new package to your SSIS project. On the control flow, add an Execute Process Task . This task will call the PSFTP executable.

Open the editor of the task. Configure it as follows:

RequireFullName : this configures if the task fails if the executable cannot be found at the specified path. If the executable is added to the PATH environment variable, you can set this to False . Executable : the name of the executable you want to execute, in our case psftp . If you don’t have psftp added to the PATH environment variable, you can either copy psftp.exe to the working directory, or you can specify the full path to the executable: C:\Program Files\PuTTY\psftp.exe. Arguments : the arguments that you want to pass to psftp. First, we specify the user and the server name, using the syntax [emailprotected] . Then we specify the password with the switch -pw. Then the switch -be is used, which means continue on error when using a batch file. We also use the -batch switch to avoid any interactive prompts. Finally, we specify a batch file containing all the SFTP commands we want to execute on the SFTP server, using the -b switch. Typically, you’ll want to use an expression to pass the value to the arguments, so you can use parameters to make them more dynamic. WorkingDirectory : this is the location where we want to put the downloaded file. WindowStyle : this needs to be set to Hidden. We don’t want any interactive windows.

A full list of all the arguments you can pass to psftp (including how to specify an alternative port) can be found here . The contents of the batch file SFTPCommands.bat are the following:

cd MSSQLTIPS
get BimlBasics.pptx

With the command cd , we specify the directory on the SFTP server we want to browse to. With the get command, we indicate which file we want to download to our working directory.

The Execute Process Task will give a warning when the psftp is not located in the working directory:

Let’s test the package. After running it, you should see the file in the working directory:

You can also verify on the SFTP server itself:

Deploying the Package to a Server

Normally you’re not going to run the package manually, but rather schedule it on a server. In this tip, we’ll use SQL Server Agent to schedule the package. There’s a bit of prep work that needs to be done:

Make sure psftp is also downloaded and installed on the server. Optionally add it to the PATH environment variable there as well. You can either use the SQL Server Agent account to execute the SSIS package, but best practice is to use a proxy account. The tip Running a SSIS Package from SQL Server Agent Using a Proxy Account explains how you can set this up. Give the proxy account read/write permissions on the working directory and execute permissions on PSFTP. Use the same folder structure for your working directory, or parameterize your set-up. Adjust any firewall settings if necessary. Deploy the SSIS package/project