I've been doing some Power BI work recently where my customers had on-premises databases that were fairly large. After building a data model in Power BI Desktop that accessed the data directly, we published it up to PowerBI.Com, installed and configured a Power BI Data Gateway, and configured scheduled data refreshes.
In one instance, the on-premises data was hosted by Postgres, and for the other, Sybase SQL Anywhere.
At some point, we decided in each case that we wanted to build a data warehouse (or data mart, if you wish) and use it for the Power BI data model, rather than the source databases directly. The primary reason was that some of the tables were quite large, and most of the data didn't change, so we wanted to load it incrementally. Sales data, in this case - we just load the warehouse staging table with the past 24-48 hours of transactions and incrementally merge it into the main table.
Neither of these customers had SQL Server, but I wanted to use SSIS packages to do the loading, and use Azure SQL database as the target location.
Now that Azure Data Factory V2 supports running SSIS packages, I delved into that configuration. Most of the docs for doing this are pretty good .
However, when it comes to customizing the Azure-SSIS integration runtime, the instructions are ok but I certainly learned a few things along the way. I've done it twice now, and I'm going to show you the things I learned here.
The Azure-SSIS integration runtime is just a docker container running windows Server of some sort. What I mean by docker container is that it is essentially a virtual machine provisioned and started when necessary, from a template containing the OS and whatever other software it needs. You don't really get any console or remote access to this container, but you can have custom configuration steps to install other software as needed.
Be prepared for a long debug cycle though. It takes about 30-40 minutes for the runtime to start, and your custom steps happen at the very end of that. Hopefully the log files that are gathered will show you enough of what happened, or what didn't happen, or what failed, for you to fix your mistake, update your scripts, and try again.
Your configuration happens via an Azure blog storage account, containing a main.cmd file that you write and whatever other files your main.cmd references.
This is what my main.cmd file looks like:
@echo off
powershell.exe -ExecutionPolicy RemoteSigned -File %~dp0\InstallNetFx35.ps1
msiexec /i %~dp0\sqlany12-64bit.msi /passive /l %CUSTOM_SETUP_SCRIPT_LOG_DIR%\sqlanyclient.txt
powershell [reflection.assembly]::Load('iAnywhere.Data.SQLAnywhere.v4.5, Version=12.0.1.41344, Culture=neutral, PublicKeyToken=f222fc4333e0d400') > %CUSTOM_SETUP_SCRIPT_LOG_DIR%\sqlanyverify.logThere are two calls to Powershell and one execution of MSIEXEC, the Microsoft installer tool. I'm going to come back to the first PowerShell call in a bit.
The MSIEXEC call runs the setup for the file sqlany12-64bit.msi. The command-line options for MSIExec are here .
/i installs the package file
/passive means the install process won't stop for any user input
/l defines the logging level and location
There's also some magic tokens in this file: %~dp0 is the current drive and directory of this script. See StackOverflow for some discussions of magic tokens .
The environment variable %CUSTOM_SETUP_SCRIPT_LOG_DIR% is a folder in the docker configuration that will be copied to your Azure Storage log folder, so anything your script puts there you will see.
In my first try, I didn't have the first powershell call (I'll let you guess what it does), and I just had the msiexec line.
When you start the Azure SSIS Integration Runtime (from the ADF portal), after about 30-40 minutes, you'll see a folder in your Azure storage container with the name main.cmd.log. In there will be a folder for the Azure runtime you just tried to start. This screen shot shows the folders created for several attempts at starting my runtime. The first part of the folder is the name of the container and its timestamp, and the part after the ## is another timestamp.
Image may be NSFW.
Clik here to view.
Inside these folders are any files that were put in the magic log directory as well as a stderr.log and stdout.log.
This is what showed up in the sqlanyclient.txt file the first time:
=== Logging started: 10/31/2018 20:41:31 ===Action start 20:41:31: INSTALL.
Action start 20:41:32: AppSearch.
Action ended 20:41:32: AppSearch. Return value 1.
Action start 20:41:32: LaunchConditions.
MSI (s) (08:64) [20:41:32:477]: Product: SQL Anywhere 12 Deployment -- .NET Framework 2.0 is required by this product. .NET Framework 2.0 is required by this product.
Action ended 20:41:32: LaunchConditions. Return value 3.
Action ended 20:41:32: INSTALL. Return value 3.
MSI (s) (08:64) [20:41:32:482]: Product: SQL Anywhere 12 Deployment -- Installation failed. MSI (s) (08:64) [20:41:32:483]: Windows Installer installed the product. Product Name: SQL Anywhere 12 Deployment. Product Version: 12.01.4403. Product Language: 1033. Manufacturer: iAnywhere Solutions. Installation success or error status: 1603.
=== Logging stopped: 10/31/2018 20:41:32 ===
The Az-SSIS docker container also failed to start, telling me that an error occured during the custom installation script. (Thanks!)
So, the setup log tells me that sqlany12-64bit.msi has a dependency on .NET Framework 2.0.
In some ways, this error message is deceiving. I went on a half-day rabbit trail, looking for the install packages for .NET Framework 2.0. It's not hard to find, but it didn't install properly, and it was very quiet about it.
At some point, my subconsious surfaced a vague memory that .NET Framework 3.5 actually includes support for the 2.0 framework. I'm glad it did, but that wasn't the end of my rabbit trail. I tried the full installer for 3.5 and it would not work either, and not give me any log entries telling me why.
After a bit more searching, I tried this:
DISM /Online /Enable-Feature /FeatureName:NetFx3 /All > %CUSTOM_SETUP_SCRIPT_LOG_DIR%\dism.log
My dism.log file contained this:
Deployment Image Servicing and Management toolVersion: 10.0.14393.0
Image Version: 10.0.14393.0
Enabling feature(s)
[ 0.1% ] [ 1.1% ] [==========================100.0%==========================]Error: 0x800f081f
The source files could not be found.Use the "Source" option to specify the location of the files that are required to restore the feat