The Data Engineer Blog

Posted on May 11, 2020

In this article, we will give a brief introduction of Hadoop and how it is integrated with SQL Server. Then, we will illustrate how to connect to the Hadoop cluster on-premises using the SSIS Hadoop connection manager and the related tasks.

Posted on May 11, 2020

In this article, we will briefly explain the Avro and ORC Big Data file formats. Then, we will be talking about Hadoop data flow task components and how to use them to import and export data into the Hadoop cluster. Then we will compare those Hadoop components with the Hadoop... [Read More]

Posted on May 5, 2020

This article is a part of a series that we are publishing on TowardsDataScience.com that aims to illustrate how to install Big Data technologies on Windows operating system.

Posted on May 4, 2020

While working on a project two years ago, I wrote a step-by-step guide to install Hadoop 3.1.0 on Ubuntu 16.04 operating system. Since we are currently working on a new project where we need to install a Hadoop cluster on Windows 10, I decided to write a guide for this... [Read More]

Posted on April 22, 2020

In this article, we will explain what JSON is, what are the SQL Server JSON functions introduced in 2016, how to index JSON values, and how to import JSON data into SQL tables.

Posted on April 18, 2020

While working on a project, we were asked to install Apache Hive on a Windows 10 operating system. Many guides were found online but unfortunately, they didn’t work. For this reason, I decided to write a step-by-step guide to help others.

Posted on April 14, 2020

Glad to annouce that we have created a NuGet package for the SchemaMapper class library. You can check it on the following link: NuGet Gallery - SchemaMapper 1.1.0

Posted on April 2, 2020

DTExec is a tool used to configure and execute SQL Server Integration Services (SSIS) packages. Many developers and database administrators use this tool, especially when automating the package execution using third-party applications or non-supported programming languages. This article will introduce this tool to you, also help you understand how and... [Read More]

Posted on March 26, 2020

In this article, we will illustrate another Biml alternative, which is the EzApi class library, and we will make a comparison between both technologies.

Posted on March 25, 2020

In this article, we will first illustrate how to create, save and execute SSIS packages using ManagedDTS in C#, then we will do a small comparison with Biml.

Posted on March 18, 2020

In this article, we will explain the File Transfer Protocol (FTP), and we will give an overview of the FTP task in SQL Server Integration Services (SSIS) and FTP connection manager.

Posted on March 18, 2020

In this article, we will explain how to store these VB or C# scripts within external files and use them within Biml scripts. Also, we will mention how to create extension methods within these C# scripts.

Posted on March 16, 2020

In this article, we will talk about sp_WhoIsActive stored procedure and how we can use it to monitor currently running activities in SQL Server.

Posted on March 16, 2020

In this article, we will explain BimlScript, which is an extension of this markup language with VB or C# scripts.

Posted on March 13, 2020

In this article, we will explain how to generate scripts from existing SSIS packages by showing all related options. Then, we will analyze the generated script and identify how each object is mentioned in the script.

Posted on March 13, 2020

In this article, we will explain how to generate SSIS packages using business intelligence markup language by providing a step-by-step guide where we will illustrate how to configure each object and how to use variables, expressions and parameters within the package.

Posted on March 4, 2020

In this article, we will give an overview of the Execute T-SQL Statement Task, and we will make a small comparison with the Execute SQL Task which is more popular.

Posted on March 2, 2020

What is the SSIS package GraphLayout? Is it readable in programmatically generated SSIS package?

Posted on February 26, 2020

In this article, we will give a brief overview of BIML by showing how it can be installed and how we can use it to generate SSIS packages using the BimlExpress tool. Finally, we will provide some good resources to learn this technology.

Posted on February 12, 2020

In this article, we will give an overview of the first two tasks provided by Integration Services (SSIS) that allows manipulating OLAP cubes and executing commands over SSAS. These two tasks are Analysis Services Processing Task and Analysis Services Execute DDL Task.

Posted on February 6, 2020

In this article, we will explain how a string to date conversion can be achieved implicitly, or explicitly in SQL Server using built-in functions such as CAST(), TRY_CAST(), CONVERT(), TRY_CONVERT() and TRY_PARSE().

Posted on January 28, 2020

SQL join multiple tables is one of the most popular types of statements executed while handling relational databases. As known, there are five types of join operations: Inner, Left, Right, Full and Cross joins. In this article, we will explain the meaning of Joins in SQL, we will describe each... [Read More]

Posted on January 23, 2020

SQL cursor is one of the most popular database objects. It is used to retrieve data from the result set of an SQL query one row at a time. Even if the cursor is not recommended from a performance perspective, they are still widely used especially when handling a small... [Read More]

Posted on January 13, 2020

This article is a supplement to the previously published article, An overview of SQL String Functions . This time, we will describe another bunch of SQL string functions used in SQL Server.

Posted on January 10, 2020

One of the most asked questions is how to link an OLAP cube with a relational database, or in another way how to join the result of an MDX query with a table stored in a relational database. In this article, we will illustrate how to link a SQL Server... [Read More]