The Data Engineer Blog

Posted on December 25, 2019

In this article, we will try to give a brief overview of the SQL string functions used in SQL Server and we will provide some examples. A string function is a function that takes a string value as an input regardless of the data type of the returned value. In... [Read More]

Posted on December 16, 2019

Radiation pollution has been always a critical concern, since it can cause a huge damage to humans and for nature. To minimize the damage, governments are collecting and monitoring radiation level using advanced systems. In the past years, Big data technologies such as distributed file systems, NoSQL databases and stream... [Read More]

Posted on December 9, 2019

In this article, we will describe SQL Server Profiler and how it can be used. In addition, we will mention the tools that are integrated with it- plans about its deprecation, and what are the alternatives?

Posted on December 4, 2019

In this article, first, we will briefly describe foreach loops and for loops. Then, we will give an overview of SSIS Foreach Loop Container and For Loop Container. Finally, we will compare these two containers.

Posted on November 22, 2019

In this article, we will give a brief overview of SSIS Balanced Data Distributor (BDD). In addition, we will give some examples and illustrate alternatives.

Posted on November 21, 2019

The Data Profiling task in SSIS is an important task that can be used to assess the quality of data sources. Unfortunately, this component is not widely used by many business intelligence developers. [Read More]

Posted on November 14, 2019

In this article, we illustrate how to use the OFFSET FETCH feature as a solution for loading large volumes of data from a relational database using a machine with limited memory and preventing an out of memory exception. We describe how to load data in batches to avoid placing a... [Read More]

Posted on October 29, 2019

SSIS Script component is a prominent strength of SQL Server Integration Services since it allows developers to implement any complex logic and utilize libraries from the powerful .NET framework or third-parties. In this article, we will give an overview of SSIS Script component, and illustrate the benefits of using it.... [Read More]

Posted on October 29, 2019

In this article, we will give an overview of using Flat Files and Raw Files in SSIS, then we will illustrate some of the differences between using these two types.

Posted on October 11, 2019

Until now, Microsoft has not included the XML Destination component in SQL Server Integration Services (SSIS). Many years ago, this component was requested on the Microsoft connect website, but it was closed as “Won’t fix.” For this reason, many workarounds and third-party components were created. In this article, we’ll talk... [Read More]

Posted on October 4, 2019

In this article, I will first give an overview of SSIS data types and data types conversion methods and then I will illustrate the difference between changing the columns data types from the Source Advanced editor and using Data Conversion Transformation.

Posted on October 4, 2019

In this article, I will first give an overview of the OLE DB, ODBC, and ADO.NET SSIS connection managers. Then I will try to illustrate the difference between them when trying to connect to SQL Server.

Posted on September 25, 2019

In this article, we will first give an overview on SSIS derived column transformation, then we will run an experiment to check if there is any difference between adding multiple expressions within one derived column transformation and adding a derived column transformation for each expression.

Posted on September 17, 2019

In this article, I will describe the difference between using output parameters and result sets within Execute SQL Task in SSIS.

Posted on September 9, 2019

In this article, I will give an overview of Execute SQL Task in SSIS and I will try to illustrate some of the differences between writing an expression to evaluate SqlStatementSource property or writing this expression within a variable and change the Execute SQL Task Source Type to variable.

Posted on August 30, 2019

In this article, I will give an overview of SSIS OLE DB Destination and SQL Server Destination and I will try to illustrate some of the difference between both destination components based on my personal experience, SSIS official documentation and some other experts experience in this domain.

Posted on August 27, 2019

In this article, I will first give an introduction about SSIS expressions, then I will describe briefly the Expression Task and how to Evaluate a variable as expression. Then I will do a comparison between these two features to illustrate the similarities and differences between them.

Posted on August 22, 2019

SQL Server Integration Services provides a wide variety of features that helps developers to build a robust Extract, Transform and Load process. After many years contributing to SSIS related tags on Stackoverflow.com, I can say that many developers have some misunderstanding about SSIS features (SSIS OLE DB Source, SSIS Expressions,... [Read More]

Posted on August 22, 2019

Handling NUL (\x00) value in flat files using SQL Server Integration Services.

Posted on August 15, 2019

While building and deploying an SSAS OLAP cube, there are two processing orders that you can choose from when you create a process operation: Parallel (All Objects will be processed in a single transaction): Used for batch processing, all tasks run in parallel inside one transaction. Sequential (Transaction mode): One... [Read More]

Posted on August 14, 2019

Using NumberFormat property while manipulating Microsoft Excel worksheets using SSIS.

Posted on July 29, 2019

How can I extract all objects mentioned in a SQL Command using C#?

Posted on June 16, 2019

Posted on June 14, 2019

Explaining the validation types performed when using OLE DB destination component in SSIS

Posted on April 25, 2019

Trying to fix out the date time format issue in SQL Server Reporting Services.