The Data Engineer Blog

Posted on July 18, 2024

The goal of this tutorial is to enable ETL developers to obtain practical knowledge to exploit SSIS in transforming and combining data through hands-on exploration to be ready for real-world challenges in managing information. We’ll examine and discuss some of the most essential components and functionalities. We will begin by... [Read More]

Posted on May 8, 2024

In this tutorial, we delve into several critical aspects of data quality management within the context of time-series data. We will use Python in this article to demonstrate working with time-series data.

Posted on April 23, 2024

In this article, we look at how to visualize time-series data using Excel with a walkthrough of how this is done with a sample dataset.

Posted on February 20, 2024

In this article, I’ll show you how to check out an app’s log using the Python Matplotlib library and see when warnings and errors happen.

Posted on January 10, 2024

In this article, I will briefly discuss how context awareness can be applied in different life sectors such as education, religion, and others.

Posted on December 12, 2023

This article explains some of the popular SQL Server monitoring tools and techniques.

Posted on September 29, 2023

This step-by-step technical article will explore how to perform data profiling on a database using SQL code. We will use a combination of SQL queries to gather various metrics and statistics about the data, such as column data types, null values, data length, and more. Ultimately, all SQL code portions... [Read More]

Posted on August 3, 2023

I will briefly remind you about the data wrangling concept in this article. Then, I will dive into the SQL Server functions used to create a custom data wrangling tool.

Posted on July 19, 2023

This article explains how implicit conversions affect SQL query performance and cause the query optimizer to perform an index scan instead of index seek while searching for a specific value.

Posted on June 30, 2023

Inadequate data quality can lead to significant financial or business opportunity losses. Still, many SQL Server data practitioners are unaware of the available data quality solutions. This article briefly explains SQL Server’s solutions to assess and improve data quality.

Posted on June 27, 2023

This article emphasizes the significance of data quality in data-driven strategies and highlights the consequences of implementing data warehousing scenarios that need more attention to data quality. It introduces the concept of data quality management and discusses the ISO/IEC 25012 standard’s fifteen data quality characteristics. The article explains the different... [Read More]

Posted on June 14, 2023

This article explains what database monitoring is and why it is essential. Then, it illustrates the different methods for monitoring MongoDB NoSQL databases.

Posted on June 13, 2023

The term data quality refers to measuring the fitness of data regarding the intended usage. Poor data quality leads to inadequate, inconsistent, and erroneous decisions that could escalate the computational cost, cause a decline in profits, and cause customer churn. Thus, data quality is crucial for researchers and industry practitioners.... [Read More]

Posted on June 13, 2023

In the big data domain, data quality assessment operations are often complex and must be implementable in a distributed and timely manner. This paper tries to generalize the quality assessment operations by providing a new ISO-based declarative data quality assessment framework (BIGQA). BIGQA is a flexible solution that supports data... [Read More]

Posted on June 3, 2023

In this article, we will examine the effectiveness of several popular AI detectors by subjecting them to a human-written paragraph and an AI-generated text. Subsequently, we will analyze and discuss the outcomes obtained from these tests.

Posted on April 24, 2023

When working with databases, it is common to encounter scenarios where you must search for data within a specific date range. Fortunately, Microsoft SQL Server provides several methods for performing these types of searches with T-SQL. This SQL tutorial illustrates some of the most common techniques for searching between two... [Read More]

Posted on April 5, 2023

In this article, we will be asking several questions about prompt engineering. It is worth noting that while ChatGPT has been trained over a vast amount of data up to 2021, prompt engineering has only recently become a buzzword in 2023.

Posted on March 29, 2023

Interviewing ChatGPT is an article series where we delve into various technical domains by conversing with ChatGPT. With its knowledge base and natural language processing capabilities, ChatGPT can provide insights and answers to various technical questions. In this article, we will be asking several questions about data engineering.

Posted on March 26, 2023

In this article, we will focus on three essential skills that every computer scientist should have in 2023 but are often overlooked: efficient Google searching, effective use of Stack Overflow for asking and answering questions, and having the ability to ask good questions to intelligent virtual assistants like ChatGPT By... [Read More]

Posted on March 15, 2023

This tutorial explains different data types that can be used to store date time values and illustrates when to use each one in scripts, stored procedures and more.

Posted on March 6, 2023

This article explains the different approaches used to insert multiple rows into SQL Server tables.

Posted on March 1, 2023

This tutorial, illustrates the different use cases and options available when using the WHERE clause and demonstrates them using the AdventureWorks2017 database.

Posted on February 10, 2023

I am very thankful for being awarded the MSSQLTips.com Rookie of the year. I am pleased that this award was based on the SSIS Bad Habits article series I published in 2022. A special thanks to Jeremy Kadlec, who encouraged me to complete this article series before the end of... [Read More]

Posted on February 6, 2023

This article illustrates how to use a Microsoft SQL Server Integration Services (SSIS) external configuration file to enable the properties of package objects to be dynamically updated at run time.

Posted on February 3, 2023

This article provides more than 50 questions that can be used within an SSIS job interview.