h3xagnh3xagn
  • blog
  • about
  • twitter
  • reddit

azure

Data Engineering

Extract dark data from SQL Server

Many systems exist today that were simply not optimised for time-series data analysis and machine learning. People may not even know the data is being collected regularly as part of day-to-day operations. This is often referred to as dark data. As the hunger for data increases, data engineers are tasked to make this dark data available.

By Coenraad Pretorius, 9 monthsAugust 21, 2022 ago
Data Engineering

Building an ETL pipeline from device to cloud (bonus part) 

Over this six plus one blog series we covered many different technologies and integrations, as we built our data pipeline. While there are standard patterns we can often use, sometimes we need to be able to flexible and use what is available.

By Coenraad Pretorius, 1 yearMay 14, 2022 ago
Data Engineering

Building an ETL pipeline from device to cloud (part 6)

In this part we set up our big data platform, Azure Data Explorer, and create ingestion events to continually ingest our transformed data. Finally, we add context to our data by creating a data model and exporting it to our data lake.

By Coenraad Pretorius, 1 yearMay 7, 2022 ago
Data Engineering

Building an ETL pipeline from device to cloud (part 5)

We discuss the importance of proper data management upfront and the need to always be in control of data sharing. We send the appropriate data automatically to our vendor via an Azure Function App.

By Coenraad Pretorius, 1 yearApril 30, 2022 ago
Data Engineering

Building an ETL pipeline from device to cloud (part 4)

We added a new uploader method to store data in Azure Blob storage and started to deal with leaky data pipelines by handling failed uploads.

By Coenraad Pretorius, 1 yearApril 24, 2022 ago

Recent Posts

  • Four steps to systematically optimise your energy consumptionJanuary 18, 2023
  • Streaming logs using RabbitMQNovember 20, 2022
  • Extract dark data from SQL ServerAugust 21, 2022
  • Give your cleverHome one voiceJuly 7, 2022
  • Create a streaming log viewer using FastAPIJune 16, 2022

Categories

  • Data Engineering
  • Data Science
  • Energy Efficiency
  • Internet of Things

Don’t miss another post

We don’t spam! You can unsubscribe anytime.

Check your inbox or spam folder to confirm your subscription.

  • blog
  • about
  • twitter
  • reddit
Hestia | Developed by ThemeIsle