With the increased need to extract and analyse data over the last couple of years and the convergence of OT and IT systems, new possibilities are available to improve the traditional industrial data stack. This has the potential not just to make data available securely, but also reduce complexity, latency, Read more
Many systems exist today that were simply not optimised for time-series data analysis and machine learning. People may not even know the data is being collected regularly as part of day-to-day operations. This is often referred to as dark data. As the hunger for data increases, data engineers are tasked to make this dark data available.
Over this six plus one blog series we covered many different technologies and integrations, as we built our data pipeline. While there are standard patterns we can often use, sometimes we need to be able to flexible and use what is available.
In this part we set up our big data platform, Azure Data Explorer, and create ingestion events to continually ingest our transformed data. Finally, we add context to our data by creating a data model and exporting it to our data lake.
We discuss the importance of proper data management upfront and the need to always be in control of data sharing. We send the appropriate data automatically to our vendor via an Azure Function App.