Data Engineering
Building a home energy monitoring dashboard for your clevrHome
Build a comprehensive home energy monitoring dashboard with solar, hot water, and mains power tracking using TimeScaleDB and Grafana.
Data Engineering
Build an OPC UA bridge in Python
Many times, during development one comes across a gap in a project that is easy to fix in essence but there are other factors at play. One such example from my work is the need for an OPC UA bridge. In this post we will explore the use cases that I have come across and build an OPC UA bridge in Python.
Data Engineering
Streaming logs using RabbitMQ
More data stacks are shifting towards microservices and communication between them are essential as it sometimes need to communicate to more than one service. Message brokers play a critical part in this architecture by allowing services to communicate by routing messages and storing them. In this post we will be creating a real-time log viewer that consolidates errors across several apps and a python app to process error logs.
Data Engineering
Extract dark data from SQL Server
Many systems exist today that were simply not optimised for time-series data analysis and machine learning. People may not even know the data is being collected regularly as part of day-to-day operations. This is often referred to as dark data. As the hunger for data increases, data engineers are tasked to make this dark data available.
Data Engineering
Create a streaming log viewer using FastAPI
Many times, you need to view the latest log entries generated by your app to check that everything is working or see what is currently broken. We will be creating a FastAPI app that will stream our latest log files entries over websockets.