Many times, you need to view the latest log entries generated by your app to check that everything is working or see what is currently broken. We will be creating a FastAPI app that will stream our latest log files entries over web sockets.

Monitoring our app

In a previous post, we built a FastAPI web server as part of a data pipeline that receives data from a field device, transforms it and uploads it to Azure. Further to that, we also made the server run as a Windows service, but now we don’t have it running as a console app. Therefore, we cannot print logs to the console (and shouldn’t do it anyway), but we do log everything in a log file. Logs files are great, but we don’t want to log into a server every time and check the last entries in the file.

Seeing that we are running a FastAPI web server, why not make an endpoint available to display our logs in a web page? We can then easily connect via any browser to see what is happening. Some log files may become large, so we don’t want to read the whole file, just the last few lines and add the newly created ones.

Streaming logs over WebSocket

WebSocket is a bidirectional, full duplex communications protocol provided over a single TCP connection. It is a stateful protocol which means the connection between the client and server is kept alive until it is terminated by either party. In comparison, HTTP is a unidirectional protocol that sends a request and receives a response, after which the connection is closed, thus it is slower¹.

The idea is to create a simple HTML endpoint and establish a web socket connection between the client and the server. Once established, we read and display the last 30 lines of a log file and update it every second.

Setting up our server

We start off by creating a simple FastAPI app and importing Request for our HTML page and WebSocket to stream our log file data. Then we add the modules for Jinja2 as we will be using it as our template engine, and we will be using Uvicorn to run our server. It is also here that we define the path and file name of our log file.

from pathlib import Path
from fastapi import FastAPI, WebSocket, Request
from fastapi.templating import Jinja2Templates
import uvicorn
import asyncio

base_dir = Path(__file__).resolve().parent
log_file = "app.log"

app = FastAPI()

templates = Jinja2Templates(directory=str(Path(base_dir, "templates")))

###
# functions and endpoints go here
###

if __name__ == "__main__":
    uvicorn.run(
        "main:app",
        host="localhost",
        port=8000,
        reload=True,
        workers=1,
        debug=True,
    )

There are two endpoints needed: one for our HTML page to display our logs in a browser and a web socket endpoint that will be providing the data.

Create our web socket endpoint

The first function we create is our log_reader. It will open our log file and read the last n-lines, which in our case would be 30. We then iterate through each line to check for specific keywords in our log entry to add custom colours tags to it. The entries will be rendered on our main HTML page in a named div tag. This enables the ERROR and WARNING messages to stand out.

async def log_reader(n=5):
    log_lines = []
    with open(f"{base_dir}/{log_file}", "r") as file:
        for line in file.readlines()[-n:]:
            if line.__contains__("ERROR"):
                log_lines.append(f'<span class="text-red-400">{line}</span><br/>')
            elif line.__contains__("WARNING"):
                log_lines.append(f'<span class="text-orange-300">{line}</span><br/>')
            else:
                log_lines.append(f"{line}<br/>")
        return log_lines

Next, we create our web socket endpoint that the client will connect to and call our log_reader function. Once the connection is established, we will read and send the logs every one-second to the client.

@app.websocket("/ws/log")
async def websocket_endpoint_log(websocket: WebSocket):
    await websocket.accept()

    try:
        while True:
            await asyncio.sleep(1)
            logs = await log_reader(30)
            await websocket.send_text(logs)
    except Exception as e:
        print(e)
    finally:
        await websocket.close()

Render our log viewer page

The next endpoint is a GET endpoint and it is our default page. If you have other pages running, you may want to make it something like "/logs". As we are using templates, we provide some context to use in our Jinja rendered index.html template.

@app.get("/")
async def get(request: Request):
    context = {"title": "FastAPI Streaming Log Viewer over WebSockets", "log_file": log_file}
    return templates.TemplateResponse("index.html", {"request": request, "context": context})

The index.html template page will be using JavaScript to establish a web socket connection from the client to our FastAPI server. We initiate the connection to our endpoint we created above and point it to: ws://<server:port>/ws/log. Note that ws:// is unsecured and you should use wss:// in production.

We add a script function that will trigger every time we receive a new message from the web socket connection. It will find a named div element called logs and replace its contents with our web socket event data, which contain some HTML tags as specified in our log_reader function.

var ws_log = new WebSocket("ws://localhost:8000/ws/log");

ws_log.onmessage = function (event) {
    var logs = document.getElementById("logs");
    var log_data = event.data;
    logs.innerHTML = log_data;
    };

At this point, the page looks like the first web page from 1991! In order to make it more presentable, we add script links to the TailwindCSS CDN and add some style to the various elements. Our final output is a fast web log viewer using random log entries.

Summary

In this blog post we created a log file viewer using FastAPI and streamed it over web sockets. We can now instantly view the last few log entries in a browser, and ensure our application is still running and doing what it is supposed to do.

The full code is available on GitHub. Inspired by blog of Amit Tallapragada.

References

1. What is web socket and how it is different from the HTTP? – GeeksforGeeks
2. Realtime Log Streaming with FastAPI and Server-Sent Events | Amit Tallapragada