šŸŒµOldMartijntje

LGTM Stack - personal attempt

3 min read

ID: I202605301410
Status: idea
Tags: Observability, OpenTelemetry

LGTM Stack - personal attempt

Warning

This note is not a guide. This note is a note where I track the steps that I am taking to try and figure out how it works. That said, this note could help you debug / find possible solutions to obscure issues.

We are going to use: Prometheus, Tempo, Loki, Grafana.

Stack overview

What is Loki

Loki serves as the centralized log aggregation solution, streamlining the collection and querying of application and system logs. Its schema-less architecture ensures flexibility, while its efficient storage design minimizes resource usage. Loki is particularly useful for debugging, searching error logs, and tracking system events without the complexity of traditional log management systems.

What is Grafana

Grafana is the visualization powerhouse of the stack, combining metrics, logs, and traces in customizable dashboards. With robust alerting capabilities and seamless integration with various data sources, Grafana enables real-time monitoring of system health and the creation of unified observability dashboards tailored to business needs.

What is Tempo

Tempo simplifies distributed tracing by tracking requests across microservices, helping teams pinpoint issues in complex environments. It integrates with OpenTelemetry for standardized instrumentation and offers lightweight storage to keep infrastructure costs in check. Tempo excels in root cause analysis and mapping service dependencies, making it an essential tool for tracing.

What is Prometheus

Prometheus is an open-source time-series database and monitoring system designed for collecting and querying metrics from distributed systems. With a pull-based scraping model and powerful query language (PromQL), Prometheus enables real-time performance monitoring and alerting. Itā€™s the industry standard for observability in containerized and microservices environments, providing efficient metric storage and flexible alerting capabilities for system health and performance tracking.

My journey

For this project Iā€™ll setup 2 containers. 1 template javascript application that is an API, and 1 template typescipt application that the javascript application interacts with through https://test.mosquitto.org/ with the 1883 port. This way I can test logging, and traceability through applications. And I just like MQTT lol.

So I created an empty repo: https://github.com/oldmartijntje/LGTM-stack-testing

Then I added 2 containers and ran the docker compose locally.

I ran docker compose up on the docker compose. this makes the containers.

Now for the actual important stuff:

  • How do I actually use the LGTM stack
  • How do I create traces, logs, etc. from code.

I open grafana on http://localhost:3050/dashboards. go to connections ā†’ data sources. Here I click ā€œadd data sourceā€ and click Prometheus and set itā€™s connection url to http://prometheus:9090. Then I did the same for Loki and Tempo, Where Loki connects to http://loki:3100 and tempo to http://tempo:3200. I left all other options like authentication empty.

Then I went to Dashboards and created a new dashboard. By adding my first Panel, I quickly got overwhelmed by all the options.

I am actually having issues with visualizing data that i am supposedly exporting.

References

Graph View