Loki

Architecture

Storage

Chunk

• The chunk is a container for log entries for a specific set of labels.

• Each label combination is a set of chunk

Index

• The index is a table of contents of where to find logs for a specific set of labels.

• There are two index formats that are currently supported as single store with index shipper:

• TSDB (recommended)

  Time Series Database (or short TSDB) is an index format originally developed by the maintainers of Prometheus for time series (metric) data.

  It is extensible and has many advantages over the deprecated BoltDB index. New storage features in Loki are solely available when using TSDB.

• BoltDB (deprecated)

  Bolt is a low-level, transactional key-value store written in Go.

Components

Distributor

• The distributor service is responsible for handling incoming push requests from clients. It’s the first step in the write path for log data. Once the distributor receives a set of streams in an HTTP request, each stream is validated for correctness , preprocess the label into beautiful format and to ensure that it is within the configured tenant (or global) limits.

• Rate-limit incoming logs based on the maximum data ingest rate per tenant. It does this by checking a per-tenant limit and dividing it by the current number of distributors. say we have 10 distributors and tenant A has a 10MB rate limit. Each distributor will allow up to 1MB/s before limiting

• Each valid stream is then sent to n ingesters in parallel, where n is the replication factor for data.

Ingester

• The ingester service is responsible for persisting data and shipping it to long-term storage (Amazon Simple Storage Service, Google Cloud Storage, Azure Blob Storage, etc.) on the write path, and returning recently ingested, in-memory log data for queries on the read path.

• Each log stream that an ingester receives is built up into a set of many “chunks” in memory and flushed to the backing storage backend at a configurable interval.

  Chunks are compressed and marked as read-only when:

  1. The current chunk has reached capacity (a configurable value).

  2. Too much time has passed without the current chunk being updated

  3. A flush occurs.

• If an ingester process crashes or exits abruptly, all the data that has not yet been flushed will be lost. Loki is usually configured to replicate multiple replicas (usually 3) of each log to mitigate this risk.

• The ingester validates that log lines are received in timestamp-ascending order. Each log has a timestamp that occurs at a later time than the log before it. When the ingester receives a log that does not follow this order, the log line is rejected and an error is returned.

• While ingesters do support writing to the filesystem through BoltDB, this only works in single-process mode as queriers need access to the same back-end store and BoltDB only allows one process to have a lock on the DB at a given time.

Query Frontend

• The query frontend is an optional service providing the querier’s API endpoints and can be used to accelerate the read path. When the query frontend is in place, incoming query requests should be directed to the query frontend instead of the queriers.

• Splits larger queries into multiple smaller queries, executing these queries in parallel on downstream queriers and stitching the results back together again

• Cache query results and reuses them on subsequent queries.

Querier

• The querier service is responsible for executing Log Query Language (LogQL) queries. The querier can handle HTTP requests from the client directly (in “single binary” mode, or as part of the read path in “simple scalable deployment”) or pull subqueries from the query frontend or query scheduler (in “microservice” mode).

• It fetches log data from both the ingesters and from long-term storage. Queriers query all ingesters for in-memory data before falling back to running the same query against the backend store.

Index Gateway

• The index gateway service is responsible for handling and serving metadata queries. Metadata queries are queries that look up data from the index. The index gateway is only used by “shipper stores”, such as single store TSDB or single store BoltDB.

• The query frontend queries the index gateway for the log volume of queries so it can make a decision on how to shard the queries.

Compactor

• The compactor service is used by “shipper stores”, such as single store TSDB or single store BoltDB, to compact the multiple index files produced by the ingesters and shipped to object storage into single index files per day and tenant.

Ruler

The ruler service manages and evaluates rule and/or alert expressions provided in a rule configuration. The rule configuration is stored in object storage (or alternatively on local file system)

Deployment mode

Single Binary

• Monolithic mode is useful for getting started quickly to experiment with Loki, as well as for small read/write volumes of up to approximately 20GB per day.

Distributed

• Running components as individual microservices provides more granularity, letting you scale each component as individual microservices, to better match your specific use case.

References

Loki overview | Grafana Loki documentationGrafana Labs