Loki
Last updated
Was this helpful?
Last updated
Was this helpful?
Each label combination is a set of chunk
There are two index formats that are currently supported as single store with index shipper:
It is extensible and has many advantages over the deprecated BoltDB index. New storage features in Loki are solely available when using TSDB.
The distributor service is responsible for handling incoming push requests from clients. It’s the first step in the write path for log data. Once the distributor receives a set of streams in an HTTP request, each stream is validated for correctness , preprocess the label into beautiful format and to ensure that it is within the configured tenant (or global) limits.
Rate-limit incoming logs based on the maximum data ingest rate per tenant. It does this by checking a per-tenant limit and dividing it by the current number of distributors. say we have 10 distributors and tenant A has a 10MB rate limit. Each distributor will allow up to 1MB/s before limiting
The ingester service is responsible for persisting data and shipping it to long-term storage (Amazon Simple Storage Service, Google Cloud Storage, Azure Blob Storage, etc.) on the write path, and returning recently ingested, in-memory log data for queries on the read path.
Each log stream that an ingester receives is built up into a set of many “chunks” in memory and flushed to the backing storage backend at a configurable interval.
Chunks are compressed and marked as read-only when:
The current chunk has reached capacity (a configurable value).
Too much time has passed without the current chunk being updated
A flush occurs.
If an ingester process crashes or exits abruptly, all the data that has not yet been flushed will be lost. Loki is usually configured to replicate multiple replicas (usually 3) of each log to mitigate this risk.
The ingester validates that log lines are received in timestamp-ascending order. Each log has a timestamp that occurs at a later time than the log before it. When the ingester receives a log that does not follow this order, the log line is rejected and an error is returned.
The query frontend is an optional service providing the querier’s API endpoints and can be used to accelerate the read path. When the query frontend is in place, incoming query requests should be directed to the query frontend instead of the queriers.
Splits larger queries into multiple smaller queries, executing these queries in parallel on downstream queriers and stitching the results back together again
Cache query results and reuses them on subsequent queries.
It fetches log data from both the ingesters and from long-term storage. Queriers query all ingesters for in-memory data before falling back to running the same query against the backend store.
The query frontend queries the index gateway for the log volume of queries so it can make a decision on how to shard the queries.
The ruler service manages and evaluates rule and/or alert expressions provided in a rule configuration. The rule configuration is stored in object storage (or alternatively on local file system)
Monolithic mode is useful for getting started quickly to experiment with Loki, as well as for small read/write volumes of up to approximately 20GB per day.
Running components as individual microservices provides more granularity, letting you scale each component as individual microservices, to better match your specific use case.
The is a container for log entries for a specific set of labels.
The is a table of contents of where to find logs for a specific set of labels.
(recommended)
Time Series Database (or short TSDB) is an originally developed by the maintainers of for time series (metric) data.
(deprecated)
is a low-level, transactional key-value store written in Go.
Each valid stream is then sent to n in parallel, where n is the for data.
While ingesters do support writing to the filesystem through BoltDB, this only works in single-process mode as need access to the same back-end store and BoltDB only allows one process to have a lock on the DB at a given time.
The querier service is responsible for executing queries. The querier can handle HTTP requests from the client directly (in “single binary” mode, or as part of the read path in “simple scalable deployment”) or pull subqueries from the query frontend or query scheduler (in “microservice” mode).
The index gateway service is responsible for handling and serving metadata queries. Metadata queries are queries that look up data from the index. The index gateway is only used by “shipper stores”, such as or .
The compactor service is used by “shipper stores”, such as or , to compact the multiple index files produced by the ingesters and shipped to object storage into single index files per day and tenant.
