Logging

Level

• Using log levels, a security analyst can decide the priority and the characteristic of log easily

• Minimum number of priority can be set to filter out some of less important log

• Here is the common logging level and its priority number

const logger = require('./logger');

logger.fatal('fatal');
logger.error('error');
logger.warn('warn');
logger.info('info');
logger.debug('debug');
logger.trace('trace');

// output:
// {"level":60,"time":1643664517737,"pid":20047,"hostname":"fedora","msg":"fatal"}
// {"level":50,"time":1643664517738,"pid":20047,"hostname":"fedora","msg":"error"}
// {"level":40,"time":1643664517738,"pid":20047,"hostname":"fedora","msg":"warn"}
// {"level":30,"time":1643664517738,"pid":20047,"hostname":"fedora","msg":"info"}
// {"level":20,"time":1643664517738,"pid":20047,"hostname":"fedora","msg":"debug"}
// {"level":10,"time":1643664517738,"pid":20047,"hostname":"fedora","msg":"trace"}

System Design

• The logging system can be mainly divided into several components

Log accumulator

• It can be placed on se­rvers or devices. The­ agents collect logs on the de­vices themselve­s. They then send the­ collected logs to a central logging syste­m

Log aggregator

• To stream processing huge log datas from different sources

• To consolidates logs from different sources into a centralized location. This centralization simplifies log management and ensures that all logs are stored in a unified manner, making it easier to search and analyze them.

• To label the log and store the logs into storage

• To integrate with analysis tool and handle the query from frontend and fetch the result from storage or cache

• To support the creation of custom monitoring and alerting rules to detect anomalies, threshold breaches, or specific patterns in log data, triggering notifications or automated actions when predefined conditions are met.

Log Visualizer

• Act as a frontend for user to output a log

• Allow user to enter query, setting up the rules and alert based on the data source

Stack

• There are common stacks/ practices for implementing the design - PLG / ELK

PLG

Promtail (P)

• Act as a log accumulator, can be installed as a daemon set on different machines

• To collect the log from different applications and then send to loki

Loki (L)

• The distributor service is responsible for handling incoming streams by clients. It’s the first stop in the write path for log data. Once the distributor receives a set of streams, each stream is validated for correctness and to ensure that it is within the configured tenant (or global) limits. Valid chunks are then split into batches and sent to multiple ingesters in parallel.
• Distributors use consistent hashing in conjunction with a configurable replication factor to determine which instances of the ingester service should receive a given stream.

• The ingester service is responsible for writing log data to long-term storage backends (DynamoDB, S3, Cassandra, etc.) on the write path and returning log data for in-memory queries on the read path.

• The querier service handles queries using the LogQL query language, fetching logs both from the ingesters and from long-term storage.

Grafana (G)

• Act as a log visualizer

ELK

Logstash (L)

• Act as a log accumulator or data pipeline. It collects logs from various sources, stream processing the data and finally sends it to Elastic search or other destinations

Elastic Search (E)

• Act as a log aggregator

• Unlike loki , it focus on storing the log data

• To index the data it receives, enabling fast and efficient search capabilities. It uses inverted index structures to build indexes on the data, allowing for quick retrieval of relevant information based on search queries.

Kibana (K)

• Act as frontend for viewing log

References

Centralized Logging Systems | System Design - GeeksforGeeksGeeksforGeeks

ELK Stack Tutorial: What is Kibana, Logstash & Elasticsearch?Guru99

Loki components | Grafana Loki documentationGrafana Labs

[DevOps]Grafana + Loki + Promtail Logging system 小試牛刀Medium