Centralised logging with Loki and Grafana
Centralised logs make it possible to follow a problem across machines. A small setup can collect Windows event logs and Linux syslog, store the events in Loki, and explore them in Grafana. The first useful outcome is not a large dashboard. It is being able to say what changed and when across every system involved in a problem, without signing in to each host in turn.
Start with a clear question
Before choosing labels, retention, or ingestion rate, write down the questions you expect centralised logs to answer. Examples: which host rejected an authentication attempt in the last hour, what a service wrote just before it stopped responding, or which machines have not checked in today. If you cannot state the question, you cannot design the pipeline to answer it.
Once the questions are clear, choose labels that let you narrow a search quickly. A label for the host's role, its operating system, or the service it runs will group events the way you actually search. Avoid labels whose value changes for every event, such as the full message text or a unique request identifier.
Keep labels stable, keep details in the body
Loki is built around the idea that labels should be few and stable, with the event content carried in the log line itself. The documentation recommends labels that describe the log source using a small set of possible values. It warns that "Loki performs very poorly when your labels have high cardinality" (Grafana Labs, Understand labels). The same page suggests aiming for 10 to 15 labels at most and choosing them based on the terms your users will actually query on.
A label such as host or role fits. A label such as request_id with a unique value per event does not. Loki does not index the text of each log line, so a detail you may want to filter on later does not have to be a label. Where a value is worth keeping but changes from event to event, the documentation points to structured metadata, and that is also where it puts the hostname of a short-lived machine.
Choose labels for the filters that help investigations, while checking how many distinct streams each label creates. A rarely used field may still matter during an incident; keeping it in the body or structured metadata preserves it without making it an indexed label. A lean label set keeps ingestion fast and queries responsive, which matters when you are investigating under time pressure.
Getting the logs in
Check which agent your current notes assume before you copy a configuration. Promtail reached end of life on 2 March 2026, and the Loki documentation now points users at Grafana Alloy (Grafana Labs, Promtail). A setup written up two years ago may still be built around an agent that no longer receives updates.
Transport affects what happens during an outage. UDP is simple, but a receiver restart or a brief network interruption can lose messages without the sender knowing. TCP gives delivery feedback while the connection is active. For delivery you cannot afford to lose, pair the transport with a disk-backed queue at the sender so events wait locally while the central store is unreachable.
Set queue limits and decide what happens when they are reached. No finite buffer absorbs an indefinite outage. When the queue is full you can drop the oldest events, drop the newest, or stop accepting new events locally. Each choice has consequences and the right one depends on what the logs are for.
Monitor the health of the pipeline itself. A quiet dashboard can mean a healthy system, or it can mean the forwarder stopped sending. Track how quickly events arrive, how much disk the store is using, and the last-seen timestamp for each source, so a silent host is noticed rather than assumed to be idle.
How long to keep logs, and who can read them
Set retention from investigation needs, storage capacity, and privacy rules. Keep only the fields and the duration that have a clear purpose. Logs often contain personal or operational details, and the longer they are kept the more they become a liability as well as an asset.
Loki's own retention machinery is worth understanding before you rely on it. Retention is carried out by the compactor, needs retention_enabled together with a delete_request_store, and only works when the index period is 24h (Grafana Labs, Retention). The detail that matters most for planning is that a change to the retention period is not retroactive, so shortening it will not remove logs that have already been ingested.
Limit access to those who need it. In a small environment a single shared account is tempting, but it makes it impossible to say who looked at what. Use named identities and record access to the log store alongside access to the systems it describes.
Alerts that point to decisions
Alerts should point to a meaningful change that needs action. Examples include repeated authentication failures above a sensible threshold, a failed backup job, or a log source going quiet unexpectedly. Group related events and add a recovery condition so the same noise does not repeat itself.
Review alert usefulness after incidents and remove rules that do not lead to a decision. An alert that fires every week and is always ignored is not a safety net. It is a distraction from the alerts that matter.
Checklist
- Write down the questions centralised logs must answer before choosing labels.
- Keep the label set small and stable, and keep details in the log body or in structured metadata.
- Check which agent the current documentation expects before copying a configuration.
- For important sources, use an acknowledged transport together with durable sender buffering where supported. Test retries and delivery after an outage; TCP alone does not prove the receiver stored an event.
- Set queue limits and decide the behaviour when they are reached.
- Monitor ingestion health and a last-seen timestamp for each source.
- Set retention from investigation needs and privacy rules, and remember that a change does not apply retroactively.
- Review alerts after incidents and remove rules that do not lead to action.