systemctl logs: A Guide to Managing Logs in Linux

When a Linux service fails, one of the first things developers check is systemctl. It’s the standard way to manage and inspect system services, and it’s often where debugging starts.

This guide covers how to work with systemctl logs, viewing, filtering, and managing them efficiently. We’ll also touch on where these methods fall short in larger environments, and how teams move beyond basic log inspection as systems scale.

What Are `systemctl` Logs?

systemctl is a tool for managing services on Linux systems that use systemd. It’s commonly used to start and stop services, check their status, and review logs.

These logs come from journald, the built-in logging service that collects messages from services, the kernel, and other sources. Using systemctl, you can quickly access logs for any service without needing to set up an external logging tool.

Why Use `systemctl` Logs?

systemctl logs give you a window into how services behave over time. They show what happened, when it happened, and which part of the system was involved.

For smaller setups or during early debugging, this is often enough. You can check if a service crashed, see the exact error message, or track what happened during startup.

Common uses:

Check why a service failed or didn’t restart
Monitor recurring warnings or errors
Catch config mistakes or permission issues
Review output from custom background services

But as your setup grows, with more nodes, containers, or dynamic workloads, scanning logs on individual machines becomes harder to manage. It’s not just about collecting logs anymore; it’s about being able to use them to find patterns, spot failures quickly, and understand how different parts of your system affect each other.

That’s where relying only on systemctl starts to fall short, and why many teams move to centralized tools that make this process less manual.

💡

If you're also looking to understand where different logs live on disk, here's a quick reference to common Linux log file locations.

Useful `journalctl` Commands

The journalctl command is your go-to tool for reading logs collected by systemd-journald. These examples cover the most useful flags and filters when you're working with Linux service logs:

Command	What it does
`journalctl`	Show all logs, newest first
`journalctl -u <service-name>`	Show logs for a specific service (e.g., `nginx`)
`journalctl -f`	Stream logs live (like `tail -f`)
`journalctl --since "YYYY-MM-DD"`	Show logs from a specific date
`journalctl -b`	Show logs from the current boot
`journalctl -p <priority>`	Filter by priority (e.g., `err` for errors)
`journalctl -n <number>`	Show the last N lines (e.g., `-n 100`)
`journalctl --vacuum-time=2weeks`	Remove logs older than two weeks
`journalctl --vacuum-size=500M`	Keep logs within a 500MB size limit

These are the basics most teams use for local debugging, especially when diagnosing failed services, reviewing reboots, or catching errors in real time.

How to View Logs Using `systemctl` (via `journalctl`)

While systemctl is used to manage services, logs are read using the journalctl command. It pulls data from the systemd journal (journald) and gives you several ways to inspect logs, whether you're checking a crash or monitoring services live.

Here’s how to use it effectively:

1. View All Logs

journalctl

This prints all logs collected by systemd, starting with the oldest and moving forward. You’ll see entries from all services, kernel messages, and system events.

For long-running systems, this output can be large; it’s better to apply filters as shown below.

2. View Logs for a Specific Service

journalctl -u nginx

Replace nginx with the name of any systemd-managed service. This shows only logs related to that service — including startup messages, runtime errors, and shutdown events.

You can also combine it with other flags for more control. For example:

journalctl -u nginx --since "2024-12-01"

Shows nginx logs from December 1st onward.

3. Watch Logs in Real Time

journalctl -u nginx -f

This is like running tail -f on a log file. New log lines will appear as they’re written — helpful during deployments or when tracking a recurring issue.

To exit, press Ctrl + C.

4. Filter Logs by Time

journalctl --since "2024-12-01" --until "2024-12-09"

This limits log output to entries within a specific date range. You can also use relative time formats like:

journalctl --since "1 hour ago"

This is useful when investigating incidents that happened recently.

5. View Logs for Boot Sessions

Each time your machine reboots, systemd starts a new "boot session." You can filter logs by boot ID:

Previous boot:

journalctl -b -1

Current boot session:

journalctl -b

You can also go back multiple sessions:

journalctl -b -2

This helps when you’re trying to debug a service that failed before the system came back up.

💡

If you're tracking down strange service behavior or unexpected reboots, this guide on Linux event logs walks through how to use logs to spot and understand those issues.

Filter Logs for Faster Debugging

On active systems, logs can grow fast. Without filters, you’ll spend more time scrolling than solving problems. Here are a few ways to cut through the noise and find what you need faster:

1. Filter by Priority

System logs are tagged with priority levels — from 0 (emergency) to 7 (debug). You can filter by level to focus on warnings or errors:

journalctl -p err

This shows only entries marked as errors (err = priority 3). You can also use:

warning (level 4)
info (level 6)
debug (level 7)

Example:

journalctl -u nginx -p warning

Shows warning-level messages for the nginx service only.

💡

Understanding these priority levels is crucial for effective logging — learn more in our log levels guide.

2. Search Within Logs Using Keywords

If you’re looking for a specific error message or phrase, use grep:

journalctl -u nginx | grep "connection failed"

This returns only lines from nginx logs that match the keyword.

You can also combine it with time or boot filters for more precision.

3. Limit the Output Size

To avoid dumping hundreds of lines, use -n to show only the most recent entries:

journalctl -n 50

This shows the last 50 log lines across all services. To do the same for a specific service:

journalctl -u nginx -n 50

Useful when reviewing recent changes or checking the tail of the logs without paging through too much output.

Practical Tips for Managing `systemctl` Logs

Logs can take up a lot of space and become noisy fast. Here's how to keep things in check without overcomplicating your setup.

1. Limit Log Size

By default, journald will keep logs until it runs out of disk space. You can cap usage by editing this config:

# /etc/systemd/journald.conf
[Journal]
SystemMaxUse=1G

This limits total journal size to 1GB. After changing it, restart the logging service:

sudo systemctl restart systemd-journald

Other options you can set:

SystemKeepFree – keeps some disk free for other processes
SystemMaxFileSize – limits individual log file size

2. Forward Logs Elsewhere

If you need logs for longer than a few days, or across multiple machines, don’t rely on local storage. You can forward them to another system using:

A remote syslog server
A log shipper like Fluentd or Logstash
A cloud logging agent (if you're on AWS/GCP)

This makes it easier to search logs later and build alerts around them.

3. Clean Up Old Logs

You can clear logs manually using built-in journalctl flags:

By disk space:

journalctl --vacuum-size=500M

By time:

journalctl --vacuum-time=2weeks

No extra tools needed, just a one-liner to keep disk usage in check.

4. Watch for Errors Automatically

Logs are only helpful if you know something went wrong. You can plug your logs into a system that tracks known patterns and sends alerts, like failed service restarts or permission errors.

Options include:

Prometheus (via exporters or custom scripts)
Last9, for setups where you want to monitor logs, metrics, and traces together, without juggling separate tools
Loki with Promtail

💡

For a closer look at how logs work on Ubuntu systems specifically, check out this guide on finding and using Ubuntu system logs.

Troubleshooting with `systemctl` Logs

When a service misbehaves, fails to start, crashes, or starts acting slow, logs are usually the fastest way to figure out what’s going on. Since systemctl pulls directly from the systemd journal, it gives you structured access to service-level logs without needing to dig through scattered log files.

Here’s how to approach common issues using systemctl logs:

1. Service Fails to Start

Start with:

journalctl -u <service-name>

This shows you logs tied to the service's startup process. Look for:

Missing config files
Port binding failures
Permission errors
Failed dependencies

These are usually near the bottom of the output. If the logs are too noisy, filter by priority:

journalctl -u <service-name> -p err

This shows only error-level entries, making it easier to isolate the failure.

2. Service Starts, But Something’s Off

If a service runs but behaves unpredictably — high CPU usage, long response times, memory leaks — check for repeated errors or warnings:

journalctl -u <service-name> --since "30 minutes ago"

You may see timeouts, retries, or low-level exceptions. These patterns help narrow down what the service is struggling with, even before you get to CPU or memory metrics.

3. Crashes After Boot or Unexpected Restarts

To debug crashes that happen during or after system startup, use boot filtering:

journalctl -b            # Logs from the current boot
journalctl -b -1         # Logs from the previous boot

This is especially useful after a crash or forced reboot — you can check what happened just before the system went down. Look for:

Kernel panics
OOM kills (out-of-memory)
Segfaults or abrupt shutdown signals

If the crash is related to a specific service, add -u <service-name> to scope the logs.

4. You’re Checking This Too Often

If you find yourself running these checks repeatedly, especially during incidents, it's time to look into centralized log and telemetry pipelines. When logs, metrics, and traces are connected, debugging gets much faster.

Last9 lets you tie service logs directly to performance metrics and request traces, so you don’t have to dig through multiple tools just to confirm what failed and when. You get the full picture: what happened, where, and how it impacted your systems.

Security and Performance Considerations

Logs aren't just helpful for debugging; they can also introduce risk if access isn’t managed properly or if log volume starts to affect system performance. Here’s what to keep in mind.

Log Access and Permissions

System logs often include sensitive information: authentication failures, internal IPs, or debug traces that expose application behavior. Access to these logs is controlled by file permissions and user groups.

To check who can access journal logs:

ls -l /var/log/journal/

By default, only users in the systemd-journal group (or root) can read detailed logs.

To grant access:

sudo usermod -a -G systemd-journal <username>

Then verify:

groups <username>

Why it's important:

Prevents accidental exposure of system internals
Limits who can see audit logs or authentication attempts
Helps meet internal compliance or audit requirements

Log Storage and System Load

Too much logging can lead to disk bloat, slower IO, or even missed entries during high load. It’s important to monitor both usage and impact.

Check current log usage:

journalctl --disk-usage

This shows how much disk space is currently used by the journal.

For real-time disk write monitoring:

sudo iotop -o

If you’re seeing sustained writes from systemd-journald, consider lowering verbosity in your services or enabling log rotation and cleanup (as described earlier).

To inspect memory use by the journal service:

systemctl status systemd-journald

Logs in Containerized Environments

If you’re running services inside containers, especially in Kubernetes, journalctl won’t give you pod-level visibility. You’ll need to switch to tools like kubectl logs for that.

Integrate `systemctl` Logs with Other Tools

Working directly with journalctl is fine for a single server. But if you're managing multiple machines, containers, or VMs, you'll want to push logs into something searchable and structured.

Here's how to wire up common tools, and how Last9 fits into the workflow.

View Logs in Grafana with Loki

You can use Loki to collect and query logs through Grafana. It's a good option if you're already running Grafana for metrics.

Step 1: Run Loki

docker run -d -p 3100:3100 grafana/loki:latest

Step 2: Set Up Promtail to Read from journald

Edit /etc/promtail/config.yml:

server:
  http_listen_port: 9080

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
  - job_name: journal
    journal:
      path: /var/log/journal
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'

Restart Promtail after changes. Logs should now appear in your Loki instance and be queryable via Grafana.

💡

Running services in Kubernetes? Here's how Kubernetes logs work and what to watch out for when debugging pods and containers.

View Logs from Docker Containers

By default, Docker uses the json-file log driver, but you can switch it to journald so container logs are captured by systemd.

Send logs to journald:

docker run --log-driver=journald my-container

Query by container name:

journalctl CONTAINER_NAME=my-container

This is useful for environments where journald is your central collector, and you don't want to depend on file-based logging.

Send Logs to Last9

If you're already using Promtail to ship logs to Loki, you can point Promtail to Last9 instead. Last9 supports ingestion via the same interfaces and lets you query logs alongside metrics and traces, without maintaining a full observability stack yourself.

Example config snippet for Promtail → Last9:

clients:
  - url: https://logs.last9.io/loki/api/v1/push
    tenant_id: your-tenant-id

This works out of the box if you're already collecting systemd logs with Promtail. Just update the push endpoint, and your logs will start showing up in Last9 — searchable by labels like _SYSTEMD_UNIT, CONTAINER_NAME, or MESSAGE.

You don’t need to manage Grafana, Loki, or a separate storage backend; Last9 takes care of log ingestion and links it with your metrics and traces. If your team already uses Grafana, Last9 also provides an embedded Grafana instance for a familiar querying and dashboard experience.

Wrapping Up

Working with systemctl and journalctl is part of day-to-day Linux operations. These tools help you inspect how services behave, find failures, and trace issues back to their source.

But local logs only take you so far. As systems scale, more nodes, more containers, more services, it gets harder to rely on manual inspection. You need a way to search across logs, link them to metrics and traces, and spot problems faster.

And Last9 helps you here. You can send logs using tools like Promtail, then connect that data to everything else in your stack, without maintaining a separate backend.

Get started with us for free today!

🤝

And, if you'd like to continue the conversation, feel free to join our community on Discord. We have a dedicated channel where you can share your specific use case and connect with other developers.

FAQs

What is the systemctl command used for?

The systemctl command is a tool for managing and controlling systemd services on Linux systems. It lets you start, stop, restart, enable, or disable services. Additionally, systemctl is used to view the status of services and check logs generated by systemd. System administrators need to ensure that services are running properly and that the system stays healthy.

How do I view logs of a specific service using systemctl?

To view logs for a specific service, you can use the journalctl command with the -u option followed by the service name. For example:

journalctl -u nginx

This command will show all the logs related to the nginx service, including error messages, startup times, and any issues that may arise during its operation.

Can I view logs in real-time using systemctl?

Yes, you can! By using the -f option with journalctl, you can view logs in real-time, just like using tail -f. For example:

journalctl -u nginx -f

This will stream the logs of the nginx service as new entries are added, allowing you to monitor the service live as it runs.

How do I filter systemctl logs by date?

You can filter logs by specific dates using the --since and --until options. For example:

journalctl --since "2024-12-01" --until "2024-12-09"

This command will display logs between December 1st and December 9th, 2024. You can also use relative time filters, such as --since "1 hour ago" or --since "yesterday", for more dynamic queries.

What are the priority levels in systemctl logs?

systemctl logs are categorized by severity levels, ranging from 0 (emergency) to 7 (debug). These priority levels allow you to filter logs based on their importance. You can use the -p option to filter logs by priority. For example:

journalctl -p err

This will display only error messages. Other priority levels include alert, critical, warning, and info. Each level helps you focus on the most urgent logs.

How can I clean up old logs using systemctl?

Logs can accumulate quickly, so it’s essential to clean them up periodically. You can remove old logs using the --vacuum-time or --vacuum-size options. For example:

journalctl --vacuum-time=2weeks

This will delete logs older than two weeks. Alternatively, you can limit the journal size to free up disk space:

journalctl --vacuum-size=500M

This will delete logs until the total journal size is below 500MB.

How do I monitor logs for multiple services at once?

You can monitor logs for multiple services by running separate journalctl commands in parallel using different terminal windows. Alternatively, you can use grep to filter logs from multiple services. For example:

journalctl -u nginx | grep "error"
journalctl -u apache2 | grep "error"

This will allow you to search for errors across different services in real-time, making it easier to monitor multiple services simultaneously.

What is the difference between systemctl and journalctl?

While systemctl is used to manage system services and control their states (such as starting, stopping, or enabling them), journalctl is a command-line tool specifically for viewing and managing logs generated by systemd services. In short, systemctl handles the management of services, while journalctl deals with logging and troubleshooting.

How can I forward systemctl logs to an external server?

To forward systemctl logs to an external server, you can configure systemd to send logs to a syslog server. Modify the journald configuration in /etc/systemd/journald.conf and set the ForwardToSyslog option to yes. For centralized log collection, tools like Fluentd or Logstash can be used to collect and forward logs to external systems. This is helpful for long-term log storage or compliance purposes.

How do I view logs for a specific boot session?

To view logs for the current boot session, use:

journalctl -b

If you want to view logs from the previous boot, use:

journalctl -b -1

This is useful for troubleshooting issues related to a specific system startup.

How do I filter logs by specific keywords or strings?

If you’re looking for specific events in the logs, you can pipe journalctl with grep to search for certain keywords. For example:

journalctl -u nginx | grep "connection failed"

This will only show logs where the phrase "connection failed" appears, making it easier to locate relevant log entries.

How can I monitor disk usage from logs?

Heavy logging can affect system performance. To monitor the disk usage of logs, use:

journalctl --disk-usage

This will show you the current disk usage of system logs. You can also use iotop it to monitor real-time disk writes, which helps you understand how logs are affecting your system’s resources.