# systemctl logs: A Guide to Managing Logs in Linux

> Learn how to manage and view systemctl logs in Linux with this guide, covering essential commands and best practices for troubleshooting.

Source: https://last9.io/blog/systemctl-logs/

When a Linux service fails, one of the first things developers check is `systemctl`. It’s the standard way to manage and inspect system services, and it’s often where debugging starts.

This guide covers how to work with `systemctl` logs, viewing, filtering, and managing them efficiently. We’ll also touch on where these methods fall short in larger environments, and how teams move beyond basic log inspection as systems scale.

**How do you view systemctl logs?** Use `journalctl -u servicename` to view logs for a specific service. Add `-f` to follow in real time, `-n 100` for the last 100 lines, or `--since "1 hour ago"` to filter by time. For a quick status check with the last few log lines inline, run `systemctl status servicename`. All systemd logs are written by `journald` and persist across reboots by default.

## What Are `systemctl` Logs?

`systemctl` is a tool for managing services on Linux systems that use `systemd`. It’s commonly used to start and stop services, check their status, and review logs.

These logs come from `journald`, the built-in logging service that collects messages from services, the kernel, and other sources. Using `systemctl`, you can quickly access logs for any service without needing to set up an external logging tool.

### Why Use `systemctl` Logs?

`systemctl` logs give you a window into how services behave over time. They show what happened, when it happened, and which part of the system was involved.

For smaller setups or during early debugging, this is often enough. You can check if a service crashed, see the exact error message, or track what happened during startup.

Common uses:

- Check why a service failed or didn’t restart
- Monitor recurring warnings or errors
- Catch config mistakes or permission issues
- Review output from custom background services

But as your setup grows, with more nodes, containers, or dynamic workloads, scanning logs on individual machines becomes harder to manage. Collecting logs is no longer enough. You need to _use_ them to find patterns, spot failures quickly, and understand how different parts of your system affect each other.

That’s where relying only on `systemctl` starts to fall short, and why many teams move to centralized tools that make this process less manual.

If you're also looking to understand where different logs live on disk, here's a quick reference to [common Linux log file locations](https://last9.io/blog/linux-log-file-locations/).

## Useful `journalctl` Commands

The `journalctl` command is your go-to tool for reading logs collected by `systemd-journald`. These examples cover the most useful flags and filters when you're working with Linux service logs:

| Command                           | What it does                                     |
| --------------------------------- | ------------------------------------------------ |
| `journalctl`                      | Show all logs, newest first                      |
| `journalctl -u <service-name>`    | Show logs for a specific service (e.g., `nginx`) |
| `journalctl -f`                   | Stream logs live (like `tail -f`)                |
| `journalctl --since "YYYY-MM-DD"` | Show logs from a specific date                   |
| `journalctl -b`                   | Show logs from the current boot                  |
| `journalctl -p <priority>`        | Filter by priority (e.g., `err` for errors)      |
| `journalctl -n <number>`          | Show the last N lines (e.g., `-n 100`)           |
| `journalctl --vacuum-time=2weeks` | Remove logs older than two weeks                 |
| `journalctl --vacuum-size=500M`   | Keep logs within a 500MB size limit              |

These are the basics most teams use for local debugging, especially when diagnosing failed services, reviewing reboots, or catching errors in real time.

## How to View Logs Using `systemctl` (via `journalctl`)

While `systemctl` is used to manage services, logs are read using the `journalctl` command. It pulls data from the systemd journal (`journald`) and gives you several ways to inspect logs, whether you're checking a crash or monitoring services live.

Here’s how to use it effectively:

#### 1\. View All Logs

```bash
journalctl
```

This prints all logs collected by `systemd`, starting with the oldest and moving forward. You’ll see entries from all services, kernel messages, and system events.

For long-running systems, this output can be large; it’s better to apply filters as shown below.

#### 2\. View Logs for a Specific Service

```bash
journalctl -u nginx
```

Replace `nginx` with the name of any systemd-managed service. This shows only logs related to that service, including startup messages, runtime errors, and shutdown events.

You can also combine it with other flags for more control. For example:

```bash
journalctl -u nginx --since "2024-12-01"
```

Shows nginx logs from December 1st onward.

#### 3\. Watch Logs in Real Time

```bash
journalctl -u nginx -f
```

This is like running `tail -f` on a log file. New log lines will appear as they’re written. This is helpful during deployments or when tracking a recurring issue.

To exit, press `Ctrl + C`.

#### 4\. Filter Logs by Time

```bash
journalctl --since "2024-12-01" --until "2024-12-09"
```

This limits log output to entries within a specific date range. You can also use relative time formats like:

```bash
journalctl --since "1 hour ago"
```

This is useful when investigating incidents that happened recently.

#### 5\. View Logs for Boot Sessions

Each time your machine reboots, systemd starts a new "boot session." You can filter logs by boot ID:

Previous boot:

```bash
journalctl -b -1
```

Current boot session:

```bash
journalctl -b
```

You can also go back multiple sessions:

```bash
journalctl -b -2
```

This helps when you’re trying to debug a service that failed before the system came back up.

If you're tracking down strange service behavior or unexpected reboots, this guide on [Linux event logs](https://last9.io/blog/linux-event-logs-your-troubleshooting-guide/) walks through how to use logs to spot and understand those issues.

## Filter Logs for Faster Debugging

On active systems, logs can grow fast. Without filters, you’ll spend more time scrolling than solving problems. Here are a few ways to cut through the noise and find what you need faster:

#### 1\. Filter by Priority

System logs are tagged with priority levels — from `0` (emergency) to `7` (debug). You can filter by level to focus on warnings or errors:

```bash
journalctl -p err
```

This shows only entries marked as errors (`err` = priority 3). You can also use:

- `warning` (level 4)
- `info` (level 6)
- `debug` (level 7)

Example:

```bash
journalctl -u nginx -p warning
```

Shows warning-level messages for the `nginx` service only.

Understanding these priority levels is crucial for effective logging. Learn more in our [log levels guide](https://last9.io/blog/log-levels-explained/).

#### 2\. Search Within Logs Using Keywords

If you’re looking for a specific error message or phrase, use `grep`:

```bash
journalctl -u nginx | grep "connection failed"
```

This returns only lines from `nginx` logs that match the keyword.

You can also combine it with time or boot filters for more precision.

#### 3\. Limit the Output Size

To avoid dumping hundreds of lines, use `-n` to show only the most recent entries:

```bash
journalctl -n 50
```

This shows the last 50 log lines across all services. To do the same for a specific service:

```bash
journalctl -u nginx -n 50
```

Useful when reviewing recent changes or checking the tail of the logs without paging through too much output.

## Practical Tips for Managing `systemctl` Logs

Logs can take up a lot of space and become noisy fast. Here's how to keep things in check without overcomplicating your setup.

#### 1\. Limit Log Size

By default, `journald` will keep logs until it runs out of disk space. You can cap usage by editing this config:

```ini
# /etc/systemd/journald.conf
[Journal]
SystemMaxUse=1G
```

This limits total journal size to 1GB. After changing it, restart the logging service:

```bash
sudo systemctl restart systemd-journald
```

Other options you can set:

- `SystemKeepFree` – keeps some disk free for other processes
- `SystemMaxFileSize` – limits individual log file size

#### 2\. Forward Logs Elsewhere

If you need logs for longer than a few days, or across multiple machines, don’t rely on local storage. You can forward them to another system using:

- A remote syslog server
- A log shipper like Fluentd or Logstash
- A cloud logging agent (if you're on AWS/GCP)

This makes it easier to search logs later and build alerts around them.

#### 3\. Clean Up Old Logs

You can clear logs manually using built-in `journalctl` flags:

By disk space:

```bash
journalctl --vacuum-size=500M
```

By time:

```bash
journalctl --vacuum-time=2weeks
```

No extra tools needed, just a one-liner to keep disk usage in check.

#### 4\. Watch for Errors Automatically

Logs are only helpful if you know something went wrong. You can plug your logs into a system that tracks known patterns and sends alerts, like failed service restarts or permission errors.

Options include:

- [Prometheus](https://last9.io/blog/prometheus-and-grafana/) (via exporters or custom scripts)
- [Last9](https://last9.io/), for setups where you want to monitor logs, metrics, and traces together, without juggling separate tools
- [Loki](https://last9.io/blog/loki-for-log-management/) with [Promtail](https://grafana.com/docs/loki/latest/send-data/promtail/)

For a closer look at how logs work on Ubuntu systems specifically, check out this guide on [finding and using Ubuntu system logs](https://last9.io/blog/ubuntu-system-logs-how-to-find-and-use-them/).

## Troubleshooting with `systemctl` Logs

When a service misbehaves, fails to start, crashes, or starts acting slow, logs are usually the fastest way to figure out what’s going on. Since `systemctl` pulls directly from the systemd journal, it gives you structured access to service-level logs without needing to dig through scattered log files.

Here’s how to approach common issues using `systemctl` logs:

#### 1\. Service Fails to Start

Start with:

```bash
journalctl -u <service-name>
```

This shows you logs tied to the service's startup process. Look for:

- Missing config files
- Port binding failures
- Permission errors
- Failed dependencies

These are usually near the bottom of the output. If the logs are too noisy, filter by priority:

```bash
journalctl -u <service-name> -p err
```

This shows only error-level entries, making it easier to isolate the failure.

#### 2\. Service Starts, But Something’s Off

If a service runs but behaves unpredictably (high CPU, long response times, memory leaks), check for repeated errors or warnings:

```bash
journalctl -u <service-name> --since "30 minutes ago"
```

You may see timeouts, retries, or low-level exceptions. These patterns help narrow down what the service is struggling with, even before you get to CPU or memory metrics.

#### 3\. Crashes After Boot or Unexpected Restarts

To debug crashes that happen during or after system startup, use boot filtering:

```bash
journalctl -b            # Logs from the current boot
journalctl -b -1         # Logs from the previous boot
```

This is especially useful after a crash or forced reboot. You can check what happened just before the system went down. Look for:

- Kernel panics
- OOM kills (out-of-memory)
- Segfaults or abrupt shutdown signals

If the crash is related to a specific service, add `-u <service-name>` to scope the logs.

#### 4\. You’re Checking This Too Often

If you find yourself running these checks repeatedly, especially during incidents, it's time to look into centralized log and telemetry pipelines. When logs, metrics, and traces are connected, debugging gets much faster.

[Last9](https://last9.io/) lets you tie service logs directly to performance metrics and request traces, so you don’t have to dig through multiple tools just to confirm what failed and when. You get the full picture: what happened, where, and how it impacted your systems.

## Security and Performance Considerations

Logs aren't just helpful for debugging; they can also introduce risk if access isn’t managed properly or if log volume starts to affect system performance. Here’s what to keep in mind.

#### Log Access and Permissions

System logs often include sensitive information: authentication failures, internal IPs, or debug traces that expose application behavior. Access to these logs is controlled by file permissions and user groups.

To check who can access journal logs:

```bash
ls -l /var/log/journal/
```

By default, only users in the `systemd-journal` group (or root) can read detailed logs.

To grant access:

```bash
sudo usermod -a -G systemd-journal <username>
```

Then verify:

```bash
groups <username>
```

Why it's important:

- Prevents accidental exposure of system internals
- Limits who can see audit logs or authentication attempts
- Helps meet internal compliance or audit requirements

#### Log Storage and System Load

Too much logging can lead to disk bloat, slower IO, or even missed entries during high load. Monitor both usage and impact.

Check current log usage:

```bash
journalctl --disk-usage
```

This shows how much disk space is currently used by the journal.

For real-time disk write monitoring:

```bash
sudo iotop -o
```

If you’re seeing sustained writes from `systemd-journald`, consider lowering verbosity in your services or enabling log rotation and cleanup (as described earlier).

To inspect memory use by the journal service:

```bash
systemctl status systemd-journald
```

#### Logs in Containerized Environments

If you’re running services inside containers, especially in Kubernetes, `journalctl` won’t give you pod-level visibility. You’ll need to switch to tools like `kubectl logs` for that.

## Integrate `systemctl` Logs with Other Tools

Working directly with `journalctl` is fine for a single server. But if you're managing multiple machines, containers, or VMs, you'll want to push logs into something searchable and structured.

Here's how to wire up common tools, and how Last9 fits into the workflow.

#### View Logs in Grafana with Loki

You can use Loki to collect and query logs through Grafana. It's a good option if you're already running Grafana for metrics.

**Step 1: Run Loki**

```bash
docker run -d -p 3100:3100 grafana/loki:latest
```

**Step 2: Set Up Promtail to Read from journald**

Edit `/etc/promtail/config.yml`:

```yaml
server:
  http_listen_port: 9080

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
  - job_name: journal
    journal:
      path: /var/log/journal
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: ["__journal__systemd_unit"]
        target_label: "unit"
```

Restart Promtail after changes. Logs should now appear in your Loki instance and be queryable via Grafana.

Running services in Kubernetes? Here's how [Kubernetes logs](https://last9.io/blog/kubernetes-logs/) work and what to watch out for when debugging pods and containers.

#### View Logs from Docker Containers

By default, Docker uses the `json-file` log driver, but you can switch it to `journald` so container logs are captured by systemd.

**Send logs to journald:**

```bash
docker run --log-driver=journald my-container

```

**Query by container name:**

```bash
journalctl CONTAINER_NAME=my-container

```

This is useful for environments where journald is your central collector, and you don't want to depend on file-based logging.

#### Send Logs to Last9

If you're already using Promtail to ship logs to Loki, you can point Promtail to [Last9](https://last9.io/docs/onboard/) instead. Last9 supports ingestion via the same interfaces and lets you query logs alongside metrics and traces, without maintaining a full observability stack yourself.

**Example config snippet for Promtail → Last9:**

```yaml
clients:
  - url: https://logs.last9.io/loki/api/v1/push
    tenant_id: your-tenant-id
```

This works out of the box if you're already collecting systemd logs with Promtail. Just update the push endpoint, and your logs will start showing up in Last9, searchable by labels like `_SYSTEMD_UNIT`, `CONTAINER_NAME`, or `MESSAGE`.

You don’t need to manage Grafana, Loki, or a separate storage backend; Last9 takes care of log ingestion and links it with your metrics and traces. If your team already uses Grafana, Last9 also provides an embedded Grafana instance for a familiar querying and dashboard experience.

## Wrapping Up

Working with `systemctl` and `journalctl` is part of day-to-day Linux operations. These tools help you inspect how services behave, find failures, and trace issues back to their source.

But local logs only take you so far. As systems scale, more nodes, more containers, more services, it gets harder to rely on manual inspection. You need a way to search across logs, link them to metrics and traces, and spot problems faster.

And Last9 helps you here. You can send logs using tools like Promtail, then connect that data to everything else in your stack, without maintaining a separate backend.

[Get started with us](https://app.last9.io/) for free today!

And, if you'd like to continue the conversation, feel free to join [our community on Discord](https://discord.com/invite/W8gMppQC4b). We have a dedicated channel where you can share your specific use case and connect with other developers.

## FAQs

#### What is the systemctl command used for?

The `systemctl` command is a tool for managing and controlling systemd services on Linux systems. It lets you start, stop, restart, enable, or disable services. Additionally, `systemctl` is used to view the status of services and check logs generated by systemd. System administrators need to ensure that services are running properly and that the system stays healthy.

#### How do I view logs of a specific service using systemctl?

To view logs for a specific service, you can use the `journalctl` command with the `-u` option followed by the service name. For example:

```
journalctl -u nginx

```

This command will show all the logs related to the `nginx` service, including error messages, startup times, and any issues that may arise during its operation.

#### Can I view logs in real-time using systemctl?

Yes, you can! By using the `-f` option with `journalctl`, you can view logs in real-time, just like using `tail -f`. For example:

```
journalctl -u nginx -f

```

This will stream the logs of the `nginx` service as new entries are added, allowing you to monitor the service live as it runs.

#### How do I filter systemctl logs by date?

You can filter logs by specific dates using the `--since` and `--until` options. For example:

```
journalctl --since "2024-12-01" --until "2024-12-09"

```

This command will display logs between December 1st and December 9th, 2024. You can also use relative time filters, such as `--since "1 hour ago"` or `--since "yesterday"`, for more dynamic queries.

#### What are the priority levels in systemctl logs?

`systemctl` logs are categorized by severity levels, ranging from `0` (emergency) to `7` (debug). These priority levels allow you to filter logs based on their importance. You can use the `-p` option to filter logs by priority. For example:

```
journalctl -p err

```

This will display only error messages. Other priority levels include `alert`, `critical`, `warning`, and `info`. Each level helps you focus on the most urgent logs.

#### How can I clean up old logs using systemctl?

Logs can accumulate quickly, so you need to clean them up periodically. You can remove old logs using the `--vacuum-time` or `--vacuum-size` options. For example:

```
journalctl --vacuum-time=2weeks

```

This will delete logs older than two weeks. Alternatively, you can limit the journal size to free up disk space:

```
journalctl --vacuum-size=500M

```

This will delete logs until the total journal size is below 500MB.

#### How do I monitor logs for multiple services at once?

You can monitor logs for multiple services by running separate `journalctl` commands in parallel using different terminal windows. Alternatively, you can use `grep` to filter logs from multiple services. For example:

```
journalctl -u nginx | grep "error"
journalctl -u apache2 | grep "error"

```

This will allow you to search for errors across different services in real-time, making it easier to monitor multiple services simultaneously.

#### What is the difference between systemctl and journalctl?

While `systemctl` is used to manage system services and control their states (such as starting, stopping, or enabling them), `journalctl` is a command-line tool specifically for viewing and managing logs generated by systemd services. In short, `systemctl` handles the management of services, while `journalctl` deals with logging and troubleshooting.

#### How can I forward systemctl logs to an external server?

To forward systemctl logs to an external server, you can configure systemd to send logs to a syslog server. Modify the `journald` configuration in `/etc/systemd/journald.conf` and set the `ForwardToSyslog` option to `yes`. For centralized log collection, tools like Fluentd or Logstash can be used to collect and forward logs to external systems. This is helpful for long-term log storage or compliance purposes.

#### How do I view logs for a specific boot session?

To view logs for the current boot session, use:

```
journalctl -b

```

If you want to view logs from the previous boot, use:

```
journalctl -b -1

```

This is useful for troubleshooting issues related to a specific system startup.

#### How do I filter logs by specific keywords or strings?

If you’re looking for specific events in the logs, you can pipe `journalctl` with `grep` to search for certain keywords. For example:

```
journalctl -u nginx | grep "connection failed"

```

This will only show logs where the phrase "connection failed" appears, making it easier to locate relevant log entries.

#### How can I monitor disk usage from logs?

Heavy logging can affect system performance. To monitor the disk usage of logs, use:

```
journalctl --disk-usage

```

This will show you the current disk usage of system logs. You can also use `iotop` it to monitor real-time disk writes, which helps you understand how logs are affecting your system’s resources.