# Linux Security Logs: Complete Guide for DevOps and SysAdmins

> A practical guide to understanding, finding, and using Linux security logs — built for DevOps, SysAdmins, and anyone managing production systems.

Source: https://last9.io/blog/linux-security-logs/

Security logs are the quiet sentinels of your Linux systems, recording critical information that can mean the difference between detecting an intrusion and discovering a breach months too late.

For most DevOps professionals and system administrators, these logs contain valuable insights that often go untapped. While they're essential for compliance, their real value lies in providing visibility into your system's security posture and operational health.

This guide covers Linux security logs from foundational concepts to advanced implementation.

## What Are Linux Security Logs?

Linux security logs are records that capture system events related to security. Think of them as your system's security cameras, constantly recording who's doing what on your servers.

These logs track login attempts, permission changes, command executions, and potential security breaches. They're your first line of defense when something goes wrong and your best tool for understanding what happened after an incident.

Different Linux distributions store logs in various locations, but `/var/log` is the standard directory where most of these files live.

If you're new to Linux logging, understanding what lives inside [`/var/log`](https://last9.io/blog/what-is-var-log/) is a good place to start.

## Key Linux Security Log Files You Should Know

Linux distributions maintain several log files that are crucial for security monitoring. Here's a comprehensive rundown:

### Auth.log/Secure - Track User Authentication Events

```
/var/log/auth.log    # Debian/Ubuntu
/var/log/secure      # RHEL/CentOS
```

This file records authentication events – successful and failed login attempts, user and group creation, and changes to user privileges. It's your first stop when investigating potential unauthorized access attempts, brute force attacks, and tracking sudo commands. To proactively safeguard production systems against unauthorized access from leaked corporate credentials, DevOps teams should supplement local access logs with automated data breach monitoring to detect exposed infrastructure assets on the dark web.

### Messages/Syslog - System-wide Event Logs

```
/var/log/messages     # RHEL/CentOS
/var/log/syslog       # Debian/Ubuntu
```

These logs contain generic system activity and non-critical messages. They're often the first place to check when troubleshooting, as they store non-kernel boot errors, application service errors, and startup messages.

### Audit.log - Detailed System Call Tracking

```
/var/log/audit/audit.log
```

The audit system records system calls and can be configured to monitor specific files, directories, or system activities. It's more granular than standard logs and is excellent for compliance requirements.

### Boot.log - System Startup Information

```
/var/log/boot.log
```

Contains all bootup messages and system startup information. Check this file when investigating improper shutdowns, unexpected reboots, or boot failures.

### Dmesg - Hardware and Driver Messages

```
/var/log/dmesg
```

Stores kernel ring buffer messages, particularly about hardware devices and their drivers. Critical for troubleshooting hardware issues or devices that aren't being detected properly.

### Kern.log - Kernel Error Tracking

```
/var/log/kern.log
```

Contains information logged by the kernel. Useful for troubleshooting kernel-related errors, custom kernel issues, hardware problems, and connectivity troubles.

### Faillog - Failed Login Attempt Records

```
/var/log/faillog
```

Records failed login attempts. Check this file to uncover attempted security breaches involving password hacking and brute-force attacks.

### Cron - Scheduled Task Execution Logs

```
/var/log/cron
```

Logs information about scheduled cron jobs, including successful executions and any errors. Important for security as compromised cron jobs can be security risks.

To track scheduled tasks and catch failures early, it helps to know where to find and read [crontab logs](https://last9.io/blog/crontab-logs/).

### Yum.log/Dpkg.log - Package Installation Tracking

```
/var/log/yum.log      # RHEL/CentOS
/var/log/dpkg.log     # Debian/Ubuntu
```

Tracks package installations and updates. Useful for checking when packages were installed or identifying potentially malicious software.

### Journal - Centralized Systemd Logs

Modern Linux distributions using systemd store logs in the journal, which can be accessed using the `journalctl` command.

### Application-specific Logs

Various applications maintain their logs that should be incorporated into your security monitoring:

#### Web Server Logs - HTTP Access and Error Tracking

```
/var/log/httpd/        # Apache on RHEL/CentOS
/var/log/apache2/      # Apache on Debian/Ubuntu
/var/log/nginx/        # Nginx
```

Web server logs track access requests, errors, and potential attack patterns like SQL injection attempts or XSS attacks.

#### Database Logs - Query and Authentication Records

```
/var/log/mysqld.log    # MySQL on RHEL/CentOS
/var/log/mysql.log     # MySQL on Debian/Ubuntu
```

Database logs record connections, queries, errors, and can be configured to track slow queries and lock issues.

#### Mail Logs - Email Delivery and Spam Detection

```
/var/log/maillog       # RHEL/CentOS
/var/log/mail.log      # Debian/Ubuntu
```

Mail server logs track email delivery, receipt, and potential spam or phishing attempts.

## How to Set Up Basic Security Logging

Getting started with security logging isn't complicated. Here's how to ensure your Linux system captures the information you need:

### 1\. Check Current Log Settings

First, see what you're already logging:

```bash
cat /etc/rsyslog.conf
```

### 2\. Configure Rsyslog for Better Security

Edit `/etc/rsyslog.conf` to enhance security logging using [rsyslog](https://www.rsyslog.com/):

```bash
# Add these lines to /etc/rsyslog.conf
auth,authpriv.*                 /var/log/auth.log
*.alert                         /var/log/alert.log
kern.*                          /var/log/kern.log
```

### 3\. Enable Audit Framework

The [audit framework](https://github.com/linux-audit/audit-userspace) provides detailed security logging:

```bash
# Install auditd
sudo apt install auditd      # Debian/Ubuntu
sudo yum install audit       # RHEL/CentOS

# Start and enable the service
sudo systemctl enable auditd
sudo systemctl start auditd
```

### 4\. Create Basic Audit Rules

Edit `/etc/audit/rules.d/audit.rules` to add basic security monitoring:

```bash
# Monitor user and group management
-w /etc/passwd -p wa -k user_modification
-w /etc/group -p wa -k group_modification

# Monitor network configuration changes
-w /etc/network/ -p wa -k network_modifications

# Monitor changes to sudoers
-w /etc/sudoers -p wa -k sudoers_changes
```

Restart the audit service to apply changes:

```bash
sudo systemctl restart auditd
```

If your app feels slow but CPU and memory look fine, it's worth checking [disk I/O on Linux](https://last9.io/blog/monitoring-disk-io-on-linux/) — it’s often the silent culprit.

## Advanced Security Logging Techniques

Once you've got the basics down, it's time to level up your security logging game:

### Create Custom Log Filters with Ausearch

Use [ausearch](https://man7.org/linux/man-pages/man8/ausearch.8.html) to create custom filters for the audit logs:

```bash
# Search for all sudo commands
ausearch -m USER_CMD -k sudo_log

# Search for specific user actions
ausearch -ua root -ts today
```

### Set Up Remote Logging with Rsyslog

Sending logs to a remote server is crucial – if an attacker compromises your system, they'll try to clear local logs. [Rsyslog](https://www.rsyslog.com/) provides reliable remote logging capabilities:

```bash
# In rsyslog.conf on the client
*.* @192.168.1.100:514  # Replace with your log server IP
```

Configure the receiving server to accept these logs:

```bash
# In rsyslog.conf on the server
$ModLoad imudp
$UDPServerRun 514
```

### Implement Log Rotation with Logrotate

Prevent logs from consuming all disk space with [logrotate](https://linux.die.net/man/8/logrotate):

```bash
# Edit /etc/logrotate.d/rsyslog
/var/log/auth.log {
    rotate 7
    daily
    compress
    missingok
    notifempty
}
```

### Combine System Logs with Application Logs

Create a unified view of security events by collecting logs from applications like web servers:

```bash
# For Apache, add to rsyslog.conf
$InputFileName /var/log/apache2/access.log
$InputFileTag apache:
$InputFileStateFile stat-apache1
$InputFileSeverity info
$InputFileFacility local6
$InputRunFileMonitor
```

## Use Log Analysis Tools for Security Monitoring

Manually parsing logs quickly becomes impossible at scale. Here are tools that can help:

### [Last9](https://last9.io/): Your All-in-One Observability Solution

When you need comprehensive observability without blowing your budget, we stand out as a solid choice. Our platform brings together metrics, logs, and traces in one place, making monitoring and troubleshooting much simpler.

We work well with OpenTelemetry and Prometheus, giving you full visibility across your infrastructure. Companies like Probo, CleverTap, and Replit trust us for their observability needs — and we’ve handled monitoring for 11 of the 20 largest live-streaming events in history.

### Elastic Stack (ELK)

The [Elastic Stack](https://last9.io/blog/elk-vs-grafana-vs-prometheus/) (Elasticsearch, Logstash, Kibana) offers powerful log analysis:

```bash
# Install Filebeat to ship logs
sudo apt install filebeat
sudo filebeat modules enable system

# Configure filebeat.yml to send logs to Elasticsearch
output.elasticsearch:
  hosts: ["localhost:9200"]
```

### Graylog

[Graylog](https://www.graylog.org/) provides structured log analysis with alert capabilities:

```bash
# Configure rsyslog to send to Graylog
*.* @graylog-server:12201;GELF
```

### OSSEC

[OSSEC](https://www.ossec.net/) is a host-based intrusion detection system that excels at log analysis:

```bash
# Install OSSEC
wget https://github.com/ossec/ossec-hids/archive/3.6.0.tar.gz
tar -xzf 3.6.0.tar.gz
cd ossec-hids-3.6.0/
./install.sh
```

## Best Practices for Linux Security Logging

Follow these guidelines to get the most from your security logs:

### Use Consistent Timestamps with NTP

Configure all servers to use [NTP (Network Time Protocol)](https://www.ntppool.org/) for time synchronization:

```bash
sudo apt install ntp
sudo systemctl enable ntp
sudo systemctl start ntp
```

### Define Log Retention Policies

Set clear policies for how long to keep logs:

```bash
# In logrotate.conf
rotate 90    # Keep logs for 90 days
```

### Create Security-Specific Log Views

Use tags to categorize security events:

```bash
# In rsyslog.conf
if $msg contains 'authentication failure' then /var/log/security_auth.log
```

### Implement Log Integrity Checks with AIDE

Use tools like [AIDE (Advanced Intrusion Detection Environment)](https://github.com/aide/aide) to verify log integrity:

```bash
sudo apt install aide
sudo aide --init
sudo cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db
```

Run regular checks:

```bash
sudo aide --check
```

When something breaks and you’re not sure where to start, [Linux event logs](https://last9.io/blog/linux-event-logs-your-troubleshooting-guide/) can point you in the right direction.

## Troubleshoot Common Security Logging Issues

Even the best logging setups can run into problems. Here's how to fix the most common issues:

### Missing Logs

If logs aren't being generated:

Check permissions with [ls](https://www.gnu.org/software/coreutils/ls):

```bash
ls -la /var/log
sudo chmod 755 /var/log
sudo chmod 640 /var/log/auth.log
```

Verify log configurations:

```bash
sudo rsyslogd -f /etc/rsyslog.conf -N1
```

Check service status using [systemctl](https://www.freedesktop.org/software/systemd/man/systemctl.html):

```bash
sudo systemctl status rsyslog
sudo systemctl status auditd
```

### Log File Too Large

When log files grow too quickly:

Filter less critical events:

```bash
# In rsyslog.conf
if $programname != 'sshd' then stop
```

Implement more aggressive [log rotation](https://linux.die.net/man/8/logrotate):

```bash
# In logrotate.conf
size 100M    # Rotate when log reaches 100MB
```

### Can't Find Specific Security Events

If you're struggling to locate security events:

Combine search tools:

```bash
ausearch -i -k user_modification | grep username
```

Try structured searching with [journalctl](https://www.freedesktop.org/software/systemd/man/journalctl.html):

```bash
journalctl _COMM=sshd --since today
```

Use [grep](https://www.gnu.org/software/grep/) for quick searches:

```bash
grep 'authentication failure' /var/log/auth.log
```

## Practical Security Log Analysis Examples

Let's look at some practical examples of security log analysis:

### Detect Brute Force Attacks

Use [grep](https://www.gnu.org/software/grep/) and [awk](https://www.gnu.org/software/gawk/) to identify potential brute force attacks:

```bash
# Find multiple failed SSH logins
grep "Failed password" /var/log/auth.log | grep "ssh" | awk '{print $11}' | sort | uniq -c | sort -nr
```

This command counts failed SSH login attempts by IP address. Once a suspicious address surfaces in that count, tracing that IP address across your other log sources, web server access logs, audit trails, mail logs, lets you piece together the attacker's full footprint instead of stopping at the SSH layer.

### Track Privilege Escalation

Monitor sudo command usage:

```bash
grep "COMMAND=" /var/log/auth.log
```

This shows all sudo commands executed on the system.

### Monitor File Permission Changes

Use [ausearch](https://linux.die.net/man/8/ausearch) to track file permission modifications:

```bash
ausearch -k file_modifications -i
```

This displays all file permission changes tracked by the audit system.

### Analyze Account Changes

Track new user account creation:

```bash
grep "new user" /var/log/auth.log
grep "add.*user" /var/log/auth.log
```

These commands help identify new user accounts.

## Automate Security Log Monitoring

Manual checking isn't realistic for busy system administrators. Let's automate:

### Create Simple Alert Scripts

Here's a basic script to alert on failed SSH attempts:

```bash
#!/bin/bash
THRESHOLD=5
COUNT=$(grep "Failed password" /var/log/auth.log | grep "ssh" | wc -l)

if [ $COUNT -gt $THRESHOLD ]; then
  echo "Alert: $COUNT failed SSH login attempts detected" | mail -s "Security Alert" admin@example.com
fi
```

### Set Up Cron Jobs for Regular Checks

Use [cron](https://man7.org/linux/man-pages/man5/crontab.5.html) to schedule regular security checks:

```bash
# Check every 15 minutes
*/15 * * * * /path/to/security_check.sh
```

### Integrate with Notification Systems

Send alerts to [Slack](https://api.slack.com/messaging/webhooks), Teams, or email:

```bash
# Using curl to send to Slack webhook
curl -X POST -H 'Content-type: application/json' --data '{"text":"Security Alert: Multiple failed logins detected"}' https://hooks.slack.com/services/YOUR/WEBHOOK/URL
```

## Conclusion

Linux security logs are your window into what's happening across your systems. When set up correctly, they transform from cryptic text files into powerful security tools.

What security logging challenges are you facing in your environment? Join our [Discord Community](https://discord.com/invite/W8gMppQC4b) to discuss with fellow engineers!

## FAQs

### Where are security logs stored in Linux?

Most security logs are stored in the `/var/log` directory. Authentication logs are in `/var/log/auth.log` (Debian/Ubuntu) or `/var/log/secure` (RHEL/CentOS). Audit logs are typically in `/var/log/audit/audit.log`.

### How long should I keep security logs?

For most organizations, keeping logs for 90 days is a good baseline. However, some compliance standards like PCI DSS require at least one year of logs. Your retention policy should balance security needs with storage constraints.

### How can I monitor logs in real-time?

Use commands like `tail -f /var/log/auth.log` for real-time monitoring. For more advanced real-time analysis, tools like `journalctl -f` or `last9` provide live monitoring capabilities.

### Should I encrypt my security logs?

For sensitive environments, encrypting logs is recommended. You can use solutions like encrypted filesystems or encrypted transport when sending logs to remote servers.

### How much disk space should I allocate for logs?

Plan for at least 1GB of log storage per server for a basic setup. High-traffic or security-sensitive systems may need 5-10GB or more. Implement log rotation to manage space effectively.

### Can I centralize logs from multiple servers?

Yes, using a central log server is a security best practice. Tools like rsyslog, Last9, Elasticsearch, or Graylog can aggregate logs from multiple sources.