Linux Security Logs: Complete Guide for DevOps and SysAdmins

Security logs are the quiet sentinels of your Linux systems, recording critical information that can mean the difference between detecting an intrusion and discovering a breach months too late.

For most DevOps professionals and system administrators, these logs contain valuable insights that often go untapped. While they're essential for compliance, their real value lies in providing visibility into your system's security posture and operational health.

This guide covers Linux security logs from foundational concepts to advanced implementation.

What Are Linux Security Logs?

Linux security logs are records that capture system events related to security. Think of them as your system's security cameras, constantly recording who's doing what on your servers.

These logs track login attempts, permission changes, command executions, and potential security breaches. They're your first line of defense when something goes wrong and your best tool for understanding what happened after an incident.

Different Linux distributions store logs in various locations, but /var/log is the standard directory where most of these files live.

Key Linux Security Log Files You Should Know

Linux distributions maintain several log files that are crucial for security monitoring. Here's a comprehensive rundown:

Auth.log/Secure - Track User Authentication Events

/var/log/auth.log    # Debian/Ubuntu
/var/log/secure      # RHEL/CentOS

This file records authentication events – successful and failed login attempts, user and group creation, and changes to user privileges. It's your first stop when investigating potential unauthorized access attempts, brute force attacks, and tracking sudo commands.

Messages/Syslog - System-wide Event Logs

/var/log/messages     # RHEL/CentOS
/var/log/syslog       # Debian/Ubuntu

These logs contain generic system activity and non-critical messages. They're often the first place to check when troubleshooting, as they store non-kernel boot errors, application service errors, and startup messages.

Audit.log - Detailed System Call Tracking

/var/log/audit/audit.log

The audit system records system calls and can be configured to monitor specific files, directories, or system activities. It's more granular than standard logs and is excellent for compliance requirements.

Boot.log - System Startup Information

/var/log/boot.log

Contains all bootup messages and system startup information. Check this file when investigating improper shutdowns, unexpected reboots, or boot failures.

Dmesg - Hardware and Driver Messages

/var/log/dmesg

Stores kernel ring buffer messages, particularly about hardware devices and their drivers. Critical for troubleshooting hardware issues or devices that aren't being detected properly.

Kern.log - Kernel Error Tracking

/var/log/kern.log

Contains information logged by the kernel. Useful for troubleshooting kernel-related errors, custom kernel issues, hardware problems, and connectivity troubles.

Faillog - Failed Login Attempt Records

/var/log/faillog

Records failed login attempts. Check this file to uncover attempted security breaches involving password hacking and brute-force attacks.

Cron - Scheduled Task Execution Logs

/var/log/cron

Logs information about scheduled cron jobs, including successful executions and any errors. Important for security as compromised cron jobs can be security risks.

Yum.log/Dpkg.log - Package Installation Tracking

/var/log/yum.log      # RHEL/CentOS
/var/log/dpkg.log     # Debian/Ubuntu

Tracks package installations and updates. Useful for checking when packages were installed or identifying potentially malicious software.

Journal - Centralized Systemd Logs

Modern Linux distributions using systemd store logs in the journal, which can be accessed using the journalctl command.

Application-specific Logs

Various applications maintain their logs that should be incorporated into your security monitoring:

Web Server Logs - HTTP Access and Error Tracking

/var/log/httpd/        # Apache on RHEL/CentOS
/var/log/apache2/      # Apache on Debian/Ubuntu
/var/log/nginx/        # Nginx

Web server logs track access requests, errors, and potential attack patterns like SQL injection attempts or XSS attacks.

Database Logs - Query and Authentication Records

/var/log/mysqld.log    # MySQL on RHEL/CentOS
/var/log/mysql.log     # MySQL on Debian/Ubuntu

Database logs record connections, queries, errors, and can be configured to track slow queries and lock issues.

Mail Logs - Email Delivery and Spam Detection

/var/log/maillog       # RHEL/CentOS
/var/log/mail.log      # Debian/Ubuntu

Mail server logs track email delivery, receipt, and potential spam or phishing attempts.

How to Set Up Basic Security Logging

Getting started with security logging isn't complicated. Here's how to ensure your Linux system captures the information you need:

1. Check Current Log Settings

First, see what you're already logging:

cat /etc/rsyslog.conf

2. Configure Rsyslog for Better Security

Edit /etc/rsyslog.conf to enhance security logging using rsyslog:

# Add these lines to /etc/rsyslog.conf
auth,authpriv.*                 /var/log/auth.log
*.alert                         /var/log/alert.log
kern.*                          /var/log/kern.log

3. Enable Audit Framework

The audit framework provides detailed security logging:

# Install auditd
sudo apt install auditd      # Debian/Ubuntu
sudo yum install audit       # RHEL/CentOS

# Start and enable the service
sudo systemctl enable auditd
sudo systemctl start auditd

4. Create Basic Audit Rules

Edit /etc/audit/rules.d/audit.rules to add basic security monitoring:

# Monitor user and group management
-w /etc/passwd -p wa -k user_modification
-w /etc/group -p wa -k group_modification

# Monitor network configuration changes
-w /etc/network/ -p wa -k network_modifications

# Monitor changes to sudoers
-w /etc/sudoers -p wa -k sudoers_changes

Restart the audit service to apply changes:

sudo systemctl restart auditd

Advanced Security Logging Techniques

Once you've got the basics down, it's time to level up your security logging game:

Create Custom Log Filters with Ausearch

Use ausearch to create custom filters for the audit logs:

# Search for all sudo commands
ausearch -m USER_CMD -k sudo_log

# Search for specific user actions
ausearch -ua root -ts today

Set Up Remote Logging with Rsyslog

Sending logs to a remote server is crucial – if an attacker compromises your system, they'll try to clear local logs. Rsyslog provides reliable remote logging capabilities:

# In rsyslog.conf on the client
*.* @192.168.1.100:514  # Replace with your log server IP

Configure the receiving server to accept these logs:

# In rsyslog.conf on the server
$ModLoad imudp
$UDPServerRun 514

Implement Log Rotation with Logrotate

Prevent logs from consuming all disk space with logrotate:

# Edit /etc/logrotate.d/rsyslog
/var/log/auth.log {
    rotate 7
    daily
    compress
    missingok
    notifempty
}

Combine System Logs with Application Logs

Create a unified view of security events by collecting logs from applications like web servers:

# For Apache, add to rsyslog.conf
$InputFileName /var/log/apache2/access.log
$InputFileTag apache:
$InputFileStateFile stat-apache1
$InputFileSeverity info
$InputFileFacility local6
$InputRunFileMonitor

Use Log Analysis Tools for Security Monitoring

Manually parsing logs quickly becomes impossible at scale. Here are tools that can help:

Last9: Your All-in-One Observability Solution

When you need comprehensive observability without blowing your budget, we stand out as a solid choice. Our platform brings together metrics, logs, and traces in one place, making monitoring and troubleshooting much simpler.

We work well with OpenTelemetry and Prometheus, giving you full visibility across your infrastructure. Companies like Probo, CleverTap, and Replit trust us for their observability needs — and we’ve handled monitoring for 11 of the 20 largest live-streaming events in history.

Elastic Stack (ELK)

The Elastic Stack (Elasticsearch, Logstash, Kibana) offers powerful log analysis:

# Install Filebeat to ship logs
sudo apt install filebeat
sudo filebeat modules enable system

# Configure filebeat.yml to send logs to Elasticsearch
output.elasticsearch:
  hosts: ["localhost:9200"]

Graylog

Graylog provides structured log analysis with alert capabilities:

# Configure rsyslog to send to Graylog
*.* @graylog-server:12201;GELF

OSSEC

OSSEC is a host-based intrusion detection system that excels at log analysis:

# Install OSSEC
wget https://github.com/ossec/ossec-hids/archive/3.6.0.tar.gz
tar -xzf 3.6.0.tar.gz
cd ossec-hids-3.6.0/
./install.sh

Best Practices for Linux Security Logging

Follow these guidelines to get the most from your security logs:

Use Consistent Timestamps with NTP

Configure all servers to use NTP (Network Time Protocol) for time synchronization:

sudo apt install ntp
sudo systemctl enable ntp
sudo systemctl start ntp

Define Log Retention Policies

Set clear policies for how long to keep logs:

# In logrotate.conf
rotate 90    # Keep logs for 90 days

Create Security-Specific Log Views

Use tags to categorize security events:

# In rsyslog.conf
if $msg contains 'authentication failure' then /var/log/security_auth.log

Implement Log Integrity Checks with AIDE

Use tools like AIDE (Advanced Intrusion Detection Environment) to verify log integrity:

sudo apt install aide
sudo aide --init
sudo cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db

Run regular checks:

sudo aide --check

Troubleshoot Common Security Logging Issues

Even the best logging setups can run into problems. Here's how to fix the most common issues:

Missing Logs

If logs aren't being generated:

Check permissions with ls:

ls -la /var/log
sudo chmod 755 /var/log
sudo chmod 640 /var/log/auth.log

Verify log configurations:

sudo rsyslogd -f /etc/rsyslog.conf -N1

Check service status using systemctl:

sudo systemctl status rsyslog
sudo systemctl status auditd

Log File Too Large

When log files grow too quickly:

Filter less critical events:

# In rsyslog.conf
if $programname != 'sshd' then stop

Implement more aggressive log rotation:

# In logrotate.conf
size 100M    # Rotate when log reaches 100MB

Can't Find Specific Security Events

If you're struggling to locate security events:

Combine search tools:

ausearch -i -k user_modification | grep username

Try structured searching with journalctl:

journalctl _COMM=sshd --since today

Use grep for quick searches:

grep 'authentication failure' /var/log/auth.log

Practical Security Log Analysis Examples

Let's look at some practical examples of security log analysis:

Detect Brute Force Attacks

Use grep and awk to identify potential brute force attacks:

# Find multiple failed SSH logins
grep "Failed password" /var/log/auth.log | grep "ssh" | awk '{print $11}' | sort | uniq -c | sort -nr

This command counts failed SSH login attempts by IP address.

Track Privilege Escalation

Monitor sudo command usage:

grep "COMMAND=" /var/log/auth.log

This shows all sudo commands executed on the system.

Monitor File Permission Changes

Use ausearch to track file permission modifications:

ausearch -k file_modifications -i

This displays all file permission changes tracked by the audit system.

Analyze Account Changes

Track new user account creation:

grep "new user" /var/log/auth.log
grep "add.*user" /var/log/auth.log

These commands help identify new user accounts.

Automate Security Log Monitoring

Manual checking isn't realistic for busy system administrators. Let's automate:

Create Simple Alert Scripts

Here's a basic script to alert on failed SSH attempts:

#!/bin/bash
THRESHOLD=5
COUNT=$(grep "Failed password" /var/log/auth.log | grep "ssh" | wc -l)

if [ $COUNT -gt $THRESHOLD ]; then
  echo "Alert: $COUNT failed SSH login attempts detected" | mail -s "Security Alert" admin@example.com
fi

Set Up Cron Jobs for Regular Checks

Use cron to schedule regular security checks:

# Check every 15 minutes
*/15 * * * * /path/to/security_check.sh

Integrate with Notification Systems

Send alerts to Slack, Teams, or email:

# Using curl to send to Slack webhook
curl -X POST -H 'Content-type: application/json' --data '{"text":"Security Alert: Multiple failed logins detected"}' https://hooks.slack.com/services/YOUR/WEBHOOK/URL

Conclusion

Linux security logs are your window into what's happening across your systems. When set up correctly, they transform from cryptic text files into powerful security tools.

FAQs

Where are security logs stored in Linux?

Most security logs are stored in the /var/log directory. Authentication logs are in /var/log/auth.log (Debian/Ubuntu) or /var/log/secure (RHEL/CentOS). Audit logs are typically in /var/log/audit/audit.log.

How long should I keep security logs?

For most organizations, keeping logs for 90 days is a good baseline. However, some compliance standards like PCI DSS require at least one year of logs. Your retention policy should balance security needs with storage constraints.

How can I monitor logs in real-time?

Use commands like tail -f /var/log/auth.log for real-time monitoring. For more advanced real-time analysis, tools like journalctl -f or last9 provide live monitoring capabilities.

Should I encrypt my security logs?

For sensitive environments, encrypting logs is recommended. You can use solutions like encrypted filesystems or encrypted transport when sending logs to remote servers.

How much disk space should I allocate for logs?

Plan for at least 1GB of log storage per server for a basic setup. High-traffic or security-sensitive systems may need 5-10GB or more. Implement log rotation to manage space effectively.

Can I centralize logs from multiple servers?

Yes, using a central log server is a security best practice. Tools like rsyslog, Last9, Elasticsearch, or Graylog can aggregate logs from multiple sources.