Python Logging Exceptions: The Setup Guide You Actually Need

Debugging a Python app can be frustrating, especially when an unexpected crash leaves behind nothing but a vague error message. A well-configured exception log can make all the difference, turning guesswork into clear insights. Here’s how to set up logging that actually helps.

Why Standard Exception Handling Isn’t Enough

When your Python app crashes, your first instinct might be :

try:
    risky_function()
except Exception as e:
    print(f"Error: {e}")

And sure, that works… until it doesn’t. Here’s why:

Print statements vanish when your program terminates
There’s no timestamp, no traceback
Good luck finding that needle in your terminal haystack
You can’t filter by severity or type

Python logging exception handling isn’t just a nice-to-have—it’s a great way to handle failures.

💡

If you’re setting up exception logging, it helps to understand the different types of errors in Python. This guide breaks them down.

How to Set Up Basic Exception Logging in Python

Let’s start with the simplest setup:

import logging

# Configure logging once at the application startup
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    filename='app.log'
)

# Then use it anywhere in your code
try:
    1 / 0
except Exception as e:
    logging.exception("Division failed")

What makes this better than print statements?

It records timestamps automatically.
It captures the full traceback, not just the error message
The log persists after your program crashes
You can easily search through it later

💡

Notice I used logging.exception() instead of logging.error(). The difference? exception() automatically includes the traceback—a cheat code for debugging.

The Five Logging Levels You Should Know

Python’s logging module gives you five severity levels. Think of them as your alert system:

Level	When to Use	Example
DEBUG	Detailed info, typically for diagnosing problems	”Connecting to database with timeout=30s”
INFO	Confirmation that things are working	”Server started on port 8000”
WARNING	Something unexpected happened, but the app still works	”Cache miss, falling back to database”
ERROR	Something failed, but the app can continue	”Couldn’t send email notification”
CRITICAL	The app is about to crash or has a serious problem	”Out of memory, shutting down”

For exception logging, you’ll typically use ERROR or CRITICAL:

try:
    connect_to_database()
except ConnectionError as e:
    logging.error("Database connection failed", exc_info=True)
except Exception as e:
    logging.critical("Unexpected error", exc_info=True)

The exc_info=True parameter tells Python to include the traceback, just like logging.exception() does.

💡

If you’re configuring exception logging, understanding log levels can help you filter the right information. This guide explains how they work.

Why Structuring Logs Helps?

Random logs are about as useful as a screen door on a submarine. Structure is key:

import logging
import json
from datetime import datetime

# Custom JSON formatter
class JsonFormatter(logging.Formatter):
    def format(self, record):
        log_data = {
            "timestamp": datetime.utcnow().isoformat(),
            "level": record.levelname,
            "message": record.getMessage(),
            "module": record.module,
            "function": record.funcName,
            "line": record.lineno
        }

        # Add exception info if present
        if record.exc_info:
            log_data["exception"] = {
                "type": record.exc_info[0].__name__,
                "message": str(record.exc_info[1]),
                "traceback": self.formatException(record.exc_info)
            }

        return json.dumps(log_data)

# Set up the logger
handler = logging.FileHandler('app.log')
handler.setFormatter(JsonFormatter())
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
logger.addHandler(handler)

# Use it in your code
try:
    some_risky_function()
except Exception as e:
    logger.error("Operation failed", exc_info=True)

This gives you structured JSON logs that you can easily parse, filter, and analyze—much better than trying to grep through text files.

Logging Exceptions to Different Destinations

Logging to files is great, but complex applications need more. Here’s how to level up:

Rotating File Handler

Logs can grow faster than your AWS bill. Rotate them:

from logging.handlers import RotatingFileHandler

handler = RotatingFileHandler(
    'app.log',
    maxBytes=10_000_000,  # 10MB
    backupCount=5
)
logger.addHandler(handler)

Sending Critical Exceptions to Slack/Discord

Wake yourself up when things go wrong:

import requests
from logging import Handler

class SlackHandler(Handler):
    def __init__(self, webhook_url):
        super().__init__()
        self.webhook_url = webhook_url

    def emit(self, record):
        if record.levelno < logging.ERROR:
            return

        log_entry = self.format(record)
        payload = {"text": f"❌ Exception caught: {log_entry}"}
        requests.post(self.webhook_url, json=payload)

# Add this handler alongside your file handler
slack_handler = SlackHandler("https://hooks.slack.com/services/YOUR/WEBHOOK/URL")
slack_handler.setLevel(logging.ERROR)  # Only send errors and criticals
logger.addHandler(slack_handler)

Logging to a Centralized Service

For teams or multi-server apps, consider these options:

# Using Sentry
import sentry_sdk
sentry_sdk.init("https://your-key@sentry.io/12345")

# Then exceptions are automatically captured
try:
    problematic_code()
except Exception as e:
    # Sentry captures this automatically
    logging.exception("Something broke")

💡

If you’re looking to improve Python logging, this guide covers how structlog can add more structure and clarity to your logs.

Capturing More Than Just the Exception

The exception itself is only half the story. Context makes debugging 10x easier:

import logging
from contextvars import ContextVar

# Create context variables
request_id = ContextVar('request_id', default=None)
user_id = ContextVar('user_id', default=None)

# Custom filter to add context to log records
class ContextFilter(logging.Filter):
    def filter(self, record):
        record.request_id = request_id.get()
        record.user_id = user_id.get()
        return True

# Set up logger with context
logger = logging.getLogger(__name__)
logger.addFilter(ContextFilter())

# In your request handler
def process_request(request):
    request_id.set("req-123")
    user_id.set("user-456")

    try:
        # Do something that might fail
        process_payment(request.amount)
    except Exception as e:
        # This log will include request_id and user_id
        logger.exception("Payment processing failed")

Now when you see that 2 AM error, you’ll know exactly which request and which user caused it.

Common Python Logging Exception Patterns

Let’s look at some practical patterns that work well:

Catch, Log, Rethrow for Library Code

def get_user_data(user_id):
    try:
        return database.fetch_user(user_id)
    except Exception as e:
        logging.exception(f"Failed to fetch user {user_id}")
        raise  # Re-raise for the caller to handle

Catch, Log, Return Default for API Endpoints

@app.route('/api/users/<user_id>')
def user_endpoint(user_id):
    try:
        user = get_user_data(user_id)
        return jsonify(user)
    except Exception:
        logging.exception(f"API error for user {user_id}")
        return jsonify({"error": "Could not retrieve user"}), 500

Catch, Log, Retry for External Services

def send_notification(user_id, message):
    retries = 3
    for attempt in range(retries):
        try:
            return notification_service.send(user_id, message)
        except TransientError as e:
            logging.warning(f"Notification attempt {attempt+1}/{retries} failed", exc_info=True)
            time.sleep(2 ** attempt)  # Exponential backoff
        except Exception as e:
            logging.exception(f"Notification to {user_id} failed permanently")
            return False

    logging.error(f"All {retries} notification attempts to {user_id} failed")
    return False

💡

Understanding exception logging is important, but following best practices ensures your logs stay useful. This guide covers key techniques.

Debugging from Python Logs: A Practical Example

Let’s say you’ve set up logging and now you’re hunting down an issue. Here’s how you might approach it:

# Your application code
import logging

logger = logging.getLogger(__name__)

def process_order(order_id, items):
    logger.info(f"Processing order {order_id} with {len(items)} items")

    try:
        for item in items:
            try:
                check_inventory(item)
                apply_discount(item)
                add_to_cart(item, order_id)
            except OutOfStockError as e:
                logger.warning(f"Item {item['id']} out of stock", exc_info=True)
                continue  # Skip this item and continue
            except Exception as e:
                logger.error(f"Failed to process item {item['id']}", exc_info=True)
                raise  # Re-raise to abort the order

        finalize_order(order_id)
        logger.info(f"Order {order_id} completed successfully")

    except Exception as e:
        logger.exception(f"Order {order_id} failed")
        # Send notification to customer service
        notify_failed_order(order_id)
        raise

From the logs, you can trace exactly:

Which order failed
Which item caused the problem
At what step did it fail (inventory, discount, or adding to cart)
The full traceback to pinpoint the code

This kind of visibility is worth its weight in gold when you’re trying to fix production issues.

3 Common Python Logging Exception Mistakes to Avoid

These mistakes will come back to haunt you:

1. Catching Too Broadly

# BAD: Catches and logs everything the same way
try:
    do_something()
except Exception as e:
    logging.exception("Error")

# GOOD: Different handling for different exceptions
try:
    do_something()
except ValueError as e:
    logging.error("Invalid value provided", exc_info=True)
except TimeoutError as e:
    logging.warning("Operation timed out, will retry", exc_info=True)
except Exception as e:
    logging.exception("Unexpected error")

2. Logging Without Context

# BAD: "Error processing data" tells you almost nothing
try:
    process_data(data)
except Exception:
    logging.exception("Error processing data")

# GOOD: Include relevant variables
try:
    process_data(data)
except Exception:
    logging.exception(f"Error processing data. Type: {type(data)}, Length: {len(data)}, First 50 chars: {str(data)[:50]}")

3. Forgetting Log Levels

# BAD: Everything is an error
try:
    cache.get(key)
except CacheMiss:
    logging.error("Cache miss")  # This isn't really an error!

# GOOD: Use appropriate levels
try:
    cache.get(key)
except CacheMiss:
    logging.info(f"Cache miss for key {key}")  # It's just information

Performance Considerations for Python Exception Logging

Logging isn’t free. Here’s how to keep it lean:

Use Lazy Evaluation for Expensive Operations

# BAD: Always evaluates the string formatting, even if DEBUG is disabled
logging.debug("User data: " + json.dumps(large_user_data))

# GOOD: Only formats if that log level is enabled
logging.debug("User data: %s", json.dumps(large_user_data))

# EVEN BETTER: Only formats AND only serializes if needed
if logging.getLogger().isEnabledFor(logging.DEBUG):
    logging.debug("User data: %s", json.dumps(large_user_data))

Batch Logs in High-Volume Environments

For applications that generate thousands of logs per second, consider batching:

from logging.handlers import QueueHandler, QueueListener
import queue

# Set up queue
log_queue = queue.Queue()
queue_handler = QueueHandler(log_queue)

# Main logger just puts messages on queue (very fast)
logger = logging.getLogger()
logger.addHandler(queue_handler)

# Listener processes from queue in background
file_handler = logging.FileHandler("app.log")
listener = QueueListener(log_queue, file_handler)
listener.start()

# Later, when shutting down:
listener.stop()

💡

If you need to use Go functionality in a Python project, this guide walks through how gopy helps bridge the gap between the two languages.

Integrating Python Exception Logging with Monitoring Tools

Modern monitoring tools can give you alerts and dashboards based on your logs:

Prometheus + Grafana

With Prometheus and Grafana, you can expose exception metrics and build dashboards to analyze error rates. Here’s a simple setup using the prometheus_client:

from prometheus_client import Counter, start_http_server
import logging

# Create metric
exception_counter = Counter('python_exceptions_total', 'Total exceptions', ['type', 'module'])

# Custom handler to increment metrics
class PrometheusHandler(logging.Handler):
    def emit(self, record):
        if record.exc_info:
            exception_type = record.exc_info[0].__name__
            exception_counter.labels(type=exception_type, module=record.module).inc()

# Add the handler
logger = logging.getLogger()
logger.addHandler(PrometheusHandler())

# Start metrics server
start_http_server(8000)

This exposes a metrics endpoint that Prometheus can scrape. With Grafana, you can visualize exceptions by type, frequency, and affected modules, helping identify recurring issues before they escalate.

💡

If you’re setting up monitoring, this guide explains how Prometheus and Grafana work together for observability.

OpenTelemetry and Last9

This setup captures exceptions and sends traces to an OpenTelemetry-compatible backend for deeper insights.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
import logging

# Set up OpenTelemetry Tracer
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)
otlp_exporter = OTLPSpanExporter(endpoint="https://otlp.last9.io")  # Replace with your Last9 endpoint
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(otlp_exporter))

# Custom logging handler to send exceptions as traces
class OpenTelemetryHandler(logging.Handler):
    def emit(self, record):
        if record.exc_info:
            with tracer.start_as_current_span("exception_logging") as span:
                span.record_exception(record.exc_info)
                span.set_attribute("exception.type", record.exc_info[0].__name__)
                span.set_attribute("exception.message", str(record.msg))
                span.set_attribute("module", record.module)

# Configure logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.ERROR)
logger.addHandler(OpenTelemetryHandler())

# Example function with an error
def faulty_function():
    try:
        1 / 0  # Intentional error
    except Exception as e:
        logger.exception("An error occurred")

faulty_function()

Why This Matters

Traces exceptions and sends them to Last9 for analysis.
Adds context like module name and error message for better debugging.
Correlates logs and traces, helping pinpoint issues in distributed systems.

Conclusion

The time you invest in setting up proper logging will pay off tenfold when you’re debugging issues at 2 AM. Start small if you’re new to this. Even basic exception logging is better than nothing. As your application grows, expand your logging strategy to match.

💡

Got questions about Python exception logging or have your own tips? Drop by our Discord community and share your experiences. We’re all trying to avoid those late-night debugging sessions together.

FAQs

How do I log exceptions in multithreaded applications?

Python’s logging module is thread-safe, but you might want to include thread IDs:

import logging
import threading

logging.basicConfig(
    format='%(asctime)s - %(threadName)s - %(levelname)s - %(message)s',
    level=logging.INFO
)

def worker():
    try:
        # Some work
        pass
    except Exception:
        logging.exception("Worker thread failed")

threads = []
for i in range(5):
    t = threading.Thread(target=worker, name=f"Worker-{i}")
    threads.append(t)
    t.start()

Should I log in the `init` method?

Generally, no. Init methods should be fast and side-effect free. Log when something actually happens:

# Better approach
class PaymentProcessor:
    def __init__(self, api_key):
        self.api_key = api_key
        self.logger = logging.getLogger(__name__)

    def process(self, amount):
        self.logger.info(f"Processing payment of ${amount}")
        try:
            # Process payment
            pass
        except Exception:
            self.logger.exception(f"Payment of ${amount} failed")
            raise

How do I deal with sensitive data in exception logs?

Redact it with a filter:

class SensitiveFilter(logging.Filter):
    def __init__(self, patterns):
        self.patterns = patterns

    def filter(self, record):
        if isinstance(record.msg, str):
            msg = record.msg
            for pattern, replacement in self.patterns:
                msg = re.sub(pattern, replacement, msg)
            record.msg = msg
        return True

# Use it
patterns = [
    (r'card_number=\d+', 'card_number=REDACTED'),
    (r'password=\w+', 'password=REDACTED')
]
logger.addFilter(SensitiveFilter(patterns))

Python Logging Exceptions: The Setup Guide You Actually Need

Contents

Why Standard Exception Handling Isn’t Enough

How to Set Up Basic Exception Logging in Python

The Five Logging Levels You Should Know

Why Structuring Logs Helps?

Logging Exceptions to Different Destinations

Rotating File Handler

Sending Critical Exceptions to Slack/Discord

Logging to a Centralized Service

Capturing More Than Just the Exception

Common Python Logging Exception Patterns

Catch, Log, Rethrow for Library Code

Catch, Log, Return Default for API Endpoints

Catch, Log, Retry for External Services

Debugging from Python Logs: A Practical Example

3 Common Python Logging Exception Mistakes to Avoid

1. Catching Too Broadly

2. Logging Without Context

3. Forgetting Log Levels

Performance Considerations for Python Exception Logging

Use Lazy Evaluation for Expensive Operations

Batch Logs in High-Volume Environments

Integrating Python Exception Logging with Monitoring Tools

Prometheus + Grafana

OpenTelemetry and Last9

Why This Matters

Conclusion

FAQs

How do I log exceptions in multithreaded applications?

Should I log in the `init` method?

How do I deal with sensitive data in exception logs?

Contents

Start observing for free. No lock-in.

OpenTelemetry · Prometheus

Datadog · New Relic · Others

Built on Open Standards

Python Logging Exceptions: The Setup Guide You Actually Need

Contents

Why Standard Exception Handling Isn’t Enough

How to Set Up Basic Exception Logging in Python

The Five Logging Levels You Should Know

Why Structuring Logs Helps?

Logging Exceptions to Different Destinations

Rotating File Handler

Sending Critical Exceptions to Slack/Discord

Logging to a Centralized Service

Capturing More Than Just the Exception

Common Python Logging Exception Patterns

Catch, Log, Rethrow for Library Code

Catch, Log, Return Default for API Endpoints

Catch, Log, Retry for External Services

Debugging from Python Logs: A Practical Example

3 Common Python Logging Exception Mistakes to Avoid

1. Catching Too Broadly

2. Logging Without Context

3. Forgetting Log Levels

Performance Considerations for Python Exception Logging

Use Lazy Evaluation for Expensive Operations

Batch Logs in High-Volume Environments

Integrating Python Exception Logging with Monitoring Tools

Prometheus + Grafana

OpenTelemetry and Last9

Why This Matters

Conclusion

FAQs

How do I log exceptions in multithreaded applications?

Should I log in the __init__ method?

How do I deal with sensitive data in exception logs?

Contents

Start observing for free. No lock-in.

OpenTelemetry · Prometheus

Datadog · New Relic · Others

Built on Open Standards

Should I log in the `init` method?