Python Logging Exceptions: The Setup Guide You Actually Need

Debugging a Python app can be frustrating, especially when an unexpected crash leaves behind nothing but a vague error message. A well-configured exception log can make all the difference, turning guesswork into clear insights. Here’s how to set up logging that actually helps.

Why Standard Exception Handling Isn't Enough

When your Python app crashes, your first instinct might be :

try:
    risky_function()
except Exception as e:
    print(f"Error: {e}")

And sure, that works... until it doesn't. Here's why:

• Print statements vanish when your program terminates
• There's no timestamp, no traceback
• Good luck finding that needle in your terminal haystack
• You can't filter by severity or type

Python logging exception handling isn't just a nice-to-have—it's a great way to handle failures.

How to Set Up Basic Exception Logging in Python

Let's start with the simplest setup:

import logging

# Configure logging once at the application startup
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    filename='app.log'
)

# Then use it anywhere in your code
try:
    1 / 0
except Exception as e:
    logging.exception("Division failed")

What makes this better than print statements?

• It records timestamps automatically.
• It captures the full traceback, not just the error message
• The log persists after your program crashes
• You can easily search through it later

The Five Logging Levels You Should Know

Python's logging module gives you five severity levels. Think of them as your alert system:

For exception logging, you'll typically use ERROR or CRITICAL:

try:
    connect_to_database()
except ConnectionError as e:
    logging.error("Database connection failed", exc_info=True)
except Exception as e:
    logging.critical("Unexpected error", exc_info=True)

The exc_info=True parameter tells Python to include the traceback, just like logging.exception() does.

Why Structuring Logs Helps?

Random logs are about as useful as a screen door on a submarine. Structure is key:

import logging
import json
from datetime import datetime

# Custom JSON formatter
class JsonFormatter(logging.Formatter):
    def format(self, record):
        log_data = {
            "timestamp": datetime.utcnow().isoformat(),
            "level": record.levelname,
            "message": record.getMessage(),
            "module": record.module,
            "function": record.funcName,
            "line": record.lineno
        }
        
        # Add exception info if present
        if record.exc_info:
            log_data["exception"] = {
                "type": record.exc_info[0].__name__,
                "message": str(record.exc_info[1]),
                "traceback": self.formatException(record.exc_info)
            }
            
        return json.dumps(log_data)

# Set up the logger
handler = logging.FileHandler('app.log')
handler.setFormatter(JsonFormatter())
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
logger.addHandler(handler)

# Use it in your code
try:
    some_risky_function()
except Exception as e:
    logger.error("Operation failed", exc_info=True)

This gives you structured JSON logs that you can easily parse, filter, and analyze—much better than trying to grep through text files.

Logging Exceptions to Different Destinations

Logging to files is great, but complex applications need more. Here's how to level up:

Rotating File Handler

Logs can grow faster than your AWS bill. Rotate them:

from logging.handlers import RotatingFileHandler

handler = RotatingFileHandler(
    'app.log',
    maxBytes=10_000_000,  # 10MB
    backupCount=5
)
logger.addHandler(handler)

Sending Critical Exceptions to Slack/Discord

Wake yourself up when things go wrong:

import requests
from logging import Handler

class SlackHandler(Handler):
    def __init__(self, webhook_url):
        super().__init__()
        self.webhook_url = webhook_url
        
    def emit(self, record):
        if record.levelno < logging.ERROR:
            return
            
        log_entry = self.format(record)
        payload = {"text": f"❌ Exception caught: {log_entry}"}
        requests.post(self.webhook_url, json=payload)
        
# Add this handler alongside your file handler
slack_handler = SlackHandler("https://hooks.slack.com/services/YOUR/WEBHOOK/URL")
slack_handler.setLevel(logging.ERROR)  # Only send errors and criticals
logger.addHandler(slack_handler)

Logging to a Centralized Service

For teams or multi-server apps, consider these options:

# Using Sentry
import sentry_sdk
sentry_sdk.init("https://your-key@sentry.io/12345")

# Then exceptions are automatically captured
try:
    problematic_code()
except Exception as e:
    # Sentry captures this automatically
    logging.exception("Something broke")

Capturing More Than Just the Exception

The exception itself is only half the story. Context makes debugging 10x easier:

import logging
from contextvars import ContextVar

# Create context variables
request_id = ContextVar('request_id', default=None)
user_id = ContextVar('user_id', default=None)

# Custom filter to add context to log records
class ContextFilter(logging.Filter):
    def filter(self, record):
        record.request_id = request_id.get()
        record.user_id = user_id.get()
        return True

# Set up logger with context
logger = logging.getLogger(__name__)
logger.addFilter(ContextFilter())

# In your request handler
def process_request(request):
    request_id.set("req-123")
    user_id.set("user-456")
    
    try:
        # Do something that might fail
        process_payment(request.amount)
    except Exception as e:
        # This log will include request_id and user_id
        logger.exception("Payment processing failed")

Now when you see that 2 AM error, you'll know exactly which request and which user caused it.

Common Python Logging Exception Patterns

Let's look at some practical patterns that work well:

Catch, Log, Rethrow for Library Code

def get_user_data(user_id):
    try:
        return database.fetch_user(user_id)
    except Exception as e:
        logging.exception(f"Failed to fetch user {user_id}")
        raise  # Re-raise for the caller to handle

Catch, Log, Return Default for API Endpoints

@app.route('/api/users/<user_id>')
def user_endpoint(user_id):
    try:
        user = get_user_data(user_id)
        return jsonify(user)
    except Exception:
        logging.exception(f"API error for user {user_id}")
        return jsonify({"error": "Could not retrieve user"}), 500

Catch, Log, Retry for External Services

def send_notification(user_id, message):
    retries = 3
    for attempt in range(retries):
        try:
            return notification_service.send(user_id, message)
        except TransientError as e:
            logging.warning(f"Notification attempt {attempt+1}/{retries} failed", exc_info=True)
            time.sleep(2 ** attempt)  # Exponential backoff
        except Exception as e:
            logging.exception(f"Notification to {user_id} failed permanently")
            return False
    
    logging.error(f"All {retries} notification attempts to {user_id} failed")
    return False

Debugging from Python Logs: A Practical Example

Let's say you've set up logging and now you're hunting down an issue. Here's how you might approach it:

# Your application code
import logging

logger = logging.getLogger(__name__)

def process_order(order_id, items):
    logger.info(f"Processing order {order_id} with {len(items)} items")
    
    try:
        for item in items:
            try:
                check_inventory(item)
                apply_discount(item)
                add_to_cart(item, order_id)
            except OutOfStockError as e:
                logger.warning(f"Item {item['id']} out of stock", exc_info=True)
                continue  # Skip this item and continue
            except Exception as e:
                logger.error(f"Failed to process item {item['id']}", exc_info=True)
                raise  # Re-raise to abort the order
                
        finalize_order(order_id)
        logger.info(f"Order {order_id} completed successfully")
        
    except Exception as e:
        logger.exception(f"Order {order_id} failed")
        # Send notification to customer service
        notify_failed_order(order_id)
        raise

From the logs, you can trace exactly:

1. Which order failed
2. Which item caused the problem
3. At what step did it fail (inventory, discount, or adding to cart)
4. The full traceback to pinpoint the code

This kind of visibility is worth its weight in gold when you're trying to fix production issues.

3 Common Python Logging Exception Mistakes to Avoid

These mistakes will come back to haunt you:

1. Catching Too Broadly

# BAD: Catches and logs everything the same way
try:
    do_something()
except Exception as e:
    logging.exception("Error")

# GOOD: Different handling for different exceptions
try:
    do_something()
except ValueError as e:
    logging.error("Invalid value provided", exc_info=True)
except TimeoutError as e:
    logging.warning("Operation timed out, will retry", exc_info=True)
except Exception as e:
    logging.exception("Unexpected error")

2. Logging Without Context

# BAD: "Error processing data" tells you almost nothing
try:
    process_data(data)
except Exception:
    logging.exception("Error processing data")

# GOOD: Include relevant variables
try:
    process_data(data)
except Exception:
    logging.exception(f"Error processing data. Type: {type(data)}, Length: {len(data)}, First 50 chars: {str(data)[:50]}")

3. Forgetting Log Levels

# BAD: Everything is an error
try:
    cache.get(key)
except CacheMiss:
    logging.error("Cache miss")  # This isn't really an error!

# GOOD: Use appropriate levels
try:
    cache.get(key)
except CacheMiss:
    logging.info(f"Cache miss for key {key}")  # It's just information

Performance Considerations for Python Exception Logging

Logging isn't free. Here's how to keep it lean:

Use Lazy Evaluation for Expensive Operations

# BAD: Always evaluates the string formatting, even if DEBUG is disabled
logging.debug("User data: " + json.dumps(large_user_data))

# GOOD: Only formats if that log level is enabled
logging.debug("User data: %s", json.dumps(large_user_data))

# EVEN BETTER: Only formats AND only serializes if needed
if logging.getLogger().isEnabledFor(logging.DEBUG):
    logging.debug("User data: %s", json.dumps(large_user_data))

Batch Logs in High-Volume Environments

For applications that generate thousands of logs per second, consider batching:

from logging.handlers import QueueHandler, QueueListener
import queue

# Set up queue
log_queue = queue.Queue()
queue_handler = QueueHandler(log_queue)

# Main logger just puts messages on queue (very fast)
logger = logging.getLogger()
logger.addHandler(queue_handler)

# Listener processes from queue in background
file_handler = logging.FileHandler("app.log")
listener = QueueListener(log_queue, file_handler)
listener.start()

# Later, when shutting down:
listener.stop()

Integrating Python Exception Logging with Monitoring Tools

Modern monitoring tools can give you alerts and dashboards based on your logs:

Prometheus + Grafana

With Prometheus and Grafana, you can expose exception metrics and build dashboards to analyze error rates. Here’s a simple setup using the prometheus_client:

from prometheus_client import Counter, start_http_server
import logging

# Create metric
exception_counter = Counter('python_exceptions_total', 'Total exceptions', ['type', 'module'])

# Custom handler to increment metrics
class PrometheusHandler(logging.Handler):
    def emit(self, record):
        if record.exc_info:
            exception_type = record.exc_info[0].__name__
            exception_counter.labels(type=exception_type, module=record.module).inc()

# Add the handler
logger = logging.getLogger()
logger.addHandler(PrometheusHandler())

# Start metrics server
start_http_server(8000)

This exposes a metrics endpoint that Prometheus can scrape. With Grafana, you can visualize exceptions by type, frequency, and affected modules, helping identify recurring issues before they escalate.

OpenTelemetry and Last9

This setup captures exceptions and sends traces to an OpenTelemetry-compatible backend for deeper insights.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
import logging

# Set up OpenTelemetry Tracer
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)
otlp_exporter = OTLPSpanExporter(endpoint="https://otlp.last9.io")  # Replace with your Last9 endpoint
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(otlp_exporter))

# Custom logging handler to send exceptions as traces
class OpenTelemetryHandler(logging.Handler):
    def emit(self, record):
        if record.exc_info:
            with tracer.start_as_current_span("exception_logging") as span:
                span.record_exception(record.exc_info)
                span.set_attribute("exception.type", record.exc_info[0].__name__)
                span.set_attribute("exception.message", str(record.msg))
                span.set_attribute("module", record.module)

# Configure logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.ERROR)
logger.addHandler(OpenTelemetryHandler())

# Example function with an error
def faulty_function():
    try:
        1 / 0  # Intentional error
    except Exception as e:
        logger.exception("An error occurred")

faulty_function()

Why This Matters

• Traces exceptions and sends them to Last9 for analysis.
• Adds context like module name and error message for better debugging.
• Correlates logs and traces, helping pinpoint issues in distributed systems.

Conclusion

The time you invest in setting up proper logging will pay off tenfold when you're debugging issues at 2 AM. Start small if you're new to this. Even basic exception logging is better than nothing. As your application grows, expand your logging strategy to match.

FAQs

How do I log exceptions in multithreaded applications?

Python's logging module is thread-safe, but you might want to include thread IDs:

import logging
import threading

logging.basicConfig(
    format='%(asctime)s - %(threadName)s - %(levelname)s - %(message)s',
    level=logging.INFO
)

def worker():
    try:
        # Some work
        pass
    except Exception:
        logging.exception("Worker thread failed")

threads = []
for i in range(5):
    t = threading.Thread(target=worker, name=f"Worker-{i}")
    threads.append(t)
    t.start()

Should I log in the __init__ method?

Generally, no. Init methods should be fast and side-effect free. Log when something actually happens:

# Better approach
class PaymentProcessor:
    def __init__(self, api_key):
        self.api_key = api_key
        self.logger = logging.getLogger(__name__)
        
    def process(self, amount):
        self.logger.info(f"Processing payment of ${amount}")
        try:
            # Process payment
            pass
        except Exception:
            self.logger.exception(f"Payment of ${amount} failed")
            raise

How do I deal with sensitive data in exception logs?

Redact it with a filter:

class SensitiveFilter(logging.Filter):
    def __init__(self, patterns):
        self.patterns = patterns
        
    def filter(self, record):
        if isinstance(record.msg, str):
            msg = record.msg
            for pattern, replacement in self.patterns:
                msg = re.sub(pattern, replacement, msg)
            record.msg = msg
        return True

# Use it
patterns = [
    (r'card_number=\d+', 'card_number=REDACTED'),
    (r'password=\w+', 'password=REDACTED')
]
logger.addFilter(SensitiveFilter(patterns))