Implementing Python Logging for Better Application Insights

Farouk Ben. - Founder at OdownFarouk Ben.()
Apr 29, 2025
Implementing Python Logging for Better Application Insights - Odown - uptime monitoring and status page

Python's logging system is one of those unsung heroes in development—reliable, always there when you need it, but often overlooked until things go sideways in production. I've been using Python's logging module for years, and it's saved my bacon more times than I can count. Let's dive into what makes Python logging so powerful and how you can use it effectively in your projects.

Table of Contents

Understanding Python's Logging System

Python's built-in logging module provides a flexible framework for emitting log messages from Python programs. It's part of the standard library, so there's nothing extra to install—just import and go.

The logging system is based on a hierarchical model with loggers, handlers, filters, and formatters. This might sound complicated at first, but it's this architecture that makes the system so flexible.

import logging

# This gets the root logger
logger = logging.getLogger()

But why use logging instead of just sprinkling print() statements throughout your code? Well, for starters:

  1. Logs can be directed to different outputs (console, files, network)
  2. Log messages can have different severity levels
  3. Logging can be configured and controlled centrally
  4. Log output can be formatted consistently
  5. Logging can be turned on/off without changing code

I once had a colleague who used print() statements for debugging a complex data processing pipeline. When we moved to production, they had to remove all those statements manually. Not fun! Proper logging would have saved hours of tedious work.

Setting Up Basic Logging

Getting started with logging is surprisingly simple. Here's the most basic setup:

2025-03-15 14:32:11,123 - __main__ - INFO - Application started
2025-03-15 14:32:11,124 - __main__ - WARNING - This is a warning message
2025-03-15 14:32:11,124 - __main__ - ERROR - This is an error message

When you run this code, you'll see output like:

2025-03-15 14:32:11,123 - __main__ - INFO - Application started
2025-03-15 14:32:11,124 - __main__ - WARNING - This is a warning message
2025-03-15 14:32:11,124 - __main__ - ERROR - This is an error message

The __name__ variable is a special Python variable that contains the name of the current module. Using it as the logger name helps you identify which module generated the log message, which becomes super helpful in larger applications.

Log Levels and When to Use Them

Python's logging module provides five standard levels, indicating the severity of events. From lowest to highest severity:

Level Numeric Value When to Use
DEBUG 10 Detailed information, typically of interest only when diagnosing problems
INFO 20 Confirmation that things are working as expected
WARNING 30 An indication that something unexpected happened, or may happen in the near future
ERROR 40 Due to a more serious problem, the software has not been able to perform some function
CRITICAL 50 A serious error indicating that the program itself may be unable to continue running

Choosing the right log level for your messages is more art than science, but here's how I think about them:

  • DEBUG: Use for information that is diagnostically helpful to people more than just developers (IT, sysadmins, etc.)
  • INFO: Use for information that's useful to application users and system administrators looking at logs to know what the application is doing
  • WARNING: Use when something unexpected happened, but the application will continue working as expected
  • ERROR: Use when the application has encountered an error that prevents some function from working but doesn't stop the entire application
  • CRITICAL: Use when the application has encountered a serious error that threatens its ability to continue running

Here's a practical example:

import logging

logging.basicConfig (level= logging.DEBUG)
logger = logging. getLogger(__name__)

def divide(x, y):
logger.debug (f"Dividing {x} by {y}")
try:
result = x / y
except ZeroDivisionError:
logger.error ("Division by zero!")
return None
else:
logger.info (f"Division result: {result}")
return result

divide(10, 2) # Should work fine
divide(10, 0) # Should trigger an error

Configuring Logging Formats

The format of your log messages matters a lot. A good format makes it easy to parse logs both visually and programmatically. The default format string is pretty basic, but you can customize it to include any information you need.

Here's a more detailed format that I often use:

import logging
from datetime import datetime

logging. basicConfig(
level= logging.INFO,
format= '%(asctime)s [%(levelname)s] %(name)s (%(filename)s: %(lineno)d) - %(message)s',
datefmt='%Y-%m-%d %H:%M:%S'
)

logger = logging.getLogger (__name__)
logger.info ("This is a detailed log message")

This will produce output like:

2025-03-15 14:35:23 [INFO] __main__ (example.py:10) - This is a detailed log message

The format string can include various attributes:

  • %(name)s: The name of the logger
  • %(levelname)s: The level of the log message
  • %(message)s: The log message itself
  • %(asctime)s: Human-readable time when the log message was created
  • %(filename)s: Filename portion of pathname
  • %(lineno)d: Line number in the source file
  • %(funcName)s: Function name
  • %(process)d: Process ID
  • %(thread)d: Thread ID

I've found that including the filename and line number is particularly helpful when debugging complex applications. It lets you quickly jump to the exact location where the log message was generated.

Working with Multiple Loggers

In larger applications, you'll often want to use multiple loggers for different components. This helps keep your logs organized and allows you to configure different logging behaviors for different parts of your application.

Loggers are organized in a hierarchy based on their names, with dots as separators. For example, a logger named "app.database" is a child of the logger named "app".

import logging

# Configure the root logger
logging.basicConfig (level= logging.WARNING)

# Create loggers for different components
logger_main = logging.getLogger ("app")
logger_main.setLevel (logging.INFO)

logger_db = logging.getLogger ("app.database")
logger_db.setLevel (logging.DEBUG)

# Create a separate logger for a different component
logger_api = logging.getLogger ("api")
logger_ api.setLevel (logging.WARNING)

# Use the loggers
logger_main.info("Main application info") # Will be shown (INFO >= INFO)
logger_db.debug("Database debug info") # Will be shown (DEBUG >= DEBUG)
logger_api.info("API info") # Won't be shown (INFO < WARNING)

Child loggers inherit settings from their parents unless configured otherwise. This hierarchical structure gives you a lot of flexibility in how you organize your logging.

Logging to Files

Logging to the console is great during development, but in production, you'll usually want to log to files. This is where handlers come in. A handler is responsible for sending log records to the appropriate destination.

Here's how to set up file logging:

import logging

# Create a logger
logger = logging.getLogger (__name__)
logger.setLevel (logging.INFO)

# Create a file handler
file_handler = logging.FileHandler('app.log')
file_handler.setLevel (logging.INFO)

# Create a formatter and add it to the handler
formatter = logging.Formatter ('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
file_handler. setFormatter (formatter)

# Add the handler to the logger
logger.addHandler (file_handler)

# Use the logger
logger.info ("This message will go to the log file")

You can also use multiple handlers to send logs to different destinations. For example, you might want to log warnings and errors to the console while logging all messages to a file:

import logging

logger = logging.getLogger (__name__)
logger.setLevel (logging.DEBUG)

# File handler for all logs
file_handler = logging.FileHandler('app.log')
file_handler.setLevel (logging.DEBUG)
file_handler.setFormatter (logging.Formatter ('%(asctime)s - %(levelname)s - %(message)s'))

# Console handler for warnings and errors
console_handler = logging.StreamHandler()
console_ handler.setLevel (logging.WARNING)
console_ handler.setFormatter (logging.Formatter ('%(levelname)s - %(message)s'))

# Add both handlers to the logger
logger.addHandler (file_handler)
logger.addHandler (console_handler)

# Use the logger
logger.debug("This goes only to the file")
logger.warning("This goes to both file and console")

Rotating Log Files

If your application runs for a long time, log files can grow very large. This is where log rotation comes in handy. The RotatingFileHandler lets you limit the size of log files and keep a certain number of backup files.

import logging
from logging.handlers import RotatingFileHandler

logger = logging.getLogger (__name__)
logger.setLevel (logging.INFO)

# Create a rotating file handler
# Max size 1MB, keep 5 backup files
handler = RotatingFile Handler(
'app.log',
maxBytes=1_000_000,
backupCount=5
)
handler.setFormatter (logging.Formatter(' %(asctime)s - %(levelname)s - %(message)s'))
logger.addHandler (handler)

# Log a bunch of messages
for i in range(10000):
logger.info (f"This is log message {i}")

This will create files named app.log, app.log.1, app.log.2, etc., as the logs rotate.

There's also the TimedRotatingFileHandler which rotates logs based on time rather than size:

from logging.handlers import TimedRotating FileHandler

# Rotate logs at midnight every day, keep 7 days of logs
handler = TimedRotating FileHandler(
'app.log',
when='midnight',
interval=1,
backupCount=7
)

I had a service once that was generating gigabytes of logs per day. Switching to rotating logs saved us from constantly running out of disk space and made log management much easier.

Logging in Different Environments

Your logging needs will vary depending on whether you're in development, testing, or production. A common pattern is to configure logging differently based on the environment:

import logging
import os

# Get the environment from an environment variable
env = os.environ.get(' ENVIRONMENT', 'development')

# Configure logging based on environment
if env == 'development':
logging.basicConfig (
level= logging.DEBUG,
format='%(asctime)s - %(levelname)s - %(message)s'
)
elif env == 'testing':
logging.basicConfig (
level= logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s',
filename='test.log'
)
elif env == 'production':
logging.basicConfig (
level= logging.WARNING,
format='%(asctime)s - %(levelname)s - %(name)s - %(message)s',
filename=' production.log'
)
else:
raise ValueError(f"Unknown environment: {env}")

logger = logging.getLogger (__name__)

You can also use configuration files to manage logging settings, which makes it easier to change settings without modifying code. Python's logging module supports configuration via dictionaries, files, or programmatic configuration.

Structured Logging with JSON

Traditional text logs are fine for human reading, but if you're using log aggregation tools like ELK (Elasticsearch, Logstash, Kibana) or Splunk, structured logging formats like JSON can be more useful.

While Python's built-in logging doesn't support JSON directly, you can use formatters to achieve this:

import json
import logging
import datetime

class JsonFormatter (logging.Formatter):
"""JSON log formatter."""

def format(self, record):
log_record = {
"timestamp": datetime.datetime. fromtimestamp (record.created) .isoformat(),
"level": record.levelname,
"name": record.name,
"message": record.getMessage(),
}

if hasattr(record, 'props'):
log_record['props'] = record.props

if record.exc_info:
log_record['exception'] = self.formatException (record.exc_info)

return json.dumps(log_record)

# Set up logger with JSON formatter
logger = logging.getLogger (__name__)
handler = logging. StreamHandler()
handler.setFormatter (JsonFormatter())
logger.addHandler (handler)
logger.setLevel (logging.INFO)

# Log with extra properties
logger.info ("User logged in", extra={"props": {"user_id": 123, "ip": "192.168.1.1"}})

This will produce a JSON log entry like:

{"timestamp": "2025-03-15T14: 42:11.123456", "level": "INFO", "name": "__main__", "message": "User logged in", "props": {
"user_id": 123, "ip": "192.168.1.1"
}
}

For serious JSON logging, consider using a dedicated library like python-json-logger which handles many edge cases and provides more features.

Common Logging Patterns and Best Practices

After working with logging for years, I've developed some patterns and practices that have served me well:

  1. Log at the right level: Be thoughtful about which level you log at. Too many DEBUG logs can obscure important information, while too few might leave you without enough context.

  2. Include context: Log messages should be self-contained and include relevant context.

# Bad
logger.info ("Operation completed")

# Good
logger.info ("User profile update completed for user_id=123")

  1. Log in the right place: Log close to where the action is happening, but not so frequently that you create noise.

  2. Use structured logging: Include key-value pairs in your logs to make them more searchable and filterable.

logger.info ("Payment processed", extra={"props": {"amount": 123.45, "payment_id": "pay_abc123"}})
  1. Log exceptions properly: Include the full traceback when logging exceptions.
try:
do_something_risky()
except Exception:
logger.exception ("Failed to do something risky") # Includes traceback
  1. Be consistent with log levels: Define what each level means for your application and stick to it.

One especially useful pattern is using a context manager for logging the start and end of operations:

import logging
import time
from contextlib import contextmanager

logger = logging.getLogger (__name__)

@contextmanager
def log_operation (operation_name):
"""Log the start and end of an operation, including its duration."""
logger.info (f"Starting {operation_name}")
start_time = time.time()
try:
yield
except Exception as e:
logger.exception (f"Error during {operation_name}: {e}")
raise
finally:
duration = time.time() - start_time
logger.info (f"Completed {operation_name} in {duration:.2f} seconds")

# Usage
with log_operation ("data processing"):
# Do some work here
process_data()

This pattern helps you track how long operations take and ensures that completion or failure is always logged, even if an exception occurs.

Debugging with Logs

Logs are one of the most powerful debugging tools at your disposal. When debugging with logs, consider:

  1. Temporarily increasing log levels: For stubborn bugs, temporarily increase the log level in the problematic area.
# Increase log level just for this module
logger = logging.getLogger (" app.problematic_module")
logger.setLevel (logging.DEBUG)
  1. Adding context to logs: Include variable values and state information in your debug logs.
logger.debug(f"Processing request with params: {params}")

  1. Using log files for post-mortem analysis: Logs let you reconstruct what happened after a crash.

  2. Creating a logging test harness: For complex issues, create a test case that reproduces the issue with detailed logging.

I once spent days tracking down an intermittent bug in a distributed system. The key insight came when I noticed in the logs that two servers were processing the same message simultaneously. Without good logging, I might never have found that race condition.

Logging in Production

Production logging has special considerations:

  1. Security: Be careful not to log sensitive information like passwords, tokens, or personal data.
# Bad
logger.info (f"User login attempt with password: {password}")

# Good
logger.info (f" User login attempt for username: {username}")

  1. Performance: Excessive logging can impact performance. Use appropriate log levels and consider async logging for high-volume applications.

  2. Storage: Plan for log storage and retention. How long will you keep logs? How will they be archived?

  3. Log aggregation: In distributed systems, use a centralized logging system to collect and analyze logs from all services.

  4. Alerting: Set up alerts for critical log messages to notify the team when something goes wrong.

A common pattern for production logging is to use a log aggregation service like ELK (Elasticsearch, Logstash, Kibana), Graylog, or a cloud service like AWS CloudWatch Logs. These services allow you to search, filter, and analyze logs from multiple sources in one place.

Third-Party Logging Libraries

While Python's built-in logging module is powerful, several third-party libraries can enhance your logging setup:

  1. loguru: A library that aims to bring enjoyable logging in Python with a simpler API.
from loguru import logger

logger.debug("This is a debug message")
  1. structlog: Structured logging made easy, with support for context-based logging.
import structlog

log = structlog. get_logger()
log.info("user logged in", user_id=123)
  1. python-json-logger: Provides a JSON formatter for the standard logging library.
from pythonjsonlogger import jsonlogger

handler = logging.StreamHandler()
handler.setFormatter (jsonlogger.JsonFormatter())
  1. sentry-sdk: Provides error tracking that helps developers monitor and fix crashes in real time.
import sentry_sdk

sentry_sdk.init("your-sentry-dsn")

try:
1 / 0
except Exception as e:
sentry_sdk.capture_ exception(e)

Each of these libraries has its strengths and might be more suitable for certain use cases than the standard library.

Common Pitfalls to Avoid

Even with a good logging setup, there are common mistakes to watch out for:

  1. Not configuring the root logger: If you only configure specific loggers, messages from other parts of your application or third-party libraries might not be captured.

  2. Logging sensitive information: Be careful not to log passwords, API keys, or personal data.

  3. Log flooding: Logging too much can obscure important information and impact performance. Use appropriate log levels.

  4. Poor log message formatting: Inconsistent or unclear log messages make debugging harder.

  5. Not handling logging exceptions: Logging itself can fail. Make sure your logging setup is robust.

  6. Using print instead of logging: Mixing logging and print statements makes log management harder.

  7. Hardcoding log configuration: Configuration should be flexible and environment-dependent.

One particular mistake I see often is forgetting that loggers propagate messages up the hierarchy. If you've set up both the root logger and a specific module logger with handlers, you might get duplicate log messages. To prevent this, set propagate=False on the specific logger:

logger = logging.getLogger ("app.module")
logger.propagate = False # Don't send logs to the parent logger

Monitoring Your Logs

Once you have a good logging system in place, you need a way to monitor and analyze those logs. Here are some approaches:

  1. Log search and analysis tools: Use tools like Kibana, Graylog, or Splunk to search and visualize logs.

  2. Alerting: Set up alerts for specific log patterns that indicate problems.

  3. Log-based metrics: Extract metrics from logs to track system behavior over time.

  4. Anomaly detection: Use machine learning to detect unusual patterns in logs that might indicate problems.

For small applications, simple tools like grep or tail -f might be sufficient. For larger systems, dedicated log management solutions become essential.

To make log monitoring easier, consider adopting a consistent log format across all your applications and services. This makes it easier to set up parsing and filtering rules in your log management system.

Conclusion

Python's logging system is a powerful tool that, when used effectively, can significantly improve your ability to debug issues and understand what's happening in your application. From basic console logging to complex distributed setups with structured logs and centralized monitoring, the logging module can scale with your needs.

If you're monitoring your application's uptime, having good logs makes it much easier to diagnose and fix issues quickly. A tool like Odown can help you keep track of your application's availability, alerting you when problems occur. Combined with good logging practices, this gives you a powerful system for maintaining reliability.

Odown's SSL certificate monitoring can also alert you before certificates expire, while its public status pages allow you to communicate transparently with users during incidents. When an incident does occur, your logs will be invaluable in diagnosing the root cause and preventing similar issues in the future.

Remember, logging isn't just about debugging—it's about observability, understanding how your system behaves in production, and gaining insights that help you improve reliability and performance. Invest time in setting up good logging practices early in your project, and you'll thank yourself later when that mysterious bug appears at 3 AM.