Logging changes how you understand your application. Good logs help you debug faster, track user behavior, and catch problems before they affect customers. Bad logs clutter your disk and confuse your team.
Most Python developers start with print() statements. They move to logging.info() without configuration. Then production breaks and the logs tell them nothing. This guide shows you how to set up logging correctly from the start.
Prerequisites
You need Python 3.8 or newer. The logging module comes with Python’s standard library. For structured logging, you’ll install python-json-logger:
pip install python-json-logger
Basic knowledge of Python functions and classes helps. You should understand how to run Python scripts and install packages with pip.
Step 1: The Logging Module Basics
Python’s logging module has five levels: DEBUG, INFO, WARNING, ERROR, and CRITICAL. Each level filters messages based on severity.
import logging
logging.debug("Detailed information for diagnosing problems")
logging.info("Confirmation that things are working")
logging.warning("Something unexpected happened")
logging.error("The software failed to perform a function")
logging.critical("The program may be unable to continue")
By default, Python only shows WARNING and above. You need to configure the logger to see INFO and DEBUG messages.
Loggers, Handlers, and Formatters
The logging system has three components. Loggers expose the interface your code uses. Handlers send log records to destinations (console, file, network). Formatters specify the layout of log messages.
import logging
# Create a logger
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
# Create console handler
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.INFO)
# Create file handler
file_handler = logging.FileHandler('app.log')
file_handler.setLevel(logging.DEBUG)
# Create formatter
formatter = logging.Formatter(
'%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
# Add formatter to handlers
console_handler.setFormatter(formatter)
file_handler.setFormatter(formatter)
# Add handlers to logger
logger.addHandler(console_handler)
logger.addHandler(file_handler)
# Now log something
logger.info("Application started")
logger.debug("Debug information here")
This code creates a logger that writes INFO and above to console, but writes everything to a file. The formatter adds timestamps and log levels to each message.
Logger Hierarchy
Loggers form a hierarchy using dots in their names. A logger named myapp.database is a child of myapp. Child loggers inherit configuration from parents unless you override it.
import logging
# Parent logger
parent = logging.getLogger('myapp')
parent.setLevel(logging.INFO)
# Child logger inherits level
child = logging.getLogger('myapp.database')
child.info("This message appears because parent is INFO")
Use __name__ as your logger name. This gives you automatic hierarchy based on your module structure.
Step 2: Configuring Loggers
Hardcoding configuration in every file gets messy fast. Python offers three better approaches: dictConfig, fileConfig, and YAML.
Using dictConfig
The dictConfig method accepts a dictionary. This works well for programmatic configuration:
import logging
import logging.config
LOGGING_CONFIG = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'standard': {
'format': '%(asctime)s [%(levelname)s] %(name)s: %(message)s'
},
},
'handlers': {
'console': {
'class': 'logging.StreamHandler',
'level': 'INFO',
'formatter': 'standard',
'stream': 'ext://sys.stdout'
},
'file': {
'class': 'logging.handlers.RotatingFileHandler',
'level': 'DEBUG',
'formatter': 'standard',
'filename': 'logs/app.log',
'maxBytes': 10485760, # 10MB
'backupCount': 5
}
},
'loggers': {
'': { # Root logger
'handlers': ['console', 'file'],
'level': 'DEBUG',
'propagate': False
},
'myapp': {
'handlers': ['console', 'file'],
'level': 'DEBUG',
'propagate': False
}
}
}
logging.config.dictConfig(LOGGING_CONFIG)
logger = logging.getLogger(__name__)
logger.info("Configuration loaded")
The RotatingFileHandler prevents your log file from growing forever. It creates new files when the current one reaches maxBytes and keeps backupCount old files.
Using YAML Configuration
YAML files are easier to read and edit than Python dictionaries. First install PyYAML:
pip install pyyaml
Create logging.yaml:
version: 1
disable_existing_loggers: false
formatters:
standard:
format: '%(asctime)s [%(levelname)s] %(name)s: %(message)s'
detailed:
format: '%(asctime)s [%(levelname)s] %(name)s.%(funcName)s:%(lineno)d - %(message)s'
handlers:
console:
class: logging.StreamHandler
level: INFO
formatter: standard
stream: ext://sys.stdout
file:
class: logging.handlers.RotatingFileHandler
level: DEBUG
formatter: detailed
filename: logs/app.log
maxBytes: 10485760
backupCount: 5
loggers:
myapp:
level: DEBUG
handlers: [console, file]
propagate: false
root:
level: INFO
handlers: [console]
Load the configuration in your code:
import logging
import logging.config
import yaml
with open('logging.yaml', 'r') as f:
config = yaml.safe_load(f.read())
logging.config.dictConfig(config)
logger = logging.getLogger(__name__)
logger.info("Logging configured from YAML")
Environment-Specific Configuration
Production and development need different settings. Use environment variables to switch configurations:
import os
import logging.config
import yaml
env = os.getenv('ENVIRONMENT', 'development')
config_file = f'logging_{env}.yaml'
with open(config_file, 'r') as f:
config = yaml.safe_load(f.read())
logging.config.dictConfig(config)
Create separate files like logging_development.yaml and logging_production.yaml. Development can use DEBUG level and console output. Production uses INFO level and file handlers.
Step 3: Structured Logging with JSON
Text logs work fine until you have thousands of lines per minute. Then you need structured logs that machines can parse. JSON is the standard format.
Why JSON Logs Matter
Traditional logs look like this:
2026-02-09 10:15:32 [INFO] myapp.api: User logged in
JSON logs contain structured data:
{
"timestamp": "2026-02-09T10:15:32.123Z",
"level": "INFO",
"logger": "myapp.api",
"message": "User logged in",
"user_id": "12345",
"ip_address": "192.168.1.100",
"request_id": "abc-123-def"
}
Now you can filter logs by user_id, track requests across services, and build dashboards.
Implementing JSON Logging
The python-json-logger library adds JSON formatting:
import logging
from pythonjsonlogger import jsonlogger
logger = logging.getLogger()
logHandler = logging.StreamHandler()
formatter = jsonlogger.JsonFormatter(
'%(timestamp)s %(level)s %(name)s %(message)s'
)
logHandler.setFormatter(formatter)
logger.addHandler(logHandler)
logger.setLevel(logging.INFO)
logger.info("User action", extra={
'user_id': '12345',
'action': 'login',
'ip_address': '192.168.1.100'
})
The extra parameter adds custom fields to your log record. These fields appear in the JSON output.
Adding Context with Contextual Loggers
You often need the same fields in many log messages. Request IDs, user IDs, and session IDs should appear in every log from that context.
import logging
from pythonjsonlogger import jsonlogger
class ContextLogger:
def __init__(self, logger, **context):
self.logger = logger
self.context = context
def _log(self, level, message, **kwargs):
extra = {**self.context, **kwargs}
self.logger.log(level, message, extra=extra)
def info(self, message, **kwargs):
self._log(logging.INFO, message, **kwargs)
def error(self, message, **kwargs):
self._log(logging.ERROR, message, **kwargs)
# Usage
base_logger = logging.getLogger(__name__)
context_logger = ContextLogger(
base_logger,
request_id='abc-123',
user_id='12345'
)
context_logger.info("Processing request")
context_logger.info("Database query", query_time=0.045)
Every log from context_logger includes the request_id and user_id. This makes tracing requests through your system much easier.
Structlog for Advanced Needs
The structlog library offers more features than python-json-logger. It handles context binding, filtering, and processing:
pip install structlog
Basic setup:
import structlog
structlog.configure(
processors=[
structlog.processors.TimeStamper(fmt="iso"),
structlog.stdlib.add_log_level,
structlog.processors.JSONRenderer()
],
context_class=dict,
logger_factory=structlog.stdlib.LoggerFactory(),
)
logger = structlog.get_logger()
logger.info("Application started", version="1.0.0")
Bind context to the logger:
logger = structlog.get_logger()
logger = logger.bind(user_id="12345", request_id="abc-123")
logger.info("User logged in")
logger.info("Profile updated", fields_changed=["email", "name"])
The bound fields appear in all subsequent log messages from that logger instance.
Step 4: Logging in Web Applications
Web frameworks need special logging setup. You want to log requests, responses, errors, and performance metrics.
Django Logging
Django includes a logging configuration in settings.py. Extend it rather than replacing it:
# settings.py
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'verbose': {
'format': '{levelname} {asctime} {module} {process:d} {thread:d} {message}',
'style': '{',
},
'json': {
'()': 'pythonjsonlogger.jsonlogger.JsonFormatter',
'format': '%(asctime)s %(levelname)s %(name)s %(message)s'
}
},
'handlers': {
'console': {
'class': 'logging.StreamHandler',
'formatter': 'verbose',
},
'file': {
'class': 'logging.handlers.RotatingFileHandler',
'filename': 'logs/django.log',
'maxBytes': 1024 * 1024 * 15, # 15MB
'backupCount': 10,
'formatter': 'json',
},
},
'loggers': {
'django': {
'handlers': ['console', 'file'],
'level': 'INFO',
'propagate': False,
},
'myapp': {
'handlers': ['console', 'file'],
'level': 'DEBUG',
'propagate': False,
},
},
}
Add request logging with middleware:
# middleware.py
import logging
import time
logger = logging.getLogger(__name__)
class RequestLoggingMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
start_time = time.time()
response = self.get_response(request)
duration = time.time() - start_time
logger.info(
"Request processed",
extra={
'method': request.method,
'path': request.path,
'status_code': response.status_code,
'duration': duration,
'user_id': getattr(request.user, 'id', None)
}
)
return response
Register the middleware in settings.py:
MIDDLEWARE = [
'myapp.middleware.RequestLoggingMiddleware',
# ... other middleware
]
Flask Logging
Flask uses Python’s standard logging. Configure it at application startup:
from flask import Flask, request, g
import logging
from pythonjsonlogger import jsonlogger
import time
app = Flask(__name__)
# Setup JSON logging
logHandler = logging.StreamHandler()
formatter = jsonlogger.JsonFormatter(
'%(asctime)s %(levelname)s %(name)s %(message)s'
)
logHandler.setFormatter(formatter)
app.logger.addHandler(logHandler)
app.logger.setLevel(logging.INFO)
@app.before_request
def before_request():
g.start_time = time.time()
@app.after_request
def after_request(response):
if hasattr(g, 'start_time'):
duration = time.time() - g.start_time
app.logger.info(
"Request processed",
extra={
'method': request.method,
'path': request.path,
'status_code': response.status_code,
'duration': duration,
'ip': request.remote_addr
}
)
return response
@app.errorhandler(Exception)
def handle_exception(e):
app.logger.error(
"Unhandled exception",
extra={
'error': str(e),
'path': request.path,
'method': request.method
},
exc_info=True
)
return "Internal Server Error", 500
FastAPI Logging
FastAPI works well with structlog. Set it up in your main application file:
from fastapi import FastAPI, Request
import structlog
import time
structlog.configure(
processors=[
structlog.processors.TimeStamper(fmt="iso"),
structlog.stdlib.add_log_level,
structlog.processors.JSONRenderer()
],
logger_factory=structlog.stdlib.LoggerFactory(),
)
app = FastAPI()
logger = structlog.get_logger()
@app.middleware("http")
async def log_requests(request: Request, call_next):
start_time = time.time()
response = await call_next(request)
duration = time.time() - start_time
logger.info(
"request_processed",
method=request.method,
path=request.url.path,
status_code=response.status_code,
duration=duration,
client_ip=request.client.host
)
return response
@app.get("/")
async def root():
logger.info("Root endpoint accessed")
return {"message": "Hello World"}
For endpoint-specific logging, use dependency injection:
from fastapi import Depends
import structlog
def get_logger(request: Request):
return structlog.get_logger().bind(
path=request.url.path,
method=request.method,
request_id=request.headers.get('X-Request-ID', 'unknown')
)
@app.get("/users/{user_id}")
async def get_user(user_id: int, logger=Depends(get_logger)):
logger.info("Fetching user", user_id=user_id)
# ... fetch user from database
logger.info("User fetched successfully", user_id=user_id)
return {"user_id": user_id}
Step 5: Log Aggregation and Monitoring
Production applications generate too many logs to read manually. Log aggregation tools collect, parse, and analyze your logs.
ELK Stack (Elasticsearch, Logstash, Kibana)
The ELK stack is popular for log management. Elasticsearch stores logs, Logstash processes them, and Kibana visualizes them.
Send logs to Logstash using the TCP handler:
import logging
import logging.handlers
logger = logging.getLogger('myapp')
logger.setLevel(logging.INFO)
# Send logs to Logstash on localhost:5000
handler = logging.handlers.SocketHandler('localhost', 5000)
logger.addHandler(handler)
logger.info("Application started")
For JSON logs, use filebeat to ship logs to Elasticsearch. Configure your application to write JSON logs to a file:
import logging
from pythonjsonlogger import jsonlogger
logger = logging.getLogger()
handler = logging.FileHandler('/var/log/myapp/app.log')
formatter = jsonlogger.JsonFormatter()
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.INFO)
Filebeat configuration (filebeat.yml):
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/myapp/*.log
json.keys_under_root: true
json.add_error_key: true
output.elasticsearch:
hosts: ["localhost:9200"]
index: "myapp-logs-%{+yyyy.MM.dd}"
Using Sentry for Error Tracking
Sentry specializes in error tracking and performance monitoring. It captures exceptions with full context:
pip install sentry-sdk
Basic setup:
import sentry_sdk
from sentry_sdk.integrations.logging import LoggingIntegration
sentry_logging = LoggingIntegration(
level=logging.INFO,
event_level=logging.ERROR
)
sentry_sdk.init(
dsn="your-sentry-dsn-here",
integrations=[sentry_logging],
traces_sample_rate=1.0,
)
Now all ERROR level logs go to Sentry automatically. Add context to errors:
import logging
logger = logging.getLogger(__name__)
try:
result = divide(10, 0)
except Exception as e:
logger.error("Division failed", extra={
'user_id': '12345',
'operation': 'divide',
'numerator': 10,
'denominator': 0
})
raise
Sentry captures the exception with all context data.
Using CloudWatch for AWS Applications
AWS CloudWatch collects logs from EC2, Lambda, and ECS. Use the watchtower library to send logs directly:
pip install watchtower
Configuration:
import logging
import watchtower
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
handler = watchtower.CloudWatchLogHandler(
log_group='myapp',
stream_name='production'
)
logger.addHandler(handler)
logger.info("Application started")
For Lambda functions, print JSON to stdout. CloudWatch captures it automatically:
import json
import logging
from pythonjsonlogger import jsonlogger
logger = logging.getLogger()
handler = logging.StreamHandler()
formatter = jsonlogger.JsonFormatter()
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.INFO)
def lambda_handler(event, context):
logger.info("Lambda invoked", extra={
'event': event,
'request_id': context.request_id
})
return {'statusCode': 200}
Common Pitfalls
Logging Too Much
Excessive logging slows your application and fills your disk. Avoid logging inside tight loops:
# Bad
for item in large_list:
logger.debug(f"Processing item: {item}")
# Good
logger.info(f"Processing {len(large_list)} items")
for item in large_list:
process(item)
logger.info("Processing complete")
Use appropriate log levels. DEBUG is for development. Production should use INFO for normal operations and WARNING for problems.
Logging Sensitive Data
Never log passwords, API keys, credit card numbers, or personal information:
# Bad
logger.info(f"User logged in with password: {password}")
# Good
logger.info(f"User logged in", extra={'user_id': user_id})
Filter sensitive fields before logging:
def sanitize_data(data):
sensitive_fields = ['password', 'api_key', 'credit_card']
return {k: '***' if k in sensitive_fields else v
for k, v in data.items()}
user_data = {'username': 'john', 'password': 'secret123'}
logger.info("User data", extra=sanitize_data(user_data))
String Formatting in Log Calls
Don’t format strings before passing them to the logger:
# Bad - formats string even if log level is too low
logger.debug(f"User {user_id} performed action {action}")
# Good - only formats if message will be logged
logger.debug("User %s performed action %s", user_id, action)
The second approach only creates the string if the log level allows the message through. This saves CPU time.
Not Using Logger Names
Always use named loggers instead of the root logger:
# Bad
import logging
logging.info("Something happened")
# Good
import logging
logger = logging.getLogger(__name__)
logger.info("Something happened")
Named loggers let you control different parts of your application separately. You can set database logs to DEBUG while keeping API logs at INFO.
Ignoring Exceptions
Always include exception information when logging errors inside except blocks:
# Bad
try:
risky_operation()
except Exception as e:
logger.error(f"Operation failed: {e}")
# Good
try:
risky_operation()
except Exception as e:
logger.error("Operation failed", exc_info=True)
The exc_info=True parameter adds the full stack trace to your log. This shows you exactly where the error occurred.
Summary
Good logging practices start with proper configuration. Use YAML or dictConfig instead of hardcoding settings. Choose appropriate log levels for different environments.
Structured logging with JSON makes logs machine-readable. Add context with extra fields. Use libraries like structlog for advanced features.
Web frameworks need request logging middleware. Capture method, path, duration, and status code for every request. Log errors with full context.
Production applications need log aggregation. The ELK stack works well for general logging. Sentry excels at error tracking. CloudWatch integrates with AWS services.
Avoid logging too much, logging sensitive data, or using string formatting in log calls. Use named loggers and always include exception information.
Start with basic logging and add complexity as you need it. When something breaks in production at 2 AM, good logs are the difference between a 10-minute fix and a 3-hour investigation.
Discussion
Leave a comment
No comments yet
Be the first to start the conversation.