log_safe
is a Python library that provides safe and efficient logging capabilities for multiprocessing applications. It ensures that logging is thread-safe and process-safe, making it ideal for complex, multi-process Python applications.
- Thread-safe and process-safe logging
- Centralized log collection from multiple processes
- Customizable logging configurations for both listener and worker processes
- Automatic initialization of logging in child processes
- Watchdog mechanism to prevent hanging listener process daemons
- Seamless integration with Python's built-in
multiprocessing
andconcurrent.futures
modules
Here's a simple example of how to use log_safe
:
from log_safe import initialize_safe_logging
import logging
import multiprocessing
initialize_safe_logging()
def worker_function():
logger = logging.getLogger()
logger.info("This is a log message from a worker process")
if __name__ == "__main__":
logger = logging.getLogger()
logger.info("Starting main process")
with multiprocessing.Pool(processes=2) as pool:
pool.apply_async(worker_function)
pool.apply_async(worker_function)
logger.info("Main process completed")
- Initialize logging before using any multiprocessing features:
from log_safe import initialize_safe_logging
initialize_safe_logging()
- Use with
multiprocessing.Pool
,ProcessPoolExecutor
, ormultiprocessing.Process
as you normally would.log_safe
automatically patches these to ensure proper logging initialization in all child processes.
You can provide custom configurations for both the listener and worker processes:
from log_safe import initialize_safe_logging
listener_config = {
# ... listener configuration ...
}
worker_config = {
# ... worker configuration ...
}
initialize_safe_logging(listener_config, worker_config)
- Always call
initialize_safe_logging()
in the main process before creating any child processes. - The library uses global variables to maintain the logging state. Be cautious when modifying global state in your application.
- The watchdog mechanism will shut down the logging system if it's idle for too long (default is 10 hours). Adjust this if needed for long-running applications.
For more detailed information, use cases, and advanced usage, please refer to the full documentation on GitHub.