Hello,

I’ve recently implemented a logging system in the Farseer-NMR project using the Python Logging library. As the project grows, the original, simplified and own made logger I had developed was no more suited and it needed to be updated.

There are many logging systems available for Python, from simple to very professional; I am not going to name them, you can easily search them out there. For the case of Farseer-NMR, I chose the Python Logging library because it is, from my understanding, one of the simplest to implement, yet powerful to use, and no more advanced functionalities were needed for the time being.

There are plenty of blogs and sites (and the official documentation itself) that explain how implement the Python Logging library in simple and more complex scripts – please read else where for that information. However, I found little information and examples on how to implement a full featured logging functionality in a complex software of multiple Python libraries. I am definitively not an expert on logging systems, and my understanding is limited to what I needed to learn for this implementation, nonetheless, I think what I’ve learned will be useful for any one looking for implementing the Python logging systems in a software of multiple libraries.

It’s the aim of this post to explain how I did it for Farseer-NMR. If any skilled reader out there wishes to propose any improvement to this method he/she will be very welcomed to post a comment, I will definitively appreciate :-)

So let’s start,

I am considering the following setup: a main.py script sets the workflow of the program where several classes in the libs/ folder interact with each other; each class is independent and must have its own logging system. The question then is: how to set up a unified logging system that can track to log of the whole program and the libraries separately?

- main.py
- libs/
    - __init__.py
    - Class1.py
    - Class2.py
    - Logger.py

1) Define a Logger class that manages the logging system configuration and initiation. The Logger class should contain:

• a dictionary class attribute (not instance attribute) with the user defined logger configuration. Bellow, I present the one I used for the Farseer-NMR project.
• The __init__() function can receive any arguments that might help on further setup of the configuration.
• a setup_log() function to initiate the logger.

Bellow the structure of the Logger class:

class Logger:
    """Logger container and configuration."""
    
    # here goes the configuration dictionary
    
    def __init__(self):
        # do init

    def setup_log(self):
        # do logging setup

Here is the configuration dictionary I use. What I want is that the logger INFO level messages are printed to the terminal and written to an external file «info.log» while the DEBUG level messages are written to the external file «debug.log». Also the «formatters» setting allow to use separate formatting settings in both «info.log» and «debug.log». For example, it is very useful that information on the file name, function and line number are printed to the «debug.log» file.

log_config = {
        "version": 1,
        "disable_existing_loggers": False,
        "formatters": {
            "debug_format": {
                "format": "%(asctime)s - %(levelname)s - %(filename)s:%(name)s:%(funcName)s:%(lineno)d - %(message)s"
            },
            "info_format": {}
        },
        
        "handlers": {
            "console": {
                "class": "logging.StreamHandler",
                "level": "INFO",
                "formatter": "info_format",
                "stream": "ext://sys.stdout"
            },
            "info_file_handler": {
                "class": "logging.handlers.RotatingFileHandler",
                "level": "INFO",
                "formatter": "info_format",
                "filename": "info.log",
                "maxBytes": 10485760,
                "backupCount": 20,
                "encoding": "utf8"
            },
            "debug_file_handler": {
                "class": "logging.handlers.RotatingFileHandler",
                "level": "DEBUG",
                "formatter": "debug_format",
                "filename": "debug.log",
                "maxBytes": 10485760,
                "backupCount": 20,
                "encoding": "utf8"
            }
        },
        
        "loggers": {},
        
        "root": {
            "level": "DEBUG",
            "handlers": ["console", "info_file_handler", "debug_file_handler"]
        }
    }

The __init__() function can received additional arguments to enhance the logger configuration. The parameter name is mandatory because it will be used to initiate the logging.getLogger(). In this example, a new_dir parameter can be optionally given. I use this parameter to update the configuration dictionary with the desired path to write the log files. Because the configuration dictionary is a class attribute, it is possible to change it the first time the Logger class is instantiated and such information will be available to all subsequent Logger instantiations.

As described elsewhere on the web, the name parameter should be the Python special __name__ variable.

    def __init__(self, name, new_dir=''):
        """
        Parameters:
            - name (str): name of the logger, use __name__ variable
            - new_dir (opt, str): the new directory to store the log files
        """
        
        if new_dir:
            self.log_config["handlers"]["info_file_handler"]["filename"] = \
                "{}/info.log".format(new_dir)
            self.log_config["handlers"]["debug_file_handler"]["filename"] = \
                "{}/debug.log".format(new_dir)
        
        self.name = name

The setup_log() function initiates the logger instance by calling logging.getLogger() and returning the Logger object.

    def setup_log(self):
        
        logging.config.dictConfig(self.log_config)
        return logging.getLogger(self.name)

So, now that the Logger class is defined, how to initiate the logging system in our program composed of several libraries?

Wherever you want to initiate the logging system just import the Logger class and temporarily instantiate a Logger object and store in a variable what is returned by the Logger.setup_log() function, which is the result of logging.getLogger() as seen above but with an initial step to apply the configuration file. If you are initiating the logging system for a Class instance just store the logger in a class instance attribute.

from libs.Logger import Logger

class Main:
	""" This is my first class."""

	def __init__(self, path, par1):
	
	# initiates log
	self.logger = Logger(__name__, path).setup_log()
	self.logger.info('this is an info message')
	self.logger.debug('this is a debug level message')

	# continue __init__

The order in which each logging logger is instantiated is maintained by definition of the Python Logging library and all the messages will converge to the same files because the log_config dictionary is shared among all the instances of Logger. For this reason and in the case of Farseer-NMR, I only initiate the Logger class with the new_dir parameter the first time the Logger is initiated, that is, in the main.py script (farseermain.py). In the other libs I simply call Logger(__name__).setup_log().

I hope this explanation helps others looking for similar implementations.

The code snippets were prepared with http://hilite.me/.

Best,

João