Friday, May 9

One tip about logging.debug() in python

The idea of logging.debug( output string ) is that, if the logging system was set as higher level, the output string will be ignored, not being output to the logging system.

One problem, if the logging.debug() is calling some functions to format the output string, that functions ARE executed, no matter what the logging system is.

For example, I used this sentence to find out the size of the critical data object in the environment:
logging.debug("  size of dataprovider1 is: %s", sys.getsizeof(cPickle.dumps(dataprovider1)))
It works fine in debug environment; then when switching to production environment, even though the logging.level was set to INFO, this sentence was not being output to the log file, this function was still being executed, and an object of 90G size was created, and crashed the production system immediately.

Solution: use the isEnableFor() function of the logging.
if logging.root.isEnabledFor(logging.DEBUG):
    logging.debug("  size of dataprovider1 is: %s", sys.getsizeof(cPickle.dumps(dataprovider1)))

Labels: