Python 日志模块的定制

Python标准logging模块中主要分为四大块内容：

Logger: 定义应用程序使用的接口

Handler: 将Loggers产生的日志输出到目的地

Filter: 对Loggers产生的日志进行过滤

Formatter: 格式化Loggers产生的日志

其中常用的是Logger, Handler, Formatter。

目标

对logging模块进行封装，使其尽量达到几个目标：

易用性：尽量保持python原生标准的用法。
扩展性：避免将所有模块集中在一个文件中进行配置，而是对不同模块继承扩展。如此可以确保修改时尽量保持简洁性，而非混淆杂乱。
文字加色：同一时间段内出现大量日志输出时，可以更清晰地识别出问题所在。
输出多个目的地：在生产代码中，并非所有日志信息都输出到console或者file中，而是日志的重要级别，将那些比较重要的发送到console，将那些不太重要但以后分析问题时可能又会用到的发送到file中。
非阻塞发送器：记录日志时如果涉及到网络（如socket, smtp），那么这个记录过程可能会阻塞当前线程，在python 3.2+，我们可以非常容易地设计一个非阻塞的发送器（当然了，非阻塞机制的内部实现大多都是队列）。

目录结构

如下的定制日志主要包括两个包，log_config和tools，目录结构如下：

log_config 模块结构

默认配置

Logging Level:

　　logging.NOTSET is default.

　　You can change it by passing ‘level=’ to BTLogger instance in your application code.

Sending Level:

　　Send logging messages to console with levels of logging.INFO and higher, and simultaneously to the disk file with levels of logging.DEBUG and higher.

　　You can change them by passing ‘level=’ to BTStreamHandler/BTFileHandler instance in log_config\__init__.py.

Log Config:

　　LOG_PATH = r'C:log'
　　LOG_NAME = r'mylogger.log'

　　You can change them in log_config\__init__.py.

Console Text Color:

　　DEBUG

　　INFO

　　WARNING

　　ERROR

　　CRITICAL

　　You can change them by passing ‘fore_color=’ and ‘back_color=’ to decorator @text_attribute.set_textcolor(fore_color, back_color) in log_config\__init__.py.

There also exists a NonBlockingHandler class in log_confighandlers.py, which will use features of version 3.2+. If you unfold it, you can get a non_blocking_handler.

Demo

ts_log_config.py:

 1 def log_test():
 2     import log_config
 3 
 4     logger = log_config.BTLogger('mylogger')
 5 
 6     logger.debug('Debug message')
 7     logger.info('Info message')
 8     logger.warning('Warning message')
 9     logger.error('Error message')
10     logger.critical('Critical message')
11 
12 def main():
13     log_test()
14 
15 if __name__ == '__main__':
16     main()

outputs:

代码实现

..log_config\__init__.py:

  1 """
  2 Wrap the classes of logging module so as to expose more flexible interfaces that application code directly uses.
  3 
  4 By default, when you do some log operations, it will log messages to console with levels of logging.INFO and higher,
  5 and simultaneously to the disk file with levels of logging.DEBUG and higher.
  6 
  7 Usages:
  8 To use log_config module, simply 'import log_config' in your code, and create a BTLogger instance, then, you can use this
  9 instance to log messages:
 10     import log_config
 11 
 12     logger = log_config.BTLogger('mylogger')
 13 
 14     logger.debug('Debug message')
 15     logger.info('Info message')
 16     logger.warning('Warning message')
 17     logger.error('Error message')
 18     logger.critical('Critical message')
 19 """
 20 
 21 import os
 22 import sys
 23 import logging
 24 from log_config import formatters
 25 from log_config import handlers
 26 from log_config import text_attribute
 27 from tools import os_tools
 28 
 29 __author__ = " "
 30 __status__ = "debugging"  # {debugging, production}
 31 __version__ = "0.1.0"  # major_version_number.minor_version_number.revision_number
 32 __date__ = " "
 33 
 34 """
 35 # Config root logger
 36 basicConfig(
 37     filename=__name__,
 38     filemode='w',
 39     format='%(asctime)s: %(levelname)s: %(message)s',
 40     datefmt='%Y-%m-%d %H:%M:%S',
 41     level='logging.NOTSET'
 42 )
 43 """
 44 
 45 # Config log path and file name.
 46 LOG_PATH = r'C:log'
 47 LOG_NAME = r'mylogger.log'
 48 filename = 'default.log' if not os_tools.makedir(LOG_PATH) else os.path.abspath(os.path.join(LOG_PATH, LOG_NAME))
 49 
 50 
 51 class BTLogger(logging.Logger):
 52     def __init__(self, name=None, level=logging.NOTSET):
 53         """
 54         Initialize the BTLogger with basic settings.
 55 
 56         :param
 57             name: Specify the logger's name. If name is None, default is root logger.
 58                    Note:
 59                         All BTLogger instances with a given name return the same logger
 60                         instance. This means that logger instances never need to be passed
 61                         between different parts of an application.
 62             level: The threshold of logging records, messages which are less severe
 63                    than level will be ignored.
 64                    If the level is set to NOTSET, than:
 65                         1> If current logger is root logger, than all messages will
 66                         be logged.
 67                         2> Or, delegate to its parent logger, until an ancestor with
 68                         a level other than NOTSET is found, or the root is reached.
 69                    Note:
 70                         The root logger is created with level logging.WARNING.
 71         :return:
 72             None
 73         """
 74         logging.Logger.__init__(self, name, level)
 75 
 76         # Create handlers
 77         # non_blocking_ch = handlers.NonBlockingHandler(handlers.BTStreamHandler(sys.stdout, logging.DEBUG)).get_handler()  # version 3.2+
 78         console_handler = handlers.BTStreamHandler(stream=sys.stdout, level=logging.INFO)
 79         file_handler = handlers.BTFileHandler(filename=filename, level=logging.DEBUG, mode='w')
 80 
 81         # Set formatters to handlers.
 82         # non_blocking_ch.setFormatter(formatters.BTStreamFormatter())  # version 3.2+
 83         console_handler.setFormatter(formatters.BTStreamFormatter())
 84         file_handler.setFormatter(formatters.BTFileFormatter())
 85 
 86         # Add the handlers to logger.
 87         # self.addHandler(non_blocking_ch)  # version 3.2+
 88         self.addHandler(console_handler)
 89         self.addHandler(file_handler)
 90 
 91     # Override methods in logging.Logger(debug, info, warning, error, critical).
 92     # @handlers.NonBlockingHandler.non_blocking     # version 3.2+
 93     @text_attribute.set_textcolor('FOREGROUND_BLACK', 'BACKGROUND_WHITE')
 94     def debug(self, msg, *args, **kwargs):
 95         self.log(logging.DEBUG, msg, *args, **kwargs)
 96 
 97     @text_attribute.set_textcolor('FOREGROUND_BLACK', 'BACKGROUND_GREEN')
 98     def info(self, msg, *args, **kwargs):
 99         self.log(logging.INFO, msg, *args, **kwargs)
100 
101     @text_attribute.set_textcolor('FOREGROUND_BLACK', 'BACKGROUND_YELLOW')
102     def warning(self, msg, *args, **kwargs):
103         self.log(logging.WARNING, msg, *args, **kwargs)
104 
105     warn = warning
106 
107     @text_attribute.set_textcolor('FOREGROUND_BLACK', 'BACKGROUND_RED')
108     def error(self, msg, *args, **kwargs):
109         self.log(logging.ERROR, msg, *args, **kwargs)
110 
111     @text_attribute.set_textcolor('FOREGROUND_BLACK', 'BACKGROUND_RED')
112     def critical(self, msg, *args, **kwargs):
113         self.log(logging.CRITICAL, msg, *args, **kwargs)
114 
115     fatal = critical

View Code

..log_configformatters.py:

 1 """
 2 Formatters specify the output format of logging messages.
 3 """
 4 
 5 from logging import Formatter
 6 
 7 
 8 # Changed in version 3.2+: The 'style' parameter was added.
 9 '''
10 class BTStreamFormatter(Formatter):
11     def __init__(self,
12                  fmt='[%(asctime)s] [%(levelname)s]: %(message)s',
13                  datefmt='%Y-%m-%d %H:%M:%S', style='%'):
14         Formatter.__init__(self, fmt, datefmt, style)
15 
16 
17 class BTFileFormatter(Formatter):
18     def __init__(self,
19                  fmt='%(asctime)s: %(levelname)s: %(message)s',
20                  datefmt='%Y/%m/%d %H:%M:%S', style='%'):
21         Formatter.__init__(self, fmt, datefmt, style)
22 '''
23 
24 # Be suitable for version 3.2-.
25 class BTStreamFormatter(Formatter):
26     def __init__(self,
27                  fmt='[%(asctime)s] [%(levelname)s]: %(message)s',
28                  datefmt='%Y-%m-%d %H:%M:%S'):
29         Formatter.__init__(self, fmt, datefmt)
30 
31 
32 class BTFileFormatter(Formatter):
33     def __init__(self,
34                  fmt='%(asctime)s: %(levelname)s: %(message)s',
35                  datefmt='%Y/%m/%d %H:%M:%S'):
36         Formatter.__init__(self, fmt, datefmt)

View Code

..log_confighandlers.py:

 1 """
 2 Handlers send logging messages(created by BTLogger) to the specific destination.
 3 """
 4 
 5 import sys
 6 import logging
 7 from logging import StreamHandler
 8 from logging import FileHandler
 9 
10 
11 class BTStreamHandler(StreamHandler):
12     """
13     Write logging records to a stream(e.g., console).
14 
15     Note that this class does not close the stream automatically,
16     as sys.stdout or sys.stderr may be used.
17     """
18     def __init__(self, stream=sys.stderr, level=logging.INFO):
19         """
20         Initialize the handler.
21 
22         :param
23             stream: If stream is not specified, sys.stderr is used.
24         :return
25             None
26         """
27         StreamHandler.__init__(self, stream)
28         self.setLevel(level)    # The threshold of handling records, default is logging.INFO.
29 
30 
31 class BTFileHandler(FileHandler):
32     """
33     Write logging records to disk files.
34 
35     Inherited from logging.FileHandler, only modify the default mode('a') to ('w').
36     """
37     def __init__(self,
38                  filename='default.log',
39                  level=logging.DEBUG,   # Handle messages with levels of logging.DEBUG and higher to file.
40                  mode='w',
41                  encoding=None,
42                  delay=False):
43         FileHandler.__init__(self, filename, mode, encoding, delay)
44         self.setLevel(level)
45 
46 
47 # New in version 3.2+: QueueHandler, QueueListener.
48 # Unfold the following block, and you will get a non_blocking_handler.
49 '''
50 from queue import Queue
51 from logging.handlers import QueueHandler
52 from logging.handlers import QueueListener
53 from functools import wraps
54 
55 class NonBlockingHandler(object):
56     """
57     Let logging handlers do their work without blocking the thread you're logging from.
58     Especially those network-based handler can block, e.g., SocketHandler, SMTPHandler.
59     """
60     def __init__(self, handler_instance):
61         self.queue = Queue(-1)
62         self.queue_handler = QueueHandler(self.queue)
63         self.handler_instance = handler_instance
64         global _queue_listener
65         _queue_listener = QueueListener(self.queue, self.handler_instance)
66 
67     def get_handler(self):
68         return self.handler_instance
69 
70     @classmethod
71     def non_blocking(cls, func):
72         @wraps(func)
73         def wrapper(*args, **kwargs):
74             _queue_listener.start()
75             res = func(*args, **kwargs)
76             _queue_listener.stop()
77             return res
78 
79         return wrapper
80 '''

View Code

..log_config ext_attribute.py:

 1 """
 2 Set the text attributes in console. Currently, you can only set font color.
 3 
 4 # -----------------------------------------
 5 # Author: 
 6 # Created: 
 7 # Modified: 
 8 # Version: 0.1.0
 9 # -----------------------------------------
10 
11 Usages:
12 When you want to design a function which outputs messages to console with different colors, simply use '@text_attribute.set_textcolor(fore_color, back_color)' to decorate your function, e.g.:
13     @text_attribute.set_textcolor('FOREGROUND_BLACK', 'BACKGROUND_RED')
14     def func(*args, **kwargs):
15         # output some messages to console
16         # ...
17 
18     func(...)
19 """
20 
21 import ctypes
22 from functools import wraps
23 
24 
25 STD_INPUT_HANDLE = -10
26 STD_OUTPUT_HANDLE = -11
27 STD_ERROR_HANDLE = -12
28 
29 
30 _CONSOLE_TEXT_COLOR = {
31     'FOREGROUND_RED': 0x04,
32     'FOREGROUND_GREEN': 0x02,
33     'FOREGROUND_BLUE': 0x01,
34     # 'FOREGROUND_INTENSITY': 0x08,
35     'FOREGROUND_WHITE': 0x07,
36     'FOREGROUND_BLACK': 0x00,
37     'FOREGROUND_YELLOW': 0x06,
38 
39     'BACKGROUND_RED': 0x40,
40     'BACKGROUND_GREEN': 0x20,
41     'BACKGROUND_BLUE': 0x10,
42     # 'BACKGROUND_INTENSITY': 0x80,
43     'BACKGROUND_WHITE': 0x70,
44     'BACKGROUND_BLACK': 0x00,
45     'BACKGROUND_YELLOW': 0x60
46 }
47 
48 
49 class TextColor(object):
50     """
51     Set console text color.
52     """
53     std_output_handle = ctypes.windll.kernel32.GetStdHandle(STD_OUTPUT_HANDLE)
54 
55     @staticmethod
56     def text_color(fore_color, back_color):
57         fore_color = 'FOREGROUND_WHITE' if fore_color not in _CONSOLE_TEXT_COLOR else fore_color
58         back_color = 'BACKGROUND_BLACK' if back_color not in _CONSOLE_TEXT_COLOR else back_color
59         return _CONSOLE_TEXT_COLOR[fore_color] | _CONSOLE_TEXT_COLOR[back_color]
60 
61     @staticmethod
62     def set(fore_color='FOREGROUND_WHITE', back_color='BACKGROUND_BLACK', handle=std_output_handle):
63         is_set = ctypes.windll.kernel32.SetConsoleTextAttribute(handle, TextColor.text_color(fore_color, back_color))
64         return is_set
65 
66     @staticmethod
67     def unset():
68         return TextColor.set()
69 
70 
71 def set_textcolor(fore_color='FOREGROUND_WHITE', back_color='BACKGROUND_BLACK'):
72 
73     def decorate(func):
74         """
75         A decorator which can set the color of console message, include foreground and background.
76         :param
77             func: The original function which will be wrapped by a wrapper.
78         :return:
79             wrapper
80         """
81         @wraps(func)
82         def wrapper(*args, **kwargs):
83             TextColor.set(fore_color, back_color)
84             res = func(*args, **kwargs)
85             TextColor.unset()
86             return res
87 
88         return wrapper
89 
90     return decorate

View Code

.. ools\__init__.py: None

.. oolsos_tools.py:

 1 """
 2 Originated from os module.
 3 
 4 # -----------------------------------------
 5 # Author: 
 6 # Created: 
 7 # Modified:
 8 # Version: 0.1.0
 9 # -----------------------------------------
10 """
11 
12 import os
13 
14 
15 def makedir(path):
16     """
17     Recursive directory creation, include all intermediate-level directories needed to contain the leaf directory.
18 
19     If specific path already exists, do nothing. Or path is not a directory, return False.
20     """
21     try:
22         os.makedirs(path, 0o777)
23         # os.makedirs(path, 0o777, False) # Version 3.2+
24     except OSError:
25         if not os.path.isdir(path):      # Path is not a valid directory.
26             return False                 # Path already exists.
27         return True
28     else:                                # Path does not exist.
29         return True

View Code