How to use Python logging QueueHandler with dictConfig in Python 3.12

Support was added for configuring the logging queue handler with dictConfig in Python 3.12. This means that we can junk all the messing around I covered in my previous article. Well, almost …

If you just want to see the result go to my GitHub project medium-queue-logging, and look at examples/main_3_23.py and examples/queue_listener_3_12.py.

The new config supports creating the queue handler (with redirection to the actual outputs, file, console, e.t.c.), the listener, and the queue.

The first attempt at the new configuration is given below. You might want to compare this to the final config in my previous article.

version: 1
formatters:
  simple:
    format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
handlers:
  console:
    class: logging.StreamHandler
    level: DEBUG
    formatter: simple
    stream: ext://sys.stdout
  file:
    class: logging.FileHandler
    filename: 'config_example_3_12.log'
    formatter: simple
  queue_listener:
    class: logging.handlers.QueueHandler
    level: DEBUG
    handlers:
      - console
      - file
loggers:
  examples.my_script1:
    level: DEBUG
    handlers:
      - queue_listener
    propagate: false
  examples.my_script2:
    level: WARNING
    handlers: 
      - queue_listener
    propagate: false
root:
  level: WARN
  handlers:
    - console

How simple! Unfortunately this doesn’t quite work. The last step in enabling queue logging is to call start on the QueueListener object. With the above configuration this doesn’t happen. That’s fine though, we can call it ourselves.

handler = logging.getHandlerByName("queue_listener")
handler.listener.start()

This works perfectly.

Auto Starting the Listener

I kinda wanted the convenience that I had before, where things just worked through the config though. It turns out that the new config is pretty comprehensive; it allows you to set up your own listener. Nice.

Let’s write a listener that starts itself.

from logging.handlers import QueueListener

class AutoStartQueueListener(QueueListener):

    def __init__(self, queue, *handlers, respect_handler_level=False):
        super().__init__(queue, *handlers, respect_handler_level=respect_handler_level)
        # Start the listener immediately.
        self.start()

Adding it to the config is super easy.

  queue_listener:
    class: logging.handlers.QueueHandler
    listener: examples.queue_listener_3_12.AutoStartQueueListener
    level: DEBUG
    handlers:
      - console
      - file

Configuring the Queue Size

In my previous article I found a way to cap the queue size. The new configuration has support for this. Here’s the final config.

  queue_listener:
    class: logging.handlers.QueueHandler
    listener: examples.queue_listener_3_12.AutoStartQueueListener
    queue:
      (): queue.Queue
      maxsize: 1000
    level: DEBUG
    handlers:
      - console
      - file

Good luck with your coding.