Draft: Use Python logging module in Python clients
When built with the USE_PYTHON_LOGGER option, which is now enabled when building Python wheels, a Python logger object from the Python logging module is used for log messages. This allows the user to control log settings like the log level and format via the usual Python tools and the special set_log_level method of the producer object could be deprecated and removed in the future.
I marked this as draft for the following reasons:
- There is a small race condition between checking if the python interpreter is finalizing and trying to acquire the GIL. It might be better to stop all background threads before the interpreter shuts down with an
atexit
hook or similar - Every log message now requires the GIL. This might be slower (needs benchmarks) and increases the chance for deadlocks (if a thread currently holding the GIL waits for a lock currently hold by a thread that wants to log something).
A better approach might be to put all log messages to a C++ queue similar to queue.Queue
and a start a dedicated thread that forwards the log messages from the queue to the Python logger object.