billiard Documentation

billiard is a standalone fork of the Python multiprocessing package. It is designed to provide fixes, improvements, and patches drawn from python-trunk, as well as specific enhancements used by the Celery project.

Key features and origins include:

• A fork of the Python 2.7 multiprocessing package.
• Includes the no-execv patch.
• Includes Pool improvements originally located in Celery.
• It is a dependency for Celery and is maintained by the Celery team.

When using multiprocessing.Queue, a process that has put items into the queue will wait to terminate until all buffered items are fed by the internal "feeder" thread to the underlying pipe. To avoid deadlocks, ensure that all items put on the queue are eventually removed by a consumer before the process is joined.

Alternatively, a child process can call Queue.cancel_join_thread() to avoid this waiting behavior.

from multiprocessing import Process, Queue

def f(q):
    q.put('X' * 1000000)

if __name__ == '__main__':
    queue = Queue()
    p = Process(target=f, args=(queue,))
    p.start()
    # WRONG: p.join() will deadlock here because the queue is full
    # p.join() 
    
    # CORRECT: Get the item before joining, or don't join
    obj = queue.get()
    p.join()

To prevent security risks associated with unpickling data from untrusted sources, Listener and Client use HMAC digest authentication.

An authkey is a bytes object used as a password. If no authkey is explicitly provided, the connection uses current_process().authkey, which is automatically inherited by any child multiprocessing.Process objects created by the current process. This allows processes in a multi-process program to share a single authentication key by default. You can also generate suitable keys using os.urandom.

A Namespace object has no public methods but has writable attributes. Its representation shows the values of its attributes.

Warning: When using a proxy for a namespace object, any attribute starting with an underscore (_) is treated as an attribute of the proxy itself, not the underlying referent.

manager = multiprocessing.Manager()
Global = manager.Namespace()
Global.x = 10
Global.y = 'hello'
Global._z = 12.3    # this is an attribute of the proxy
print(Global)        # Namespace(x=10, y='hello')

While Unix processes can sometimes access shared resources via global variables, it is best practice to pass objects (like Lock, Semaphore, etc.) as arguments to the child process constructor. This ensures compatibility with Windows and prevents the object from being garbage collected in the parent process while the child is still using it.

from multiprocessing import Process, Lock

def f(l):
    # ... do something using "l" ...
    pass

if __name__ == '__main__':
    lock = Lock()
    for i in range(10):
        # Pass the lock explicitly via args
        Process(target=f, args=(lock,)).start()

To create a custom manager, subclass BaseManager and use the register classmethod to associate a name with a class or callable. This allows you to expose custom objects through the manager proxy.

from multiprocessing.managers import BaseManager

class MathsClass:
    def add(self, x, y):
        return x + y
    def mul(self, x, y):
        return x * y

class MyManager(BaseManager):
    pass

# Register the class with a name
MyManager.register('Maths', MathsClass)

if __name__ == '__main__':
    manager = MyManager()
    manager.start()
    maths = manager.Maths()
    print(maths.add(4, 3))         # prints 7
    print(maths.mul(7, 8))         # prints 56

You can run a manager server on one machine and connect to it from others. The server requires an address (IP/hostname and port) and an authkey for security.

# --- SERVER SIDE ---
from multiprocessing.managers import BaseManager
import queue

q = queue.Queue()
class QueueManager(BaseManager): pass
QueueManager.register('get_queue', callable=lambda: q)

# Listen on all interfaces at port 50000
m = QueueManager(address=('', 50000), authkey='abracadabra')
s = m.get_server()
s.serve_forever()

# --- CLIENT SIDE ---
from multiprocessing.managers import BaseManager

class QueueManager(BaseManager): pass
QueueManager.register('get_queue')

# Connect to the server
m = QueueManager(address=('foo.bar.org', 50000), authkey='abracadabra')
m.connect()
queue = m.get_queue()
queue.put('hello')

If you are creating a frozen Windows executable (using tools like pyinstaller, py2exe, or cx_Freeze) that uses multiprocessing, you must call freeze_support() immediately after the if __name__ == '__main__': block. Failure to do so will result in a RuntimeError when the executable runs.

On a normal Python interpreter, freeze_support() has no effect.

from multiprocessing import Process, freeze_support

def f():
    print('hello world!')

if __name__ == '__main__':
    freeze_support()
    Process(target=f).start()