aioitertools

aioitertools: itertools and more for AsyncIO

aioitertools provides asynchronous versions of the standard library's itertools functions, designed for use with asyncio. It allows you to perform complex iterator manipulations (like grouping, combinations, and permutations) on asynchronous iterables without blocking the event loop.

aioitertools emulates the standard itertools module by offering the same function signatures but implemented as async generators. These functions support both standard iterables and async iterables as inputs, and can handle both regular functions and coroutines.

from aioitertools import chain, islice

async def generator1():
    yield 1
    yield 2

async def generator2():
    yield 3
    yield 4

# chain combines async generators
async for value in chain(generator1(), generator2()):
    print(value)

# islice provides slicing for async iterables
async for value in islice(generator1(), 2, None, 2):
    print(value)

aioitertools shadows standard library builtins to provide asynchronous versions of familiar functions. It is designed to be fully compatible with both standard iterables and async iterators, providing a unified interface. You can use functions like iter, next, map, and zip to interact with mixed-type iterable objects asynchronously.

from aioitertools import iter, next, map, zip

something = iter(...) 
# 'something' can be a standard iterable or an async iterator

first_item = await next(something)

async for item in iter(something):
    ...

# map works with coroutines
async def fetch(url):
    # ... implementation ...
    return data

async for value in map(fetch, MANY_URLS):
    ...

# zip works with mixed types
async for a, b in zip(something, something_else):
    ...

aioitertools requires Python 3.9 or newer. You can install it from PyPI using pip.

$ pip install aioitertools

Use gather_iter(itr, return_exceptions=False, limit=-1) to gather results from an iterable instead of passing them as positional arguments (*args).

This is a convenience wrapper around gather(). The values within the iterable do not necessarily have to be awaitable; they will be handled via maybe_await internally.

# Example usage pattern
results = await gather_iter(my_async_iterable, limit=5)

Yield values from the iterable until the predicate evaluates to False. The predicate can be a standard function or a coroutine.

def pred(x):
    return x < 4

async for item in takewhile(pred, range(8)):
    print(item)  # 0, 1, 2, 3

Use as_generated(iterables, return_exceptions=False) to consume multiple async iterables (like async generators) concurrently, yielding values in the order they are produced.

It creates a separate task for each iterable and uses a single queue to intermix the results.

Error Handling:

• If return_exceptions=False (default): Any exception raised by an underlying iterable will be raised immediately, cancelling all pending iterables and tasks.
• If return_exceptions=True: Exceptions are yielded as results, and execution continues until all iterables are fully consumed.

async def generator(x):
    for i in range(x):
        yield i

gen1 = generator(10)
gen2 = generator(12)

async for value in as_generated([gen1, gen2]):
    # intermixed values yielded from gen1 and gen2
    ...

The zip(*itrs) function yields tuples of items from the provided mixed iterables. It continues yielding until the shortest iterable is exhausted. It uses asyncio.gather to fetch the next item from all iterables concurrently.

async for a, b, c in zip(i, j, k):
    ...

These functions consume a mixed iterable and return a standard Python collection. Because they consume the entire iterable, they must be awaited.

• await list(itr): Returns a list of items.
• await tuple(itr): Returns a tuple of items.
• await set(itr): Returns a set of items.

# Examples
await list(range(5))    # [0, 1, 2, 3, 4]
await tuple(range(5))   # (0, 1, 2, 3, 4)
await set([1, 2, 2])   # {1, 2}

Yield a tuple of items from mixed iterables until all are consumed. If shorter iterables are exhausted, the fillvalue (defaults to None) will be used until all iterables are exhausted.

a = range(3)
b = range(5)

async for a_val, b_val in zip_longest(a, b, fillvalue=-1):
    print(a_val, b_val)  # (0, 0), (1, 1), (2, 2), (-1, 3), (-1, 4)

Both functions accept either standard functions or coroutines as predicates.

• dropwhile(predicate, iterable): Drops all items until the predicate evaluates False, then yields all remaining items.
• filterfalse(predicate, iterable): Yields items only when the predicate evaluates to False.

# dropwhile
def pred(x):
    return x < 4

async for item in dropwhile(pred, range(6)):
    print(item)  # 4, 5, 6

# filterfalse
async for item in filterfalse(pred, range(6)):
    print(item)  # 4, 5