Coroutines

Notes based on Fluent Python, summarized and rephrased in my own words.[file:24]

From generators to coroutines

The verb "to yield" has two meanings: produce and give way. For Python generators, both are true:[file:24]

• yield item produces a value to the caller (e.g. next(gen)), and
• it gives way by pausing the generator and returning control to the caller until the next request.[file:24]

Classic generators are pull-based: the caller repeatedly pulls values by calling next().

Coroutines use the same yield keyword but in a different role:

• yield often appears on the right-hand side of an assignment, e.g. data = yield.[file:24]
• A coroutine can both yield values and receive values sent from the caller using .send(value).[file:24]
• In practice, the caller "pushes" data into the coroutine.[file:24]

yield can also be used without sending or receiving data; in all cases it is a control-flow mechanism that enables cooperative multitasking: coroutines voluntarily yield control so that other coroutines can run.[file:24]

Key evolutions that enable modern coroutines:[file:24]

• Callers can use .send(value) to push data into a generator; the value becomes the result of the yield expression inside the generator. This is what lets a generator behave like a coroutine.
• Generators can now return value; the value is carried in the StopIteration.value attribute when the generator finishes.[file:24]
• yield from syntax lets a generator delegate work to subgenerators with far less boilerplate.[file:24]

Basic behavior of a generator used as a coroutine

A minimal coroutine:

def simple_coroutine():
    print('-> coroutine started')
    x = yield
    print('-> coroutine received:', x)

Usage:[file:24]

my_coro = simple_coroutine()

next(my_coro)       # prime: runs until the first `yield`
# -> coroutine started

my_coro.send(42)
# -> coroutine received: 42
# raises StopIteration when the function body ends

Important points:[file:24]

• Before the first yield, the coroutine is in state GEN_CREATED. You must prime it by calling next(coro) or coro.send(None) so it runs to the first yield.
• After priming, the coroutine is GEN_SUSPENDED at a yield expression and ready to receive data via .send(value).
• When control reaches the end of the function body, the generator is GEN_CLOSED and further .send or next calls raise StopIteration.[file:24]

A slightly richer example:

from inspect import getgeneratorstate


def simple_coro2(a):
    print('-> Started: a =', a)
    b = yield a
    print('-> Received: b =', b)
    c = yield a + b
    print('-> Received: c =', c)


my_coro2 = simple_coro2(14)
getgeneratorstate(my_coro2)        # 'GEN_CREATED'

next(my_coro2)                     # prints, returns 14
getgeneratorstate(my_coro2)        # 'GEN_SUSPENDED'

my_coro2.send(28)                  # prints, returns 42
my_coro2.send(99)                  # prints, then StopIteration
getgeneratorstate(my_coro2)        # 'GEN_CLOSED'

Overall picture: yield exposes the expression’s value to the caller, and .send(value) provides a value back into the suspended yield expression so the coroutine can continue.[file:24]

Example: computing a running average with a coroutine

We can implement a moving average using a coroutine instead of a closure:[file:24]

def averager():
    total = 0.0
    count = 0
    average = None
    while True:
        term = yield average
        total += term
        count += 1
        average = total / count

Notes:[file:24]

• This is an infinite loop: as long as the caller keeps sending values, the coroutine keeps updating the average.
• It only stops when the caller calls .close() or when the coroutine becomes unreachable and is garbage-collected.

Usage:[file:24]

coro_avg = averager()
next(coro_avg)          # prime coroutine; returns initial None

coro_avg.send(10)       # 10.0
coro_avg.send(30)       # 20.0
coro_avg.send(5)        # 15.0

Each .send(value) call resumes execution at term = yield average, where term becomes the sent value.[file:24]

Priming coroutines with a decorator

Because coroutines generally must be primed before first use, it’s convenient to hide this step behind a decorator:[file:24]

from functools import wraps


def coroutine(func):
    """Decorator: automatically prime a generator-based coroutine."""
    @wraps(func)
    def primer(*args, **kwargs):
        gen = func(*args, **kwargs)
        next(gen)
        return gen
    return primer

Applying it:[file:24]

@coroutine
def averager():
    total = 0.0
    count = 0
    average = None
    while True:
        term = yield average
        total += term
        count += 1
        average = total / count

from inspect import getgeneratorstate
getgeneratorstate(averager())   # 'GEN_SUSPENDED' — already primed

Now users can call averager() directly and start sending values without manually calling next() first.[file:24]

Terminating coroutines and handling exceptions

An unhandled exception inside a coroutine bubbles up to the caller that invoked next() or .send().[file:24]

coro_avg = averager()
next(coro_avg)

coro_avg.send(40)    # 40.0
coro_avg.send(50)    # 45.0

coro_avg.send('spam')  # TypeError inside the coroutine
# further sends raise StopIteration because the coroutine is closed
coro_avg.send(60)      # StopIteration

From the coroutine’s perspective:[file:24]

• If it does not catch an exception, it terminates immediately.
• After termination (GEN_CLOSED), it cannot be resumed.

A common pattern is to designate a sentinel value that signals termination, such as None, Ellipsis, or a special marker type.[file:24]

Injecting exceptions with throw() and closing with close()

Generator objects support two extra methods to interact with coroutines:[file:24]

• generator.throw(exc_type[, exc_value[, traceback]])

  • Causes the coroutine to raise the given exception at the suspended yield.

• generator.close()

  • Causes a GeneratorExit to be raised at the suspended yield.

You can catch these inside the coroutine to perform custom cleanup or logging.

Example of handling a custom exception:[file:24]

class DemoException(Exception):
    """Custom exception type for this demo."""


def demo_exc_handling():
    print('-> coroutine started')
    while True:
        try:
            x = yield
        except DemoException:
            print('*** DemoException handled. Continuing...')
        else:
            print('-> coroutine received: {!r}'.format(x))
    # This line is never reached because only an unhandled exception
    # breaks the loop, and such an exception terminates the coroutine.

Usage:[file:24]

exc_coro = demo_exc_handling()
next(exc_coro)          # start

exc_coro.send(11)       # prints received 11
exc_coro.send(22)       # prints received 22

exc_coro.throw(DemoException)  # handled inside, coroutine continues

exc_coro.close()        # triggers cleanup if implemented

If you throw an exception the coroutine does not handle, it terminates and the exception propagates to the caller:[file:24]

exc_coro = demo_exc_handling()
next(exc_coro)
exc_coro.send(11)
exc_coro.throw(ZeroDivisionError)  # not handled; coroutine closes

You can use a try/finally block inside the coroutine to guarantee cleanup at termination:[file:24]

class DemoException(Exception):
    pass


def demo_finally():
    print('-> coroutine started')
    try:
        while True:
            try:
                x = yield
            except DemoException:
                print('*** DemoException handled. Continuing...')
            else:
                print('-> coroutine received: {!r}'.format(x))
    finally:
        print('-> coroutine ending')

Returning values from coroutines

Since Python 3.3, generators can return value. In generator-based coroutines, this is used to send a final result back to the caller when the coroutine ends normally.[file:24]

Example:

from collections import namedtuple
Result = namedtuple('Result', 'count average')


def averager():
    total = 0.0
    count = 0
    average = None
    while True:
        term = yield
        if term is None:
            break    # normal termination
        total += term
        count += 1
        average = total / count
    return Result(count, average)

Usage:[file:24]

coro_avg = averager()
next(coro_avg)

coro_avg.send(10)
coro_avg.send(30)
coro_avg.send(6.5)

try:
    coro_avg.send(None)
except StopIteration as exc:
    result = exc.value

print(result)  # Result(count=4, average=14.125)

Key idea:[file:24]

• The coroutine does not yield the final result.
• Instead, the value passed to return is attached to the StopIteration exception object as .value.
• The caller must catch StopIteration to retrieve the result (unless using yield from, which does this automatically).

yield from and delegation to subgenerators

yield from does more than flatten nested loops—it creates a two-way channel between the outermost caller and the innermost subgenerator.[file:24]

Simple flattening examples:[file:24]

def gen():
    for c in 'AB':
        yield c
    for i in range(1, 3):
        yield i

# With `yield from`:

def gen():
    yield from 'AB'
    yield from range(1, 3)


def chain(*iterables):
    for it in iterables:
        yield from it

list(chain('ABC', range(3)))   # ['A', 'B', 'C', 0, 1, 2]

When you write yield from x, Python first calls iter(x) to get an iterator, then:

• Forwards values from the subiterator to the caller.
• Forwards .send() and .throw() calls from the caller into the subgenerator.
• Captures the StopIteration.value from the subgenerator and uses it as the value of the yield from expression.[file:24]

Terminology:[file:24]

• Delegating generator: the generator that contains yield from <iterable>.
• Subgenerator: the generator produced from <iterable>.
• Caller: the external code that drives the delegating generator.

The main benefit: yield from lets you decompose complex generator-based coroutines into smaller, composable pieces without writing a lot of boilerplate to pass values and exceptions back and forth.[file:24]

Example: delegating to a subgenerator with yield from

Consider again the averaging coroutine, this time as a subgenerator:[file:24]

from collections import namedtuple
Result = namedtuple('Result', 'count average')


def averager():   # subgenerator
    total = 0.0
    count = 0
    average = None
    while True:
        term = yield
        if term is None:
            break
        total += term
        count += 1
        average = total / count
    return Result(count, average)

A delegating generator can use yield from to run averager and capture its result:[file:24]

def grouper(results, key):   # delegating generator
    while True:
        results[key] = yield from averager()

Finally, a caller drives the whole process:[file:24]

def main(data):
    results = {}
    for key, values in data.items():
        group = grouper(results, key)
        next(group)            # prime the grouper
        for value in values:
            group.send(value)
        group.send(None)       # signal end of this group
    report(results)


def report(results):
    for key, result in sorted(results.items()):
        group, unit = key.split(';')
        print('{:2} {:5} averaging {:.2f}{}'.format(
              result.count, group, result.average, unit))

With a dataset like this:[file:24]

data = {
    'girls;kg': [40.9, 38.5, 44.3, 42.2, 45.2, 41.7, 44.5, 38.0, 40.6, 44.5],
    'girls;m':  [1.6, 1.51, 1.4, 1.3, 1.41, 1.39, 1.33, 1.46, 1.45, 1.43],
    'boys;kg':  [39.0, 40.8, 43.2, 40.8, 43.1, 38.6, 41.4, 40.6, 36.3],
    'boys;m':   [1.38, 1.5, 1.32, 1.25, 1.37, 1.48, 1.25, 1.49, 1.46],
}

if __name__ == '__main__':
    main(data)

The output summarizes averages per group and unit.[file:24]

What yield from does here:[file:24]

• Forwards all sent values (except the sentinel None) directly into averager.
• Receives the Result from averager when it returns and stores it in results[key].
• Lets the caller interact with averager as if it were talking directly to it, even though communication passes through grouper.

Semantics of yield from (high level)

At a conceptual level, yield from subgen:

• Forwards all values yielded by the subgenerator to the caller.
• Forwards values sent by the caller (via .send()) to the subgenerator.
• When the subgenerator terminates with return value, catches the StopIteration(value) and uses value as the value of the yield from expression.
• Forwards most exceptions from the caller down to the subgenerator via .throw().
• Handles GeneratorExit and .close() by closing the subgenerator appropriately.[file:24]

The result is a clean delegation mechanism that composes complex coroutine workflows out of simpler building blocks.