Using async generators as data streams

One of the most recent additions to ECMAScript/JavaScript are async generators. In case you’re not already familiar with them, generators are basically special functions which can ‘return’ multiple times using a yield statement like this:

function* makeRangeGenerator(start, end) {
let n = 0;
for (let i = start; i < end; i++) {
n += 1;
yield i;
}
return n;
}
let rangeGenerator = makeRangeGenerator(0, 3);
console.log(rangeGenerator.next()); // { value: 0, done: false }
console.log(rangeGenerator.next()); // { value: 1, done: false }
console.log(rangeGenerator.next()); // { value: 2, done: false }
console.log(rangeGenerator.next()); // { value: 3, done: true }

Generator functions are different from regular functions in at least two fundamental ways:

  • A generator function can yield (~return) values multiple times without exiting and losing its internal state.
  • A caller can push values back into the generator function at the point where it last yielded — the generator will resume execution from that point until it encounters the next yield.

Note that you can also iterate over the output of a generator using a for-of loop like this:

for (let integer of rangeGenerator) {
console.log(integer);
}

^ This is slightly cleaner than using rangeGenerator.next() — In this case our integer variable will be the raw value instead of the object {value: …, done: false}.

Sometimes, we may want to pass values into a generator like this:

function* pushPullGenerator() {
let n = 0;
while (true) {
n = yield n * 2;
}
}
let gen = pushPullGenerator();
console.log(gen.next());  // { value: 0, done: false }
console.log(gen.next(2)); // { value: 4, done: false }
console.log(gen.next(3)); // { value: 6, done: false }
console.log(gen.next(4)); // { value: 8, done: true }

^ Here we’re passing values to the generator using gen.next(…) calls. Note that the first invocation of gen.next() doesn’t have any arguments; this is because the first yield in the generator is different from the others; it behaves more like a plain return statement. Subsequent calls to yield, however, will both push a value out and then pull a new value into the generator function (at the point where the yield occurred).

Coming up with use cases for generators might not be obvious at first. If overused, generators could make code more difficult to follow. However, if used correctly, generators can offer some unique capabilities. In this guide, we will look at a specific kind of generator; the async generator.

Async generators (as in async/await) are like regular generators except that instead of yielding normal values, they yield Promise objects which resolve asynchronously. This allows us to do some useful things; in particular, async generators are ideal for representing asynchronous streams of data.

If you’ve used RxJS Observables before, you should already be familiar with the idea of consuming asynchronous streams of data in a declarative (reactive) way. Async generators can be used as an alternative to Observables; one of the main benefits of using async generators over Observables is that they will make your code look more like regular synchronous JavaScript; this should make it more succinct and readable.

Here is an example of an async generator which streams some integers:

function waitForDelay(delay) {
return new Promise((resolve) => {
setTimeout(() => {
resolve();
}, delay);
});
}
async function* createRandomStream(randomness) {
let n = 0;
while (true) {
let randomDelay = Math.round(Math.random() * randomness);
await waitForDelay(randomDelay);
yield n++;
}
}
async function startConsumingStream() {
let randomStream = createRandomStream(1000);
for await (let value of randomStream) {
console.log(value);
}
}
startConsumingStream();

^ The randomStream in this case is a stream of integers which increments 0, 1, 2, 3, 4, … — it’s only random in the sense that the delay period between each iteration of the generator will be a random value between 0 and 1000 milliseconds; the result is that you can see the logging of numbers accelerate and decelerate randomly. Note that, in this case, we’ve used a for-await-of loop to iterate over the async generator; this is the async equivalent of the for-of loop which we used earlier to iterate over a normal generator. As a challenge; you could try to modify the code above to use a trigonometric function like Math.sin(…) to generate the delay value; it shouldn’t be too difficult.

Random streams are fun, but what about something more useful? For example; maybe we want each iteration of our generator to correspond to an event; like receiving a message from a user.

We can do something like this:

let triggerMessageReceived = () => {};
function waitForNextMessage() {
return new Promise((resolve) => {
triggerMessageReceived = resolve;
});
}
async function* messageStream() {
while (true) {
yield await waitForNextMessage();
}
}
async function startConsumingMessageStream() {
let randomStream = messageStream();
for await (let value of randomStream) {
console.log(value);
}
}
startConsumingMessageStream();
setTimeout(() => {
triggerMessageReceived('Hello');
}, 500);
setTimeout(() => {
triggerMessageReceived('world');
}, 1000);
setTimeout(() => {
triggerMessageReceived('!!!');
}, 3000);

^ The trick here is to assign the resolve function from the Promise in the waitForNextMessage function to our own triggerMessageReceived function; that way it can be invoked anytime from anywhere else in our code — the message in this case is triggered by a setTimeout but it could also have come from any other source such as an HTTP request or a WebSocket message… Note that because we’re using async/await, it doesn’t cost any CPU to wait indefinitely.

One more thing…

For real-world use cases, you may want to wrap your async generator streams into proper OOP objects; in that case, you may want to expose your async generators from Iterable objects.


Using async generators as data streams was originally published in Hacker Noon on Medium, where people are continuing the conversation by highlighting and responding to this story.

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: