Learn advanced node js, Backpressure

Sometimes the data coming from a readable stream is too fast for a writable stream to handle. Using our water bucket example, imagine pouring water into the hose. So we’re pouring water from our source bucket to our destination. Now let’s say we’re pouring the water too fast and the hose begins to back up, what should we do, should we keep pouring the water in? Well doing that will cause an overflow and we’ll loose all of our water. Instead what we actually want to do is stop pouring water in until it’s drained. So here we have the hose, it’s full, our source has stopped pouring water in until all that water can make it to the destination and then we’ll start pouring water in from our source again.

And again once our hose has been full we’ll stop pouring the water in and we’ll repeat this process until all of the water pipes through the hose from our source to our destination. Whenever we actually have a full hose, this is referred to as back pressure, and how much water our hose can handle is referred to as the high water mark. By setting a large high water mark essentially what you’re doing is creating a very big hose that can handle all of the water that’s passed to it. Let’s take a look at this in a code example.

In the last lesson we built a writable stream that we can use to actually copy the data from powder day to our copy. And what we have going on here is a little bit of back pressure. We can’t see that it’s happening in the terminal but if we were to take a look at memory we could see back pressure causing problems. What we’re going to do is actually handle that. I’m inside of our exercise files, I’m in chapter two, lesson five looking at the start folder and here’s the file that we created in the last lesson. Our writestream.write method will let us know whether the hose is full or not.

It will return a true or false value as to whether it can handle more data. I can capture that value as a result. If this result is false that means that our hose is full. What I want to do is say if we don’t have a result then our hose is full so I need to tell our read stream to stop pouring water in it. So we can pause the read stream so we’re not going to actually read any more data until our hose drains. We’ll do a little console log here just so we can see that we have back pressure.

So how do we know when to start pouring data in again? Well we can listen for a drain event on the write stream. When our write stream drains this function will fire and we can handle that event with this callback. Console.log, drain, and we’ll also tell our read stream to resume. This process will continue until all of the data has been passed from our read stream to our write stream.

Let’s take a look. So I’m coming over here to the terminal and we’ll type node dot. And what you’ll notice is we have a lot of back pressure. Every time there was back pressure instead of creating more memory or taking up more memory now we simply pause the read stream until the write steam can handle that much data. Now when we create the write stream we have the ability to decide how thick our hose is. We can do that by setting the high water mark. In the options right after the file I can send an object of options and I’m going to set the high water mark.

Now if I set this to some arbitrary number then we can allow for a larger hose so that’s how many bites we’ll have in our high water mark or how many bits we’ll have. Let’s go ahead and try to run this again. So node dot, so because we have a larger hose we don’t have as much back pressure. Again if I were to set this very large we won’t see any back pressure at all because our internal buffers for the write stream have enough space to handle all of the video.

Again without setting this high water mark we don’t want to take up too much memory with our application so we can handle that through handling back pressure. That means pausing our read stream until our write stream is ready to handle more data.

Leave a Reply

Notify of