Bastibeckr

Recent Blog Comments By Bastibeckr

• Using Transform Streams To Manage Backpressure For Asynchronous Tasks In Node.js

  Posted on Mar 12, 2017 at 11:32 AM

  @Ben, This was my first contact to ndjson and googled "node read huge file line by line". I knew I would run into memory issues if I try to parse all the data at once. And if I would use standard JSON-arrays, one parse error would destroy the whole file. I like the concept of one-object-... read more »

• Using Transform Streams To Manage Backpressure For Asynchronous Tasks In Node.js

  Posted on Mar 11, 2017 at 3:11 PM

  Very elegant solution. I had to deal with large amounts (10s of GB) ndjson data recently. The data came from elasticsearch and had to be inserted into mongodb. I ended up with Node's readline - interface. On each 'line' event, I paused the stream, parsed the JSON, transform some of the data, upser... read more »