Someday you will need to import a large file to mongo or some other DBMS ,
\nYou start writing code for achieving same using NodeJS, but when you run memory usage start increasing and either core of your CPU is at 100%.<\/p>\n
<\/p>\n
var fs = require('fs'),\r\n readline = require('readline'),\r\n stream = require('stream'),\r\n instream = fs.createReadStream(\"SomeFile\"),\r\n k,\r\n buffer = [],\r\n c = 0,\r\n sCount = 0,\r\n lines = 0;\r\nconsole.log(\"Importing...\");\r\nrl.on('line', function (line) {\r\n ++lines;\r\n k = line.split(\",\");\r\n\r\n buffer.push({p:k[0],q:k[1]});\r\n c++;\r\n\r\n if (c == 100000) {\r\n rl.pause(); \/\/Pause the reader this won't help, as till system pauses some lines are already read.\r\n insertDocument(buffer, function () {\r\n \r\n sCount += buffer.length;\r\n buffer = []; \/\/clear buffer\r\n c = 0;\r\n rl.resume();\r\n })\r\n }\r\n\r\n});\r\n\r\nrl.on('close', function () {\r\n insertDocument(buffer, function () {});\r\n console.log(\"\\nProcessed [ Lines:\" + lines + \"]\");\r\n});\r\n<\/pre>\n\n- Since nodejs is Async language you start getting `line` events without pausing for previous event to finish, You might get 10000 lines per second, depends on Disk Activity.<\/li>\n
- Now we created a buffer so that less calls should be made to database, but still till the database processes request node may push same buffer 100 times for processing.<\/li>\n
- To solve the above problem you might think pausing the reader till it records are not processed, Cool !, but it won’t work as expected because readline doesn’t pauses immediately, some events are still fired, so you might miss those – https:\/\/nodejs.org\/api\/readline.html#readline_rl_pause<\/a><\/li>\n
- You might think to add a delay of few ms, but it isn’t scalable solution, Bit easy solution is to create another buffer and use a bit to indicate start of insertion operation, events after that can be processed into new buffer and when database finished change the bit copy the buffer.<\/em><\/li>\n<\/ol>\n
A better solution would be making async call. Mongo (MONGOIMPORT), Mysql (MYSQLDUMP) have inbuilt functions to process large CSV file
\n<\/strong><\/p>\nmongoimport --db DATABASE --collection COLLECTION_NAME --type csv --headerline --file SOME_LARGE.csv<\/code><\/p>\nit works efficiently with a nice CLI showing progress.<\/p>\n","protected":false},"excerpt":{"rendered":"
Someday you will need to import a large file to mongo or some other DBMS , You start writing code for achieving same using NodeJS, but when you run memory usage start increasing and either core of your CPU is at 100%.<\/p>