Node.js 在 Nodejs 中解析大型 JSON 文件

要逐行处理文件，只需将文件的读取与作用于该输入的代码分离.你可以通过缓冲你的输入直到你到达一个换行符来实现这一点.假设每行有一个JSON对象(基本上是格式B):

var stream = fs.createReadStream(filePath, {flags: 'r', encoding: 'utf-8'});
var buf = '';

stream.on('data', function(d) {
    buf += d.toString(); // when data is read, stash it in a string buffer
    pump(); // then process the buffer
});

function pump() {
    var pos;

    while ((pos = buf.indexOf('\n')) >= 0) { // keep going while there's a newline somewhere in the buffer
        if (pos == 0) { // if there's more than one newline in a row, the buffer will now start with a newline
            buf = buf.slice(1); // discard it
            continue; // so that the next iteration will start with data
        }
        processLine(buf.slice(0,pos)); // hand off the line
        buf = buf.slice(pos+1); // and slice the processed data off the buffer
    }
}

function processLine(line) { // here's where we do something with a line

    if (line[line.length-1] == '\r') line=line.substr(0,line.length-1); // discard CR (0x0D)

    if (line.length > 0) { // ignore empty lines
        var obj = JSON.parse(line); // parse the JSON
        console.log(obj); // do something with the data here!
    }
}

每次文件流从文件系统接收数据时，都会将其存储在缓冲区中，然后调用pump.

如果缓冲区中没有换行符，pump将直接返回，而不做任何操作.下次流获取数据时，更多的数据(可能还有一个换行符)将被添加到缓冲区，然后我们将拥有一个完整的对象.

如果有换行符，pump从开始到换行符将缓冲区切掉，并将其交给process.然后再次判断缓冲区中是否有另一个换行符(while循环).通过这种方式，我们可以处理当前块中读取的所有行.

Finally, process is called once per input line. If present, it strips off the carriage return character (to avoid issues with line endings – LF vs CRLF), and then calls JSON.parse one the line. At this point, you can do whatever you need to with your object.

注意，JSON.parse对于它接受的输入是严格的；必须引用标识符和字符串值.换句话说，{name:'thing1'}将抛出一个错误；你必须使用{"name":"thing1"}.

因为一次内存中的数据不会超过一块，所以这将是非常高效的内存.它也会非常快.快速测试显示我在15毫秒内处理了10000行.