A section of my Node.js application involves receiving a string as input from the user and storing it in a JSON file. JSON itself obviously has no limit on this, but is there any upper bound on the amount of text that Node can process into JSON?

Note that I am not using MongoDB or any other technology for the actual insertion - this is native stringification and saving to a .json file using fs.

推荐答案

V8 (the JavaScript engine node is built upon) until very recently had a hard limit on heap size of about 1.9 GB.

Node v0.10 is stuck on an older version of V8 (3.14) due to breaking V8 API changes around native addons. Node 0.12 will update to the newest V8 (3.26), which will break many native modules, but opens the door for the 1.9 GB heap limit to be raised.

So as it stands, a single node process can keep no more than 1.9 GB of JavaScript code, objects, strings, etc combined. That means the maximum length of a string is under 1.9 GB.

You can get around this by using Buffers, which store data outside of the V8 heap (but still in your process's heap). A 64-bit build of node can pretty much fill all your RAM as long as you never have more than 1.9 GB of data in JavaScript variables.


All that said, you should never come anywhere near this limit. When dealing with this much data, you must deal with it as a stream. You should never have more than a few megabytes (at most) in memory at one time. The good news is node is especially well-suited to dealing with streaming data.

你应该问自己几个问题:

  • What kind of data are you actually receiving from the user?
  • 为什么要以JSON格式存储它?
  • 把千兆字节塞进JSON真的是个好主意吗?(答案是否定的.)
  • What will happen with the data later, after it is stored? Will your code read it? Something else?

你发布的问题实际上是相当模糊的,与你真正想要实现的目标有关.要获得更具体的建议,请使用更多信息更新您的问题.

如果您希望数据永远不会那么大,只需在输入上抛出10MB或更大的合理限制,将其全部缓冲,然后使用JSON.stringify.

If you expect to deal with data any larger, you need to stream the input straight to disk. Look in to transform streams if you need to process/modify the data before it goes to disk. For example, there are modules that deal with streaming JSON.

Json相关问答推荐

使用相同的密钥值来命名Json并使用Jolt重命名密钥

JOLT将对象名作为新属性添加到主体中

Vega图表计数聚合如果数据值为空数组则不显示任何内容,如何解决此问题?

NIFI-我需要数组的信息,但只需要第一个信息

Allof Indide的JSON模式之一

VBA json按特定属性名称提取所有数据

如何在Swift中使用JSON编码器的泛型

如何使用PowerShell从ExchangeOnline命令执行中获得JSON输出

GitHub Pages无法正确显示我的项目

如何使用jq按键 Select 并获取整个json输出来更改json中的多个值

如何在循环中传递列表(会话(字符串属性))以在 Gatling Scala 中创建批量 Json 请求

为什么根据其他工具,来自 aws rds 的 JSON 在 Docker 中格式错误运行?

如何在 Dart 中与多个 map (字典)相交

如何在 onClick 事件处理程序中识别在同一 map 上绘制的多个多边形中的哪个(使用 react-leaflet)被单击?

如何解决名为 null 的map值

如何使用 gson 调用默认反序列化

使用 @ResponseBody 自定义 HttpMessageConverter 来做 Json 事情

[__NSCFNumber 长度]:发送到实例 UITableView 的无法识别的 Select 器

有没有办法在 angular.json 中扩展配置?

使用新的 Net Core 3.0 Json 时忽略属性