"modern" because that definition may change over time (and specifically I mean desktop browsers)

"handle" because that may vary depending on machine configurations/memory, but specifically I mean a general use case.


这个问题出现在我试图解决的一个涉及大型数据集的特殊问题上.

Essentially, whenever a change is made to a particular dataset I get the full dataset back and I have to render this data in the browser.

例如,在一个websocket上,我得到一个push事件,它告诉我一个数据集有变化,然后我必须用HTML呈现这个数据集,方法是抓取一个现有的DOM元素,复制它,使用类名或其他元素标识符用这个集合中的数据填充元素,然后将它添加回DOM.

Keep in mind that any object (JSON) in this dataset may have as many as 1000+ child objects, and there may be as many as 10,000+ parent objects, so as you can see there may be an instance where the returned dataset is upwards towards 1,000,000 => 10,000,000 data points or more.

Now the fun part comes when I have to render this stuff. For each data point there may be 3 or 4 tags used to render and style the data, and there may be event listeners for any of these tags (maybe on the parent container to lighten things up using delegation).

总而言之,可能需要呈现大量传入的信息,我正试图找出处理这种情况的最佳方法.

理想情况下,您只希望呈现具有更改的单个数据点的更改,而不是重新呈现整个集合,但这可能不是一个选项,因为后端是如何设计的.

My main concern here is to understand the limitations of the browser/DOM and looking at this problem through the lense of the frontend. There are some changes that should happen on the backend for sure (data design, caching, pagination), but that isnt the focus here.

This isn't a typical use case for HTML/DOM, as I know there are limitations, but what exactly are they? Are we still capped out at about 3000-4000 elements?


I've got a number of related subquestions for this that I'm actively looking up but I thought it'd be nice to share some thoughts with the rest of the stackoverflow community and try to pool some information together about this issue.

What is "reasonable" amount of DOM elements that a modern browser can handle before it starts becoming slow/non-responsive?

How can I benchmark the number of DOM elements a browser can handle?

What are some 100 for handling large datasets that need to be rendered (besides pagination)?

Are templating frameworks like mustache and handlebars more performant for rendering html from data/json (on the frontend) than using jQuery or Regular Expressions?

推荐答案

这是一个只有在统计学上有头脑的答案才能准确而全面的问题.

Why

The appropriate equation is this, where N is the number of nodes, bytesN is the total bytes required to represent them in the DOM, the node index range is n ∈ [0, N), bytesOverhead is the amount of memory used for a node with absolute minimum attribute configuration and no innerHTML, and bytesContent is the amount of memory used to fill such a minimal node.

bytesN = ∑N (bytesContentn + bytesOverheadn)

The value requested in the question is the maximum value of N in the worst case handheld device, operating system, browser, and operating conditions. Solving for N for each permutation is not trivial. The equation above reveals three dependencies, each of which could drastically alter the answer.

Dependencies

  1. node 的平均大小取决于每个 node 中用于保存内容的平均字节数,例如UTF-8文本、属性名称和值或缓存信息.
  2. DOM对象的平均开销取决于管理每个文档的DOM表示的HTTP用户代理.W3C的Document Object Model FAQ条声明:"虽然所有DOM实现都应该是可互操作的,但它们在代码大小、memory demand和单个操作的性能上可能会有很大的差异."
  3. 可用于DOM表示的内存取决于默认使用的浏览器(根据浏览器手持设备供应商或用户的喜好而有所不同)、默认浏览器的用户覆盖、操作系统版本、手持设备的内存容量、常见后台任务和其他内存消耗.

The Rigorous Solution

可以运行测试来确定(1)和(2)手持设备上使用的每个常见http用户代理.可以通过配置web服务器的日志(log)机制来获得任何给定站点的用户代理的分布,如果默认情况下HTTP_user_代理不存在,则将其放置在日志(log)中,然后剥离日志(log)中除该字段以外的所有字段,并计算每个值的实例.

The number of bytes per character would need to be tested for both attributes values and UTF-8 inner text (or whatever the encoding) to get a clear pair of factors for calculating (1).

可用的记忆也需要在各种常见条件下进行测试,这本身就是一个重大的研究项目.

Select 的N的特定值必须为零才能处理实际的最坏情况,因此可以 Select 特定百分比的内容、 node struct 和运行时条件的典型情况.例如,可以使用某种形式的随机in situ(在正常环境条件下)研究对病例进行抽样,发现满足95%病例的N.

也许可以用上述方式测试一组 case ,并将结果放在表格中.这将是对您问题的直接回答.

我猜,一个受过良好教育的移动软件工程师,在flare上学习数学,尤其是统计学,需要五周的全职时间才能得到合理的结果.

A More Practical Estimation

One could guess the worst case scenario. With a few full days of research and a few proof-of-concept apps, this proposal could be refined. Absent of the time to do that, here's a good first guess.

考虑允许用于DOM的1千兆字节的蜂窝电话,因为正常操作条件使用4千兆字节中的3千兆字节用于上述目的.可以假设 node 的平均内存消耗如下所示,以获得大致的数字.

  • 2 bytes per character for 40 characters of inner text per node
  • 2 bytes per character for 4 attribute values of 10 characters each
  • 每个字符1个字节,用于4个属性名称,每个属性名称4个字符
  • 160 bytes for the C/C++ node overhead in the less efficient cases

In this case Nworst_case, the worst case max nodes,

= 1,024 X 1,024 X 1,024
  / (2 X 40  +  2 X 4 X 10  +  1 X 4 X 4  +  160)

= 3,195,660 . 190,476.

但是,如果完全可以避免的话,我不会在拥有300万个DOM node 的浏览器中构建文档.考虑采用下面更常见的做法.

Common Practice

最好的解决方案是远远低于N最糟糕的情况,并使用标准HTTP设计技术将 node 总数减少到可能的程度.

  • 减少任何给定页面上显示内容的大小和复杂性,这也提高了视觉和概念的清晰度.
  • 从服务器请求最少量的数据,使用窗口技术延迟尚不可见的内容,或以精心计划的方式平衡响应时间和内存消耗.
  • 使用异步调用来帮助实现上述极简主义.

Json相关问答推荐

当有2个嵌套数组时展平复杂的JSON

无法从MongoDB集合中检索正确的文档

重构JOLT代码以获得预期输出

使用 jolt 变换展平嵌套 JSON - 非常复杂

当并非所有子对象都有 Select 器字段时 Select

报告重复的对象键

使用 jq 获取所有嵌套键和值

在 postgres 14 中将记录转换为所需的 json 格式

Groovy JsonBuilder 在for循环中添加数组元素

在 JOLT 中重新排列值

阅读 JSON 正文 Firebase 云函数

如何在 Android Studio 中将 List 字段作为空列表[]返回?

Python Flask-Restful POST 不采用 JSON 参数

在 bash 中将 CSV 转换为 JSON

Rails 中奇怪的 JSON Javascript 问题

jQuery AJAX 和 JSON 格式

如何在 Perl 中将简单的哈希转换为 json?

如何使用 Jackson 注释从 HttpResponse 反序列化 JSON 对象?

无法将空值放入 JSON 对象

设置对象不是 JSON 可序列化的