Node.js 如何获取需要加载cheerio的网站部分数据

这是一个Reaction/Next.js应用程序，这意味着数据不在axios请求的静态HTML中，而是在页面加载后由JS添加到DOM中.单页应用程序(SPA)的数据通常通过API端点进入，如果不安全，您通常可以直接访问该端点.

在本例中，数据是(幸运的)<script id="__NEXT_DATA__">，JS在页面加载后使用它来创建您在dev工具中看到的可见元素.您可以按如下方式获取数据:

const axios = require("axios");
const cheerio = require("cheerio");
require("util").inspect.defaultOptions.depth = null;

const url = "<Your URL>";

axios.get(url).then(response => {
  const html = response.data;
  const $ = cheerio.load(html);
  const payload = $("#__NEXT_DATA__").first().text();
  const {data} = JSON.parse(JSON.parse(payload).props.initialState)
    .cryptocurrency.listingLatest;
  console.log(data);
});

该 struct 是压缩的，没有标头.如果希望将标头映射到数据以使其更具可读性，则可以:

const payload = $("#__NEXT_DATA__").first().text();
const {data} = JSON.parse(
  JSON.parse(payload).props.initialState
).cryptocurrency.listingLatest;
const [{keysArr}, ...rest] = data;
const withKeys = rest.map(e =>
  Object.fromEntries(
    e.map((e, i) => [keysArr[i] ?? "unknown", e])
  )
);
console.log(withKeys.slice(0, 10));

现在，下面的代码显示了与站点前几列类似的数据:

const summary = withKeys.map(e => ({
  "id": e.id,
  "name": e.name,
  "symbol": e.symbol,
  "price": e["quote.USD.price"],
  "1h": e["quote.USD.percentChange1h"],
  "24h": e["quote.USD.percentChange24h"],
  "marketCap": e["quote.USD.marketCap"],
}));
console.log(summary);
console.log(summary.length); // => 100

输出:

[
  {
    id: 1,
    name: 'Bitcoin',
    symbol: 'BTC',
    price: 28422.366435538406,
    '1h': -0.02014955,
    '24h': 5.28633725,
    marketCap: 549443765021.2035
  },
  {
    id: 1027,
    name: 'Ethereum',
    symbol: 'ETH',
    price: 1809.5420505966479,
    '1h': 0.00783376,
    '24h': 3.55526047,
    marketCap: 221440656815.19766
  },
  {
    id: 825,
    name: 'Tether',
    symbol: 'USDT',
    price: 0.9998792896202411,
    '1h': -0.00074249,
    '24h': -0.02658359,
    marketCap: 79511295635.83586
  },
  // ...
]