Html 在R中从网页上的特定iframe中提取图形数据

从页面源代码中，您只能找到动态元素的占位符或锚点，例如，对于特定的iframe，有:

<div id="comp-iw3d16s21" class="comp-iw3d16s21 _xg6_p"></div>

所有这些动态元素都是由浏览器中的JavaScript呈现的，这个过程的一部分是由assets资源 管理器协调的，它将id-s(如comp-iw3d16s21)转换为特定页面版本的url-s.部分回应看起来像这样:

                "comp-iw3d16s21": {
                    "url": "https://www-immostat-com.filesusr.com/html/ae915e_6e3e7e9d19bc8d5af1f3e4b96ae5c686.html",
                    "translations": {
                        "title": "Embedded Content"
                    }
                }

从技术上讲，你可以自己提出这个请求，提取iframe个URL，尽管它包括35个参数，要让它在下一个页面发布中发挥作用看起来像是一个适当的挑战:

https://siteassets.parastorage.com/pages/pages/thunderbolt?appDefinitionIdToSiteRevision=%7B%2214bcded7-0066-7c35-14d7-466cb3f09103%22%3A%22855%22%7D&beckyExperiments=specs.thunderbolt.supportSpxInEEMappers%3Atrue%2Cspecs.thunderbolt.one_cell_grid_display_flex%3Atrue%2Cspecs.thunderbolt.MediaContainerAndPageBackgroundMapper%3Atrue%2Cspecs.thunderbolt.catharsis_theme_optimize_css%3Atrue%2Cspecs.thunderbolt.WRichTextSemanticClasses%3Atrue%2Cspecs.thunderbolt.ghostify_hidden_comps%3Atrue%2Cspecs.thunderbolt.edixIsInFirstFold%3Atrue%2Cspecs.thunderbolt.catharsis_theme%3Atrue%2Cspecs.thunderbolt.DatePickerPortal%3Atrue%2Cspecs.thunderbolt.native_css_mappers_popups%3Atrue%2Cspecs.thunderbolt.wowImageRelayout%3Atrue%2Cspecs.thunderbolt.useElementoryRelativePath%3Atrue%2Cspecs.thunderbolt.new_responsive_layout_render_all_breakpoints%3Atrue%2Cspecs.thunderbolt.mesh_css_catharsis%3Atrue%2Cspecs.thunderbolt.DDMenuMigrateCssCarmiMapper%3Atrue%2Cspecs.thunderbolt.responsiveShapeDividersPublic%3Atrue%2Cspecs.thunderbolt.compsMeasuresCss_catharsis%3Atrue%2Cspecs.thunderbolt.customElemCollapsedheight%3Atrue%2Cspecs.thunderbolt.url_hierarchy%3Atrue%2Cspecs.thunderbolt.scaleprop%3Atrue%2Cspecs.thunderbolt.interactionsOverrides%3Atrue%2Cspecs.thunderbolt.displayRefComponentsAsBlock%3Atrue%2Cspecs.thunderbolt.pinned_layout_css_catharsis%3Atrue%2CuseTranslatedUrlSlugs%3Atrue%2Cspecs.thunderbolt.responsiveLayout_optimize_css%3Atrue%2Cspecs.thunderbolt.theme_fonts_colors_catharsis%3Atrue%2Cspecs.thunderbolt.catharsis_fontFaces%3Atrue&contentType=application%2Fjson&deviceType=Desktop&dfCk=6&dfVersion=1.2684.0&disableStaticPagesUrlHierarchy=false&editorName=Unknown&experiments=bv_remove_add_chat_viewer_fixer%2Cdm_linkTargetDefaults%2Cdm_removePageDataUnderTranslations%2Cdm_runTranslationsPageUriSeoFixer&externalBaseUrl=https%3A%2F%2Fwww.immostat.com&fileId=5745cace.bundle.min&formFactor=desktop&hasTPAWorkerOnSite=false&isConsentPolicyActive=true&isHttps=true&isInSeo=false&isMultilingualEnabled=false&isPremiumDomain=true&isTrackClicksAnalyticsEnabled=false&isUrlMigrated=true&isWixCodeOnPage=false&isWixCodeOnSite=false&language=fr&languageResolutionMethod=QueryParam&metaSiteId=6d08802f-53b1-41d3-8efe-e9ff693936c5&module=thunderbolt-features&originalLanguage=fr&pageId=ae915e_8a4ec4fca009e211728288dc101e4786_1143.json&quickActionsMenuEnabled=false&registryLibrariesTopology=%5B%7B%22artifactId%22%3A%22editor-elements%22%2C%22namespace%22%3A%22wixui%22%2C%22url%22%3A%22https%3A%2F%2Fstatic.parastorage.com%2Fservices%2Feditor-elements%2F1.11361.0%22%7D%2C%7B%22artifactId%22%3A%22editor-elements%22%2C%22namespace%22%3A%22dsgnsys%22%2C%22url%22%3A%22https%3A%2F%2Fstatic.parastorage.com%2Fservices%2Feditor-elements%2F1.11361.0%22%7D%5D&remoteWidgetStructureBuilderVersion=1.238.0&siteId=64b052e8-197b-46d4-8d7f-2954e7e9a6a6&siteRevision=1144&staticHTMLComponentUrl=https%3A%2F%2Fwww-immostat-com.filesusr.com%2F&useSandboxInHTMLComp=false&viewMode=desktop

我个人倾向于使用无头浏览器，这种浏览器可以执行站点的JavaScript并呈现最终用户所看到的页面；在这样的环境中判断定制的Java脚本也是非常方便的.这里的个人喜好是{chromote}，但它相对类似于RSe/Se.

我相信这个过程是可以优化的，但它是这样的:

• 用chromote加载页面，网站的脚本将被执行，所有那些可通过您的Web浏览器元素判断器访问的元素都是可访问的；
• 用JavaScript+XPath查找iframe，使用带有预定义文本的标题作为锚，提取iframe url；
• 加载url(包括Google图表的Java脚本)，并用{rvest}解析内容，这对于通过CSS Select 器和XPath提取元素很方便；
• 抓取一些负责为折线图设置数据序列的JavaScript，并对其进行修改以输出该序列的JSON字符串.
• 在chromote‘S JS运行时判断修改后的JS(我们已经将其启动并运行，所以我们不妨使用它)
• Parse在R中返回JSON，将list转换为data.Frame

library(rvest)
library(stringr)
library(chromote)

# create new Chrome session, wait until page is loaded + few more moments
b <- ChromoteSession$new()
{
  b$Page$navigate("https://www.immostat.com/market-data")
  b$Page$loadEventFired()
  Sys.sleep(.5)
}

# evaluate JavaScript in Chromote:
# xpath to find iframe in relation to the h2 element with specific text content,
# "Office Investment Prices (gross € psqm)",
# extract src atribute from the first matched iframe
iframe_src <- b$Runtime$evaluate('
var xpath = \'//h2[text()="Office Investment Prices (gross € psqm)"]/../following-sibling::div/wix-iframe/div/iframe\';
   document.evaluate(
     xpath, 
     document, null, XPathResult.UNORDERED_NODE_ITERATOR_TYPE, null )
   .iterateNext()
   .getAttribute("src")')$result$value
iframe_src
#> [1] "https://www-immostat-com.filesusr.com/html/ae915e_6e3e7e9d19bc8d5af1f3e4b96ae5c686.html"

如果从JavaScript中提取数据序列，您可以在这里停止并忽略其余部分 已经开始运作了

# with rvest extract script element from iframe page source, we only need element containing  "var dataIDF",
# split it by linefeed,
# get a line from jsvsacript with line chart data,
# modify resulting javascript so it would return google.visualization.arrayToDataTable() argument as 
# JSON string, i.e.
# var dataIDF = google.visualization.arrayToDataTable([['Quarter','Greater Paris Region'],['Q1 2006',4490],...)
# becomes: 
# JSON.stringify([['Quarter','Greater Paris Region'],['Q1 2006',4490],...)
array_stringify_js <- read_html(iframe_src) |>
  html_element(xpath = "//script[contains(., 'var dataIDF')]") |>
  html_text() |>
  str_split("\n") |>
  unlist() |>
  str_subset("var dataIDF") |>
  str_replace("var.*arrayToDataTable", "JSON.stringify")
str_trunc(array_stringify_js, 80)
#> [1] "\tJSON.stringify([['Quarter','Greater Paris Region'],['Q1 2006',4490],['Q2 200..."

# use existing JS runtime to evalute that frankenscript we just created,
# parse resulting JSON with jsonlite, result is a list
chart_data_lst <- b$Runtime$evaluate(array_stringify_js)$result$value |> 
  jsonlite::parse_json() 
str(chart_data_lst[1:3])
#> List of 3
#>  $ :List of 2
#>   ..$ : chr "Quarter"
#>   ..$ : chr "Greater Paris Region"
#>  $ :List of 2
#>   ..$ : chr "Q1 2006"
#>   ..$ : int 4490
#>  $ :List of 2
#>   ..$ : chr "Q2 2006"
#>   ..$ : int 4631

# returned list structure could use some tweaking, 
# we want list of named lists and items with same names should share same types,
# this structure can be converted to data.frame with proper column types,  
# without having to deal with numeric values turned into strings;

# use the first list item for names for all others, drop the first element, 
# bind list of named lists to data.frame
chart_data <- lapply(chart_data_lst[-1], setNames, chart_data_lst[[1]]) |>
  do.call(rbind, args = _) |>
  as.data.frame()

结果:

head(chart_data)
#>   Quarter Greater Paris Region
#> 1 Q1 2006                 4490
#> 2 Q2 2006                 4631
#> 3 Q3 2006                 4803
#> 4 Q4 2006                 4837
#> 5 Q1 2007                 4953
#> 6 Q2 2007                 5064