Json 如何一次加载无限滚动中的所有条目以解析python中的HTML

发布于01月09日

I am trying to extract information from this page. The page loads 10 items at a time, and I need to scroll to load all entries (for a total of 100). I am able to parse the HTML and get the information that I need for the first 10 entries, but I want to fully load all entries before parsing the HTML.

我正在使用python、请求和BeautifulSoup.当页面加载前10个条目时，我解析页面的方式如下:

from bs4 import BeautifulSoup
import requests
s = requests.Session()
r = s.get('https://medium.com/top-100/december-2013')
page = BeautifulSoup(r.text)

But this only loads the first 10 entries. So I looked at the page and got the AJAX request used to load the subsequent entries and I get a response but it's in the a funky JSON and I'd rather use the HTML parser instead of parsing JSON. Here's the code:

from bs4 import BeautifulSoup
import requests
import json
s = requests.Session()
url = 'https://medium.com/top-100/december-2013/load-more'
payload = {"count":100}
r = s.post(url, data=payload)
page = json.loads(r.text[16:]) #skip some chars that throw json off

这给了我数据，但它是在一个非常长且复杂的JSON中，我更愿意加载页面上的所有数据，并简单地解析HTML.此外，呈现的HTML提供了比JSON响应更多的信息(即作者的姓名而不是模糊的用户ID等).有一个类似的问题here，但没有相关的答案.理想情况下，我希望进行POST调用，并请求HTML并对其进行解析，但我无法做到这一点.

import time from selenium import webdriver from selenium.webdriver.common.keys import Keys browser = webdriver.Chrome() browser.get("https://medium.com/top-100/december-2013") time.sleep(1) elem = browser.find_element_by_tag_name("body") no_of_pagedowns = 20 while no_of_pagedowns: elem.send_keys(Keys.PAGE_DOWN) time.sleep(0.2) no_of_pagedowns-=1 post_elems = browser.find_elements_by_class_name("post-item-title") for post in post_elems: print post.text

When Your Mother Says She’s Fat When “Life Hacking” Is Really White Privilege As tendências culturais dos anos 2000 adiantadas pelo É o Tchan na década de 90 Coming Out as Biracial Como ganhar discussões com seus parentes de direita neste Natal How to save local bookstores in two easy steps Welcome to Dinovember How to Piss Off Your Barista The boy whose brain could unlock autism CrossFit’s Dirty Little Secret Welcome to Medium Here’s How the Military Wasted Your Money in 2013 Why I Wear Nail Polish The day of High School I’ll never forget 7 Reasons Buffalonians Shouldn’t Hate Snow Dear Guy Who Just Made My Burrito: Is the Mona Lisa Priceless? Please stop live tweeting people’s private conversations Your Friends and Rapists Eight things you can live without The Value of Content 40 Ways To Make Life Simple Again Manila-Beijing-Washington: Things I Wish Someone Had Told Me When I Was Learning How to Code Dear Ticketmaster, Steve Jobs Danced To My Song 11 Things I Wish I Knew When I Started My Business Bullish: Benevolent Sexism and “That Guy” Who Makes Everything Awkward Advice to a College Music Student Silver Gyninen joutui sotaan Imagining the Post-Antibiotics Future Which side are you on? Put it away, junior. Casual Predation The sad little iPhone commercial How Node.js is Going to Replace JavaScript Why you should have your heart broken into a million little pieces. How to Write Emails Like a CEO Designing Products That Scale How radioactive poison became the assassin’s weapon of choice Why do people hate CrossFit? We (Still) Need Feminism 10 Advanced Hearthstone Arena Tips Let It Full-Bleed What Medium Is For How a Small Force of Finnish Ski Troops Fought Off a Massive Soviet Army An Introvert’s Guide to Better Presentations Mandela The Terrorist Why You Should have a Messy Desk Why I’m Not a TEDx Speaker Fonts have feelings too You Don’t Want Your Thanksgiving to Go Like This What I’ve Learned in My First Month as a VC Why Quantity Should be Your Priority My Airbnb story I Wanna Date You Like An Animal The GIF Guide to Getting Paid How We Discovered the Underground Chinese App Market First Images of a Heart Injected with Liquid Metal Beyonce Broke the Music Business “View mode” approach to responsive web design Sometimes You Will Forget Your Mom Has Cancer Darkness Ray Beams Invisibility From A Distance Why Work As We Know It May Be Immoral Staying Ahead of the Curve The Geekiest Game Ever Made Has Been Released In Germany The Dirty Secret Behind the Salesforce $1M Hackathon I’m a really good impostor Mathematical Model of Zombie Epidemics Reveals Two Types of Living-Dead Infections The Heartbreak Kid 200 Things I’m Not Racist But— Duel of the Superbattleships 23 and You The Seattle NO I’m a vaccine refuser. There, I said it. The Year We Broke Everything How to make a DIY home alarm system with a raspberry pi and a webcam Strike While the App is Hot How to Fall In (and Out) of Love: Why did Google make an ad for promoting “Search” in India where it has over 97% market share? A Holiday Message From Jesus Revealed: The Soviet Union’s $1 Billion ‘Psychotronic’ Arms Race with the US Postmortem of a Venture-backed Startup The 1.x Crore Myth The “Getting Shit Done” Sleep Cycle Is the F-35 Joint Strike Fighter the New F-4? Can the F-35 Win a Dogfight? Responsive Photosets Fightball: Millennials vs Boomers The iconicity of “peaceful resistance” How We Make Chocolate Five Ships of the Chinese Navy You Really Ought to Know About Glassholes and Black Rock City Bad News for U.S. Warplane Pilots: Russia’s New Dogfighting Missile Can’t Miss How Antisec Died 10 ways you’ll probably f**k up your startup UPDATED: Finding the unjustly homeless, and teaching them to code. Technology hasn’t Changed Us. What I’ve learned from fatherhood

Json 如何一次加载无限滚动中的所有条目以解析python中的HTML

推荐答案

Json相关问答推荐

Golang JSON Date Tim.Date()测试请求

如何使用表键名称GROUP_BY

如何用JQ更改空/布尔/数字的 colored颜色？

使用JQ在数组中 Select 第一个具有匹配项的项

有没有办法让serde_json正确/不正确地处理NaN、inf和-inf(IEEE 754特殊标准)？

转换为Json时忽略内部对象中的数组

Oracle json 对象的最后一个值不在引号中

将 colly 包输出文本添加到 golang 中的映射

是否可以在有条件的情况下将 json 对象转换为 JOLT 中的数组？

如何将从嵌套 Select 返回的空值转换为空数组？

序列化特定类型时如何使 JSON.Net 序列化程序调用 ToString()？

JSON Schema 与 XML Schema 的比较及其future

json.decoder.JSONDecodeError：期望值：第 1 行第 1 列(字符 0)

将带有数据和文件的 JSON 发布到 Web Api - jQuery / MVC

我可以使用空字符串作为对象标识符吗？

在 Java 中比较两个 JSON 文件的最佳方法

有 Json 标签但未导出

你如何在 Arrays of Arrays 上 OPENJSON

杰克逊自动将 Joda DateTime 格式化为 ISO 8601 格式

如何在所有子项中查询具有特定值的属性的firebase