I am trying to scrape data from "https://lol.inven.co.kr/dataninfo/match/teamList.php" (sorry the website is not in English)
When I don't clikc the orange squre button, the site does not have "detailTable"
When clicking the button, "detailTable" shows up, which I actually want to scrape.
With this dummy webpage
Since it is the dummy webpage, the request URL (POST) does not have any rule.
在这种情况下，单击onclick按钮("detailTable")时出现的内容是Selenium吗？我是否可以将BeautifulSoup和请求结合使用？
欢迎提出任何 idea .提前感谢您！
您可以从post请求中获取所有数据，但首先需要获取matchcode个值.
然后，发出一个POST请求并解析返回的XML.
例如:
import re
import time
import requests
from bs4 import BeautifulSoup
headers = {
"Content-Type": "application/x-www-form-urlencoded;charset=utf-8",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0"
}
with requests.Session() as s:
soup = (
BeautifulSoup(
s.get(
"https://lol.inven.co.kr/dataninfo/match/teamList.php",
headers=headers,
).text,
"lxml"
).select("#lolDbMatchTeamList .listDetail a")
)
match_ids = [re.search(r"(\d+)", a["onclick"]).group(1) for a in soup]
for match_id in match_ids[:1]: # just getting one match here
r = s.post(
f"https://lol.inven.co.kr/dataninfo/match/match_detail.xml.php?dummy={int(time.time())}",
data={"matchcode": match_id},
)
player_names = (
"\n".join(
p.find("playername").getText() for p
in BeautifulSoup(r.text, features="xml").select("player")
)
)
print(player_names)
这将打印第一场比赛的球员姓名.
RNG Ming
RNG Wei
RNG Xiaohu
RNG GALA
RNG Ming
T1 Zeus
T1 Oner
T1 Faker
T1 Gumayusi
T1 Keria
这里有一个完整的XML，您可以在每POST个请求中获得它.
<?xml version="1.0" encoding="UTF-8"?>
<resultdata>
<match>7345</match>
<player>
<code>125062</code><winloss>1</winloss><side>1</side><teamcode>412</teamcode><teamname>Royal Never Give Up</teamname><playercode>900165</playercode><playername>RNG Ming</playername><mname>Ming</mname><champ>155</champ><champicon>Gwen_Square_0</champicon><champname>그웬</champname><spell1icon>lol_spell_ghost</spell1icon><spell1name>유체화</spell1name><spell2icon>lol_spell_teleport</spell2icon><spell2name>순간이동</spell2name><kill>3</kill><death>0</death><assist>5</assist><ka>53</ka><kd>3.6</kd><asd>6</asd><kda>9.6</kda> </player>
<player>
<code>125063</code><winloss>1</winloss><side>1</side><teamcode>412</teamcode><teamname>Royal Never Give Up</teamname><playercode>9002581</playercode><playername>RNG Wei</playername><mname>Wei</mname><champ>34</champ><champicon>LeeSin_Square_0_2</champicon><champname>리 신</champname><spell1icon>lol_spell_flash</spell1icon><spell1name>점멸</spell1name><spell2icon>lol_spell_smite</spell2icon><spell2name>강타</spell2name><kill>2</kill><death>1</death><assist>11</assist><ka>87</ka><kd>2</kd><asd>11</asd><kda>13</kda> </player>
<player>
<code>125064</code><winloss>1</winloss><side>1</side><teamcode>412</teamcode><teamname>Royal Never Give Up</teamname><playercode>99012</playercode><playername>RNG Xiaohu</playername><mname>Xiaohu</mname><champ>113</champ><champicon>Lissandra_Square_0_1</champicon><champname>리산드라</champname><spell1icon>lol_spell_flash</spell1icon><spell1name>점멸</spell1name><spell2icon>lol_spell_teleport</spell2icon><spell2name>순간이동</spell2name><kill>2</kill><death>2</death><assist>10</assist><ka>80</ka><kd>1</kd><asd>5</asd><kda>6</kda> </player>
<player>
<code>125065</code><winloss>1</winloss><side>1</side><teamcode>412</teamcode><teamname>Royal Never Give Up</teamname><playercode>9002583</playercode><playername>RNG GALA</playername><mname>GALA</mname><champ>64</champ><champicon>Tristana_Square_0_4</champicon><champname>트리스타나</champname><spell1icon>lol_spell_cleanse</spell1icon><spell1name>정화</spell1name><spell2icon>lol_spell_flash</spell2icon><spell2name>점멸</spell2name><kill>5</kill><death>0</death><assist>3</assist><ka>53</ka><kd>6</kd><asd>3.6</asd><kda>9.6</kda> </player>
<player>
<code>125066</code><winloss>1</winloss><side>1</side><teamcode>412</teamcode><teamname>Royal Never Give Up</teamname><playercode>900165</playercode><playername>RNG Ming</playername><mname>Ming</mname><champ>135</champ><champicon>Rakan_Square_0</champicon><champname>라칸</champname><spell1icon>lol_spell_ignite</spell1icon><spell1name>점화</spell1name><spell2icon>lol_spell_flash</spell2icon><spell2name>점멸</spell2name><kill>3</kill><death>0</death><assist>8</assist><ka>73</ka><kd>3.6</kd><asd>9.6</asd><kda>13.2</kda> </player>
<player>
<code>125067</code><winloss>2</winloss><side>2</side><teamcode>223</teamcode><teamname>T1</teamname><playercode>9002552</playercode><playername>T1 Zeus</playername><mname>Zeus</mname><champ>100</champ><champicon>Jayce_Square_0_1</champicon><champname>제이스</champname><spell1icon>lol_spell_flash</spell1icon><spell1name>점멸</spell1name><spell2icon>lol_spell_teleport</spell2icon><spell2name>순간이동</spell2name><kill>1</kill><death>4</death><assist>0</assist><ka>33</ka><kd>0.3</kd><asd>0</asd><kda>0.3</kda> </player>
<player>
<code>125068</code><winloss>2</winloss><side>2</side><teamcode>223</teamcode><teamname>T1</teamname><playercode>9002559</playercode><playername>T1 Oner</playername><mname>Oner</mname><champ>80</champ><champicon>MonkeyKing_Square_0_2</champicon><champname>오공</champname><spell1icon>lol_spell_flash</spell1icon><spell1name>점멸</spell1name><spell2icon>lol_spell_smite</spell2icon><spell2name>강타</spell2name><kill>1</kill><death>2</death><assist>2</assist><ka>100</ka><kd>0.5</kd><asd>1</asd><kda>1.5</kda> </player>
<player>
<code>125069</code><winloss>2</winloss><side>2</side><teamcode>223</teamcode><teamname>T1</teamname><playercode>135</playercode><playername>T1 Faker</playername><mname>Faker</mname><champ>89</champ><champicon>Ahri_Square_0_2</champicon><champname>아리</champname><spell1icon>lol_spell_cleanse</spell1icon><spell1name>정화</spell1name><spell2icon>lol_spell_flash</spell2icon><spell2name>점멸</spell2name><kill>0</kill><death>5</death><assist>2</assist><ka>67</ka><kd>0</kd><asd>0.4</asd><kda>0.4</kda> </player>
<player>
<code>125070</code><winloss>2</winloss><side>2</side><teamcode>223</teamcode><teamname>T1</teamname><playercode>9002255</playercode><playername>T1 Gumayusi</playername><mname>Gumayusi</mname><champ>129</champ><champicon>Jhin_Square_0</champicon><champname>진</champname><spell1icon>lol_spell_exhaust</spell1icon><spell1name>탈진</spell1name><spell2icon>lol_spell_flash</spell2icon><spell2name>점멸</spell2name><kill>1</kill><death>1</death><assist>1</assist><ka>67</ka><kd>1</kd><asd>1</asd><kda>2</kda> </player>
<player>
<code>125071</code><winloss>2</winloss><side>2</side><teamcode>223</teamcode><teamname>T1</teamname><playercode>900280</playercode><playername>T1 Keria</playername><mname>Keria</mname><champ>144</champ><champicon>yummi_square_0</champicon><champname>유미</champname><spell1icon>lol_spell_exhaust</spell1icon><spell1name>탈진</spell1name><spell2icon>lol_spell_heal</spell2icon><spell2name>회복</spell2name><kill>0</kill><death>3</death><assist>2</assist><ka>67</ka><kd>0</kd><asd>0.7</asd><kda>0.7</kda> </player>
</resultdata>