如何使用BeautifulSoup从多个URL检索数据(Python只返回最后一行)

您正在覆盖每个循环中的结果-在list中存储结果以在另一个for-loop中迭代这些结果，或者直接抓取所需的信息-请注意，您只是从每个航空公司的第一个 comments 页面获得 comments ，要获得所有 comments ，您必须实现另一个loop来迭代每个航空公司的所有页面(查看示例后了解它的 idea ).

示例集中在您的操作中描述的第一页，并将结果存储在字典列表中，您只需将其转换为数据帧即可:

import requests
import pandas as pd

base_url = 'https://www.airlinequality.com/airline-reviews/'

endings = ['american-airlines', 'delta-air-lines', 'united-airlines',
           'southwest-airlines', 'alaska-airlines']

results = []
data = []

for ending in endings:
    url = base_url + ending
    r = requests.get(url)
    soup = BeautifulSoup(r.content, 'html.parser')
    for e in soup.select('article[itemprop="review"]'):
        data.append({
            'title': e.h2.text,
            'airline': ending,
            'rating':e.select_one('span[itemprop="ratingValue"]').text
        })

data看起来像:

[{'title': '"nothing but a headache"',
  'airline': 'american-airlines',
  'rating': '1'},
 {'title': '“provide vegan options”',
  'airline': 'american-airlines',
  'rating': '2'},
 {'title': '“created so much stress and hassle”',
  'airline': 'american-airlines',
  'rating': '1'},...]

将data转换为数据帧:

pd.DataFrame(data)

How to get "all results"

为了让你知道如何处理所有的结果，判断一下额外的while-loop个结果在做什么-请记住，对你抓取的网站要温和，也要有一些延迟:

for ending in endings:
    url = f'https://www.airlinequality.com/airline-reviews/{ending}/page/1/?sortby=post_date%3ADesc&pagesize=100'
    while True:
        r = requests.get(url)
        soup = BeautifulSoup(r.content, 'html.parser')
        for e in soup.select('article[itemprop="review"]'):
            data.append({
                'title': e.h2.text,
                'airline': ending,
                'rating':e.select_one('span[itemprop="ratingValue"]').text
            })
        if soup.select_one('article.comp_reviews-pagination ul li:last-of-type a'):
            url = base_url + soup.select_one('article.comp_reviews-pagination ul li:last-of-type a').get('href')
        else:
            break