Html 如何从 div 内的属性中提取 href 和文本

以下是获取这些数据的一种方法:

import requests
from bs4 import BeautifulSoup as bs
import pandas as pd

pd.set_option('display.max_columns', None)
pd.set_option('display.max_colwidth', None)

headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36'
}

s = requests.Session()
s.headers.update(headers)

big_list = []
for x in range(9):
    try:
        r = s.get(f'https://pll.harvard.edu/catalog/free?page={x}')
        is_next_page = bs(r.text, 'html.parser').select_one('li[class="pager__item pager__item--next pagination-next"]')
        courses = bs(r.text, 'html.parser').select('div[class="views-row"]')
        for course in courses:
            course_name = course.select_one('h3[class="field__item"]').get_text(strip=True, separator=' ')
            course_url = 'https://pll.harvard.edu' + course.select_one('h3[class="field__item"] a').get('href')
            big_list.append((course_name, course_url))
    except Exception as e:
        print(e)
        break
df = pd.DataFrame(set(big_list), columns = ['title', 'url'])
print(df)

结果为终端:

    title   url
0   CS50's Understanding Technology     https://pll.harvard.edu/course/cs50s-understanding-technology
1   Global News & Technology Leadership in Challenging Times    https://pll.harvard.edu/course/global-news-technology-leadership-challenging-times
2   China’s First Empires and the Rise of Buddhism  https://pll.harvard.edu/course/chinas-first-empires-and-the-rise-of-buddhism
3   Deploying TinyML    https://pll.harvard.edu/course/deploying-tinyml
4   Science & Cooking: From Haute Cuisine to Soft Matter Science (chemistry)    https://pll.harvard.edu/course/science-and-cooking
...     ...     ...
123     PredictionX: John Snow and the Cholera Epidemic of 1854     https://pll.harvard.edu/course/predictionx-john-snow-and-cholera-epidemic-1854
124     CS50's Computer Science for Business Professionals  https://pll.harvard.edu/course/cs50s-computer-science-business-professionals
125     CS50's Understanding Technology     https://pll.harvard.edu/course/cs50s-understanding-technology-0/2023-05
126     Innovating in Health Care   https://pll.harvard.edu/course/innovating-health-care
127     Data Science: Productivity Tools    https://pll.harvard.edu/course/data-science-productivity-tools/2023-10

128 rows × 2 columns

有关详细信息，请参阅requests和BeautifulSoup的文档.