Python3.x 公开数据中的卫星图像网页抓取优化

此代码可以下载压缩后的 map 文件.

import time
import requests
from bs4 import BeautifulSoup
import os

def download_url(url, save_path, chunk_size=128):
    r = requests.get(url, stream=True)
    with open(save_path, 'wb') as fd:
        for chunk in r.iter_content(chunk_size=chunk_size):
            fd.write(chunk)

def get_file_name(url):
    tokens = url.split("&")
    for token in tokens:
        if(token[:2] == 'f='):
            return token[2:]
    return ''

# Start timer
start_time = time.time()
print("Start time: ", start_time)

# create image directory     
image_directory = 'images'
isExist = os.path.exists(image_directory)
if not isExist:
   os.makedirs(image_directory)


# get zip URL and file name
start_sheet = 44744
end_sheet = 44844 # you need to change with 74331, I just test 100 range
total_download = 0
for index in range(start_sheet, end_sheet):
    template = "https://geoportaal.maaamet.ee/index.php?lang_id=2&plugin_act=otsing&page_id=662&&kaardiruut={sheet_number:n}&andmetyyp=ortofoto_eesti_rgb"
    webpage = template.format(sheet_number = index)
    response = requests.get(webpage)
    if (response.status_code == 200):
        soup = BeautifulSoup(response.content, "html.parser")
        link = soup.find("a")
        if link is not None:
            url = 'https://geoportaal.maaamet.ee/' + link['href']
            file_name = get_file_name(url)
            print(file_name)
            # save zip file
            download_url(url, './' + image_directory + '/' + get_file_name(url))
            total_download = total_download + 1
# End timer
end_time = time.time()

# Calculate elapsed time
elapsed_time = end_time - start_time
print("Elapsed time: ", elapsed_time)
print("Total Download zip files: ", total_download)

完成后的结果

If unzip, you can see the geoTIFF file.

主旨

当您指向此URL时

https://geoportaal.maaamet.ee/eng/Maps-and-Data/Orthophotos/Download-Orthophotos-p662.html

标明了页码范围

Map sheet numbers of 1:10000 scale are between 44744 to 74331.

在Chrome(或Firefox)中，如果按F12键，则会显示Dev Tool.

在"网络"选项卡上可以看到"https Call at header"选项卡.

打开此屏幕后，当您按search键搜索页码(44744)时，可以看到请求URL.

这是模板URL.

https://geoportaal.maaamet.ee/index.php?lang_id=2&plugin_act=otsing&page_id=662&&kaardiruut=44744&andmetyyp=ortofoto_eesti_rgb&_=1686945341505

kaardiruut参数是switch 板号的关键.

kaardiruut=44744

下载一个程序会增加它的数值，用于更改另一个区域.