Python 并行解析大量文件并进行汇总

根据 comments ，这里的示例使用multiprocessing.Pool.

每个进程逐行读取一个文件，并将结果发送回主进程进行收集.

import re
import multiprocessing
from collections import Counter

pat = re.compile(r"([A-Z]{2,})=(.+?)(?=[A-Z]{2,}=|$)")


def process_file(filename):
    c = Counter()
    with open(filename, "r") as f_in:
        for line in f_in:
            d = dict(pat.findall(line))
            if "ADDRESS" in d:
                c[d["ADDRESS"]] += 1

    # send partial result back to main process:
    return c


if __name__ == "__main__":

    # you can get filelist for example from `glob` module:
    files = ["file1.txt", "file2.txt"]

    final_counter = Counter()

    with multiprocessing.Pool(processes=4) as pool:
        # iterate over files and update the final Counter:
        for result in pool.imap_unordered(process_file, files):
            final_counter.update(result)

    # print final Counter:
    for k, v in final_counter.items():
        print("{:<20} {}".format(k, v))

打印:

123 Fake St          2
234 Fake St          2

注:您可以使用tqdm模块来获得漂亮的进度条，例如:

...
from tqdm import tqdm
...

for result in pool.imap_unordered(process_file, tqdm(files)):

...