嗨,我引用了以下问题,因为它与我想要实现的目标类似,但是,我收到了一个错误,我似乎找不到答案,所以寻求一些帮助

Combining multithreading and multiprocessing with concurrent.futures

以下是我的测试代码:

from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
import numpy as np
from os import cpu_count
from functools import partial

num_list = range(0,1000)
  
def test(x):
    x**2
             
def multithread(f, lst):
    print('Thread running')
    with ThreadPoolExecutor() as thread_executor:
        thread_executor.map(f, lst)

def multiprocesser(lst, f, n_processors=cpu_count()//2):
    chunks = np.array_split(lst, n_processors)
    with ProcessPoolExecutor(max_workers=n_processors) as mp_executor:
        mp_executor.map(partial(multithread, f), chunks)

if __name__ == '__main__':
    multiprocesser(num_list, test)
Process SpawnProcess-31:
Traceback (most recent call last):
  File "C:\Users\Test_user\Anaconda3\envs\test_env\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "C:\Users\Test_user\Anaconda3\envs\test_env\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Test_user\Anaconda3\envs\test_env\lib\concurrent\futures\process.py", line 237, in _process_worker
    call_item = call_queue.get(block=True)
  File "C:\Users\Test_user\Anaconda3\envs\test_env\lib\multiprocessing\queues.py", line 122, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'multithread' on <module '__main__' (built-in)>
Process SpawnProcess-32:
Traceback (most recent call last):
  File "C:\Users\Test_user\Anaconda3\envs\test_env\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "C:\Users\Test_user\Anaconda3\envs\test_env\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Test_user\Anaconda3\envs\test_env\lib\concurrent\futures\process.py", line 237, in _process_worker
    call_item = call_queue.get(block=True)
  File "C:\Users\Test_user\Anaconda3\envs\test_env\lib\multiprocessing\queues.py", line 122, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'multithread' on <module '__main__' (built-in)>

所以我没有指定线程的数量(我看不出有理由指定线程池执行器).我很难理解这个错误的实际含义以及我如何修复它.任何帮助都将不胜感激.

推荐答案

该错误可能源于错误地调用了多线程().

试试这个:

from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
import numpy as np
from os import cpu_count
from functools import partial

num_list = range(0,1000)

def test(x):
    x**2
               
def multithread(f, lst):
    print('Thread running')
    with ThreadPoolExecutor() as thread_executor:
        thread_executor.map(f, lst)

def multiprocesser(lst, f, n_processors=cpu_count()//2):
    chunks = np.array_split(lst, n_processors)
    with ProcessPoolExecutor(max_workers=n_processors) as mp_executor:
        mp_executor.map(partial(multithread, f), chunks)

if __name__ == '__main__':
    multiprocesser(num_list, test)

Python相关问答推荐

用Python获取HTML Span类中的数据

Python panda拆分列保持连续多行

通过交换 node 对链接列表进行 Select 排序

Python -根据另一个数据框中的列编辑和替换数据框中的列值

Python Hashicorp Vault库hvac创建新的秘密版本,但从先前版本中删除了密钥

如何在Deliveryter笔记本中从同步上下文正确地安排和等待Delivercio代码中的结果?

点到面的Y距离

时间序列分解

连接两个具有不同标题的收件箱

按顺序合并2个词典列表

如何让程序打印新段落上的每一行?

如何使用它?

从一个系列创建一个Dataframe,特别是如何重命名其中的列(例如:使用NAs/NaN)

Django REST Framework:无法正确地将值注释到多对多模型,不断得到错误字段名称字段对模型无效'<><>

如何使用OpenGL使球体遵循Python中的八样路径?

python—telegraph—bot send_voice发送空文件

使用字典或列表的值组合

如何在Gekko中使用分层条件约束

有没有办法让Re.Sub报告它所做的每一次替换?

如何在PythonPandas 中对同一个浮动列进行逐行划分?