Python3.x python struct.error： 'i' 格式需要 2147483648 <= number <= 2147483647

进程之间的通信协议使用pickling，并且pickle数据以pickle数据的大小作为前缀.对于您的方法，all arguments together被作为一个对象进行酸洗.

您生成了一个对象，当pickle大于i struct格式化程序(一个四字节有符号整数)时，它打破了代码所做的假设.

您可以将数据帧的读取委托给子进程，只发送加载数据帧所需的元数据.它们的总大小接近1GB，太多的数据无法在进程之间通过管道共享.

引用Programming guidelines section人的话:

Better to inherit than pickle/unpickle

当使用spawn或forkserver启动方法时，multiprocessing中的许多类型需要可 Select ，以便子进程可以使用它们.However, one should generally avoid sending shared objects to other processes using pipes or queues. Instead you should arrange the program so that a process which needs access to a shared resource created elsewhere can inherit it from an ancestor process.

如果您不是在Windows上运行，并且使用spawn或forkserver方法，则可以将数据帧作为全局before加载，以启动子进程，此时子进程将通过正常的操作系统写入内存页共享机制"继承"数据.

请注意，在Python 3.8中，对于非Windows系统，这个限制被提高到了无符号长(8字节)，因此现在可以发送和接收4EiB个数据.参见this commit，以及Python第#35152和#17560期.

如果无法升级，无法利用资源继承，并且没有在Windows上运行，请使用以下修补程序:

import functools
import logging
import struct
import sys

logger = logging.getLogger()


def patch_mp_connection_bpo_17560():
    """Apply PR-10305 / bpo-17560 connection send/receive max size update

    See the original issue at https://bugs.python.org/issue17560 and 
    https://github.com/python/cpython/pull/10305 for the pull request.

    This only supports Python versions 3.3 - 3.7, this function
    does nothing for Python versions outside of that range.

    """
    patchname = "Multiprocessing connection patch for bpo-17560"
    if not (3, 3) < sys.version_info < (3, 8):
        logger.info(
            patchname + " not applied, not an applicable Python version: %s",
            sys.version
        )
        return

    from multiprocessing.connection import Connection

    orig_send_bytes = Connection._send_bytes
    orig_recv_bytes = Connection._recv_bytes
    if (
        orig_send_bytes.__code__.co_filename == __file__
        and orig_recv_bytes.__code__.co_filename == __file__
    ):
        logger.info(patchname + " already applied, skipping")
        return

    @functools.wraps(orig_send_bytes)
    def send_bytes(self, buf):
        n = len(buf)
        if n > 0x7fffffff:
            pre_header = struct.pack("!i", -1)
            header = struct.pack("!Q", n)
            self._send(pre_header)
            self._send(header)
            self._send(buf)
        else:
            orig_send_bytes(self, buf)

    @functools.wraps(orig_recv_bytes)
    def recv_bytes(self, maxsize=None):
        buf = self._recv(4)
        size, = struct.unpack("!i", buf.getvalue())
        if size == -1:
            buf = self._recv(8)
            size, = struct.unpack("!Q", buf.getvalue())
        if maxsize is not None and size > maxsize:
            return None
        return self._recv(size)

    Connection._send_bytes = send_bytes
    Connection._recv_bytes = recv_bytes

    logger.info(patchname + " applied")