Python3.x 使用大型多个数据集，其中每个数据集包含多个值 Pytorch

通过实现Pytorch提供的定制dataset类，我们需要实现三个方法，以便Pytorch loader可以处理您的数据

• __len__
• __getitem__
• __init__

让我们来看看如何分别实现它们中的每一个.

• __init__个

如上所述，主要思想是使用prefix sum一次性"处理"所有数据集，因此逻辑是，无论何时以后需要访问特定索引，只需查看prefix_sum_idx即可查看此idx出现的位置.

在上图中，假设我们需要访问索引150.由于前缀sum，我们现在可以知道第二个.pickle文件中存在150个.我们仍然需要一个快速的机制来知道idx在prefix_sum_idx中的位置.这将在__getitem__中解释

• __getitem__个

判断bisect\u right()文档以了解其工作方式的详细信息，但只需返回排序列表中最右边的位置，即可插入给定元素并保持其排序.在我们的方法中，我们只对以下问题感兴趣，"为了获得适当的数据，我应该访问哪个文件".更重要的是，它在O(log n)

• __len__个

为了获得数据集的长度，我们循环遍历中的每个文件，并累积结果，如图__init__所示.

完整的代码示例如下所示:

import pickle
import torch
import torch.nn as nn
import numpy
import os 
import bisect
from torch.utils.data import Dataset, DataLoader
from src.data.make_dataset import main
from torch.nn import functional as F

class dataset(Dataset):
  def __init__(self):

    # Original Data has the following format
    """
    dict_object = 
    {
        "x":[],
        "y":[]
    }
    """
    DIRECTORY = "data/raw"
    self.dataset_file_name = os.listdir(DIRECTORY)
    self.dataset_file_name_index = 0
    self.dataset_length =0
    self.prefix_sum_idx = list()
    # Loop over each file and calculate the length of overall dataset
    # you might need to check if file_name is file
    for file_name in os.listdir(DIRECTORY):
    with (open(f'{DIRECTORY}/{file_name}', "rb")) as openfile:
        dict_object = pickle.load(openfile)
        curr_page_sum = len(dict_object["x"]) + len(dict_object["y"])
        self.prefix_sum_idx.append(curr_page_sum)
        self.dataset_length += curr_page_sum
    # prefix sum so we have an idea of where each index appeared in which file. 
    for i in range (1,len(self.prefix_sum_idx)):
    self.prefix_sum_idx[i] = self.prefix_sum_idx[i] + self.prefix_sum_idx[i-1]

    assert self.prefix_sum_idx[-1] == self.dataset_length
    self.x = []
    self.y = []

    



def read_pickle_file(self, idx):
 file_name = self.dataset_file_name[idx]
 dict_object = dict()
 with (open(f'{YOUR_DIRECTORY}/{file_name}', "rb")) as openfile:
     dict_object = pickle.load(openfile)

 self.x = dict_object['x']
 self.y = #some logic here
 ......
 # Some logic here....


def __getitem__(self,idx):

 # Similar to C++ std::upper_bound - O(log n)
 temp = bisect.bisect_right(self.prefix_sum_idx, idx)

 self.read_pickle_file(temp)
 local_idx = idx - self.prefix_sum_idx[temp] 

 return self.x[local_idx],self.y[local_idx]



def __len__(self):
 return self.dataset_length


large_dataset = dataset()
train_size = int (0.8 * len(large_dataset))
validation_size = len(large_dataset) - train_size

train_dataset, validation_dataset = torch.utils.data.random_split(large_dataset, [train_size, validation_size])
validation_loader = DataLoader(validation_dataset, batch_size=64, num_workers=4, shuffle=False)
train_loader = DataLoader(train_dataset,batch_size=64, num_workers=4,shuffle=False)