Python 寻找一种更快的方法在数据框中创建新列，其中包含来自另一列行的字典值

Pd.json_Normize使用起来更方便，但它被证明是最慢的.列表生成器已经成为所有生成器中速度最快的.下面是pd.json_Normize代码.并使用不同的方法进行测试.

import pandas as pd

aaa = [[{'date': None, 'dateTime': '2021-08-12T09:30'}],
[{'date': None, 'dateTime': '2021-08-12T10:00'}],
[{'date': None, 'dateTime': '2021-08-18T11:00'}],
[{'date': None, 'dateTime': '2023-08-09T14:00'}],
[{'date': None, 'dateTime': '2023-08-09T15:00'}],
[{'date': None, 'dateTime': '2023-08-10T11:30'}],
[{'date': None, 'dateTime': '2023-08-10T16:00'}],
[{'date': None, 'dateTime': '2023-08-10T17:00'}]]


calFilteredDf = pd.DataFrame(aaa)

print(calFilteredDf)

calFilteredDf = pd.json_normalize(calFilteredDf[0])
calFilteredDf['startTime'] = calFilteredDf['dateTime']

print(calFilteredDf)

输入

                                                0
0  {'date': None, 'dateTime': '2021-08-12T09:30'}
1  {'date': None, 'dateTime': '2021-08-12T10:00'}
2  {'date': None, 'dateTime': '2021-08-18T11:00'}
3  {'date': None, 'dateTime': '2023-08-09T14:00'}
4  {'date': None, 'dateTime': '2023-08-09T15:00'}
5  {'date': None, 'dateTime': '2023-08-10T11:30'}
6  {'date': None, 'dateTime': '2023-08-10T16:00'}
7  {'date': None, 'dateTime': '2023-08-10T17:00'}

输出

   date          dateTime         startTime
0  None  2021-08-12T09:30  2021-08-12T09:30
1  None  2021-08-12T10:00  2021-08-12T10:00
2  None  2021-08-18T11:00  2021-08-18T11:00
3  None  2023-08-09T14:00  2023-08-09T14:00
4  None  2023-08-09T15:00  2023-08-09T15:00
5  None  2023-08-10T11:30  2023-08-10T11:30
6  None  2023-08-10T16:00  2023-08-10T16:00
7  None  2023-08-10T17:00  2023-08-10T17:00

是的，的确，json_Normizize的速度是原来的两倍.下面是使用Apply、json_Normize、Transform、List Generator的代码.

now = datetime.datetime.now()
for i in range(10000):
    calFilteredDf[0].apply(lambda x: x['dateTime'])

time_ = datetime.datetime.now() - now
print('apply', time_)

now = datetime.datetime.now()
for i in range(10000):
    pd.json_normalize(calFilteredDf[0])

time_ = datetime.datetime.now() - now
print('json_normalize', time_)


now = datetime.datetime.now()
for i in range(10000):
    calFilteredDf[0].transform(lambda x: x['dateTime'])

time_ = datetime.datetime.now() - now
print('transform', time_)

now = datetime.datetime.now()
for i in range(10000):
    a = [i['dateTime'] for i in calFilteredDf[0]]

time_ = datetime.datetime.now() - now
print('list generator', time_)

输出

apply 0:00:01.707580
json_normalize 0:00:03.666553
transform 0:00:01.896933
list generator 0:00:00.056657

输出结果是使用列表生成器.