Python 在一列可以有重复项的PYSpark中连接2个数据帧

收集每个ID的唯一条件

users = df1.groupby('ID').agg(F.collect_set('CONDITION').alias('CONDITION'))

# +---+------------+
# | ID|   CONDITION|
# +---+------------+
# |  1|[C, E, B, A]|
# |  2|   [C, B, D]|
# +---+------------+

将CONDITIONS_DATA与用户数据框连接，其中连接条件必须满足集合成员资格测试

cond = F.expr("size(array_intersect(conditions, CONDITION)) = size(conditions)")
result = conditions_data.join(users, on=cond, how='left')

# +-----+----------+---+------------+
# |sl_no|conditions| ID|   CONDITION|
# +-----+----------+---+------------+
# |   s1|    [A, B]|  1|[C, E, B, A]|
# |   s2|    [C, D]|  2|   [C, B, D]|
# |   s3|    [B, C]|  1|[C, E, B, A]|
# |   s3|    [B, C]|  2|   [C, B, D]|
# +-----+----------+---+------------+

收集Condition_Data中每个唯一行的用户

result = result.groupby(*conditions_data.columns).agg(F.collect_list('ID').alias('ID'))

# +-----+----------+------+
# |sl_no|conditions|    ID|
# +-----+----------+------+
# |   s1|    [A, B]|   [1]|
# |   s2|    [C, D]|   [2]|
# |   s3|    [B, C]|[1, 2]|
# +-----+----------+------+