Python 如何按属性键过滤RDD，然后使用pyspark应用函数

没有任何关键字的最小分数:

score_rdd = my_rdd.map(lambda my_dict: my_dict['exam_score']/100)
print(score_rdd.min())
# 0.76

这是一个使用PYSPARK RDD的解决方案:

score_rdd = my_rdd.map(lambda my_dict: (my_dict['class'], my_dict['exam_score']/100))
print(score_rdd.collect())
# [('classA', 0.9), ('classA', 0.78), ('classB', 0.91), ('classB', 0.76), ('classA', 0.88), ('classA', 0.77), ('classB', 0.86), ('classB', 0.89)]
from builtins import min # import python function: min
class_min_rdd = score_rdd.reduceByKey(min)
print(class_min_rdd.collect())
# [('classA', 0.77), ('classB', 0.76)]

这是一个使用PYSPARK数据帧的解决方案:

from pyspark.sql.types import *
from pyspark.sql.functions import *

df = my_rdd.toDF()
df.printSchema()
# root
#  |-- age: long (nullable = true)
#  |-- class: string (nullable = true)
#  |-- exam_score: long (nullable = true)
#  |-- id: string (nullable = true)
#  |-- name: string (nullable = true)
df = df.withColumn('score', col('exam_score')/100).groupBy('class').agg(min('score').alias('score'))
df.show(10, False)
# +------+-----+
# |class |score|
# +------+-----+
# |classB|0.76 |
# |classA|0.77 |
# +------+-----+
print(df.collect())
# [Row(class='classB', score=0.76), Row(class='classA', score=0.77)]