判断Pyspark生成的SQL查询

假设在PySpark中没有直接的方法来实现这一点，我能想出的最好方法是:

import pandas as pd
import pandas_to_sql
from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.getOrCreate()

input_data = [
    (100, 2, 3),
    (200, 4, 5),
    (300, 6, 7),
]
pd_df = spark.createDataFrame(input_data, ["first_col", "second_col", "third_col"]).toPandas()

table_name = "this_table"
df = pandas_to_sql.wrap_df(pd_df, table_name)
df[df["first_col"]>100][["second_col", "third_col"]].get_sql_string()

输出:

'SELECT (second_col) AS second_col, (third_col) AS third_col FROM this_table WHERE ((first_col > 100)) '

我知道这并不完全是"转换"PySpark到SQL.但是，如果您熟悉PySpark和Pandas，您可以暂时将其用作您的用例的解决方案.

目前使用的图书馆是pandas_to_sql.它不是生产准备，似乎没有积极维护，所以使用它在您自己的意愿.

除此之外，我不认为有任何简单的方法来做你正在寻找的事情.

否则，如果您只想查看物理计划，而不是精确的SQL，则可以使用explain()，如下所示:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.getOrCreate()

data = [
    (100, 2, 3),
    (200, 4, 5),
    (300, 6, 7),
]

spark_df = spark.createDataFrame(data, ["first_col", "second_col", "third_col"])
spark_df = spark_df.filter(col("first_col")>100).select("second_col", "third_col")
spark_df.explain()

输出:

== Physical Plan ==
*(1) Project [second_col#1L, third_col#2L]
+- *(1) Filter (isnotnull(first_col#0L) AND (first_col#0L > 100))
   +- *(1) Scan ExistingRDD[first_col#0L,second_col#1L,third_col#2L]