Python3.x 如何从拼图分区数据集中读取数据到Polar

您可以使用from_arrow()方法:

(
    pl.from_arrow(pq_df).lazy()
    .group_by("date")
    .agg(
        pl.col("price").sum().alias("grouped_sum"),
        pl.col("price").count().alias("grouped_count")
    ).collect(streaming=True)
)

┌──────┬─────────────────────┬───────┐
│ Name ┆ date                ┆ price │
│ ---  ┆ ---                 ┆ ---   │
│ str  ┆ datetime[ms]        ┆ i64   │
╞══════╪═════════════════════╪═══════╡
│ GHI  ┆ 2023-01-29 00:00:00 ┆ 1800  │
│ JKL  ┆ 2023-01-29 00:00:00 ┆ 2100  │
│ ABC  ┆ 2024-01-01 00:00:00 ┆ 1000  │
│ DEF  ┆ 2024-01-10 00:00:00 ┆ 1500  │
└──────┴─────────────────────┴───────┘

但正确的方法可能是使用scan_parquet()种功能，它允许您扫描路径:

(
    pl.scan_parquet(r"test_pl/*/*.parquet")
    .group_by("date")
    .agg(
        pl.col("price").sum().alias("grouped_sum"),
        pl.col("price").count().alias("grouped_count")
    ).collect(streaming=True)
)

┌────────────┬─────────────┬───────────────┐
│ date       ┆ grouped_sum ┆ grouped_count │
│ ---        ┆ ---         ┆ ---           │
│ str        ┆ i64         ┆ u32           │
╞════════════╪═════════════╪═══════════════╡
│ 2024-01-01 ┆ 1000        ┆ 1             │
│ 2024-01-10 ┆ 1500        ┆ 1             │
│ 2023-01-29 ┆ 3900        ┆ 2             │
└────────────┴─────────────┴───────────────┘

此外，您还可以始终使用duckdb:

duckdb.sql("""
    select
        a.date,
        sum(a.price) as grouped_sum,
        sum(a.price) as grouped_count
    from read_parquet('test_pl/*/*.parquet') as a
    group by
        a.date
""").pl()

┌────────────┬─────────────┬───────────────┐
│ date       ┆ grouped_sum ┆ grouped_count │
│ ---        ┆ ---         ┆ ---           │
│ date       ┆ f64         ┆ i64           │
╞════════════╪═════════════╪═══════════════╡
│ 2024-01-01 ┆ 1000.0      ┆ 1             │
│ 2024-01-10 ┆ 1500.0      ┆ 1             │
│ 2023-01-29 ┆ 3900.0      ┆ 2             │
└────────────┴─────────────┴───────────────┘