我想把postgres表读入数据工程管道的数据框中.我用气流来安排这些任务.我在Airflow中创建了一个名为postgres_product_db的连接,并try 使用get_pandas_df
来获取记录.
db_hook = PostgresHook('postgres_product_db')
fetch_item = db_hook.get_pandas_df(request)
但它把错误当作
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)
完整错误日志(log):
[2022-04-05, 06:46:27 UTC] {taskinstance.py:1774} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/azureuser/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 174, in execute
return_value = self.execute_callable()
File "/home/azureuser/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 188, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/home/azureuser/airflow/dags/foodstar_store1_pricing_update_program.py", line 81, in fetch_inventory
inventory = db_hook.get_pandas_df(request)
File "/home/azureuser/.local/lib/python3.8/site-packages/airflow/hooks/dbapi.py", line 138, in get_pandas_df
return psql.read_sql(sql, con=conn, params=parameters, **kwargs)
File "/home/azureuser/.local/lib/python3.8/site-packages/pandas/io/sql.py", line 566, in read_sql
return pandas_sql.read_query(
File "/home/azureuser/.local/lib/python3.8/site-packages/pandas/io/sql.py", line 2094, in read_query
data = self._fetchall_as_list(cursor)
File "/home/azureuser/.local/lib/python3.8/site-packages/pandas/io/sql.py", line 2108, in _fetchall_as_list
result = cur.fetchall()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)
在正常情况下,为了克服这个问题,我设置了编码
conn=psycopg2.connect(
database="xxxxx", user='xxxxx', password='xxxx', host='xxxx.xxx.xxx.xx.xxx', port='5432')
conn.set_client_encoding('UNICODE')
cur=conn.cursor()
但找不到任何选项来设置Postgreshake中的客户端_编码.连接中有一个额外选项,我试图将其设置为{ encode: 'UNICODE' }
.但这也带来了错误.有人能帮忙吗?