给定一个包含几十列或更多列的CSV,如何创建一个"模式",用于PostgreSQL中的CREATE TABLE SQL表达式,以便与复制工具一起使用?
我看到了很多关于复制工具和基本的创建表表达式的例子,但是对于手动创建模式的列数可能过高的情况,没有详细介绍.
给定一个包含几十列或更多列的CSV,如何创建一个"模式",用于PostgreSQL中的CREATE TABLE SQL表达式,以便与复制工具一起使用?
我看到了很多关于复制工具和基本的创建表表达式的例子,但是对于手动创建模式的列数可能过高的情况,没有详细介绍.
如果CSV不是太大,并且在本地机器上可用,那么csvkit是最简单的解决方案.它还包含许多用于处理CSV的其他实用程序,因此一般来说,这是一个需要了解的基础知识.
在shell中输入最简单的内容:
$ csvsql myfile.csv
将打印出所需的CREATE TABLE
SQL命令,该命令可以使用输出重定向保存到文件中.
如果您还提供了连接字符串,csvsql
将一次性创建表并上传文件:
$ csvsql --db "$MY_DB_URI" --insert myfile.csv
还有一些选项可以指定正在使用的SQL和CSV的风格.它们记录在内置帮助中:
$ csvsql -h
usage: csvsql [-h] [-d DELIMITER] [-t] [-q QUOTECHAR] [-u {0,1,2,3}] [-b]
[-p ESCAPECHAR] [-z MAXFIELDSIZE] [-e ENCODING] [-S] [-H] [-v]
[--zero] [-y SNIFFLIMIT]
[-i {access,sybase,sqlite,informix,firebird,mysql,oracle,maxdb,postgresql,mssql}]
[--db CONNECTION_STRING] [--query QUERY] [--insert]
[--tables TABLE_NAMES] [--no-constraints] [--no-create]
[--blanks] [--no-inference] [--db-schema DB_SCHEMA]
[FILE [FILE ...]]
Generate SQL statements for one or more CSV files, create execute those
statements directly on a database, and execute one or more SQL queries.
positional arguments:
FILE The CSV file(s) to operate on. If omitted, will accept
input on STDIN.
optional arguments:
-h, --help show this help message and exit
-d DELIMITER, --delimiter DELIMITER
Delimiting character of the input CSV file.
-t, --tabs Specifies that the input CSV file is delimited with
tabs. Overrides "-d".
-q QUOTECHAR, --quotechar QUOTECHAR
Character used to quote strings in the input CSV file.
-u {0,1,2,3}, --quoting {0,1,2,3}
Quoting style used in the input CSV file. 0 = Quote
Minimal, 1 = Quote All, 2 = Quote Non-numeric, 3 =
Quote None.
-b, --doublequote Whether or not double quotes are doubled in the input
CSV file.
-p ESCAPECHAR, --escapechar ESCAPECHAR
Character used to escape the delimiter if --quoting 3
("Quote None") is specified and to escape the
QUOTECHAR if --doublequote is not specified.
-z MAXFIELDSIZE, --maxfieldsize MAXFIELDSIZE
Maximum length of a single field in the input CSV
file.
-e ENCODING, --encoding ENCODING
Specify the encoding the input CSV file.
-S, --skipinitialspace
Ignore whitespace immediately following the delimiter.
-H, --no-header-row Specifies that the input CSV file has no header row.
Will create default headers.
-v, --verbose Print detailed tracebacks when errors occur.
--zero When interpreting or displaying column numbers, use
zero-based numbering instead of the default 1-based
numbering.
-y SNIFFLIMIT, --snifflimit SNIFFLIMIT
Limit CSV dialect sniffing to the specified number of
bytes. Specify "0" to disable sniffing entirely.
-i {access,sybase,sqlite,informix,firebird,mysql,oracle,maxdb,postgresql,mssql}, --dialect {access,sybase,sqlite,informix,firebird,mysql,oracle,maxdb,postgresql,mssql}
Dialect of SQL to generate. Only valid when --db is
not specified.
--db CONNECTION_STRING
If present, a sqlalchemy connection string to use to
directly execute generated SQL on a database.
--query QUERY Execute one or more SQL queries delimited by ";" and
output the result of the last query as CSV.
--insert In addition to creating the table, also insert the
data into the table. Only valid when --db is
specified.
--tables TABLE_NAMES Specify one or more names for the tables to be
created. If omitted, the filename (minus extension) or
"stdin" will be used.
--no-constraints Generate a schema without length limits or null
checks. Useful when sampling big tables.
--no-create Skip creating a table. Only valid when --insert is
specified.
--blanks Do not coerce empty strings to NULL values.
--no-inference Disable type inference when parsing the input.
--db-schema DB_SCHEMA
Optional name of database schema to create table(s)
in.
其他几种工具也可以进行模式推断,包括:
每一个都具有将CSV(和其他格式)读入表格数据 struct (通常称为DataFrame或类似 struct )的功能,从而推断过程中的列类型.然后,他们可以使用其他命令来写出等效的SQL模式,或者直接将数据帧上传到指定的数据库中.工具的 Select 将取决于数据量、存储方式、CSV的特性、目标数据库和您喜欢使用的语言.