我有大量的文件和解析器.我要做的是go 掉所有非utf-8符号,并将数据放入mongodb.
with open(fname, "r") as fp:
for line in fp:
line = line.strip()
line = line.decode('utf-8', 'ignore')
line = line.encode('utf-8', 'ignore')
不知怎的,我还是犯了个错误
bson.errors.InvalidStringData: strings in documents must be valid UTF-8:
1/b62010montecassianomcir\xe2\x86\x90ta0\xe2\x86\x90008923304320733/290066010401040101506055soccorin
我不明白.有什么简单的方法吗?
UPD:Python和Mongo似乎不同意Utf-8有效字符串的定义.