C#新手.

我正在try 编写一些代码,它将从一个AWS帐户复制S3存储桶内容,并使用AWS SDK将它们放入另一个S3帐户.

我的问题是,虽然文件名出现在目标存储桶中,但内容/文件似乎已损坏. 作为测试,我让它复制并移动了一个图像. 源映像是193KB,但目标存储桶中的文件是339KB,这似乎有点小.

以下是我的代码. 以ListAndCopyObjects方法作为入口点开始. 客户端,已在代码的另一部分中定义了目标客户端. 如有任何帮助,我们不胜感激. 我已经在这个问题上苦苦思索好几天了.

public static async Task ListAndCopyObjects(string bucketName, string folderName, AmazonS3Client client)
        {
            // Read all bucket files
            Console.WriteLine("Listing Objects");
            var listObjects = client.Paginators.ListObjectsV2(new ListObjectsV2Request()
            {
                BucketName = bucketName,
                Prefix = folderName,
                StartAfter = folderName
            });
            
            Console.WriteLine("Loop through Files");
            
            await foreach (var response in listObjects.Responses)
            {
                Console.WriteLine($"HttpStatusCode: {response.HttpStatusCode}");
                Console.WriteLine($"Number of Keys: {response.KeyCount}");
                foreach (var entry in response.S3Objects)
                {
                    Console.WriteLine($"Key = {entry.Key} Size = {entry.Size}");
                    //Download file data
                    NameValueCollection fileContent = await DownloadObject(client, bucketName, entry.Key);
                    
                    Console.WriteLine(fileContent);
                    //Upload file
                    await UploadObject(destinationClient, Environment.GetEnvironmentVariable("DESTINATION_BUCKET"),
                        entry.Key, fileContent);
                }
            }
        }

        private static async Task<bool> UploadObject(AmazonS3Client client, string bucketName, string objectName, 
            NameValueCollection fileContents)
        {
            var request = new PutObjectRequest
            {
                BucketName = bucketName,
                Key = objectName,
                ContentBody = fileContents["content"],
                ContentType = fileContents["contentType"]
            };

            var response = await client.PutObjectAsync(request);
            if (response.HttpStatusCode == System.Net.HttpStatusCode.OK)
            {
                Console.WriteLine($"Successfully uploaded {objectName} to {bucketName}.");
                return true;
            }
            else
            {
                Console.WriteLine($"Could not upload {objectName} to {bucketName}.");
                return false;
            }
        }
        
        private static async Task<NameValueCollection> DownloadObject(AmazonS3Client client, string bucketName, string objectName)
        {
            Console.WriteLine("Downloading File");
            NameValueCollection fileData = new NameValueCollection();
            try
            {
                GetObjectRequest request = new GetObjectRequest
                {
                    BucketName = bucketName,
                    Key = objectName
                };
                using (GetObjectResponse response = await client.GetObjectAsync(request))
                using (Stream responseStream = response.ResponseStream)
                using (StreamReader reader = new StreamReader(responseStream))
                {
                    string title = response.Metadata["x-amz-meta-title"];
                    string contentType = response.Headers["Content-Type"];
                    Console.WriteLine("Object metadata, Title: {0}", title);
                    Console.WriteLine("Content type: {0}", contentType);

                    fileData["contentType"] = contentType;
                    fileData["content"] = reader.ReadToEnd();
                    
                    return fileData;
                }
            }
            catch (AmazonS3Exception e)
            {
                Console.WriteLine("Error encountered ***. Message:'{0}' when writing an object", e.Message);
            }
            catch (Exception e)
            {
                Console.WriteLine("Unknown encountered on server. Message:'{0}' when writing an object", e.Message);
            }

            return fileData;
        }

推荐答案

Probable cause for corruption

您的问题很可能来自于将任何文件视为UTF8编码的文本.文本编码通常需要特定的二进制数据排列,try 将任何二进制数据转换为文本将是"可行的",但通常(取决于编码)在写回二进制数据时会导致更改(损坏)数据.

Solution

由于您是新手,我已try 对您的代码进行尽可能少的更改以使其正常工作,但请查看我提出的改进建议.

因为您想传输任何类型的数据,所以您可能想要考虑使用可以保存二进制数据的东西.因为PutObjectRequest分代表Stream分,所以MemoryStream分似乎是个不错的 Select .我已经更新了您的DownloadObject方法,将数据下载到MemoryStream,并将其保存在您的NameValueCollection中,我不得不将其更改为Dictionary<string, object>:

private static async Task<Dictionary<string, object>> DownloadObject(AmazonS3Client client, string bucketName, string objectName)
{
    Console.WriteLine("Downloading File");
    Dictionary<string, object> fileData = new Dictionary<string, object>();
    try
    {
        GetObjectRequest request = new GetObjectRequest
        {
            BucketName = bucketName,
            Key = objectName
        };

        var contentStream = new MemoryStream();
        using (GetObjectResponse response = await client.GetObjectAsync(request))
        using (Stream responseStream = response.ResponseStream)
        {
            string title = response.Metadata["x-amz-meta-title"];
            string contentType = response.Headers["Content-Type"];
            Console.WriteLine("Object metadata, Title: {0}", title);
            Console.WriteLine("Content type: {0}", contentType);


            await responseStream.CopyToAsync(contentStream);
            contentStream.Position = 0;

            fileData["contentType"] = contentType;
            fileData["content"] = contentStream;
        }

        return fileData;
    }
    catch (AmazonS3Exception e)
    {
        Console.WriteLine("Error encountered ***. Message:'{0}' when writing an object", e.Message);
    }
    catch (Exception e)
    {
        Console.WriteLine("Unknown encountered on server. Message:'{0}' when writing an object", e.Message);
    }

    return fileData;
}

然后我已经更新了您的UploadObject方法,以使用PutObjectRequest‘S InputStream方法的流:

private static async Task<bool> UploadObject(AmazonS3Client client, string bucketName, string objectName,
    Dictionary<string, object> fileContents)
{
    var request = new PutObjectRequest
    {
        BucketName = bucketName,
        Key = objectName,
        InputStream = (Stream)fileContents["content"],
        ContentType = (string)fileContents["contentType"]
    };

    var response = await client.PutObjectAsync(request);
    if (response.HttpStatusCode == System.Net.HttpStatusCode.OK)
    {
        Console.WriteLine($"Successfully uploaded {objectName} to {bucketName}.");
        return true;
    }
    else
    {
        Console.WriteLine($"Could not upload {objectName} to {bucketName}.");
        return false;
    }
}

您还需要将复制方法中的行更改为Dictionary<string, object>var:

var fileContent = await DownloadObject(client, bucketName, entry.Key);

Suggested improvements

我觉得这段代码最大的问题是,虽然每段代码都被很好地分开,但最终您必须将整个文件读入内存.如果文件很小,这不会太糟糕,但如果文件很大,它可能会占用大量RAM.

可能的解决方案:

  1. 使用CopyObjectRequestCopyObjectAsync方法代替.这里的问题是,这限制了您复制5GB的文件.
  2. 打开从现有文件读取的流,并将其直接传递给InputStream,它应该只通过您的服务器流式传输文件,而不是一次性将整个文件加载到内存中.记住仍然把它放在using中,以便在失败时正确地处理它,尽管您需要在它只在上传完成/失败后才被处理的地方这样做.
  3. 与此无关,但如果你是从谷歌存储等其他服务进行复制,你可以读取文件的部分内容,并使用S3的S分部分上传功能上传它们.上载所有部件后,将其标记为已完成,以使其成为完整文件.

另一件事是,使用NameValueCollection甚至Dictionary<string, object>会带来出错的可能性.比方说,如果你把"content"拼错了"cntent"怎么办?这将导致运行时异常.创建一个小类来存储您的信息可能会更好.假设您的C#语言版本是>;=8.0,您也可以对此使用记录(例如private record FileData(string contentType, Stream dataStream);),然后像使用常规类一样使用它.

Csharp相关问答推荐

如何将ref T*重新解释为ref nint?

禁用AutoSuggestBox项目更改时的动画?

. NET 8 HttpClient post参数将其情况更改为camel'

ASP.NET Core:如何在IPageFilter中注入ApplicationDbContext

Blazor EventCallback<;MyType<;T>;>;

应用程序启动器,可 Select 物理屏幕

如何使用新的Microsoft.IdentityModel.JsonWebToken创建JwtSecurityToken?

从ASP.NET Core中的枚举字段填充 Select 选项时,将默认的第一个选项添加为 Select 元素

JSON空引用异常仅在调试器中忽略try-Catch块,但在其他上下文中捕获得很好

从.Net 6 DLL注册和检索COM对象(Typelib导出:类型库未注册.(异常来自HRESULT:0x80131165))

如何注册类使用多级继承与接口

在IAsyncEnumerable上先调用,然后跳过(1)可以吗?

如何管理Azure认证客户端响应和证书 fingerprint

为什么方法的值在SELECT方法中不会更改?

是否可以将Collectionview中的数组与ObservableCollection绑定?

使用未赋值的、传递的局部变量

无法将.Net Framework 4.8.1升级到.Net 7

反编译源代码时出现奇怪的字符

根据运行时值获取泛型类型的字典

C#:我需要根据换行符拆分字符串,而不是在字符串中用双引号分隔换行符