[RESOLVED] Read 4gb file into FileStream and copy to MemoryStream

[JuggaloBrotha]

VS2015, .Net 4.0
Used NuGet to get the latest AWSSDK.S3 v3.3.5.3 and AWSSDK.Core v3.3.8 references.

I'm trying to use the AWS SDK to upload a file from our server to an Amazon S3 account which works just fine for files 2gb or smaller but one client we generate a file that when it's gzipped ends up being around 3.71gb and our upload routine errors out because File.ReadAllBytes() only handles up to 2gb.

I've tried creating the FileStream manually and using .CopyTo() to copy it into the MemoryStream before passing it into the AWK SDK PutObjectRequest, but I'm still getting this exception:

System.IO.IOException: Stream was too long.
   at AmazonS3Utilities.UploadFile.UploadFile(String FilePath, String BucketPath, String UsrName, String Pass, RegionEndpoint EndPt) in C:\TFS\DataTeam Projects\Utilities\AmazonS3Utilities\AmazonS3Utilities\UploadFile.vb:line 123
   at AmazonS3Utilities.UploadFile.UploadFile(String FilePath, String BucketPath) in C:\TFS\DataTeam Projects\Utilities\AmazonS3Utilities\AmazonS3Utilities\UploadFile.vb:line 90
   at UploadFileS3.Main.Main(String[] Args) in C:\TFS\DataTeam Projects\Utilities\AmazonS3Utilities\UploadFileS3\Main.vb:line 48

Here's my vb.net code:

Public Function UploadFile(FilePath As String, BucketPath As String, UsrName As String, Pass As String, EndPt As RegionEndpoint) As Boolean
        Dim Output As Boolean = False
 
        UsrName = UsrName.Trim()
        Pass = Pass.Trim()
        BucketPath = BucketPath.Trim()
        FilePath = FilePath.Trim()
 
        If UsrName.Length > 0I Then
            If Pass.Length > 0I Then
                If File.Exists(FilePath) Then
                    Dim s3Client As AmazonS3Client = Nothing
                    Dim memStream As MemoryStream = Nothing
                    Dim fileStream As FileStream = Nothing
                    Try
                        s3Client = New AmazonS3Client(UsrName, Pass, EndPt)
                        'memStream = New MemoryStream(File.ReadAllBytes(FilePath))
                        memStream = New MemoryStream
                        fileStream = New FileStream(FilePath, FileMode.Open, FileAccess.Read)
                        fileStream.CopyTo(memStream)
 
                        Dim rr As New PutObjectRequest() With {.BucketName = BucketPath, .Key = Path.GetFileName(FilePath), .InputStream = memStream}
                        Dim response As PutObjectResponse = s3Client.PutObject(rr)
 
                        Output = response.HttpStatusCode = HttpStatusCode.OK
                    Catch ex As Exception
                        Throw ex
                    Finally
                        If fileStream IsNot Nothing Then
                            fileStream.Close()
                            fileStream.Dispose()
                        End If
 
                        If memStream IsNot Nothing Then
                            memStream.Close()
                            memStream.Dispose()
                        End If
 
                        If s3Client IsNot Nothing Then
                            s3Client.Dispose()
                        End If
                    End Try
                Else
                    Throw New FileNotFoundException($"File {Path.GetFileName(FilePath)} was not found")
                End If
            Else
                Throw New ArgumentNullException("Password cannot be blank or all spaces")
            End If
        Else
            Throw New ArgumentNullException("UserName cannot be blank or all spaces")
        End If
 
        Return Output
    End Function

I do see that .Net offers a BufferedStream, could I somehow use the FileStream.CopyTo() into a BufferedStream that will then .CopyTo() the MemoryStream?

[techgnome]

no need....
https://msdn.microsoft.com/en-us/lib...v=vs.110).aspx
The CopyTo has a buffer size parameter .... so what you'll want to do is simply loop until you hit the EOS (end of stream) each time specifying a buffer size.

-tg

[JuggaloBrotha]

no need....
https://msdn.microsoft.com/en-us/lib...v=vs.110).aspx
The CopyTo has a buffer size parameter .... so what you'll want to do is simply loop until you hit the EOS (end of stream) each time specifying a buffer size.

-tg

Is there an example showing this?
I have a severe lack of working with stream so this all seems confusing to me still.

[techgnome]

actually... now that I look at the code closer... you're taking the file stream, copying it to a memory stream and then passing the memory stream to the request call... any reason for that? InputStream should be able to accept any IStream ... including a FileStream object... it should then be responsible for buffering and reading the stream directly.

what if you change this:

                        s3Client = New AmazonS3Client(UsrName, Pass, EndPt)
                        'memStream = New MemoryStream(File.ReadAllBytes(FilePath))
                        memStream = New MemoryStream
                        fileStream = New FileStream(FilePath, FileMode.Open, FileAccess.Read)
                        fileStream.CopyTo(memStream)
 
                        Dim rr As New PutObjectRequest() With {.BucketName = BucketPath, .Key = Path.GetFileName(FilePath), .InputStream = memStream}
                        Dim response As PutObjectResponse = s3Client.PutObject(rr)

to this:

                        s3Client = New AmazonS3Client(UsrName, Pass, EndPt)
                        fileStream = New FileStream(FilePath, FileMode.Open, FileAccess.Read)
 
                        Dim rr As New PutObjectRequest() With {.BucketName = BucketPath, .Key = Path.GetFileName(FilePath), .InputStream = fileStream}
                        Dim response As PutObjectResponse = s3Client.PutObject(rr)

-tg

[JuggaloBrotha]

actually... now that I look at the code closer... you're taking the file stream, copying it to a memory stream and then passing the memory stream to the request call... any reason for that? InputStream should be able to accept any IStream ... including a FileStream object... it should then be responsible for buffering and reading the stream directly.

what if you change this:

                        s3Client = New AmazonS3Client(UsrName, Pass, EndPt)
                        'memStream = New MemoryStream(File.ReadAllBytes(FilePath))
                        memStream = New MemoryStream
                        fileStream = New FileStream(FilePath, FileMode.Open, FileAccess.Read)
                        fileStream.CopyTo(memStream)
 
                        Dim rr As New PutObjectRequest() With {.BucketName = BucketPath, .Key = Path.GetFileName(FilePath), .InputStream = memStream}
                        Dim response As PutObjectResponse = s3Client.PutObject(rr)

to this:

                        s3Client = New AmazonS3Client(UsrName, Pass, EndPt)
                        fileStream = New FileStream(FilePath, FileMode.Open, FileAccess.Read)
 
                        Dim rr As New PutObjectRequest() With {.BucketName = BucketPath, .Key = Path.GetFileName(FilePath), .InputStream = fileStream}
                        Dim response As PutObjectResponse = s3Client.PutObject(rr)

-tg

I just came here to post that I had (eventually) noticed that the parameter it accepts is "Stream" and I am able to pass it the FileStream directly and it works fine.
The examples I had followed a couple of weeks ago when I first made this all used a MemoryStream and thus showed reading a FileStream into a MemoryStream which kept their examples aligned.

So my final code is:

Public Function UploadFile(FilePath As String, BucketPath As String, UsrName As String, Pass As String, EndPt As RegionEndpoint) As Boolean
        Dim Output As Boolean = False
 
        UsrName = UsrName.Trim()
        Pass = Pass.Trim()
        BucketPath = BucketPath.Trim()
        FilePath = FilePath.Trim()
 
        If UsrName.Length > 0I Then
            If Pass.Length > 0I Then
                If File.Exists(FilePath) Then
                    Dim s3Client As AmazonS3Client = Nothing
                    Dim fileStream As FileStream = Nothing
 
                    Try
                        s3Client = New AmazonS3Client(UsrName, Pass, EndPt)
 
                        fileStream = New FileStream(FilePath, FileMode.Open, FileAccess.Read)
 
                        Dim rr As New PutObjectRequest() With {.BucketName = BucketPath,
                                                               .Key = Path.GetFileName(FilePath),
                                                               .InputStream = fileStream
                                                              }
                        Dim response As PutObjectResponse = s3Client.PutObject(rr)
 
                        Output = response.HttpStatusCode = HttpStatusCode.OK
                    Catch ex As Exception
                        Throw ex
                    Finally
                        If fileStream IsNot Nothing Then
                            fileStream.Close()
                            fileStream.Dispose()
                        End If
 
                        If s3Client IsNot Nothing Then
                            s3Client.Dispose()
                        End If
                    End Try
                Else
                    Throw New FileNotFoundException($"File {Path.GetFileName(FilePath)} was not found")
                End If
            Else
                Throw New ArgumentNullException("Password cannot be blank or all spaces")
            End If
        Else
            Throw New ArgumentNullException("UserName cannot be blank or all spaces")
        End If
 
        Return Output
    End Function

[techgnome]

sweet! Glad it worked.

-tg