[RESOLVED] Read 4gb file into FileStream and copy to MemoryStream
VS2015, .Net 4.0
Used NuGet to get the latest AWSSDK.S3 v3.3.5.3 and AWSSDK.Core v3.3.8 references.
I'm trying to use the AWS SDK to upload a file from our server to an Amazon S3 account which works just fine for files 2gb or smaller but one client we generate a file that when it's gzipped ends up being around 3.71gb and our upload routine errors out because File.ReadAllBytes() only handles up to 2gb.
I've tried creating the FileStream manually and using .CopyTo() to copy it into the MemoryStream before passing it into the AWK SDK PutObjectRequest, but I'm still getting this exception:
Code:
System.IO.IOException: Stream was too long.
at AmazonS3Utilities.UploadFile.UploadFile(String FilePath, String BucketPath, String UsrName, String Pass, RegionEndpoint EndPt) in C:\TFS\DataTeam Projects\Utilities\AmazonS3Utilities\AmazonS3Utilities\UploadFile.vb:line 123
at AmazonS3Utilities.UploadFile.UploadFile(String FilePath, String BucketPath) in C:\TFS\DataTeam Projects\Utilities\AmazonS3Utilities\AmazonS3Utilities\UploadFile.vb:line 90
at UploadFileS3.Main.Main(String[] Args) in C:\TFS\DataTeam Projects\Utilities\AmazonS3Utilities\UploadFileS3\Main.vb:line 48
Here's my vb.net code:
vb.net Code:
Public Function UploadFile(FilePath As String, BucketPath As String, UsrName As String, Pass As String, EndPt As RegionEndpoint) As Boolean
Dim Output As Boolean = False
UsrName = UsrName.Trim()
Pass = Pass.Trim()
BucketPath = BucketPath.Trim()
FilePath = FilePath.Trim()
If UsrName.Length > 0I Then
If Pass.Length > 0I Then
If File.Exists(FilePath) Then
Dim s3Client As AmazonS3Client = Nothing
Dim memStream As MemoryStream = Nothing
Dim fileStream As FileStream = Nothing
Try
s3Client = New AmazonS3Client(UsrName, Pass, EndPt)
'memStream = New MemoryStream(File.ReadAllBytes(FilePath))
memStream = New MemoryStream
fileStream = New FileStream(FilePath, FileMode.Open, FileAccess.Read)
fileStream.CopyTo(memStream)
Dim rr As New PutObjectRequest() With {.BucketName = BucketPath, .Key = Path.GetFileName(FilePath), .InputStream = memStream}
Dim response As PutObjectResponse = s3Client.PutObject(rr)
Output = response.HttpStatusCode = HttpStatusCode.OK
Catch ex As Exception
Throw ex
Finally
If fileStream IsNot Nothing Then
fileStream.Close()
fileStream.Dispose()
End If
If memStream IsNot Nothing Then
memStream.Close()
memStream.Dispose()
End If
If s3Client IsNot Nothing Then
s3Client.Dispose()
End If
End Try
Else
Throw New FileNotFoundException($"File {Path.GetFileName(FilePath)} was not found")
End If
Else
Throw New ArgumentNullException("Password cannot be blank or all spaces")
End If
Else
Throw New ArgumentNullException("UserName cannot be blank or all spaces")
End If
Return Output
End Function
I do see that .Net offers a BufferedStream, could I somehow use the FileStream.CopyTo() into a BufferedStream that will then .CopyTo() the MemoryStream?
Re: Read 4gb file into FileStream and copy to MemoryStream
no need....
https://msdn.microsoft.com/en-us/lib...v=vs.110).aspx
The CopyTo has a buffer size parameter .... so what you'll want to do is simply loop until you hit the EOS (end of stream) each time specifying a buffer size.
-tg
Re: Read 4gb file into FileStream and copy to MemoryStream
Quote:
Originally Posted by
techgnome
Is there an example showing this?
I have a severe lack of working with stream so this all seems confusing to me still.
Re: Read 4gb file into FileStream and copy to MemoryStream
actually... now that I look at the code closer... you're taking the file stream, copying it to a memory stream and then passing the memory stream to the request call... any reason for that? InputStream should be able to accept any IStream ... including a FileStream object... it should then be responsible for buffering and reading the stream directly.
what if you change this:
Code:
s3Client = New AmazonS3Client(UsrName, Pass, EndPt)
'memStream = New MemoryStream(File.ReadAllBytes(FilePath))
memStream = New MemoryStream
fileStream = New FileStream(FilePath, FileMode.Open, FileAccess.Read)
fileStream.CopyTo(memStream)
Dim rr As New PutObjectRequest() With {.BucketName = BucketPath, .Key = Path.GetFileName(FilePath), .InputStream = memStream}
Dim response As PutObjectResponse = s3Client.PutObject(rr)
to this:
Code:
s3Client = New AmazonS3Client(UsrName, Pass, EndPt)
fileStream = New FileStream(FilePath, FileMode.Open, FileAccess.Read)
Dim rr As New PutObjectRequest() With {.BucketName = BucketPath, .Key = Path.GetFileName(FilePath), .InputStream = fileStream}
Dim response As PutObjectResponse = s3Client.PutObject(rr)
-tg
Re: Read 4gb file into FileStream and copy to MemoryStream
Quote:
Originally Posted by
techgnome
actually... now that I look at the code closer... you're taking the file stream, copying it to a memory stream and then passing the memory stream to the request call... any reason for that? InputStream should be able to accept any IStream ... including a FileStream object... it should then be responsible for buffering and reading the stream directly.
what if you change this:
Code:
s3Client = New AmazonS3Client(UsrName, Pass, EndPt)
'memStream = New MemoryStream(File.ReadAllBytes(FilePath))
memStream = New MemoryStream
fileStream = New FileStream(FilePath, FileMode.Open, FileAccess.Read)
fileStream.CopyTo(memStream)
Dim rr As New PutObjectRequest() With {.BucketName = BucketPath, .Key = Path.GetFileName(FilePath), .InputStream = memStream}
Dim response As PutObjectResponse = s3Client.PutObject(rr)
to this:
Code:
s3Client = New AmazonS3Client(UsrName, Pass, EndPt)
fileStream = New FileStream(FilePath, FileMode.Open, FileAccess.Read)
Dim rr As New PutObjectRequest() With {.BucketName = BucketPath, .Key = Path.GetFileName(FilePath), .InputStream = fileStream}
Dim response As PutObjectResponse = s3Client.PutObject(rr)
-tg
I just came here to post that I had (eventually) noticed that the parameter it accepts is "Stream" and I am able to pass it the FileStream directly and it works fine.
The examples I had followed a couple of weeks ago when I first made this all used a MemoryStream and thus showed reading a FileStream into a MemoryStream which kept their examples aligned.
So my final code is:
vb.net Code:
Public Function UploadFile(FilePath As String, BucketPath As String, UsrName As String, Pass As String, EndPt As RegionEndpoint) As Boolean
Dim Output As Boolean = False
UsrName = UsrName.Trim()
Pass = Pass.Trim()
BucketPath = BucketPath.Trim()
FilePath = FilePath.Trim()
If UsrName.Length > 0I Then
If Pass.Length > 0I Then
If File.Exists(FilePath) Then
Dim s3Client As AmazonS3Client = Nothing
Dim fileStream As FileStream = Nothing
Try
s3Client = New AmazonS3Client(UsrName, Pass, EndPt)
fileStream = New FileStream(FilePath, FileMode.Open, FileAccess.Read)
Dim rr As New PutObjectRequest() With {.BucketName = BucketPath,
.Key = Path.GetFileName(FilePath),
.InputStream = fileStream
}
Dim response As PutObjectResponse = s3Client.PutObject(rr)
Output = response.HttpStatusCode = HttpStatusCode.OK
Catch ex As Exception
Throw ex
Finally
If fileStream IsNot Nothing Then
fileStream.Close()
fileStream.Dispose()
End If
If s3Client IsNot Nothing Then
s3Client.Dispose()
End If
End Try
Else
Throw New FileNotFoundException($"File {Path.GetFileName(FilePath)} was not found")
End If
Else
Throw New ArgumentNullException("Password cannot be blank or all spaces")
End If
Else
Throw New ArgumentNullException("UserName cannot be blank or all spaces")
End If
Return Output
End Function
Re: Read 4gb file into FileStream and copy to MemoryStream
:thumb:
sweet! Glad it worked.
-tg
