-
Jul 20th, 2012, 09:52 AM
#1
Thread Starter
Hyperactive Member
[RESOLVED] download single file in segment .. error
below is the code for downloading file into segment ..
vb Code:
Public Class part_down Const ChunkSizeConst As Integer = 1024 * 4 Public Delegate Sub _delprogressupdate(ByVal update As String, ByVal row As Integer) Public Event progressreport As _delprogressupdate Public start As Int32 Public finish As Int32 Public path As String Public url As String Public row As Integer Public Sub part_download2() Dim req As HttpWebRequest Dim resp As Net.HttpWebResponse req = WebRequest.Create(url) req.UserAgent = "NotZilla" req.Timeout = -1 req.AddRange("bytes", start, finish) resp = req.GetResponse Dim netstream As IO.Stream = resp.GetResponseStream Dim Buffer(ChunkSizeConst - 1) As Byte Dim BytesRead As Long Dim totalbytesread As Long = 0 Dim file As FileStream If start = 0 Then file = New FileStream(path, FileMode.OpenOrCreate, FileAccess.Write, FileShare.ReadWrite) Else file = New FileStream(path, FileMode.Append, FileAccess.Write, FileShare.ReadWrite) End If Do BytesRead = netstream.Read(Buffer, 0, ChunkSizeConst) file.Write(Buffer, 0, BytesRead) 'RaiseEvent writetodisk(Buffer, BytesRead, file) totalbytesread += BytesRead RaiseEvent progressreport((totalbytesread * 100 / resp.ContentLength).ToString("n2") & "%", row) Loop While BytesRead > 0 End Sub End Class
in form1
vb Code:
Private Function DownLoad(ByVal url As String, ByVal path As String) 'create a request Dim req As Net.HttpWebRequest = DirectCast(Net.WebRequest.Create(url), Net.HttpWebRequest) req.UserAgent = "NotZilla" req.AddRange(0) Dim resp As Net.HttpWebResponse = DirectCast(req.GetResponse, Net.HttpWebResponse) Dim thr(NumThreads - 1) As Thread Dim thrclass(NumThreads - 1) As part_down 'get a response Dim size As Long = resp.ContentLength Dim block As Int32 = size \ NumThreads + ChunkSizeConst - ((size \ NumThreads) Mod ChunkSizeConst) 'Dim support_multi As Boolean = False 'get its stream 'Dim BufString As New System.Text.StringBuilder() Dim i As Integer For i = 0 To NumThreads - 1 thrclass(i) = New part_down() thrclass(i).url = url thrclass(i).path = path.Replace(IO.Path.GetFileNameWithoutExtension(path), IO.Path.GetFileNameWithoutExtension(path) & i.ToString) thrclass(i).row = i AddHandler thrclass(i).progressreport, AddressOf statusupdate Next thrclass(0).start = 0 thrclass(0).finish = block - 1 If resp.StatusCode = Net.HttpStatusCode.PartialContent Then ' support_multi = True For i = 0 To NumThreads - 1 thr(i) = New Thread(AddressOf thrclass(i).part_download) thr(i).Start() addnewitemto_listview(SetBytes(thrclass(i).finish - thrclass(i).start)) 'part_download() If i < NumThreads - 1 Then thrclass(i + 1).start = thrclass(i).finish If i = NumThreads - 2 Then thrclass(i + 1).finish = size Else thrclass(i + 1).finish = thrclass(i).finish + block - 1 'thrclass(i + 1).finish += block - 1 End If End If Next For i = 0 To NumThreads - 1 thr(i).Join() Next MsgBox("Completed") End If End Function
after that i Applied binary joining suggested by stanav
Originally Posted by stanav
Try something like this:
Code:
Private Sub JoinFiles(ByVal fileChunks As List(Of String), ByVal output As String, Optional ByVal deleteFileChunksWhenDone As Boolean = False)
Try
Using outStream As New IO.FileStream(output, System.IO.FileMode.Create, System.IO.FileAccess.Write, System.IO.FileShare.None)
Using writer As New IO.BinaryWriter(outStream)
For i As Integer = 0 To fileChunks.Count - 1
Using inStream As New IO.FileStream(fileChunks(i), System.IO.FileMode.Open)
Using reader As New IO.BinaryReader(inStream)
writer.Write(reader.ReadBytes(reader.BaseStream.Length))
End Using
End Using
Next
End Using
End Using
If deleteFileChunksWhenDone = True Then
For Each chunk As String In fileChunks
IO.File.Delete(chunk)
Next
End If
Catch ex As Exception
MessageBox.Show(ex.Message())
End Try
End Sub
Usage example:
Code:
'Create a list of files to be joined. The files are added to the list in the order to be joined.
Dim files2Join As New List(Of String)
files2Join.Add("C:\test\New Folder\New Folder\hehe1.flv")
files2Join.Add("C:\test\New Folder\New Folder\hehe2.flv")
files2Join.Add("C:\test\New Folder\New Folder\hehe3.flv")
'Set a path for the joined output file
Dim outputFile As String = "C:\test\New Folder\New Folder\hehe.flv"
'Join the files and delete the original files when done.
JoinFiles(files2Join, outputFile, True)
But after joinng output was not right
dunno what's wrong
thanks
Last edited by CatchItBaby; Jul 20th, 2012 at 10:03 AM.
-
Jul 20th, 2012, 10:19 AM
#2
Re: download single file in segment .. error
I believe this part in the part_download2 method is what causes the corruption of the file after joining:
Code:
Dim file As FileStream
If start = 0 Then
file = New FileStream(path, FileMode.OpenOrCreate, FileAccess.Write, FileShare.ReadWrite)
Else
file = New FileStream(path, FileMode.Append, FileAccess.Write, FileShare.ReadWrite)
End If
Besides, the whole thing is very poorly written...
Please try this code I just wrote. Since I'm at work right now I our firewall blocks all video sites, I can't test download a video file. However, I downloaded a few zip files using it and it seems to work correctly.
Anyway, if you found a bug in my code, please report back and I'm more than happy to try to help you fix it (when I can, of course )
This is the Download class
vb Code:
Imports System.Net Imports System.IO Public Class Download Const ChunkSize As Integer = 1024 * 4 Public StartPosition As Long = 0 Public DownloadLength As Long = 0 Public FilePath As String = String.Empty Public DownloadUrl As String = String.Empty Public Name As String = String.Empty Public DownloadResponse As HttpWebResponse = Nothing Public DownloadStream As IO.Stream = Nothing Public IsComplete As Boolean = False Public Delegate Sub _delprogressupdate(ByVal sender As Object, ByVal e As Download.DownloadProgress) Public Event ProgressReport As _delprogressupdate Public Delegate Sub _deldownloadcompleted(ByVal sender As Object, ByVal e As Download.DownloadProgress) Public Event DownloadCompleted As _deldownloadcompleted Public Sub DownloadChunk() If Not String.IsNullOrEmpty(DownloadUrl) Then Dim req As HttpWebRequest = Net.HttpWebRequest.Create(DownloadUrl) With req .UserAgent = "NotZilla" req.Timeout = -1 req.AddRange("bytes", StartPosition, StartPosition + DownloadLength) End With DownloadResponse = req.GetResponse() DownloadStream = DownloadResponse.GetResponseStream() DownloadIt() End If End Sub Public Sub DownloadIt() Dim buffer(ChunkSize - 1) As Byte Dim bytesRead As Long = 0 Dim totalBytesRead As Long = 0 Dim stopFlag As Boolean = False Using fs As New IO.FileStream(FilePath, FileMode.Create, FileAccess.Write, FileShare.None) Do bytesRead = DownloadStream.Read(buffer, 0, ChunkSize) fs.Write(buffer, 0, bytesRead) totalBytesRead += bytesRead RaiseEvent ProgressReport(Me, New DownloadProgress(Me.FilePath, totalBytesRead, DownloadLength - totalBytesRead)) Loop While totalBytesRead < DownloadLength End Using DownloadStream.Close() DownloadResponse.Close() Me.IsComplete = True RaiseEvent DownloadCompleted(Me, New DownloadProgress(Me.FilePath, totalBytesRead, DownloadLength - totalBytesRead)) End Sub Public Class DownloadProgress Private _filePath As String Private _bytesDownloaded As Long Private _bytesRemain As Long Private _percentCompleted As Double Public ReadOnly Property FilePath() As String Get Return _filePath End Get End Property Public ReadOnly Property BytesDownloaded() As Long Get Return _bytesDownloaded End Get End Property Public ReadOnly Property BytesRemaining() As Long Get Return _bytesRemain End Get End Property Public ReadOnly Property PercentCompleted() As Double Get Return _percentCompleted End Get End Property Public Sub New(ByVal path As String, ByVal bytesRead As Long, ByVal bytesRemaining As Long) _filePath = path _bytesDownloaded = bytesRead _bytesRemain = bytesRemaining _percentCompleted = bytesRead * 100 / (bytesRead + bytesRemaining) End Sub End Class End Class
And usage implementation. You can paste the whole thing in your form's code
vb Code:
Private downloadList As New List(Of Download) Private Sub DoParallelDownload(ByVal url As String, ByVal saveTo As String) Me.downloadList.Clear() 'We 1st need to find out how big the file is and whether the download can be split Dim req As Net.HttpWebRequest = Net.HttpWebRequest.Create(url) req.UserAgent = "NotZilla" req.AddRange(0) Dim resp As Net.HttpWebResponse = req.GetResponse() Dim downloadSize As Long = resp.ContentLength Dim netStream As IO.Stream = resp.GetResponseStream() If resp.StatusCode = Net.HttpStatusCode.PartialContent Then 'Yes, we can split this download Dim numberOfThreads As Integer = 5 Dim aDownload As Download = Nothing Dim thrds(numberOfThreads - 1) As Threading.Thread Dim thdStart As Threading.ThreadStart = Nothing Dim downloadLength As Long = downloadSize \ numberOfThreads For i As Integer = 0 To numberOfThreads - 1 aDownload = New Download() aDownload.DownloadUrl = url aDownload.FilePath = saveTo & ".dlpart" & i.ToString("000") aDownload.StartPosition = i * downloadLength aDownload.DownloadLength = downloadLength aDownload.Name = IO.Path.GetFileName(aDownload.FilePath) If i = numberOfThreads - 1 Then aDownload.DownloadLength += downloadSize Mod numberOfThreads End If AddHandler aDownload.ProgressReport, AddressOf Download_ProgressReport AddHandler aDownload.DownloadCompleted, AddressOf Download_Completed If i = 0 Then aDownload.DownloadResponse = resp aDownload.DownloadStream = netStream thdStart = New Threading.ThreadStart(AddressOf aDownload.DownloadIt) Else thdStart = New Threading.ThreadStart(AddressOf aDownload.DownloadChunk) End If thrds(i) = New Threading.Thread(thdStart) thrds(i).Start() Me.downloadList.Add(aDownload) Next Else 'No, we can't split this download so we have to download it using 1 thread. Dim dl As New Download() dl.DownloadUrl = url dl.FilePath = saveTo dl.StartPosition = 0 dl.DownloadLength = downloadSize dl.Name = IO.Path.GetFileName(dl.FilePath) dl.DownloadResponse = resp dl.DownloadStream = netStream AddHandler dl.ProgressReport, AddressOf Download_ProgressReport AddHandler dl.DownloadCompleted, AddressOf Download_Completed Dim thdStart As New Threading.ThreadStart(AddressOf dl.DownloadIt) Dim thd As New Threading.Thread(thdStart) thd.Start() Me.downloadList.Add(dl) End If End Sub Private Sub Download_ProgressReport(ByVal sender As Object, ByVal e As Download.DownloadProgress) 'Your own code to report progress here. This progress report is for 1 chunk. If you want the overall progress, you'll have to calculate it yourself. Dim progress As String = String.Format("Download status for file {0}{4}BytesDownloaded: {1}{4}BytesRemaining: {2}{4}PercentCompleted: {3}{4}", _ IO.Path.GetFileName(e.FilePath), e.BytesDownloaded, e.BytesRemaining, e.PercentCompleted, Environment.NewLine) Debug.Write(progress) End Sub Private Sub Download_Completed(ByVal sender As Object, ByVal e As Download.DownloadProgress) Dim filePath As String = e.FilePath Dim doneCount As Integer = 0 Dim fileList As New List(Of String) For Each item As Download In Me.downloadList If item.FilePath = filePath Then item.IsComplete = True End If If item.IsComplete = True Then doneCount += 1 End If fileList.Add(item.FilePath) Next If doneCount = Me.downloadList.Count Then If doneCount = 1 Then 'This file was downloaded in whole. No need to do the joining. Else 'All file chunks are downloaded. Start joining them using the DownloadList to get the file paths of those chunks fileList.Sort() Dim folder As String = IO.Path.GetDirectoryName(filePath) Dim fileName As String = IO.Path.GetFileNameWithoutExtension(filePath) Me.JoinFiles(fileList, IO.Path.Combine(folder, fileName), True) Debug.WriteLine("Done joining files") End If End If End Sub Private Sub JoinFiles(ByVal fileChunks As List(Of String), ByVal output As String, Optional ByVal deleteFileChunksWhenDone As Boolean = False) Try Using outStream As New IO.FileStream(output, System.IO.FileMode.Create, System.IO.FileAccess.Write, System.IO.FileShare.None) Using writer As New IO.BinaryWriter(outStream) For i As Integer = 0 To fileChunks.Count - 1 Using inStream As New IO.FileStream(fileChunks(i), System.IO.FileMode.Open) Using reader As New IO.BinaryReader(inStream) writer.Write(reader.ReadBytes(reader.BaseStream.Length)) End Using End Using Next End Using End Using If deleteFileChunksWhenDone = True Then For Each chunk As String In fileChunks IO.File.Delete(chunk) Next End If Catch ex As Exception MessageBox.Show(ex.Message()) End Try End Sub
And finally, usage example:
Code:
'In a button click or something similar, you do:
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
Dim url As String = "the url to the file to be downloaded here"
Dim savePath As String = "the full path to where it should be saved after done done loading here"
Me.DoParallelDownload(url, savePath)
End Sub
Enjoy
Let us have faith that right makes might, and in that faith, let us, to the end, dare to do our duty as we understand it.
- Abraham Lincoln -
-
Jul 20th, 2012, 10:56 AM
#3
Thread Starter
Hyperactive Member
Re: download single file in segment .. error
having same problem
- after joining video file was not playing (or stopped in middle .. played via vlc media player)
- and i tried one rar archive and after completition when i opened rar file it shows unexpected end or archive error
-
Jul 20th, 2012, 11:26 AM
#4
Re: download single file in segment .. error
You you have a download url that I can use to test?
Let us have faith that right makes might, and in that faith, let us, to the end, dare to do our duty as we understand it.
- Abraham Lincoln -
-
Jul 20th, 2012, 11:59 AM
#5
Re: download single file in segment .. error
OK... I think I found where the bug was and corrected it. Although I haven't tested downloading a .rar or video file, but it looks good with zip files. The MD5 hash of the downloaded file matches that of the original file.
And here is the revised code:
vb Code:
Imports System.Net Imports System.IO Public Class Download Const ChunkSize As Integer = 1024 * 4 Public StartPosition As Long = 0 Public DownloadLength As Long = 0 Public FilePath As String = String.Empty Public DownloadUrl As String = String.Empty Public Name As String = String.Empty Public DownloadResponse As HttpWebResponse = Nothing Public DownloadStream As IO.Stream = Nothing Public IsComplete As Boolean = False Public Delegate Sub _delprogressupdate(ByVal sender As Object, ByVal e As Download.DownloadProgress) Public Event ProgressReport As _delprogressupdate Public Delegate Sub _deldownloadcompleted(ByVal sender As Object, ByVal e As Download.DownloadProgress) Public Event DownloadCompleted As _deldownloadcompleted Public Sub DownloadChunk() If Not String.IsNullOrEmpty(DownloadUrl) Then Dim req As HttpWebRequest = Net.HttpWebRequest.Create(DownloadUrl) With req .UserAgent = "NotZilla" req.Timeout = -1 req.AddRange("bytes", StartPosition, StartPosition + DownloadLength) End With DownloadResponse = req.GetResponse() DownloadStream = DownloadResponse.GetResponseStream() DownloadIt() End If End Sub Public Sub DownloadIt() Dim buffer(ChunkSize - 1) As Byte Dim bytes2Download As Long = ChunkSize Dim bytesRemain As Long = DownloadLength Dim bytesRead As Long = 0 Dim totalBytesRead As Long = 0 Using fs As New IO.FileStream(FilePath, FileMode.Create, FileAccess.Write, FileShare.None) Do bytesRemain = DownloadLength - totalBytesRead If bytesRemain < ChunkSize Then bytes2Download = bytesRemain End If bytesRead = DownloadStream.Read(buffer, 0, bytes2Download) fs.Write(buffer, 0, bytesRead) totalBytesRead += bytesRead RaiseEvent ProgressReport(Me, New DownloadProgress(Me.FilePath, totalBytesRead, DownloadLength - totalBytesRead)) Loop While totalBytesRead < DownloadLength End Using DownloadStream.Close() DownloadResponse.Close() Me.IsComplete = True RaiseEvent DownloadCompleted(Me, New DownloadProgress(Me.FilePath, totalBytesRead, DownloadLength - totalBytesRead)) End Sub Public Class DownloadProgress Private _filePath As String Private _bytesDownloaded As Long Private _bytesRemain As Long Private _percentCompleted As Double Public ReadOnly Property FilePath() As String Get Return _filePath End Get End Property Public ReadOnly Property BytesDownloaded() As Long Get Return _bytesDownloaded End Get End Property Public ReadOnly Property BytesRemaining() As Long Get Return _bytesRemain End Get End Property Public ReadOnly Property PercentCompleted() As Double Get Return _percentCompleted End Get End Property Public Sub New(ByVal path As String, ByVal bytesRead As Long, ByVal bytesRemaining As Long) _filePath = path _bytesDownloaded = bytesRead _bytesRemain = bytesRemaining _percentCompleted = bytesRead * 100 / (bytesRead + bytesRemaining) End Sub End Class End Class
And the implementation code (some changes were made in the thread creating loop and the joinfiles sub)
vb Code:
Private downloadList As New List(Of Download) Private Sub DoParallelDownload(ByVal url As String, ByVal saveTo As String) Me.downloadList.Clear() 'We 1st need to find out how big the file is and whether the download can be split Dim req As Net.HttpWebRequest = Net.HttpWebRequest.Create(url) req.UserAgent = "NotZilla" req.AddRange(0) Dim resp As Net.HttpWebResponse = req.GetResponse() Dim downloadSize As Long = resp.ContentLength Dim netStream As IO.Stream = resp.GetResponseStream() If resp.StatusCode = Net.HttpStatusCode.PartialContent Then 'Yes, we can split this download Dim numberOfThreads As Integer = 5 Dim aDownload As Download = Nothing Dim thrds(numberOfThreads - 1) As Threading.Thread Dim thdStart As Threading.ThreadStart = Nothing Dim downloadLength As Long = downloadSize \ numberOfThreads For i As Integer = 0 To numberOfThreads - 1 aDownload = New Download() aDownload.DownloadUrl = url aDownload.FilePath = saveTo & "." & (i + 1).ToString("000") aDownload.StartPosition = i * downloadLength aDownload.DownloadLength = downloadLength aDownload.Name = IO.Path.GetFileName(aDownload.FilePath) If i = numberOfThreads - 1 Then aDownload.DownloadLength += downloadSize Mod numberOfThreads End If AddHandler aDownload.ProgressReport, AddressOf Download_ProgressReport AddHandler aDownload.DownloadCompleted, AddressOf Download_Completed If i = 0 Then aDownload.DownloadResponse = resp aDownload.DownloadStream = netStream thdStart = New Threading.ThreadStart(AddressOf aDownload.DownloadIt) Else thdStart = New Threading.ThreadStart(AddressOf aDownload.DownloadChunk) End If thrds(i) = New Threading.Thread(thdStart) thrds(i).Start() Me.downloadList.Add(aDownload) Next Else 'No, we can't split this download so we have to download it using 1 thread. Dim dl As New Download() dl.DownloadUrl = url dl.FilePath = saveTo dl.StartPosition = 0 dl.DownloadLength = downloadSize dl.Name = IO.Path.GetFileName(dl.FilePath) dl.DownloadResponse = resp dl.DownloadStream = netStream AddHandler dl.ProgressReport, AddressOf Download_ProgressReport AddHandler dl.DownloadCompleted, AddressOf Download_Completed Dim thdStart As New Threading.ThreadStart(AddressOf dl.DownloadIt) Dim thd As New Threading.Thread(thdStart) thd.Start() Me.downloadList.Add(dl) End If End Sub Private Sub Download_ProgressReport(ByVal sender As Object, ByVal e As Download.DownloadProgress) 'Your own code to report progress here. This progress report is for 1 chunk. If you want the overall progress, you'll have to calculate it yourself. Dim progress As String = String.Format("Download status for file {0}{4}BytesDownloaded: {1}{4}BytesRemaining: {2}{4}PercentCompleted: {3}{4}", _ IO.Path.GetFileName(e.FilePath), e.BytesDownloaded, e.BytesRemaining, e.PercentCompleted, Environment.NewLine) Debug.Write(progress) End Sub Private Sub Download_Completed(ByVal sender As Object, ByVal e As Download.DownloadProgress) Dim filePath As String = e.FilePath Dim doneCount As Integer = 0 Dim fileList As New List(Of String) For Each item As Download In Me.downloadList If item.FilePath = filePath Then item.IsComplete = True End If If item.IsComplete = True Then doneCount += 1 End If fileList.Add(item.FilePath) Next If doneCount = Me.downloadList.Count Then If doneCount = 1 Then 'This file was downloaded in whole. No need to do the joining. Else 'All file chunks are downloaded. Start joining them using the DownloadList to get the file paths of those chunks fileList.Sort() Dim folder As String = IO.Path.GetDirectoryName(filePath) Dim fileName As String = IO.Path.GetFileNameWithoutExtension(filePath) Me.JoinFiles(fileList, IO.Path.Combine(folder, fileName), False) Debug.WriteLine("Done joining files") End If End If End Sub Private Sub JoinFiles(ByVal fileChunks As List(Of String), ByVal output As String, Optional ByVal deleteFileChunksWhenDone As Boolean = False) Try Using outStream As New IO.FileStream(output, System.IO.FileMode.Create, System.IO.FileAccess.Write, System.IO.FileShare.None) For i As Integer = 0 To fileChunks.Count - 1 Using inStream As New IO.FileStream(fileChunks(i), System.IO.FileMode.Open) Dim buffer(inStream.Length - 1) As Byte inStream.Read(buffer, 0, buffer.Length) outStream.Write(buffer, 0, buffer.Length) End Using Next End Using If deleteFileChunksWhenDone = True Then For Each chunk As String In fileChunks IO.File.Delete(chunk) Next End If Catch ex As Exception MessageBox.Show(ex.Message()) End Try End Sub
Please try again with this new code and give any feedback you have.
Let us have faith that right makes might, and in that faith, let us, to the end, dare to do our duty as we understand it.
- Abraham Lincoln -
-
Jul 20th, 2012, 01:00 PM
#6
Thread Starter
Hyperactive Member
Re: [RESOLVED] download single file in segment .. error
Thanks working fine now
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
Click Here to Expand Forum to Full Width
|