[2008] Multithreading

[youngbucks]

I have a code here to download a list of files (Same one i posted in CodeBank) but i currently use it to download huge files but it freezes and im thinking i should move it to a backgroundworker, and i want to know how would i report the progress accurately and same way i do here while multithreading?

Imports System.IO
Imports System.Net
Imports Microsoft.VisualBasic.FileIO.FileSystem
 
Public Class Form1
    Private Const sfilename As String = "c:\Users\Home\Desktop\list_of_files.txt"
    Private Const url As String = "http://www.hotlinkfiles.com/files"
    Private Const spath As String = "c:\Users\Home\Desktop\test"
    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        'CHECK FOR LIST
        If Not My.Computer.FileSystem.FileExists(sfilename) Then
            MsgBox("No list found, exiting application", MsgBoxStyle.Critical)
            Exit Sub
            Application.Exit()
        End If
        Dim sr As New IO.StreamReader(sfilename)
        Dim line As String = sr.ReadLine()
        Dim req As Net.WebRequest
        Dim resp As IO.Stream
        Dim out As IO.BinaryWriter
        Do While line IsNot Nothing
            req = Net.HttpWebRequest.Create(url & line)
            Dim progmax As Net.HttpWebResponse = req.GetResponse()
            resp = req.GetResponse().GetResponseStream()
            '===========================
            'CREATES DIRECTORY IF NEEDED BEFORE IT CALLS TO OPEN A NEW FILESTREAM AND THERE IS NONE
            '===========================
            Dim filepath As String = spath & line
            If Not Directory.Exists(filepath.Replace(Path.GetFileName(filepath), String.Empty)) Then Directory.CreateDirectory(filepath.Replace(Path.GetFileName(filepath), String.Empty))
            '===========================
            out = New IO.BinaryWriter(New IO.FileStream(spath & line, IO.FileMode.OpenOrCreate))
            Dim buf(4096) As Byte
            Dim k As Int32 = resp.Read(buf, 0, 4096)
            Do While k > 0
                out.Write(buf, 0, k)
                k = resp.Read(buf, 0, 4096)
                '===================
                'READ FROM THE BUFFER AND REPORT PROGRESS TO THE PROGBAR
                '===================
                Progbar.Maximum = progmax.ContentLength
                If Progbar.Value + k <= Progbar.Maximum Then
                    Progbar.Value += k
                Else
                    Progbar.Value = Progbar.Maximum
                End If
            Loop
 
            resp.Close()
            out.Close()
            line = sr.ReadLine()
        Loop
    End Sub
End Class

[Krenshau]

You would use a delegate to trigger the update of the progres bar.

This article might help.

http://www.developer.com/net/net/article.php/3555551

[youngbucks]

I know i would use a delegate but im trying to find out how exactly im going to run this information correctly through a background worker.

If Progbar.Value + k <= Progbar.Maximum Then
                    Progbar.Value += k
                Else
                    Progbar.Value = Progbar.Maximum
                End If

In that code it all runs on one thread and all the calculation for the progress is done there also so im trying to figure out how to multithread but have the same information being passed around.

[jmcilhinney]

What you're asking for is the primary reason for the backgroundWorker's existence. The class has a ReportProgress method and a ProgressChanged event specifically for the purpose. You calculate the current progress on the worker thread and then call ReportProgress. That raises the ProgressChanged event on the UI thread so you can update a ProgressBar in the event handler. Follow the BackgroundWorker link in my signature for an example.

[youngbucks]

How would i pass the maximum progressbar amount using the backgroundworker jmcilhinney? I looked at your example but it doesnt declare a progressbar.maximum as it seems ur using the default 100.

[jmcilhinney]

How would i pass the maximum progressbar amount using the backgroundworker jmcilhinney? I looked at your example but it doesnt declare a progressbar.maximum as it seems ur using the default 100.

You really don't need to. Just use 100 as the maximum and calculate the current progress as a percentage. If you really want to use an absolute maximum then you can pass that to the second parameter of ReportProgress.

[youngbucks]

Whats the second parameter? This?

BackgroundWorker1.ReportProgress(k, progressbarmaximum)

[jmcilhinney]

Have you read the MSDN documentation for the ReportProgress method and the ProgressChanged event?

[youngbucks]

I just did and mustve overlooked because i found nothing about second parameter. When i read the ReportProgress it talkes about reporting the progress as an integer and userstate as a string. As for the progresschanged only about rasing the event of progress changed.

[jmcilhinney]

The second parameter can be anything you want. It is type Object specifically for that purpose. You may have seen an example that passes a String but that's just an example. If you want to pass an Integer containing the maximum progress then that's exactly what you pass. If you want to pass a ListViewItem that was created from the field values of a DataRow then that's exactly what you pass. Etc., etc.

[youngbucks]

How would i pass a integer as it though? I tried this and it doent work so how?

Copyingbackgroundworker.ReportProgress(k, 100)

And

ProgressBar1.Maximum = TryCast(e.UserState, Integer)

[jmcilhinney]

Um, yes it does work. That code isn't going to make any change though because you're setting the Maximum to 100, which is the same value it already is. Try this:

1. Create a new WinForms project.
2. Add two Buttons, a ProgressBar and a BackgroundWorker.
3. Set the BGW's WorkerReportsProgress property to True.
4. Add this code:

Private Sub Button1_Click(ByVal sender As System.Object, _
                          ByVal e As System.EventArgs) Handles Button1.Click
    MessageBox.Show(Me.ProgressBar1.Maximum.ToString())
End Sub

Private Sub Button2_Click(ByVal sender As System.Object, _
                          ByVal e As System.EventArgs) Handles Button2.Click
    Me.BackgroundWorker1.RunWorkerAsync()
End Sub

Private Sub BackgroundWorker1_DoWork(ByVal sender As System.Object, _
                                     ByVal e As System.ComponentModel.DoWorkEventArgs) Handles BackgroundWorker1.DoWork
    Me.BackgroundWorker1.ReportProgress(0, 999)
End Sub

Private Sub BackgroundWorker1_ProgressChanged(ByVal sender As Object, _
                                              ByVal e As System.ComponentModel.ProgressChangedEventArgs) Handles BackgroundWorker1.ProgressChanged
    Me.ProgressBar1.Maximum = CInt(e.UserState)
End Sub

5. Run the project.
6. Click Button1 and observe the value.
7. Click Button2.
8. Click Button1 again and observe the value.

Tada!

[youngbucks]

I got that to work here but im having another problem, i get to change the max value but its changed too late so the actual progress seen is never more that 1%.

Imports System.IO
Imports System.Net
Imports Microsoft.VisualBasic.FileIO.FileSystem
 
Public Class Form1
    Private Const sfilename As String = "c:\Users\Home\Desktop\list_of_files.txt"
    Private Const url As String = "http://www.hotlinkfiles.com/files"
    Private Const spath As String = "c:\Users\Home\Desktop\test"
 
    Private Sub BackgroundWorker1_DoWork(ByVal sender As System.Object, ByVal e As System.ComponentModel.DoWorkEventArgs) Handles BackgroundWorker1.DoWork
        Try
            'CHECK FOR LIST
            If Not My.Computer.FileSystem.FileExists(sfilename) Then
                MsgBox("No list found, exiting application", MsgBoxStyle.Critical)
                Exit Sub
                Application.Exit()
            End If
            Dim sr As New IO.StreamReader(sfilename)
            Dim line As String = sr.ReadLine()
            Dim req As Net.WebRequest
            Dim resp As IO.Stream
            Dim out As IO.BinaryWriter
            Do While line IsNot Nothing
                req = Net.HttpWebRequest.Create(url & line)
                Dim progmax As Net.HttpWebResponse = req.GetResponse()
                resp = req.GetResponse().GetResponseStream()
                '===========================
                'CREATES DIRECTORY IF NEEDED BEFORE IT CALLS TO OPEN A NEW FILESTREAM AND THERE IS NONE
                '===========================
                Dim filepath As String = spath & line
                If Not Directory.Exists(filepath.Replace(Path.GetFileName(filepath), String.Empty)) Then Directory.CreateDirectory(filepath.Replace(Path.GetFileName(filepath), String.Empty))
                '===========================
                out = New IO.BinaryWriter(New IO.FileStream(spath & line, IO.FileMode.OpenOrCreate))
                Dim buf(4096) As Byte
                Dim k As Int32 = resp.Read(buf, 0, 4096)
                Do While k > 0
                    out.Write(buf, 0, k)
                    k = resp.Read(buf, 0, 4096)
                    '===================
                    'READ FROM THE BUFFER AND REPORT PROGRESS TO THE PROGBAR
                    '===================
                    Dim max As Integer = progmax.ContentLength
                    'ProgBar.Maximum = progmax.ContentLength
                    'If ProgBar.Value + k <= ProgBar.Maximum Then
                    BackgroundWorker1.ReportProgress(k + k, max)
                    'ProgBar.Value += k
                    'Else
                    'BackgroundWorker1.ReportProgress(max, max)
                    ' ProgBar.Value = ProgBar.Maximum
                    'End If
                Loop
 
                resp.Close()
                out.Close()
                line = sr.ReadLine()
            Loop
        Catch ex As Exception
            MsgBox(ex.ToString)
            Exit Sub
        End Try
    End Sub
 
    Private Sub BackgroundWorker1_ProgressChanged(ByVal sender As System.Object, ByVal e As System.ComponentModel.ProgressChangedEventArgs) Handles BackgroundWorker1.ProgressChanged
        Try
            Me.ProgBar.Maximum = CInt(e.UserState)
            ProgBar.Value = e.ProgressPercentage
        Catch ex As Exception
            MsgBox(ex.ToString)
        End Try
    End Sub
 
    Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
        BackgroundWorker1.RunWorkerAsync()
    End Sub
End Class

[jmcilhinney]

Why do you have exception handling in your ProgressChanged event handler? What possible exception could you be handling? You only add exception handling when an exception can reasonably be thrown, which it can't in that case.

Why do you need to get the maximum progress each time through the loop? Once it's set is it ever going to change? No it's not, so you only need to set it once.

Try calling Refresh on your ProgressBar in the ProgressChanged event handler to force it to repaint.

[youngbucks]

Refresh doesnt seem to help at all and i put "BackgroundWorker1.ReportProgress(k + k, max)" to report progress as ive seen in your example so i am unaware of any other way to set the progressbar maximum than as u have seen.

[jmcilhinney]

Think outside the box. The only way to get information to the ProgressChanged event handler is to call ReportProgress but who says you have to pass the maximum value every time?

Public Class Form1
 
    Private Sub BackgroundWorker1_DoWork(ByVal sender As System.Object, _
                                         ByVal e As System.ComponentModel.DoWorkEventArgs) Handles BackgroundWorker1.DoWork
        Me.BackgroundWorker1.ReportProgress(0, 999)
 
        For i As Integer = 1 To 999
            Me.BackgroundWorker1.ReportProgress(i)
        Next
    End Sub
 
    Private Sub BackgroundWorker1_ProgressChanged(ByVal sender As Object, _
                                                  ByVal e As System.ComponentModel.ProgressChangedEventArgs) Handles BackgroundWorker1.ProgressChanged
        If e.UserState Is Nothing Then
            Me.ProgressBar1.Value = e.ProgressPercentage
        Else
            Me.ProgressBar1.Maximum = CInt(e.UserState)
        End If
 
        Me.ProgressBar1.Refresh()
    End Sub
 
    Private Sub Form1_Shown(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Shown
        Me.BackgroundWorker1.RunWorkerAsync()
    End Sub
 
End Class

[youngbucks]

I fixed my code to match what you have up there but now when i run it the form hampers drastically, it freezes majority of the time and 1/10 it unfreezes and you see a big chunk of the progress change so i cant keep up to see if its reporting the progress properly. Adding Me.Refresh and ProgressBar1.Refresh does nothing. Could it be that the values im working with are too big?

[jmcilhinney]

This is what debugging is for. Add some code that will output the time just before calling ReportProgress, then watch the Output window to see whether what you see there matches what you see in the UI. If you want to know if your chunks are too big then test with smaller chunks. Do what you can for yourself first because we can only do so much without seeing the code in action.

[youngbucks]

Question, how come when i use kleinma's downloader it doesnt freeze when i set it to download huge files? Is it cuz its a class? Im thinking of modifying it to work like i have up top but without the lag.

[youngbucks]

I used kleinma's webfiledownloader class and modified it to download the list of files without a backgroundworker and it downloads huge files one after each other without any lag or delay. Why did mine lag and this one didnt? :S

This is the code i currently use.

Imports System.IO
 
Public Class form1
    Private Processcompletion As Boolean = True
    Private Const url As String = "http://www.hotlinkfiles.com/files"
    Private Const txtdownloadto As String = "C:\Users\Home\Desktop\test"
    Private SR As New IO.StreamReader("C:\users\Home\Desktop\list_of_files.txt")
    Private WithEvents _Downloader As WebFileDownloader
    Private Sub cmdDownload_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles cmdDownload.Click
        Dim currentline As String = SR.ReadLine()
        Do While currentline IsNot Nothing
            Try
                If Not Processcompletion = True Then Application.Exit()
                Dim filepath As String = txtdownloadto & currentline
                If Not Directory.Exists(filepath.Replace(Path.GetFileName(filepath), String.Empty)) Then Directory.CreateDirectory(filepath.Replace(Path.GetFileName(filepath), String.Empty))
                Dim newurl As String = url & currentline.Replace("\", "/")
                Dim destination As String = txtdownloadto & currentline
                _Downloader = New WebFileDownloader
                _Downloader.DownloadFileWithProgress(newurl, destination)
                currentline = SR.ReadLine()
            Catch ex As Exception
                MsgBox(ex.Message)
            End Try
        Loop
    End Sub
    Private Sub _Downloader_FileDownloadSizeObtained(ByVal iFileSize As Long) Handles _Downloader.FileDownloadSizeObtained
        ProgBar.Value = 0
        ProgBar.Maximum = Convert.ToInt32(iFileSize)
    End Sub
    Private Sub _Downloader_FileDownloadComplete() Handles _Downloader.FileDownloadComplete
        ProgBar.Value = ProgBar.Maximum
        Processcompletion = True
    End Sub
    Private Sub _Downloader_FileDownloadFailed(ByVal ex As System.Exception) Handles _Downloader.FileDownloadFailed
        Processcompletion = False
        MsgBox("An error has occured during download: " & ex.Message)
    End Sub
    Private Sub _Downloader_AmountDownloadedChanged(ByVal iNewProgress As Long) Handles _Downloader.AmountDownloadedChanged
        ProgBar.Value = Convert.ToInt32(iNewProgress)
        lblProgress.Text = WebFileDownloader.FormatFileSize(iNewProgress) & " of " & WebFileDownloader.FormatFileSize(ProgBar.Maximum) & " downloaded"
        Application.DoEvents()
    End Sub
End Class

Edit, i went back through the code looking at the difference and it seems application.doevents() was the trick to not freezing. Why?

[jmcilhinney]

Going back to your own code, here's how you were calling ReportProgress:

BackgroundWorker1.ReportProgress(k + k, max)

That's not right though because k is not the total amount read so far. It's just the amount read in the last pass. That amount will be the same every pass until the last one so your ProgressBar is always going to show the same value.

It's for reasons like this that you MUST debug. You cannot just assume that you're using the correct data and you can't always see that you're not just by looking at the code. You have to run the code and see what values your variables contain at the time. Doing that you'd have seen that you were simply setting the Value of the ProgressBar to 8192 every time and never increasing it.

Now, those comments are based on the most recent code you posted. If you'd followed mu instructions you should have changed that code since then but I don't know if you had changed that aspect of it.

[Shaggy Hiker]

Application.DoEvents will allow the program to process all pending events, such as repaints, button clicks, etc. It comes with a cost, though, as it is not a cheap method to call (cost in time, of course). Multithreading isn't necessarily free, either, but it CAN take advantage of multiple cores, which will be far more efficient than a solution using DoEvents, as long as multiple cores are available.