[2008] Using .NET async methods

[obi1kenobi]

I have noticed that the .NET framework contains extensive amounts of asynchronous methods (identified by the Begin and End keywords). They usually take an AsyncCallback (or other callbacks) and an Object as parameters. Unfortunately, given my limited experience with multithreading, I was unable to understand the MSDN documentation on the proper use of callbacks and am now left dead in the water with my current project. How do I use the callback objects?

Can anyone please show me a proper way to fill an array of bytes by using multiple threads (or the ThreadPool) for a sub which takes two Integers as parameters? When I try to add arguments to a method after the AddressOf keyword, it tries to treat the method as an array, resulting in an error.

The scenario is as follows:
A byte array needs to be filled with data. The data is received from a server via the WebResponse stream. Since the server must be contacted several times, possibly with different URIs every time, the interaction must be concurrent. Then each response must be written at the appropriate position in the byte array. In the end, the byte array must be returned to the calling thread.

I know that it is unusual to write to an array concurrently, but it's an application requirement. Unless someone proposes a better method to get the data concurrently and then write it in an array with the same effect, of course.

[MaximilianMayrhofer]

You're basically talking about something like a torrent download? Where different parts of the file are downloaded from different users? You wouldn't write to the byte array at the same time from different threads. You would create as many arrays as you have URI's, and fill each array at appropriate positions with the appropriate info. Then when all are finished you add them together.

[obi1kenobi]

Well the file itself doesn't exist "as is" anywhere, instead it is created from multiple parts from multiple users, so I guess you could say it's similar to a torrent.

I considered the option you are proposing, however I abandoned it because I could not track the progress of each array. How would I know when a certain array is filled and ready to be transferred to the master array? AFAIK it should be related to my callback question in my previous post...

[MaximilianMayrhofer]

Could you post some of the code you are using in the WebResponse? And how is this data retrieved? Is each part a consecutive group of bytes? With a bit more data, I think I can help you out.

[obi1kenobi]

ATM I do not have access to my code since it is on a different pc from the one I'm using. However, I will try to provide as much info as possible and if you still need the code then I will post it as soon as I can.

The WebResponse contains a stream object in one of its properties, which in turn is populated by plain binary data. I use a BinaryReader to retrieve the bytes of data from the stream and place them in an array. It would be nice if the order of the bytes could be kept between WebResponses (e.g. the first response would return the first series of bytes etc.), however it is not necessary and may be avoided if it complicates matters.

[jmcilhinney]

Have you read this?

[jmcilhinney]

Having read your first post more carefully it seems to me that you should build this solution up in two parts. First, use the ThreadPool to write to your array, e.g.

Module Module1
 
    Sub Main()
        Dim stream As IO.Stream
        Dim array As Byte()
 
        Threading.ThreadPool.QueueUserWorkItem(AddressOf WriteToArray, _
                                               New ArrayWriteData With {.Array = array, _
                                                                        .Offset = 0, _
                                                                        .Count = 0, _
                                                                        .Stream = stream})
        Threading.ThreadPool.QueueUserWorkItem(AddressOf WriteToArray, _
                                               New ArrayWriteData With {.Array = array, _
                                                                        .Offset = 0, _
                                                                        .Count = 0, _
                                                                        .Stream = stream})
        Threading.ThreadPool.QueueUserWorkItem(AddressOf WriteToArray, _
                                               New ArrayWriteData With {.Array = array, _
                                                                        .Offset = 0, _
                                                                        .Count = 0, _
                                                                        .Stream = stream})
    End Sub
 
    Private Sub WriteToArray(ByVal state As Object)
        Dim awd As ArrayWriteData = DirectCast(state, ArrayWriteData)
 
        awd.Stream.Read(awd.Array, awd.Offset, awd.Count)
    End Sub
 
End Module
 
 
Friend Structure ArrayWriteData
    Public Array As Byte()
    Public Offset As Integer
    Public Count As Integer
    Public Stream As IO.Stream
End Structure

Obviously that code won't work as is. You'd have to actually create an array for a start, plus use proper values for the offset and count each time. Also, your streams will come from your WebResponses.

Once you've got that part working properly, then you can look at using BeginGetResponse to get your responses asynchronously instead of using GetResponse to get them synchronously.

[MaximilianMayrhofer]

Your webresponses contain streams of bytes.

Lets say that you have a empty 200 byte array. And you are retrieving streams from 4 locations. One stream has bytes 0-49, one has 50-99, one has 100-149 and one has 150-199. How does the program know which stream contains the bytes for which location?

[obi1kenobi]

@jmcilhinney:
Thanks for the great link, I'll try to make the most of it.

The code you provided enabled me to understand the use of the ThreadPool much better. I'll try it in my project as soon as possible. Just one thing - how about using GetResponse in the sub executed by the ThreadPool? Wouldn't it be a cleaner solution than using BeginGetResponse before starting the ThreadPool WorkItems? I am unexperienced, so I'm asking for your opinion before I do anything I might regret later

@MaximilianMayrhofer:
Well when a WorkItem is enqueued in the ThreadPool, the appropriate arguments will be passed, ensuring that the bytes read from its Stream will be placed at their appropriate positions. The point is that the WorkItem will query the server, get the info and write it to the predetermined location.

[obi1kenobi]

Just one thing left to solve now I think. How do I detect when all the tasks in the ThreadPool have finished executing? Since I need to pass the finished byte array along...

Is WaitHandle the way to go or do you have another suggestion?

EDIT:
Ok I made some changes to the code, so now it uses WaitHandles and AutoResetEvents. The problem is that I'm getting a NotSupportedException on the WaitHandle.WaitAll(waitHandles) line, saying the following: "WaitAll for multiple handles on a STA thread is not supported.". I've added the <MTAThreadAttribute()> attribute before the method using WaitHandles, however this has no effect. Any ideas?

[obi1kenobi]

Bump. This issue is a serious one and I cannot solve it by myself. I need your help!