[RESOLVED] Task(Of TResult).Factory.ContinueWith with chain of functions, methods and results

[JayJayson]

Hi everyone and thanks for reading..

I am getting myself mixed up while thinking about chaining multiple methods together using a single task.

Okay, so, I am passing multiple methods and their respective argument (always a single object) into a collection. When all the methods I want to run have been added to the collection, I run the methods in turn using a single task for all of the methods in the collection.

So for example, I am able to run the methods in turn by using the following:

'Me.Token is a CancellationToken used to exit the task if a cancellation was requested.
'Item is a Class with two properties [.Method is an Action(Of Object) and .Argument is an Object]
Return Task.Factory.StartNew(
    Sub()
        Me.Token.ThrowIfCancellationRequested()
 
        For Each Item As TaskItem In Me.List
            Task.Factory.StartNew(Item.Method, Item.Argument, Me.Token, TaskCreationOptions.AttachedToParent, TaskScheduler.Current)
        Next Item
    End Sub, Me.Token)

I would now like to extend on this and to run a series of Functions in a chained task and to collect the results of each function within that chain. My understanding is that you can pass the result of one function to the parameter of the next by using the ContinueWith method however to also collect the results I think I would need to do this with the use of a Task(Of TResult).

So what I would like to know is - if I start with the following (which will run the first function from within my collection and collect the result):

Dim T As Task(Of Object) = Task(Of Object).Factory.StartNew(Me.List.Item(0).Method, Me.List.Item(0).Argument, Me.Token, TaskCreationOptions.AttachedToParent, TaskScheduler.Current)

What do I need to do to allow ContinueWith to run each of my remaining functions while passing the TResult of the previous function to the next.

For iChild As Integer = 1 To Me.Count - 1
    ' What should go here.
    T = T.Factory.ContinueWith...
Next iChild

I am really struggling to understand the MSDN documentation on this (it wants a func(of object, object) for the first parameter of the ContinueWith method and I am not sure how to go about that). I cannot find an example close enough to what I am doing in order to follow it.


I hope I have made sense, but let me know if I need to be any clearer.

Cheers,
Jay

[Sitten Spynne]

My first thought is this is overcomplicating things. All programs are essentially chains of function calls. This is what a "chain" of functions collecting results might look like:

Function DoChain() As Something
    Dim result = GetFinalResult(...)
    Return result
End Function

Function GetFinalResult(...) As Something
    Dim accumulatedResults = Step1(...)
    accumulatedResults = Step2(...)
    ...
    Return accumulatedResults
End Function

Tasks weren't specifically made for this. They were made for running functions asynchronously. Continuations weren't created for building chains of functions that feed each other. They were built for specifying what to do when a task completes. Both of these together implement the Async/Await keywords that make BackgroundWorker look obsolete.

That doesn't mean you can't use them to build chains of functions, but it might mean it will be harder than it looks.

Before I get too much deeper, I want to make sure I understand what you're trying to do. I wrote some code above that looks like how you'd write it without tasks. Can you write a little bit of code without Tasks that shows what you're trying to do? Then we could come up with a better idea of if and how Tasks might help.

By the way, which VS are you using? Task.Factory is VS2010 syntax, I think? It's a little more streamlined in newer versions.

[JayJayson]

Thanks Sitten Spynne. I have read a lot of your helpful posts over the years and appreciate you taking the time to contribute.

The first thing I will add, that will be relevant here, is that my overall aim is to create a DLL that can chain any combination of methods/functions together and pass parameters between them (if necessary) and then raise an event back to the control that called the DLL once the tasks have all completed.

Also I am using VS2015.

So as an example this is a scenario I am working on at the mo (without tasks):

Public Class Parameters
  Public Property MySQLConnectionString As String
  Public Property ID As Integer
  Public Property Query As String
  Public Property Result As String
  Public Property Success As Boolean
End Class
 
Public Sub Method(Param As Object)
  'Connect to a MySQLDatabase and SELECT multiple columns into a datatable.
  Dim MethodParams As Parameters = CType(Param, Parameters)
  Dim DT As DataTable = Nothing
  '...
  'E.g. DT could end up with two columns, 'ID' and 'Value'.
  '...
  For Each DRow As DataRow In DT.Rows
    '...
    'Loop through the datatable and perform a lengthy function for each row.
    '...
    Dim ChainParams As New Parameters
    ChainParams.MySQLConnectionString = MethodParams.MySQLConnectionString
    ChainParams.ID = CInt(DRow.Item("ID"))
    ChainParams.Query = "QueryBasedOnDRow.Item('Value')"
    ChainOne(ChainParams)
  Next DRow
  '...
  'Wait on the completion of all the Chained Methods
  '...
   'Raise a Completion Event and attach a class with some stats as a parameter based on the results collected from the Chained functions.
End Sub
 
Public Function ChainOne(Param As Object) As Parameters
  'Connect to a MySQLDatabase and run a query from Method().
  Dim Params As Parameters = CType(Param, Parameters)
  Params.Result = "ResultFromLengthyMySQLDatabaseQuery"
 
  Return Params
End Function
 
Public Function ChainTwo(Param As Object) As Parameters
  'Perform final part of the chain and record some results.
  Dim Params As Parameters = CType(Param, Parameters)
 
  If CBool("SomeLogic") = True Then
    '...
    'Update the MYSQL Table with the result of ChainTwo into the row with the ID from Method().
    '...
    Params.Success = True
  Else
    Params.Success = False
  End If
 
  'Collect some stats for a completion event.
  Return Params
End Function

In this scenario the Method retrieves over 10,000 rows of data from a MySQL table. Then it needs to run a query for each of those rows and update the results back to the table in the correct row. What got me started on this journey into tasks, is that I want to be able to run several of the 'Chained' functions on simultaneous threads because each can take several seconds to run. However at the same time I want to limit how many threads will run at once so I can control how many database connections are being used. I started to use collections to control this after I ran into difficulty limiting the number of tasks used AND allowing them to take a parameter and produce a result.

So my aim (in this particular scenario) is to start Method() on a task of it's own to keep the UI thread from becoming unresponsive. When I loop through the DataTable in MethodOne I want to add each reference of the ChainOne() function and it's associated parameter to a collection. For example there may be 8 collections, representing 8 threads.

Each collection has a method to loop through it's items and run the ChainOne function's in turn and thus keeps them all running on a single task per collection. But - I have this working fine where I do not need to pass a parameter from a ChainOne function to a ChainTwo function and then collect a result from the ChainTwo function to pass back to an event raised by the collection when it finishes all of the functions.

I hope this makes sense, as my head is still spynning (sorry, pun intended) from thinking this over.

Finally, and as I said at the beginning, because I want this to be part of a DLL I want to be able to make sure I can simply call upon it with any combination of Methods and Functions and therefore finding a specific approach for the specific scenario I mentioned is not my only goal here.

Thanks,
Jay

[Sitten Spynne]

I have some ideas. I'm sorry it's taking me so long, I'm having a hard time shaking an old project where I did this out of my head. In that project, customers could add new functions, so our code had to handle any combination of types, including some we'd obviously never seen. That was really, really hard, and it's making me view this problem as really, really hard. I still need some time to think about it, but here's some ideas.

The simplest implementation of a chain of functions is an array of some delegate.

Dim operations As Func(Of Double, Double)() = { Add, Subtract }

Dim result As Double = 0
For Each operation In Operations
    result = operation(result)
Next

I thought about a lot of things at this point. One of them's important, but let's ignore it because it's a problem with every implementation. The next thing I thought about was, "How would this be represented via tasks and continuations?" Using the imaginary Add() and Subtract() above, and pretending VB syntax isn't stupid about line endings:

Dim chainTask = Task(Of Double).Run(Add)
    .ContinueWith(Sub(parent) Subtract(parent.Result))

Dim result = chainTask.Result

That looks inferior to an array, to me, because by the time we've got a fifth "chain" things look ridiculous. It does handle exceptions for us, but when I think about "Is this 20-element array in the right order?" at debug time vs. "Are the continuations in the right order?" I barf thinking about Tasks.

This is where things can get REALLY complicated, because it's very hard to put functions with different signatures into a data structure AND maintain integrity.

What I mean is it's hard to represent this chain:

Separate an integer into its digits. Sum the digits, then multiply the digits by three.

(Integer -> Integer()) -> (Integer(), Integer) -> (Integer, Integer)

It's not impossible. But it's also a lot easier if you can narrow it to things like, "I'll only ever use Integer", or "If Double is at the front, Double will be at the end". I keep trying to come up with examples, and they always end up with that kind of complexity.

Tell me, does this sound more like your use case? (I know it says 'a customer'. Assume this is happening in parallel with many customers.)

I retrieve a customer. I gather some sales information and modify the customer's status in some way. I gather some other information and update the customer again. Finally, I save the updated customer to a database.

(Id, Customer) -> (Customer, Customer) -> (Customer, Customer) -> Action(Customer)

That sort of workflow is much more reasonable to implement. If you agree that all the steps in the middle should have the same input/output types, I think we're in business with a type that looks something like this:

Public Class CustomerWorkflow

    Public Sub Execute(ByVal id As Integer)
        Dim customer = FetchCustomer(id)

        customer = ExecuteSteps(customer)

        SaveCustomer(customer)
    End Sub

    ...
End Class

I think that setup can still benefit from task parallelization, but I want to make sure this covers it before I talk about it. I've got a lot of other stuff to do today so it might be tomorrow morning!

[JayJayson]

Hi, the customer workflow suggestion does look more like the kind of implementation I am trying to achieve. My plan is to always use the same Class as input and output from one function to the next throughout the chain of tasks.

Also, please take your time as this is purely for personal projects so I am not in any rush.

Thanks,
Jay

[FunkyDexter]

I haven't had time to take in the whole thread but I think you're looking for the command pattern

edit>...and here's a handy link on passing parameters between commands

[Sitten Spynne]

That's more or less going to be my suggestion, especially if you don't have a lot of variety in parameters.

Here's a watered-down version of a really common class in modern development: RelayCommand:

Public Class RelayCommand
    
    Private Func(Of Customer, Customer) _execute

    Public Sub New(ByVal Func(Of Customer, Customer) execute)
        _execute = execute
    End Sub

    Public Function Execute(ByVal customer As Customer) As Customer
        Return _execute(customer)
    End Function
End Class

It is, essentially, a Func(Of Customer, Customer). The idea is you can derive more specialized classes that do a little more work. Anyway, your "pipeline" here will consist of several of those commands chained together, with a start/end step:

Public Class Pipeline

    Private _commands As List(Of RelayCommand)

    Public Sub AddCommand(...)

    Public Sub Execute(...)
        Dim customer = GetCustomer(...)

        For Each command In _commands
            customer = command.Execute(customer)
        Next

        SaveCustomer(customer)
    End Sub

End Class

You can add a lot of things here. Sophisticated error tracking, forking chains of commands, etc. But this is the basic flow. You were trying to parallelize it a bit, earlier. There's also a lot of different ways you could do it. But the dang easiest is something like this, that'd execute every step as a separate task:

Public Async Sub Execute(...)
    Dim customer = GetCustomer(...)

    For Each command In _commands
        customer = Await command.Execute(customer)
    Next

    SaveCustomer(customer)
End Sub

I'm still a little shaky on the right syntax. I think to make that work, RelayCommand needs this change:

Public Class AsyncRelayCommand
    
    Private Func(Of Customer, Customer) _execute

    Public Sub New(ByVal Func(Of Customer, Customer) execute)
        _execute = execute
    End Sub

    Public Function Execute(ByVal customer As Customer) As Task(Of Customer)
        Return Task.Run(Sub() _execute(customer))
    End Function
End Class

There's a lot of ways to go from here. The key concept is building classes that represent a process, and a class that executes a chain of them.

[JayJayson]

Thank you Sitten Spynne, that explanation and example code was amazing. Don't get me wrong, I had to think it over a bit until something clicked and I was able to comprehend from it what I need to do - but I managed to piece the following test together with the help of your post, Option Strict and a little bit of further googling:

Private Async Sub TestButton_Click(sender As Object, e As EventArgs) Handles TestButton.Click
  Dim Pipe As New PipeLine
  Pipe.Add(New RelayCommand(AddressOf Test1))
  Pipe.Add(New RelayCommand(AddressOf Test2))
  Pipe.Add(New RelayCommand(AddressOf Test3))
 
  Dim Result As String = Await Pipe.Execute("Testing")
  MessageBox.Show(Result)
End Sub
 
Public Function Test1(Input As String) As String
  Return String.Concat(Input, "-1")
End Function
 
Public Function Test2(Input As String) As String
  Return String.Concat(Input, "-2")
End Function
 
Public Function Test3(Input As String) As String
  Return String.Concat(Input, "-3")
End Function
 
Public Class PipeLine : Inherits CollectionBase
  Public Sub Add(Command As RelayCommand)
    Me.List.Add(Command)
  End Sub
 
  Public Async Function Execute(Param As String) As Task(Of String)
    For Each CMD As RelayCommand In Me.List
      Param = Await CMD.Execute(Param)
    Next CMD
 
    Return Param
  End Function
End Class
 
Public Class RelayCommand
  Private Command As Func(Of String, String)
 
  Public Sub New(CMD As Func(Of String, String))
    Me.Command = CMD
  End Sub
 
  Public Function Execute(Param As String) As Task(Of String)
    Return Task.Run(Function() Me.Command(Param))
  End Function
End Class

It simply forms the string "Testing-1-2-3".

I would appreciate you looking it over to verify I have got this right, or if indeed there is perhaps another way that may be better for one reason or another. Although the fact that I get the result I expect from this code tells me it should be fine in practice. Of course, it will need to be adapted for my DLL so that it can raise events back to the control where the functions will reside but I think I have that covered.

Massive thanks for your help