Results 1 to 13 of 13

Thread: [RESOLVED] This is probably a Math question

  1. #1

    Thread Starter
    Hyperactive Member babyekc's Avatar
    Join Date
    Jul 2004
    Location
    planet earth
    Posts
    270

    Resolved [RESOLVED] This is probably a Math question

    Hi All,

    Say I got records:
    Id: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

    Now I got multiple apps running, retrieving the same record set as above.
    How do I split the load based on Id? Example:

    If 2 apps running:
    App 1: process record - Id 1, 3, 5, 7, 9, 11, 13, 15, 17, 19
    App 2: process record - Id 2, 4, 6, 8, 10, 12, 14, 16, 18, 20

    If 3 apps running:
    App 1: process record - Id 1, 4, 7, 10, 13, 16, 19
    App 2: process record - Id 2, 5, 8, 11, 14, 17, 20
    App 3: process record - Id 3, 6, 9, 12, 15, 18

    If 4 apps running:
    App 1: process record - Id 1, 5, 9, 13, 17
    App 2: process record - Id 2, 6, 10, 14, 18
    App 3: process record - Id 3, 7, 11, 15, 19
    App 4: process record - Id 4, 8, 12, 16, 20

    Appreciate if anyone can give any idea. Thanks.
    *wink wink*

    _babyekc

  2. #2
    eXtreme Programmer .paul.'s Avatar
    Join Date
    May 2007
    Location
    Chelmsford UK
    Posts
    25,464

    Re: This is probably a Math question

    It'd make more sense to just split the load based on remaining records. That is instead of assigning anything, just take the top record and process it, then mark it as processed.

  3. #3

    Thread Starter
    Hyperactive Member babyekc's Avatar
    Join Date
    Jul 2004
    Location
    planet earth
    Posts
    270

    Re: This is probably a Math question

    It's basically a load balancer, the multiple apps are exactly the same. I just want to eliminate the possibility of multiple apps processing the same record.
    *wink wink*

    _babyekc

  4. #4
    Super Moderator jmcilhinney's Avatar
    Join Date
    May 2005
    Location
    Sydney, Australia
    Posts
    110,297

    Re: This is probably a Math question

    I agree with .paul. that it makes most sense to simply treat the list as a queue and have each processor take the next item in the queue. One advantage of that is that, if some records take longer to process than others, you won't end up with idle processors waiting for others to finish their assigned load. If you're talking about different processes, rather than different threads in the same process, then you'd need to use a mutex to ensure that you didn't get two processes trying to dequeue the same item.

  5. #5
    Super Moderator jmcilhinney's Avatar
    Join Date
    May 2005
    Location
    Sydney, Australia
    Posts
    110,297

    Re: This is probably a Math question

    If you're determined to go with your original idea, you could use something like this to create the sublists based on the number of sublists you want:
    vb.net Code:
    1. Private Function Partition(Of T)(sourceList As IEnumerable(Of T), count As Integer) As List(Of T)()
    2.     Dim destinationLists = Enumerable.Range(1, count).
    3.                                       Select(Function(n) New List(Of T)).
    4.                                       ToArray()
    5.     Dim listIndex = 0
    6.  
    7.     For Each item As T In sourceList
    8.         destinationLists(listIndex).Add(item)
    9.  
    10.         'Move to the next list and wrap from the last to the first.
    11.         listIndex = (listIndex + 1) Mod count
    12.     Next
    13.  
    14.     Return destinationLists
    15. End Function

  6. #6
    Angel of Code Niya's Avatar
    Join Date
    Nov 2011
    Posts
    8,598

    Re: This is probably a Math question

    Quote Originally Posted by jmcilhinney View Post
    If you're talking about different processes, rather than different threads in the same process, then you'd need to use a mutex to ensure that you didn't get two processes trying to dequeue the same item.
    Somehow, I'm smelling some kind of relational database involved in this. Database engines like SQL Server have their own mechanisms for managing concurrency which he can "piggyback" on top of for synchronization of his queue. All he has to do is check for concurrency related exceptions propagated from the database. The DB engine will take care of keeping things in sync.

    Of course if no database backend is involved then, yea, a mutex would be the way to go for synchronization.
    Treeview with NodeAdded/NodesRemoved events | BlinkLabel control | Calculate Permutations | Object Enums | ComboBox with centered items | .Net Internals article(not mine) | Wizard Control | Understanding Multi-Threading | Simple file compression | Demon Arena

    Copy/move files using Windows Shell | I'm not wanted

    C++ programmers will dismiss you as a cretinous simpleton for your inability to keep track of pointers chained 6 levels deep and Java programmers will pillory you for buying into the evils of Microsoft. Meanwhile C# programmers will get paid just a little bit more than you for writing exactly the same code and VB6 programmers will continue to whitter on about "footprints". - FunkyDexter

    There's just no reason to use garbage like InputBox. - jmcilhinney

    The threads I start are Niya and Olaf free zones. No arguing about the benefits of VB6 over .NET here please. Happiness must reign. - yereverluvinuncleber

  7. #7

    Thread Starter
    Hyperactive Member babyekc's Avatar
    Join Date
    Jul 2004
    Location
    planet earth
    Posts
    270

    Re: This is probably a Math question

    OK it's like this.

    1) I got 2 AWS APP servers running (and possible more, because of 'Auto-Scaling feature' from AWS) to connect to a AWS DB server
    2) This 2 APP servers are installed with same Windows Service and Web App/API (same settings and same configurations)
    3) I used SQLDependency to check if there's any changes in a particular table
    * Select all pending records > lets say 100 records
    4) If there's changes, I used Signal R to tell the 2 AWS APP Servers
    * New records inserted
    5) 2 AWS APP Servers will then reload the list records
    * Select all pending records (now 101 records) > this is to refresh the list and check which records is DUE (based on time)
    6) Now both APP Servers has the same list of records (101)
    7) There's a timer on the APP Server Windows Service, to check if any record is DUE (this is on the Windows Service local, not querying the DB)
    8) Assuming 11 records is DUE on 13:00:00, Id: 1000, 1001, 1002, 1003, 1004, 1005, 1006, 1007, 1008, 1009 and 1010
    9) When the clock hit 13:00:00, there will be 11 records need to process for both APP Servers Windows Service


    Now in here, I want to eliminate APP Server 1 and APP Server 2 processing the same record.
    There is this POSSIBILITY that both APP Server Windows Service timer hit on the same time, and retrieve record Id: 1000 - to process


    If there's a math formula (which I'm asking in here on the 1st post), the above possibility can be eliminated and also both APP Servers are SHARING the loads.
    The Signal R will know there are 2 connected clients, and will then tell:

    APP Server 1 - process records:
    1000, 1002, 1004, 1006, 1008, 1010

    APP Server 2 - process records:
    1001, 1003, 1005, 1007, 1009
    *wink wink*

    _babyekc

  8. #8
    PowerPoster PlausiblyDamp's Avatar
    Join Date
    Dec 2016
    Location
    Pontypool, Wales
    Posts
    2,458

    Re: This is probably a Math question

    If you are already using AWS could you not use SQS and add a message for each record that needs to be processed? The two app servers would just grab the next message in the queue, process it and then get the next message. It might not be exactly the order you specified but that should mean each message is only processed once.

  9. #9

    Thread Starter
    Hyperactive Member babyekc's Avatar
    Join Date
    Jul 2004
    Location
    planet earth
    Posts
    270

    Re: This is probably a Math question

    OK basically I can do it with:

    Code:
    For lLoop = (Id) To 1 Step (Total APP Server) * -1
            If lLoop <= (Total APP Server) Then
              TextBox3.Text = lLoop
            End If
          Next
    But if the Id is large number, say - 273982374, it will take some some times.

    Is there any math formula to replicate the above code?
    *wink wink*

    _babyekc

  10. #10
    Super Moderator jmcilhinney's Avatar
    Join Date
    May 2005
    Location
    Sydney, Australia
    Posts
    110,297

    Re: This is probably a Math question

    You really should get over the idea that there is some magic maths that is going to do this for you.

  11. #11
    Fanatic Member
    Join Date
    Feb 2013
    Posts
    985

    Re: This is probably a Math question

    im a little out of my league here but if i was to tackle this problem id do it like this.....

    have a process monitoring everything, lets call it the controller.

    this process is responsible for delegating workloads, if only a small workload is present it opens a single worker process to handle it, if a large workload is present it opens several process and delegates a small amount of work to each one.....

    forgive my ignorance im just thinking about the logic here not about the classes or objects your currently using
    Yes!!!
    Working from home is so much better than working in an office...
    Nothing can beat the combined stress of getting your work done on time whilst
    1. one toddler keeps pressing your AVR's power button
    2. one baby keeps crying for milk
    3. one child keeps running in and out of the house screaming and shouting
    4. one wife keeps nagging you to stop playing on the pc and do some real work.. house chores
    5. working at 1 O'clock in the morning because nobody is awake at that time
    6. being grossly underpaid for all your hard work


  12. #12

    Thread Starter
    Hyperactive Member babyekc's Avatar
    Join Date
    Jul 2004
    Location
    planet earth
    Posts
    270

    Re: This is probably a Math question

    OK, i'm basically solve the problem, it might not be the beautiful solution:

    Code:
    Private Function CalculateId(ByVal pId As Int64, ByVal pStep As Integer) As Integer
    
        Dim lId As Int64 = pId
        Dim lLength As Integer = Len(lId.ToString)
    
        Dim lStepToMultiply As String = "1"
        If lLength > 3 Then
    
          If Val(lId.ToString.First) > pStep Then
            For lLoop = 1 To lLength - 1
              lStepToMultiply = lStepToMultiply & "0"
            Next
          Else
            For lLoop = 1 To lLength - 2
              lStepToMultiply = lStepToMultiply & "0"
            Next
          End If
    
          Dim lStep As Int64 = Convert.ToInt64(lStepToMultiply)
          For lLoop = lId To lStep Step lStep * pStep * -1
            lId = lLoop
          Next
    
          If lLength > 3 Then
            lId = CalculateId(lId, pStep)
          End If
    
        End If
    
        For lLoop = lId To 1 Step pStep * -1
          If lLoop <= pStep Then
            lId = lLoop
          End If
        Next
    
        Return lId
    
      End Function
    *wink wink*

    _babyekc

  13. #13
    PowerPoster PlausiblyDamp's Avatar
    Join Date
    Dec 2016
    Location
    Pontypool, Wales
    Posts
    2,458

    Re: This is probably a Math question

    Quote Originally Posted by babyekc View Post
    OK, i'm basically solve the problem, it might not be the beautiful solution:

    Code:
    Private Function CalculateId(ByVal pId As Int64, ByVal pStep As Integer) As Integer
    
        Dim lId As Int64 = pId
        Dim lLength As Integer = Len(lId.ToString)
    
        Dim lStepToMultiply As String = "1"
        If lLength > 3 Then
    
          If Val(lId.ToString.First) > pStep Then
            For lLoop = 1 To lLength - 1
              lStepToMultiply = lStepToMultiply & "0"
            Next
          Else
            For lLoop = 1 To lLength - 2
              lStepToMultiply = lStepToMultiply & "0"
            Next
          End If
    
          Dim lStep As Int64 = Convert.ToInt64(lStepToMultiply)
          For lLoop = lId To lStep Step lStep * pStep * -1
            lId = lLoop
          Next
    
          If lLength > 3 Then
            lId = CalculateId(lId, pStep)
          End If
    
        End If
    
        For lLoop = lId To 1 Step pStep * -1
          If lLoop <= pStep Then
            lId = lLoop
          End If
        Next
    
        Return lId
    
      End Function
    This seems an awful lot of work and effort to produce a very basic load balancer, if the work is taking varying amounts of time you could find one app has exhausted all it's items while the other one has plenty left. It will prevent any scalability without rewriting your logic every time an extra app is added and won't take into account instance failures or other problems.

    Why don't you just use a queue? jmcilhinney mentioned this earlier and AWS has this kind of functionality built in anyway.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width