-
Nov 21st, 2017, 11:10 PM
#1
Thread Starter
Hyperactive Member
[RESOLVED] This is probably a Math question
Hi All,
Say I got records:
Id: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Now I got multiple apps running, retrieving the same record set as above.
How do I split the load based on Id? Example:
If 2 apps running:
App 1: process record - Id 1, 3, 5, 7, 9, 11, 13, 15, 17, 19
App 2: process record - Id 2, 4, 6, 8, 10, 12, 14, 16, 18, 20
If 3 apps running:
App 1: process record - Id 1, 4, 7, 10, 13, 16, 19
App 2: process record - Id 2, 5, 8, 11, 14, 17, 20
App 3: process record - Id 3, 6, 9, 12, 15, 18
If 4 apps running:
App 1: process record - Id 1, 5, 9, 13, 17
App 2: process record - Id 2, 6, 10, 14, 18
App 3: process record - Id 3, 7, 11, 15, 19
App 4: process record - Id 4, 8, 12, 16, 20
Appreciate if anyone can give any idea. Thanks.
-
Nov 21st, 2017, 11:36 PM
#2
Re: This is probably a Math question
It'd make more sense to just split the load based on remaining records. That is instead of assigning anything, just take the top record and process it, then mark it as processed.
- Coding Examples:
- Features:
- Online Games:
- Compiled Games:
-
Nov 21st, 2017, 11:46 PM
#3
Thread Starter
Hyperactive Member
Re: This is probably a Math question
It's basically a load balancer, the multiple apps are exactly the same. I just want to eliminate the possibility of multiple apps processing the same record.
-
Nov 22nd, 2017, 12:37 AM
#4
Re: This is probably a Math question
I agree with .paul. that it makes most sense to simply treat the list as a queue and have each processor take the next item in the queue. One advantage of that is that, if some records take longer to process than others, you won't end up with idle processors waiting for others to finish their assigned load. If you're talking about different processes, rather than different threads in the same process, then you'd need to use a mutex to ensure that you didn't get two processes trying to dequeue the same item.
-
Nov 22nd, 2017, 12:49 AM
#5
Re: This is probably a Math question
If you're determined to go with your original idea, you could use something like this to create the sublists based on the number of sublists you want:
vb.net Code:
Private Function Partition(Of T)(sourceList As IEnumerable(Of T), count As Integer) As List(Of T)() Dim destinationLists = Enumerable.Range(1, count). Select(Function(n) New List(Of T)). ToArray() Dim listIndex = 0 For Each item As T In sourceList destinationLists(listIndex).Add(item) 'Move to the next list and wrap from the last to the first. listIndex = (listIndex + 1) Mod count Next Return destinationLists End Function
-
Nov 22nd, 2017, 01:53 AM
#6
Re: This is probably a Math question
Originally Posted by jmcilhinney
If you're talking about different processes, rather than different threads in the same process, then you'd need to use a mutex to ensure that you didn't get two processes trying to dequeue the same item.
Somehow, I'm smelling some kind of relational database involved in this. Database engines like SQL Server have their own mechanisms for managing concurrency which he can "piggyback" on top of for synchronization of his queue. All he has to do is check for concurrency related exceptions propagated from the database. The DB engine will take care of keeping things in sync.
Of course if no database backend is involved then, yea, a mutex would be the way to go for synchronization.
-
Nov 22nd, 2017, 02:38 AM
#7
Thread Starter
Hyperactive Member
Re: This is probably a Math question
OK it's like this.
1) I got 2 AWS APP servers running (and possible more, because of 'Auto-Scaling feature' from AWS) to connect to a AWS DB server
2) This 2 APP servers are installed with same Windows Service and Web App/API (same settings and same configurations)
3) I used SQLDependency to check if there's any changes in a particular table
* Select all pending records > lets say 100 records
4) If there's changes, I used Signal R to tell the 2 AWS APP Servers
* New records inserted
5) 2 AWS APP Servers will then reload the list records
* Select all pending records (now 101 records) > this is to refresh the list and check which records is DUE (based on time)
6) Now both APP Servers has the same list of records (101)
7) There's a timer on the APP Server Windows Service, to check if any record is DUE (this is on the Windows Service local, not querying the DB)
8) Assuming 11 records is DUE on 13:00:00, Id: 1000, 1001, 1002, 1003, 1004, 1005, 1006, 1007, 1008, 1009 and 1010
9) When the clock hit 13:00:00, there will be 11 records need to process for both APP Servers Windows Service
Now in here, I want to eliminate APP Server 1 and APP Server 2 processing the same record.
There is this POSSIBILITY that both APP Server Windows Service timer hit on the same time, and retrieve record Id: 1000 - to process
If there's a math formula (which I'm asking in here on the 1st post), the above possibility can be eliminated and also both APP Servers are SHARING the loads.
The Signal R will know there are 2 connected clients, and will then tell:
APP Server 1 - process records:
1000, 1002, 1004, 1006, 1008, 1010
APP Server 2 - process records:
1001, 1003, 1005, 1007, 1009
-
Nov 22nd, 2017, 04:37 AM
#8
Re: This is probably a Math question
If you are already using AWS could you not use SQS and add a message for each record that needs to be processed? The two app servers would just grab the next message in the queue, process it and then get the next message. It might not be exactly the order you specified but that should mean each message is only processed once.
-
Nov 22nd, 2017, 04:38 AM
#9
Thread Starter
Hyperactive Member
Re: This is probably a Math question
OK basically I can do it with:
Code:
For lLoop = (Id) To 1 Step (Total APP Server) * -1
If lLoop <= (Total APP Server) Then
TextBox3.Text = lLoop
End If
Next
But if the Id is large number, say - 273982374, it will take some some times.
Is there any math formula to replicate the above code?
-
Nov 22nd, 2017, 04:51 AM
#10
Re: This is probably a Math question
You really should get over the idea that there is some magic maths that is going to do this for you.
-
Nov 22nd, 2017, 06:30 AM
#11
Fanatic Member
Re: This is probably a Math question
im a little out of my league here but if i was to tackle this problem id do it like this.....
have a process monitoring everything, lets call it the controller.
this process is responsible for delegating workloads, if only a small workload is present it opens a single worker process to handle it, if a large workload is present it opens several process and delegates a small amount of work to each one.....
forgive my ignorance im just thinking about the logic here not about the classes or objects your currently using
Yes!!!
Working from home is so much better than working in an office...
Nothing can beat the combined stress of getting your work done on time whilst
1. one toddler keeps pressing your AVR's power button
2. one baby keeps crying for milk
3. one child keeps running in and out of the house screaming and shouting
4. one wife keeps nagging you to stop playing on the pc and do some real work.. house chores
5. working at 1 O'clock in the morning because nobody is awake at that time
6. being grossly underpaid for all your hard work
-
Nov 22nd, 2017, 06:41 AM
#12
Thread Starter
Hyperactive Member
Re: This is probably a Math question
OK, i'm basically solve the problem, it might not be the beautiful solution:
Code:
Private Function CalculateId(ByVal pId As Int64, ByVal pStep As Integer) As Integer
Dim lId As Int64 = pId
Dim lLength As Integer = Len(lId.ToString)
Dim lStepToMultiply As String = "1"
If lLength > 3 Then
If Val(lId.ToString.First) > pStep Then
For lLoop = 1 To lLength - 1
lStepToMultiply = lStepToMultiply & "0"
Next
Else
For lLoop = 1 To lLength - 2
lStepToMultiply = lStepToMultiply & "0"
Next
End If
Dim lStep As Int64 = Convert.ToInt64(lStepToMultiply)
For lLoop = lId To lStep Step lStep * pStep * -1
lId = lLoop
Next
If lLength > 3 Then
lId = CalculateId(lId, pStep)
End If
End If
For lLoop = lId To 1 Step pStep * -1
If lLoop <= pStep Then
lId = lLoop
End If
Next
Return lId
End Function
-
Nov 22nd, 2017, 07:19 AM
#13
Re: This is probably a Math question
Originally Posted by babyekc
OK, i'm basically solve the problem, it might not be the beautiful solution:
Code:
Private Function CalculateId(ByVal pId As Int64, ByVal pStep As Integer) As Integer
Dim lId As Int64 = pId
Dim lLength As Integer = Len(lId.ToString)
Dim lStepToMultiply As String = "1"
If lLength > 3 Then
If Val(lId.ToString.First) > pStep Then
For lLoop = 1 To lLength - 1
lStepToMultiply = lStepToMultiply & "0"
Next
Else
For lLoop = 1 To lLength - 2
lStepToMultiply = lStepToMultiply & "0"
Next
End If
Dim lStep As Int64 = Convert.ToInt64(lStepToMultiply)
For lLoop = lId To lStep Step lStep * pStep * -1
lId = lLoop
Next
If lLength > 3 Then
lId = CalculateId(lId, pStep)
End If
End If
For lLoop = lId To 1 Step pStep * -1
If lLoop <= pStep Then
lId = lLoop
End If
Next
Return lId
End Function
This seems an awful lot of work and effort to produce a very basic load balancer, if the work is taking varying amounts of time you could find one app has exhausted all it's items while the other one has plenty left. It will prevent any scalability without rewriting your logic every time an extra app is added and won't take into account instance failures or other problems.
Why don't you just use a queue? jmcilhinney mentioned this earlier and AWS has this kind of functionality built in anyway.
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
Click Here to Expand Forum to Full Width
|