Results 1 to 17 of 17

Thread: dumb question

  1. #1

    Thread Starter
    Hyperactive Member
    Join Date
    Jan 2013
    Posts
    486

    dumb question

    In general are link statements slower than writing your own code? I will be running for millions of times.
    Thank you
    George

  2. #2
    PowerPoster PlausiblyDamp's Avatar
    Join Date
    Dec 2016
    Location
    Pontypool, Wales
    Posts
    2,458

    Re: dumb question

    Quote Originally Posted by georgesutfin View Post
    In general are link statements slower than writing your own code? I will be running for millions of times.
    Thank you
    George
    I am guessing you mean Linq rather than link - if I am wrong then the rest of what I say is going to make little sense...

    The most basic answer I can give is "it depends".

    There are certainly overheads involved in Linq compared to doing things by hand, however the quality of the code you write in either case will be a big decider on the overall performance of your code. Another influence will be the choice of technology you are using linq with (EF, Linq2Sql etc) as the performance of that layer is also going to impact things.

    I would at least have a look at the following two links to get a better feel for things before making any decisions
    https://msdn.microsoft.com/en-us/lib...v=vs.113).aspx
    https://docs.microsoft.com/en-us/dot...considerations

    However, without knowing the amount of data, the database platform, the type of data access (queries, inserts etc.) you will be performing there isn't going to be a simple answer to this one.

    Personally I tend to find EF and Linq makes for a much quicker and easier development experience and I avoid having to spend a lot of time actually wring SQL, messing with parameters and data types etc - this lets me get on with just writing code that works. If there are areas of my code that are under performing then once I have identified them I can either look at the Linq I have written more closely and look to optimising it (Lazy loading / Eager loading trade-offs, tuning the DB correctly and so on) to see if the performance can be improved. Failing that then I would generally consider rewriting these queries as SQL Stored Procs / Views but still call them via EF and Linq anyway.

  3. #3
    Smooth Moperator techgnome's Avatar
    Join Date
    May 2002
    Posts
    34,532

    Re: dumb question

    Agreed... like a lot of things "it depends" ... I worked on a system a number of years ago where we did a lot of iterations over rows in a datatable. The convention at the time was to loop through all the rows, test for a condition, if true, then process that row. The size of the datatable was on the order of tens of thousands... and it was ridiculously slow. I spent some time with it, and found that with a little bit of LINQ (to object) and some lambdas, I could test for the rows I wanted and select them out into a subset and iterate only through those - despite conventional wisdom to the contrary, the performance increase was significant, dropping from a couple minutes to a couple seconds. Since we were doing these kinds of operations on the data multiple times in a row, the savings added up fast.

    The best way to know for sure in your case is to try it out. Try both methods, set up comparable operations using both methods, and time them. Run several iterations, 100's if you can. At least a dozen. And see what comes out on top. Unfortunately this isn't one of those one size fits all situations. It has to be handled case by case.

    -tg
    * I don't respond to private (PM) requests for help. It's not conducive to the general learning of others.*
    * I also don't respond to friend requests. Save a few bits and don't bother. I'll just end up rejecting anyways.*
    * How to get EFFECTIVE help: The Hitchhiker's Guide to Getting Help at VBF - Removing eels from your hovercraft *
    * How to Use Parameters * Create Disconnected ADO Recordset Clones * Set your VB6 ActiveX Compatibility * Get rid of those pesky VB Line Numbers * I swear I saved my data, where'd it run off to??? *

  4. #4
    Super Moderator Shaggy Hiker's Avatar
    Join Date
    Aug 2002
    Location
    Idaho
    Posts
    38,989

    Re: dumb question

    You will have to test. I have yet to see a circumstance where LINQ is not slower, but TG has mentioned his example before, and I don't doubt it. Quite possibly, the size of the datatable may have much to do with it. Every test I have done has been on datatables with a few dozen rows at most. In those tests LINQ was always significantly slower, but iterating through such a collection is going to be fast in all cases. Once you get much larger, other things come into play. For example, iterating through a loop is significantly slower than getting an item from a Dictionary. An uninspired implementation of a Dictionary lookup would involve iterating through a collection, so it shouldn't be faster. The dramatic increase in performance indicates that a Dictionary is set up in such a fashion that a faster algorithm, like a binary tree search, is being used. Similarly, it may be that a LINQ statement does more than a simple iteration. That may incur an additional cost such that doing that for a small collection is more expensive, but you get that cost back and then some if the collection gets large enough. I don't know that this is the case, so you'd have to do a bit of testing for yourself.
    My usual boring signature: Nothing

  5. #5
    Smooth Moperator techgnome's Avatar
    Join Date
    May 2002
    Posts
    34,532

    Re: dumb question

    And for the record - this really isn't a dumb question. It's actually a good question. And it's one we should all be asking every time we use LINQ or iterate through data. One theory on why LINQ should be faster is that it can be optimized under the covers w/o you needing to change your code. If MS tweaks the logic and issues an update to the FW, you just got an instant upgrade w/o needing to do anything.
    However, as Shaggy noted, and as I noted, nothing beats testing testing testing testing and more testing.

    -tg
    * I don't respond to private (PM) requests for help. It's not conducive to the general learning of others.*
    * I also don't respond to friend requests. Save a few bits and don't bother. I'll just end up rejecting anyways.*
    * How to get EFFECTIVE help: The Hitchhiker's Guide to Getting Help at VBF - Removing eels from your hovercraft *
    * How to Use Parameters * Create Disconnected ADO Recordset Clones * Set your VB6 ActiveX Compatibility * Get rid of those pesky VB Line Numbers * I swear I saved my data, where'd it run off to??? *

  6. #6

  7. #7
    You don't want to know.
    Join Date
    Aug 2010
    Posts
    4,578

    Re: dumb question

    It's not dumb, but it's extremely hard to answer in general. This is why so many experts are typing so many words.

    The compiler knows how to optimize For loops and other basic primitives. For many reasons, arrays can perform better than more complex data structures.

    LINQ uses enumerators, which require several method calls to accomplish what a single array index achieves. It also depends heavily upon lambdas, which are delegates, and delegate calls are always a bit slower than normal method calls. It can also involve closures and other mechanisms that further increase the levels of indirection of every operation.

    I haven't really seen many people point out LINQ was the performance bottleneck in their code. It is definitely slower, but even across millions of iterations I'm suspicious the magnitude doesn't approach "bottleneck". I'm sure someone, somewhere has a workload so great that removing LINQ from it saved them a lot of time/money.

    The only way to find out if it's too great a burden for you is to try. "Millions of iterations" is generally small potatoes in software. Consider that your CPU has multiple cores, each capable of performing billions of operations per second.

    It's generally easiest to write code that uses LINQ first. Then you can profile it. If it's too slow, you can ask other experts what they think and how to optimize it. But I'd argue the most important thing is "get something that works correctly as soon as possible, even if it works too slowly".

    You can refactor correct, slow code to correct, fast code. Writing correct, fast code takes more than twice as long, because "fast" often means "not intuitive".
    This answer is wrong. You should be using TableAdapter and Dictionaries instead.

  8. #8
    Bad man! ident's Avatar
    Join Date
    Mar 2009
    Location
    Cambridge
    Posts
    5,398

    Re: dumb question

    Maybe worth adding, that LINQ, execution of a query is usually deferred until the moment when you actually request the data.

  9. #9

    Thread Starter
    Hyperactive Member
    Join Date
    Jan 2013
    Posts
    486

    Re: dumb question

    Thank you all for your useful input!
    I am simulating a deck of cards, shuffling the cards and dealing the first 5 cards.
    I am barely started. That said my own code is faster the linq.
    Code:
    Option Explicit On
    Option Strict On
    Public Class Form1
        Private rnd As New Random
        Private watch As New Stopwatch
        Private deck(51) As Integer
        Private newdeck(51) As Integer
        Private hold(51) As Integer
        Private hand(4) As Integer
        Private locate As Integer
    
        Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
            Dim counter As Integer = 0
            For y = 1 To 4
                For x = 1 To 13
                    deck(counter) = x * 10 + y
                    counter += 1
                Next
            Next
            watch.Start()
            For x = 1 To 1000000
                Dim shuffle As Integer = 0
                Array.Copy(deck, hold, 52)
                While shuffle <> 52
                    locate = rnd.Next(52)
                    If hold(locate) <> 0 Then
                        deck(shuffle) = hold(locate)
                        hold(locate) = 0
                        shuffle += 1
                    End If
                End While
                hand = deck.Take(5).ToArray
            Next
            'For x = 1 To 1000000
            '    deck = deck.OrderBy(Function(o) rnd.Next()).ToArray()
            '    hand = deck.Take(5).ToArray
            'Next
            watch.Stop()
            For w = 0 To 4
                TextBox1.Text += hand(w).ToString + ", "
            Next
            For w = 0 To 51
                TextBox2.Text += deck(w).ToString + ", "
            Next
            TextBox3.Text = watch.Elapsed.ToString
    
            TextBox1.Text += Int(deck(3) / Int(10)).ToString + "****"
            'TextBox1.Text += (deck(3) Mod 10).ToString + "****"
        End Sub
    End Class

  10. #10
    Super Moderator Shaggy Hiker's Avatar
    Join Date
    Aug 2002
    Location
    Idaho
    Posts
    38,989

    Re: dumb question

    HA! If you added a watch to watch, you'd have a variable watch that was a watch variable.

    A couple (more serious) points:

    1) This is odd: Int(deck(3) / Int(10)). You're casting an integer to an integer, which likely has no impact. There is also an integer division operator (\ rather than /), so you might write (deck(3)\10) for the same result.

    2) That shuffle is pretty inefficient. The first time through the While loop, hold will be full (no zeroes), so you'll hit on the first try. The second time through the loop, you have a 1 in 52 chance of wasting the loop by hold(locate) being 0. After the second successful time through the loop, then the third will have a 2 in 52 chance of wasting the loop, and so on. In theory, you should be spending a great deal of time spinning around that loop hitting an increasingly common case of 0s. In fact, you really only need to perform the loop 51 times, because there is no choice remaining for the final card. That, alone, should remove an average of 52 wasted loops.

    There are better ways to shuffle. I don't know what is the most efficient, though I'd be inclined to try a couple different alternatives. One thing I'd try would be to use a List rather than an array and remove each item as it was chosen. This may not prove efficient at all, because the cost of the removal may be higher than the efficiency gained by the fact that there won't be ANY wasted loops in that case.

    I also once used a design where, when I hit a zero, I then looked either up or down from that point until I found a valid entry and used that. This is not truly random, though, because the odds of picking any one value depends on whether it is bordered by a zero, and how many zeroes are beside it. So, it isn't a true shuffle because it isn't truly random, but I wasn't shuffling a deck, but choosing from a much larger set where being truly random didn't matter so much.
    My usual boring signature: Nothing

  11. #11

    Thread Starter
    Hyperactive Member
    Join Date
    Jan 2013
    Posts
    486

    Re: dumb question

    Shaggy Hiker
    As to your point number 1) I knew it was wrong, but this was the only way I could get the correct answer using the wrong division operator. I used the wrong operator because I've never used the integer division operator before, thus I was basically ignorant of it's use. Thanks to you, that won't happen again.
    Your point number 2) is dead on. I am aware of the gross inefficiency, but in my time trials using this verses the linq statement
    Code:
     deck = deck.OrderBy(Function(o) rnd.Next()).ToArray()
    in a million shuffles, linq is about 1 second slower. I will try a list.

  12. #12
    Bad man! ident's Avatar
    Join Date
    Mar 2009
    Location
    Cambridge
    Posts
    5,398

    Re: dumb question

    How does your teacher assign you homework if you have to learn it?

    can you upload your homework question.....

    Btw we wont do your homework.

  13. #13

    Thread Starter
    Hyperactive Member
    Join Date
    Jan 2013
    Posts
    486

    Re: dumb question

    I am not in school this is for my private use only

  14. #14
    Super Moderator Shaggy Hiker's Avatar
    Join Date
    Aug 2002
    Location
    Idaho
    Posts
    38,989

    Re: dumb question

    Quote Originally Posted by georgesutfin View Post
    Shaggy Hiker
    As to your point number 1) I knew it was wrong, but this was the only way I could get the correct answer using the wrong division operator. I used the wrong operator because I've never used the integer division operator before, thus I was basically ignorant of it's use. Thanks to you, that won't happen again.
    Your point number 2) is dead on. I am aware of the gross inefficiency, but in my time trials using this verses the linq statement
    Code:
     deck = deck.OrderBy(Function(o) rnd.Next()).ToArray()
    in a million shuffles, linq is about 1 second slower. I will try a list.
    Point number one was just for your information. I'm not sure that there is actually a performance benefit to integer division over double division, but I assume that avoiding the conversion has SOME benefit. In theory, floating point division should be faster these days than integer division, but in .NET I believe the two are essentially equal, though I haven't looked in years.

    Point number 2 is kind of cool. I have always found LINQ to be slower, but what you have shown is that, in this case, an inefficient loop is STILL faster. This is one of those odd situations where you can keep playing with timing to come up with more efficient algorithms.
    My usual boring signature: Nothing

  15. #15
    You don't want to know.
    Join Date
    Aug 2010
    Posts
    4,578

    Re: dumb question

    Well, his LINQ statement is needlessly wasteful. .ToArray() is always an "allocate and copy".

    Also that sort algorithm is a "monkey sort", which means its runtime is very hard to analyze and is probably different every time you run it.

    OrderBy() expects you to give it a function that given an object, returns a sortable key. It likely uses QuickSort, which is O(n log(n)). Unfortunately, it assumes the function will return the same thing for the same object every time, and if you're returning a random number that's not true.

    So in theory, if the RNG doesn't behave itself, it could take 100 iterations to sort a 3-item list. Or it could take 1. Or 1,000,000.

    Or maybe not. Maybe OrderBy() is smart enough to cache the values. We'd have to look under the hood to tell. Either way, "shuffle by abusing a sorting algorithm" is cute but never efficient.

    LINQ was not built to shuffle things. It was built to filter, transform, map, and perform several other operations. If you want to shuffle, use Fisher-Yates-Knuth. Shuffling with LINQ is for stupid pet tricks.
    This answer is wrong. You should be using TableAdapter and Dictionaries instead.

  16. #16

    Thread Starter
    Hyperactive Member
    Join Date
    Jan 2013
    Posts
    486

    Re: dumb question

    It is interesting that you feel that way about the linq statement. I don't know linq at all. the statement was provided by a heavy hitter on this forum. I've never heard of the Fisher-Yates-Knuth shuffle and will check it out. I have finished the list shuffle as Shaggy suggested. It is much faster than the monkey sort by about 50%. It will do 1000000 shuffles in about 4 seconds. I am not 100% sure it is perfect.
    Code:
    Option Explicit On
    Option Strict On
    Public Class Form1
        Private rnd As New Random
        Private watch As New Stopwatch
        Private deck As New List(Of Integer)
        Private newdeck As New List(Of Integer)
        Private hand As New List(Of Integer)
        Private locate As Integer
    
        Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
            For y = 1 To 4
                For x = 1 To 13
                    newdeck.Add(x * 10 + y)
                Next
            Next
            watch.Start()
            For y = 1 To 1000000
                deck.Clear()
                For x = 0 To 51
                    locate = rnd.Next(newdeck.Count)
                    deck.Add(newdeck(locate))
                    newdeck.RemoveAt(locate)
                Next
                newdeck.AddRange(deck)
            Next
            hand = deck.Take(5).ToList
            watch.Stop()
            For w = 0 To 4
                TextBox1.Text += hand(w).ToString + ", "
            Next
            For Each card As Integer In deck
                TextBox2.Text += card.ToString + ", "
            Next
            TextBox3.Text = watch.Elapsed.ToString
            ''TextBox1.Text += (deck(3) \ 10).ToString + "****"
            ''TextBox1.Text += (deck(3) Mod 10).ToString + "****"
        End Sub
    End Class
    Please share your thoughts, I am off to explore Fisher-Yates-Knuth.

  17. #17
    You don't want to know.
    Join Date
    Aug 2010
    Posts
    4,578

    Re: dumb question

    Some "heavy hitters" still think "do it on one line" qualifies as a decent quality of software engineering.

    Programming is some kind of Eastern philosophy. Everything we do represents a tension between factors like "clarity" and "performance" and "terseness". Code that executes in one line goes for that "terseness" quality, but it's not always clear how it works nor is it always performant. I often write code in more lines than I "should" because I value clarity higher than terseness. Sometimes I cut some lines I "shouldn't" because I feel the terseness is more important than clarity at that spot.

    Forum answers don't often have enough context to perfectly balance all of the factors like a true sensei can. They have to provide a fast answer that's easy to understand. It's meant to help you make progress, but you're also meant to meditate on the solution and see if it can be improved. He who programs by copy and paste is doomed to never achieve enlightenment.
    This answer is wrong. You should be using TableAdapter and Dictionaries instead.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width