[RESOLVED] Deployed 64 bit app takes long time to load, but only the first call

[HumePeabody]

I have an application that calls my GetCSVData method of an object. When I'm running on my development machine the time between calling this function and entering the function is negligible. However, when I compile it as a 64 bit application and execute the code on a client machine, it takes 67 seconds just to go into the function. My development machine is only 32 bit, so I cannot test this out in a 64 bit development mode. I have something like this set up <pseudocode>:

Function A
blah blah blah
msgbox("Calling GETCSVData")
object.GetCSVData(...)
blah blah blah
End Function A

Then in the Object definition
Public Sub GetCSVData(...)
Msgbox("Starting GetCSVData")
...rest of code...

The first time I call this from within the code as a result of a button click, it takes 67 seconds between the two message boxes. There are no other threads running at this point that I control. The next time(s), it is instantaneous, even if called from another instantiation of object. I'm going to try to put a thread running in the background on startup just to call this function, but I would love to know how to fix it properly. Any thoughts are welcome. Thanks in advance.

Hume

RESOLUTION: Having Resume commands in the error handler inexplicably seemed to cause the issue. Commenting those out resulted in the function loading much faster. Special thanks to Grimfort, Shaggy Hiker, and formlesstree4 for their help!!

[formlesstree4]

If you can post the actual code, perhaps we can help more?

[Shaggy Hiker]

That's some pretty weird behavior. That much of a delay sounds like something is getting loaded as a result of the call. That would also account for the performance going back to normal the second time. The delay is too long for pretty nearly any JIT compilation happening that I have seen, though the way you stated the behavior, it remains a remote possibility.

What is 'object'? Is it really an Object and you are using late binding to call the method, or is it some type of class that you created? Is there anything unusual about it, such as it being a Singleton?

[HumePeabody]

Some more information on the problem.

Before it calls the GetCSVData the application is using 40,172 KB of memory (per the Windows Task Manager). Once it calls GetCSVData it peaks out at a whopping 1.3 GB of memory and then eventually backs down to 81,436 KB at the first line of the GetCSVData function itself. The problem does occur on multiple 64 Bit Client machines.

It is compiled as an x64 application, but under 32 bit Vista. Maybe compiling it under Windows 7, 64 bit would be better. I can try that tomorrow.

'Object' in the pseudo code is a tarpDataSet object which is my own Class. The GetCSVData is one of the heavy lifting functions that reads an end users output file and stores the data in the class. It is about 4000 lines of code and all this happens before any variable declarations (the msgbox is the first line after the sub declaration).

Not sure what is meant by "Singleton". It is one item in a collection of tarpDataSets based on what the user has requested through the GUI. Nothing exotic about it. I'm going to try this as a 32 bit application and see what happens as well.

[HumePeabody]

Further information:

The problem does not occur for a 32 bit installation of the application on same client machine where it does have issues with 64 bit installation. Still compiling both x86 and x64 application on 32 bit Vista machine though...

[formlesstree4]

I'd like to know some details of GetCSVData() if possible. I think the issue might lie there, but it could be a red herring.

[HumePeabody]

What do you want to know? It is 4000 lines long, but all this time is between a msgbox just before the call to the method and a msgbox which is the first line of code in the method, before any variable have been declared or execution lines have been reached. In short the method parses through the file specified by the user and extracts the data requested from 10 or so different output formats supported. This main loop to process the file is about the first half. The second half is processing on the back end to make sure that everything is populated and of similar format once the data is made available for further processing. I'm happy to give out information that can help, but since this occurs between essentially two lines of executable code (one just outside the method and one just inside) I'm struggling to see how all the lines afterwards (the meat of the method) play a role. I could see it if the function required gobs of memory, but seeing that it goes back down to a reasonable memory size after it reached the inner msgbox suggests that the GetCSVData code is not the obvious problem. Let me know what I could provide that may help, but I do think the code itself is a red herring as you say, especially since it only happens on the first call to the function and only on a 64 bit compilation

Hume

[HumePeabody]

What do you want to know? It is 4000 lines long, but all this time is between a msgbox just before the call to the method and a msgbox which is the first line of code in the method, before any variable have been declared or execution lines have been reached. In short the method parses through the file specified by the user and extracts the data requested from 10 or so different output formats supported. This main loop to process the file is about the first half. The second half is processing on the back end to make sure that everything is populated and of similar format once the data is made available for further processing. I'm happy to give out information that can help, but since this occurs between essentially two lines of executable code (one just outside the method and one just inside) I'm struggling to see how all the lines afterwards (the meat of the method) play a role. I could see it if the function required gobs of memory, but seeing that it goes back down to a reasonable memory size after it reached the inner msgbox suggests that the GetCSVData code is not the obvious problem. Let me know what I could provide that may help, but I do think the code itself is a red herring as you say, especially since it only happens on the first call to the function and only on a 64 bit compilation

Hume

[formlesstree4]

Then let's work backwards. What calls this method?

Also, does this issue happen on all 64 bit computers or just a specific test case? It could be a corrupt installation of .NET if that's the case.

[HumePeabody]

What ends up calling this method is a button click, which uses a FileOpen to allow the user to specify a filename. This then assigns the filename to a TextBox. When the TextChanged event gets fired for that text box, this GetCSVData function gets called with the necessary arguments for the currently selected DataSet specified by the user. The issue happens on at least 2 64 bit client machines. I cannot say if it happens on more or not yet. Also, correct me if I'm wrong but there is not a 64 bit .NET framework and a separate 32 bit .NET framework is there? This works fine on those same two machines as a 32 bit install. It also happens if I take a separate isolated thread at the very beginning of the application and create a temporary tarpDataSet object and call the GetCSVData method running independently of the main thread. I think my next best hope is to try to compile this fro 64 bit on a true 64 bit OS and see what happens then...

Hume

[formlesstree4]

Wait wait. What happens when you create a tarpDataSet Object? Is there any special code in the constructor?

[HumePeabody]

Nothing major in the tarpDataSet constructor. The tarpDataset instance is created long before the user specifies a file name for it. The long time to load and the high memory usage are literally between (just before calling the method) and (the first line of the method). Nothing else is running that I'm controlling. I'm very curious about compiling on a 64 bit OS, which I will try tonight as soon as I get home from my day job. I'm also going to try out a test case of temporarily removing all the code in the GetCSVData method and see if the problem persists.

[formlesstree4]

Nothing major in the tarpDataSet constructor. The tarpDataset instance is created long before the user specifies a file name for it. The long time to load and the high memory usage are literally between (just before calling the method) and (the first line of the method). Nothing else is running that I'm controlling. I'm very curious about compiling on a 64 bit OS, which I will try tonight as soon as I get home from my day job. I'm also going to try out a test case of temporarily removing all the code in the GetCSVData method and see if the problem persists.

There shouldn't be a difference. Heck, if you want to make sure there isn't a difference, change your target mode from (probably) AnyCpu to x86 as that'll force it to run 32 bit on 64.

[HumePeabody]

Setting it to AnyCPU caused me problems since it is compiled on 32 bit VISTA and then believed to be a 64 bit application when deployed under 64 bit Windows 7 client. It would crash immediately if I compiled it as AnyCPU and deployed it to the 64 bit Windows 7 client. Hence the explicit x86 or x64 on compile. Currently, I explicitly set it to x86 or x64 depending on what I'm trying to see in the deployment. If it does not make a difference to compile on a 64 bit OS, then I will nearly be out of bullets and may just have to accept the delay (although that is not optimal for my customers). It does work; it just takes longer than it should...

[formlesstree4]

Try just setting to x86 and deploying it to a x64 bit platform.

[HumePeabody]

That is what I have been doing. Switching between x86 and x64 and seeing how it behaves on a 64 bit platform. I still wonder if the 32 bit Vista OS that it is being compiled under is the issue

[formlesstree4]

That is what I have been doing. Switching between x86 and x64 and seeing how it behaves on a 64 bit platform. I still wonder if the 32 bit Vista OS that it is being compiled under is the issue

So you are getting the same issue regardless of the target platform?? o.o

[HumePeabody]

No, I only have the issue if I compile as x64

Compile as x86 on 32 bit Vista, deploy to Win 7 64 Pro - works fine
Compile as x64 on 32 bit Vista, deploy to Win 7 64 Pro - has delay
Compile as x64 on 64 bit Win 7 Home edition, deploy to Win 7 64 Pro - has delay, but seems a bit less

I removed all code from the function, save the declaration and then tried it and it does not have the delay on x64. So hte next things I will try are to add back in part of the function and see what happens. I also want to do a binary file compare on the 32 bit Vista compiled exe and the 64 bit Win 7 compiled exe and see if they are different.

Is there any way to find out how much memory a function/subroutine is using in memory? That is likely way beyond my knowledge at this point, but it would be interesting to compare how much memory is needed to load the GetCSVData method for 32 bit and 64 bit compilation, but it should not be the huge number I'm seeing I suspect.

Hume

[formlesstree4]

So why bother compiling to x64 if it works fine as x86? I agree that's weird behavior, but I think you should just compile to x86 period.

[HumePeabody]

I want to compile as x64to take advantage of larger allocatable memory. I have some users with particularly large data files that run into memory problems and compiling to x64 should hopefully address those users. I plan to offer both 64 bit and 32 bit versions of my product. This may just have to be a "feature" that my x64 users have to deal with. This function often gets called twice; once for preprocessing to get some information from the users file (once specified) and then processing to read in all the data (when producing the final output from my app). I could consider moving the preprocessing functionality into its own smaller function since that is the one that needs to be immediately available and then calling the larger one that is causing the issue in the background to allow it to make it through once before being needed for a production run.

[formlesstree4]

I want to compile as x64to take advantage of larger allocatable memory. I have some users with particularly large data files that run into memory problems and compiling to x64 should hopefully address those users. I plan to offer both 64 bit and 32 bit versions of my product. This may just have to be a "feature" that my x64 users have to deal with. This function often gets called twice; once for preprocessing to get some information from the users file (once specified) and then processing to read in all the data (when producing the final output from my app). I could consider moving the preprocessing functionality into its own smaller function since that is the one that needs to be immediately available and then calling the larger one that is causing the issue in the background to allow it to make it through once before being needed for a production run.

Until you really need it, just worry about getting x86 working. If you're not pushing the limits, then don't worry

[HumePeabody]

I'm already pushing limits for some customers files; that's why I want to offer the 64 bit version. I know that I have come across some files that will hit the OutOfMemory exception and am hoping this will solve that problem. This may just have to be a trade they live with. If you work with smaller files, you can use the issue free version; if not, well then if you get an answer in the end and it takes more time it is better than getting no answer at all. But I would still like to understand why this is happening

[formlesstree4]

I'm already pushing limits for some customers files; that's why I want to offer the 64 bit version. I know that I have come across some files that will hit the OutOfMemory exception and am hoping this will solve that problem. This may just have to be a trade they live with. If you work with smaller files, you can use the issue free version; if not, well then if you get an answer in the end and it takes more time it is better than getting no answer at all. But I would still like to understand why this is happening

Ahhh now that makes sense. Hmmm...I would ask to actually see this source code myself in a PM, but that's up to you. I honestly have no idea why it would be doing this without taking a look at things.

[HumePeabody]

Let me see if I can gen something up that shows it. It is 5+ years of development and I don't want to burden you or expose me to more than necessary ;-).

[formlesstree4]

Let me see if I can gen something up that shows it. It is 5+ years of development and I don't want to burden you or expose me to more than necessary ;-).

It's quite alright. You have me rather curious now!

[HumePeabody]

OK, some new information. I finally got a chance to try things out on my Win 7 Home Edition 64 bit OS machine. It actually has this delay in the debug mode and only when I change the configuration from x86 to x64. It is also throwing back an exception that I need to try to trace where it is hitting (since it does not break there): "A first chance exception of type 'System.IO.FileNotFoundException' occurred in mscorlib.dll". I googled this and it looks like quite a few folks have had this issue. When I am in x86 mode, it took 640 ms to get into the function. When I switch over to x64, it takes 112413 ms to reach the same point. NO OTHER CHANGE AT ALL!!! So, that exception is doing something that is sucking the resources from running the code. I never saw this on my development machine, since it cannot run a 64 bit version on the 32 bit Vista OS. Stay tuned and I'll let you know what else I can dig up. And, btw, thanks for your help with this. I do greatly appreciate it.

Update: I did find the offending code that was causing the FileNotFoundException and removed it, but it still has the delay

[formlesstree4]

So something else is causing the issue...this is very interesting! Glad you found the offending code for the FileNotFoundException. Every little bit helped.

[HumePeabody]

I'm going to slowly start stripping out "unnecessary" parts of the code for troubleshooting this issue and systematically see what happens with load/run times. Once I get it down to a sufficiently skeletal frame that still has the problem, I'll send it to you. If you want to see the behavior as x64 and x86 for a compiled, deployed application, I'd be happy to provide you a link over PM to the install package (assuming you trust my code ;-) If not, I completely understand not wanting to install software that you don't know where it came from...we can't be too careful these days...

[dbasnett]

Looking around the net it seems that others have this problem. The common solution I have found has to do with NGEN and causing it to do an update. Here is one such answer:

"At a command console enter this:

cd C:\Windows\Microsoft.NET\Framework64\v4.0.30319

ngen update
"

I believe that you must run the command prompt at the highest level. I don't understand what it does, but others have found it helpful.

[Shaggy Hiker]

An exception of this sort would cause the behavior you are seeing on the first run, but it is very peculiar that it would happen only the first time. That makes it sound like there is some kind of internal flaw where something internal wasn't initialized correctly, failed the first time, then got initialized correctly and worked after that. That leads to this question: Does this happen in a production installation, and have you tested that, or only in development? That would be ideal, but I'm not sure you have actually stated whether you have gotten as far as deployment testing.

The other question I have is whether or not this happens at program launch, or as a result of some user interaction? If it is the latter, one odd hack you might consider is launching a background process that makes the call during startup. If that incurred the penalty, then you might be able to throw out the background process and all subsequent foreground processes initiated by the user would become "the second time", which you say works fine.

[HumePeabody]

I fixed the issue with the FileNotFoundException (it was an oversight by me where the text stream that would have been created for the non existent file was ignored anyways). So that problem is retired at this point.

I did put in a thread at the very start as you suggested to try to get it through the problem section as quickly as possible, but one of the first things a user might do is to provide a file at which point the GetCSVData is called to extract part of the information from the file that the user needs without processing everything. It is only at user interaction (except for the background thread I hoped would get through it quickly). I started to see it in a production deployment on a 64 bit machine (although it was compiled and published under 32 bit Vista as Debug and not Release (not sure what the real difference is at deployment between Debug/Release)). But I could only run it in 32 bit mode in development, since my development machine is a 32 bit OS. I just recently tried it in development on a 64 bit OS and saw the behavior that has plagued me with 64 bit.

I'm considering making a separate function for the file preprocessing and the actual processing, which are currently both handled by my GetCSVData with different arguments passed in. If I could get a smaller function that doesn't have this problem, then the 2 minutes to load the main function may be tolerable running in the background while the user is working around in the front end GUI. But I am still troubled by this behavior only happening when I compile for 64 bit and no sign of it for 32 bit.

I ran the ngen update and then tried my app again and same behavior. Needs about 1.4 GB of memory and takes 113 s to get into the GetCSVData subroutine that needs less than 1 s with a 32 bit platform compile option.

Hume

[HumePeabody]

I took some time tonight and stripped out just about all functionality from my application. It is down to pretty bare bones and still demonstrates the problem when compiled as x64, but not as x86. I can send the source along to anyone who is curious about this. PM me and we'll go from there. Thanks for any help folks are willing to give.

[HumePeabody]

I downloaded VS2012 and tried my simplified code there and have the same problem with it taking a long time to load. Hacking out parts of it seems to indicate that the number of lines of code and the time to load are related, but not necessarily linear. I tried 4 configurations: no code in the function at all, it takes about 0 seconds to load. With only main loop 41.780 s; No main loop (only post processing) 5.69 s; and the whole thing 107.25 s. All of this is for an x64 compile. Running it twice serially in a thread gives 0 ms to load them each a second time. If I do the same thing for x86 it is 0, 330 ms, 92 ms, and 522 ms with 0 to reload a second time. On a recommendation from another user, I also tried it without all the optional arguments to the function, but that had no impact on the delay.

[Grimfort]

I will take a look-see if you don't mind. I have used mem-profiler in the past to nail down similar issues.

[Grimfort]

I have no idea why, but removing the Resume line at the button of the sub fixed it for me... I can't understand what most of your code is doing, but possibly this should be in a loop with exception handling rather than a goto and resume?

End If
Exit Sub
' Resume <<
End Sub

[Shaggy Hiker]

Depending on where that Resume line was, standard exception handling might show that you are throwing a series of exceptions the first time.

[HumePeabody]

The Resume I put in was generally for debugging purposes when it would go to the generic error handler, I could drag the current line to the resume to have it go back to where the offending line was. I agree that structured exception handling would be better overall. I'll try to remove the resume in the app tonight and see what happens. Curiosity is killing me right now!!!

[Grimfort]

Depending on where that Resume line was, standard exception handling might show that you are throwing a series of exceptions the first time.

There were no exceptions being raised, stepping into this sub actually took over 2 mins on my laptop under x64, on x32 it took no time at all. At one point I thought it was working OK on framework 4.5 x64, but then could not repeat it, it did this for all .net version from 2.0 up to 4.5 in my tests. The code was a bit odd and confusing, for example, you have classes within classes within classes, for which I could see no real purpose. You are inheriting collections, whereas you could possibly just create the generic collections instead. Of course I don't know the history of this code and know how things can build up. I would say to split all that code in the GetCSVData method into separate methods, you have actually commented out sections 1,2,3-7 etc and most of your context seemed to be class variables. But hey, if it works...

[HumePeabody]

Brilliant! Sure enough 102000 ms to load with the Resume and 2500 without it. I will replace all my resumes with a 'DebugResume so that I can re-enable them when debugging and leave them out when compiling for production mode. As for the structure of the code, this is a small excerpt from a much larger program. That program has many collections of varying object types (classes), some of which are inter-dependent. To troubleshoot this issue, I stripped out as much as I could to still reproduce the problem. The GetCSVdata is meant to be a single interface to handle multiple formats that the user could throw at it as far as output files to process, without having to have a separate reader for each format type. The entire project is a hierarchy of objects kind of like XL with XLApp, XLbook, XLsheet, XLchart, etc, just for the purposes my program provides. Thanks to all who took a look at this. On my list of things to try, "remove resumes" was somewhere around 200000; I would have never even thought to try that. I will mark this one resolved as the workaround, but the behavior is still strange and I would love an explanation from MS as to what is really going on under the hood.

[Grimfort]

I'm going to throw out a guess, that the engine will parse the entire method upfront to work out the possible paths through, and that resume along with the gotos is causing some confusion. I would suggest adding #If Debug statements to help with debugging pathing.