[RESOLVED] High CPU usage

[Luinox86]

Hi guys,

I am programmer of a project (VB6), my program reads lots of data from a big text file, parse the information and start some mathematics calculation on every single line of the data.

Anyway, a run of my program takes like 10minutes and every second a calculation is being applied, so while my program is running the CPU usage is always up to 50% so causes my Windows XP to malfunction and slow down. Is there anyway to reduce the CPU usage? I am using DoEvents after each reading and parsing of the data and also after each calculation.

Any hints used to solve this kind of problem is appreciated guys.

[dilettante]

A program like this will burn all of the CPU it can get, and when single-threaded that's most of one CPU, hyperthread, or core depending on the machine architecture.

Higher-priority tasks will pre-empt it after each time slice, but in most machines most of the tasks run at roughly the same priority. So if your process "hogs" close to 100% of each slice it will tend to starve the other tasks on the machine.

About the only things you can really do are either optimize the code (lots of which might be inefficient I/O) or else give up your timeslices more quickly.

The latter is what DoEvents() calls and Sleep() calls do. DoEvents() does a lot of extra work though, running around handling pending window messages and some other housekeeping chores. It is going to slow your program down quite a bit, but will improve system responsiveness by a little at least. "Inside" every DoEvents() function call you perform a Sleep 0 call, which is where the timeslice is given up.


People often write the main loop of such crunching so that every N iterations they call DoEvents() and then call Sleep M, where N in iterations and M in milliseconds require trial and error tuning. In such cases they want the DoEvents() housekeping but also want to stay off the ready queue for a longer time to let other tasks use more of the processor.

The tuning and tweaking is mostly art and little science. It tends to have so-so results because DoEvents() still involves a lot of wasted CPU cycles, and making M too large extends the time required for the whole job by huge amounts.


What you need here is a background worker thread. Sadly there are no features in VB6 that make such a thing simple and reliable to implement. There are a few techniques you can use involving an ActiveX EXE which with some effort can provide you additional works threads in its own process. Few people do this though because it tends to be a bit awkward.

The advantage of background workers is that even though they still need to give up the CPU more often than running in a tight loop they don't need to service window messages. That may be too small an advantage to help you by itself here.


Anything you can do is going to slow down your data crunching, with the possible exception of optimization. Your PC just isn't as powerful as you imagine. Most programs normally only do little bits of processing then go idle for a while. People do some silly things they think are cheap but that eat tons of CPU.

Normally pushing hard for optimization doesn't matter because over the life of the program you aren't doing very much. You have a case here where being so cavalier is hurting you.

I would look hard at your disk reading and parsing logic.


Even better, test with a smaller set of data while running a profiler. That can tell you what is worth optimizing. It is also the best place to begin.

http://www.aivosto.com/vbwatch-shots.html#profg is free for profiling Projects of 10 modules or less.

[Luinox86]

Thank you very much dilettante,

Your information was so useful, though, I am not familiar with background worker thread and I have to look into it.

By the time being, the logic of parsing the text file is not very important (I think) because that's only the first step. I am writing a chess game analyzer which opens a PGN file (database of chess games).

The part that I am reading the file is not taking too much effort, as I just open the file, capture all the information in an array and close the file. Then further analysis is only internally.

The matter is that I have lots and lots of text manipulations which modifying it would take for ever. Also many nested loops which I can't remove them. About the modification, my code is around 15,000 lines and any modification should be applied into all modules and function which takes lots of time.


So what I got from your text is that you think that the better way is not to look for tricks to run the program faster and instead trim my code as much as possible, isn't it? And yep! change my laptop of course!

I will look into the options you just mentioned and profiling. Thanks again.

[si_the_geek]

It sounds like reading the file probably is ignorable.

The slowest (and most CPU intensive) parts are likely to be the text manipulations and nested loops, and there are various ways that they could potentially be improved based on what they do, and how well optimised they are already.

The loops will probably be the easiest to improve, as they are unlikely to require changes to the rest of the code.


In terms of hogging the CPU, one way to avoid that (which will probably slow the program down) is to move the code to a timer - but I would recommend leaving that until after you have optimised the code a bit.


If you want help with optimising, show us a section of code that is a bottleneck at the moment (which the profiler will probably make clear to you).

[dilettante]

Don't go out and buy a new PC right away. The reduction in overall elapsed time might be minimal.

So much depends on what you're really doing. Often you can find things you can avoid doing repeatedly, or things you can remove from inner loops (er, which is often the same thing). Or choose a thriftier data representation that doesn't need so much crunching, like replacing String values by numeric codes that fit in a Long or Integer or even a Byte.

There are other places where you can spend more memory but get tighter, faster code.

But profiling should help you find the "hot spots" worth the effort. It helps if you can find a way for the program to stop early if every case normally must run a long time. Those profilers write a lot to disk and then analyze it, so you usually wouldn't want to let this run all the way though. It will take forever and use a ton of disk.

[MartinLiss]

In the 'Project' menu choose <Your Project> Properties… and then go to the 'Compile' tab. Once there make sure you have 'Compile to Native Code' and 'Optimize for Fast Code' selected.

[Luinox86]

Thank you si_the_geek, dilettante and MartinLiss,

si_the_geek, can you explain a bit more how can I use a loop in a timer? I don't get the logic.

dilettante, I think you are right, actually right now I don't have time for profiling as I am under time limitations, but as soon as I get I will get into the profiling and show you the results. But you are absolutely right about the text manipulations, you know how a chess board works, imagine a board that has 64 cells, I have two methods to mention each particular cell, one is by integer 0 to 63 and another is by coordinates A8 to H1, unfortunately from the first day, timing and CPU usage was not my concern and all the functions I have written are based on the coordinates (like 30 functions work as they take the coordinates then convert it into integer) which I think is one of the bottlenecks.

I think as soon as possible I should just take out a part of my code and try to simplify it with converting the text coordinates into integer values, then I will show all of you that piece of code and see which can help it.

MartinLiss, I am going to try the two options you mentioned, but can you explain how does it work practically? I mean logically it 'changes' my code or just changes the method of 'compiling to exe' if I tick those options?
EDIT: MartinLiss, I just checked my properties and its already set to be 'Compile to native code' as well as 'Optimize for fast code'.

Thanks everyone.

[MartinLiss]

From MSDN

Optimize for Fast Code


Maximizes the speed of compiled executable files by instructing the compiler to favor speed over size.

When the compiler translates Visual Basic statements into machine code, there are often many different sequences of machine code that can correctly represent a given statement or construct. Sometimes these differences offer trade-offs of size versus speed. Selecting this option ensures that when the compiler recognizes such alternatives it will always generate the fastest code sequence possible, even when that may increase the size of the compiled program.

[si_the_geek]

si_the_geek, can you explain a bit more how can I use a loop in a timer? I don't get the logic.

The idea is that the timer would replace the loop, and any DoEvents/Sleep calls.

As a basic example, if you currently have code like this:

...
  For intLoop = 1 to 1000

    DoSomething
    DoSomethingElse

    Sleep 100
    DoEvents
  Next x
  MsgBox "Finished"
End Sub

...you would replace it with a variable at the top of the code file (eg: Private m_intLoop as Integer), and code like this:

...
  m_intLoop = 1
  Timer1.Enabled = True
  Timer1.Interval = 100
End Sub

Sub Timer1_Timer

  DoSomething
  DoSomethingElse

  m_intLoop = m_intLoop + 1
  If m_intLoop = 1000 Then
    MsgBox "Finished"
    Timer1.Enabled = False
  End If
End Sub

The value of Timer1.Interval can be changed to alter the balance between CPU % used and the total amount of time to run, and you can even change that while the program is running (if you create an interface for the user to alter it).

[dilettante]

Another option is to use a worker process since you can't easily use worker threads in VB6.

This gets a little complicated, because to write an effective worker in VB6 you'll want to relink it for the Console Subsystem to free it from all of the message processing. Then you have the matter of passing data between the two processes, which for a child Console Subsystem process would be most naturally via redirected standard I/O streams.

However if you just need to get a blob of data ready to crunch on and get a blob back after the worker is done, you might just communicate via an input and an output disk file. You can also start the child/worker with command line parameters if that is useful.


To create the worker program, write it with no Forms. Just use static modules and Classes as required, with Sub Main() as the entrypoint. You can relink it as required after compiling by dragging and dropping the EXE onto the file:

LinkConsole.vbs

Option Explicit
'LinkConsole.vbs
'
'This is a WSH script used to make it easier to edit a
'compiled VB6 EXE using LINK.EXE to create a console
'mode program.
'
'Drag the EXE's icon onto the icon for this file, or
'execute it from a command prompt as in:
'
' LinkConsole.vbs <EXEpath&file>
'
'Be sure to set up the LINK path to match your VB6 installation.

If WScript.Arguments.Count < 1 Then
  WScript.Echo "Missing EXE parameter." & vbNewLine _
             & "Use LinkConsole.vbs <EXEpath&file>" & vbNewLine _
             & "or drag EXE onto this script's icon."
Else
  With CreateObject("WScript.Shell")
    .Run """C:\Program Files\Microsoft Visual Studio\VB98\LINK.EXE""" _
       & " /EDIT /SUBSYSTEM:CONSOLE " _
       & """" & WScript.Arguments(0) & """"
  End With
  WScript.Echo "Complete!"
End If

Do that again every time you recompile the worker process program.

From there you just need to use a Shell-and-Wait technique in your GUI parent program (which itself needs no special treatment). However even so...

... you will want to put something like a Sleep 10 somewhere in the loops to give up the CPU a bit every so often. No DoEvents() required.

[Luinox86]

Thanks si_the_geek and dilettante and MartinLiss,

si_the_geek, I should have to give the timer-loop a try to get the logic completely and play around using the code you have given a bit to be familiar with it. Thank you.

dilettante, frankly, I will have some questions when applying your method. I should have to study that to get the logic first,

By now guys, I am using DoEvents in my program and no "sleep", I know what the DoEvents does, it takes the handler from the program and checks if other Windows programs needs it, so the system wouldn't hang for the 10 minutes or so and it looks active. But I am not totally clear how to replace them with the sleep as dilettante mentioned, sleep pauses the execution for the determined milliseconds or what? And it's usage is just like the DoEvents?

What I am doing with the DoEvents is for example in a for loop which has heavy calculations in each round, I put the DoEvents after each run so after getting each result it would have a breath-break to take over other actions, but how the sleep would work? Shall I put the sleep at the same place as the DoEvents in my code?

[dilettante]

No, it isn't as easy as substituting Sleep for DoEvents.


These articles provide a simplified view of the topic:

How To Determine the Differences Between DoEvents and Sleep

Definition of DoEvents in Visual Basic for Applications

Effect of DoEvents in Visual Basic OLE Server DLLs

As Mr. Mackey says: DoEvents is bad, m'kay?


A more advanced approach involves SetWaitableTimer combined with DoEvents. It has most of the risks of DoEvents alone or DoEvents with Sleep, but it has the advantages of (a.) not wasting as much CPU by calling DoEvents when not required and (b.) not throwing away unused CPU cycles by Sleeping when not required.

How To Use SetWaitableTimer With Visual Basic

[coolcurrent4u]

be careful when using doevents, if you search the forum for " doevents2" or "better doevents", you'll see that the vb6 native doevents is evil

[Luinox86]

Thank you guys. I will look into the matters you mentioned, frankly, I have been under extreme time pressure and I have to suspend worrying about the speed to do some other stuff. I will get back to you guys and try your suggestions as soon as possible.

Thanks all.