How do I improve (reduce) my app memory and cpu usage
Hello guys,
I am developing an application that manipulates big strings in memory, basically the app filter, sort, extract some strings from plain text files.
The text files am currently working of is 4MB, but am putting a maximum files size of 10 MB in mind as i work.
First i have an open feature, so my app loads the text file to a textbox, and when i click on a button, it starts to do its work.
Observing a compiled version of my app, its cpu usage goes up as high as 99%:eek:, and the memory usage jumps high to 100MB:eek:.
My system has 767MB memory 2.4GH processor power.
In order to make the GUI responsive i put the code in a dll, and before starting the process here is the code i use
vb Code:
'create buffer
sTemp = Space$(Len(txtOut.Text)) ' this line take a minute of more for 3MB text
'copy from textbox to buffer
retval = lstrcpy(sTemp, txtOut.Text) ' this take about 15 seconds for the same 3mb text
where i need your help is to improve this routines so as to take less cpu and memory usage
Is there a way i can assign max memory my app can use?
Please any ideas are welcomed
Re: How do I improve (reduce) my app memory and cpu usage
Quote:
Originally Posted by
coolcurrent4u
In order to make the GUI responsive i put the code in a dll, ...
Since a typical DLL runs in-process and in VB6 on the same thread this isn't going to accomplish what you're hoping for.
Another thing to consider is that approaches that work with smaller amounts of data don't always scale well to large sets of data. "Sucking it all into RAM" is one of these, though as RAM has gotten plentiful the threshold of "large" has moved quite high.
Anything you can do to reduce the amount of data you work with early on should help. For example any filtering or extracting you can do before sorting will make your sorting take a lot fewer resources.
Anything you can do with your algorithms that reduces the number of times you have to "touch" (test, modify, copy, or move) the same data item should help.
There is also the issue of memory management and "chunk sizes" that can come into the picture. Asking for too many or too large arrays (and a String is essentially an array in most ways that matter) can mean a performance hit. Working with a huge number of tiny chunks of data can be a performance problem too though.
Those guidelines are so vague as to be almost useless I admit. However without knowing more specifics it is hard to make specific suggestions.
Some of the factors must be given different amounts of weight depending on the ecosystem your application lives in. "Suck it all into RAM" can fail miserably for server applications, where multiple copies must co-exist in RAM (and with other applications). Such an approach might work fine for systems that are otherwise idle though - like a desktop PC.
I think you know where some of your trouble spots are. But you might also consider running a profiler against your application to see where more subtle bottlenecks are.
Re: How do I improve (reduce) my app memory and cpu usage
This site has string helpers: http://www.xbeat.net/vbspeed/i_Dope.htm
Like, instead of Space$() you could use SysAllocStringByteLen.
Re: How do I improve (reduce) my app memory and cpu usage
Quote:
Originally Posted by
dilettante
Since a typical DLL runs in-process and in VB6 on the same thread this isn't going to accomplish what you're hoping for.
Another thing to consider is that approaches that work with smaller amounts of data don't always scale well to large sets of data. "Sucking it all into RAM" is one of these, though as RAM has gotten plentiful the threshold of "large" has moved quite high.
Anything you can do to reduce the amount of data you work with early on should help. For example any filtering or extracting you can do before sorting will make your sorting take a lot fewer resources.
Anything you can do with your algorithms that reduces the number of times you have to "touch" (test, modify, copy, or move) the same data item should help.
There is also the issue of memory management and "chunk sizes" that can come into the picture. Asking for too many or too large arrays (and a String is essentially an array in most ways that matter) can mean a performance hit. Working with a huge number of tiny chunks of data can be a performance problem too though.
Those guidelines are so vague as to be almost useless I admit. However without knowing more specifics it is hard to make specific suggestions.
Some of the factors must be given different amounts of weight depending on the ecosystem your application lives in. "Suck it all into RAM" can fail miserably for server applications, where multiple copies must co-exist in RAM (and with other applications). Such an approach might work fine for systems that are otherwise idle though - like a desktop PC.
I think you know where some of your trouble spots are. But you might also consider running a profiler against your application to see where more subtle bottlenecks are.
What profiler do you recommend? The Profiler i found does not help
Re: How do I improve (reduce) my app memory and cpu usage
I like VB Watch myself, and they have an evaluation version that can handle small projects (10 modules I believe). If you have VB/VS 6.0 Enterprise there is a profiler included too (APE is its name as I recall).
I'm not sure what other commercial products are still available though.
Re: How do I improve (reduce) my app memory and cpu usage
Thank you dilettante,
I'll look at vbwatch trail. I am using visual basic professional, so it doesn't have APE. An i dont think APE is for testing individual application stress/problems see here.
any other suggestions would be welcomed
Re: How do I improve (reduce) my app memory and cpu usage
Quote:
Originally Posted by
dilettante
Since a typical DLL runs in-process and in VB6 on the same thread this isn't going to accomplish what you're hoping for.
so do i use activex exe instead of dll?
Quote:
Originally Posted by
dilettante
Anything you can do to reduce the amount of data you work with early on should help. For example any filtering or extracting you can do before sorting will make your sorting take a lot fewer resources.
Usually i follow this step when working with strings
Functions
Code:
Extract > sort > remove duplicates > filter > group
what would be your advice here?
Quote:
Originally Posted by
dilettante
Anything you can do with your algorithms that reduces the number of times you have to "touch" (test, modify, copy, or move) the same data item should help.
do you have better algorithms for the above named functions that would be faster?
Re: How do I improve (reduce) my app memory and cpu usage
Allowing Users to Interrupt Tasks is the classic approach to background processing. There is also the DoEvents hack, but of course it has its risks.
Other people offload background processing to another program, either a console program "connected" via anonymous pipes or an ActiveX EXE rigged for background processing on its own thread.
As far as data reduction goes, you do what you can. Not having duplicates is best. If you do have them though you may be forced to sort first, and take the performance hit. Prevention is usually cheaper than "cure" when possible.
"Better" algorithms are entirely situational. An example though might be to used hash-based keyed access for locating items in a large list instead of sequentially searching every time you need an item. Binary searches work well if your lists are ordered appropriately. Linked-lists can be better than moving long chunks of data when adjusting the sequence of a list, etc. Sorting is a performance killer when you don't truly require a fully sorted list. Not that any of that is news.
It is hard to make specific suggestions without considering specific cases. I suspect that's why others are holding back their comments so far.
Re: How do I improve (reduce) my app memory and cpu usage
I'll past the code am using so far, maybe you'll see some bugs. They are all in a class module
Here is the extract function
'
vb Code:
extract Items from text
Public Sub Extract(ByRef Value As String)
Dim inGroup As Integer
If sSeperator = vbNullString Then
RaiseEvent Error("No Pattern defined", vbObjectError + 500)
Exit Sub
End If
RaiseEvent Status("Extracting", 1)
'Set the pattern by using the Pattern property.
objRegExp.Pattern = sPattern
' Set Case Insensitivity.
objRegExp.IgnoreCase = True
'Set global applicability.
objRegExp.Global = True
'Test whether the String can be compared.
If (objRegExp.Test(Value) = True) Then
'Get the matches.
Set colMatches = objRegExp.Execute(Value)
' Execute search.
For Each objMatch In colMatches
Do_Events
' Iterate Matches collection.
Append objMatch.Value & sSeperator
Next
lCount = colMatches.Count
End If
Set colMatches = Nothing
End Sub
Here is the append function
vb Code:
Private Sub Append(s As String)
Dim L As Long
L = StrLen + Len(s)
While L > Len(Buffer)
Buffer = Buffer & Space$(BufferSize)
Wend
CopyMemory ByVal StrPtr(Buffer) + (StrLen * 2), ByVal StrPtr(s), LenB(s)
StrLen = StrLen + Len(s)
End Sub
please let me know what you find.
Thank you for you response so far
