How do I improve (reduce) my app memory and cpu usage

[coolcurrent4u]

Hello guys,

I am developing an application that manipulates big strings in memory, basically the app filter, sort, extract some strings from plain text files.

The text files am currently working of is 4MB, but am putting a maximum files size of 10 MB in mind as i work.

First i have an open feature, so my app loads the text file to a textbox, and when i click on a button, it starts to do its work.

Observing a compiled version of my app, its cpu usage goes up as high as 99%, and the memory usage jumps high to 100MB.

My system has 767MB memory 2.4GH processor power.

In order to make the GUI responsive i put the code in a dll, and before starting the process here is the code i use

'create buffer
 sTemp = Space$(Len(txtOut.Text)) ' this line take a minute of more for 3MB text
'copy from textbox to buffer
 retval = lstrcpy(sTemp, txtOut.Text) ' this take about 15 seconds for the same 3mb text

where i need your help is to improve this routines so as to take less cpu and memory usage

Is there a way i can assign max memory my app can use?

Please any ideas are welcomed

[dilettante]

In order to make the GUI responsive i put the code in a dll, ...

Since a typical DLL runs in-process and in VB6 on the same thread this isn't going to accomplish what you're hoping for.


Another thing to consider is that approaches that work with smaller amounts of data don't always scale well to large sets of data. "Sucking it all into RAM" is one of these, though as RAM has gotten plentiful the threshold of "large" has moved quite high.

Anything you can do to reduce the amount of data you work with early on should help. For example any filtering or extracting you can do before sorting will make your sorting take a lot fewer resources.

Anything you can do with your algorithms that reduces the number of times you have to "touch" (test, modify, copy, or move) the same data item should help.

There is also the issue of memory management and "chunk sizes" that can come into the picture. Asking for too many or too large arrays (and a String is essentially an array in most ways that matter) can mean a performance hit. Working with a huge number of tiny chunks of data can be a performance problem too though.


Those guidelines are so vague as to be almost useless I admit. However without knowing more specifics it is hard to make specific suggestions.

Some of the factors must be given different amounts of weight depending on the ecosystem your application lives in. "Suck it all into RAM" can fail miserably for server applications, where multiple copies must co-exist in RAM (and with other applications). Such an approach might work fine for systems that are otherwise idle though - like a desktop PC.


I think you know where some of your trouble spots are. But you might also consider running a profiler against your application to see where more subtle bottlenecks are.

[FireXtol]

This site has string helpers: http://www.xbeat.net/vbspeed/i_Dope.htm

Like, instead of Space$() you could use SysAllocStringByteLen.

[coolcurrent4u]

Since a typical DLL runs in-process and in VB6 on the same thread this isn't going to accomplish what you're hoping for.


Another thing to consider is that approaches that work with smaller amounts of data don't always scale well to large sets of data. "Sucking it all into RAM" is one of these, though as RAM has gotten plentiful the threshold of "large" has moved quite high.

Anything you can do to reduce the amount of data you work with early on should help. For example any filtering or extracting you can do before sorting will make your sorting take a lot fewer resources.

Anything you can do with your algorithms that reduces the number of times you have to "touch" (test, modify, copy, or move) the same data item should help.

There is also the issue of memory management and "chunk sizes" that can come into the picture. Asking for too many or too large arrays (and a String is essentially an array in most ways that matter) can mean a performance hit. Working with a huge number of tiny chunks of data can be a performance problem too though.


Those guidelines are so vague as to be almost useless I admit. However without knowing more specifics it is hard to make specific suggestions.

Some of the factors must be given different amounts of weight depending on the ecosystem your application lives in. "Suck it all into RAM" can fail miserably for server applications, where multiple copies must co-exist in RAM (and with other applications). Such an approach might work fine for systems that are otherwise idle though - like a desktop PC.


I think you know where some of your trouble spots are. But you might also consider running a profiler against your application to see where more subtle bottlenecks are.

What profiler do you recommend? The Profiler i found does not help

[dilettante]

I like VB Watch myself, and they have an evaluation version that can handle small projects (10 modules I believe). If you have VB/VS 6.0 Enterprise there is a profiler included too (APE is its name as I recall).

I'm not sure what other commercial products are still available though.

[coolcurrent4u]

Thank you dilettante,

I'll look at vbwatch trail. I am using visual basic professional, so it doesn't have APE. An i dont think APE is for testing individual application stress/problems see here.

any other suggestions would be welcomed

[coolcurrent4u]

Since a typical DLL runs in-process and in VB6 on the same thread this isn't going to accomplish what you're hoping for.

so do i use activex exe instead of dll?

Anything you can do to reduce the amount of data you work with early on should help. For example any filtering or extracting you can do before sorting will make your sorting take a lot fewer resources.

Usually i follow this step when working with strings

Functions

Extract > sort > remove duplicates > filter > group

what would be your advice here?

Anything you can do with your algorithms that reduces the number of times you have to "touch" (test, modify, copy, or move) the same data item should help.

do you have better algorithms for the above named functions that would be faster?

[dilettante]

Allowing Users to Interrupt Tasks is the classic approach to background processing. There is also the DoEvents hack, but of course it has its risks.

Other people offload background processing to another program, either a console program "connected" via anonymous pipes or an ActiveX EXE rigged for background processing on its own thread.


As far as data reduction goes, you do what you can. Not having duplicates is best. If you do have them though you may be forced to sort first, and take the performance hit. Prevention is usually cheaper than "cure" when possible.


"Better" algorithms are entirely situational. An example though might be to used hash-based keyed access for locating items in a large list instead of sequentially searching every time you need an item. Binary searches work well if your lists are ordered appropriately. Linked-lists can be better than moving long chunks of data when adjusting the sequence of a list, etc. Sorting is a performance killer when you don't truly require a fully sorted list. Not that any of that is news.


It is hard to make specific suggestions without considering specific cases. I suspect that's why others are holding back their comments so far.

[coolcurrent4u]

I'll past the code am using so far, maybe you'll see some bugs. They are all in a class module

Here is the extract function

'

extract Items from text
Public Sub Extract(ByRef Value As String)
    Dim inGroup      As Integer
    If sSeperator = vbNullString Then
        RaiseEvent Error("No Pattern defined", vbObjectError + 500)
        Exit Sub
    End If
    RaiseEvent Status("Extracting", 1)
    'Set the pattern by using the Pattern property.
    objRegExp.Pattern = sPattern
    ' Set Case Insensitivity.
    objRegExp.IgnoreCase = True
    'Set global applicability.
    objRegExp.Global = True
    'Test whether the String can be compared.
    If (objRegExp.Test(Value) = True) Then
        'Get the matches.
        Set colMatches = objRegExp.Execute(Value)
        
        '  Execute search.
        For Each objMatch In colMatches
            Do_Events
            ' Iterate Matches collection.
            Append objMatch.Value & sSeperator
        Next
        lCount = colMatches.Count
    End If
    Set colMatches = Nothing
End Sub

Here is the append function

Private Sub Append(s As String)
 
    Dim L      As Long
   
    L = StrLen + Len(s)
   
    While L > Len(Buffer)
        Buffer = Buffer & Space$(BufferSize)
    Wend
   
    CopyMemory ByVal StrPtr(Buffer) + (StrLen * 2), ByVal StrPtr(s), LenB(s)
    StrLen = StrLen + Len(s)
   
End Sub

please let me know what you find.

Thank you for you response so far