[Resolved] Does anyone know why the VB Replace function doesn't work?

[cafeenman]

I'm parsing web pages. Basically I replace all the carriage-return/line feeds with a space character. I replace all the word separators with a space character. Then I replace the double-spaces with a space character. And so on until I have a list of words and phrases.

All that works fine and happens pretty quickly. I'm timing everything and I'm pleased with the speed.

Eventually I have a list of comma-separated words and phrases with no spaces before or after the commas.

I then loop through a list of common words (the, and, is, at, etc.) and build a token as follows:

"," & commonword(n) & ","

Then I replace that word in the string. I loop through the commonwords removing each one.

My final string is turned into an array.

After that I count how many times each word that remains is used.

Basically what all this does is give me a list of not common words sorted by number of times they appear in the page.

Screenshot

What I noticed was that, for example, the word "The" is replaced several times, it's not replaced in all cases. I checked the string before replacement and after replacement and it's in the string like this,

etc,The,etc,etc,The,etc.

So there's no reason for it not to be replaced.

I've also converted everything to lower case for the comparison so that's not an issue.

To resolve it I've used a Do-Loop but it shouldn't be necessary unless I don't understand something about how replace is supposed to work.

For count, I'm using the default of -1 which according to MSDN, replaces all instances.

Here's the code:

Private Function RemoveWords(ByRef Words() As String) As Long
Dim C() As String
Dim sText As String
Dim n As Long
Dim s As String
Dim nStart As Long
Dim nEnd As Long
Dim nCount As Long
Dim nMostCount As Long

nStart = GetTickCount

C = Project_V2.CommonWords

If ArrayInitialized(C) And ArrayInitialized(Words) Then

  ' Rejoin string using a comma delimiter.
  sText = Join(Words, ",")

  ' Check entire token by including commas on both sides so partial words aren't removed.
  For n = LBound(C) To UBound(C)

    ' Create token.
    s = "," & Trim$(C(n)) & ","
    nCount = 0
    ' For whatever reason not all instances of token are removed in one replace operation.
    Do While InStr(1, sText, s, vbTextCompare)
      nCount = nCount + 1
      sText = Replace(sText, s, ",", 1, -1, vbTextCompare)
    Loop
    If nCount > nMostCount Then nMostCount = nCount
  Next n
  Debug.Print nMostCount
  
  ' Remove leading and trailing commas.
  If Left$(sText, 1) = "," Then sText = Right$(sText, Len(sText) - 1)
  If Right$(sText, 1) = "," Then sText = Left$(sText, Len(sText) - 1)

  Words = Split(sText, ",", -1, vbTextCompare)
End If

nEnd = GetTickCount

MsgBox (nEnd - nStart) / 1000 & " seconds to remove common words."

End Function

nMostCount is returning a value of 2 for several exceptionally verbose web pages.

Can anyone tell me why I have to call Replace multiple times?

[jmsrickland]

If I use this:

sText = Replace("etc,The,etc,etc,the,etc", "the", "blah", , , vbTextCompare)

then sText = "etc,blah,etc,etc,blah,etc"

If I use this:

sText = Replace("etc,The,etc,etc,the,etc", "the", ",", 1, -1, vbTextCompare)

then sText = "etc,,,etc,etc,,,etc"

So I don't understand your problem.

[cafeenman]

The problem is that without the do loop there are a lot of "the" left.

One of my pages has the word "the" 660 times.

After one replace using the exact code above but without the loop there were still something like 40 instances of "the" left in the string.

I don't get it either.

[cafeenman]

This is the page with all the "the's" in it.

http://www.airfieldmodels.com/faq/co...techniques.htm

[cafeenman]

Ok, this is being a pain but here's part of what was left after one replacement. Note that the word "the" is in here and it exactly matches the token, ",the,"

============

,90,degrees,the,strongest,webs,45,degrees,spars,directions,—,words,plywood,45,degrees,vertical,lamin ate,balsa,lighter,i’m,ocd,i’m,motivated,unless,it’s,critical,wing,absolute,highest,strength,i’m,stic king,vertical,webs,added,link,provided,shear,webs,thanks,adjusto,wing,jig,mention,site,magnetic,tape r,prefer,magnetic,tapered,prefer,wing,jig,that’s,goes,tool,having,variety,gives,“just,right”,tool,gi ven,circumstance,solder,clevises,loss,tried,burnt,blobby,mess,funniest,clevis,pushrod,landed,burnt,n ike's,thinking,throwing,solder,clevises,using,z,bends,pain,—,used,somebody,showed,solder,problems,—, cooked,joints,solder,didn’t,burns,areas,learned,kinds,colorful,words,they’d,you,solder,alcohol,conta iner,soak,clevis,pipe,cleaner,alcohol,the,clevis,

==============

Out of 660 initial instances of the word, the, 43 were left after one replace operation. By looping they all go away. Note the first word is "90 degrees" and the last word is "clevis"

[cafeenman]

Now here is the same exerpt after looping. All instances of "the" are gone. Same first and last word (90 degrees and clevis).

==========

,90,degrees,strongest,webs,45,degrees,spars,directions,—,words,plywood,45,degrees,vertical,laminate, balsa,lighter,i’m,ocd,i’m,motivated,unless,it’s,critical,wing,absolute,highest,strength,i’m,sticking ,vertical,webs,added,link,provided,shear,webs,thanks,adjusto,wing,jig,mention,site,magnetic,taper,pr efer,magnetic,tapered,prefer,wing,jig,that’s,goes,tool,having,variety,gives,“just,right”,tool,given, circumstance,solder,clevises,loss,tried,burnt,blobby,mess,funniest,clevis,pushrod,landed,burnt,nike' s,thinking,throwing,solder,clevises,using,z,bends,pain,—,used,somebody,showed,solder,problems,—,cook ed,joints,solder,didn’t,burns,areas,learned,kinds,colorful,words,they’d,solder,alcohol,container,soa k,clevis,pipe,cleaner,alcohol,clevis,

[cafeenman]

OK, I just confirmed using MS Word. Without replacing there are 666 instances of "the" in the page I linked to. After one replacement there are 43 instances left. After two replacements they're all gone.

[jmsrickland]

I created a text file with over 600 "THE"s in it. I loaded the file and replaced all "THE"s with "BLAH"s and saved the new file. There were no "THE"s in the file and there were over 600 "BLAH"s instead.

The only thing I can think of is that your file has some control codes in it and one of them may stop the Replace() function from continueing on beyond the control code.

AFTER POST EDIT:

I tried the HTML source from that link. All "THE"s and "The"s were replaced with "BLAH"

I also used your text from post #5. Same thing. There are no "The"s in the results.

[techgnome]

jms... that's because the string in post 5 is after the first replace (I count two occurrences -at least - of ",the" in the text) ... I'd like to know what it was BEFORE the first round of replacing.

-tg

[jmsrickland]

jms... that's because the string in post 5 is after the first replace (I count two occurrences -at least - of ",the" in the text) ... I'd like to know what it was BEFORE the first round of replacing.

-tg

I know that but it makes no difference. I tried all variations. And, like you, I would like to know what was BEFORE too. Like I said before the only thing I can think of is there could be some control code that terminates the Replace() function before it reaches the end of the string.

All I can say is that in all cases (even the HTML from the link he was using) I had no problems whatsoever.

[cafeenman]

I tried posting the original string but it won't let me - it's 52K characters long. I'll save it to a text file and upload it. I can't remember if this forum allows upload of text though.

I'll also upload the common words.

[cafeenman]

File names should be obvious.

Common Words are what I'm removing.
Original string has punctuation and such stripped.
One replace is the original string after one replacement.
Looped replace is after multiple replacements.

[cafeenman]

By the way, I had code that worked before but I was looping through it in code to remove the tokens and it took about 13 seconds to get the result.

This morning I decided to see what happens if I use replace instead. Was a little tricky because it would remove portions of words. For example, if I had the word "theory" I'd be left with "ory" which is why I went with the before and after commas.

looping through the words in the array inside a loop of common words was just too slow for longer pages.

My timings for the new function using the looped replace is 1.38 seconds to achieve the same thing.

[cafeenman]

jmsrickland - when you replaced from the page source it did some bad things too, right? I mean it replaced anything with the combination, "the" in it?

It would probably be a lot more efficient for me to replace the common words before doing anything else but I need some way of making sure it only replaces whole words. The VB Replace function doesn't have that option.

[jcis]

When you do this:
Replace(Me.Text1, ",the,", ",")
..and in the Text there is something like this: ",the,the," the first run will let ",the," and this must be remove in the next run of replace and so on, so the definitive way to solve this is doing multiple replaces until nothing is found:

    Do While InStr(Text1.Text, ",the,")
        Me.Text1 = Replace(Me.Text1, ",the,", ",")
    Loop

This is because one comma is taken for first replace, but then replace itself adds another comma making ",the," appear again. Multiple Replaces should fix it.

EDIT: in this case with this text, the while will do 2 loops.

[cafeenman]

OK, that would explain it and that is what's fixing it. Guess it's time to move on.

Thanks!

[jmsrickland]

Well I used the comma just like OP did and I didn't have any problems with the replace.

[cafeenman]

jmsrickland - I'll bet the reason we're getting different results is because you're only looking for "the".

I'm looping through (common words are sorted alphabetically) so by the time I get to "the" there may be two of them in succession which causes the problem.

[iam_clint]

I would recommend using a regular expression... it will be faster and better

\b means whole word match

Example replace (\bthe\b) with "" with global setting on and it will do the replacement
or all your key words at once
(\bthe\b|\bmeta\b|\byoub) you can keep going on... the regex will be 10x faster than vb replace.

but you should be able to see how you can make one giant replace string by looping through your words making a pattern string.

Example

InitialString = "the cat in the hat walked a million miles in the sunny sun sunshine. the the the people was in the the place"
Dim rx As New RegExp

With rx
.Pattern = "(\bthe\b|\byou\b|\bcat\b|\bhat\b|\bin\b|\bplace\b)"
.IgnoreCase = True
.Global = True
End With

ReplacedString = rx.Replace(InitialString, "WEEEE")
MsgBox ReplacedString

Regex is by far better than vb's replace function

Must add microsoft VBScript Regular Expressions 5.5 to your project references

[cafeenman]

You know... I keep forgetting about regular expressions.

These common words get added a little at a time and are saved in the project file (for my application, not the VP project file).

I could save the whole thing as a regular expression every time a new common word is added and load the whole thing as a big string to do it all at once instead of building the expression at run time or looping through one word at a time.

[cafeenman]

Thanks for everyone's help with this. Unfortunately, I'm not getting regular expressions to work like I wanted them to.

It has cut the time to about 1.8 seconds from whatever it was before... I think about 3.5 seconds, so roughly half the time. I'm building the RegExp string in advance (when the form loads). It doesn't noticably change the form load time and it lessens whatever time it takes from the parsing operation.

But the problem is that it sees apostrophes as word separators so I get left with a lot of things like, 's, 've, etc.

This is the function where the regular expression string is constructed:

Note that when I'm checking the timing I'm commenting out the Clipboard code. Probably doesn't matter.

This code takes twice the time on my laptop as my desktop.

Public Function BuildRegExCommonWords() As String
Dim s As String
Dim n As Long
Dim C() As String

C = Project_V2.CommonWords

If Not ArrayInitialized(C) Then Exit Function

s = "("

For n = LBound(C) To UBound(C) - 1
  s = s & "\b" & C(n) & "\b|"
Next n

BuildRegExCommonWords = s & "\b" & C(UBound(C)) & "\b)"

Clipboard.Clear
Clipboard.SetText BuildRegExCommonWords

End Function

This is function I posted yesterday as it's rewritten:

Private Function RemoveWords(ByRef Words As String, RegExCommonWords) As String
Dim C() As String
Dim sText As String
Dim n As Long
Dim s As String
Dim nStart As Long
Dim nEnd As Long
Dim nCount As Long
Dim RegEx As New RegExp

' Words is passed as a string instead of an array because this function is called after punctuation is stripped but before the page is parsed.

nStart = GetTickCount

C = Project_V2.CommonWords

If Not ArrayInitialized(C) Or (Words = vbNullString) Then Exit Function

With RegEx
  .Pattern = RegExCommonWords
  .IgnoreCase = True
  .Global = True
  s = .Replace(Words, vbNullString)
  RemoveWords = s
  Clipboard.Clear
  Clipboard.SetText s
  nEnd = GetTickCount
  MsgBox (nEnd - nStart) / 1000 & " seconds to remove common words."
 End With

End Function

[iam_clint]

Thanks for everyone's help with this. Unfortunately, I'm not getting regular expressions to work like I wanted them to.

It has cut the time to about 1.8 seconds from whatever it was before... I think about 3.5 seconds, so roughly half the time. I'm building the RegExp string in advance (when the form loads). It doesn't noticably change the form load time and it lessens whatever time it takes from the parsing operation.

But the problem is that it sees apostrophes as word separators so I get left with a lot of things like, 's, 've, etc.

This is the function where the regular expression string is constructed:

Note that when I'm checking the timing I'm commenting out the Clipboard code. Probably doesn't matter.

This code takes twice the time on my laptop as my desktop.

Public Function BuildRegExCommonWords() As String
Dim s As String
Dim n As Long
Dim C() As String

C = Project_V2.CommonWords

If Not ArrayInitialized(C) Then Exit Function

s = "("

For n = LBound(C) To UBound(C) - 1
  s = s & "\b" & C(n) & "\b|"
Next n

BuildRegExCommonWords = s & "\b" & C(UBound(C)) & "\b)"

Clipboard.Clear
Clipboard.SetText BuildRegExCommonWords

End Function

This is function I posted yesterday as it's rewritten:

Private Function RemoveWords(ByRef Words As String, RegExCommonWords) As String
Dim C() As String
Dim sText As String
Dim n As Long
Dim s As String
Dim nStart As Long
Dim nEnd As Long
Dim nCount As Long
Dim RegEx As New RegExp

' Words is passed as a string instead of an array because this function is called after punctuation is stripped but before the page is parsed.

nStart = GetTickCount

C = Project_V2.CommonWords

If Not ArrayInitialized(C) Or (Words = vbNullString) Then Exit Function

With RegEx
  .Pattern = RegExCommonWords
  .IgnoreCase = True
  .Global = True
  s = .Replace(Words, vbNullString)
  RemoveWords = s
  Clipboard.Clear
  Clipboard.SetText s
  nEnd = GetTickCount
  MsgBox (nEnd - nStart) / 1000 & " seconds to remove common words."
 End With

End Function

  s = s & "\b" & replace(C(n), "'", "\'") & "\b|"

escape the apostrophe by using \ so i'm becomes i\'m

[jmsrickland]

jmsrickland - I'll bet the reason we're getting different results is because you're only looking for "the".

I kind of thought that was what you were asking about.

I'm looping through (common words are sorted alphabetically) so by the time I get to "the" there may be two of them in succession which causes the problem.

No. That does not cause a problem. Multiple successions of a word does not cause the Replace() function from not replacing each occurance of the word.

I have tried every possibility and in all possibilities the Replace() function has replaced the word.

[cafeenman]

OK, but only because you told me to.

Do you know of any good references for regular expressions using vb6? I've noticed there's an extra argument when using .net (three instead of two).

A lot of what I found is hard to follow. I'd like just something that lists the syntax and special characters without paragraphs of explanation.

[iam_clint]

actually a ' is a non word so something like this will fail

Words
i i'm
Regex (\bi\b, \bi\'m\b)
will return
i
and the i from i'm
because ' is a non word boundary.... not sure how to fix it other than replacing all apostrophe's from the string and regex.


http://www.regular-expressions.info/wordboundaries.html

([\S\s]*i*[\s]|[\S\s]*i\'m*[\s\S])

Seemed to work but it matches spaces also so instead of just "i" it matches "i "

[cafeenman]

Not only that but I'm finding I have two different kinds of apostrophes.

’ and '

Seems kind of stupid to have that as a word boundary when it's part of a whole lot of words.

[cafeenman]

I kind of thought that was what you were asking about.

No. That does not cause a problem. Multiple successions of a word does not cause the Replace() function from not replacing each occurance of the word.

I have tried every possibility and in all possibilities the Replace() function has replaced the word.

OK, I'm not sure what it is but you see the code and I'm not getting it in one replace operation.

On the plus side I know one way that absolutely fixes it. Not as elegant as I'd like but it works.

I made a copy of the code before playing with regexp so if I can't get it working I can fall back on what I know works.

[cafeenman]

This is what I'm trying but like the man said, it won't work. The uploaded image shows the result.

Public Function BuildRegExCommonWords() As String
Dim s As String
Dim n As Long
Dim C() As String
Dim sTmp As String

C = Project_V2.CommonWords

If Not ArrayInitialized(C) Then Exit Function

s = "("

For n = LBound(C) To UBound(C) - 1
  's = s & "\b" & C(n) & "\b|"
  sTmp = Replace(C(n), "'", "\'")
  sTmp = Replace(sTmp, "’", "\’")
  s = s & "\b" & sTmp & "\b|"
Next n

s = s & "\b" & C(UBound(C)) & "\b)"

BuildRegExCommonWords = s

End Function

[iam_clint]

This is what I'm trying but like the man said, it won't work. The uploaded image shows the result.

Public Function BuildRegExCommonWords() As String
Dim s As String
Dim n As Long
Dim C() As String
Dim sTmp As String

C = Project_V2.CommonWords

If Not ArrayInitialized(C) Then Exit Function

s = "("

For n = LBound(C) To UBound(C) - 1
  's = s & "\b" & C(n) & "\b|"
  sTmp = Replace(C(n), "'", "\'")
  sTmp = Replace(sTmp, "’", "\’")
  s = s & "\b" & sTmp & "\b|"
Next n

s = s & "\b" & C(UBound(C)) & "\b)"

BuildRegExCommonWords = s

End Function

i'm not an expert in regex but i'll see if i can figure something else out

[cafeenman]

I don't know what to do with that. Can you put it in context please?

[jmsrickland]

I used your Original String.txt file and used one Replace() function the replace the word "the" and it did. There are no 'the's in the text file.

One thing about multiple successions of the same word however should really not be replaced since that would also cause the letters 'the' to be replaced within another word.

Example:

"This sentence contains the word the.

Replace 'the' with 'JMS'

Results = "This sentence contains JMS word JMS.

But consider this:

"This sentence contains the word further"

Results = "This sentence contains the word furJMSr"

Not exactly what your looking for.

But:

"This sentence contains the word thethethe"

Results = "This sentence contains the word JMSJMSJMS"

Again not what you want.

In other words, the replace should only work on a single occurance of the word 'the'. So you need some way to distinguish between 'the' and 'further' and 'thethethe'

[cafeenman]

That's why I used a comma before and after each word.

s = ",blah,the,blah,blah,the,etc,blah,the,"

Then I add a comma before and after each common word.

Replace (s,",the,")

and that still left me the's. ???

[iam_clint]

I don't know what to do with that. Can you put it in context please?

After You do your first regex replace it leaves 'll 'm 't things like that
Do another regex replace like this (\B\'[\w]+|\B\’[\w]+|\B\'\B|\B\’\B) after the first one has ran

This will get rid of anything that has an apostrophe without a word infront of it.

Then you can use (\s{1,}) to replace any spaces with a comma or whatever you feel like using

End result on your text file

lighter    OCD       motivated   unless   critical  wing  absolute highest strength    sticking   vertical webs  added  link  provided   shear webs  Thanks          Adjusto wing jig   mention   site   magnetic       taper  prefer     magnetic

See edits above

[jmsrickland]

That's why I used a comma before and after each word.

s = ",blah,the,blah,blah,the,etc,blah,the,"

Then I add a comma before and after each common word.

Replace (s,",the,")

and that still left me the's. ???

You shouldn't use the comma as a delimiter. Use a hex code instead; like &HFF eventhough the comma didn't hinder your approach.

You must have some other reason why you can't get the replace to work with one time when I can do it no matter what the input is.

So your're telling me that in the above sentence s = ",blah,the,blah,blah,the,etc,blah,the," you cannot get all occurances of the word 'the' to be replaced?

I used that exact string and here is what i get

",blahJMSblah,blahJMSetc,blahJMS"

using this replace function

Replace(sText, ",the,", "JMS", 1, -1, vbTextCompare)

BTW: Exactly what version of VB are you using and what OS are you using?

[iam_clint]

My test function worked fine

    Dim c() As String
    Dim s As String
    Dim f() As String
    
    c = Split(RichTextBox2.Text, ",")
    s = "("
    For n = LBound(c) To UBound(c) - 1
        's = s & "\b" & C(n) & "\b|"
        If c(n) <> "" Then
            sTmp = Replace(c(n), "'", "\'")
            sTmp = Replace(sTmp, "’", "\’")
            s = s & "\b" & sTmp & "\b|"
        End If
    Next n
    s = s & ")"
    Dim rx As New RegExp
    With rx
        .Pattern = s
        .IgnoreCase = True
        .Global = True
    End With
    ReplacedString = rx.Replace(RichTextBox1.Text, "")
    s = "(\B\'[\w]+|\B\’[\w]+|\B\'\B|\B\’\B|\B\-\B|\B\""\B)"
    With rx
        .Pattern = s
        .IgnoreCase = True
        .Global = True
    End With
    ReplacedString = rx.Replace(ReplacedString, "")
    f = Split(ReplacedString, ",")
    For n = LBound(f) To UBound(f) - 1
        If f(n) <> "" Then
            List1.AddItem (f(n))
        End If
    Next n

Richtextbox2 was your common words
richtextbox1 was your original string

list1 was just any normal list.

[jcis]

Are you sure that using RegExp is the way to go? Is Replace in a While loop too slow for you? if not, i don't think the extra code + new reference in the Project is really justified.

[iam_clint]

Are you sure that using RegExp is the way to go? Is Replace in a While loop too slow for you? if not, i don't think the extra code + new reference in the Project is really justified.

On performance I think its very justified. when you hit a page that has several thousands of words and run it you will see a huge difference in time.

Replacing in a while loop will tax the cpu when its not needed and make your app look locked up for longer periods. If other people are going to use this I think its justified.

Its sort of like saying lets walk because it works instead of getting in your car and driving. Makes sense when going a small distance but how about 100 miles?

[cafeenman]

Alright...

When using iam_clint's code I still have some stray characters. I don't know what they are because they show up as little thick lines which means they're line-feeds or something that doesn't display properly.

I added some code to give me the ascii value and had to do a replace for each of those.

By the time it worked the time to do the long page is 4.7 to 5.2 seconds.

My original code takes from 3.4 to 3.7 seconds.

I'm sure regular expressions can do it faster but I don't know enough about it to optimize the pattern string so I don't have to run it a half-dozen times.

But now that I've worked with it a bit I can see where it will come in handy fairly frequently.

So for now I'm going to stick with my original code and when I have more experience with regular expressions I'll revisit and see if I can get it right.

As far as the app goes, I've already written up a rough draft to distribute with it.

Basically I'm writing it for me but cleaned up enough for public consumption. It's very limited in what it does but it will make maintaining my site a lot easier with fewer opportunities to overlook things.

So while it's going to be freeware, I don't plan to spend a lot of time adding features I won't personally use.

Anyway, thanks again for everyone's efforts. I'm not done with this but I need to move along. I did a major revision this week by moving a lot of stuff to a class module and broke the app everywhere. I'm still hunting down all the bugs I created. It will be a better app because of the change but I have a lot to do.

[cafeenman]

Well, I finally finished my parsing app. I'm not uploading the code but the application is here in both winrar and zip format for anyone who wants to try it out.

I found that my code is much faster than use regular expressions. Don't know why but it's pretty fast. For normal length documents the parsing happens in less than a second on my computer when "Show Progress Window" is de-selected.

I have a 661 page document of quotes. There is a double-return between quotes and all other crap removed. It has 1.19 million characters and parses in about 20 seconds. I don't think that's too bad.

http://www.airfieldmodels.com/other/...ParseToy20.zip

http://www.airfieldmodels.com/other/...ParseToy20.rar

If you decide to download and play with it let me know what you think.

Thanks!

[anhn]

Back to your problem in the original code in post#1. (You resurrect it after 17 months!?!)
This works with one pass of Replace() that avoid the problem of consecutive common words (they may or may not be the same word):

  sText = "," & Join(Words, ",,") & ","
  For n = LBound(C) To UBound(C)
    s = "," & Trim$(C(n)) & ","
    sText = Replace(sText, s, "", 1, -1, vbTextCompare)
  Next n
  sText = Left$(Mid$(sText, 2), Len(sText) - 2)
  Words = Split(sText, ",,")