[RESOLVED] Adding a pause to my webpage parse function

[FONT=Verdana]I coded a small app for my web community last year. It goes through my users database (and up to 100 pages of usernames to a text file) So that I can look at the usernames & make sure nobody is using fowl/colorful usernames that might offend others. So that i can later remove them if needed.

Now that I have a lot of users, I don't want to parse all 100 pages of usernames so quickly & I would like to add a pause so that I don't bog down my server. Can you help me figure out how to write in a 'pause' to only collect 1 page of user-id's per 20 seconds?? Thanks! (I'm still new to this coding thing! Have no idea how to make pauses etc.)

Right now it just goes through & collects the pages as quickly as they load.......
Here is my 'ParseUsers' function code, it's called from 'button1' which then calls on 'BackgroundWorker1', which calls on the 'ParseUsers' function.

Code:

---

    Private Sub ParseUsers()
        Dim pageSrc As String = GetStuff("http://www.MY-FORUM.com/my-search/index.htm?get_post_detail_form_sub=2&got_it=1&p=1")
        Dim maxPages As Integer = GetMaxPages(pageSrc)
        ProgressBar1.Maximum = maxPages
        Profilelist.AddRange(ExtractUsers(pageSrc))
        For i As Integer = 2 To maxPages
            pageSrc = GetStuff("http://www.MY-FORUM.com/my-search/index.htm?get_post_detail_form_sub=2&got_it=1&p=" & i.ToString)
            Profilelist.AddRange(ExtractUsers(pageSrc))
            ProgressBar1.Value = i
        Next
        IO.File.WriteAllLines("USERS.txt", Profilelist.ToArray)
    End Sub

---

Any help/suggestions/code would be greatly appreciated :duck::duck::goose:

The easiest all purpose delay is ..

vb.net Code:

Dim t As UInt64 = Now.Ticks
        Do Until Now.Ticks - t >= d * 10 ^ 7 'where d is required delay in seconds
        Loop

Awesome thank you very much...
Now i guess...
Can anyone give me an idea of how/where to implement this pause? (i.e. inside the function, in the background worker itself, etc.)
I'm at a total loss where to get started

Well if the goal is to pause between reading pages from the server the only place to put it would appear to be immediately after ....

ProgressBar1.Value = i

... in the Parse sub. As a word of warning though, I should tell you that 20 seconds is an eternity in computing terms, and if a pause is necessary at all then 2-5 seconds is more than adequate for most purposes, especially if you're just spinning the wheels as it were.

Quote:

---

Originally Posted by dunfiddlin

Well if the goal is to pause between reading pages from the server the only place to put it would appear to be immediately after ....

ProgressBar1.Value = i

... in the Parse sub. As a word of warning though, I should tell you that 20 seconds is an eternity in computing terms, and if a pause is necessary at all then 2-5 seconds is more than adequate for most purposes, especially if you're just spinning the wheels as it were.

---

The goal is to add a pause between 'each request' for a page from the web server. So if that's what you meant then ok, I will make sure to make it around 5 seconds then!

Quote:

---

Originally Posted by dunfiddlin

The easiest all purpose delay is ..

vb.net Code:

Dim t As UInt64 = Now.Ticks
        Do Until Now.Ticks - t >= d * 10 ^ 7 'where d is required delay in seconds
        Loop

---

This is a more compact equivalent:-

vbnet Code:

Threading.Thread.Sleep(d * 1000)

Quote:

---

Originally Posted by Niya

This is a more compact equivalent:-

vbnet Code:

Threading.Thread.Sleep(d * 1000)

---

You can't use Thread.Sleep in a background worker (or at least you couldn't last time I looked into it ... I doubt that it's changed, however).

Quote:

---

Originally Posted by dunfiddlin

You can't use Thread.Sleep in a background worker (or at least you couldn't last time I looked into it ... I doubt that it's changed, however).

---

Ok so what does the 10 & 7 equal out to be in..... Do Until Now.Ticks - t >= d * 10 ^ 7

I'm not familiar with that structure...I'm used to writing out seconds in timers like 75000 = 75 seconds

Quote:

---

Originally Posted by jalexander

Ok so what does the 10 & 7 equal out to be in..... Do Until Now.Ticks - t >= d * 10 ^ 7

I'm not familiar with that structure...I'm used to writing out seconds in timers like 75000 = 75 seconds

---

10 to the power of 7 (or 10,000,000 to you) ticks per second.

dupe

Worked perfectly! Thank you so much @dunfiddlin ! You just solved a problem I've had now for months...this is why I love this forum & all the awesome people on here <3 <3 love love love you all!
1 - question though .... so 'd' = number of seconds to pause right...do I write 5 seconds as '5' or as '5000' (as within a timer). Btw thanks also for '20' seconds is a life-time in computing tip I'll keep that in mind because when I thought about it...it made a lot of sense! <3

& thanks also to @Niya for the compact version!

You guys are seriously the best. When I start college this fall I'll have all of you to thank...for being so far ahead of everyone else in the 101 course! :blush: <3

[Deleted Double Post]

Quote:

---

Originally Posted by dunfiddlin

You can't use Thread.Sleep in a background worker (or at least you couldn't last time I looked into it ... I doubt that it's changed, however).

---

This is not true. Thread.Sleep can pause any thread.

Quote:

---

Originally Posted by Niya

This is not true. Thread.Sleep can pause any thread.

---

Well, no it can't. It can't pause the background worker because that isn't a thread within the strict definition of the relevant namespace. Many have tried. All have failed.

Quote:

---

'd' = number of seconds

---

Not milliseconds!

Quote:

---

Originally Posted by dunfiddlin

Well, no it can't. It can't pause the background worker because that isn't a thread within the strict definition of the relevant namespace. Many have tried. All have failed.

---

And I'm telling you that you're wrong. See for yourself:-

vbnet Code:

'
    Private Sub BackgroundWorker1_DoWork(ByVal sender As System.Object, ByVal e As System.ComponentModel.DoWorkEventArgs) Handles BackgroundWorker1.DoWork
        For i = 1 To 1000
            Threading.Thread.Sleep(100)
            SetLabelText(i)
        Next
    End Sub
 
    Private Sub SetLabelText(ByVal text As String)
 
        If Label1.InvokeRequired Then
            Label1.Invoke(New Action(Of String)(AddressOf SetLabelText), text)
        Else
            Label1.Text = text
        End If
 
    End Sub
 
    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        BackgroundWorker1.RunWorkerAsync()
    End Sub

Start a new project and place the above code in Form1 along with a Label, Button and a BackgroundWorker. Run the program and press the button.

I'm curious...you said I could use the following

Code:

---

Threading.Thread.Sleep(d * 1000)

---

i.e. to pause for 5 seconds i would use...

Code:

---

Threading.Thread.Sleep(5 * 1000)

---

but in the code above you used...

Code:

---

Threading.Thread.Sleep(100)

---

does that mean i could simply put in my code...

Code:

---

Threading.Thread.Sleep(5000)

---

To pause for 5 seconds?

Also, can this code be used anywhere in a sub, or does it need to be in specific places....like right before the end of a sub etc. Or does it all depend?

Thanks :)

It can be used anywhere. And yes you can use 5000 directly.