[RESOLVED] Adding a pause to my webpage parse function

[jalexander]

[FONT=Verdana]I coded a small app for my web community last year. It goes through my users database (and up to 100 pages of usernames to a text file) So that I can look at the usernames & make sure nobody is using fowl/colorful usernames that might offend others. So that i can later remove them if needed.

Now that I have a lot of users, I don't want to parse all 100 pages of usernames so quickly & I would like to add a pause so that I don't bog down my server. Can you help me figure out how to write in a 'pause' to only collect 1 page of user-id's per 20 seconds?? Thanks! (I'm still new to this coding thing! Have no idea how to make pauses etc.)

Right now it just goes through & collects the pages as quickly as they load.......
Here is my 'ParseUsers' function code, it's called from 'button1' which then calls on 'BackgroundWorker1', which calls on the 'ParseUsers' function.

    Private Sub ParseUsers()
        Dim pageSrc As String = GetStuff("http://www.MY-FORUM.com/my-search/index.htm?get_post_detail_form_sub=2&got_it=1&p=1")
        Dim maxPages As Integer = GetMaxPages(pageSrc)
        ProgressBar1.Maximum = maxPages
        Profilelist.AddRange(ExtractUsers(pageSrc))
        For i As Integer = 2 To maxPages
            pageSrc = GetStuff("http://www.MY-FORUM.com/my-search/index.htm?get_post_detail_form_sub=2&got_it=1&p=" & i.ToString)
            Profilelist.AddRange(ExtractUsers(pageSrc))
            ProgressBar1.Value = i
        Next
        IO.File.WriteAllLines("USERS.txt", Profilelist.ToArray)
    End Sub

Any help/suggestions/code would be greatly appreciated :goose:

[dunfiddlin]

The easiest all purpose delay is ..

Dim t As UInt64 = Now.Ticks
        Do Until Now.Ticks - t >= d * 10 ^ 7 'where d is required delay in seconds
        Loop

[jalexander]

Awesome thank you very much...
Now i guess...
Can anyone give me an idea of how/where to implement this pause? (i.e. inside the function, in the background worker itself, etc.)
I'm at a total loss where to get started

[dunfiddlin]

Well if the goal is to pause between reading pages from the server the only place to put it would appear to be immediately after ....

ProgressBar1.Value = i

... in the Parse sub. As a word of warning though, I should tell you that 20 seconds is an eternity in computing terms, and if a pause is necessary at all then 2-5 seconds is more than adequate for most purposes, especially if you're just spinning the wheels as it were.

[jalexander]

Well if the goal is to pause between reading pages from the server the only place to put it would appear to be immediately after ....

ProgressBar1.Value = i

... in the Parse sub. As a word of warning though, I should tell you that 20 seconds is an eternity in computing terms, and if a pause is necessary at all then 2-5 seconds is more than adequate for most purposes, especially if you're just spinning the wheels as it were.

The goal is to add a pause between 'each request' for a page from the web server. So if that's what you meant then ok, I will make sure to make it around 5 seconds then!

[Niya]

The easiest all purpose delay is ..

Dim t As UInt64 = Now.Ticks
        Do Until Now.Ticks - t >= d * 10 ^ 7 'where d is required delay in seconds
        Loop

This is a more compact equivalent:-

Threading.Thread.Sleep(d * 1000)

[dunfiddlin]

This is a more compact equivalent:-

Threading.Thread.Sleep(d * 1000)

You can't use Thread.Sleep in a background worker (or at least you couldn't last time I looked into it ... I doubt that it's changed, however).

[jalexander]

You can't use Thread.Sleep in a background worker (or at least you couldn't last time I looked into it ... I doubt that it's changed, however).

Ok so what does the 10 & 7 equal out to be in..... Do Until Now.Ticks - t >= d * 10 ^ 7

I'm not familiar with that structure...I'm used to writing out seconds in timers like 75000 = 75 seconds

[dunfiddlin]

Ok so what does the 10 & 7 equal out to be in..... Do Until Now.Ticks - t >= d * 10 ^ 7

I'm not familiar with that structure...I'm used to writing out seconds in timers like 75000 = 75 seconds

10 to the power of 7 (or 10,000,000 to you) ticks per second.

[jalexander]

dupe

[jalexander]

Worked perfectly! Thank you so much @dunfiddlin ! You just solved a problem I've had now for months...this is why I love this forum & all the awesome people on here <3 <3 love love love you all!
1 - question though .... so 'd' = number of seconds to pause right...do I write 5 seconds as '5' or as '5000' (as within a timer). Btw thanks also for '20' seconds is a life-time in computing tip I'll keep that in mind because when I thought about it...it made a lot of sense! <3

& thanks also to @Niya for the compact version!

You guys are seriously the best. When I start college this fall I'll have all of you to thank...for being so far ahead of everyone else in the 101 course! <3

[Niya]

[Deleted Double Post]

[Niya]

You can't use Thread.Sleep in a background worker (or at least you couldn't last time I looked into it ... I doubt that it's changed, however).

This is not true. Thread.Sleep can pause any thread.

[dunfiddlin]

This is not true. Thread.Sleep can pause any thread.

Well, no it can't. It can't pause the background worker because that isn't a thread within the strict definition of the relevant namespace. Many have tried. All have failed.

[dunfiddlin]

'd' = number of seconds

Not milliseconds!

[Niya]

Well, no it can't. It can't pause the background worker because that isn't a thread within the strict definition of the relevant namespace. Many have tried. All have failed.

And I'm telling you that you're wrong. See for yourself:-

'
    Private Sub BackgroundWorker1_DoWork(ByVal sender As System.Object, ByVal e As System.ComponentModel.DoWorkEventArgs) Handles BackgroundWorker1.DoWork
        For i = 1 To 1000
            Threading.Thread.Sleep(100)
            SetLabelText(i)
        Next
    End Sub
 
    Private Sub SetLabelText(ByVal text As String)
 
        If Label1.InvokeRequired Then
            Label1.Invoke(New Action(Of String)(AddressOf SetLabelText), text)
        Else
            Label1.Text = text
        End If
 
    End Sub
 
    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        BackgroundWorker1.RunWorkerAsync()
    End Sub

Start a new project and place the above code in Form1 along with a Label, Button and a BackgroundWorker. Run the program and press the button.

[jalexander]

I'm curious...you said I could use the following

Threading.Thread.Sleep(d * 1000)

i.e. to pause for 5 seconds i would use...

Threading.Thread.Sleep(5 * 1000)

but in the code above you used...

Threading.Thread.Sleep(100)

does that mean i could simply put in my code...

Threading.Thread.Sleep(5000)

To pause for 5 seconds?

Also, can this code be used anywhere in a sub, or does it need to be in specific places....like right before the end of a sub etc. Or does it all depend?

Thanks

[Niya]

It can be used anywhere. And yes you can use 5000 directly.