Retrieving data from a webpage

[Falz]

Hi

Im a beginner to Vb.net .I would like to know how to get data from a webpage. Am I supposed to use ASP.Net?


Thanks

Matt

[jmcilhinney]

You'll need to give more information. Do you want to display a Web page in a browser window or in your own application? Do you want to get some data from a page that is already displayed in a browser? Do you want to get the source for a page from a Web server without displying the rendered page?

[mimminito]

i am after doing the same sort of thing. i have posted a topic on it. i want to get data from a webpage and store it into a database or text file maybe? i want to get the titles of applications from a website, www.pspupdates.com in the files section. is it possible to only extract the titles of the applications and save them into a text file?

[Atheist]

mimminito, heres a quick example i wrote that retrieves the application titles from that website (In VB 2005):

(add a listbox called listbox1 to your form)

Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
        If IO.File.Exists(Application.StartupPath & "\heh.txt") Then
            IO.File.Delete(Application.StartupPath & "\heh.txt")
        End If
        My.Computer.Network.DownloadFile("http://dl.qj.net/Gaming-Consoles-PSP/catid/106", Application.StartupPath & "\heh.txt")
        Dim sr As New IO.StreamReader(Application.StartupPath & "\heh.txt")
        Dim strLines() As String = Strings.Split(sr.ReadToEnd, Environment.NewLine)
        sr.Close()
        IO.File.Delete(Application.StartupPath & "\heh.txt")
        Dim temp1 As Integer
        Dim temp2 As Integer
        For Each str As String In strLines
            str = Trim(str)
            If str.StartsWith("<td height=" & Chr(34) & "15" & Chr(34) & "><a href") Then
                temp1 = str.IndexOf("class=" & Chr(34) & "filename" & Chr(34) & ">") + ("class=" & Chr(34) & "filename" & Chr(34) & ">").Length
                temp2 = str.IndexOf("</a>") - (str.IndexOf("class=" & Chr(34) & "filename" & Chr(34) & ">") + ("class=" & Chr(34) & "filename" & Chr(34) & ">").Length)
                ListBox1.Items.Add(str.Substring(temp1, temp2))
            End If
        Next
    End Sub

[mimminito]

Cheers Atheist. I have got Visual Basic 2005 Express Edition and Visual Studio 2005. When i try the code in Visual Basic 2005 Express Edition i get an error, 'Statement is not valid in namespace' Any suggestions? Thanks for the code though i think this will work. I am fairly new to this type of coding, so i may sound a little stupid sorry

[mimminito]

I have got it to work thankyou very much. I will reply soon with my results

[mimminito]

Atheist, i was wondering if you could explain your coding a little more for me. I would like to understand it a little more, then i should be able to work on it myself. Thanks if you can

[mimminito]

i got a lil problem now. i can make out most of your coding now, (apart from why a text file is created and then deleted) and i am able to change what i need. but when i go to say the development section on the webpage, the source code is diff to the other page. there is no 'class filename' only a href, so how can i go about getting the applications name?

[Atheist]

Hello again
First the page is downloaded as a textfile, then its read into a string. Since we have all the text in the string we dont need the textfile anymore, thats why its deleted (And if its not deleted the code will generate an error the next time its run)

The development page...

First of we begin with the basics:

If IO.File.Exists(Application.StartupPath & "\heh.txt") Then
            IO.File.Delete(Application.StartupPath & "\heh.txt")
        End If
        My.Computer.Network.DownloadFile("http://dl.qj.net/Gaming-Consoles-PSP/catid/106", Application.StartupPath & "\heh.txt")
        Dim sr As New IO.StreamReader(Application.StartupPath & "\heh.txt")
        Dim strLines() As String = Strings.Split(sr.ReadToEnd, Environment.NewLine)
        sr.Close()
        IO.File.Delete(Application.StartupPath & "\heh.txt")

Then we continue with this code:

Dim blnFound As Boolean = False
        Dim intTemp_1 As Integer
        Dim i As Integer
        Dim q As Integer
        For Each str As String In strLines
            If blnFound = False Then 'If it has not found the application table
                If Trim(str) = "<table width=" & Chr(34) & "476" & Chr(34) & " class=" & Chr(34) & "contentcell" & Chr(34) & ">" Then
                    blnFound = True 'Application table found!
                End If
            Else 'If it has found the application table
                If Trim(str).StartsWith("<td><a href") Then 'If the line starts with <td><a href
                    intTemp_1 = 0
                    For i = 0 To str.Length 'Loop through the line and stop on the second >
                        If str.Substring(i, 1) = ">" Then
                            intTemp_1 = intTemp_1 + 1
                        End If
                        If intTemp_1 = 2 Then
                            Exit For
                        End If
                    Next
                    For q = i To str.Length ' Loop through the line (starting on the position of the second >, to find the first <
                        If str.Substring(q, 1) = "<" Then
                            Exit For
                        End If
                    Next
                    If str.Substring((i + 1), (q - i) - 1) = "Users questions" Then
                        Exit For ' Exit if we have come to User questions (we dont want that do we)
                    End If
                    If Not str.Substring((i + 1), (q - i) - 1) = "" Then
                        ListBox1.Items.Add(str.Substring((i + 1), (q - i) - 1))
                    End If
                End If
                End If
        Next

There! Just ask if you wonder something