Retrieving information from website?

[TeachMeVB]

Hey
Im a noob coder, not gonna hide it, but we're all noobs at some point right?

Anyway... To you the program that i am making may seem wierd, but i didn't come here for your opinions on that...
I came here for your help

My problem is as follows:
I have watched a few tutorials about setting info in textboxes online, and retrieving info...

WebBrowser1.Document.GetElementById("[NAME OF ELEMENT]").GetAttr...

But how do i get the info i want, when the "element" is nameless, and idless?

The site im trying to get info from is: http://vidstatsx.com/PewDiePie/videos-most-recent
Im trying to retrieve the number of videos uploaded.

If anyone could provide me with the code i need to get that number, and an explanation for every line of code (so that i actually learn it, not just copy), i would be very happy Thanks in advance

[dunfiddlin]

Would you prefer not to have a visible WebBrowser (it seems kinda pointless retrieving the information if it's sitting right there in front of you but chacon a son gout)

[TeachMeVB]

Would you prefer not to have a visible WebBrowser (it seems kinda pointless retrieving the information if it's sitting right there in front of you but chacon a son gout)

Yeah... But i can take care of that myself... lol... :P
Im not THAT noob! I don't need you to design the whole program for me! I just need the code that lets me retrieve the info of how many videos has been uploaded. http://vidstatsx.com/PewDiePie/videos-most-recent

[Pyth007]

Instead of using GetElementById or GetElementByName, you may have to use a more general function like GetElementsByTagName and dig down from there. Looking at the page's source, you should be able to get a table with a particular class name (it's class is "vp_detailed" so you can check for it with "if webElement.className="vp_detailed" Then..."). This is the table that has the stats that you are looking for (I believe). The first row in that table has the number of Uploads with the 1st cell being the title and the 2nd cell being the actual number.

If you are actually using MSHTML, then for the table elements, you can actually cast the entire table that you are looking for as an HTMLTable and iterate through the rows and cells using a for...each loop, or if there is a specific row / cell that you are looking for, get the .item(#) from the corresponding collection of the parent. For example:

Dim webTableCollection as mshtml.IHTMLElementCollection = webDocument.getElementsByTagName("table") ' Note, webDocument is the web page opened as a mshtml.IHTMLDocument3
For Each webElement as mshtml.IHTMLElement In webTableCollection
      If webElement.className="vp_detailed" Then ' Note that the page has more with this class name, but what you are looking for is the first table with this class
            Dim webTable as mshtml.HTMLTable = DirectCast(webElement, mshtml.HTMLTable)
            Dim webFirstRow as mshtml.HTMLTableRow = DirectCast(webTable.rows.item(0), mshtml.HTMLTableRow) ' Gets the first row of this table
            Dim webSecondCell as mshtml.HTMLTableCell = DirectCast(webFirstRow.cells(1), mshtml.HTMLTableCell) ' Gets the 2nd cell of the 1st row, which holds the # of Uploads. Using .item short-cut here
            Dim strUploads as string = webSecondCell.innerText
            Exit For
      End If
Next

[TeachMeVB]

Instead of using GetElementById or GetElementByName, you may have to use a more general function like GetElementsByTagName and dig down from there. Looking at the page's source, you should be able to get a table with a particular class name (it's class is "vp_detailed" so you can check for it with "if webElement.className="vp_detailed" Then..."). This is the table that has the stats that you are looking for (I believe). The first row in that table has the number of Uploads with the 1st cell being the title and the 2nd cell being the actual number.

If you are actually using MSHTML, then for the table elements, you can actually cast the entire table that you are looking for as an HTMLTable and iterate through the rows and cells using a for...each loop, or [...]

I get all the table, and vp_detailed stuff :P Thank you for clearing that up
However, what i don't get, is the thing about MSHTML... I get an error on the code when i input it.
(The error is marked on the following
mshtml.IHTMLElementCollection
mshtml.IHTMLElement
The error is the same for both of them (Type [...] is not defined)

I guess i should import a function or something? I don't know :P That's why I need some of you guys to teach me Thanks for all the help so far

[dunfiddlin]

Project > Add Reference > .Net tab > Microsoft.mshtml

[TeachMeVB]

Project > Add Reference > .Net tab > Microsoft.mshtml

I can't add that reference... it's not on the list :S What do i do?

[dunfiddlin]

I take it you don't have Office installed? Looks like we'll have to find you a solution in plain old HTML then.

[TeachMeVB]

I take it you don't have Office installed? Looks like we'll have to find you a solution in plain old HTML then.

What do you mean office?
Microsoft office? (If that's what u mean, i got it :P)

Anyway, found this reply on a thread/topic about adding MSHTML:
The MSHTML import no longer exists in anything after VB2005. You can do everything using the normal webbrowser control that you can using MSHTML. MSHMTL is just outdated.

[Pyth007]

Humph... I guess it makes sense since the code I was was relying upon to pose my response was made back when using VB2005... I guess I'll also have to learn about the new webbrowser control.... There may be sites that explain how to convert MSHTML code to the newer version... But regardless of the syntax, the logic behind my code should help.

[Legjendat]

OK, my code will do the job, but that's for as long as they don't change the page source and the order of the elements in that page, in which case my code won't work, as it is not dynamic, but rather static.

Public Class Form1
 
    Public Function GetBetween(ByVal StringSearch As String, ByVal FirstString As String, _
ByVal SecondString As String, Optional ByVal SearchLength As Integer = 1) As String
 
        Dim TempLength As Long
        SearchLength = InStr(SearchLength, StringSearch, FirstString)
 
        If SearchLength > 0 Then
            SearchLength += Len(FirstString)
            TempLength = InStr(SearchLength, StringSearch, SecondString)
            If TempLength > SearchLength Then
                GetBetween = Trim(Mid(StringSearch, SearchLength, TempLength - SearchLength))
            End If
        End If
 
    End Function
 
    Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
        Try
            My.Computer.Network.DownloadFile("http://vidstatsx.com/PewDiePie/videos-most-recent", _
Environment.GetFolderPath(Environment.SpecialFolder.Desktop) & "\lalalalala.html")
        Catch ex As Exception
            'Throw whatever error msg u want here.
        End Try
 
        Dim PageSource As String
        Using myStreamReader As System.IO.StreamReader = System.IO.File.OpenText(Environment.GetFolderPath _
(Environment.SpecialFolder.Desktop) & "\lalalalala.html")
            PageSource = myStreamReader.ReadToEnd()
        End Using
 
        Dim Quotes As String = """"
        Dim findString As String = GetBetween(PageSource, "Uploads:</h3></td><td class=" & Quotes & _ 
"rank green" & Quotes & " width=", _
"</td><td class=" & Quotes & "pad" & Quotes & ">")
 
        Dim pxSizeLength As Integer = GetBetween(findString, Quotes, Quotes).Length
        findString = findString.Remove(0, 3 + pxSizeLength)
'findString is the number of uploads, you can remove the comma in it if you want by replacing it with nothing.
        IO.File.Delete(Environment.GetFolderPath(Environment.SpecialFolder.Desktop) & "\lalalalala.html")
 
    End Sub
End Class

[dunfiddlin]

What do you mean office?
Microsoft office? (If that's what u mean, i got it :P)

Anyway, found this reply on a thread/topic about adding MSHTML:
The MSHTML import no longer exists in anything after VB2005. You can do everything using the normal webbrowser control that you can using MSHTML. MSHMTL is just outdated.

Interesting, as I've never had anything than VS 2010 on this machine and one of the sources for the mshtml.dll is in the Office 2007 binaries. I'm really not sure that mshtml is outdated and I know for certain that you can't do everything in the webbrowser without it, notably use that very browser in edit mode!

[TeachMeVB]

Interesting, as I've never had anything than VS 2010 on this machine and one of the sources for the mshtml.dll is in the Office 2007 binaries. I'm really not sure that mshtml is outdated and I know for certain that you can't do everything in the webbrowser without it, notably use that very browser in edit mode!

I searched my computer for "mshtml.dll".
Found a file in the windows/system32 folder. Will this work?

[TeachMeVB]

OK, my code will do the job, but that's for as long as they don't change the page source and the order of the elements in that page, in which case my code won't work, as it is not dynamic, but rather static.

Public Class Form1
 
    Public Function GetBetween(ByVal StringSearch As String, ByVal FirstString As String, _
ByVal SecondString As String, Optional ByVal SearchLength As Integer = 1) As String
 
        Dim TempLength As Long
        SearchLength = InStr(SearchLength, String [...]
 
        Dim pxSizeLength As Integer = GetBetween(findString, Quotes, Quotes).Length
        findString = findString.Remove(0, 3 + pxSizeLength)
'findString is the number of uploads, you can remove the comma in it if you want by replacing it with nothing.
        IO.File.Delete(Environment.GetFolderPath(Environment.SpecialFolder.Desktop) & "\lalalalala.html")
 
    End Sub
End Class

Thanks, but i didn't get the code... How do i make the number of uploads appear as text in label1?
(Sorry, but this is to advanced for me :P Liked the mshtml better :P)

[TeachMeVB]

Tried using the "mshtml.dll" found in C:\Windows\System32

... didn't work...

But i'm gonna google a bit and see if i can find a downloadable mshtml.dll for VB...

[Legjendat]

Thanks, but i didn't get the code... How do i make the number of uploads appear as text in label1?
(Sorry, but this is to advanced for me :P Liked the mshtml better :P)

OK, so the code in GetBetween is not to be touched, it is just a function that gets a certain string between two other strings, like you have

Dim numbers as string = "One Two Three"
'You want to get the string that says "two" so you write somewhere (in a button's click event for example)
MsgBox(GetBetween(numbers, "One", "Three"))

'What I did basically is I downloaded the source code of that webpage and I used that function GetBetween to get the text between what was before the Uploads number in the age source and what was after it. Given that the source code might change if say they change the "Uploads:" to "Uploads Number:", my code wouldn't work as it would look to find the number of uploads between the word "Uploads:" whereas that word has become "UploadsNumber:" thus not finding it. Anyway, I doubt that would change much, so for the moment it is safe to use.

The real job is done on the button1_click event that happens when you click the button. I use GetBetween once to extract the number of uploads with a bunch of other text before it, so I use GetBetween another time to extract the number of uploads without that other text before it, but in order to get that, I get the length of the text that's before it and I remove the text before it since I know it's length. Anyway, if you want the job done and don't want to look much into the details, just do whatever you want with the string findString.

findString is your number of uploads, so use it as you want to, assign it to a label, a textbox, whatever. Like say MyLabel.Text = findString
Btw I'd just like to say that my method looks complicated not because it's advanced, but because I'm a noob like you and couldn't find a better way of doing it.

[Pyth007]

Actually my mshtml.dll was as a COM reference... See if you can find Project --> Add Reference --> COM Tab --> Microsoft HTML Object Library

Yeah, you can also use string manipulation (usually regular expressions / regex are used since they work very well for pattern recognition), but since the OP had mentioned some DOM functions, I thought that it'd be easier to understand using other DOM functions. Using the DOM method may be a little more dynamic in that the web master for a particular site may change some positions around on the page, but if some of the objects are named / id'ed then it's possible to start from there and work in. Usually a page may change its structure, but if they have any dynamic data being added to it, it becomes too much work to completely change the format. Either the web designer will name actual elements for easy reference for plugging in the dynamic data, or they'll have some other static means to get to that point (eg name the table, div, or some other container that holds the data). Thus if you can figure out how the web designer is getting at the data, you can use the DOM to delve down and pull out specific pieces of data too.

[TeachMeVB]

Actually my mshtml.dll was as a COM reference... See if you can find Project --> Add Reference --> COM Tab --> Microsoft HTML Object Library

Yeah, you can also use string manipulation [...] DOM to delve down and pull out specific pieces of data too.

Found the reference Gonna try the code soon. Will edit this message after i've tried to see it it worked Thanks

...:::EDIT:::...
Tried the code, got an error wich made me unable to run/debug the program. Even when i tried to build the program it failed.

The error was shown on this line of coding:
Dim webTableCollection As mshtml.IHTMLElementCollection = webDocument.getElementsByTagName("table")
The acutal error was highlighted as:
webDocument
The error that showed was:
"webDocument is not declared. It may be inaccessible due to it's protection level."

Other then that the code has no errors... Any ideas? :P (As I said im a noob, i have no idea how to fix this :P)

If it helps, here is the entire code for my program: (And yes, i have added a WebBrowser)

Public Class Form1

    Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
        WebBrowser1.Navigate("http://vidstatsx.com/PewDiePie/videos-most-recent")

        Dim webTableCollection As mshtml.IHTMLElementCollection = webDocument.getElementsByTagName("table") ' Note, webDocument is the web page opened as a mshtml.IHTMLDocument3
        For Each webElement As mshtml.IHTMLElement In webTableCollection
            If webElement.className = "vp_detailed" Then ' Note that the page has more with this class name, but what you are looking for is the first table with this class
                Dim webTable As mshtml.HTMLTable = DirectCast(webElement, mshtml.HTMLTable)
                Dim webFirstRow As mshtml.HTMLTableRow = DirectCast(webTable.rows.item(0), mshtml.HTMLTableRow) ' Gets the first row of this table
                Dim webSecondCell As mshtml.HTMLTableCell = DirectCast(webFirstRow.cells(1), mshtml.HTMLTableCell) ' Gets the 2nd cell of the 1st row, which holds the # of Uploads. Using .item short-cut here
                Dim strUploads As String = webSecondCell.innerText
                My.Settings.Videos = strUploads
                Exit For
            End If
        Next

        Label1.Text = "Uploaded videos: " & My.Settings.Videos
    End Sub
End Class

[Pyth007]

If you look at the comment I had at the end of that line, you'll see that webDocument needs to be created and opened as an IHTMLDocument3 interface... So here's two methods that has everything you'll need. First, there's a button click event that does the bulk of the work (you could put this into your Form.load code above, noting that I did change the last part of what to do with the result). There is also GetHTML, a function that downloads a web page and returns a string of the HTML code. The button's sub uses this GetHTML function and then creates a new HTMLDocument that incorporates the IHTMLDocument3 interface. The GetHTML function uses System.Net so you'll also have to add the line "Imports System.Net" to the top of your code. One last thing... When I ran this, I did get a pop-up about a possible security risk; I just hit continue and it worked fine. Not sure how to get around this, however, so that it acts smoother....

Here's the code:

Imports System.Net
Imports mshtml
....
    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        Me.Cursor = Cursors.WaitCursor
        Dim webElement As mshtml.IHTMLElement
        Dim strURL As String = "http://vidstatsx.com/PewDiePie/videos-most-recent"
        Dim strHTML As String = MiscMethods.GetHTML(strURL)
        If String.IsNullOrEmpty(strHTML) Then Exit Sub

        ' Load HTML into parsable HTML document
        Dim webDocument As mshtml.IHTMLDocument3 = New HTMLDocumentClass()
        webDocument.write(strHTML)
        webDocument.close()

        Dim webTableCollection As mshtml.IHTMLElementCollection = webDocument.getElementsByTagName("table") ' Note, webDocument is the web page opened as a mshtml.IHTMLDocument3
        For Each webElement In webTableCollection
            If webElement.className = "vp_detailed" Then ' Note that the page has more with this class name, but what you are looking for is the first table with this class
                Dim webTable As mshtml.HTMLTable = DirectCast(webElement, mshtml.HTMLTable)
                Dim webFirstRow As mshtml.HTMLTableRow = DirectCast(webTable.rows.item(0), mshtml.HTMLTableRow) ' Gets the first row of this table
                Dim webSecondCell As mshtml.HTMLTableCell = DirectCast(webFirstRow.cells.item(1), mshtml.HTMLTableCell) ' Gets the 2nd cell of the 1st row, which holds the # of Uploads. Using .item short-cut here
                Dim strUploads As String = webSecondCell.innerText
                MsgBox("Uploads: " & strUploads, MsgBoxStyle.Information, "Result")
                Exit For
            End If
        Next
        Me.Cursor = Cursors.Default
    End Sub

    Function GetHTML(ByVal pageUrl As String) As String
        Dim s As String = ""
        Try
            Dim request As System.Net.HttpWebRequest = WebRequest.Create(pageUrl)
            Dim response As HttpWebResponse = request.GetResponse()
            Using reader As StreamReader = New StreamReader(response.GetResponseStream())
                s = reader.ReadToEnd()
            End Using
        Catch ex As Exception
            MsgBox("FAIL: " + ex.Message)
        End Try
        Return s
    End Function

<edit> I almost forgot to mention that when I had tried my original code I posted, it broke at the line that begins with "Dim webSecondCell"; I guess the webFirstRow.cells doesn't have a default property so I couldn't use the shortcut... you have to spell out the full "webFirstRow.cells.item(1)". I wanted to mention that so you didn't just make the changes at the top an leave that line alone.

[TeachMeVB]

If you look at the comment I had [...] ention that so you didn't just make the changes at the top an leave that line alone.

Well... got several errors with the new code... most of them were the same (see picture)
-----------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------
Also, i didn't understand most of what you'd written along with the code (It's starting to get a bit much at once :P (Im a slow learner).. and a noob :P)

Why does this have to be so hard :/

[Pyth007]

Ack! Sorry! I had put a bunch of methods like GetHTML() into my own module MiscMethods that I can then use in other programs. Just remove that part if you've placed the GetHTML function in the same class as the rest of your program (like how it's written above). So the line should just read "Dim strHTML as String = GetHTML(strURL)".

What else were you confused on? You said what I'd written along with the code; does this mean the comments I added to the code, or was it my ramblings in the post?

[TeachMeVB]

Ack! Sorry! I had put a bunch of methods like GetHTML() into my own module MiscMethods that I can then use in other programs. Just remove that part if you've placed the GetHTML function in the same class as the rest of your program (like how it's written above). So the line should just read "Dim strHTML as String = GetHTML(strURL)".

What else were you confused on? You said what I'd written along with the code; does this mean the comments I added to the code, or was it my ramblings in the post?

Thanks for the reply :P Took a while so i was starting to think noone would awnser... (im quite an impatient guy :P)
Your reply made 1 less error, and i managed to remove one myself (not sure how i did it :P)

Anyway, there is one remaining:
--------------------------------------------------------------------------------------------
||
--------------------------------------------------------------------------------------------

Pls tell me how to fix this (I may PM you in the future, you seem like a very good and helpful coder).

[dunfiddlin]

The rather less fancy but more reliable way ... (no mshtml required)

Dim wc As New Net.WebClient
        Dim s = wc.DownloadString("http://vidstatsx.com/PewDiePie/videos-most-recent")
        Dim wb As New WebBrowser With {.DocumentText = "", .ScriptErrorsSuppressed = True} ' don't care about scripts so ignore errors
        wb.Document.Write(s)
        Dim td As HtmlElement = CType((From tr In wb.Document.GetElementsByTagName("h3") Where CType(tr, HtmlElement).InnerText.Contains("Uploads")).FirstOrDefault, HtmlElement)
        TextBox1.Text = td.Parent.NextSibling.InnerText ' the result pops out here
 
' you can do more searches of the source HTML here before
 
        wb.Dispose()
 
' losing the webbrowser object

[TeachMeVB]

The rather less fancy but more reliable way ... (no mshtml required)

Dim wc As New Net.WebClient
        Dim s = wc.DownloadString("http://vidstatsx.com/PewDiePie/videos-most-recent")
        Dim wb As New WebBrowser With {.DocumentText = "", .ScriptErrorsSuppressed = True} ' don't care about scripts so ignore errors
        wb.Document.Write(s)
        Dim td As HtmlElement = CType((From tr In wb.Document.GetElementsByTagName("h3") Where CType(tr, HtmlElement).InnerText.Contains("Uploads")).FirstOrDefault, HtmlElement)
        TextBox1.Text = td.Parent.NextSibling.InnerText ' the result pops out here
 
' you can do more searches of the source HTML here before
 
        wb.Dispose()
 
' losing the webbrowser object

Thanks No errors However... I saved it with the new code, clicked "Build [Program name]" and ran it (debug) and the label that is ment to show the number of vids, doesn't change. It's set to 0 at the moment, and it was suppost to change to the current number of vids, but no :/

Here is the complete code i got: (got some other stuff to but that doesn't influence this problem.)

Imports System.Net
Imports System.IO
Public Class Form1

Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs)
        UpdateVids()
    End Sub

    Private Sub UpdateVids()
        Dim wc As New Net.WebClient
        Dim s = wc.DownloadString("http://vidstatsx.com/PewDiePie/videos-most-recent")
        Dim wb As New WebBrowser With {.DocumentText = "", .ScriptErrorsSuppressed = True} ' don't care about scripts so ignore errors
        wb.Document.Write(s)
        Dim td As HtmlElement = CType((From tr In wb.Document.GetElementsByTagName("h3") Where CType(tr, HtmlElement).InnerText.Contains("Uploads")).FirstOrDefault, HtmlElement)
        Label2.Text = td.Parent.NextSibling.InnerText ' the result pops out here

        ' you can do more searches of the source HTML here before

        wb.Dispose()

        ' losing the webbrowser object
    End Sub

[dunfiddlin]

Well that's because the code isn't executed!

Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Me.Load

[TeachMeVB]

Well that's because the code isn't executed!

Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Me.Load

o.O Well that makes sence... But usually when i double click on form1 it creates the full sub for form1_Load... why didn't it this time? :O
(No need to awnser that.) Thanks for the help guys You're awesome