PHP User Warning: fetch_template() calls should be replaced by the vB_Template class. Template name: bbcode_highlight in ..../includes/functions.php on line 4197
Web scraper stopped by popup-VBForums
Results 1 to 4 of 4

Thread: Web scraper stopped by popup

  1. #1

    Thread Starter
    New Member
    Join Date
    May 2012
    Posts
    7

    Web scraper stopped by popup

    Hello!

    I recently wrote an outlook macro in VBA that creates a internet explorer object, navigates to a secure web page, logs in, and scrapes data from that page. It works beautifully.

    The only problem.is that occasionally this page displays a notification within a pop up window. That causes an error in the scrape method because the active window is now the popup, where those elements do not exist.

    Does any one have an idea on how I can either bypass/block the popup, or keep the main window active? I'm sure the solution is simple, and it's probably method im unaware of.

    Thanks!!

  2. #2
    PowerPoster
    Join Date
    Dec 2004
    Posts
    24,568

    Re: Web scraper stopped by popup

    is the popup a webpage or a dialog?
    i do my best to test code works before i post it, but sometimes am unable to do so for some reason, and usually say so if this is the case.
    Note code snippets posted are just that and do not include error handling that is required in real world applications, but avoid On Error Resume Next

    dim all variables as required as often i have done so elsewhere in my code but only posted the relevant part

    come back and mark your original post as resolved if your problem is fixed
    pete

  3. #3

    Thread Starter
    New Member
    Join Date
    May 2012
    Posts
    7

    Re: Web scraper stopped by popup

    Quote Originally Posted by westconn1 View Post
    is the popup a webpage or a dialog?
    It pops up in a new internet explorer window in front of the VBA actived window.

    Here is my code. Please note that this is an Outlook Macro.

    Code:
     Sub ImpoprtData()
    
    Dim sched As String
    Dim ie As New InternetExplorer
    Dim oHTML_Element As IHTMLElement
         
    ie.Visible = False
    
    ie.navigate "my url goes here"
    
    Do
    DoEvents
    Loop Until ie.readyState = READYSTATE_COMPLETE
    
    Dim doc As HTMLDocument
    Set doc = ie.document
    
    'If I am already logged in, it will return an error. So this part skips the login in that event' 
    
        On Error GoTo Errorclear:
        
          'set the input fields'
            doc.all.UserName.Value = "my user name"
            doc.all.Password.Value = "my password"
            
            'this part inputs the fields and clicks the submit button for me'
            
            For Each oHTML_Element In doc.getElementsByTagName("input")
            If oHTML_Element.Type = "submit" Then
            oHTML_Element.Click
            End If
            Next
            
    Errorclear:
    
    'the pop up appears now, as soon as I am logged in' 
      
    'this is the scrape function to pull the web elements. I usually encounter the error here, since it appears that VBA is trying to pull data from the pop up instead' 
    
    sched = Trim(doc.getElementsByTagName("td")(37).innerText)
    
    'after this, there is a lot more code that uses the web data to create outlook calendar items. But that part works perfectly so I left it out.'
    Please help!! Circumventing this pop up is driving me crazy... I may become an alcoholic before I'm done.

  4. #4
    PowerPoster
    Join Date
    Dec 2004
    Posts
    24,568

    Re: Web scraper stopped by popup

    you can not rely on web sites keeping the same design for their pages for any length of time, they are continuously updating and changing the way they work, you can expect to have to update your code reasonably often

    it would be possible to automate the new IE window, as preventing it from appearing may just stop the page from loading at all

    you can try like
    vb Code:
    1. set sh = createobject("shell.application")
    2. set w = sh.windows(sh.windows.count)
    3. ' you may need to wait here while the page completes
    4. schd = Trim(w.document.getElementsByTagName("td")(37).innerText)
    i do my best to test code works before i post it, but sometimes am unable to do so for some reason, and usually say so if this is the case.
    Note code snippets posted are just that and do not include error handling that is required in real world applications, but avoid On Error Resume Next

    dim all variables as required as often i have done so elsewhere in my code but only posted the relevant part

    come back and mark your original post as resolved if your problem is fixed
    pete

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Featured


Click Here to Expand Forum to Full Width