Results 1 to 6 of 6

Thread: WebClient and Webbrowser

  1. #1

    Thread Starter
    Member
    Join Date
    Sep 2013
    Posts
    41

    WebClient and Webbrowser

    Hello all,

    I programmed a small application to extract data from a webpage. All I was using this data for was to check if it had been updated.

    The first issue with this is that you need to log on to get to the webpage. I was originally using WebClient and the login was the Authentic Login type, the pop up box type basically. This was quite easy and worked well!

    After a while things stopped working, this was because parts of the page were changed via JavaScript and WebClient doesn't take this into account. It just uses the raw HTML. I looked into things and found the site used a JSON file that contains all the data. I changed my software to use this, and of course speeded things up a lot!

    Then, the site changed to form based login. WebClient didn't work, as it needs to return cookies. I looked into WebBrowser, and finally got the correct HTML. A few bugs but basically works. The problem is the speed. It has to load the entire page, and some of them are quite large with images. It would seem you can't download a file via WebBrowser silently, and doesn't look like you can display a JSON file in a browser.

    I have recently found information on CookieAwareWebClient, which I hope to test soon.

    My question is, is there anyway to do what I want to do? Using the JSON file would be best as it is so quick.

    Many thanks

  2. #2
    Superbly Moderated NeedSomeAnswers's Avatar
    Join Date
    Jun 2002
    Location
    Manchester uk
    Posts
    2,660

    Re: WebClient and Webbrowser

    and doesn't look like you can display a JSON file in a browser.
    Firstly yes you can, if you just browse to a Json file in a web browser it will show the files contents, but ...

    I am not completely clear on what your doing here, can you post your code? the relevant part of it?

    Its seems like you have tried several different methods.

    Is it the login that has slowed down?

    as if you can login and grab the Json then, Json is very quick and simple to work with !
    Please Mark your Thread "Resolved", if the query is solved & Rate those who have helped you



  3. #3

    Thread Starter
    Member
    Join Date
    Sep 2013
    Posts
    41

    Re: WebClient and Webbrowser

    Hello. Thanks for the reply. Sorry for the long story. Probably best to stick to what I am currently trying.
    I will try navigating to the JSON file, but I'm sure it keeps prompting to download or view.

    What I would like to do is log onto a webpage, using a form based login, and grab a JSON file.
    I am trying the CookieAwareWebClient version I have found. I realise the WebClient variable name isn't the best.. Need to change it...

    Code:
    Dim WebClient As New CookieAwareWebClient
    Dim LoginInfo As New Specialized.NameValueCollection
    LoginInfo.Add("uid", "xxx")
    LoginInfo.Add("pwd", "xxx")
               
    Dim res = WebClient.UploadValues(WebUrl, "POST", LoginInfo)
    
    Dim resbody = (New System.Text.UTF8Encoding).GetString(res)
    
    MsgBox(resbody)
    The variable "res" returns the HTML for the login page. I think I know my problem, I'm not sure the site has an actual login page. The login page looks like it is dynamically created when there is no log cookie or something. I need to find a actual login page I think. I have also checked the submit form HTML and added any hidden values.

    I had all this working before, as the site used the popup login window type, and WebClient handled this ok, but then they changed it. I only want to do this so I can easily check for updated sections.

    Many thanks

  4. #4
    You don't want to know.
    Join Date
    Aug 2010
    Posts
    4,578

    Re: WebClient and Webbrowser

    You should ask the site to implement an API or, barring that, ask them to give you better access to the data. What is happening looks like they don't want you to access the data in an automated fashion, so every time you find a way around their login they update the site to thwart you. This is going to continue forever, they're going to make it harder and harder to log in. So you may as well ask for an automation angle that they will bless.
    This answer is wrong. You should be using TableAdapter and Dictionaries instead.

  5. #5
    Super Moderator Shaggy Hiker's Avatar
    Join Date
    Aug 2002
    Location
    Idaho
    Posts
    38,988

    Re: WebClient and Webbrowser

    I second the API suggestion. What you have already encountered is the reason that dealing with web pages is the worst idea aside from doing nothing (and doing nothing is sometimes better, too). Web pages change a LOT. Sometimes this is security, often it is just because I've never known a web page creator who didn't like to tinker incessantly. The only thing that allows some pages to remain unchanged is that the people who created them get called away to other things. Therefore, trying to work with a web page is an exercise it trying to hit a constantly moving target. That's what API solutions are all about: They don't change, or change very rarely. Of course, it means that the site owner has to be willing to create an API solution, but they often are, so it's worth a shot. After all, they have taken the time, effort, and cost, of putting up a web page to get information out to people, so you are just asking for a different means.
    My usual boring signature: Nothing

  6. #6

    Thread Starter
    Member
    Join Date
    Sep 2013
    Posts
    41

    Re: WebClient and Webbrowser

    Thanks for the replies. Yes, it look likes they don't want me to get the data!! I have found a way to read the JSON, I found it on another forum. A slight adjust of the registry causes .JSON files to display in the IE. As the WebBrowser in VB seems to use IE, it worked in there also. Its only for my use, so not a problem. A lot quicker now! Thanks again

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width