Results 1 to 4 of 4

Thread: [RESOLVED] I need pro HELP with HTTPClient or HTTPWebRequest!

  1. #1

    Thread Starter
    New Member
    Join Date
    Jul 2013
    Posts
    14

    Resolved [RESOLVED] I need pro HELP with HTTPClient or HTTPWebRequest!

    I'm building a windows application that will connect to a website using private proxies and do stuff on that site. I want to use the new HTTPClient to do that. I have the next scenario for the login part:

    Code:
    1 - I GET the login page. The login page set a cookie (I need to store that).
    2 - In the code of the login page I find some dynamic parameters that I scrape using Regex.
     I create the POST url using the parameters that I scraped and an user and a password.
    3 - I use the cookie that the login page stored and  I POST to the website (the created URL).
    4 - I receive the content of the response and if the login is done correctly I see in the code a redirect url (the http web request will not follow this redirect!) and a cookie is stored.
    5- I scrape the redirect url and I use the cookies stored with the POST and do a GET request to the redirect url. Another final cookie is stored (overwritten the last one)
    I have tried to do this with httpwebrequest but after the POST I get the response and when I load that response in a browser I see "your browser has the cookies functionality turned off" so I obviously do not know how to store my cookies and use them with the next request then store the new cookies and use them with the last request ...

    Now I try to do all over again with HTTPClient ... does anyone can help me with an example using proxy (with proxy user and proxy password)for all 3 requests an with cookie functionality ??

    Please HELP!!!

    I do not need links to msdn, I have read A LOT, I have experimented A LOT, I did not sleep in the last 48 hours reading and testing so I'm trying my luck on the forum. May be someone will want to share a working code that do something similar (with HTTPWebRequest or the new HTTPClient) so I can test and find out why my code do not use the first cookie to POST the data (I have serialized the cookie and I know that the response from the first GET is saving the cookie, I save it on my hard drive) ...


    EDIT: After some sleep I "discovered" why all the "wonderful" examples of HTTPClient out there are written in C# and why I could not find HTTPClient in my VStudio 2012: HTTPClient do not exist in VB.NET, is only implemented for C# C++and JS ... so please help me with HTTPWebRequests (my initial approach)...
    EDIT: strange thing - if I start a project in C# I have httpclient by default and in vb I do not have it, but I was able to install in now in VB using NuGet manager... so there is httpclient support in vb.net after all
    Last edited by eSolution; Jul 8th, 2013 at 02:33 AM.

  2. #2
    Bad man! ident's Avatar
    Join Date
    Mar 2009
    Location
    Cambridge
    Posts
    5,398

    Re: I need pro HELP with HTTPClient or HTTPWebRequest!

    post your code

  3. #3

    Thread Starter
    New Member
    Join Date
    Jul 2013
    Posts
    14

    Re: I need pro HELP with HTTPClient or HTTPWebRequest!

    Quote Originally Posted by ident View Post
    post your code
    Hi ident!
    This is my code - the version that return: "cookies functionality disabled, please enable cookies"

    Code:
    Imports System.Net
    Imports System.Text
    Imports System.IO
    Imports System.Runtime.Serialization.Formatters.Binary
    Imports System.Text.RegularExpressions
    Imports System.Web
    
    Namespace Functions
        Public Class Functions
    
            Public xuser As String
            Public xpassword As String
            Public xproxy As String
            Public xport As String
            Public xproxypassword As String
            Public xproxyuser As String
            Public xphonenumber As String
    
            Public logincookies As CookieContainer
    
            Dim UA As String = "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:22.0) Gecko/20100101 Firefox/22.0"
    
    
            Public Function login(ByVal user As String, ByVal password As String, ByVal proxy As String, ByVal port As String, ByVal proxyuser As String, ByVal proxypassword As String, ByVal phonenumber As String)
                xuser = user
                xpassword = password
                xproxy = proxy
                xport = port
                xproxypassword = proxypassword
                xproxyuser = proxyuser
                xphonenumber = phonenumber
    
                logincookies = New CookieContainer
                'start the login phase
                'initiate the GET procedure for the login page
                'set the login url
                Dim loginpage As String = "https://theloginpageURL"
    
                'the GET request
                Dim request As HttpWebRequest = DirectCast(WebRequest.Create(loginpage), HttpWebRequest)
                request.UserAgent = UA
                request.Referer = "http://thepagethatreferedmeURL"
                request.KeepAlive = True
    
                'set the cookies for the request to the public cookie container
                request.CookieContainer = New CookieContainer()
                request.CookieContainer = logincookies
    
                'get the response from the server
                Dim response As HttpWebResponse = DirectCast(request.GetResponse(), HttpWebResponse)
    
                logincookies.Add(response.Cookies)
    
                'start the stream reader to read the response
                Dim reader As New StreamReader(response.GetResponseStream())
                Dim loginpagecontent As String = reader.ReadToEnd
    
                '-------------------------------
                'read the login page and get the parameters
    
                Dim pageGetLogin As String = loginpagecontent
    
                'Get the parameters
                Dim param1 As String = "this is a parameter value"
                Dim param2 As Object
                Dim param2fin As String
                param2 = Regex.Match(pageGetLogin, "(?<=param2""\s* value="")[\w\W]*?(?="")")
                param2fin = param2.ToString
                Dim param3 As String = "this is a parameter value"
                Dim param4 As String = "this is a parameter value"
                '----------------------------------
    
                'creating the POST request
                'declaring the encoding type
                Dim encoding As New UTF8Encoding
    
                'creating the post data
                'creating the Request
                Dim postLoginRequest As HttpWebRequest = DirectCast(WebRequest.Create("https://website.com/loginservice?param1=" & param1 & "&param2=" & param2fin & "&param3=" & param3 & "&param4=" & param4 & "&Email=" & Email & "&Passwd=" & Passwd & "&signIn=" & signIn), HttpWebRequest)
                'postLoginRequest.CookieContainer = logincookies
                'postLoginRequest.CookieContainer = cookieContainer
                postLoginRequest.Method = "POST"
                postLoginRequest.KeepAlive = True
                postLoginRequest.AllowAutoRedirect = True
                postLoginRequest.MaximumAutomaticRedirections = 50
                postLoginRequest.ContentType = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
                postLoginRequest.Referer = "https://theloginpageURL"
                postLoginRequest.UserAgent = UA
                postLoginRequest.ContentLength = 0
                postLoginRequest.AutomaticDecompression = DecompressionMethods.GZip
    
                postLoginRequest.CookieContainer = New CookieContainer
                postLoginRequest.CookieContainer = logincookies
    
                'getting the response to the POST request
    
                Dim postLoginResponse As HttpWebResponse
                postLoginResponse = DirectCast(postLoginRequest.GetResponse(), HttpWebResponse)
    
                logincookies.Add(postLoginResponse.Cookies)
    
                Dim postLoginReader As New StreamReader(postLoginResponse.GetResponseStream())
                Dim loginResponsePageContent As String = postLoginReader.ReadToEnd
    
    
                'show the parameters to check if they are find corectly - to comment out later
                MessageBox.Show(param1 & "--" & param2 & "--" & user & "--" & xport & "--" & xproxyuser & "--" & xproxypassword)
    
    
                'serialize the cookies to bin format and save them to the disk
                '----------------------------------
    
                'set the name for the cookie file
                Dim filename As String = "d:/" & "cookie.dat"
                'delete the old file if it exists
                If (File.Exists(filename)) Then
                    File.Delete(filename)
                End If
                'serialize and write the file
                Dim stream As FileStream = File.Create(filename)
                Dim formatter As New BinaryFormatter()
                formatter.Serialize(stream, response.Cookies)
                stream.Close()
    
    
                'Return the functions values
                'Return loginpagecontent
                Return loginResponsePageContent
            End Function
    
    
        End Class
    End Namespace
    I have replaced all the URL's in the code and most of the commented code that I used in later trials. I removed the parameters also and I replaced them with similar ones (also Iformated the post url to match the real one in aspect)

    I call this class from a button on a form and I supply the response variable to a web browser component just to see how the response look (I'm to tired to read the code in a text editor by now ) )

    The cookie that is saved is the first cookie that the login page set so I know that the first cookie is saved in the container.
    The GET part is working as it should and I receive the response and I parse it and obtain all the parameters, I can verify that I have them by showing them in a messagebox.show()
    The POST it seem to be ok but I receive the "cookies disabled" message on the response html...
    Please show me where I,m doing things wrong and how to fix it...
    I can't post the real URL's and the entire code as I would broke the rules of this forum and I don't want that (and I don't want any more discussions with other users about the rules ... I think an experienced programmer will spot the mistakes from this code)

    Thank You!!

    PS - this code does not have the next GET as I can't receive a good response from the POST so I can't find the redirect URL and make the next GET request
    Last edited by eSolution; Jul 8th, 2013 at 04:51 AM.

  4. #4

    Thread Starter
    New Member
    Join Date
    Jul 2013
    Posts
    14

    Re: I need pro HELP with HTTPClient or HTTPWebRequest!

    After another full day of searching and experimenting I have solved the problem using HTTPClient ... Now I have to play with the HTTPClientHandler to be able to use the cookies with another clients instances and also to use private proxies ... It seems that nobody is using HTTPClient and .net fr 4.5 ... no one answered my question here and on another 3 major forums

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width