[RESOLVED] HTTP GET Request

[kiwi1342]

Hi,

I searched around the forum but I couldn't find the right answer for my problem.

I am trying to make a GET request to a .php page. To get to this page I have to pass before to a login page (username and password then).

My application has a browser from where I manually login and then with one button I want to do this GET request but not from the browser. I need to create a GET request using the cookie and Session ID from the browser in my app.

I think I need to create this GET with the original header from the browser but I am new to vb and I am trying to figure out how:

1. create a simple GET request
2. add cookie, session ID, headers to the GET request
3. get the page source (I already use the browser.document but I'd like to retrieve it from the get request).

Can someone pls help me or point me to the right direction?

Thanks for your help and sorry for my bad english

[brin351]

The webRequest object can make get/post requests and return the response (no browser). You can also add a cookies collection but the session I'm not sure about, you'd have to look into the documentation about that.

[kiwi1342]

thanks for the reply but I was wondering if someone of you might post some simple example. How to make a simple get request with no header.

I have then to understand how to add cookies, session id and headers to it.

[cicatrix]

Dim wr As Net.WebRequest = Net.HttpWebRequest.Create("http://www.google.com")
Dim response As Net.WebResponse = wr.GetResponse ' Contains header, cookies and response stream
Dim responseStream As IO.Stream = response.GetResponseStream ' Gets the response stream

But it's very strange for a login page to use GET.

[kiwi1342]

Dim wr As Net.WebRequest = Net.HttpWebRequest.Create("http://www.google.com")
Dim response As Net.WebResponse = wr.GetResponse ' Contains header, cookies and response stream
Dim responseStream = response.GetResponseStream ' Gets the response stream

But it's very strange for a login page to use GET.

no well it's not for the login process. I will login manually, once logged in from the browser in my app I will need to make a get request NOT from the browser but using the cookie and session id used in the browser.

IN the ecample above can I add custom headers, cookies etc?

thanks

[cicatrix]

Yes, you can. But web doesn't work that way. To illustrate my point simply go to the login page with Internet explorer and login. Then go to your other page using Firefox, Opera or Chrome (any other browser).
You'll see that you won't be logged in there since generally your login information is stored in a cookie file and the web-server 'remembers' you by the cookie it sent you on logon. A different browser (and your code will be considered as a different browser) will have other session id and different set of cookies (you won't be able to re-use those from your login).

You will have to login using the same application.

[kiwi1342]

Yes, you can. But web doesn't work that way. To illustrate my point simply go to the login page with Internet explorer and login. Then go to your other page using Firefox, Opera or Chrome (any other browser).
You'll see that you won't be logged in there since generally your login information is stored in a cookie file and the web-server 'remembers' you by the cookie it sent you on logon. A different browser (and your code will be considered as a different browser) will have other session id and different set of cookies (you won't be able to re-use those from your login).

You will have to login using the same application.

doh, are you sure? I found a way to get my browser cookies and I see the session ID is present and the main cookie too. In theory if I add them to a custom http request I should be able to get it working.

[cicatrix]

In theory... well maybe you will be able to accomplish that but wouldn't it be easier just to login to the website using the same application? All you need is to send a POST request with your credentials and save the cookies from httpwebresponse.

[kiwi1342]

well, I have to say that in my app once you log in, you have to navigate through the website. If I login manually and then I do the POST to login again I get disconnected from the previous session making impossible to navigate.

I was reading this internale post but it's quite difficult for me to understand but it seems it's similar to my case:
http://www.vbforums.com/showthread.p...t=http+request

[cicatrix]

You don't have to login through a browser in my scenario. You can have two textboxes (login/password) on your form and send these credentials in a POST request to login. Then you can browse your site.

[kiwi1342]

You don't have to login through a browser in my scenario. You can have two textboxes (login/password) on your form and send these credentials in a POST request to login. Then you can browse your site.

hmm I might try. How can I do that? But once I do the POST can I navigate with the browser manually?

[cicatrix]

The same, but before obtaining the response stream you should write your name=value pairs in the request stream and provide cookies for each subsequent GET request.

Your login page contains a form which has the url of the post request and names of the fields you need to provide (name, password, sometimes SID, etc). You use them to send POST data back to the server.

The code should look something like this:

' 

        Dim request As Net.WebRequest = Net.HttpWebRequest.Create("http://www.google.com")

        request.Method = "POST"
        ' Form up the headers using request.Headers.Add method here.

        ' Write the POST stream
        Using wrt As New IO.StreamWriter(request.GetRequestStream)
            wrt.Write("user=USERNAME&password=PASSWORD") ' You should substitute fields from the html form here.
            wrt.Flush()
        End Using

        Dim response As Net.WebResponse = request.GetResponse ' Contains header, cookies and response stream
        Dim responseStream As IO.Stream = response.GetResponseStream ' Gets the response stream
        ' Get cookies from the response and keep them for later use.

P.S. Use Fiddler web-debugger to intercept your browser-made login request and see what's inside the post stream.
Fiddler is a very sophisticated tool that can help you greatly for web debugging (it's freely downloadable from the developer's site).

[KilSoft]

Use this function, it makes it a lot easier.

' EasyHttp.vb class file
Public Class EasyHttp
    Public Enum HTTPMethod As Short
        HTTP_GET = 0
        HTTP_POST = 1
    End Enum
 
    Public Shared Function Send(ByVal URL As String, _
        Optional ByVal PostData As String = "", _
        Optional ByVal Method As HTTPMethod = HTTPMethod.HTTP_GET, _
        Optional ByVal ContentType As String = "")
 
        Dim Request As HttpWebRequest = WebRequest.Create(URL)
        Dim Response As HttpWebResponse
        Dim SW As StreamWriter
        Dim SR As StreamReader
        Dim ResponseData As String
 
        ' Prepare Request Object
        Request.Method = Method.ToString().Substring(5)
 
        ' Set form/post content-type if necessary
        If (Method = HTTPMethod.HTTP_POST AndAlso PostData >< "" AndAlso ContentType = "") Then
            ContentType = "application/x-www-form-urlencoded"
        End If
 
        ' Set Content-Type
        If (ContentType >< "") Then
            Request.ContentType = ContentType
            Request.ContentLength = PostData.Length
        End If
 
        ' Send Request, If Request
        If (Method = HTTPMethod.HTTP_POST) Then
            Try
                SW = New StreamWriter(Request.GetRequestStream())
                SW.Write(PostData)
            Catch Ex As Exception
                Throw Ex
            Finally
                SW.Close()
            End Try
        End If
 
        ' Receive Response
        Try
            Response = Request.GetResponse()
            SR = New StreamReader(Response.GetResponseStream())
            ResponseData = SR.ReadToEnd()
        Catch Wex As System.Net.WebException
            SR = New StreamReader(Wex.Response.GetResponseStream())
            ResponseData = SR.ReadToEnd()
            Throw New Exception(ResponseData)
        Finally
            SR.Close()
        End Try
 
        Return ResponseData
    End Function
End Class

Examples of use:

Response.Write(EasyHttp.Send("http://yahoo.com"))
Response.Write(EasyHttp.Send("http://search.yahoo.com/bin/search", "p=test"))

You only have to add the cookies to the function.

KilSoft

[kiwi1342]

The same, but before obtaining the response stream you should write your name=value pairs in the request stream and provide cookies for each subsequent GET request.

Your login page contains a form which has the url of the post request and names of the fields you need to provide (name, password, sometimes SID, etc). You use them to send POST data back to the server.

The code should look something like this:

' 

        Dim request As Net.WebRequest = Net.HttpWebRequest.Create("http://www.google.com")

        request.Method = "POST"
        ' Form up the headers using request.Headers.Add method here.

        ' Write the POST stream
        Using wrt As New IO.StreamWriter(request.GetRequestStream)
            wrt.Write("user=USERNAME&password=PASSWORD") ' You should substitute fields from the html form here.
            wrt.Flush()
        End Using

        Dim response As Net.WebResponse = request.GetResponse ' Contains header, cookies and response stream
        Dim responseStream As IO.Stream = response.GetResponseStream ' Gets the response stream
        ' Get cookies from the response and keep them for later use.

P.S. Use Fiddler web-debugger to intercept your browser-made login request and see what's inside the post stream.
Fiddler is a very sophisticated tool that can help you greatly for web debugging (it's freely downloadable from the developer's site).

Thanks for your kind help I already use fiddler, it's a great program and I don't have any sort of issue in understanding it.

From the code you posted there is the line:

' Form up the headers using request.Headers.Add method here.

It's empty and I guess I should leave it so (I'll do my test later on) but I wanted to ask you if there is a guide explaining what fields should I add to it, I mean, is there a special syntax that I shall use?

[kiwi1342]

Use this function, it makes it a lot easier.

' EasyHttp.vb class file
Public Class EasyHttp
    Public Enum HTTPMethod As Short
        HTTP_GET = 0
        HTTP_POST = 1
    End Enum
 
    Public Shared Function Send(ByVal URL As String, _
        Optional ByVal PostData As String = "", _
        Optional ByVal Method As HTTPMethod = HTTPMethod.HTTP_GET, _
        Optional ByVal ContentType As String = "")
 
        Dim Request As HttpWebRequest = WebRequest.Create(URL)
        Dim Response As HttpWebResponse
        Dim SW As StreamWriter
        Dim SR As StreamReader
        Dim ResponseData As String
 
        ' Prepare Request Object
        Request.Method = Method.ToString().Substring(5)
 
        ' Set form/post content-type if necessary
        If (Method = HTTPMethod.HTTP_POST AndAlso PostData >< "" AndAlso ContentType = "") Then
            ContentType = "application/x-www-form-urlencoded"
        End If
 
        ' Set Content-Type
        If (ContentType >< "") Then
            Request.ContentType = ContentType
            Request.ContentLength = PostData.Length
        End If
 
        ' Send Request, If Request
        If (Method = HTTPMethod.HTTP_POST) Then
            Try
                SW = New StreamWriter(Request.GetRequestStream())
                SW.Write(PostData)
            Catch Ex As Exception
                Throw Ex
            Finally
                SW.Close()
            End Try
        End If
 
        ' Receive Response
        Try
            Response = Request.GetResponse()
            SR = New StreamReader(Response.GetResponseStream())
            ResponseData = SR.ReadToEnd()
        Catch Wex As System.Net.WebException
            SR = New StreamReader(Wex.Response.GetResponseStream())
            ResponseData = SR.ReadToEnd()
            Throw New Exception(ResponseData)
        Finally
            SR.Close()
        End Try
 
        Return ResponseData
    End Function
End Class

Examples of use:

Response.Write(EasyHttp.Send("http://yahoo.com"))
Response.Write(EasyHttp.Send("http://search.yahoo.com/bin/search", "p=test"))

You only have to add the cookies to the function.

KilSoft

wow thanks a lot guys. I will try all this stuff out in few hours. Thanks again for your help!!!

[kiwi1342]

Use this function, it makes it a lot easier.

' EasyHttp.vb class file
Public Class EasyHttp
    Public Enum HTTPMethod As Short
        HTTP_GET = 0
        HTTP_POST = 1
    End Enum
 
    Public Shared Function Send(ByVal URL As String, _
        Optional ByVal PostData As String = "", _
        Optional ByVal Method As HTTPMethod = HTTPMethod.HTTP_GET, _
        Optional ByVal ContentType As String = "")
 
        Dim Request As HttpWebRequest = WebRequest.Create(URL)
        Dim Response As HttpWebResponse
        Dim SW As StreamWriter
        Dim SR As StreamReader
        Dim ResponseData As String
 
        ' Prepare Request Object
        Request.Method = Method.ToString().Substring(5)
 
        ' Set form/post content-type if necessary
        If (Method = HTTPMethod.HTTP_POST AndAlso PostData >< "" AndAlso ContentType = "") Then
            ContentType = "application/x-www-form-urlencoded"
        End If
 
        ' Set Content-Type
        If (ContentType >< "") Then
            Request.ContentType = ContentType
            Request.ContentLength = PostData.Length
        End If
 
        ' Send Request, If Request
        If (Method = HTTPMethod.HTTP_POST) Then
            Try
                SW = New StreamWriter(Request.GetRequestStream())
                SW.Write(PostData)
            Catch Ex As Exception
                Throw Ex
            Finally
                SW.Close()
            End Try
        End If
 
        ' Receive Response
        Try
            Response = Request.GetResponse()
            SR = New StreamReader(Response.GetResponseStream())
            ResponseData = SR.ReadToEnd()
        Catch Wex As System.Net.WebException
            SR = New StreamReader(Wex.Response.GetResponseStream())
            ResponseData = SR.ReadToEnd()
            Throw New Exception(ResponseData)
        Finally
            SR.Close()
        End Try
 
        Return ResponseData
    End Function
End Class

Examples of use:

Response.Write(EasyHttp.Send("http://yahoo.com"))
Response.Write(EasyHttp.Send("http://search.yahoo.com/bin/search", "p=test"))

You only have to add the cookies to the function.

KilSoft

Ok I am trying using this function and I have few issues:
the first is that using the line

Response.Write(EasyHttp.Send("http://yahoo.com"))

I get tge Response bit aligned as Not declared.

The other problem is the post line

Response.Write(EasyHttp.Send("http://search.yahoo.com/bin/search", "p=test"))

Well I guess this is the post but no, this is a GET function which doesn't actually send p=test (I checked using fiddler). Maybe I am missing something.
Can you help me please?

thanks again

[cicatrix]

Don't use Response.Write, try just like this:

EasyHttp.Send("http://search.yahoo.com/bin/search", "p=test")

[kiwi1342]

Don't use Response.Write, try just like this:

EasyHttp.Send("http://search.yahoo.com/bin/search", "p=test")

thanks for the quick response

I already tried that and it's a GET request, not a post :/

[cicatrix]

Does the declaration of the EasyHttp.Send method tell you anything? Especially the red part:

  Public Shared Function Send(ByVal URL As String, _
        Optional ByVal PostData As String = "", _
        Optional ByVal Method As HTTPMethod = HTTPMethod.HTTP_GET, _
        Optional ByVal ContentType As String = "")

If no method is specified it would generate a GET request.

[kiwi1342]

Does the declaration of the EasyHttp.Send method tell you anything? Especially the red part:

  Public Shared Function Send(ByVal URL As String, _
        Optional ByVal PostData As String = "", _
        Optional ByVal Method As HTTPMethod = HTTPMethod.HTTP_GET, _
        Optional ByVal ContentType As String = "")

If no method is specified it would generate a GET request.

You are absolutely right. I was in fact reading line by line and I was focused on that part. I just had to press space after the "p=test" and VB helped me out suggesting the method.

Now I have to try to get the streamdata and then add the cookies (which is the most difficult part to me as I can not find any reference on the internet).

thx

[kiwi1342]

up please.

KilSoft how do I read the response? I keep getting an error in using

Response.Write(....

I am trying now adding the cookies to my request. Now a newbie question:
If I understand correctly a http request follow these steps:

1. create a url
2. add cookie to a container which automatically attach to the next http request (I think and I am not sure about this step)
3. start the request
4. wait for response

Am I right about step 2?

Thanks

[kiwi1342]

up please.

KilSoft how do I read the response? I keep getting an error in using

Response.Write(....

I am trying now adding the cookies to my request. Now a newbie question:
If I understand correctly a http request follow these steps:

1. create a url
2. add cookie to a container which automatically attach to the next http request (I think and I am not sure about this step)
3. start the request
4. wait for response

Am I right about step 2?

Thanks

ok I'll answer to myself! Yes adding a cookieContainer to the request will automatically attach it to the GET/POST.

I managed to add cookies at the KilSoft http function but I am still missing few information which usually appear when clicked manually on a link. I'll test some more and I'll get back soon

[kiwi1342]

I solved it!!!!!

Thanks a lot to all of you guys. Really, big thanks!

So what I've done is:

1. get the cookies which are in my browser (this after I logged in so I have the Session ID too)
2. create a cookie container and add them to it
3.create custom headers so that in flidder look like a real GET

 Public Class EasyHttp
        Public Enum HTTPMethod As Short
            HTTP_GET = 0
            HTTP_POST = 1
        End Enum

        Public Shared Function Send(ByVal URL As String, _
            Optional ByVal PostData As String = "", _
            Optional ByVal Method As HTTPMethod = HTTPMethod.HTTP_GET, _
            Optional ByVal ContentType As String = "")

            Dim Request As HttpWebRequest = WebRequest.Create(URL)
            Dim Response As HttpWebResponse
            Dim SW As StreamWriter
            Dim SR As StreamReader
            Dim ResponseData As String

            ' create custom headers to make it look real
            Request.Accept = "image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/xaml+xml, application/vnd.ms-xpsdocument, application/x-ms-xbap, application/x-ms-application, */*"
            Request.Referer = URL
            Request.Headers.Add(HttpRequestHeader.AcceptLanguage, "it")
            Request.Headers.Add("UA-CPU", "x86")
            Request.Headers.Add(HttpRequestHeader.AcceptEncoding, "deflate")
            Request.UserAgent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; InfoPath.1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E)"
            Request.KeepAlive = True



            'Create the cookie container
            Request.CookieContainer = New CookieContainer()

            Try
                Dim IECookies As String()


                'take the cookie from the browser where I am logged in
                Dim YumYum = Form1.browser0.Document.Cookie

                
                ' I have to split the cookie removing ; and =
                IECookies = YumYum.Split(New String() {"; "}, StringSplitOptions.RemoveEmptyEntries)
                
                For i As Integer = 0 To IECookies.Length - 1
                    Dim TheCookie As String() = IECookies(i).Split(New Char() {"="c}, 2, StringSplitOptions.RemoveEmptyEntries)

                    ' for each cookie found create a new one for the request
                    Request.CookieContainer.Add(New Uri(URL), _
                New Cookie(TheCookie(0), TheCookie(1)))

                Next

            Catch ex As Exception
                MessageBox.Show(ex.Message)
            End Try



            ' Prepare Request Object
            Request.Method = Method.ToString().Substring(5)

            ' Set form/post content-type if necessary
            If (Method = HTTPMethod.HTTP_POST AndAlso PostData <> "" AndAlso ContentType = "") Then
                ContentType = "application/x-www-form-urlencoded"
            End If

            ' Set Content-Type
            If (ContentType <> "") Then
                Request.ContentType = ContentType
                Request.ContentLength = PostData.Length
            End If


            ' Send Request, If Request
            If (Method = HTTPMethod.HTTP_POST) Then
                Try
                    SW = New StreamWriter(Request.GetRequestStream())
                    SW.Write(PostData)
                Catch Ex As Exception
                    Throw Ex
                Finally
                    SW.Close()
                End Try
            End If

            ' Receive Response
            Try
                Response = Request.GetResponse()
                SR = New StreamReader(Response.GetResponseStream())
                ResponseData = SR.ReadToEnd()
            Catch Wex As System.Net.WebException
                SR = New StreamReader(Wex.Response.GetResponseStream())
                ResponseData = SR.ReadToEnd()
                Throw New Exception(ResponseData)
            Finally
                SR.Close()

            End Try

            Return ResponseData

        End Function


    End Class

and as explained I use the following method to call it:

GET

EasyHttp.Send("http://www.lol.com")

POST

EasyHttp.Send("http://www.lol.com",  "p=test", 2)

I found very usefull the following threads/guide

http://msdn.microsoft.com/en-gb/libr...(v=VS.95).aspx

http://msdn.microsoft.com/en-gb/libr...(v=VS.90).aspx

http://www.vbforums.com/showthread.php?t=589448

http://msdn.microsoft.com/en-gb/libr...(v=VS.90).aspx

http://www.vbforums.com/showthread.p...r+http+request


Hope it helps