[RESOLVED] Submit form using HttpWebRequest

[lintz]

The below code allows me to login to a site using HttpWebRequest however I can't login to the site https://www.comsec.com.au (which is the one I want to login to). At the bottom of the code I write the source to a file and upon opening it simply shows the login page.

After looking at the source I believe the issue is caused by the form calling a javascript funciton to actually submit the form when the 'Login' button is clicked. Has anyone had this type of situation in the past and if so what was the solution to get the form to submit?

Dim url As String = "https://www.comsec.com.au/default.aspx"
        Dim data As String = "UcHeader1$LoginName=MyLogin&UcHeader1$Password=MyPassword&UcHeader1$StartIn="""
    
        Dim buffer As Byte() = Encoding.UTF8.GetBytes(data)
        Dim result As String = ""
        Dim req As HttpWebRequest = DirectCast(WebRequest.Create(url), HttpWebRequest)
        req.Method = "POST"
        req.ContentType = "application/x-www-form-urlencoded"
        req.ContentLength = buffer.Length
        'req.Proxy = new WebProxy(proxy, true); // ignore for local addresses
        req.CookieContainer = New CookieContainer()
        ' enable cookies
        Dim reqst As Stream = req.GetRequestStream()
        ' add form data to request stream
        reqst.Write(buffer, 0, buffer.Length)
        reqst.Flush()
        reqst.Close()
 
        Console.WriteLine(vbLf & "Posting data to " & url)
        Dim res As HttpWebResponse = DirectCast(req.GetResponse(), HttpWebResponse)
        ' send request,get response
        Console.WriteLine(vbLf & "Response stream is: " & vbLf)
        Dim resst As Stream = res.GetResponseStream()
        ' display HTTP response
        Dim sr As New StreamReader(resst)
        result = sr.ReadToEnd()
        Using writer As System.IO.StreamWriter = New StreamWriter("C:\startOut.html")
            writer.Write(result)
        End Using

[lintz]

I've searched the web all day and still can't find a soltion.....anyone able to help me out?

[mb2000inc]

Did you ever get this solved? I have a similar situation and was curious to know how you solved it. Please let me know.

Thanks
-mb

[Zeljko]

maybe it's irrelevant but i'm curious to know this:
as U said that this approach works to some other sites but not on this one can you test if you can open it like this from your webbrowser:

'data to send:
'Dim url As String = "https://www.comsec.com.au/default.aspx"
'Dim data As String = "UcHeader1$LoginName=MyLogin&UcHeader1$Password=MyPassword&UcHeader1$StartIn="""

data to type in your browser to test if its working:
https://www.comsec.com.au/default.aspxUcHeader1$LoginName=MyLogin&UcHeader1$Password=MyPassword&UcHeader1$StartIn="

When I use stuff like this with user pass in url encoded I try to test it first in my browser to see if its working...
But in your case I see that this site uses secured http connection so IMO this is not possible because you don't have in your console app needed certificates that says that you are an trusted connection...
That site must first evaluate that certificate to continue, then is your name/pass evaluated and only then you can get in...

[Zeljko]

Some reading:
HTTPS protocol
Certificate on that site is Commonwealth Securities Limited Certificate which is verified by VeriSign.

[lintz]

mb2000inc, yes I have managed to solve my issue with a work around. This is how I did it...

1. Added a hidden websbrowser control to my form and loaded the page.
2. Simulated enteing my login details and clicking the login button via code.
3. Once logged in I grabbed all the cookies that got set (about 15 of them!) and added them to a CookieContainer.
4. Added the CookieContainer to a HttpWebRequest and then now I can get the source of any page as if I had loaded the page in my browser and selected View Page Source

[Zeljko]

nice workaround

[mb2000inc]

Excellent and impressive. I will give that a shot with my issue.

[Megalith]

mb2000inc, yes I have managed to solve my issue with a work around. This is how I did it...

1. Added a hidden websbrowser control to my form and loaded the page.
2. Simulated enteing my login details and clicking the login button via code.
3. Once logged in I grabbed all the cookies that got set (about 15 of them!) and added them to a CookieContainer.
4. Added the CookieContainer to a HttpWebRequest and then now I can get the source of any page as if I had loaded the page in my browser and selected View Page Source

well i done something similar to this but i'm stuck at point 3., in my case i have 13 cookies in my cookie folder, how do i add these cookies to the cookiecontainer? is it a case of reading these with a filestream? some example code would be appreciated for this step

[lintz]

This the code I'm using for step 3.....HTH

'Define cookie container
Public ocookies As CookieContainer = New CookieContainer
Public YumYum As String 'the cookies
 
'Get the cookies once you have logged in from the WB control
Public Sub WB1_DocumentCompleted(ByVal sender As System.Object, ByVal e As System.Windows.Forms.WebBrowserDocumentCompletedEventArgs) Handles WB1.DocumentCompleted
 
        If Me.WB1.ReadyState = WebBrowserReadyState.Complete Then
            WebPageLoaded = True
 
            'read the cookies that have been saved by the webpage
            YumYum = WB1.Document.Cookie
           
        End If
 
 
    End Sub
 
'Then add cooklies to cookie container
Private Sub AddCookiesToCookieJar()
 
        Try
            Dim IECookies As String()
            IECookies = YumYum.Split(New String() {"; "}, StringSplitOptions.RemoveEmptyEntries)
            For i As Integer = 0 To IECookies.Length - 1
                Dim TheCookie As String() = IECookies(i).Split(New Char() {"="c}, 2, StringSplitOptions.RemoveEmptyEntries)
 
               ocookies.Add(AddComsecCookies(TheCookie(0), TheCookie(1)))
               
            Next
 
 
        Catch ex As Exception
            MessageBox.Show(ex.Message)
        End Try
 
    End Sub
 
 
Private Function AddComsecCookies(ByVal name As String, ByVal value As String) As Cookie
        Dim oc As New Cookie
        Dim ExpireDate As String = "#" & DateAdd(DateInterval.Day, 1, Today.Date) & " 1:30:00 PM#"
        oc.Domain = [url]www.comsec.com.au[/url]
        oc.Expires = ExpireDate '("#11/3/2009 1:17:10 PM#")
        oc.Name = name
        oc.Path = "/"
        oc.Secure = False
        oc.Value = value
 
        Return oc
    End Function

[Megalith]

ty lintz great solution to this issue , I found out btw how to do this using just httpwebrequest but it only gets 10 of my 13 cookies :s this method will get them all

[kiwi1342]

Hi, I am sorry to reopen this post but I have a very similar issue and I am trying to figure out your 4th step.

I grabbed all my cookies and I see from your code that you split them one by one. What it's not clear to me is how do you use the cookie container and how you send the GET request (as it's not shown in the code). I am sorry but I am quite a newbie in vb

Thanks for your help

[kiwi1342]

ok I solved it. the solution is here:

http://www.vbforums.com/showthread.php?t=614217