We have a site that was scraping a site to gather all models available of a product. The 3rd party site recently changed the website so it now uses ajax for users to select the manufacturer and then once they select that it loads a dropdown with products using ajax.
I currently was using httpwebrequest for all requests (see below).
Now in fiddler the post appears to be done with ajax. I tried to send the post data the normal way but it didn't like that.Code:Public Function fnRequest(ByVal sPOSTData As String, Optional ByVal bAutoRedirect As Boolean = False) As String Dim uriSite As Uri Dim sReturn As String Dim srReader As StreamReader Dim sTemp As String sReturn = String.Empty Try ' Setup request uriSite = New Uri(m_sURL) m_hwrRequest = DirectCast(WebRequest.Create(uriSite), HttpWebRequest) m_hwrRequest.Referer = m_sReferer m_hwrRequest.UserAgent = m_sUserAgent m_hwrRequest.AllowAutoRedirect = bAutoRedirect m_hwrRequest.AllowWriteStreamBuffering = True m_hwrRequest.KeepAlive = False m_hwrRequest.Accept = "image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms-powerpoint, application/vnd.ms-excel, application/msword, application/x-shockwave-flash, */*" If Not (m_ccCookies Is Nothing) Then If m_ccCookies.Count > 0 Then m_hwrRequest.CookieContainer = New CookieContainer() m_hwrRequest.CookieContainer.Add(m_ccCookies) End If End If If Not (sPOSTData Is Nothing) AndAlso sPOSTData.Length > 0 Then Dim stWS As Stream Dim aeEnc As ASCIIEncoding Dim baBuf As Byte() aeEnc = New ASCIIEncoding() baBuf = aeEnc.GetBytes(sPOSTData) m_hwrRequest.Method = "POST" m_hwrRequest.ContentLength = baBuf.Length m_hwrRequest.ContentType = "application/x-www-form-urlencoded" stWS = m_hwrRequest.GetRequestStream() stWS.Write(baBuf, 0, baBuf.Length) stWS.Close() 'm_hwrRequest.AllowAutoRedirect = True End If m_hwrResponse = DirectCast(m_hwrRequest.GetResponse(), HttpWebResponse) srReader = New StreamReader(m_hwrResponse.GetResponseStream()) sReturn = srReader.ReadToEnd() srReader.Close() '------------------------------------------------------------------ ' capture the redirect location from the header '------------------------------------------------------------------ Try Dim wbHCol As WebHeaderCollection = m_hwrResponse.Headers Dim i As Integer For i = 0 To wbHCol.Count - 1 Dim header As String = wbHCol.GetKey(i) Dim values As String() = wbHCol.GetValues(header) If values.Length > 0 AndAlso header.ToLower = "location" Then Location1 &= values(0) End If Next Catch Location1 = String.Empty End Try '------------------------------------------------------------------ If Not m_hwrResponse.Headers("Set-Cookie") Is Nothing Then Dim ccContainer As New CookieContainer() ccContainer = New CookieContainer() ccContainer.SetCookies(m_hwrResponse.ResponseUri, m_hwrResponse.Headers("Set-Cookie")) sTemp = m_hwrResponse.Headers("Set-Cookie").ToString m_ccCookies.Add(ccContainer.GetCookies(m_hwrResponse.ResponseUri)) End If Me.Referer = m_hwrResponse.ResponseUri.AbsoluteUri 'close response connection m_hwrResponse.Close() Catch ex As Exception lblError.Text &= ex.Message.ToString End Try Return sReturn End Function #End Region
Does anyone have an example of how to do this? To get an idea go to http://www.chiefmfg.com/ and see Mount finder and select Projector.
Thanks to any info in advance.




Reply With Quote