[RESOLVED] Problem downloading JSon as string

[.paul.]

I have an app, that subscribes to an API in the form of a JSon file. Recently the format of this JSon file has changed, and now i can't even download it...

Dim client As New WebClient
Dim rawResponseString As String = client.DownloadString("https://www.gov.uk/bank-holidays.json")

Can anyone tell me what the problem is?

Thanks

[peterst]

User agent requirements?

[.paul.]

User agent requirements?

I don’t see that being the problem. It’s a public api. I have permission to use it, but since the format change, nothing but problems

[peterst]

It works with browser. It works with cURL. Just try adding user agent in header. Many services were updated last months with that requirement.

[.paul.]

I don’t specify a header. How would I do that with WebClient.DownloadString?

[.paul.]

Dim client As New WebClient
client.Headers.Add("User agent requirements");
Dim rawResponseString As String = client.DownloadString("https://www.gov.uk/bank-holidays.json")

I haven’t tried it yet, but I’m fairly sure this isn’t how to do it…

[peterst]

I am on phone so can't test, but you can read here about it: https://developer.mozilla.org/en-US/...ers/User-Agent

Use the proper strings. No need to copy Mozilla identifier, as you can use anything you want.

[.paul.]

I am on phone so can't test, but you can read here about it: https://developer.mozilla.org/en-US/...ers/User-Agent

Use the proper strings. No need to copy Mozilla identifier, as you can use anything you want.

I googled it. Think I’ve found what I need… Thanks

[.paul.]

Ok. I tried...

Dim client As New WebClient
client.Headers.Add(HttpRequestHeader.UserAgent, "ASP_UK_Holidays.")
Dim rawResponseString As String = client.DownloadString("https://www.gov.uk/bank-holidays.json")

Getting error...

System.Net.WebException: 'The underlying connection was closed: An unexpected error occurred on a send.'

IOException: Received an unexpected EOF or 0 bytes from the transport stream.

[peterst]

This is the proper way to set the user agent string:

Dim http = New WebClient()
http.Headers.Add("User-Agent", "ASP_UK_Holidays.")
Dim json As String = http.DownloadString("https://www.gov.uk/bank-holidays.json")
Console.WriteLine(json)

And the json is returned without errors.

[.paul.]

This is the proper way to set the user agent string:

Dim http = New WebClient()
http.Headers.Add("User-Agent", "ASP_UK_Holidays.")
Dim json As String = http.DownloadString("https://www.gov.uk/bank-holidays.json")
Console.WriteLine(json)

And the json is returned without errors.

It still breaks on the DownloadString line. Same error

[peterst]

It seems you are banned for some reason. Change your IP and try again.

[.paul.]

Turns out i wasn't banned. I changed my app. framework to 4.5.2 and established a secure connection...

ServicePointManager.Expect100Continue = True
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12

[peterst]

Well, going this way, I think it is better to target .NET Framework 4.8 as latest, updated and most compatible with latest networking trends. Also it is supported on Win 7+ (well, 4.5.2 works on Vista+).

Or if it is a new project - directly target .NET 6 (where WebClient is obsolete as there is HttpClient...).

[.paul.]

I got it working on an alternative workstation running Win7 pro, running VS2017 (hence 4.7.2). It needs Tls13. That’s the all important part that got it working…

[peterst]

There was another thread about GitHub API, where minimum TLS version is required, setting of User-Agent and accepted content type (Accept header).

Many public services were changed to stop endless outdated bots - first by closing the connection via TLS requirement. Then if the user agent is missing (outdated bots or lazy authors) - reject. Some other specific header values missing? Reject. This requires devs to check regularly for API changes and apply to their tools to be compliant.