[RESOLVED] Simple JSON table extract needed

[Sitten Spynne]

I need to come back here and look at Sitten's code. Now that I know how this works, I want to see how JSON.NET will make it simpler. What I have seems pretty simple to me so I can't imagine how it could be better.

This line of code from my post does everything your code in #31 does and more:

Dim parsedResults = JsonConvert.DeserializeObject(Of Example)(json)

It takes the entire JSON string, decides what objects to create, and property-by-property creates the objects the JSON represents. That's half the goal of szlamany's #34. (The other half is "output it as some HTML".)

I'm very, very frustrated because I don't know how to make it easier than "make one call and everything is done for you". It makes me feel like you didn't even look at the code, so I'm not even sure why I came back to demonstrate more.

Seriously, the thread came full circle. After properly demonstrating how to use a brick to drive nails, it was pointed out straight-faced that the brick would be nicer if it came with a handle.

[szlamany]

This line of code from my post does everything your code in #31 does and more:

To be clear though - your one line of code requires the object definitions - right?

Which means the format of the JSON is somewhat constrained by the format of the objects - is that also true?

If the JSON format is way off compared to the pre-created object formats does it fail completely??

[cory_jackson]

To be clear though - your one line of code requires the object definitions - right?

Which means the format of the JSON is somewhat constrained by the format of the objects - is that also true?

If the JSON format is way off compared to the pre-created object formats does it fail completely??

This is what I was thinking. But I don't know enough to ask in a educated manor.
I'm struggling with how objects exist without being defined. So this serializer creates all the objects. In the IDE I can expand them and see them all and all the data in every record. But they're not dimensioned in my code. So I imagine there are a bunch of bytes of data in memory with pointers but of unknown, and variable, form (class). Like I need the right grid to overlay (class) on top of them to make sense and use it. IOW the serializer creates all the bytes for all the objects within it's object. But it still seems we need objects in our code to reference them and do anything with them. I don't see how JSON.NET could create dimensioned objects in my code for unknown objects of unknown type.
This is the question in my head. There's so much I don't know or understand, so I need to try it and experiment with JSON.NET. I'm curious as to my misunderstanding of things that leads to what seems to be an impossible feat. I'm sure i'm just not seeing it correctly.
I do like the creation of objects for the objects in the JSON. Unwittingly I have done exactly that in this program. But I'm still declaring objects and variables to point to the objects and variables in the JSON object. And testing that they exist.

[Sitten Spynne]

To be clear though - your one line of code requires the object definitions - right?

Which means the format of the JSON is somewhat constrained by the format of the objects - is that also true?

JSON is JavaScript Object Notation. In JS an object is JSON, and JSON is an object. You can't change one without changing the other. So yes.

If the JSON format is way off compared to the pre-created object formats does it fail completely??

This is not a sensible question when the problem is framed properly, and your solution is just as vulnerable.

JSON is irrelevant here. The last point is true of XML, CSV files, or any serialization format: if your object model changes, the format changes, and vice versa. If there is a change, your parsing code has to adapt.

Good companies do NOT change APIs in ways that break their clients without a lot of warning. If we're writing code to consume a JSON API, we assume it's going to remain relatively constant, just like we assume if we're going to use the .NET Framework MS isn't going to break the classes very often.

So if you mean to imply, "Well, will your code parse ANY JSON, or just the JSON it's intended to parse?", I ask you to consider if the Dictionary approach is as flexible as you think. But first, let me answer how Newtonsoft handles that.

By default, Newtonsoft is lenient. If the JSON has a property my object doesn't have, it will be ignored. If my object has a property the JSON doesn't have, that property will be ignored. I can configure the Newtonsoft types to validate this and throw exceptions, but I don't have to. Generally if a JSON API changes, it adds properties, so "ignore unrecognized properties" is a safe behavior.

(If I want to be REALLY generic, I can use Newtonsoft to get a Dictionary, but their parser API is also public. It works sort of like XmlTextReader. Awkward, but powerful.)

This is also true of Dictionary code: if you write something generic that does something "for every property", you might start using new properties you didn't forsee. If you are more specific and use String keys, you might crash if a property is removed. There's NEVER a good reason to use integer-based indices with Dictionaries, but that's even worse than String keys because an add/remove makes us grab the wrong values.

If anything about the JSON changes, you have to revisit those assumptions. So do I.

But what I don't like about the Dictionary approach is the part AFTER you parse. There was a reason to read the data. You want to do stuff with it. if you want to do any kind of sensible querying, you need to convert that dictionary to an object. Go back and look at my code that makes queries. Here's a query to "print the address of all people with "Jackson" in their name:

Dim jacksons = From record In parsedResults.Records
                       Where record.OwnerName1.Contains("Jackson")
                       Select record

For Each jackson In Jacksons
    Console.WriteLine("{0} lives at:", jackson.OwnerName1)
    Dim address = jackson.OwnerAddress
    Console.WriteLine(address.Line1)
    Console.WriteLine(address.Line2)
    Console.WriteLine(address.Line3)
    Console.WriteLine("{0}, {1} {2}", address.City, address.State, address.Zip)

If I had a Dictionary instead of my object tree, I'd be inclined to write:

Dim jacksons = From record In parsedResults("Records").Values.Cast(Of Dictionary(Of String, Object))
                    Let ownerName1 = record.OwnerName1.ToString()
                    Where ownerName1.Contains("Jackson")
                    Select record

For Each jackson in jacksons.Values.Cast(Of Dictionary(Of String, Object))()
    Console.WriteLine("{0} lives at:", jackson("OwnerName1"))
    Dim address = CType(jackson.OwnerAddress, Dictionary(Of String, Object))
    Console.WriteLine(address("Line1"))
    Console.WriteLine(address("Line2"))
    Console.WriteLine(address("Line3"))
    Console.WriteLine("{0}, {1} {2}", address("City"), address("State"), address("Zip"))

The casts are a burden, and you have to think about them. You can get around it... if you declare an object model and parse the dictionaries into that object model. But that's "taping a handle to the brick": if you need to work with the data as objects it's best to use a serialization framework that does the conversion for you.

It only gets worse as the JSON gets more complex.

[cory_jackson]

I just had an epiphany. When I'm done working today, instead of learning JSON.NET, I'm going to take Sitten's example and try to do it with System.Web.Script.Serialization.JavaScriptSerializer. That should make it apparent how the JSON.NET is superior and take less time for me. As it is now, the arguments for what JSON.NET does sound like everything JavaScriptSerializer does. But this will highlight the differences. You can hold off comments until then if you prefer. I just need to spend a little time on this.

[Sitten Spynne]

This is what I was thinking. But I don't know enough to ask in a educated manor.
I'm struggling with how objects exist without being defined. So this serializer creates all the objects. In the IDE I can expand them and see them all and all the data in every record. But they're not dimensioned in my code. So I imagine there are a bunch of bytes of data in memory with pointers but of unknown, and variable, form (class). Like I need the right grid to overlay (class) on top of them to make sense and use it. IOW the serializer creates all the bytes for all the objects within it's object. But it still seems we need objects in our code to reference them and do anything with them. I don't see how JSON.NET could create dimensioned objects in my code for unknown objects of unknown type.
This is the question in my head. There's so much I don't know or understand, so I need to try it and experiment with JSON.NET. I'm curious as to my misunderstanding of things that leads to what seems to be an impossible feat. I'm sure i'm just not seeing it correctly.
I do like the creation of objects for the objects in the JSON. Unwittingly I have done exactly that in this program. But I'm still declaring objects and variables to point to the objects and variables in the JSON object. And testing that they exist.

Let's make a really simple example. Part of the problem is trying to use a complex example.

Let's say we have just this JSON:

{
    "firstName": "Bob"
    "lastName": "Thebuilder"
}

If you use Dictionary-based code, you'll get an object graph that looks something like this:

Dictionary
    { Key: "firstName", Value: "Bob" },
    { Key: "lastName", Value: "Thebuilder" }

That makes sense. The deserializer made a dictionary with a key/value pair for every name/value pair. The value ends up being an Object in every case, so to work with it as Strings you need to call .ToString() on it. Every time.

So you might decide you want to get rid of that burden and convert the dictionary into an object.

Public Class Person
    Public Property FirstName As String
    Public Property LastName As String
End Class

Dim parsed = ' <code to parse the JSON to a Dictionary(Of String, Object)

Dim thePerson As New Person()
thePerson.FirstName = parsed("firstName")
thePerson.LastName = parsed("lastName")

Now you have an object.

That is exactly what Newtonsoft does, but it skips the Dictionary:

Dim parsed = JsonConvert.DeserializeObject(Of Person)(json)

If we were to imagine the pseudocode of DeserializeObject(), it would look like:

Dim dictionary = <convert the JSON to a dictionary>

Dim newObject = New <whatever object>()

For Each <name/value pair> In dictionary
    If newObject has a property with this name:
        Try to convert the value to the right type and if it succeeds:
            Set that property to this value.
Next

Return newObject

So it's doing the work you'd have to do yourself anyway. There's no magic to it, really. You look at the JSON and think about what objects would convert to/from it, then create the Class definitions. When you tell it to deserialize, it does its best to match properties for you and create the objects for you.

You never have "unknown object" of "unknown type". You have JSON that you believe is a Person, and you tell Newtonsoft to please create a Person based on that JSON. If you give it the wrong JSON, you either get an exception or a malformed Person back, depending on your configuration. This is true of any parser: if I try to force Word to open a file it expects me to give it a Word file. If I rename a DOOM save file "sav1.docx" and feed it to Word I'm not going to get a sensible Word document. So, too, if you tell a deserializer "Here's a Person" and you give it the JSON for a Twitter Tweet, you won't get a Person. You'll get a mess.

This is identical to what you did with the Dictionary. You didn't create the Dictionary yourself. You asked a serializer to create it, then populate it. In fact, that's an alternative way to go about creating objects with Newtonsoft:

Dim p As New Person()
JsonConvert.PopulateObject(p, json)

That code says, "Hey, this JSON has at least some of the properties that match this object. Please try to make the matches for me.". You can do some interesting things with it, but I think it's more elegant to use DeserializeObject().

[szlamany]

This is not a sensible question when the problem is framed properly, and your solution is just as vulnerable.

The whole point of my question was to basically level the comparison.

You had a comparison that you stated that one line of code was equivalent to the other pile of code.

But you have more than one line of code. You make CLASS's to type the data - and your method takes the JSON string and pushes the data into those classes doing all the CAST'ing just once under the hood.

A manual attempt does all those exact same things - just not in the same order - and certainly not as efficiently (as needing to repeat .ToString() all over the place for example).

With that said - your method makes great sense for a large complex JSON, imo.

For something simple - which I believe is what the OP has - the dictionary manual method is sufficient, again imo.

I don't have a dog in this show - I never deserialize anything myself. And I personally write business-logic agnostic applications - so the ability to pre-define classes is not really an option.

I'm coding web methods the JSON deserializing is being done by the IIS process and simply served to me as parameters...

    <WebMethod()> _
    <ScriptMethod(ResponseFormat:=ResponseFormat.Json, UseHttpGet:=False)> _
    Public Function SavePopupService(ByVal toddtype As String
                                     , ByVal fromddtype As String
                                     , ByVal fromwho As String _
                                    , ByVal parentfromddtype As String _
                                    , ByVal parentfromwho As String _
                                    , ByVal popupkey As String _
                                    , ByVal mode As String _
                                    , ByVal sguid As String _
                                    , ByVal username As String _
                                    , ByVal source As IList(Of Dictionary(Of String, String)) _
                                    , ByVal objReturn As Dictionary(Of String, String) _
                                    ) As String

JSON coming in on the AJAX call looks like:

{"toddtype": "asdf", "fromddtype": "dfsdf"..., "source": [{"xxx": "vvv"}..., "objReturn": {"aaa": "bbb"...

In a way I have some pre-declared aspects - strongly typed and always expected - and some extensible stuff...

[Sitten Spynne]

Yes, when you frame the problem that way I don't really have a reason to beat on it other than "continuing the argument".

All of my problems tend to grow in complexity after I release them. It starts with a simple, "I need to know if this feature is enabled for this user" and within a month becomes, "We need to use this as the default way to get all user settings, and the customer needs to be able to use role-based configuration to set them." So I tend to avoid "works for simple problems" solutions like the plague because every time I use them, I have to rip them out. It makes me kind of irrationally fearful of those kinds of solutions.

But you painted a picture where I would agree you can tell me exactly where in your project there are bigger fish to fry.

One thing that confounds the heck out of me in Newtonsoft:

There's this weirdo type called ExpandoObject in .NET. You can only use it with "dynamic" variables in C#, but in VB it's sufficient to be in a file with Option Strict Off. It is a bonkers, JavaScript-like object that's effectively a Dictionary but uses magic to treat its keys like properties. So this would work:

Dim bonkers As New ExpandoObject()
bonkers.Name = "George Washington" ' Creates a String property
bonkers.Id = 10 ' Creates an Integer property
bonkers.MainForm = New Form() ' Sure, have a Form property.
...

There are some types in Newtonsoft's library that can deserialize JSON to ExpandoObject, and THAT would be incredible for very complex types. But the only time I've ever tried to use it, I encountered error after error and gave up. I even delved into their source code to try and find the issue, and all I could figure was I'd found a serious bug/omission.

[szlamany]

ExpandoObjects appear to be the more complex items you can put into a JSON string - such as a function itself.

Check out this JS code - it returns a JSON object with FUNCTION's as the values in the K/V pairs...

var serial_maker = function() {
    var prefix = '';
    var seq = 0;
    return {
        set_prefix: function(p) {
            prefix = String(p);
        },
        set_seq: function(s) {
            seq = s;
        },
        gensym: function() {
            var result = prefix + seq;
            seq += 1;
            return result;
        }
    };
};

Would Newtonsoft make an ExpandoObject out of that?

[Sitten Spynne]

I don't know the answer to that question because I don't think I've ever found a REST API that returned functions.

The official JSON spec doesn't allow functions as a type. "object" is a set of string/value pairs, and "value" can be string, number, array, object, true, false, or null. In this case keep in mind "object' is just another "set of string/value pairs" as defined by the JSON grammar.

So if the MS parser supports functions, that's an extension they decided to support themselves. I don't know if Newtonsoft also supports it. They might say "we support as close to the JSON spec as we can and thus don't support this". They might say "We support it, but support it a different way".

Thing is I lied a little when I said JSON is a JS object. The code you posted returns "a JavaScript object literal" which is converted to "a JavaScript object" which isn't quite the same thing as "a JSON string". I'm curious what happens if you pass that object to JSON.Stringify(). It looks like Node.js doesn't serialize functions:

> var json = {
... "test": function() { console.log('hi') },
... "test2": 10
... }
undefined
> JSON.stringify(json)
'{"test2":10}'

I got similar results in the Safari console:

> var json = { "test": function() { console.log('hi') }, "test2": 10 }
< undefined
> JSON.stringify(json)
< "{\"test2\":10}"

So if you needed to serialize/deserialize functions, you're in a special case where you definitely need the MS types since they support that notion. It seems incredibly dangerous to do so to me, though, as it's very difficult to verify user-given functions are safe to execute. What happens if I pass this object using a malicious script?

{
    set_prefix: function(p) {
        while(true) { }
    },
    ...
}

[szlamany]

The official JSON spec doesn't allow functions as a type. "object" is a set of string/value pairs, and "value" can be string, number, array, object, true, false, or null

Which makes sense since JSON is just a subset of the JavaScript language itself.

Maybe that is why I kind of see them as one and the same...

JS quote: "In JavaScript, functions are first-class objects, because they can have properties and methods just like any other object."

[Sitten Spynne]

It's pretty confusing in that the code uses the same syntax as the serialization format, so it's not clear that the code can do more things than the serialization format. This is kind of typical of JS. There's at least two different ways to define a class, and I think there's 3 in reality. But now we're in the weeds

But VB and XML are kind of the same way these days. In code you can write XML literals, which are allowed to use VB For loops and many other programming constructs to build up a full document. But when you serialize a type with XmlSerializer, you only get the properties, and if you try to put VB's XML literal syntax in an XML file it won't be valid XML.

[cory_jackson]

For the benefit of others learning and belated thanks to Sitten Spynne.
I had another project requiring a more complicated JSON package I needed to deserialize and the method I used before was creating a lot of code and confusion. I had more time so I read up more on JSON's format and the System.Web.Script.Serialization.JavaScriptSerializer on MS. using dictionaries and ArrayLists works too, but Sitten Spynne's approach is the correct way and is elegant. I want to restate here how it works.
The JSON package you have is structured just like a custom object in VB.NET but using a small set of variables, objects, and arrays. The deserializer is designed to pump the the whole shebang into one object with properties matching the key names in the JSON. Make public properties like in Sitten Spynne's example. Things like text, integer, are obvious to make properties for. There will be some that have an object too in curly braces. In those cases, create an object that has public properties again matching the key names. Finally some will be arrays in the JSON. WHen you get one of those, it is often an array of objects. Again, make objects with properties that match the key names, but in the parent object, create a generic "List of" your custom class. In my case I have created an object that contains a hierarchy of properties that are variables, arrays, and classes that is the same structure as the JSON package. When you de-serialize, it populated your object all the way down flushing out all the arrays, objects, and such.
One extra thing I learned was that if there is information you don't need in the JSON, omit that property. It doesn't complain if you are missing some and only flushes out properties you have defined.