JSON Recursion – Designing a Self-Describing Document System

jsonrecursion

I'm trying to devise a document system for a specific domain involving simple objects (People, Companies, Invoices, etc.) that would also be able to completely describe itself. This self-description ability would ideally be recursive. The document system will be based on JSON, but I believe the principle in question applies to any structured notation.

To illustrate what I mean, let's say I have the following JSON document that describes a person (company, invoice, etc. are basically all the same, only with different properties):

{
  "Name": "John Doe",
  "Email": "john.doe@example.org",
  "BirthDate": "1976-07-04"
}

Then, I have a second JSON document that describes the structure of the first one. Let's call this a "Level 1 Meta-document":

{
  "Type": "Person",
  "Properties": {
    "Name": { "Type": "String", "Required": true },
    "Email": { "Type": "String" },
    "BirthDate": { "Type": "Date" }
  }
}

This would be simple enough, if not for the requirement, that system should also be able to fully describe this meta-document.

To put it in more general terms: I'm looking for a way to define a self-sufficient "Level N Meta-document", that would be able to describe a structure of the "Level N-1 Meta-documents".

NOTE: I'd be willing to go with a solution for N = 2, but instinct tells me that a true solution for N = 2 would also work for any N. Now that I think of it, this may be more of a math puzzle than programming one. 🙂

Is this even possible? If yes, can you give me some examples? If not, what are my other options?

EDIT: I've included a naive example of how a "Level 2 Meta-Document" would look, based on the above:

{
  "Type": "MetaLevel2",
  "Properties": {
    "Properties": { "Type": "Hash", "Required": true }
  }
}

The problem with this is that it doesn't describe the object that describes the property details (i.e. the one with the "Type" and "Required" attributes).

If I were to include description of those, I'd have to add another attribute to the very same object I'm trying to describe:

{
  "Type": "MetaLevel2",
  "Properties": {
    "Properties": {
      "Type": "Hash",
      "Required": true,
      "ValueProperties": { "Type": "String", "Required": "Boolean" }
    }
  }
}

Unfortunately, this throws me into recursive problem, because I now lack the description for "ValueProperties". In fact, for every new attribute I invent on level N, I have a problem describing it on level N+1 without introducing yet another attribute that needs description.

What I'm looking for is a solution that wouldn't suffer from this problem.

To be clear: I'm aware of XSD, but I'm not sure how to apply its principles to my case. Unless I'm missing something, XSD would suffer from the same recursive problem. This gives me reason to believe I have a problem with the approach itself.

Best Answer

There's IETF draft for JSON Schema. That sounds like what you want to use. Also check it out at Wikipedia.

Related Solutions

C# DataContracts – Using Subclasses for DataContracts with JSON

Did you author the service which you are posting to? If so, I would recommend changing it, so that you post SetState data to something like /api/sample/State, and post SetProperty data to /api/sample/Property. Then you would have separate controller actions for each service method, with each receiving only the desired parameters. If the service isn't yours and you can't modify it however, then obviously you're a bit stuffed. Otherwise, read this: http://www.asp.net/web-api/overview/web-api-routing-and-actions/routing-and-action-selection

Update

If you definitely want to retain a single action, at the same address, there is something you can do. You can define your controller action to take an interface / base class-typed parameter, and have the json deserializer use type information which you provide in your json to determine which class to rehydrate with your data, for example:

public abstract class ThingBase
{
    public int instanceId { get ; set; }
    public int actionId { get; set; }
}

public class SetStateThing : ThingBase
{
    public string desiredState { get; set; }
}

public class SetPropertyThing : ThingBase
{
    public string propName { get; set; }
    public string propValue { get; set; }
}

public class MyController
{
    public HttpResponseMessage Post([FromBody]ThingBase thing)
    {
        //use thing
    }
}

Your json would become, for example:

{
$type: "MyAssembly.SetStateThing, MyAssembly",
instanceId: 12312,      
actionId: 12345, // Maps to a SetState action
desiredState: "CURATION"
}

{
$type: "MyAssembly.SetPropertyThing, MyAssembly",
instanceId: 12312,      
actionId: 25354, // Maps to a SetProperty action
propName: "PropA",
propValue: "user-input" 
}

The trick is to set up the serializer settings correctly (in your WebApiConfig.cs Register(HttpConfiguration config) method for example):

config.Formatters.JsonFormatter.SerializerSettings.TypeNameHandling = TypeNameHandling.Objects;

I used this with an interface, but an abstract base class seems a better fit for your scenario.

Inserting JSON document with `.` in key to MongoDB

There are a few alternatives:

1. Replace dots by a dash.

This would be my favorite approach, as it keeps the structure explicit enough.

Since according to you, “it is pretty much a one time insertion,” it should be relatively simple to check if it doesn't break anything (i.e. there is already a same key with a dash). For other situations, doing those checks programmatically requires to write some code, but is still a relatively easy task.

2. Replace dots by a Unicode dot character such as U+FF0E.

I would strongly advise against this approach, since it would lead to massive debugging headaches down the road. Letting someone who uses the resulting JSON somewhere in the code far away from MongoDB to guess that a dot is not really a dot is a good way to waste literally weeks of someone's time. Keep such Unicode tricks to the hackers who want to trick someone into thinking that a character is a different one.

3. Use BSON.

Since you claim that you “would mostly retrieve the entire document rather than querying for parts of it,” this approach doesn't have major drawbacks in your case. Although, you said “mostly,” which means that sometimes, you will retrieve only parts of the document.

In general, the drawback is that you won't be able to search through the document or to load only a part of it.

4. Use a standard encoding, such as Base64.

Converting the problematic keys (or all keys, depending on the ratio between problematic and non-problematic ones) to Base64 or hexadecimal could be a viable solution, with the benefit of being rather explicit: most developers would recognize Base64 or hexadecimal values at a glance.

The drawback is the increased memory footprint, as well as the necessity to encode and decode the keys when using them.

5. Set `check_keys` to `false`.

I would strongly advise against this approach, since is would make the data query ambiguous, and waste hours or days trying to figure out why a specific query doesn't do what you imagined it should be doing. Dot is a reserved character and the check is here to protect you; by telling MongoDB to skip the check, you will only postpone the moment where you'll have to deal with a conflict between the syntax of MongoDB and the reserved character used in a key.