A format for Graph Streaming

[mbastian]

Good point, thanks

We were discussing today with Andre and decided we will start with a JSON prototype, as our primary objective is portability. However as it was said the serialization technology is not the real problem, and other library like protobuf could be used and compatible, as far as the Java Objects which are serialized don't change.

So we have to find a serialization library able to serialize in JSON, some pointers?

[tcc]

have you looked at BSON, http://bsonspec.org/, it's a binary version of JSON.
It's used as the format for mongodb and there are existing java, ruby, c++ libraries for de/serializing

[panisson]

Hi all!

As Mathieu already pointed, in the last days I started with a simple JSON prototype. For the java serialization I'm using the implementation available at json.org (http://www.json.org/java/index.html).

As I said, this is a preliminary implementation, and is completely open to changes. It is just to give us some idea of how the events would appear in the JSON format. The description of the current implementation is available at http://wiki.gephi.org/index.php/Specifi ... ing_Format. I listed also some open problems and some ways to face it, and I would like to have your opinions about it:
- how to add support to filters
- how to support identifiers to events
- and other improvements (array-type attributes, composite attributes, graph attributes, etc. etc...)
I'll wait for your suggestions.

[eduramiba]

Hi!

For the event identifiers, I don't know if it is possible, but you could use the normal format and, when the events have an identifier, then they must be in the same HTTP request, just before the event. Maybe something like

Code: Select all

{"event-id":"xyz"}
{"an":{"A":{"label":"Streaming Node A","size":2}}}

And as I see a format for complex events as applying changes using filters can be really difficult while not providing so much traffic or performance difference. So I think you could create special events for the most common of these events that delete multiple nodes based on their id range, for example. But for other multiple events, since I guess normally the JSON is going to be written by another program, sending a lot of events is no problem.

[panisson]

For event identifiers you are right, I would just put it in the same JSON object:

Code: Select all

{
    "id": "xyz",
    "an": {
        "A": {
            "label": "Streaming Node A",
            "size": 2
        }
    }
}

In this way the event identifier cannot be confused with node/edge identifiers, and the absence of the "id" attribute should not be a problem when identifiers are not required.

For filters it is a bit more difficult, and I think you are right in the fact we could create special predefined filters for the most common filtering operations. As an example, the following Json object should change all nodes with id 1-100 to a size 3:

Code: Select all

{
    "id": "xyz",
    "cn": {
        "filter": {
            "range-id": {
                "start-id": 1,
                "end-id": 100 
            } 
        },
        "attributes": {
            "size": 3 
        } 
    } 
}

The problem with this format is that someone could confuse "filter" and "attributes" with object identifiers, and try to change nodes with these ids.
Other suggestions?