Porting of Gython

[admin]

This is the thread for asking more details about the Porting of Gython proposal.

[luizribeiro]

Do you think we could implement this proposal as a MuTanT console as described on my proposal idea?

Since JSR 223 is probably going to be supported by Jython sometime soon (AFAIK currently it seems to experience some crashes), we could implement this proposal as a MuTanT console and simply import the GUESS modules on the Jython ScriptEngine (once they get ported to the latest Jython version).

The good thing about this is that Gephi's console wouldn't be tethered to Jython, since we could use virtually any ScriptEngine that supports JSR 223 (Groovy, Gremlin, etc).

Are there any specific reasons for using just Gython? It looks like GUESS' website hasn't been updated in a while and the latest beta version dates from 2007.

Edit: sorry if I'm wrong about something related to GUESS, I've just downloaded it's source code for the first time.

[mbastian]

Yes this is a very good point. Thanks to you, I just learnt about MuTanT and it seems very powerful and more generic than what we are proposing.

The reason we are looking at GUESS is because the scripting language is neat and easy to use. Moreover, we will have the fantastic guidance of it's author, Eytan Adar, who will be co-mentoring this project

The genericity to other languages than Python is a point we considered, but we didn't found any generic solution so far. Using this stack seems to solve this problem! However, we need to know more about possible isuses, is everything we want to do with Gython possible? And we need Blueprints for that, so it has to be included in the project. If we could achieve Blueprints, and a generic ScriptEngine during this summer, that would be fantastic!

[eytanadar]

MuTanT looks cool and it would be nice to switch between languages (I had a similar feature in GUESS that let you switch between Gython and R). I don't see any reason why Gython couldn't work under MuTanT.

My experience with GUESS has been that people prefer a concise language for working with the graph structures. Some of it is syntactic-sugar, but because many of the users don't have a strong programming background the overhead of learning a general language is daunting. Because Gython is an extended domain-specific-language it gives you a lot of shorthand for graph structures (while maintaining backward compatibility with Jython/Python).

The other feature of the GUESS console, which I haven't seen elsewhere is the ability to brush and click (with the mouse) on text in the console to control what you see on the screen. This mixed-mode interaction is efficient for interactive exploration tasks. You can take a look at the video to see this feature (or download GUESS).

[luizribeiro]

First of all, thanks for the great replies. It's really nice to have such feedback while writing my proposal.

After having a closer look at GUESS and the Blueprints API over the last couple of days, I have came up with some new observations that I'll try to expose on this post while commenting what you have said.

I don't see any reason why Gython couldn't work under MuTanT.

If we're really going to implement a MuTanT console + Gython, the student working on this would have to (aside from other more obvious tasks like working on the console's UI, etc):

1. Create an implementation for Gephi on Blueprints;
2. Upgrade Gython so that it's compatible with the latest Jython stable version (2.5.2), since it's the only version that seems to have decent JSR 223 support. Although, I'm not sure if MuTanT devs are going to re-enable Jython support anytime soon (they've removed it because they were experiencing crashes);
3. Refactor Gython's source code so that it uses the Blueprints API to access the graph structure, not via the com.hp.hpl.guess.Graph object.

The first item probably won't demand too much work and could be done in a couple of weeks, since it is just the work of creating implementations of the Graph, Edge and Vertex interfaces from Blueprints that should access the graph structure from Gephi. Also, since Blueprints API treats all edges as directed edges, a workaround would be necessary for this issue, so that we can create undirected edges via console (which shouldn't be hard at all). At the stage after the completion of this task, I believe the console plugin would be already usable by languages like Groovy, JRuby and Gremlin.

The second and third items are the ones that seem to demand much more work IMHO. After looking at GUESS' source code from the latest CVS revision, I still have no idea where to start the second item. It seems Jython 2.1.0 is unpacked inside GUESS' source tree, right? Also, I see that the Guess class exports the display, graph and other objects to the PythonInterpreter. But I still don't get how the queries like (dept == 'HR') are implemented. Could you give me some guidance with respect to this? I'm not yet familiar with how Jython extensions work at all and I'm trying to get to know GUESS' source code better, so that I'm familiar enough with it to know which classes and interfaces have to be exported to Gephi.

Also, I'm not sure if there's enough time during GSoC to work on all these mentioned tasks.

My experience with GUESS has been that people prefer a concise language for working with the graph structures.

After using GUESS, I have to agree with you. Probably most users would prefer if the console language was less "programming-oriented". Also, since Gython is just a Jython extension, we still have all the power of Jython on the console, which is more than enough in my opinion.

With this in mind, I'm not sure if the genericity to other languages via MuTanT justifies the amount of work of upgrading and refactoring Gython's source code (and the risk of not being able to finish it during the project's timeline). I'll be submitting the first version of my GSoC proposal next week and I'm still unsure if we should give MuTanT a shot or not. What do you think?

[taynaud]

Hello,

Sorry to appear like that in the thread, I do not read gephi forum very regularly. I am currently writing a similar python plugin on my spare time and the main difficulty is clearly not the language of the console but the api.

Writting a console using one JSR 223 language is done(I use now jython), and after that changing the language is trivial. In my code, that I plan to release in a few days, changing the language is actually changing three variables in the java code.

The gephi api to transform the graph, apply layout and other stuff is a pain for a non java programmer. Model view controller is great for the full application, but in a script environnement it is just awfull. Every action requires at least three lines of complicated code, and netbeans is not behind . Thus, I advise to focus first on one language, with one nice api, and gython api is excellent. Mutant can come after. I have looked to mutant that I did not knew. It seems to implement the jsr 223, so porting to mutant is very easy. But changing the exported api to script is more difficult and I think the student should focus on this. Another issue is that mutant documentation seems very limited.

I have not experienced difficulty with jython so far, do you have precise example ?

[luizribeiro]

Writting a console using one JSR 223 language is done(I use now jython), and after that changing the language is trivial. In my code, that I plan to release in a few days, changing the language is actually changing three variables in the java code.

This is good news! I've been also already experimenting on creating a prototype of a console UI for Gephi using Jython, to see how hard it would be to integrate Gython into it. I'm really looking forward to your release, I'm sure it will be very helpful.

It seems to implement the jsr 223, so porting to mutant is very easy. But changing the exported api to script is more difficult and I think the student should focus on this. Another issue is that mutant documentation seems very limited.

Actually, the problem isn't porting to MuTanT. The main problem with using ScriptEngines is that Gython seems to be currently using Jython 2.1.0, which lacks ScriptEngine support. If we're going to use a generic ScriptEngine, we first need to upgrade Gython to the latest Jython stable release.

I have not experienced difficulty with jython so far, do you have precise example ?

I haven't experienced any problems with Jython 2.5 actually. When I say that Jython's JSR 223 support sems to have problems, I'm referring to this commit on MuTanT's git repository. The authors opted to disable Jython support because of it's "poor implementation of ScriptEngine" (although I have no idea on what kind of problems they've been experiencing).

[eytanadar]

Thanks for the continued interest in this.

I guess I should have specified: I see no _theoretical_ reason why Gython could be integrated with MuTanT...

In answer to how Gython/GUESS are implemented:

1) As far as I know, there is no way to "extend" Jython by writing a module or something like that. Gython extends Jython by modifying the core language/implementation. So while it is backward compatible, it is still a branch from the Jython tree (which is why it is included with GUESS and also why it hasn't been upgraded to a newer version of Jython). Now, there might be a way to do this through patching so that it is also forward compatible with new releases, but I'm not sure how stable this is. I also don't know of any modern language that supports this style of extension (Lisp/ML probably don't count ), but if there is, I'd be happy to consider extending _those_ languages instead.

2) Operators are added to Jython by modifying the JavaCC spec files. There aren't that many new operators but we do have a few that are useful (e.g. -> <-> <- 'like' and some range operators)

3) Jython "set" objects (e.g., PySequence) were also modified to support things like (dept == 'HR').color = red by iterating over all the objects in the set and "setting" things or "getting" (as appropriate). This part always felt like a hack to me, but it worked. (happy to consider alternatives)

4) The query functionality is implemented through the Field and Query objects and operator overloading. So "dept" is actually an instance of a Field object and when you do dept == 'HR', the == operator invokes .isEqual which is overloaded. So dept == 'HR' is actually dept.isEqual('HR'). Similarly, freq > 50 would be freq.gt(50) (or something similar). The trick is instead of returning a boolean we return a ComparisonQuery object which is basically a set. Query objects are also overloaded so that Query1 & Query2 returns a CompoundQuery. When you go to access whatever is in the Query object it will _then_ execute the query as appropriate. In the case of GUESS this means walking through the Query tree and transforming things into a text string suitable for SQL and then running it through the DBServer. In the case of Gephi, this might mean transforming the Query into the appropriate compound blueprint calls.

5) When running GUESS, the system pushes a bunch of variables into the interpreter namespace to make sure various objects are available. So all columns in the database get their own Field objects, nodes get added in, etc. Same thing happens when you create new objects. The interpreter was also extended a little bit (includes a setImmutable method) to support variables that can't be overwritten (we didn't want users overwriting the "g" object by mistake).

6) The console supports the mouseover stuff by keeping an interval tree that keeps track of where certain things were output (to the console). This is a little bit of a hack, but works mostly ok. I'd be happy to consider alternatives, but this was done based on the particular console implementation I used.

7) There's some other junk added to the Jython core that makes it so that GUESS can run in the browser as an applet with the interpreter (safely). This wasn't supported by Jython (and may still not be). This part isn't crucial. It's nice to be able to run things on the Web, but I don't know how often GUESS actually gets used in this mode (people have a hard time compiling or getting certificates or whatever).

Anyway, I can get into more specific details about where to look for/at things but maybe this is enough to get you thinking about it.

[luizribeiro]

Thank you very much for all the valuable information and the time spent on this, Eytan. Your post has clarified a lot my vision of how Gython works.

As a side-note, about the alternative for the third item you wrote (the hack for iterating over the set of edges/vertices, setting the specified attribute), I see a way to do this: you can create a container class in Python (something like EdgesCollection) and just overload the __getattr__, __setattr__ and __delattr__ methods, so that they iterate over all elements in the set, setting the corresponding attribute (or getting a set of unique values for the attribute, in the case of __getattr__). This way, the queries would return an EdgesCollection object. Although, this is almost the same solution as yours, but it needs no modifications to Jython's core (which is good, in my opinion).

Sadly, after reading your post I see that upgrading Gython is even harder than I thought. I don't believe there's enough time for this during GSoC's three months timeline. However, porting the current source code of Gython to Gephi is completely viable.

Is there a real need for having backwards Gython-syntax portability? I mean, this might be a silly idea, but I'm pretty sure we could come up with a somewhat similar API that doesn't need modifications to Jython's core (and the whole documentation for it, of course) during a summer. In my mind, this API could do things like "(dept == 'HR').color = blue" in a way like this: "getVertices(dept = 'HR').color = blue". We could use the "hack" I mentioned above to achieve this (that's how I came up with this idea).

The upside of this is that we could have Jython 2.5.2 compatibility during the summer, which means ScriptEngines support.

I completely understand if you're not interested in modifying the API or having a different syntax from Gython's, since GUESS already has a established user-base that probably isn't interested in learning new commands. And please, don't get me wrong: I have no problems on porting Gython's current source code to Gephi, I'm just brainstorming here.

[eytanadar]

Personally, I would be totally happy to do the port over a longer period of time. So using the GSOC'11 time-frame to get some initial porting done (maybe as you describe) and maybe start doing some re-architecting and better documentation as we go that could then lead to the upgrade at a later point.