Statistics Reports and HTML5 Charts

[mbastian]

This is the thread for asking more details about the Statistics Reports and HTML5 Charts proposal.

http://wiki.gephi.org/index.php/Google_ ... ML5_Charts

[vikash23]

Hello Everybody,

For the past two days I have been trying to explore the features of Gephi and its working. I have also gone through the problem statement of the project. As of now, I have understood that the projects demands creation of dynamic charts for displaying reports generated by running various statistics algorithms as it would provide more information and will be easier for user to visualize and study the charts. Further a Report API needs to be created to implement the features mentioned in the idea.

I wanted to know that which parts of code do i need to refer in order to have a proper understanding of the report generation process in Gephi. Please do provide me the link if any proper online documentation is available.

[luizribeiro]

Hi Vikash,

Thanks for you interest on this! It's good that you're already exploring Gephi. Be sure to fork/clone the gephi repository to get used to our development environment. Also, don't forget to follow the steps and answer the questions from the questionnaire.

Yes, you are correct. The student that is going to work on this project will have to design a Report API.

At the moment, the reports are generated in a very simple way:

The dialog that shows the HTML reports is on the UIComponents module, on the SimpleHTMLReport class. The plan is replacing this simple HTML report viewer with something that supports HTML5 and JavaScript, so we can have more dynamic reports (instead of the current JFreeChart static images). The reference for getting WebKit to work under the NetBeans Platform is on Geertjan's Blog.

Metrics and statistics are implemented by classes that implement the Statistics interface. One such example is the PageRank implementation. At the moment, all the reports are generated in HTML-encoded Strings by the getReport methods of these classes.

This tutorial should be an interesting read too, as it gives a practical example of designing a new feature for Gephi.

You can find more information about the Statistics API here.

This should help you to work on your proposal. Let me know if you have any questions!

[vikash23]

Hi Luiz,

Thanks a lot for you reply. I have been going through the links you posted above.
I have already build Gephi from source code using netbeans. I have set up an SSH key for my account and forked the repository on Github.
I plan to send the answers to questionnaire once a basic draft for my proposal is ready.

I have been working on a basic layout of the proposal and have got some problem which I would like to discuss with you.

1. First of all we need to have an embedded browser for Gephi.
Inorder to implement this i have decided to go with javafx web browser for netbeans as mentioned in the Geertjan's Blog. I think its a great idea to have it in Gephi so that users can view various web components. Moreover it will be easy to implement it in Gephi as it is built on top of netbeans.

2. Next up is the creation of Report Api.
As, mentioned on the ideas page the api should allow building of reports with HTML Templates. I am not very sure about these templates i.e Which ones to be used for the report creation and their design?
I have also gone through the visualization libraries mentioned on ideas page i.e D3 and RaphaelJS. Usage of these libraries for reports will surely improve the quality and user experience as they are very interactive.

I am still not clear about the implementation of reports generated by various statistics.As you told the reports are sent as HTML string by the getReport() methods of these classes. I wanted to know that once we shift to interactive reports based on HTML5 and javascript, how will these getReport() methods be affected.
I think we have two options. which are as follow:
We can use the datasets of a statistics to generate the reports and charts using visualization libraries and return them to be displayed in the browser.
The second option we have is that, we can return the datasets of each statistics. Using these datasets we can generate charts and reports in some other class, that can be displayed in browser.

I am also not sure about the design of the Report api. Do you want the creation of sample templates for table, charts and insights that will allow users to generate these components by simply entering the required values ?
I will be glad if you could provide me a basic idea of Report api.

[LeoLin]

Hi,

I am another student who also have interests on this topic.

In my understanding, the report api should return no more a static html string, but a data structure that HTML page can read, I think JSON object may be a good idea. But I am not sure how to build a bridge between javascript and java in JAVAFX browser.( Use JSP or something like that?)
One way is to append the result into the html template as a variable and output the html, for example:

<script>
var result = [......];//a javascript array
</script>

so the javascript in html can be something like

<script>
jQuery('#div_a').chart(result);
</script>

by doing this I think we can have a interacting html page and we can implement lots of interesting feature on it such as compare two result or real-time variable changing for the algorithm, also user can write their own html template and just use the variable we provide (it would be better to wrap in a function).

[luizribeiro]

Hi Vikash and Leo (thank you for joining us, btw),

Well, I'm open to suggestions on how to design the Report API, but here's how I imagine it at the moment:

It believe it would be nice if we had a full representation of a Report in the form of plain-old Java objects (the getReport method would return this instead of a String). Reports would be composed of some attributes like the report's title and a list of the "report components". By "report components" I mean the charts, tables and text sections that are present in the report.

To understand better what I mean, let's take a look at the Modularity statistic (run it and take a look at its report before continuing, if you haven't already). On that report, the report's title is obviously "Modularity Report", the report components would be a "table" for showing the Parameters, another "table" for showing the Results, a chart of "Size Distribution" and two text sections for the references for the algorithm and the resolution. The code for building this report would look something like this:

Code: Select all

/* Warning: this is just for explanation purposes, the methods and classes' names would obviously be different and should follow the Gephi's guidelines for implementing new features (see the tutorial I linked in my first post of this topic) */

public Report getReport() {
    Report modularityReport = new Report("Modularity Report");

    ReportTable parametersTable = new ReportTable();
    parametersTable.setData(...); // insert data into the table somehow
    modularityReport.appendComponent(parametersTable);

    // ... create and add the other components to the report ...

    return modularityReport;
}

I believe this is a good way because it simplifies the report building process and abstracts the layout of the report from the content/data. If someone is implementing a new statistic/metric on Gephi, he/she would only have to know how to use the Report API to build the report. Besides, by abstracting the content from the layout, all the reports would have a familiar look (this is where the template goes in).

I think this would solve the problem of constructing the reports. To render/print the report into the JavaFX browser, we would have some sort of "report printer" that would build the HTML source (and pretty much everything else like any necessary JSON objects, dynamic Javascript code, etc) based on the report object. I am okay with the idea of appending the charts' data into the HTML template like Leo suggested.

The idea of having a HTML template is just to make sure all the reports have the same layout. I'm not sure which is the best library to do this in this case. I suggested StringTemplate on the wiki because it's simple and it looks like it would do the job (honestly I have never used it, you're free to suggest a different approach).

Hope this helps to answer your questions and gives you something to think of. Again, this is just my two cents and NOT a specification of how the Reports API should work. Feel free to share your thoughts and even suggest a whole new approach!

[luizribeiro]

To other students that are interested on this proposal, here's an IRC log from a chat with Vikash I had today on the #gephi channel at Freenode. He asked a few questions to me that might be interesting for other students too:

Code: Select all

[00:35] < vikash23> luizribeiro: ping
[00:41] < luizribeiro> vikash23: hey
[00:42] < vikash23> luizribeiro: i have been working on my proposal and have decide to use D3.js to create reports. it basically allows to bind data to DOM objects and apply transformations.
[00:42] < vikash23> luizribeiro: is it okay to go with D3
[00:43] < luizribeiro> sure, D3 is fine
[00:43] < luizribeiro> did you read my last post on the forums?
[00:44] < vikash23> luizribeiro: yes i read you post on the forum, and i am going about the proposal as you suggested. i.e getReturn() function returing a java object for the attributes of a report.
[00:45] < luizribeiro> vikash23: cool! hope you find that information useful
[00:46] < vikash23> luizribeiro: i have one more question regarding saving and browsing reports. what formats do you need to save the reports ? will it work if we save it in normal html format or do we need to export it as a pdf.
[00:49] < luizribeiro> vikash23: well, I don't think exporting as pdf is necessary on a first moment. but I would model everything in a way that we could extend the API later to export reports as PDF files
[00:51] < luizribeiro> for example, creating a ReportPrinter interface and implementing many different report printers based on it (e.g. HTML5ReportPrinter, PDFReportPrinter)
[00:52] < luizribeiro> what do you think?
[00:53] < vikash23> ok. i think this is the right way to do it.
[00:54] < vikash23> luizribeiro: luizribeiro: ok. one more question.i have this publishReport() function to publish the generated html reports to the web browser. is it fine to save this in the temp folder and the browse can retrieve these files to display them.
[00:54] < vikash23> *browser
[00:57] < luizribeiro> yeah, I think that should be okay.
[00:57] < luizribeiro> actually, I don't remember how it is currently done. did you had a chance to take a look at it?
[00:58] < luizribeiro> I mean, if at the moment the code uses temporary files
[00:59] < vikash23> luizribeiro: currently the static png images created using the jFreeChart is stored in the temp folder and its path is written to html files.
[01:00] < luizribeiro> okay, I don't see why using temp files would be a problem. :)
[01:01] < vikash23> luizribeiro: great !!, can i apply for two projects under gephi, as i am also interested in legend module project and done some research related to it.
[01:02] < vikash23> luizribeiro: most of the organizations do allow students to apply for multiple projects.
[01:02] < luizribeiro> sure, no problem
[01:02] < luizribeiro> by the way, I just found a Report class on the ImportAPI module that can be of inspiration to us: https://github.com/gephi/gephi/blob/master/ImportAPI/src/org/gephi/io/importer/api/Report.java
[01:04] < vikash23> luizribeiro: ok. thanks a lot. one last question. i have not fixed any of the bugs yet and i dont think its possible for me to do it before 6th i.e. the last day for proposal submission. is it fine if i do it after 6th.
[01:05] < luizribeiro> no problem
[01:07] < luizribeiro> would you mind if I posted a log of this conversation to the forum topic? to make this information available to other students as well?
[01:08] < vikash23> luizribeiro: no not at all. after all open source it all about sharing ;) 
[01:08] < luizribeiro> cool, thanks!

Speaking of bugs, here's a low hanging fruit from our issues tracker: https://github.com/gephi/gephi/issues/463

Looking forward to your proposals!