[FIXED] Can Gephi handle large GEXF files?

[prasoon]

I've generated a graph from email traffic http://prasoon.blogspot.com/2010/04/soc ... r-and.html and while trying to load it in Gephi, I faced issues with large gexf files http://prasoon.blogspot.com/2010/06/dat ... ocial.html.

My graph is static and directed and has ~ 70,000 nodes and ~ 500,000 edges. I noticed that if I tried loading more than 40,000 nodes and 250,000 edges (gexf file size > ~ 100MB?), Gephi stopped responding while loading the graph.

I have increased the memory Gephi allocates for JVM at startup to 1.4GB and I'm using a souped-up Windows 7 desktop (6GB+ memory, dual-code processor, advanced graphics card etc.)

Has anyone else faced this issue?

[mbastian]

Hi,

I'm not surprised about this problem, as I have same issues with large graphs like this. This is due to our current implementation of our GEXF parser, which basically eats a lot of memory (we use DOM...). Hopefully I started a new GEXF parser, using STAX which is keeping a very low memory footprint. I will continue the work on this parser as soon as possible (if someone wants to help me...feel free) and include it in the next release.

I'm not sure if it will make it, but you can try to use a 64-bit JVM and increase memory up to 4GB.

[prasoon]

Thanks for your note. Good to know that its a known issue and will be addressed. I'll try a 64 bit machine for now and wait for your fix. cheers!

[jtarbell]

I too have attempted to load a very large GEFX file into Gephi without success.

The data is a network of shops on Etsy.com connected by their favorites. The file is approximately 2.1Gb, with 376,737 shops (nodes) and 28,172,904 edges.

I knew this was a huge amount of data, but considered it worth a go on my 64bit Core2 Quad processor Windows 7 machine. Gephi's process memory rose to the JVM's limit then hovered there with no response.

My post here is to express my best wishes for the continued development of Gephi. Please also accept my deep gratitude for such an amazing program.

[mbastian]

Hello, there is a memory use issue with the current GEXF parser, filed as bug 596872 so I think it's normal memory starve for such a network size. Hopefully this bug is now fixed and will be deployed in the next release, 0.7beta, coming in about 15 days.

However this is quite huge dataset, especially with the number of edges. I think you need at least 8 or 16GB of ram to make it work properly in gephi. Of course visualization will be slow but you can perform filtering to reduce the dataset, and/or analysis.

Thanks for your wishes