Large Dataset best setup?

[kindahl]

Hi there,

I would like to visualise a large direct non-dynamic dataset. I know it's been done before, so I just need to figure out how:

I have a dataset of about 90k nodes and 800k edges (i know, i know, please spare me the "visualisation for large data that size is useless" talk... once i get this visualised i'll be filtering out what i need based on what visually looks relevant)

I am using a mac running 10.11.6, retina display, processor: 1.1, memory 8GB, graphics intel hd 5300 1536mb... i know it's not powerful enough to perform the task at hand...

Gephi on my device is the latest and works great on my device for smaller datasets. What i need to figure out is what kind of compute I need to apply modularity and force atlas 2 + the basic statistical algorithms seamlessly on such a large dataset, as well as a rough estimate on how long it will take (days, hours, instant [wishful thinking]?).

Any suggestions or advice would be totally appreciated.

Thanks...

[kindahl]

So attempted to run this via my institution's cluster computing... it is much faster but I am still facing interface issues where the graphics are all distorted... their recommendation is to use GraphViz and I am beginning to lose faith in Gephi

[eduramiba]

Distorted? Can you share some screenshots?

[kindahl]

Distorted? Can you share some screenshots?

Eduramiba, thanks for answering. I've attached a screenshot of Gephi as it is X forwarded via XQuartz...
What do you think?

[eduramiba]

Looks like typical graphic driver problems.
First things to try:
1. Force software rendering with https://superuser.com/questions/106056/ ... -rendering LIBGL_ALWAYS_SOFTWARE=1 or similar
2. Make sure you use Oracle JDK, not openjdk

[kindahl]

Looks like typical graphic driver problems.
First things to try:
1. Force software rendering with https://superuser.com/questions/106056/ ... -rendering LIBGL_ALWAYS_SOFTWARE=1 or similar
2. Make sure you use Oracle JDK, not openjdk

Thank you! I'll give it a whack. I am not sure how to do this using cluster computing, but will report back once I figure it out.

[kindahl]

Looks like typical graphic driver problems.
First things to try:
1. Force software rendering with https://superuser.com/questions/106056/ ... -rendering LIBGL_ALWAYS_SOFTWARE=1 or similar
2. Make sure you use Oracle JDK, not openjdk

Thank you! I'll give it a whack. I am not sure how to do this using cluster computing, but will report back once I figure it out.

eduramiba, Gephi is using the Oracle JDK on the cluster compute, but I am still facing the same issue. I am not sure what forcing the software to render is as suggested in point 1 since I am not that familiar with unix shell ... would you be able to give me a hint?

Cheers!

[kindahl]

oh boy... I keep getting this:

java.lang.ArrayIndexOutOfBoundsException: -1
at java.awt.font.LineBreakMeasurer.nextOffset(LineBreakMeasurer.java:364)
at java.awt.font.LineBreakMeasurer.nextLayout(LineBreakMeasurer.java:440)
at java.awt.font.LineBreakMeasurer.nextLayout(LineBreakMeasurer.java:413)
at org.gephi.ui.components.richtooltip.BasicRichTooltipPanelUI$RichTooltipPanelLayout.layoutContainer(BasicRichTooltipPanelUI.java:387)
at java.awt.Container.layout(Container.java:1510)
at java.awt.Container.doLayout(Container.java:1499)
at java.awt.Container.validateTree(Container.java:1695)
at java.awt.Container.validateTree(Container.java:1704)
at java.awt.Container.validateTree(Container.java:1704)
at java.awt.Container.validateTree(Container.java:1704)
at java.awt.Container.validateTree(Container.java:1704)
at java.awt.Container.validate(Container.java:1630)
at javax.swing.Popup.reset(Popup.java:164)
at javax.swing.PopupFactory$HeavyWeightPopup.getHeavyWeightPopup(PopupFactory.java:351)
at javax.swing.PopupFactory.getHeavyWeightPopup(PopupFactory.java:278)
at javax.swing.PopupFactory.getPopup(PopupFactory.java:233)
at com.apple.laf.ScreenPopupFactory._getHeavyWeightPopup(Native Method)
at com.apple.laf.ScreenPopupFactory.getPopup(ScreenPopupFactory.java:74)
at org.gephi.ui.components.richtooltip.RichTooltip.showTooltip(RichTooltip.java:305)
at org.gephi.desktop.layout.LayoutPanel$2.mouseEntered(LayoutPanel.java:118)
at java.awt.Component.processMouseEvent(Component.java:6542)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3324)
at java.awt.Component.processEvent(Component.java:6298)
at java.awt.Container.processEvent(Container.java:2236)
at java.awt.Component.dispatchEventImpl(Component.java:4889)
at java.awt.Container.dispatchEventImpl(Container.java:2294)
at java.awt.Component.dispatchEvent(Component.java:4711)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4888)
at java.awt.LightweightDispatcher.retargetMouseEnterExit(Container.java:4676)
at java.awt.LightweightDispatcher.trackMouseEnterExit(Container.java:4654)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4505)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4466)
at java.awt.Container.dispatchEventImpl(Container.java:2280)
at java.awt.Window.dispatchEventImpl(Window.java:2746)
at java.awt.Component.dispatchEvent(Component.java:4711)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758)
at java.awt.EventQueue.access$500(EventQueue.java:97)
at java.awt.EventQueue$3.run(EventQueue.java:709)
at java.awt.EventQueue$3.run(EventQueue.java:703)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:80)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:90)
at java.awt.EventQueue$4.run(EventQueue.java:731)
at java.awt.EventQueue$4.run(EventQueue.java:729)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:80)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:728)
at org.netbeans.core.TimableEventQueue.dispatchEvent(TimableEventQueue.java:159)
[catch] at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)


any clues?

[kindahl]

here's the error from gephi running on compute platform:
java.lang.ClassCastException: sun.awt.image.BufImgSurfaceData cannot be cast to sun.java2d.xr.XRSurfaceData
at sun.java2d.xr.XRPMBlitLoops.cacheToTmpSurface(XRPMBlitLoops.java:145)
at sun.java2d.xr.XrSwToPMBlit.Blit(XRPMBlitLoops.java:353)
at sun.java2d.SurfaceDataProxy.updateSurfaceData(SurfaceDataProxy.java:498)
at sun.java2d.SurfaceDataProxy.replaceData(SurfaceDataProxy.java:455)
at sun.java2d.SurfaceData.getSourceSurfaceData(SurfaceData.java:233)
at sun.java2d.pipe.DrawImage.renderImageCopy(DrawImage.java:566)
at sun.java2d.pipe.DrawImage.copyImage(DrawImage.java:67)
at sun.java2d.pipe.DrawImage.copyImage(DrawImage.java:1014)
at sun.java2d.pipe.ValidatePipe.copyImage(ValidatePipe.java:186)
at sun.java2d.SunGraphics2D.drawImage(SunGraphics2D.java:3318)
at sun.java2d.SunGraphics2D.drawImage(SunGraphics2D.java:3296)
at javax.swing.ImageIcon.paintIcon(ImageIcon.java:425)
at org.netbeans.core.ui.notifications.FlashingIcon$MyIcon.paintIcon(FlashingIcon.java:353)
at javax.swing.plaf.basic.BasicLabelUI.paint(BasicLabelUI.java:167)
at javax.swing.plaf.ComponentUI.update(ComponentUI.java:161)
at javax.swing.JComponent.paintComponent(JComponent.java:780)
at javax.swing.JComponent.paint(JComponent.java:1056)
at javax.swing.JComponent.paintChildren(JComponent.java:889)
at javax.swing.JComponent.paint(JComponent.java:1065)
at javax.swing.JComponent.paintToOffscreen(JComponent.java:5210)
at javax.swing.BufferStrategyPaintManager.paint(BufferStrategyPaintManager.java:290)
at javax.swing.RepaintManager.paint(RepaintManager.java:1272)
at javax.swing.JComponent._paintImmediately(JComponent.java:5158)
at javax.swing.JComponent.paintImmediately(JComponent.java:4969)
at javax.swing.RepaintManager$4.run(RepaintManager.java:831)
at javax.swing.RepaintManager$4.run(RepaintManager.java:814)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
at javax.swing.RepaintManager.paintDirtyRegions(RepaintManager.java:814)
at javax.swing.RepaintManager.paintDirtyRegions(RepaintManager.java:789)
at javax.swing.RepaintManager.prePaintDirtyRegions(RepaintManager.java:738)
at javax.swing.RepaintManager.access$1200(RepaintManager.java:64)
at javax.swing.RepaintManager$ProcessingRunnable.run(RepaintManager.java:1732)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:756)
at java.awt.EventQueue.access$500(EventQueue.java:97)
at java.awt.EventQueue$3.run(EventQueue.java:709)
at java.awt.EventQueue$3.run(EventQueue.java:703)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:726)
at org.netbeans.core.TimableEventQueue.dispatchEvent(TimableEventQueue.java:159)
[catch] at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)