Netflow Data and Gephi
-
- Posts:10
- Joined:06 Mar 2011 12:57 [phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1275: count(): Parameter must be an array or an object that implements Countable
Hi Guys,
A while back you helped me get my netflow data working on Gephi, with weights. As a reminder I ended up using DL format :
<src_ip>, <dst_ip>, <number of packets>
This all works beautifully
Now I would like to add time to the picture. In netflow there is a start_time and an end_time for the flow. I would like to use this to get a timeline dynamic graph.
I don't think my DL format will do this, and I have seen some examples for spreadsheet CSV format but that does not include weight. Here is my question: what is the best data input format for my netflow data? There is a lot of it so I would prefer not to have a markup format, but would prefer a CSV format if possible.
Like this??
<src_ip>, <dst_ip>, <number of packets>, <start_time>, <end_time>
From the spreadsheet CSV example I can see "<(1,10]>" as an example of time but used with a node.
Can I do a spreadsheet like this:
Source, Target, Weight, Time Interval
<src_ip>, <dst_ip>, <number of packets>, "(start_time, end_time)"
Can I do a normal CSV format (as opposed to a spreadsheet) ?
Thanks
A while back you helped me get my netflow data working on Gephi, with weights. As a reminder I ended up using DL format :
<src_ip>, <dst_ip>, <number of packets>
This all works beautifully
Now I would like to add time to the picture. In netflow there is a start_time and an end_time for the flow. I would like to use this to get a timeline dynamic graph.
I don't think my DL format will do this, and I have seen some examples for spreadsheet CSV format but that does not include weight. Here is my question: what is the best data input format for my netflow data? There is a lot of it so I would prefer not to have a markup format, but would prefer a CSV format if possible.
Like this??
<src_ip>, <dst_ip>, <number of packets>, <start_time>, <end_time>
From the spreadsheet CSV example I can see "<(1,10]>" as an example of time but used with a node.
Can I do a spreadsheet like this:
Source, Target, Weight, Time Interval
<src_ip>, <dst_ip>, <number of packets>, "(start_time, end_time)"
Can I do a normal CSV format (as opposed to a spreadsheet) ?
Thanks
- eduramiba
- Gephi Code Manager
- Posts:1064
- Joined:22 Mar 2010 15:30
- Location:Madrid, Spain [phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1275: count(): Parameter must be an array or an object that implements Countable
Re: Netflow Data and Gephi
Hi,
Yes, Spreadsheet supports weights and dynamic attributes like Time intervals.
Your CSV file like that should be fine.
Steps for import are:
Eduardo
Yes, Spreadsheet supports weights and dynamic attributes like Time intervals.
Your CSV file like that should be fine.
Steps for import are:
- Create new project
- Go to data laboratory
- Optionally import nodes data to nodes table, before edges
- Import spreadsheet to edges table, making sure your columns are named 'Source', 'Target' and 'Weight' (optional). Case insensitive. Also remember to select types for your columns, specially TimeInterval for your 'Time interval' column.
Eduardo
-
- Posts:10
- Joined:06 Mar 2011 12:57 [phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1275: count(): Parameter must be an array or an object that implements Countable
Re: Netflow Data and Gephi
Excellent, thanks for the quick reply. I will try that today and report back.
Cheers
Cheers
-
- Posts:10
- Joined:06 Mar 2011 12:57 [phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1275: count(): Parameter must be an array or an object that implements Countable
Re: Netflow Data and Gephi
Hi,
Well the import of the CSV file for the node table worked just fine.
I did 'import spreadsheet' and selected 'node table' set the data times and it all worked. I have a node table with all of my data, and with 'node id' added.
Problem is that I can't see how to add the 'edge' table as per your list below. When I try to repeat the above and select 'edge table' and load the spreadsheet it complains:
"Edges table needs a 'Source' and 'Target' column with node ids
The node ids were added to the node table when I did the first import, but they are not in the spreadsheet?
Here is the first two lines of the CSV file:
'source','target','weight','time interval'
11.11.11.11,22.22.22.22,1,"<(1335972406,1336577206)>"
Thanks for any help.
Well the import of the CSV file for the node table worked just fine.
I did 'import spreadsheet' and selected 'node table' set the data times and it all worked. I have a node table with all of my data, and with 'node id' added.
Problem is that I can't see how to add the 'edge' table as per your list below. When I try to repeat the above and select 'edge table' and load the spreadsheet it complains:
"Edges table needs a 'Source' and 'Target' column with node ids
The node ids were added to the node table when I did the first import, but they are not in the spreadsheet?
Here is the first two lines of the CSV file:
'source','target','weight','time interval'
11.11.11.11,22.22.22.22,1,"<(1335972406,1336577206)>"
Thanks for any help.
eduramiba wrote:Hi,
Steps for import are:
- Create new project
- Go to data laboratory
- Optionally import nodes data to nodes table, before edges
- Import spreadsheet to edges table, making sure your columns are named 'Source', 'Target' and 'Weight' (optional). Case insensitive. Also remember to select types for your columns, specially TimeInterval for your 'Time interval' column.
-
- Posts:10
- Joined:06 Mar 2011 12:57 [phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1275: count(): Parameter must be an array or an object that implements Countable
Re: Netflow Data and Gephi
Is it because when I try to import the 'edge table' it is not recognising the 'source' and 'target' as node ids because they are strings?
They are unique strings (representing an IP address) but is the edge table expecting an integer?
They are unique strings (representing an IP address) but is the edge table expecting an integer?
bob_the_web wrote: Here is the first two lines of the CSV file:
'source','target','weight','time interval'
11.11.11.11,22.22.22.22,1,"<(1335972406,1336577206)>"
- eduramiba
- Gephi Code Manager
- Posts:1064
- Joined:22 Mar 2010 15:30
- Location:Madrid, Spain [phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1275: count(): Parameter must be an array or an object that implements Countable
Re: Netflow Data and Gephi
No, it accepts strings.
I guess the problem is your CSV header is:
Try to change it to:
In CSV you use double quotes to make sure some text is correctly recognised, but not single quotes.
Eduardo
I guess the problem is your CSV header is:
Code: Select all
'source','target','weight','time interval'
Code: Select all
source,target,weight,time interval
Eduardo
-
- Posts:10
- Joined:06 Mar 2011 12:57 [phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1275: count(): Parameter must be an array or an object that implements Countable
Re: Netflow Data and Gephi
Yes, that was it. All working now - thanks so much!
Final question: I don't really want to use a time interval, I would rather use a 'first seen' datetime. In the example below I achieve this by converting first seen datetime to unix timestamp and then add a big time offset to make sure the node stays - creating an artificial interval.
Can I use just a first seen datetime instead of interval? If not this will do.
Thanks
Final question: I don't really want to use a time interval, I would rather use a 'first seen' datetime. In the example below I achieve this by converting first seen datetime to unix timestamp and then add a big time offset to make sure the node stays - creating an artificial interval.
Can I use just a first seen datetime instead of interval? If not this will do.
Thanks
eduramiba wrote:No, it accepts strings.
I guess the problem is your CSV header is:Try to change it to:Code: Select all
'source','target','weight','time interval'
In CSV you use double quotes to make sure some text is correctly recognised, but not single quotes.Code: Select all
source,target,weight,time interval
Eduardo
- phreakocious
- Gephi Plugin Developer
- Posts:25
- Joined:03 Aug 2010 15:15
- Contact:
Re: Netflow Data and Gephi
I believe you can set the end time to 'null' to make it persist forever. (this works in Java) ... Hope that helps!
- eduramiba
- Gephi Code Manager
- Posts:1064
- Joined:22 Mar 2010 15:30
- Location:Madrid, Spain [phpBB Debug] PHP Warning: in file [ROOT]/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1275: count(): Parameter must be an array or an object that implements Countable
Re: Netflow Data and Gephi
Hi,
You can write this <[2.0, Infinity]> for example.
Eduardo
You can write this <[2.0, Infinity]> for example.
Eduardo