So you’ve been using ROS to record data from a robot that you use? And you have the data in a rosbag file? And you’ve spent a while googling to find out how to extract images, data, imu readings, gps positions, etc. out of said rosbag file?
This post provides a tool to extract data to CSV format for a number of ROS message types. It was initially written for data analysis of messages in MATLAB, but applies to Python, R, SAS, Excel, or SPSS where you need it.
ROS (robot operating system) is a software system gaining popularity in robotics for control and automation.
ROS records data in binary .bag files, or bagfiles for short. Getting data out of so-called bagfiles for analysis in MATLAB, Excel, or isn’t the easiest thing in the world.ย I’ve put together a small ROS package to extract data from ROS bag files and create CSV files for use in other applications.
** update 6th July 2015 – This code has now been added to Github atย https://github.com/shanealynn/ros_csv_extraction/ **
Thus far, the data extraction tool is compatible with the following ROS message types:
- sensor_msgs/Image
- sensor_msgs/Imu
- sensor_msgs/LaserScan
- sensor_msgs/NavSatFix
- gps_common/gpsVel
- umrr_driver/radar_msg (this was a type used by the CRUISE vehicle (see below))
To install the data extraction tool, download theย zipย file, extract it somewhere on your ROS_PACKAGE_PATH, and run rosmake data_extraction before using.
The tool can be used in two different ways:
1.) Extract all compatible topics in a bag file
# Extract all message rosrun data_extract extract_all.py -b <path_to_bag_file> -o <path_to_output_dir>
2.) Extract a single topic
# Extract a certain topic rosrun data_extraction extract_topic.py -b <path_to_bag_file> -o <path_to_output_csv_file> -t <topic_name>
This program was created during a six month research proejct completed at the University of Technology Sydney on their CRUISE project. (CAS Research UTE for Intelligence, Safety, and Exploration), One of my main tasks was to transition the data collection software on the CRUISE vehicle to ROS, a popular Robot Operating System. I was working at the timeย under an Australian Endeavour award, researching withย Dr. Sarath Kodagodaย on pedestrian detection systems for autonomous vehicles using radar signals. We investigated a range of classification techniques (support vector machines, naive bayes classifiers, decision trees etc) to determine which obstacles were most likely pedestrians.
The idea was to use ROS as a data collection tool, syncing data from a number of SICK laser rangefinders, cameras, inertial measurement units, radar etc.
If you find it useful, let me know. If you have any additions / suggestions, let me know too.
Hi,
I ‘m using your code to extract laserscan, but I would like to extract other topics that are not in your list. Do I just need to add them to the allowedTopics list, or there is somthing else.
I would also like to know if each topic is stored in a single csv or there is a way to store them all together.
Thank you!
Hey Atia. Good to hear that someone is finding the code useful. I think you will need to look at the code and define an output format for any topics not currently supported. The tool exports each topic to individual CSVs since each have different numbers of variables, sample times, and variable names. Let me know how you get on. Would be happy to provide a link / instructions to people if you create more code on this!
Great work! This is just what I was looking to find.
I have the same question as Atia. I’m brand new to ROS and want to get all my data out. Your script is great. I want to add more, custom, topics. Do you have any suggestions for how to do that? Right now I have an error about “no handlers could be found for logger ‘rosout’ “.
Darren. Seems this bit of code is useful. To be honest, I haven’t looked at ROS in about 2 years, so I’m a bit out of the loop. But as far as i remember, you’ll need to look at the C++ included files that I’ve written, and follow the same format for whatever custom topic that you want to extract. It’ll be up to you to define the data format coming out etc. If you do end up adding any more topics – please do send on the details to me and I can share them here too.
I am not receiving any file from the following command: rosrun data_extraction extract_all.py -b /home/daniel/data.bag -o /home/daniel
Also, my bag file has the topics /tf_P3AT (like an imu with xyzrpy) and /temperature (single point data).
Appreciate this post. Let me try it out.
Hi ,
thanks for the plugin. I have modified to extract data from geometry_msg/vector3 format. if you are ok with sharing in a public repo, i would like to share your code in github. Please let me know if you have any questions.
Hey! Great stuff. I actually have a github account that I can upload to if its of any use. Will pass on the details! Would be good to get your changes too.
I have now put this code on Github so that it can be easily cloned and adjusted – please do submit any changes if you add topics or augment the functionality. – https://github.com/shanealynn/ros_csv_extraction/