Working with CSV Files from the Command Line
April 30th, 2009 by Mitch Frazier in
How to extract and manipulate CSV data from the command line.
Download in .ogv format
__________________________
Mitch Frazier is an Associate Editor for Linux Journal and the Web Editor for linuxjournal.com.
Special Magazine Offer -- Free Gift with Subscription
Receive a free digital copy of Linux Journal's System Administration Special Edition as well as instant online access to current and past issues. CLICK HERE for offer
Linux Journal: delivering readers the advice and inspiration they need to get the most out of their Linux systems since 1994.
Delicious
Digg
StumbleUpon
Reddit
Facebook
Post to TwitterSubscribe now!
The Latest
Newsletter
Tech Tip Videos
Recently Popular
From the Magazine
January 2010, #189
You say potato, I say potahto, you say ham, I say amateur... you see where I'm going with this? Ok, maybe not, Amateur Radio, that's where and that's what this month's issue focus is. What you might ask is the connection between Amateur Radio and Linux? Well Linux may be the only O/S out there with an AX.25 packet radio protocol driver, and it's had it since forever. So blow the dust off your license and start reading.
If Ham's not your favorite food, don't despair there are plenty of other articles in this month's issue including, but not limited to, Firewall Builder, Cucumber, Vimperator, port knocking with knockd, building appliances with Linux and Xen, and using Twitter from the command line.
I recommend the csv module
On April 30th, 2009 Anonymous (not verified) says:
I recommend the csv module in python.
http://docs.python.org/library/csv.html
It is really easy to use and handles fields with embedded newlines, quotes, commas with flying colors!
Give it a shot some time, you won't be disappointed :).
-TH
Big caveat
On April 30th, 2009 Administra.tion.ca (not verified) says:
Most CSV files I get these days have quoted strings containing commas inside the quoted field between the separators.
You'd better be very sure of your CSV contents if you're going to use cut like this.
It'll fail on this CSV file:
1,Cat food,55.69,05/04/2009
2,Power bill,149.75,03/04/2009
3,Rent,"1,350.00",01/04/2009
Use Grep
On May 2nd, 2009 Mitch Frazier says:
My original thinking was to use grep:
to extract the email but I decided on cut to simplify the "look" of it a bit for the video.
Of course your sample doesn't include any emails so that wouldn't work... so you'd have to adapt/change it to extract the right field.
A perhaps more robust method is to convert the CSV file to use tabs rather than commas. With tabs cut should work most of the time. Still might be that you'd have quoted tabs, but unless your CSV came from a database dump that seems unlikely.
__________________________Mitch Frazier is an Associate Editor for Linux Journal and the Web Editor for linuxjournal.com.
How will grep help us parse?
On May 5th, 2009 Rob Russell (not verified) says:
That grep will identify lines that are CSV, but won't parse out the values. And yes, I deal with tons of huge CSV files that go in and out of databases, so it's important that I get my scripts right.
The only thing I've found that comes close to what I need is http://perlmeme.org/tutorials/parsing_csv.html -- so I'm in the process of rewriting that in to something bash can invoke from inside shell scripts.
Forgot an option
On May 5th, 2009 Mitch Frazier says:
Forgot the -o option:
that will show you only the part that matched.
__________________________Mitch Frazier is an Associate Editor for Linux Journal and the Web Editor for linuxjournal.com.
But that's still not "parsing"
On May 7th, 2009 Rob Russell (not verified) says:
Sure, that'll extract an email address, but it doesn't fit any accepted definition of "parsing," since it assumes that you have prior knowledge of what's in the field and can't properly handle cases of having more than one @ in any line.
You're handling strings of text of which you can predict formatting, but you are certainly not using techniques that are transferrable to the general case of "I need to parse a CSV file."
That's Correct
On May 8th, 2009 Mitch Frazier says:
You're 100% right, neither cut nor grep are generalized solutions for parsing CSV files, but then again I never said they were. As far as I know I never used the word parse, that was your word.
What I meant to say was that grep is both a floor wax and a dessert topping..png)
__________________________Mitch Frazier is an Associate Editor for Linux Journal and the Web Editor for linuxjournal.com.
No need for uniq
On April 30th, 2009 Gumnos (not verified) says:
GNU
sorthas a--unique(-u) argument that checks sorts and checks for uniqueness in the same pass. So you can just use "sort -uf" or "sort --ignore-case --unique" skippinguniqaltogether.-Tim
Thanks
On May 2nd, 2009 Mitch Frazier says:
Yes it does and that's why I always say one should re-RTFM once in a while to refresh one's memory and check for new features.
.png)
I might also add that one shouldn't jump to the the conclusion that the -u option makes uniq obsolete: uniq has some other useful options that come in handy once in a while. So re-RTFM applies to uniq also.
__________________________Mitch Frazier is an Associate Editor for Linux Journal and the Web Editor for linuxjournal.com.
Beat me to it
On May 1st, 2009 mattcen says:
I'm glad somebody mentioned this; I found out about that a couple of months ago. Much better than having to pipe to yet another process :D
Post new comment