Home ›

Working with CSV Files from the Command Line

Apr 30, 2009  By Mitch Frazier
 in
FAIL (the browser should render some flash content, not this).

How to extract and manipulate CSV data from the command line.

Download in .ogv format

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Thanks a lot! Such a concise

qubodup's picture
Submitted by qubodup (not verified) on Fri, 05/07/2010 - 09:00.

Thanks a lot! Such a concise and "let's get started without talking log" video helps me a lot to learn efficiently in the CLI!

I recommend the csv module

Anonymous's picture
Submitted by Anonymous (not verified) on Thu, 04/30/2009 - 21:42.

I recommend the csv module in python.

http://docs.python.org/library/csv.html

It is really easy to use and handles fields with embedded newlines, quotes, commas with flying colors!

Give it a shot some time, you won't be disappointed :).

-TH

Big caveat

Administra.tion.ca's picture
Submitted by Administra.tion.ca (not verified) on Thu, 04/30/2009 - 20:50.

Most CSV files I get these days have quoted strings containing commas inside the quoted field between the separators.

You'd better be very sure of your CSV contents if you're going to use cut like this.

It'll fail on this CSV file:

1,Cat food,55.69,05/04/2009
2,Power bill,149.75,03/04/2009
3,Rent,"1,350.00",01/04/2009

Use Grep

Mitch Frazier's picture
Submitted by Mitch Frazier on Sat, 05/02/2009 - 09:31.

My original thinking was to use grep:

   grep -i '[^,]*@[^,]*' stuff.csv ...

to extract the email but I decided on cut to simplify the "look" of it a bit for the video.

Of course your sample doesn't include any emails so that wouldn't work... so you'd have to adapt/change it to extract the right field.

A perhaps more robust method is to convert the CSV file to use tabs rather than commas. With tabs cut should work most of the time. Still might be that you'd have quoted tabs, but unless your CSV came from a database dump that seems unlikely.

Mitch Frazier is an Associate Editor for Linux Journal.

How will grep help us parse?

Rob Russell's picture
Submitted by Rob Russell (not verified) on Tue, 05/05/2009 - 07:47.

That grep will identify lines that are CSV, but won't parse out the values. And yes, I deal with tons of huge CSV files that go in and out of databases, so it's important that I get my scripts right.

The only thing I've found that comes close to what I need is http://perlmeme.org/tutorials/parsing_csv.html -- so I'm in the process of rewriting that in to something bash can invoke from inside shell scripts.

Forgot an option

Mitch Frazier's picture
Submitted by Mitch Frazier on Tue, 05/05/2009 - 10:27.

Forgot the -o option:

  grep -o -i '[^,]*@[^,]*' stuff.csv ...

that will show you only the part that matched.

Mitch Frazier is an Associate Editor for Linux Journal.

But that's still not "parsing"

Rob Russell 's picture
Submitted by Rob Russell (not verified) on Thu, 05/07/2009 - 21:35.

Sure, that'll extract an email address, but it doesn't fit any accepted definition of "parsing," since it assumes that you have prior knowledge of what's in the field and can't properly handle cases of having more than one @ in any line.

You're handling strings of text of which you can predict formatting, but you are certainly not using techniques that are transferrable to the general case of "I need to parse a CSV file."

That's Correct

Mitch Frazier's picture
Submitted by Mitch Frazier on Fri, 05/08/2009 - 21:56.

You're 100% right, neither cut nor grep are generalized solutions for parsing CSV files, but then again I never said they were. As far as I know I never used the word parse, that was your word.

What I meant to say was that grep is both a floor wax and a dessert topping.

Mitch Frazier is an Associate Editor for Linux Journal.

No need for uniq

Gumnos's picture
Submitted by Gumnos (not verified) on Thu, 04/30/2009 - 14:57.

GNU sort has a --unique (-u) argument that checks sorts and checks for uniqueness in the same pass. So you can just use "sort -uf" or "sort --ignore-case --unique" skipping uniq altogether.

-Tim

Thanks

Mitch Frazier's picture
Submitted by Mitch Frazier on Sat, 05/02/2009 - 09:47.

Yes it does and that's why I always say one should re-RTFM once in a while to refresh one's memory and check for new features.

I might also add that one shouldn't jump to the the conclusion that the -u option makes uniq obsolete: uniq has some other useful options that come in handy once in a while. So re-RTFM applies to uniq also.

Mitch Frazier is an Associate Editor for Linux Journal.

Beat me to it

mattcen's picture
Submitted by mattcen on Fri, 05/01/2009 - 09:41.

I'm glad somebody mentioned this; I found out about that a couple of months ago. Much better than having to pipe to yet another process :D

--
Regards,
Matthew Cengia

Webcast
More Resources
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers

Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.

Learn More

Sponsored by AMD

White Paper
More Resources
Red Hat White Paper: Using an Open Source Framework to Catch the Bad Guy

Built-in forensics, incident response, and security with Red Hat Enterprise Linux 6

Every security policy provides guidance and requirements for ensuring adequate protection of information and data, as well as high-level technical and administrative security requirements for a system in a given environment. Traditionally, providing security for a system focuses on the confidentiality of the information on it. However, protecting the data integrity and system and data availability is just as important. For example, when processing United States intelligence information, there are three attributes that require protection: confidentiality, integrity, and availability.

Learn more about catching the bad guy in this free white paper.

Learn More

Sponsored by DLT Solutions