Loading
Working with CSV Files from the Command Line
White Paper
Fabric-Based Computing Enables Optimized Hyperscale Data Centers
Today’s modular x86 servers are compute-centric, designed as a least common denominator to support a wide range of IT workloads. Those generic, virtualized IT workloads have much different resource optimization requirements than hyperscale and cloud applications. They have resulted in a “one size fits all” enterprise IT architecture that is not optimized for a specific set of IT workloads, and especially not emerging hyperscale workloads, such as web applications, big data, and object storage. In this report, you will learn how shifting the focus from traditional compute-centric IT architectures to an innovative disaggregated fabric-based architecture can optimize and scale your data center.
Sponsored by AMD
White Paper
Red Hat White Paper: Using an Open Source Framework to Catch the Bad Guy
Built-in forensics, incident response, and security with Red Hat Enterprise Linux 6
Every security policy provides guidance and requirements for ensuring adequate protection of information and data, as well as high-level technical and administrative security requirements for a system in a given environment. Traditionally, providing security for a system focuses on the confidentiality of the information on it. However, protecting the data integrity and system and data availability is just as important. For example, when processing United States intelligence information, there are three attributes that require protection: confidentiality, integrity, and availability.
Learn more about catching the bad guy in this free white paper.
Sponsored by DLT Solutions
- RSS Feeds
- Making Linux and Android Get Along (It's Not as Hard as It Sounds)
- New Products
- Drupal Is a Framework: Why Everyone Needs to Understand This
- A Topic for Discussion - Open Source Feature-Richness?
- Home, My Backup Data Center
- Validate an E-Mail Address with PHP, the Right Way
- Trying to Tame the Tablet
- New Products
- Tech Tip: Really Simple HTTP Server with Python
- git-annex assistant
2 hours 3 min ago - direct cable connection
2 hours 25 min ago - Agreed on AirDroid. With my
2 hours 35 min ago - I just learned this
2 hours 39 min ago - enterprise
3 hours 10 min ago - not living upto the mobile revolution
6 hours 1 min ago - Deceptive Advertising and
6 hours 36 min ago - Let\'s declare that you have
6 hours 37 min ago - Alterations in Contest Due
6 hours 38 min ago - At a numbers mindset, your
6 hours 40 min ago
Enter to Win an Adafruit Prototyping Pi Plate Kit for Raspberry Pi
It's Raspberry Pi month at Linux Journal. Each week in May, Adafruit will be giving away a Pi-related prize to a lucky, randomly drawn LJ reader. Winners will be announced weekly.
Fill out the fields below to enter to win this week's prize-- a Prototyping Pi Plate Kit for Raspberry Pi.
Congratulations to our winners so far:
- 5-8-13, Pi Starter Pack: Jack Davis
- 5-15-13, Pi Model B 512MB RAM: Patrick Dunn
- Next winner announced on 5-21-13!
Free Webinar: Linux Backup and Recovery
Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.
In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.
Developer Poll
Comments
Thanks a lot! Such a concise
Thanks a lot! Such a concise and "let's get started without talking log" video helps me a lot to learn efficiently in the CLI!
I recommend the csv module
I recommend the csv module in python.
http://docs.python.org/library/csv.html
It is really easy to use and handles fields with embedded newlines, quotes, commas with flying colors!
Give it a shot some time, you won't be disappointed :).
-TH
Big caveat
Most CSV files I get these days have quoted strings containing commas inside the quoted field between the separators.
You'd better be very sure of your CSV contents if you're going to use cut like this.
It'll fail on this CSV file:
1,Cat food,55.69,05/04/2009
2,Power bill,149.75,03/04/2009
3,Rent,"1,350.00",01/04/2009
Use Grep
My original thinking was to use grep:
to extract the email but I decided on cut to simplify the "look" of it a bit for the video.
Of course your sample doesn't include any emails so that wouldn't work... so you'd have to adapt/change it to extract the right field.
A perhaps more robust method is to convert the CSV file to use tabs rather than commas. With tabs cut should work most of the time. Still might be that you'd have quoted tabs, but unless your CSV came from a database dump that seems unlikely.
Mitch Frazier is an Associate Editor for Linux Journal.
How will grep help us parse?
That grep will identify lines that are CSV, but won't parse out the values. And yes, I deal with tons of huge CSV files that go in and out of databases, so it's important that I get my scripts right.
The only thing I've found that comes close to what I need is http://perlmeme.org/tutorials/parsing_csv.html -- so I'm in the process of rewriting that in to something bash can invoke from inside shell scripts.
Forgot an option
Forgot the -o option:
that will show you only the part that matched.
Mitch Frazier is an Associate Editor for Linux Journal.
But that's still not "parsing"
Sure, that'll extract an email address, but it doesn't fit any accepted definition of "parsing," since it assumes that you have prior knowledge of what's in the field and can't properly handle cases of having more than one @ in any line.
You're handling strings of text of which you can predict formatting, but you are certainly not using techniques that are transferrable to the general case of "I need to parse a CSV file."
That's Correct
You're 100% right, neither cut nor grep are generalized solutions for parsing CSV files, but then again I never said they were. As far as I know I never used the word parse, that was your word.
What I meant to say was that grep is both a floor wax and a dessert topping..png)
Mitch Frazier is an Associate Editor for Linux Journal.
No need for uniq
GNU
sorthas a--unique(-u) argument that checks sorts and checks for uniqueness in the same pass. So you can just use "sort -uf" or "sort --ignore-case --unique" skippinguniqaltogether.-Tim
Thanks
Yes it does and that's why I always say one should re-RTFM once in a while to refresh one's memory and check for new features.
.png)
I might also add that one shouldn't jump to the the conclusion that the -u option makes uniq obsolete: uniq has some other useful options that come in handy once in a while. So re-RTFM applies to uniq also.
Mitch Frazier is an Associate Editor for Linux Journal.
Beat me to it
I'm glad somebody mentioned this; I found out about that a couple of months ago. Much better than having to pipe to yet another process :D
--
Regards,
Matthew Cengia