More Working with CSV Files from the Command Line

FAIL (the browser should render some flash content, not this).

How to extract and manipulate CSV data using the command line.

Download in .ogv format

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Process Substitution

mattcen's picture

I've never seen this 'Process Substitution' done before! I just had a quick look at it in the Bash man page. It's something that would've been very used to me in a few occasions over the past few months. Perhaps I haven't heard of it because it's not part of the Posix standard?

Very cool! Thanks for showing us.

Regards,
Matt.

--
Regards,
Matthew Cengia

Useful even if not often used

Mitch Frazier's picture

Not something I use very often, it seems a bit "involved" so it doesn't always occur to me at first, but it is useful.

According to wikipedia ksh, zsh, and rc also support process substitution.

Mitch Frazier is an Associate Editor for Linux Journal.

text

Anonymous Blub's picture

these videos are great. i'd love to be able to follow up and learn for myself.

is there a chance you could post the session text so we could copy/paste/play?

thanks, AB

Good Idea

Mitch Frazier's picture

Here's a shell script that contains the code from this video and the previous one:

#!/bin/bash

# Extract emails:
# Could also do: grep -i -o '[^,]*@[^,]*' stuff.csv
cut -f 3 -d , stuff.csv
echo

# Use emails to extract lines:
for i in $(cut -f 3 -d , stuff.csv | sort --ignore-case | uniq --ignore-case)
do
        grep -i -F "$i" stuff.csv | head -n 1
done
echo

# Make sure posix mode is OFF so process substitution works "<(...)" syntax.
set +o posix

# Use emails to de-dupe a file and then extract lines:
for i in $(comm -23 \
                <(cut -f 3 -d , stuff.csv | sort --ignore-case | uniq --ignore-case) \
                <(cut -f 3 -d , nostuff.csv | sort --ignore-case | uniq --ignore-case))
do
        grep -i -F "$i" stuff.csv | head -n 1
done

Here's stuff.csv:

1,a,joe@example.com,e
2,b,jim@example.com,f
3,c,sally@example.com,g

And nostuff.csv:

3,c,sally@example.com,g

Mitch Frazier is an Associate Editor for Linux Journal.

the comm command is great.

Anonymous's picture

the comm command is great. set difference on the command line has always been useful to me. thanks for the example!

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix