Python Scripts as a Replacement for Bash Utility Scripts

To demonstrate the power of combining Python scripts in a modular and piped fashion, let's expand further on the problem space. Let's find the top five users of the service. head is a command that allows you to specify a certain number of lines to display of the standard input it is given. Adding this to the command chain gives the following:

$ cat names.log | python | sort -rn | head -n 5

This prints only the top five users and ignores the rest. Similarly, to get the five users who use the service least, you can use the tail command, which takes the same arguments. The result of the Python command being printed to standard output allows you to build and extend upon its functionality.

To demonstrate the modularity of this script, let's once again change the problem space. The service also generates a comma-separated value (CSV) log file that contains a list of e-mail addresses and the comments that each e-mail address made about the service. Here's an example entry:

"", "This service is great."

The task is to provide a way for the service to send a thank-you message to the top ten users in terms of comment frequency. First, you need a script that can read and print a certain column of CSV data. The standard library of Python provides a CSV reader. The Python script below completes this goal:

#!/usr/bin/env python
# CSV module that comes with the Python standard library
import csv
import sys

if __name__ == "__main__":
    # The CSV module exposes a reader object that takes
    # a file object to read. In this example, sys.stdin.
    csvfile = csv.reader(sys.stdin)

    # The script should take one argument that is a column number.
    # Command-line arguments are accessed via sys.argv list.
    column_number = 0
    if len(sys.argv) > 1:
            column_number = int(sys.argv[1])

    # Each row in the CSV file is a list with each 
    # comma-separated value for that line.
    for row in csvfile:
            print row[column_number]

This script can parse the CSV data and return in plain text the column that is supplied as a command-line argument. It uses print instead of sys.stdout.write, as print, by default, uses standard out as its output file.

Let's add this script to the chain. The new script is chained with the others to print out a list of e-mail addresses and their comment frequencies using the command listed below (the .csv log file is assumed to be called emailcomments.csv and the new Python script,

$ cat emailcomments.csv | python | 
 ↪python | sort -rn | head -n 5

Next, you need a way to send an e-mail. In the Python standard library of functions, you can import smtplib, which is a module that allows you to connect to an SMTP server to send mail. Let's write a simple Python script that uses this library to send a message to each of the top ten e-mail addresses found already:

#!/usr/bin/env python
import smtplib
import sys


GMAIL_EMAIL = "Your Gmail Email Goes Here"
GMAIL_PASSWORD = "Your Gmail Password Goes Here"

def initialize_smtp_server():
    This function initializes and greets the smtp server.
    It logs in using the provided credentials and returns 
    the smtp server object as a result.
    smtpserver = smtplib.SMTP(GMAIL_SMTP_SERVER, GMAIL_SMTP_PORT)
    smtpserver.login(GMAIL_EMAIL, GMAIL_PASSWORD)
    return smtpserver

def send_thank_you_mail(email):
    to_email = email
    from_email = GMAIL_EMAIL
    subj = "Thanks for being an active commenter"
    # The header consists of the To and From and Subject lines
    # separated using a newline character
    header = "To:%s\nFrom:%s\nSubject:%s \n" % (to_email,
            from_email, subj)
    # Hard-coded templates are not best practice.
    msg_body = """
    Hi %s,

    Thank you very much for your repeated comments on our service.
    The interaction is much appreciated.

    Thank You.""" % email
    content = header + "\n" + msg_body
    smtpserver = initialize_smtp_server()
    smtpserver.sendmail(from_email, to_email, content)

if __name__ == "__main__":
    # for every line of input.
    for email in sys.stdin.readlines():

This Python script supports contacting any SMTP server, whether local or remote. For ease of use, I have included Gmail's SMTP server, and it should work, provided you give the scripts the correct Gmail credentials. The script uses the functions provided to send mail in smtplib. This again demonstrates the power of using Python at this level. Something like SMTP interaction is easy and readable in Python. Equivalent shell scripts are messy, and such libraries are not as easily accessible, if they exist at all.

In order to send the e-mails to the top ten users sorted by comment frequency, first you must isolate only the e-mail column of the output of column names. To isolate a certain column in Linux, you use the cut command. In the example below, the commands are given in two separate chains. For ease of use, I wrote the output into a temporary file, which can be loaded into the second chain. This simply makes the process more readable (the Python script for sending mail is referred to as

$ cat emailcomments.csv | python | 
 ↪python | sort -rn > /tmp/comment_freq
$ cat /tmp/comment_freq | head -n 10 | cut -f2 | 

This shows the real power of Python as a utility in a chain of bash commands such as this. Writing scripts that accept input from standard input and write any data out to standard out, allows the developer to chain commands such as these together quickly and easily with a link in the chain often being a Python program. This philosophy of designing a small application that services one purpose fits nicely with the flow of commands being used here.


Richard Delaney is a software engineer with Demonware Ireland. Richard works on back-end Web services using Python and the Django Web framework. He has been an avid Linux user and evangelist for the past five years.


Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.


Anonymous's picture

Great article, I’ve saved as a favorite this web site so with any luck , I will see a lot more on this subject matter in the foreseeable future! giraffe prints

Great post. I must say thanks

Raza's picture

Great post. I must say thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.
hot girls

WONDERFUL Post.thanks for

dennishobson's picture

WONDERFUL Post.thanks for share..more wait .. …There are certainly a lot of details like that to take into consideration. That is a great point to bring up. I offer the thoughts above as general inspiration but clearly there are questions like the one you bring up where the most important thing will be working in honest good faith. I don?t know if best practices have emerged around things like that, but I am sure that your job is clearly identified as a fair game. Both boys and girls feel the impact of just a moment’s pleasure, for the rest of their lives.

Between that system, the

laulan's picture

Between that system, the taillights – which according to Audi, transforms the car’s “rear end into a large, continuous light surface, with innumerable small points of light flickering 12W LED work light like a swarm of fish” — there’s also headlamp development for which to account. On the volition of the gorgeous, serpentine headlamps on its current models, the brand has gracefully become one of the most coveted imports in suburbs and city alike.

This article opens my mind

Rumah Inicio's picture

This article opens my mind about certain things that I don't know about Linux, thanks for the sharing and I hope I am allowed to return to your page again. How to last longer in bed

Reply to comment | Linux Journal's picture

This post will help the internet visitors for
setting up new website or even a weblog from
start to end.


how to last longer in bed's picture

Heya i'm for the primary time here. I came across this board and I to find It truly helpful & it helped me out a lot. I am hoping to give something again and aid others like you helped me.

more this great post

tipsinfomuda's picture

excellent issues altogether, you simply won a new reader. What would you recommend in regards to your submit that you just made some days in the past? Any certain? Ultrabook Terbaru and also Konsumen Cerdas Paham Perlindungan Konsumen - Iconia PC Tablet dengan Windows 8 - Mau Bikin Website + Hosting Murah AbizZ? Ke aja!

Very efficiently written post

how to last longer in bed's picture

Very efficiently written post. It will be useful to anyone who usess it, including myself. Keep doing what you are doing - i will definitely read more posts.

It will do all the logging

alextony's picture

It will do all the logging and graphing the inexpensive software does but with the Ford specific bundle you can command engine actuators (I think) and run diagnostic routines. Very powerful diagnostic tool. That leaves scanners. There are many many inexpensive ones. Given the other options available, I would skip the cheap ones. Dual Core Tablet PC

Perl is used that way for

CK's picture

Perl is used that way for DECADES, pretty hany with it perl -ne 'print' way of calling.
This is one of the few cases where perl is much handy that python.

Incorporating is one of the

incorporate's picture

Incorporating is one of the best ways a business owner can protect his or her personal assets. Most people choose to incorporate solely for this reason, but there are other advantages as well. For example, the corporate business structure saves you money in taxes, provides greater business flexibility, avoids audit chances, better itemization and lets you more easily raise capital.

Thanks for sharing this useable article

raspberryketonesideeffects's picture

You leave so useful information, I'm sure i am very happy to make a high comment here.
raspberryketonesideeffects | meratol reviews | proactols | capsiplex plus reviews |

Unconvencing Objective

Anonymous's picture

Wow! The article topic is about how difficult it is to interpret shell scripts and then renders, yet another convoluted syntactical language. The problem with most script languages are that, they lack strict programming semantics. Granted most newbie programmers are educated using C++, which is main hype around Python, but the real problem of simplicity, isn't being resolved. If a language offers a multitude of ways to code, it forces the programmer to open a book again. How many programmers can honestly say, they are fluent in all high/low level languages? I suggest to you that "C" is the basic language we all know and that, TCL is the most compatible syntactically, is shell universal, not to mention strict and simple. It has all the good qualities of a scripting language, an interpreter (type tclsh), can be used on a shell command line to pipe with other shell commands, has a good GUI package/s, has a great debugger, not to mention adheres to strict syntactical rules. The problem with programmers these days are that they forget the KISS philosophy.

Thanks for sharing this useable article

raspberryketonesideeffects's picture

I think this is a real great article post.Really looking forward to read more. Want more.
raspberryketonesideeffects | meratol reviews | proactols | capsiplex plus reviews |

Thanks for sharing this useable article

raspberryketonesideeffects's picture

I think this is a real great article post.Really looking forward to read more. Want more.
raspberryketonesideeffects | meratol reviews | proactols | capsiplex plus reviews |

It sound that this is a

obdii's picture

It sound that this is a perfect combination between mobile phones and automotive, no doubt, android industries have begun to enter our life, and I have been thinking recently, if Obd2 auto diagnostic software can be integrated into mobile applications, it will greatly change our lives, and even bring about a revolution again. Now many shop began selling android Obd2 auto diagnostic software product, I was no exception, welcome to my shop to see what the latest android Obd2 auto diagnostic software products. VOLVO VCADS Pro 2.4

It seems interesting thanks

Web Design Firm's picture

It seems interesting thanks for sharing this useful information.

It allows you to try the new

car diagnostic tool's picture

It allows you to try the new code, in a rapid way of explanation. This allows developers to modify the idea, without having to write a complete program output to a file.

That is very interesting

reillyariel's picture

That is very interesting Smile I love reading and I am always searching for informative information like this. This is exactly what I was looking for. Thanks for sharing this great article
free cell phone spy software
cell spyware

Aftermarket products can

Tony lincle's picture

Aftermarket products can cause for Check Engine lights to light. This is only caused by products improperly installed or manufactured. We have heard stories about the Check Engine Light illuminating when swapping out exhausts or intakes, but these codes being set will not affect your power output. Truck Diagnostic Software

Very well made and very good

Voyance sérieuse par mail's picture

Very well made and very good records, very pleasant to go good luck soon.

new text to read

Anonymous's picture

This kind of text is invaluable. Where can I get more information? sanscredit

Reply to comment | Linux Journal

Cathern's picture

Nice blog! Is your theme custom made or did you download it from somewhere?

A design like yours with a few simple adjustements would really make my blog jump out.
Please let me know where you got your theme. Appreciate it

Reply to comment | Linux Journal

how pick up women at the club's picture

Your way of telling everything in this paragraph is
genuinely good, all can effortlessly understand it, Thanks a lot.

Reply to comment | Linux Journal

Port Phillip Bay Fishing Charters's picture

Thnx for providing these details within your website.

Reply to comment | Linux Journal

how to pick up girls san andreas's picture

I do not even know how I ended up here, but I thought this
post was great. I don't know who you are but certainly you are going to a famous blogger if you are not already ;) Cheers!

Reply to comment | Linux Journal

skin disorders hypothyroidism's picture

Hi there! I know this iѕ somеwhаt off topic but
I was wonԁering which blog plаtfοrm aге you using for this ωebѕite?

I'm getting sick and tired of Wordpress because I've hаԁ prοblems with hackеrѕ аnԁ I'm looking at alternatives for another platform. I would be awesome if you could point me in the direction of a good platform.

AWK as a half-way solution

Magic Banana's picture

As other readers mention, the proper solution is:
$ sort names.log | uniq -c

Anyway, one often needs some more complicated processing and AWK really is a fantastic language to effectively process text files. It is a mix between the Shell (with $i as the ith field, pipes, redirection, etc.), C (variables, arithmetic, hash maps, loops, etc.) and sed (line matching w.r.t. regexps, functions implementing the sed's 's' command, etc.). The command above becomes, in AWK:
$ awk '{ ++names[$0] }
END { for (name in names) print name, names[name] }' names.log

Pretty straightforward, isn't it?

tiny bug

perfectionatic's picture

Great article. I noticed a tiny bug in the last python script. GMAIL_EMAIL shouldn't be there.


aishavax's picture

Great article! This is the type of information that are meant to be shared around the net.
Disgrace on Google for now not positioning this submit upper!
Come on over and seek advice from my site . Thank you studio lighting and mind machine


Felipe's picture

UNIX/Linux is all about choices. I pick between Python and Bash all the time (with a sprinking of awk). The more serious the task or the more likely the script will have a long life, the more likely it is that I will use Python.

The one thing I will say to support the shell and pipes approach is how easy it is to incrementally debug your work. At any step, replacing the remainder of the pipeline with more gives you a quick way to see how you are doing. While tossing in print statements in Python is not hard, it just is more typing.

Depending on your programming knowledge, the right approach for you will probably be different. But, that said, if you are building something that is not write-only (that is, write the code, run it and throw it away) picking a scripting language such as Python or Ruby will generally pay off in the long run.

Reply to comment | Linux Journal

try this blog page's picture

I every time useԁ to study paragrаph in news papегs but now аs I am
a user of internet thus from now Ι am usіng net
for агticlеs or reνiews, thanks to web.

Perl is used that way for

hmepas's picture

Perl is used that way for DECADES, pretty hany with it perl -ne 'print' way of calling.

This is one of the few cases where perl is much handy that python.

Excelent article. I will be

Anonymous's picture

Excelent article. I will be using more Python in bash soon.

why not +x it's picture

No need to pipe to "python", just make sure you have the hashbang in your .py file, chmod it executable, then you can pipe your file directly to "":

cat nameslog |

Would seem a bit more elegant to me.

Guys, it's really strange to

Anonymous's picture

Guys, it's really strange to post a code in 2013 without _any_ syntax highlighting, especillay in "_linux_journal".

Thanks for the tips

anonymous's picture

I recently made the leap into python programming and I have to say that it is a easy language to use. Forcing code structure in a language was pure genious if you ask me.

I like the fact that core "modules" are included out of the box helps when you need to write a program that must function across hundreds of linux servers. You don't have this luxury with Perl. Using yum or aptget on servers with different OS patch levels, or lacking internet connections, firewall issues, etc is too much of a headache.
Shell scripting can be a pain too depending on what shell you are using (Bash, Korn, etc) and the personal preferences of the admin responsible for that box. The minute differences in syntax can cause hours of troubleshooting due to spaces, braces, brackets, character case...

Python is defintely a great tool to use if you need to write scipts/programs that must be used widely and interpreted by many.

I see the point in showing

Anonymous's picture

I see the point in showing the use of piping, but "cat names.log | sort | uniq" is dumb: it's the same as "sort -u names.log".

Hey of course you are

Richy Delaney's picture

Hey of course you are right,

this was written only to illustrate piping.

Often I find it easier to chain a ton of commands together rather than rememeber each flag for each binary. Even still, your idea is cleaner and better.

python -s

Roberto S.'s picture

I often use Perl with "-e", so I can hack quick snippets of code that do the work easier than some shell scripting. The same is possible in Python, using "python -s". Like:

cat blabla.log | python -s 'import sys; a= ... ;' | less

Which is handier, IMHO, than writing a file for things that you'll use only once or twice.


Anonymous's picture

Use Ruby. Trust me, it's much better than Python! And since Ruby 1.9, it's faster, too, as the default interpreter has changed from the MRI to YARV.

Though honestly, for most things, shell scripts are easier to write than scrips in full-blown programming languages (and the line between programming and scripting is getting increasingly blurred)...

could be done in one line with perl

Anonymous's picture

without even writing a script, using "perl -ne"

sort -u

Anonymous's picture

$ cat names.log | sort | uniq | wc -l


Are you trying to make things look complicated?

sort -u names.log | wc -l

is all you need.

Hey, Thanks for reading the

Richy Delaney's picture


Thanks for reading the article.

sort -u names.log | wc -l

is certainly a nicer way to write that, however it is the way it is in the article for a number of reasons. First I wanted to illustrate piping.

Also, I would mention that "sort -u" seems a little less clear to me than "sort | uniq"

as a result I would be inclined to go for the later. You will find in a lot of the bsd operating systems, the unix commands have less command line arguments.


An excellent post. The post

Raza's picture

An excellent post. The post affects a lot of urgent issues in our minds. We can not be indifferent to these problems. Your article gives the light in which we are able to watch our real life. Keep it up. Find me a lover

Command line argument options

Anonymous's picture

Not to be snarky, but regarding "You will find in a lot of the bsd operating systems, the unix commands have less command line arguments.":

This site is, is it not?

One would think the articles are geared towards the Linux, and not any *nix, community. It doesn't hurt to revise an article after it's been published, especially when it'll remain published indefinitely.

If using shortened switches clouds readability, one may always resort to the longer switches, e.g., --unique rather than -u for the sort command.

uniq -c

Hermann Schwärzler's picture

Thank you for this article. It was really interesting. I am planning to consider python the next time I am going to write a shell-script.

But for the records: You write on Page 1

The uniq command simply removes duplicates but gives no information on how many duplicates there are.

If you use uniq -c the output will be the same as with your every unique value is preceded by the number of its occurrences.


You can try install IPython

Anonymous's picture

You can try install IPython for shell, it very easy

Another cool feature i used

kle_py's picture

Another cool feature i used some time ago is
i could build and test my script in windows, then copy it over to a Linux/Unix box and it worked the same way ;)