Python Scripts as a Replacement for Bash Utility Scripts
To demonstrate the power of combining Python scripts in a modular and
piped fashion, let's expand further on the problem space. Let's find
the top five users of the service.
head is a command
that allows you to
specify a certain number of lines to display of the standard input it
is given. Adding this to the command chain gives the following:
$ cat names.log | python namescount.py | sort -rn | head -n 5
This prints only the top five users and ignores the rest. Similarly, to
get the five users who use the service least, you can use the
tail command, which
takes the same arguments. The result of the Python command
being printed to standard output allows you to build and extend upon
To demonstrate the modularity of this script, let's once again change the problem space. The service also generates a comma-separated value (CSV) log file that contains a list of e-mail addresses and the comments that each e-mail address made about the service. Here's an example entry:
"email@example.com", "This service is great."
The task is to provide a way for the service to send a thank-you message to the top ten users in terms of comment frequency. First, you need a script that can read and print a certain column of CSV data. The standard library of Python provides a CSV reader. The Python script below completes this goal:
#!/usr/bin/env python # CSV module that comes with the Python standard library import csv import sys if __name__ == "__main__": # The CSV module exposes a reader object that takes # a file object to read. In this example, sys.stdin. csvfile = csv.reader(sys.stdin) # The script should take one argument that is a column number. # Command-line arguments are accessed via sys.argv list. column_number = 0 if len(sys.argv) > 1: column_number = int(sys.argv) # Each row in the CSV file is a list with each # comma-separated value for that line. for row in csvfile: print row[column_number]
This script can parse the CSV data and return in plain text the
column that is supplied as a command-line argument. It uses
Let's add this script to the chain. The new script is chained with the others to print out a list of e-mail addresses and their comment frequencies using the command listed below (the .csv log file is assumed to be called emailcomments.csv and the new Python script, csvcolumn.py):
$ cat emailcomments.csv | python csvcolumn.py | ↪python namescount.py | sort -rn | head -n 5
Next, you need a way to send an e-mail. In the Python standard library of functions, you can import smtplib, which is a module that allows you to connect to an SMTP server to send mail. Let's write a simple Python script that uses this library to send a message to each of the top ten e-mail addresses found already:
#!/usr/bin/env python import smtplib import sys GMAIL_SMTP_SERVER = "smtp.gmail.com" GMAIL_SMTP_PORT = 587 GMAIL_EMAIL = "Your Gmail Email Goes Here" GMAIL_PASSWORD = "Your Gmail Password Goes Here" def initialize_smtp_server(): ''' This function initializes and greets the smtp server. It logs in using the provided credentials and returns the smtp server object as a result. ''' smtpserver = smtplib.SMTP(GMAIL_SMTP_SERVER, GMAIL_SMTP_PORT) smtpserver.ehlo() smtpserver.starttls() smtpserver.ehlo() smtpserver.login(GMAIL_EMAIL, GMAIL_PASSWORD) return smtpserver def send_thank_you_mail(email): to_email = email from_email = GMAIL_EMAIL subj = "Thanks for being an active commenter" # The header consists of the To and From and Subject lines # separated using a newline character header = "To:%s\nFrom:%s\nSubject:%s \n" % (to_email, from_email, subj) # Hard-coded templates are not best practice. msg_body = """ Hi %s, Thank you very much for your repeated comments on our service. The interaction is much appreciated. Thank You.""" % email content = header + "\n" + msg_body smtpserver = initialize_smtp_server() smtpserver.sendmail(from_email, to_email, content) smtpserver.close() if __name__ == "__main__": # for every line of input. for email in sys.stdin.readlines(): send_thank_you_mail(email)
This Python script supports contacting any SMTP server, whether local or remote. For ease of use, I have included Gmail's SMTP server, and it should work, provided you give the scripts the correct Gmail credentials. The script uses the functions provided to send mail in smtplib. This again demonstrates the power of using Python at this level. Something like SMTP interaction is easy and readable in Python. Equivalent shell scripts are messy, and such libraries are not as easily accessible, if they exist at all.
In order to send the e-mails to the top ten users sorted by comment
frequency, first you must isolate only the e-mail column of the output of
column names. To isolate a certain column in Linux, you use the
command. In the example below, the commands are given in two
separate chains. For ease of use, I wrote the output into a temporary
file, which can be loaded into the second chain. This simply makes the
process more readable (the Python script for sending mail is referred
to as sendemail.py):
$ cat emailcomments.csv | python csvcolumn.py | ↪python namescount.py | sort -rn > /tmp/comment_freq $ cat /tmp/comment_freq | head -n 10 | cut -f2 | ↪python sendemail.py
This shows the real power of Python as a utility in a chain of bash commands such as this. Writing scripts that accept input from standard input and write any data out to standard out, allows the developer to chain commands such as these together quickly and easily with a link in the chain often being a Python program. This philosophy of designing a small application that services one purpose fits nicely with the flow of commands being used here.
Richard Delaney is a software engineer with Demonware Ireland. Richard works on back-end Web services using Python and the Django Web framework. He has been an avid Linux user and evangelist for the past five years.
Free DevOps eBooks, Videos, and more!
Regardless of where you are in your DevOps process, Linux Journal can help!
We offer here the DEFINITIVE DevOps for Dummies, a mobile Application Development Primer, and advice & help from the expert sources like:
- Linux Journal
- Geek Guide: The DevOps Toolbox
- Nmap—Not Just for Evil!
- Download "The DevOps Toolbox: Tools and Technologies for Scale and Reliability"
- High-Availability Storage with HA-LVM
- Resurrecting the Armadillo
- DNSMasq, the Pint-Sized Super Dæmon!
- Real-Time Rogue Wireless Access Point Detection with the Raspberry Pi
- Localhost DNS Cache
- March 2015 Issue of Linux Journal: System Administration
- Days Between Dates: the Counting