Python Scripts as a Replacement for Bash Utility Scripts
To demonstrate the power of combining Python scripts in a modular and
piped fashion, let's expand further on the problem space. Let's find
the top five users of the service.
head is a command
that allows you to
specify a certain number of lines to display of the standard input it
is given. Adding this to the command chain gives the following:
$ cat names.log | python namescount.py | sort -rn | head -n 5
This prints only the top five users and ignores the rest. Similarly, to
get the five users who use the service least, you can use the
tail command, which
takes the same arguments. The result of the Python command
being printed to standard output allows you to build and extend upon
To demonstrate the modularity of this script, let's once again change the problem space. The service also generates a comma-separated value (CSV) log file that contains a list of e-mail addresses and the comments that each e-mail address made about the service. Here's an example entry:
"firstname.lastname@example.org", "This service is great."
The task is to provide a way for the service to send a thank-you message to the top ten users in terms of comment frequency. First, you need a script that can read and print a certain column of CSV data. The standard library of Python provides a CSV reader. The Python script below completes this goal:
#!/usr/bin/env python # CSV module that comes with the Python standard library import csv import sys if __name__ == "__main__": # The CSV module exposes a reader object that takes # a file object to read. In this example, sys.stdin. csvfile = csv.reader(sys.stdin) # The script should take one argument that is a column number. # Command-line arguments are accessed via sys.argv list. column_number = 0 if len(sys.argv) > 1: column_number = int(sys.argv) # Each row in the CSV file is a list with each # comma-separated value for that line. for row in csvfile: print row[column_number]
This script can parse the CSV data and return in plain text the
column that is supplied as a command-line argument. It uses
Let's add this script to the chain. The new script is chained with the others to print out a list of e-mail addresses and their comment frequencies using the command listed below (the .csv log file is assumed to be called emailcomments.csv and the new Python script, csvcolumn.py):
$ cat emailcomments.csv | python csvcolumn.py | ↪python namescount.py | sort -rn | head -n 5
Next, you need a way to send an e-mail. In the Python standard library of functions, you can import smtplib, which is a module that allows you to connect to an SMTP server to send mail. Let's write a simple Python script that uses this library to send a message to each of the top ten e-mail addresses found already:
#!/usr/bin/env python import smtplib import sys GMAIL_SMTP_SERVER = "smtp.gmail.com" GMAIL_SMTP_PORT = 587 GMAIL_EMAIL = "Your Gmail Email Goes Here" GMAIL_PASSWORD = "Your Gmail Password Goes Here" def initialize_smtp_server(): ''' This function initializes and greets the smtp server. It logs in using the provided credentials and returns the smtp server object as a result. ''' smtpserver = smtplib.SMTP(GMAIL_SMTP_SERVER, GMAIL_SMTP_PORT) smtpserver.ehlo() smtpserver.starttls() smtpserver.ehlo() smtpserver.login(GMAIL_EMAIL, GMAIL_PASSWORD) return smtpserver def send_thank_you_mail(email): to_email = email from_email = GMAIL_EMAIL subj = "Thanks for being an active commenter" # The header consists of the To and From and Subject lines # separated using a newline character header = "To:%s\nFrom:%s\nSubject:%s \n" % (to_email, from_email, subj) # Hard-coded templates are not best practice. msg_body = """ Hi %s, Thank you very much for your repeated comments on our service. The interaction is much appreciated. Thank You.""" % email content = header + "\n" + msg_body smtpserver = initialize_smtp_server() smtpserver.sendmail(from_email, to_email, content) smtpserver.close() if __name__ == "__main__": # for every line of input. for email in sys.stdin.readlines(): send_thank_you_mail(email)
This Python script supports contacting any SMTP server, whether local or remote. For ease of use, I have included Gmail's SMTP server, and it should work, provided you give the scripts the correct Gmail credentials. The script uses the functions provided to send mail in smtplib. This again demonstrates the power of using Python at this level. Something like SMTP interaction is easy and readable in Python. Equivalent shell scripts are messy, and such libraries are not as easily accessible, if they exist at all.
In order to send the e-mails to the top ten users sorted by comment
frequency, first you must isolate only the e-mail column of the output of
column names. To isolate a certain column in Linux, you use the
command. In the example below, the commands are given in two
separate chains. For ease of use, I wrote the output into a temporary
file, which can be loaded into the second chain. This simply makes the
process more readable (the Python script for sending mail is referred
to as sendemail.py):
$ cat emailcomments.csv | python csvcolumn.py | ↪python namescount.py | sort -rn > /tmp/comment_freq $ cat /tmp/comment_freq | head -n 10 | cut -f2 | ↪python sendemail.py
This shows the real power of Python as a utility in a chain of bash commands such as this. Writing scripts that accept input from standard input and write any data out to standard out, allows the developer to chain commands such as these together quickly and easily with a link in the chain often being a Python program. This philosophy of designing a small application that services one purpose fits nicely with the flow of commands being used here.
Richard Delaney is a software engineer with Demonware Ireland. Richard works on back-end Web services using Python and the Django Web framework. He has been an avid Linux user and evangelist for the past five years.
Practical Task Scheduling Deployment
July 20, 2016 12:00 pm CDT
One of the best things about the UNIX environment (aside from being stable and efficient) is the vast array of software tools available to help you do your job. Traditionally, a UNIX tool does only one thing, but does that one thing very well. For example, grep is very easy to use and can search vast amounts of data quickly. The find tool can find a particular file or files based on all kinds of criteria. It's pretty easy to string these tools together to build even more powerful tools, such as a tool that finds all of the .log files in the /home directory and searches each one for a particular entry. This erector-set mentality allows UNIX system administrators to seem to always have the right tool for the job.
Cron traditionally has been considered another such a tool for job scheduling, but is it enough? This webinar considers that very question. The first part builds on a previous Geek Guide, Beyond Cron, and briefly describes how to know when it might be time to consider upgrading your job scheduling infrastructure. The second part presents an actual planning and implementation framework.
Join Linux Journal's Mike Diehl and Pat Cameron of Help Systems.
Free to Linux Journal readers.Register Now!
- Murat Yener and Onur Dundar's Expert Android Studio (Wrox)
- SUSE LLC's SUSE Manager
- My +1 Sword of Productivity
- Managing Linux Using Puppet
- Non-Linux FOSS: Caffeine!
- Tech Tip: Really Simple HTTP Server with Python
- SuperTuxKart 0.9.2 Released
- Parsing an RSS News Feed with a Bash Script
- Doing for User Space What We Did for Kernel Space
- Google's SwiftShader Released
With all the industry talk about the benefits of Linux on Power and all the performance advantages offered by its open architecture, you may be considering a move in that direction. If you are thinking about analytics, big data and cloud computing, you would be right to evaluate Power. The idea of using commodity x86 hardware and replacing it every three years is an outdated cost model. It doesn’t consider the total cost of ownership, and it doesn’t consider the advantage of real processing power, high-availability and multithreading like a demon.
This ebook takes a look at some of the practical applications of the Linux on Power platform and ways you might bring all the performance power of this open architecture to bear for your organization. There are no smoke and mirrors here—just hard, cold, empirical evidence provided by independent sources. I also consider some innovative ways Linux on Power will be used in the future.Get the Guide