Pythonic Parsing Programs
The Cover Your Bases Answer
The simplest solution to parsing is to parse a file-like object. Again for the newbies, I say file-like because Python has duck typing. According to Wikipedia this term, duck typing, comes from the saying of an old poet—James Whitcomb Riley:
When I see a bird that walks like a duck and swims like a duck and quacks like a duck, I call that bird a duck.
But I prefer the scene in Monty Python and the Holy Grail, when wise Bedemir uses his powers of logic to determine whether witches are made of wood and therefore flammable—fit to be burned:
BEDEMIR: So, how do we tell whether she is made of wood? VILLAGER #1: Build a bridge out of her. ... BEDEMIR: Aah, but can you not also build bridges out of stone? VILLAGER #2: Oh, yeah. BEDEMIR: Does wood sink in water? VILLAGER #1: No, no. VILLAGER #2: It floats! It floats! BEDEMIR: What also floats in water? VILLAGER #1: Bread! VILLAGER #2: Apples! VILLAGER #3: Very small rocks! VILLAGER #1: Cider! VILLAGER #2: Great gravy! VILLAGER #1: Cherries! VILLAGER #2: Mud! VILLAGER #3: Churches -- churches! VILLAGER #2: Lead -- lead! ARTHUR: A duck. CROWD: Oooh. BEDEMIR: Exactly! So, logically..., VILLAGER #1: If... she.. weighs the same as a duck, she's made of wood. BEDEMIR: And therefore--? VILLAGER #1: A witch!
(Courtesy of sacred-texts.com/neu/mphg/mphg.htm)
We can apply the same
principle—duck typing—to file objects. If an object behaves like
a file—has a
method—then it is a file. By having this interface we can easily
deal with file-like objects, but also string data, and filenames as
will be illustrated.
Implementing the Interface
Another interesting note from
the table above is that most of the parsing mechanisms are not tied
to a class. Most are functions that create an object, populate it
from data within the file and return the object (
being the exception as it has no function, though the
module has both a
function and a
method on the
class). In addition isolating parsing from the data adheres to the
Single Responsibility Principle.
The author’s opinion is that
an overloaded function or method (such as
which accepts both files and filenames, or
which accepts both strings or files) is a violation of “Explicit
is better than implicit” and “In the face of ambiguity, refuse
the temptation to guess”. These methods are not taking advantage
of duck typing, but magic typing, going beyond Postel’s
Law. The Python language makes a hard distinction about the
difference between an Iterable and an Iterator. A
similar comparison could be made between a filename, and a file
object. One is a duck and the other is a duck factory.
A Simple Example
The example below parses the
contents of the Unix
file and stores the result in a list within the
from collections import namedtuple User = namedtuple('User', 'name, pw, id, gid, groups, home, shell') class Passwd(object): def __init__(self): self.users =  def parse(fp): pw = Passwd() for line in fp: data = line.strip().split(':') pw.users.append(User(*data)) return pw
A couple of notes regarding the implementation:
parsefunction populates a
Passwdobject has one responsibility, knowing about the users. It is not responsible for parsing
If a user wanted to create a
object from a filename—
# from filename with open('/etc/passwd') as fin: pw = parse(fin)
function nor the
object needs to worry about opening or closing the file.
If the data from the file was
already in memory, utilizing the built-in module
enables parsing from that end:
# from string from StringIO import StringIO as sio fin = sio('''pulse:x:113:121:PulseAudio daemon,,,:/var/run/pulse:/bin/false\nsaned:x:114:123::/home/saned:/bin/false''') pw = parse(fin)
Advantages of File-interfaces
Hopefully this long-winded post has provided you some insight into parsing files in Python. Prior art is found in the standard library and 3rd party modules. If you want to be fancy and all accommodating, you can, but in reality you are probably just adding unnecessary complexity.
To review a parsing method that relies only on a file interface is:
- accepts generators, iterators that are potentially very long, in
addition to files, sockets, and
- Easier to manage - you don’t want to be in the business of handling and closing people’s files
- Usable for users with a filename
- Usable for users with data from a file
Go forth and parse!
Matt Harrison (@__mharrison__) is a Python developer at Fusion-io where he helps build data analysis tools.
Practical Task Scheduling Deployment
July 20, 2016 12:00 pm CDT
One of the best things about the UNIX environment (aside from being stable and efficient) is the vast array of software tools available to help you do your job. Traditionally, a UNIX tool does only one thing, but does that one thing very well. For example, grep is very easy to use and can search vast amounts of data quickly. The find tool can find a particular file or files based on all kinds of criteria. It's pretty easy to string these tools together to build even more powerful tools, such as a tool that finds all of the .log files in the /home directory and searches each one for a particular entry. This erector-set mentality allows UNIX system administrators to seem to always have the right tool for the job.
Cron traditionally has been considered another such a tool for job scheduling, but is it enough? This webinar considers that very question. The first part builds on a previous Geek Guide, Beyond Cron, and briefly describes how to know when it might be time to consider upgrading your job scheduling infrastructure. The second part presents an actual planning and implementation framework.
Join Linux Journal's Mike Diehl and Pat Cameron of Help Systems.
Free to Linux Journal readers.Register Now!
- Google's SwiftShader Released
- SUSE LLC's SUSE Manager
- My +1 Sword of Productivity
- Managing Linux Using Puppet
- Murat Yener and Onur Dundar's Expert Android Studio (Wrox)
- Non-Linux FOSS: Caffeine!
- Interview with Patrick Volkerding
- SuperTuxKart 0.9.2 Released
- Tech Tip: Really Simple HTTP Server with Python
- Parsing an RSS News Feed with a Bash Script
With all the industry talk about the benefits of Linux on Power and all the performance advantages offered by its open architecture, you may be considering a move in that direction. If you are thinking about analytics, big data and cloud computing, you would be right to evaluate Power. The idea of using commodity x86 hardware and replacing it every three years is an outdated cost model. It doesn’t consider the total cost of ownership, and it doesn’t consider the advantage of real processing power, high-availability and multithreading like a demon.
This ebook takes a look at some of the practical applications of the Linux on Power platform and ways you might bring all the performance power of this open architecture to bear for your organization. There are no smoke and mirrors here—just hard, cold, empirical evidence provided by independent sources. I also consider some innovative ways Linux on Power will be used in the future.Get the Guide