Pythonic Parsing Programs

The Cover Your Bases Answer

The simplest solution to parsing is to parse a file-like object. Again for the newbies, I say file-like because Python has duck typing. According to Wikipedia this term, duck typing, comes from the saying of an old poet—James Whitcomb Riley:

When I see a bird that walks like a duck and swims like a duck and quacks like a duck, I call that bird a duck.

But I prefer the scene in Monty Python and the Holy Grail, when wise Bedemir uses his powers of logic to determine whether witches are made of wood and therefore flammable—fit to be burned:

	BEDEMIR:  So, how do we tell whether she is made of wood?
	VILLAGER #1:  Build a bridge out of her.
	BEDEMIR:  Aah, but can you not also build bridges out of stone?
	VILLAGER #2:  Oh, yeah.
	BEDEMIR:  Does wood sink in water?
	VILLAGER #1:  No, no.
	VILLAGER #2:  It floats!  It floats!
	BEDEMIR:  What also floats in water?
	VILLAGER #1:  Bread!
	VILLAGER #2:  Apples!
	VILLAGER #3:  Very small rocks!
	VILLAGER #1:  Cider!
	VILLAGER #2:  Great gravy!
	VILLAGER #1:  Cherries!
	VILLAGER #2:  Mud!
	VILLAGER #3:  Churches -- churches!
	VILLAGER #2:  Lead -- lead!
	ARTHUR:  A duck.
	CROWD:  Oooh.
	BEDEMIR:  Exactly!  So, logically...,
	VILLAGER #1:  If... she.. weighs the same as a duck, she's made of wood.
	BEDEMIR:  And therefore--?
	VILLAGER #1:  A witch!

(Courtesy of

We can apply the same principle—duck typing—to file objects. If an object behaves like a file—has a read method—then it is a file. By having this interface we can easily deal with file-like objects, but also string data, and filenames as will be illustrated.

Implementing the Interface

Another interesting note from the table above is that most of the parsing mechanisms are not tied to a class. Most are functions that create an object, populate it from data within the file and return the object (ConfigParser.readfp being the exception as it has no function, though the ElementTree module has both a parse function and a parse method on the ElementTree class). In addition isolating parsing from the data adheres to the Single Responsibility Principle.

The author’s opinion is that an overloaded function or method (such as ElementTree.parse which accepts both files and filenames, or pyyaml.load which accepts both strings or files) is a violation of “Explicit is better than implicit” and “In the face of ambiguity, refuse the temptation to guess”. These methods are not taking advantage of duck typing, but magic typing, going beyond Postel’s Law. The Python language makes a hard distinction about the difference between an Iterable and an Iterator. A similar comparison could be made between a filename, and a file object. One is a duck and the other is a duck factory.

A Simple Example

The example below parses the contents of the Unix /etc/passwd file and stores the result in a list within the Passwd object:

	from collections import namedtuple
	User = namedtuple('User', 'name, pw, id, gid, groups, home, shell')
	class Passwd(object):
	    def __init__(self):
	        self.users = []
	def parse(fp):
		pw = Passwd()
	    for line in fp:
	        data = line.strip().split(':')
	    return pw

A couple of notes regarding the implementation:

  • A parse function populates a Passwd object
  • Passwd object has one responsibility, knowing about the users. It is not responsible for parsing

If a user wanted to create a Passwd object from a filename—/etc/passwd—this is easy:

	# from filename
	with open('/etc/passwd') as fin:
	    pw = parse(fin)

Neither the parse function nor the Passwd object needs to worry about opening or closing the file.

If the data from the file was already in memory, utilizing the built-in module StringIO enables parsing from that end:

	# from string
	from StringIO import StringIO as sio
	fin = sio('''pulse:x:113:121:PulseAudio daemon,,,:/var/run/pulse:/bin/false\nsaned:x:114:123::/home/saned:/bin/false''')
	pw = parse(fin)

Advantages of File-interfaces

Hopefully this long-winded post has provided you some insight into parsing files in Python. Prior art is found in the standard library and 3rd party modules. If you want to be fancy and all accommodating, you can, but in reality you are probably just adding unnecessary complexity.

To review a parsing method that relies only on a file interface is:

  • Scalable - accepts generators, iterators that are potentially very long, in addition to files, sockets, and StringIO
  • Easier to manage - you don’t want to be in the business of handling and closing people’s files
  • Usable for users with a filename
  • Usable for users with data from a file

Go forth and parse!


Matt Harrison (@__mharrison__) is a Python developer at Fusion-io where he helps build data analysis tools.


Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Not work

Cipto Junaedy's picture

hi, iam try it and something error at any line. please review it

"Pythonic Parsing Programs"

JollyOutdoor's picture

"Pythonic Parsing Programs" is a wonderful post, very thanks @Matt Harrison. i'm a php and java fan, i use magento make a site:

What about error handling?

Dror Harari's picture

Great article - my only issue is the lack of any attention to errors. Errors do occur and while in this particular example (the passwd file) one may assume it is valid (because the system is using it?), this is not the case in general. All too often, parsing failures are ignored resulting in corrupted data or otherwise we get some out-of-context error message 'bad format' or something similar.

Since in the pattern presented there is no way to know from where the data came, at minimum it would be nice to count the lines so the error message could be useful (a column may be even better).

Then the return from the parse function should be (pw,error) so the caller can always get the error (I know it reminds of Go but it is useful).


'bad format' or something

kevinguru's picture

'bad format' or something similar." would you please explain more detail about this ?

When parsing files, you

Dror Harari's picture

When parsing files, you invariably hit upon cases where the input does not adhere to the specified grammar (e.g. when parsing a line of the passwd file, the user ID is specified in Hex rather than in decimal). In such a case, if one does not check, one may end up working on wrong data and the result could be anything from benign to tragic. If one does check (as one should), one need a way to communicate why the parsing failed and where to the caller. That was the essence of my comment (along with the hint on how it can be approached).


Reply to comment | Linux Journal

Aluminium Trawler Yacht's picture

Hi there, I found your blog by the use of Google while looking
for a similar topic, your web site came up, it appears to be like great.
I have bookmarked it in my google bookmarks.
Hi there, simply became alert to your blog through Google, and
found that it's really informative. I am going to watch out for brussels. I'll appreciate should you continue this in future.
A lot of people can be benefited from your writing. Cheers!

Geek Guide
The DevOps Toolbox

Tools and Technologies for Scale and Reliability
by Linux Journal Editor Bill Childers

Get your free copy today

Sponsored by IBM

8 Signs You're Beyond Cron

Scheduling Crontabs With an Enterprise Scheduler
On Demand
Moderated by Linux Journal Contributor Mike Diehl

Sign up and watch now

Sponsored by Skybot