Pythonic Parsing Programs

The Cover Your Bases Answer

The simplest solution to parsing is to parse a file-like object. Again for the newbies, I say file-like because Python has duck typing. According to Wikipedia this term, duck typing, comes from the saying of an old poet—James Whitcomb Riley:

When I see a bird that walks like a duck and swims like a duck and quacks like a duck, I call that bird a duck.

But I prefer the scene in Monty Python and the Holy Grail, when wise Bedemir uses his powers of logic to determine whether witches are made of wood and therefore flammable—fit to be burned:


	BEDEMIR:  So, how do we tell whether she is made of wood?
	VILLAGER #1:  Build a bridge out of her.
	...
	BEDEMIR:  Aah, but can you not also build bridges out of stone?
	VILLAGER #2:  Oh, yeah.
	BEDEMIR:  Does wood sink in water?
	VILLAGER #1:  No, no.
	VILLAGER #2:  It floats!  It floats!
	BEDEMIR:  What also floats in water?
	VILLAGER #1:  Bread!
	VILLAGER #2:  Apples!
	VILLAGER #3:  Very small rocks!
	VILLAGER #1:  Cider!
	VILLAGER #2:  Great gravy!
	VILLAGER #1:  Cherries!
	VILLAGER #2:  Mud!
	VILLAGER #3:  Churches -- churches!
	VILLAGER #2:  Lead -- lead!
	ARTHUR:  A duck.
	CROWD:  Oooh.
	BEDEMIR:  Exactly!  So, logically...,
	VILLAGER #1:  If... she.. weighs the same as a duck, she's made of wood.
	BEDEMIR:  And therefore--?
	VILLAGER #1:  A witch!
	

(Courtesy of sacred-texts.com/neu/mphg/mphg.htm)

We can apply the same principle—duck typing—to file objects. If an object behaves like a file—has a read method—then it is a file. By having this interface we can easily deal with file-like objects, but also string data, and filenames as will be illustrated.

Implementing the Interface

Another interesting note from the table above is that most of the parsing mechanisms are not tied to a class. Most are functions that create an object, populate it from data within the file and return the object (ConfigParser.readfp being the exception as it has no function, though the ElementTree module has both a parse function and a parse method on the ElementTree class). In addition isolating parsing from the data adheres to the Single Responsibility Principle.

The author’s opinion is that an overloaded function or method (such as ElementTree.parse which accepts both files and filenames, or pyyaml.load which accepts both strings or files) is a violation of “Explicit is better than implicit” and “In the face of ambiguity, refuse the temptation to guess”. These methods are not taking advantage of duck typing, but magic typing, going beyond Postel’s Law. The Python language makes a hard distinction about the difference between an Iterable and an Iterator. A similar comparison could be made between a filename, and a file object. One is a duck and the other is a duck factory.

A Simple Example

The example below parses the contents of the Unix /etc/passwd file and stores the result in a list within the Passwd object:


	from collections import namedtuple
	
	User = namedtuple('User', 'name, pw, id, gid, groups, home, shell')
	
	class Passwd(object):
	    def __init__(self):
	        self.users = []
	        
	        
	def parse(fp):
		pw = Passwd()
	    for line in fp:
	        data = line.strip().split(':')
	        pw.users.append(User(*data))
	    return pw
	

A couple of notes regarding the implementation:

  • A parse function populates a Passwd object
  • Passwd object has one responsibility, knowing about the users. It is not responsible for parsing

If a user wanted to create a Passwd object from a filename—/etc/passwd—this is easy:


	# from filename
	with open('/etc/passwd') as fin:
	    pw = parse(fin)
	

Neither the parse function nor the Passwd object needs to worry about opening or closing the file.

If the data from the file was already in memory, utilizing the built-in module StringIO enables parsing from that end:


	# from string
	from StringIO import StringIO as sio
	fin = sio('''pulse:x:113:121:PulseAudio daemon,,,:/var/run/pulse:/bin/false\nsaned:x:114:123::/home/saned:/bin/false''')
	pw = parse(fin)
	

Advantages of File-interfaces

Hopefully this long-winded post has provided you some insight into parsing files in Python. Prior art is found in the standard library and 3rd party modules. If you want to be fancy and all accommodating, you can, but in reality you are probably just adding unnecessary complexity.

To review a parsing method that relies only on a file interface is:

  • Scalable - accepts generators, iterators that are potentially very long, in addition to files, sockets, and StringIO
  • Easier to manage - you don’t want to be in the business of handling and closing people’s files
  • Usable for users with a filename
  • Usable for users with data from a file

Go forth and parse!

______________________

Matt Harrison (@__mharrison__) is a Python developer at Fusion-io where he helps build data analysis tools.

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Not work

Cipto Junaedy's picture

hi, iam try it and something error at any line. please review it

"Pythonic Parsing Programs"

JollyOutdoor's picture

"Pythonic Parsing Programs" is a wonderful post, very thanks @Matt Harrison. i'm a php and java fan, i use magento make a site:
http://www.jollyoutdoor.com/

What about error handling?

Dror Harari's picture

Great article - my only issue is the lack of any attention to errors. Errors do occur and while in this particular example (the passwd file) one may assume it is valid (because the system is using it?), this is not the case in general. All too often, parsing failures are ignored resulting in corrupted data or otherwise we get some out-of-context error message 'bad format' or something similar.

Since in the pattern presented there is no way to know from where the data came, at minimum it would be nice to count the lines so the error message could be useful (a column may be even better).

Then the return from the parse function should be (pw,error) so the caller can always get the error (I know it reminds of Go but it is useful).

/d

'bad format' or something

kevinguru's picture

'bad format' or something similar." would you please explain more detail about this ?

When parsing files, you

Dror Harari's picture

When parsing files, you invariably hit upon cases where the input does not adhere to the specified grammar (e.g. when parsing a line of the passwd file, the user ID is specified in Hex rather than in decimal). In such a case, if one does not check, one may end up working on wrong data and the result could be anything from benign to tragic. If one does check (as one should), one need a way to communicate why the parsing failed and where to the caller. That was the essence of my comment (along with the hint on how it can be approached).

/d

Reply to comment | Linux Journal

Aluminium Trawler Yacht's picture

Hi there, I found your blog by the use of Google while looking
for a similar topic, your web site came up, it appears to be like great.
I have bookmarked it in my google bookmarks.
Hi there, simply became alert to your blog through Google, and
found that it's really informative. I am going to watch out for brussels. I'll appreciate should you continue this in future.
A lot of people can be benefited from your writing. Cheers!

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix