Converting e-Books to Open Formats

E-books are a disappointing flurry of vendor-specific formats. Get them converted to HTML to view on your choice of device.

Books in digital format, also known as e-books, can be read on devices lacking the power and screen space to afford a regular Web browser. Several publishers, not to mention projects such as Project Gutenberg, have provided thousands of new and classic titles in digital format. The problem is both the hardware—be it generic PDAs or dedicated devices—and the whole e-book publishing industry are much more fragmented than are PCs and Web browsers. Therefore, it is probable that the e-book you recently bought will not be readable ten years from now—nor tomorrow, should you decide to use a laptop or change PDAs. To help combat this fragmentation, this article discusses some existing command-line tools that can convert the most popular e-book formats to ASCII or HTML.

Practically no tools exist now to export e-book formats to PDF or OpenDocument, the new OASIS standard used in OpenOffice.org, but this is not necessarily a big deal. Once text is in ASCII or HTML format, it easily can be moved to plain-text or PDF format by using a text browser such as w3m or programs such as html2ps. If you go this route for conversion, you are able to do it today, and because it's an open format, 20 years from now too.

PalmDoc

On PalmOS, the original and most common e-book format is PalmDoc, also called AportisDoc or simply Doc, even though it has nothing to do with Microsoft Word's .doc format. Doc, recognizable by the extensions .pdb (Palm Database) or .prc (Palm Resource Code), basically is a PalmPilot database composed of records strung together. This standard has spun off several variants, including MobiPocket, which adds embedded HTML markup tags to the basic format.

Each Palm e-book is divided into three sections: the header, a series of text records and a series of bookmark records. Normally, the header is 16 bytes wide. Some Doc readers may extend the width at run time to hold additional custom information. By default, the header contains data such as the total length of the uncompressed text, the position currently viewed in the document and an array of two-byte unsigned integers giving the uncompressed size of each text record. Usually, the maximum size for this kind of records is 4,096 bytes, and each one of them is compressed individually.

The bookmark records are composed of a 16-byte name and a 4-byte offset from the beginning of text. Because bookmarks are optional, many Doc e-books don't contain them, and most Doc readers support alternative—that is, non-portable—methods to specify them. Other reader-specific extensions might include category, version numbers and links between e-books. Almost always, this information is stored outside the .pdb or .rc file. Therefore, you should not expect to preserve this kind of data when converting your e-books.

Pyrite Publisher, formerly Doc Toolkit, is a set of content conversion tools for the Palm platform. Currently, only some text formats can be converted, but functionality can be extended to support new ones by way of Python plugins. Pyrite Publisher can download the documents to convert directly from the Web; it also can download set bookmarks directly to the output database. The package, which requires Python 2.1 or greater, can be used from the command line or through a wxWindows-based GUI. The software is available for Linux and Windows in both source and binary format. Should you choose the latter option, remember that compiled versions expect Python to be in /usr. The Linux version can install converted files straight to the PDA using JPilot or pilot-link.

Pyrite installed and ran flawlessly on Fedora Core 2. Unlike the other command-line converters presented below, however, Pyrite can save only in ASCII format, not in HTML. The name of the executable is pyrpub. The exact command for converting .pdb files uses this syntax:


pyrpub -P TextOutput -o don_quixote.txt \
Don_Quixote.pdb

Pyrite can be enough if all you want to do is quickly index a digital library. On the other hand, it is almost trivial to reformat the result to make it more readable in a browser. The snippet of Perl code in Listing 1, albeit ugly, was all it took to produce the version of Don Quixote shown in Figure 1.

Figure 1. A PalmDoc file converted to HTML for viewing in a browser.

The script loads the whole ASCII text previously generated with Publisher, and every time it finds two new lines in a row, it replaces them with HTML paragraph markers. The result then is printed to standard output and properly formatted as basic HTML. To change justification, fonts and colors, you simply need to paste your favourite stylesheet right after the <html><body> line.

OpenOffice.org 2.0, expected to be released in spring 2005, will be able to save text in .pdb format. If it also is able to read such files, its mass conversion feature (File→AutoPilot→Document Converter) would solve the problem nicely. I have tried to do this with the 1.9.m65 preview, but all I got was a General input/output error pop-up message. Hopefully, this functionality will be added to future versions.

______________________

Articles about Digital Rights and more at http://stop.zona-m.net CV, talks and bio at http://mfioretti.com

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Pyrite Publisher

Anonymous's picture

Evidently not all pdb files are created equally, and Pyrite has a certain aversion to Peanut files (PNRdPPrs). It cannot open these files.
error: couldn't find a way to convert from PDB:PPrs/PNRd
I have scoured the web looking for an alternative and have yet to find one.

There is another *open* format, and a reader

Terry's picture

Regarding the author's article in general, and this comment in particular...

"An ebook supplier could charge enough to cover royalties, operating expense and a modest profit. In a model where the ebooks are all in a standard format, they could be formatted as needed, including drm controls, on demand to any format needed by the client for whatever ebook format they needed."

...(and please excuse me that for now, the app is Windows-based).

There is another "open standard" which has been around for a while, but ignored; it is called OpenReader Format.

Meanwhile, there was a free but proprietary app, using its propietary format, called ThoutReader. It was an awesome app (and format) for the Windows platform! It allowed you to do all this, and more:

  • increase and decrease font size
  • search text, table of contents, or both
  • add "notes" (literally notes you write in the middle of the text)
  • share your notes publicly
  • download public notes
  • keep a record of all your e-book downloads in your account
  • re-download both free and commercial downloads from any computer
  • categorize your e-books

There were many free and some decent commercial e-books. The best free ones were technical manuals, such as those for PHP and MySQL. Accounts are free, but credit cards are needed to buy commercial e-books and to upload public "notes".

Then the creators, OSoft, were hired by "Teachers Without Borders" to customize the app for them. At that time, they decided to a) go open-source, b) support the OpenReader Format, and c) rename the app "dotReader" after a pioneering woman in computer science.

dotReader has an area that can be customized via application customization, or by the e-book itself (to support a small advertisement area and bring down the costs of textbooks). dotReader has the old features and some additions. A conversion tool is being developed. It reads OpenReader Format, the old ThoutReader format (buggy, though, depending on the particular ebook), and a few other formats.

Now that it is open-source, someone will port it to Linux, I'm sure. Meanwhile, those who already sold their commercial e-books in ThoutReader format should be the first to offer them in OpenReader Format as well. dotReader allows for DRM support while maintaining an open file format.

Reportedly, textbook publishers really don't like the current cost situation either. Their high costs in the U.S. (as compared to U.K.) are apparently because of the way a traditional sales force is employed (even required) to get the textbooks used by academia and offered in college bookstores, and because of the high travel expenses in a much bigger country.

These publishers' stated goal, reportedly, is to keep textbooks $30 and under.

These developments should give the industry, and software makers, the kick in the pants they need. This is great for the industry and awesome for comsumers!

You can download dotReader from osoft.com

My reader

Jonas Minnberg's picture

I am currently writing my own ebook reader and for that I am using a subset of HTML as source format. I am interested to share ideas on what should be in such a subset (the trick is to support all essential features without making it complex to handle). Feel free to mail me (jonas ALPHA nightmode DOT org) if you are intereseted in this subject.

To comment on the existing formats I must say that PDF is one of the worst because of the great difficulty of converting it back to something managable, even when it is not DRM-protected. And from what I've seend from OPF it looks too big and over-engineered to be handled smoothly.

A format for fiction say, does not need that much formatting - it should not be necessary to implement HTML4.0 just to parse running text...

MobiPocket

Del's picture

Are MobiPocket and Rocket Ebook files really the same thing? Rocket Ebook appears to be a non-DRM'ed format, whereas MobiPocket files are DRM-ed. At least the stores that I've seen selling MobiPocket files appear to need the key off the MobiPocket software you're using (extracted from the PDA) and the files appear to be keyed to this software.

Will the rbmake utilities really touch DRM-ed MobiPocket files?

When open formats are not reader-friendly

Ursula's picture

This is excellent advice for all e-book lovers dreaming of re-reading a book in maybe ten years' time!
Just a note on the side: formats must not only be free but also user-friendly. The Gutenberg project makes so many good books available but they take a lot of time reformatting if you actually want to read them. Here, simple raster images or bit-maps might be the solution. On www.books4free.org you can find copyright-free books in a reader-friendly format.

> Here, simple raster images

Petr Bren's picture

> Here, simple raster images or bit-maps might be the solution.

Blind people could not read these on their braille displays.

for CHM files...

Anonymous's picture

archmage (archmage.sf.net) works great to convert to html and xchm (xchm.sf.net) is a great viewer

Non public book-formats

Mark Springer's picture

I really hate this properitary file formats. Have to install a new viewer all the time 8-(((

The only open format in my eyes is the PDF format. Dont know ... do all the compatible printer-drivers have to pay license fees to Adobe ???

Re: Non public ...

Judy Winter's picture

Open Office uses the PDF-ESImpress. So it looks to me, that there is no patent on this format.

hardware ebook readers

Anonymous's picture

As the owner of an RCA ebook reader, I can say that the hardware is an excellent way to carry large volumes of reference material in a portable manner. Gemstar shot itself in the foot by pricing the ebook versions too high. I use rbmake to convert html ebooks for my reb1100 and it works quite well, however, many titles that I need are pdf format and are not readily converted to html for input to rbmake. I like the idea of a common open standard that can be readily converted to whatever proprietary standard is needed.
On the flip side, the publishing industry favors DRM and proprietary standards. The main problem is that the digital rights are controlled by the publisher and not by the original author/artist. The publishing industry should wake up and realize that the average ebook reading joe can figure out that the ebook publisher doesn't have the manufacturing and distribution costs associated with producing paperback copies of the same material. An ebook supplier could charge enough to cover royalties, operating expense and a modest profit. In a model where the ebooks are all in a standard format, they could be formatted as needed, including drm controls, on demand to any format needed by the client for whatever ebook format they needed.

Same to me: you REALLY miss plucker

Anonymous's picture

with plucker I've got the web in my palm...

From the author, about: missing plucker

M. Fioretti's picture

you REALLY miss plucker
No. Plucker is presented as " the best offline Web and e-book viewer for PalmOS"; that's why, even though I knew it, I didn't mention it.
The theme of the article is how to convert existing ebooks in PDA-only and/or proprietary formats to something open like plain text, HTML, OASIS or (in the future) OEBPS.
Not how to read or port content in open formats to a PDA.
Ciao,
Marco Fioretti

Regarding PalmOS Platform you

Anonymous's picture

Regarding PalmOS Platform you miss plucker...

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState