Reading File Metadata with extract and libextractor

Don't just guess about a file's characteristics in a search. Use specific extractor plugins to build an accurate database of files.

This is a rather precise description of the text for a German speaker. The supported languages at the moment are Danish (da), German (de), English (en), Spanish (es), Italian (it) and Norwegian (no). Supporting other languages merely is a question of adding free dictionaries in an appropriate character set. Further options are described in the extract man page; see man 1 extract.

Using libextractor in Your Projects

Listing 5 shows the code of a minimalistic program that uses libextractor. Compiling minimal.c requires passing the option -lextractor to GCC. The EXTRACTOR_KeywordList is a simple linked list containing a keyword and a keyword type. For details and additional functions for loading plugins and manipulating the keyword list, see the libextractor man page, man 3 libextractor. Java programmers should know that a Java class that uses JNI to communicate with libextractor also is available.

Writing Plugins

The most complicated thing about writing a new plugin for libextractor is writing the actual parser for a specific format. Nevertheless, the basic pattern is always the same. The plugin library must be called libextractor_XXX.so, where XXX denotes the file format of the plugin. The library must export a method libextractor_XXX_extract, with the following signature shown in Listing 6.

The argument filename specifies the name of the file being processed. data is a pointer to the typically mmapped contents of the file, and size is the file size. Most plugins do not make use of the filename and simply parse data directly, starting by verifying that the header of the data matches the specific format.

prev is the list of keywords extracted so far by other plugins for the file. The function is expected to return an updated list of keywords. If the format does not match the expectations of the plugin, prev is returned. Most plugins use a function such as addKeyword (Listing 7) to extend the list.

A typical use of addKeyword is to add the MIME type once the file format has been established. For example, the JPEG-extractor (Listing 8) checks the first bytes of the JPEG header and then either aborts or claims the file to be a JPEG. The strdup in the code is important, because the string will be deallocated later, typically in EXTRACTOR_freeKeywords(). A list of supported keyword classifications, in the example EXTRACTOR_MIMETYPE can be found in the extractor.h header file.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Test this tool online

Anonymous's picture

Online metadata reader is using libextractor. Might be handy if someone wants to test the results first without installing libextractor.

Can this extract index

Anonymous's picture

Can this extract index information of PDF files?

Extracting titles from word documents on linux

Anonymous's picture

WORD DOCS ARE SUPPORTED

A quick scan of the examples on this page initally made it seem to me as if the extract program does not support Microsoft Word Documents.

Closer inspection reveals that extracting metadata from Office documents is supported.


[foo@localhost ~]$ extract foo.doc
mimetype - application/vnd.ms-files
os - Win32
organization - Foo Publishing
page count - 1
modification date - Tue Sep 6 16:10:00 2005
software - Microsoft Office Word
version - 3
format - ABC123
keywords - SCADA, Cryptographic Protection, Communications
author - ABC123 Task Group
subject - Cryptographic Protection of SCADA Communications
title - ABC123 Draft 3
[foo@localhost ~]$

Missing strdup()?

Aron Stansvik's picture

"The strdup in the code is important, because the string will be deallocated later, typically in EXTRACTOR_freeKeywords()."

If that strdup() is so important, then where is it? ;)

strdup necessary

Mike W's picture

The strdup() referred to is in Listing 8 !!
R-E-A-D M-O-R-E C-A-R-E-F-U-L-L-Y !

Right there?

Christian_Grothoff's picture

The strdup can either be in addKeyword or, as in the article, before the call to addKeyword:

addKeyword(&prev,
strdup("image/jpeg"),
EXTRACTOR_MIMETYPE);

I'm also not aware of any strdup's missing (at the moment) in the actual source, so I'm not sure what your comment refers to. :-)

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState