Reading File Metadata with extract and libextractor

Don't just guess about a file's characteristics in a search. Use specific extractor plugins to build an accurate database of files.

This is a rather precise description of the text for a German speaker. The supported languages at the moment are Danish (da), German (de), English (en), Spanish (es), Italian (it) and Norwegian (no). Supporting other languages merely is a question of adding free dictionaries in an appropriate character set. Further options are described in the extract man page; see man 1 extract.

Using libextractor in Your Projects

Listing 5 shows the code of a minimalistic program that uses libextractor. Compiling minimal.c requires passing the option -lextractor to GCC. The EXTRACTOR_KeywordList is a simple linked list containing a keyword and a keyword type. For details and additional functions for loading plugins and manipulating the keyword list, see the libextractor man page, man 3 libextractor. Java programmers should know that a Java class that uses JNI to communicate with libextractor also is available.

Writing Plugins

The most complicated thing about writing a new plugin for libextractor is writing the actual parser for a specific format. Nevertheless, the basic pattern is always the same. The plugin library must be called, where XXX denotes the file format of the plugin. The library must export a method libextractor_XXX_extract, with the following signature shown in Listing 6.

The argument filename specifies the name of the file being processed. data is a pointer to the typically mmapped contents of the file, and size is the file size. Most plugins do not make use of the filename and simply parse data directly, starting by verifying that the header of the data matches the specific format.

prev is the list of keywords extracted so far by other plugins for the file. The function is expected to return an updated list of keywords. If the format does not match the expectations of the plugin, prev is returned. Most plugins use a function such as addKeyword (Listing 7) to extend the list.

A typical use of addKeyword is to add the MIME type once the file format has been established. For example, the JPEG-extractor (Listing 8) checks the first bytes of the JPEG header and then either aborts or claims the file to be a JPEG. The strdup in the code is important, because the string will be deallocated later, typically in EXTRACTOR_freeKeywords(). A list of supported keyword classifications, in the example EXTRACTOR_MIMETYPE can be found in the extractor.h header file.



Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Test this tool online

Anonymous's picture

Online metadata reader is using libextractor. Might be handy if someone wants to test the results first without installing libextractor.

Can this extract index

Anonymous's picture

Can this extract index information of PDF files?

Extracting titles from word documents on linux

Anonymous's picture


A quick scan of the examples on this page initally made it seem to me as if the extract program does not support Microsoft Word Documents.

Closer inspection reveals that extracting metadata from Office documents is supported.

[foo@localhost ~]$ extract foo.doc
mimetype - application/
os - Win32
organization - Foo Publishing
page count - 1
modification date - Tue Sep 6 16:10:00 2005
software - Microsoft Office Word
version - 3
format - ABC123
keywords - SCADA, Cryptographic Protection, Communications
author - ABC123 Task Group
subject - Cryptographic Protection of SCADA Communications
title - ABC123 Draft 3
[foo@localhost ~]$

Missing strdup()?

Aron Stansvik's picture

"The strdup in the code is important, because the string will be deallocated later, typically in EXTRACTOR_freeKeywords()."

If that strdup() is so important, then where is it? ;)

strdup necessary

Mike W's picture

The strdup() referred to is in Listing 8 !!
R-E-A-D M-O-R-E C-A-R-E-F-U-L-L-Y !

Right there?

Christian_Grothoff's picture

The strdup can either be in addKeyword or, as in the article, before the call to addKeyword:


I'm also not aware of any strdup's missing (at the moment) in the actual source, so I'm not sure what your comment refers to. :-)

Geek Guide
The DevOps Toolbox

Tools and Technologies for Scale and Reliability
by Linux Journal Editor Bill Childers

Get your free copy today

Sponsored by IBM

8 Signs You're Beyond Cron

Scheduling Crontabs With an Enterprise Scheduler
On Demand
Moderated by Linux Journal Contributor Mike Diehl

Sign up now

Sponsored by Skybot