Filesystem Indexing with libferris
Normalization of EA in a Relational Database
To resolve a query on a normalized EA, four tables are involved. The docmap table stores the file's URL and a synthetic integral key, the docid. The attrmap table stores the name of an EA and assigns a synthetic integral key, the attrid. One of many valuemap tables are used, depending on the type of the value, but they all follow a similar style. For example, strlookup assigns varchar values a synthetic integral key, the vid. And finally, a join table, docattrs, joins a docid with an attrid and a vid to record that a file has an attribute with a given value. Thus, to resolve a query (width<=800) if the width EA is normalized requires looking up the attrid and vid to join in the docattrs table, producing a list of docids that have a width satisfying the query.
Normalized EAs are stored directly into the docmap table as a column. To resolve the width query above, the relational database index on the docmap.width column is used to find the matching docids directly. This normalization is a space-time trade-off. Generally, for EAs for which you are planning to search often, you would consider inlining them. Many EAs that are not part of a stat(2) call or deemed very interesting should be left indexed in the attrmap, valuemap and docattrs tables.
For this example, I use my user name and a dbname of lj. The second command below creates the EA index using the non-interactive fcreate tool. The third command then adds all the JPEG files in my shared image directory to that index. You also can use feaindexadd with the -d option to list file paths explicitly on the command line. Without -d, feaindexadd tries to recurse into the paths you supply:
$ mkdir /tmp/ea-index $ fcreate --create-type=eaindexpostgresql \ --target-path=/tmp/ea-index dbname=lj user=ben # if you have setup new db, append db-exists=1 $ find /usr/share/backgrounds/images \ -name "*.jpg" \ | feaindexadd -P /tmp/ea-index --filelist-stdin
My image directory contains 42 JPEG images. Here, I query the index:
$ feaindexquery -P /tmp/ea-index '(width>=640)' Found 34 matches at the following locations: file:///usr/share/backgrounds/images/dewdop_leaf.jpg ... $ feaindexquery -P /tmp/ea-index '(size>=100k)' Found 42 matches at the following locations: file:///usr/share/backgrounds/images/dewdop_leaf.jpg ... $ feaindexquery -P /tmp/ea-index \ '(&(width<=800)(size>=100k))' Found 19 matches at the following locations: file:///usr/.../images/space/apollo08_earthrise.jpg ...
The EA index query syntax is based on “The String Representation of LDAP Search Filters” as described in RFC 2254. This is a simple syntax, providing a small set of comparative operators to make lvalue operator rvalue terms and a means to combine these terms with Boolean and (&), or (|) and not (!) operations. All terms are contained in parentheses, with operators preceding their arguments. The operators are kept simple: == for equality, <= and >= for value ranges and =~ for regex matches.
The ODBC (optionally) and PostgreSQL (always) EA indexing plugins allow you to store many versions of EAs for a file in the index. Having many versions of metadata for a file allows you to query for files based on the EA values those files once had.
To use this functionality, you have to select a time range to match the search against when querying by using a special EA. The time-restricting EAs are atime, ferris-current-time, multiversion-mtime and multiversion-atime. The last two EAs match against the mtime and atime for the file you are seeking. The ferris-current-time EA for a version of a file's index data is the time when that file was being indexed. If no time range is selected, only the latest version of metadata for each file is considered when executing a query.
Time restrictions can be given as a string, and libferris tries its best to work out the format of your time string. In the tests/timeparsing directory of the libferris distribution is a timeparse tool that accepts time values and tells you what libferris makes of your time string. More details on the permissible time strings are given in the libferris FAQ item (see Resources).
The following example of a time-based query looks for all image files that were indexed over a year ago with a given width range:
$ feaindexquery -P /tmp/ea-index \ '(&(width>=1600)(ferris-current-time<=1 year ago))'
If a large image file was indexed two years ago and subsequently replaced with a thumbnail image and re-indexed, the above query returns the file. This is because one of its versions of metadata matches the given query.
Handling the time restriction for EA queries by using the same interface as querying on EA values allows you to use all the standard query mechanisms to select your matching time range. For example, I could select documents that were indexed in 2003 with a given width or those with a specific owner that were modified in the last month:
## note, all one line $ feaindexquery -P /tmp/ea-index ' (| (& (width>=1600)(ferris-current-time>=begin 2003) (ferris-current-time<=end 2003) ) (& (owner-name==sarusama) (multiversion-mtime>=end last month) ) )
Practical Task Scheduling Deployment
July 20, 2016 12:00 pm CDT
One of the best things about the UNIX environment (aside from being stable and efficient) is the vast array of software tools available to help you do your job. Traditionally, a UNIX tool does only one thing, but does that one thing very well. For example, grep is very easy to use and can search vast amounts of data quickly. The find tool can find a particular file or files based on all kinds of criteria. It's pretty easy to string these tools together to build even more powerful tools, such as a tool that finds all of the .log files in the /home directory and searches each one for a particular entry. This erector-set mentality allows UNIX system administrators to seem to always have the right tool for the job.
Cron traditionally has been considered another such a tool for job scheduling, but is it enough? This webinar considers that very question. The first part builds on a previous Geek Guide, Beyond Cron, and briefly describes how to know when it might be time to consider upgrading your job scheduling infrastructure. The second part presents an actual planning and implementation framework.
Join Linux Journal's Mike Diehl and Pat Cameron of Help Systems.
Free to Linux Journal readers.Register Now!
- SUSE LLC's SUSE Manager
- Murat Yener and Onur Dundar's Expert Android Studio (Wrox)
- My +1 Sword of Productivity
- Managing Linux Using Puppet
- Non-Linux FOSS: Caffeine!
- Doing for User Space What We Did for Kernel Space
- SuperTuxKart 0.9.2 Released
- Google's SwiftShader Released
- Parsing an RSS News Feed with a Bash Script
- Rogue Wave Software's Zend Server
With all the industry talk about the benefits of Linux on Power and all the performance advantages offered by its open architecture, you may be considering a move in that direction. If you are thinking about analytics, big data and cloud computing, you would be right to evaluate Power. The idea of using commodity x86 hardware and replacing it every three years is an outdated cost model. It doesn’t consider the total cost of ownership, and it doesn’t consider the advantage of real processing power, high-availability and multithreading like a demon.
This ebook takes a look at some of the practical applications of the Linux on Power platform and ways you might bring all the performance power of this open architecture to bear for your organization. There are no smoke and mirrors here—just hard, cold, empirical evidence provided by independent sources. I also consider some innovative ways Linux on Power will be used in the future.Get the Guide