FS-Cache and FUSE for Media Playback QoS
Listing 5. Entire FUSE Shim C++ Class
#include <fuselagefs/fuselagefs.hh>
using namespace Fuselage;
using namespace Fuselage::Helpers;
#include <aio.h>
#include <errno.h>
#include <string>
#include <iostream>
using namespace std;
...
class CustomFilesystem
:
public Delegatefs
{
typedef Delegatefs _Base;
off_t m_oldOffset;
off_t m_startNextAIOOffset;
enum
{
aio_buffer_sz = 8 * 1024 * 1024,
aio_consume_window = aio_buffer_sz / 2,
debug_readahread_aio = false
};
char aio_buffer[ aio_buffer_sz ];
void schedule_readahread_aio( int fd,
off_t offset, bool forceNewReadAHead )
{
if( m_startNextAIOOffset <= offset
|| forceNewReadAHead )
{
cerr << "Starting an async read request"
<< " at offset:" << offset << endl;
ssize_t retval; ssize_t nbytes;
struct aiocb arg;
bzero( &arg, sizeof (struct aiocb));
arg.aio_fildes = fd;
arg.aio_offset = offset;
arg.aio_buf = (void *) aio_buffer;
arg.aio_nbytes = aio_buffer_sz;
arg.aio_sigevent.sigev_notify = SIGEV_NONE;
retval = aio_read( &arg );
if( retval < 0 )
cerr << "error starting aio request!"
<< endl;
m_startNextAIOOffset = offset
+ aio_consume_window;
if( debug_readahread_aio )
{
while ( (retval = aio_error( &arg ) )
== EINPROGRESS )
{}
cerr << "aio_return():"
<< aio_return( &arg )
<< endl;
}
}
}
public:
CustomFilesystem()
:
_Base(),
m_startNextAIOOffset( 0 ),
m_oldOffset( -1 )
{
}
virtual int fs_read( const char *path,
char *buf, size_t size,
off_t offset, struct fuse_file_info *fi)
{
cerr << "fs_read() offset:" << offset
<< " sz:" << size << endl;
int fd = fi->fh;
bool forceNewReadAHead = false;
if( (offset - size) != m_oldOffset )
{
cerr << "possible seek() between read()s!"
<< endl;
forceNewReadAHead = true;
aio_cancel( fd, 0 );
}
schedule_readahread_aio( fd, offset,
forceNewReadAHead );
m_oldOffset = offset;
return _Base::fs_read( path, buf,
size, offset, fi );
}
};
Listing 6. Makefile for the FUSE Shim
nfs-fuse-readahead-shim: nfs-fuse-readahead-shim.cpp
g++ nfs-fuse-readahead-shim.cpp \
-o nfs-fuse-readahead-shim \
-D_FILE_OFFSET_BITS=64 -lfuselagefs
A simple application that reads from a given file at a predetermined rate can verify that the cache is being populated as expected, as shown in Listing 7. There isn't a great deal of error checking going on, but things that would cause grief, such as failed read()s, are reported to the console. The application repeatedly reads 4KB chunks at a time from a nominated file and throws away the result. Every 256KB status is reported, so that the application can be closed knowing roughly what byte of the file was last read.
Listing 7. simpleread.cpp Reads from argv[1] at a Nominated usec Rate in argv[2]
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#include <iostream>
#include <sstream>
using namespace std;
int main( int argc, char** argv )
{
cerr << "opening argv[1]:" << argv[1] << endl;
long offset = 0;
int fd = open( argv[1], O_RDONLY );
unsigned long usec = 10000;
if( argc > 2 )
{
stringstream ss;
ss << argv[2];
ss >> usec;
}
cerr << "using delay of usec:" << usec << endl;
const int bufsz = 4096;
char buf[ bufsz ];
bool error = false;
while( true )
{
ssize_t rc = read( fd, buf, bufsz );
if( rc > 0 )
{
if( error )
{
cerr << "reading resumed" << endl;
}
error = false;
offset += rc;
}
else if( rc == 0 )
{
cerr << "end of file" << endl;
exit(0);
}
else
{
error = true;
cerr << "read error:" << errno
<< " at offset:" << offset
<< endl;
}
usleep( usec );
if( offset % (1024*256) == 0 )
cerr << "offset:" << offset << endl;
}
return 0;
}
As shown in Listing 8, we first clean out the cache directory and restart cachefilesd. Then, the NFS share is mounted and the FUSE shim run against it to create a /Cache-HomeMovies directory. The FUSE executable is told to remain in the foreground, which stops FUSE from running it as a dæmon, allowing standard output and standard error of the FUSE filesystem to be displayed. We use bash to put the nfs-fuse-readahead-shim into the background (though still having its standard outputs redirected into a capture file) and run the simpleread for a little more than 500KB of data. Then, both the simpleread and nfs-fuse-readahead-shim are stopped to investigate whether the cache has been populated as expected.
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Sponsored by AMD
Built-in forensics, incident response, and security with Red Hat Enterprise Linux 6
Every security policy provides guidance and requirements for ensuring adequate protection of information and data, as well as high-level technical and administrative security requirements for a system in a given environment. Traditionally, providing security for a system focuses on the confidentiality of the information on it. However, protecting the data integrity and system and data availability is just as important. For example, when processing United States intelligence information, there are three attributes that require protection: confidentiality, integrity, and availability.
Learn more about catching the bad guy in this free white paper.
Sponsored by DLT Solutions
| Designing Electronics with Linux | May 22, 2013 |
| Dynamic DNS—an Object Lesson in Problem Solving | May 21, 2013 |
| Using Salt Stack and Vagrant for Drupal Development | May 20, 2013 |
| Making Linux and Android Get Along (It's Not as Hard as It Sounds) | May 16, 2013 |
| Drupal Is a Framework: Why Everyone Needs to Understand This | May 15, 2013 |
| Home, My Backup Data Center | May 13, 2013 |
- New Products
- Linux Systems Administrator
- Senior Perl Developer
- Technical Support Rep
- UX Designer
- Web & UI Developer (JavaScript & j Query)
- Designing Electronics with Linux
- Dynamic DNS—an Object Lesson in Problem Solving
- Using Salt Stack and Vagrant for Drupal Development
- Making Linux and Android Get Along (It's Not as Hard as It Sounds)
Enter to Win an Adafruit Pi Cobbler Breakout Kit for Raspberry Pi

It's Raspberry Pi month at Linux Journal. Each week in May, Adafruit will be giving away a Pi-related prize to a lucky, randomly drawn LJ reader. Winners will be announced weekly.
Fill out the fields below to enter to win this week's prize-- a Pi Cobbler Breakout Kit for Raspberry Pi.
Congratulations to our winners so far:
- 5-8-13, Pi Starter Pack: Jack Davis
- 5-15-13, Pi Model B 512MB RAM: Patrick Dunn
- 5-21-13, Prototyping Pi Plate Kit: Philip Kirby
- Next winner announced on 5-27-13!
Featured Jobs
| Linux Systems Administrator | Houston and Austin, Texas | Host Gator |
| Senior Perl Developer | Austin, Texas | Host Gator |
| Technical Support Rep | Houston and Austin, Texas | Host Gator |
| UX Designer | Austin, Texas | Host Gator |
| Web & UI Developer (JavaScript & j Query) | Austin, Texas | Host Gator |
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?




1 hour 18 min ago
3 hours 11 min ago
10 hours 5 min ago
10 hours 21 min ago
12 hours 12 min ago
18 hours 4 min ago
22 hours 36 min ago
22 hours 37 min ago
1 day 37 min ago
1 day 9 hours ago