LiS: Linux STREAMS
The input/output system in UNIX is far from simple and involves many different modules: networking involves different protocol stages arranged in protocol stacks; terminal I/O involves different “line disciplines” stacked over (perhaps network) devices. All those modules perform some processing on existing I/O data flows.
In Linux and most BSD systems, I/O modules live inside the kernel and the relations between them are more or less hard-wired into the code. As an example, the TCP/IP protocol stack is a carefully programmed set of modules with strong interrelations. It is designed to work well on typical configurations.
STREAMS is a flexible input/output system, initially designed to overcome the inflexibility found in previous UNIX systems (see Resources 1). It is an alternative to sockets and is used in most commercial UNIX versions. Some sort of STREAMS is needed if we ever want networking software from systems such as Solaris, Unixware, etc. to run off-the-shelf on Linux.
A STREAM (see Figure 1) is, in essence, a dynamically configurable stack of modules. Each module does some processing on a data flow as it goes from the device to the user or vice-versa. The user perceives a STREAM as a file. It is handled with the usual open, read, write, ioctl and close system calls. Data written by the user is packaged into messages, which are sent downstream. Data read by the user comes from messages sent upstream by an underlying device driver.
A couple of additional system calls, putpmsg and getpmsg, allow the user to send and receive STREAMS messages directly. Yet another system call, poll, provides an alternate interface for select. Therefore, each STREAM is composed of these elements:
A mandatory STREAM head talks to the user process doing I/O. The head fills the gap between user system calls and the message flow. Thus, a write into the STREAM is handled by the head by sending a message downstream. Conversely, a data message going upstream is used by the head to service read system calls on the STREAM.
A (possibly empty) stack of STREAM modules typically performs some computation on messages passing by and forwards them either upstream or downstream. For example, IP on X.25 encapsulation (IXE) could be implemented as a STREAMS module; IP packets would be (de)encapsulated as they pass by the IXE module. A terminal-line-discipline module is another example; typed characters can be cooked as they cross the line-discipline module. A packet-sniffer module could thus be used as a diagnostic or debugging tool.
A mandatory STREAM driver interconnects the STREAM to the device sitting below it. STREAM drivers can also be software only; for example, a STREAMS driver could be used to implement an SNMP MIB for the kernel, or a driver could be written to emulate the behaviour of a true hardware driver for development purposes.
A nice property of STREAMS is that different modules (or drivers) can be decoupled quite easily. Hence, they could be developed independently by different people who don't know the actual protocol stack where they will be used, provided the interfaces between the various modules and drivers are well-defined. STREAMS includes standard interfaces for the transport, network and data link layers. In addition, modules can be dynamically “pushed” onto (and popped off) the STREAM, which is a very convenient feature.
Finally, special multiplexor drivers allow several STREAMS to be multiplexed into another one (or ones). The ip module in Figure 2 is a multiplexor. In this example, it multiplexes both TCP and IP messages using either an Ethernet driver or an IP-on-X.25 driver. A full STREAMS network can be built (see Figure 2), and many different protocol stacks can be set up dynamically for operation.
|Speed Up Your Web Site with Varnish||Jun 19, 2013|
|Non-Linux FOSS: libnotify, OS X Style||Jun 18, 2013|
|Containers—Not Virtual Machines—Are the Future Cloud||Jun 17, 2013|
|Lock-Free Multi-Producer Multi-Consumer Queue on Ring Buffer||Jun 12, 2013|
|Weechat, Irssi's Little Brother||Jun 11, 2013|
|One Tail Just Isn't Enough||Jun 07, 2013|
- Speed Up Your Web Site with Varnish
- Containers—Not Virtual Machines—Are the Future Cloud
- RSS Feeds
- Lock-Free Multi-Producer Multi-Consumer Queue on Ring Buffer
- Non-Linux FOSS: libnotify, OS X Style
- Linux Systems Administrator
- Weechat, Irssi's Little Brother
- Senior Perl Developer
- Technical Support Rep
- UX Designer
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?