DIPC: The Linux Way of Distributed Programming
Before Linux, powerful UNIX operating systems were considered a luxury. Linux made it possible for ordinary people to have access to an affordable and reliable computing platform. The only problem is that Linux was originally based on decades-old designs (see Resources 7), making it less attractive for more technically minded users. Linux's answer to this problem is either port and adaptation or introduction of newer concepts.
Building multi-computers (see Resources 1) and programming them are among the more popular research subjects and demand for them is rapidly rising. Any solution to distributed programming under Linux should keep up with one of Linux's more important features: availability to ordinary users.
Linux already had symmetric multi-processing capabilities. However, it did not provide enough standard facilities for distributed software development. Programmers and users had to resort to different add-on packages and various programming models to write or use distributed software. The mechanisms provided by these packages usually differed greatly from one another, each requiring users to learn some new material which was not of any use to them when migrating to other methods. Many also required detailed involvement of the programmer in the process of transferring data over the network; an example is the PVM (Parallel Virtual Machine) software (see Resources 8). The need for a simpler distributed programming model, usable by more programmers, was obvious.
DIPC (Distributed Inter-Process Communication) is a software-only solution for enabling people to build and program multiple computers easily. Each node can be an ordinary personal computer. These nodes must be connected to each other by a TCP/IP (see Resources 3) network. It does not use network broadcasting, which helps it work in networks without such capabilities. Also, no assumption of a synchronized clock is made. These features mean that DIPC could be used in a wide area network (WAN).
Right from the start, it was decided that ease of application programming and the simplicity of the DIPC itself should be among the most important factors in the system design, even if it were to mean some loss in performance. This decision was backed by the observation that computing and telecommunications equipment speeds are improving rapidly, while training and programming times for distributed applications are not.
In DIPC, UNIX System V IPC mechanisms (see Resources 4), consisting of semaphores, messages and shared memories, are modified to function in a network environment. This means that installing DIPC requires changing and recompiling the kernel. Here, the same system calls used to provide communication between processes running on the same computer can be used to allow the communication of processes running on different machines. There is no new system call for the application programmer's use. There is also no library to be linked to the application code, and no need for any modifications in compilers. DIPC could be used with any language that allows access to the operating system's system calls. It is completely camouflaged in the kernel.
The result is that DIPC supports both the message passing and the distributed shared memory paradigms of distributed programming, providing more options for the application programmer (see Resources 5). It also allows the processes to share only selected parts of their address space in order to reduce the problems of false sharing.
It was decided to implement DIPC in the user space as much as possible, with minimal changes to the kernel. This leads to a cleaner and simpler design, but in a monolithic operating system such as Linux it has the drawback of requiring frequent copy operations between kernel and user address spaces (see Resources 2). As UNIX does not allow user space processes to access and change kernel data structures at will, DIPC must have two parts. The more important part is a program named dipcd, which runs with superuser privileges; dipcd forks several processes to do its work. The other part is inside the kernel giving dipcd work and letting it see and manipulate kernel data. The two parts use a private system call to exchange data. This system call must not be used by other processes in the system.
DIPC provides easy data transfer over the network and assumes that the code to use these data already resides at the suitable places. This is justifiable when one considers that in most cases, the program's code changes much less frequently than the data.
DIPC is only concerned with providing mechanisms for distributed programming. The policies, e.g., how a program is parallelized, or where an application program's processes should run, are determined by the programmer or the end user.
Practical Task Scheduling Deployment
July 20, 2016 12:00 pm CDT
One of the best things about the UNIX environment (aside from being stable and efficient) is the vast array of software tools available to help you do your job. Traditionally, a UNIX tool does only one thing, but does that one thing very well. For example, grep is very easy to use and can search vast amounts of data quickly. The find tool can find a particular file or files based on all kinds of criteria. It's pretty easy to string these tools together to build even more powerful tools, such as a tool that finds all of the .log files in the /home directory and searches each one for a particular entry. This erector-set mentality allows UNIX system administrators to seem to always have the right tool for the job.
Cron traditionally has been considered another such a tool for job scheduling, but is it enough? This webinar considers that very question. The first part builds on a previous Geek Guide, Beyond Cron, and briefly describes how to know when it might be time to consider upgrading your job scheduling infrastructure. The second part presents an actual planning and implementation framework.
Join Linux Journal's Mike Diehl and Pat Cameron of Help Systems.
Free to Linux Journal readers.Register Now!
- SourceClear Open
- SUSE LLC's SUSE Manager
- My +1 Sword of Productivity
- Tech Tip: Really Simple HTTP Server with Python
- Parsing an RSS News Feed with a Bash Script
- Doing for User Space What We Did for Kernel Space
- Managing Linux Using Puppet
- Returning Values from Bash Functions
- SuperTuxKart 0.9.2 Released
- Non-Linux FOSS: Caffeine!
With all the industry talk about the benefits of Linux on Power and all the performance advantages offered by its open architecture, you may be considering a move in that direction. If you are thinking about analytics, big data and cloud computing, you would be right to evaluate Power. The idea of using commodity x86 hardware and replacing it every three years is an outdated cost model. It doesn’t consider the total cost of ownership, and it doesn’t consider the advantage of real processing power, high-availability and multithreading like a demon.
This ebook takes a look at some of the practical applications of the Linux on Power platform and ways you might bring all the performance power of this open architecture to bear for your organization. There are no smoke and mirrors here—just hard, cold, empirical evidence provided by independent sources. I also consider some innovative ways Linux on Power will be used in the future.Get the Guide