DIPC: The Linux Way of Distributed Programming
DIPC enables the creation of clusters of PCs. Computers in the same cluster could work together to solve a problem. DIPC's clusters are logical entities, meaning they are independent of any physical network characteristics. Computers could be added or deleted from a cluster without the need to change any of the network parameters. Several clusters may exist in the same physical network, but each computer can belong to at most one of them. Computers on the same cluster can even be connected to each other by a WAN. As far as DIPC is concerned, computers in one cluster never interact with computers in other clusters.
In normal System V IPC, processes specify numerical keys to gain access to the same IPC structure (see Resources 4). They can then use these structures to communicate with each other. A key normally has a unique meaning in only one computer. DIPC makes the IPC keys globally known. Here, if the application programmer wishes, a key can have the same meaning in more than one machine. Processes on different computers can communicate with each other the same way they did in a single machine.
Information about all the IPC keys in use is kept by one of dipcd's processes called the referee. Each cluster has only one referee. In fact, it is having the same referee that places computers in the same cluster. All other processes in the cluster refer to this one to find out if a key is in use. The referee is DIPC's name server. Besides many other duties, the referee also makes sure that only one computer at a time will attempt to create an IPC structure with a given key value, hence the name. Using a central entity simplifies the design and implementation but can become a bottleneck in large configurations. Finding a remedy to this problem is left to the time when DIPC is actually running in such configurations.
Users may need to run some programs (e.g., utilities) in all the computers in the system at the same time, and these programs may need to use the same IPC keys. This could create interference. To prevent any unwanted interactions, distributed IPC structures are declared by programmers. The programmer must specify a flag to do this. The structures are local by default. The mentioned flag is the only thing the programmer should do to create a distributed program. The rest is like ordinary System V IPC programming. Other than this flag to keep DIPC compatible with older programs, the system is totally transparent to programmers.
DIPC's programming model is simple and quite similar to using ordinary System V IPC. First, a process creates and initializes the needed IPC structures. After that, other processes are started to collaborate on the job. All of them can access the same IPC structures and exchange data. These processes are usually executing in remote machines, but they could also be running on the same computer, meaning distributed programs can be written on a single machine and later run on real multi-computers.
An important point about DIPC is that no other UNIX facility is changed to work in a distributed environment. Thus, programmers cannot use system calls, such as fork, which create a process in the local computer.
The fact that DIPC programs use numerical keys to transfer data means they do not need to know where the corresponding IPC structures are located. DIPC makes sure that processes find the needed resources just by using the specified keys. The resources could be located in different computers during different runs of a distributed program. This logical addressing of resources makes the programs independent of any physical network characteristics.
Simple techniques allow the mapping from logical computing resources needed by a program to physical resources to be done with no need to remake the program. As DIPC programs do not need to use any physical network addresses, they do not need recompiling to run in new environments. Of course, this does not prevent the programmer from choosing to make his program dependent on some physical system characteristics. For example, he could hard code a computer address in his code. DIPC programmers are discouraged from doing this type of coding.
When dipcd is not running, the kernel parts of DIPC are short circuited, causing the system to behave like a normal Linux operating system. As a result, users can easily disable the distributed system. Also, normal Linux kernels are not affected by DIPC programs, meaning there is no need to change and recompile these programs when they are to be executed in single computers with no DIPC support.
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?
|Using Salt Stack and Vagrant for Drupal Development||May 20, 2013|
|Making Linux and Android Get Along (It's Not as Hard as It Sounds)||May 16, 2013|
|Drupal Is a Framework: Why Everyone Needs to Understand This||May 15, 2013|
|Home, My Backup Data Center||May 13, 2013|
|Non-Linux FOSS: Seashore||May 10, 2013|
|Trying to Tame the Tablet||May 08, 2013|
- RSS Feeds
- Making Linux and Android Get Along (It's Not as Hard as It Sounds)
- Using Salt Stack and Vagrant for Drupal Development
- New Products
- Validate an E-Mail Address with PHP, the Right Way
- Drupal Is a Framework: Why Everyone Needs to Understand This
- A Topic for Discussion - Open Source Feature-Richness?
- Download the Free Red Hat White Paper "Using an Open Source Framework to Catch the Bad Guy"
- Home, My Backup Data Center
- Tech Tip: Really Simple HTTP Server with Python
- Please correct the URL for Salt Stack's web site
36 min 12 sec ago
- Android is Linux -- why no better inter-operation
2 hours 51 min ago
- Connecting Android device to desktop Linux via USB
3 hours 20 min ago
- Find new cell phone and tablet pc
4 hours 18 min ago
5 hours 46 min ago
- Automatically updating Guest Additions
6 hours 55 min ago
- I like your topic on android
7 hours 42 min ago
- Reply to comment | Linux Journal
8 hours 3 min ago
- This is the easiest tutorial
14 hours 17 min ago
- Ahh, the Koolaid.
19 hours 56 min ago