Three Ways to Web Server Concurrency

A Multithreading Server

The multithreading_server shown in Listing 3 avoids the context-switch downside of the forking_server but faces challenges of its own. Each process has at least one thread of execution. A single multithreaded process has multiple threads. The threading_server is multithreaded.

Listing 3. threading_server.c


#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <netinet/in.h>
#include <signal.h>
#include <pthread.h>
#include "utils.h"

/* thread routine */
void* handle_client(void* client_ptr) {
  pthread_detach(pthread_self()); /* terminates on return */

  /* read/write socket */
  int client = *((int*) client_ptr);

  /* request */
  char buffer[BUFF_SIZE + 1];
  bzero(buffer, sizeof(buffer));
  int bytes_read = recv(client, buffer, sizeof(buffer), 0); 
  if (bytes_read < 0) error_msg("Problem with recv call", false);

  /* response */
  char response[BUFF_SIZE * 2]; 
  bzero(response, sizeof(response));
  generate_echo_response(buffer, response);
  int bytes_written = send(client, response, strlen(response), 0); 
  if (bytes_written < 0) error_msg("Problem with send call", false);

  close(client); 

  return NULL;
} /* detached thread terminates on return */

int main() {  
  char buffer[BUFF_SIZE + 1];      
  
  struct sockaddr_in client_addr;
  socklen_t len = sizeof(struct sockaddr_in);

  /* listening socket */
  int sock = create_server_socket(false);

  /* connections */
  while (true) {
    int client = accept(sock, 
			(struct sockaddr*) &client_addr, 
			&len);
    if (client < 0) error_msg("Problem accepting a 
        ↪client request", true);

    announce_client(&client_addr.sin_addr);

    /* client handler */
    pthread_t tid;
    int flag = pthread_create(&tid,          /* id */
			      NULL,          /* attributes */
			      handle_client, /* routine */
			      &client);      /* routine's arg */
    if (flag < 0) error_msg("Problem creating thread", false);
  } 

  return 0; 
}

The threading_server mimics the division-of-labor strategy in the forking_server, but the client handlers are now threads within a single process instead of forked child processes. This difference is huge. Thanks to COW, separate processes effectively have separate address spaces, but separate threads within a process share one address space.

When a client connects, the threading_server delegates the handling to a new thread:


pthread_create(&tid,          /* id */
	       NULL,          /* attributes */
               handle_client, /* routine */
               &client);      /* arg to routine */

The thread gets a unique identifier and executes a thread routine—in this case, handle_client. The threading_server passes the client socket to the thread routine, which reads from and writes to the client.

How could the WordGame be ported to the forking_server? This server must ensure one WordGame instance per client. The single WordGame:


WordGame game; /* one instance */

could become an array of these:


WordGame games[BACKLOG]; /* BACKLOG == max clients */

When a client connects, the threading_server could search for an available game instance and pass this to the client-handling thread:


int game_index = get_open_game(); /* called in main so thread safe */

In the function main, the threading_server would invoke get_open_game, and each client-handling thread then would have access to its own WordGame instance:


games[game_index].socket = client;   
pthread_create(&tid,                 /* id */
               NULL,                 /* attributes */
               handle_client,        /* routine */
               &games[game_index]);  /* WordGame arg */

A WordGame local to the thread_routine also would work:


void* handle_client(void* client_ptr) {
  WordGame game; /* local so thread safe */
  /* ... */
}

Each thread gets its own copy of locals, which are thereby threadsafe. Of importance is that the programmer rather than the system ensures one WordGame per client.

The threading_server would be more efficient with a thread pool. The pre-forking strategy used in FastCGI for processes extends nicely to threads.

A Polling Server

Listing 4 is a polling_server, which resembles the forking_server in some respects and the threading_server in others.

Listing 4. polling_server.c


#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <netinet/in.h>
#include <signal.h>
#include <sys/epoll.h>
#include <fcntl.h>
#include <errno.h>
#include "utils.h"

#define MAX_BUFFERS (BACKLOG + 1) /* max clients + listener */

int main() {
  char buffer[BUFF_SIZE + 1];      

  /* epoll structures */
  struct epoll_event event,     /* server2epoll */
    event_buffers[MAX_BUFFERS]; /* epoll2server */
  
  int epollfd = epoll_create(MAX_BUFFERS); /* arg just a hint */ 
  if (epollfd &lt; 0) error_msg("Problem with epoll_create", 
      ↪true);

  struct sockaddr_in client_addr;
  socklen_t len = sizeof(struct sockaddr_in);

  int sock = create_server_socket(true); /* non-blocking */

  /* polling */
  event.events = EPOLLIN | EPOLLET; /* incoming, edge-triggered */
  event.data.fd = sock;             /* register listener */
  if (epoll_ctl(epollfd, EPOLL_CTL_ADD, sock, &event) < 0) 
    error_msg("Problem with epoll_ctl call", true);
  
  /* connections + requests */
  while (true) {
    /* event count */
    int n = epoll_wait(epollfd, event_buffers, MAX_BUFFERS, -1);
    if (n < 0) error_msg("Problem with epoll_wait call", true);

    /*
       -- If connection, add to polling: may be none or more
       -- If request, read and echo 
    */
    int i;
    for (i = 0; i < n; i++) {
      /* listener? */
      if (event_buffers[i].data.fd == sock) {
	while (true) {
	  socklen_t len = sizeof(client_addr);
	  int client = accept(sock,
			      (struct sockaddr *) &client_addr, 
			      &len);

	  /* no client? */
	  if (client < 0 && (EAGAIN == errno || 
              ↪EWOULDBLOCK == errno)) break;
	  
	  /* client */
	  fcntl(client, F_SETFL, O_NONBLOCK); /* non-blocking */
	  event.events = EPOLLIN | EPOLLET;   /* incoming, 
                                                 edge-triggered */
	  event.data.fd = client;
	  if (epoll_ctl(epollfd, EPOLL_CTL_ADD, client, &event) < 0)
	    error_msg("Problem with epoll_ctl ADD call", false);	  
	  
	  announce_client(&client_addr.sin_addr);
	}
      }
      /* request */
      else {
	bzero(buffer, sizeof(buffer));
	int bytes_read = recv(event_buffers[i].data.fd, buffer,
        ↪sizeof(buffer), 0); 

	/* echo request */
	if (bytes_read < 0) {
	  char response[BUFF_SIZE * 2]; 
	  bzero(response, sizeof(response));
	  generate_echo_response(buffer, response);
	  int bytes_written = 
	    send(event_buffers[i].data.fd, response, 
            ↪strlen(response), 0); 
	  if (bytes_written < 0) error_msg("Problem with 
              ↪send call", false);
	
	  close(event_buffers[i].data.fd); /* epoll stops 
                                              polling fd */
	}  
      } 
    } 
  } 

  return 0;
}

The polling_server is complicated:


while (true)         /* listening loop */
  for (...)          /* event loop */
     if (...)        /* accepting event? */
       while (true)  /* accepting loop  */
     else            /* request event */

This server executes as one thread in one process and so must support concurrency by jumping quickly from one task (for example, accepting connections) to another (for example, reading requests). These nimble jumps are among nonblocking I/O operations, in particular calls to accept (connections) and recv (requests).

The polling_server's call to accept returns immediately:

  • If there are no clients waiting to connect, the server moves on to check whether there are requests to read.

  • If there are waiting clients, the polling_server accepts them in a loop.

The polling_server uses the epoll system library, declaring a single epoll structure and an array of these:


struct epoll_event              
      event,                     /* from server to epoll */
      event_buffers[MAX_EVENTS]; /* from epoll to server */

The server uses the single structure to register interest in connections on the listening socket and in requests on the client sockets. The epoll library uses the array of epoll structures to record such events. The division of labor is:

  • The polling_server registers events of interest with epoll.

  • The epoll library records detected events in epoll_event structures.

  • The polling_server handles epoll-detected events.

The polling_server is interested in incoming (EPOLLIN) events and in edge-triggered (EPOLLET) rather than level-triggered events. The distinction comes from digital logic design but examples abound. A red traffic light is a level-triggered event signaling that a vehicle should remain stopped, whereas the transition from green to red is an edge-triggered event signaling that a vehicle should come to a stop. The polling_server is interested in connecting and requesting events when these first occur.

A for loop iterates through detected events. Above the loop, the statement:


int n = epoll_wait(epollfd, event_buffers, MAX_EVENTS, -1);

gets an event count, where the events are either connections or requests.

My polling_server takes a shortcut. When the server reads a request, it reads only the bytes then available. Yet the server might require several reads to get the full request; hence, the server should buffer the partials until the request is complete. I leave this fix as an exercise for the reader.

How could the WordGame be ported to the polling_server? This server, like the threading_server, must ensure one WordGame instance per client and must coordinate a client's access to its WordGame. On the upside, the polling_server is single-threaded and thereby threadsafe. Unlike the forking_server, the polling_server does not incur the cost of context switches among forked children.

Conclusions

Which is the best way to client concurrency? A reasoned answer must consider traditional multiprocessing and multithreading, together with hybrids of these. The evented I/O way that epoll exemplifies also deserves study. In the end, the selected method must meet the challenges of supporting concurrency across real-world Web applications under real-world conditions.

Resources

The three Web servers together with an iterative_server are available at http://condor.depaul.edu/mkalin.

For more on Node.js, see: http://nodejs.org.

______________________

Martin Kalin is a professor at the College of Computing and Digital Media at DePaul University, Chicago, Illinois. He earned his PhD at Northwestern University. Martin has co-authored various books on C and C++, authored a book on Java and, most recent

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Gevent

AnilG's picture

Twisted and Node.js have developed the "polling" approach and real life examples always outperform.

Programming is different but complexity now dealt with by libraries such as gevent.

I can't imagine when a process driven server would be preferred unless there was no chance of requiring scale or performance.

Function

garycarnay02's picture

The primary function of a web server is to deliver web pages on the request to clients using the Hypertext Transfer Protocol (HTTP). This means delivery of HTML documents and any additional content that may be included by a document, such as images, style sheets and scripts.

A user agent, commonly a web browser or web crawler, initiates communication by making a request for a specific resource using HTTP and the server responds with the content of that resource or an error message if unable to do so. The resource is typically a real file on the server's secondary memory, but this is not necessarily the case and depends on how the web server is implemented.
Aménagement Sur Mesure

context switch takes 5ms

Anonymous's picture

Your claim that a context switch takes 5 to 10ms (on the optimistic size), can you please enlighten us on the details of your computer.

Though it is one of those things difficult to quantify in all circumstances, it can take anywhere from few hundred nano seconds to few thousand micro seconds in general.

http://www.cs.rochester.edu/u/cli/research/switch.pdf

Great information,hope it

Wild Nat's picture

Great information,hope it works.

http://www.decoralamerica.com

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState