Book Excerpt: DevOps Troubleshooting: Linux Server Best Practices
What you will notice as you refresh this page is that the objects in the scoreboard should change during each request. This scoreboard is handy when you want to keep an eye on your web server; just continually refresh the page. During a spike, you are able to see new processes get spawned, switch to W to serve the request, and then, if the spike in traffic subsides, those processes slowly change to _ and ultimately . as they are no longer needed.
Generally speaking, when you access the server status page, you do so from the command line while logged into the web server. This lets you restrict what hosts can view the page while still providing all the information you need. Now, by default, if you were to run curl against the regular server status page, you would get HTML output. However, if you pass the auto option to the server-status page, you will get text output that’s more useful for both command-line viewing and parsing by scripts:
$ curl http://localhost/server-status?auto % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 117 586 117 586 0 0 2579 0 --:--:-- --:--:-- --:--:-- 2579117 586 11 7 586 0 0 1905 0 --:--:-- --:--:-- --:--:-- 0 Total Accesses: 2343235 Total kBytes: 265925501 CPULoad: .0773742 Uptime: 6801454 ReqPerSec: .34452 BytesPerSec: 40036.7 BytesPerReq: 116210 BusyWorkers: 53 IdleWorkers: 28 Scoreboard: WW_W__W_W__W_K_W_W_K___WWWWW_WWKWWWW_WWWWWWWWWWWWWWWWWWKKKKKK_KW_.WC.CW_____K__ ____....................................................................................... ........................................................................................... ........................................................................................... ................................................
When you want to monitor the status page of a server in the command line, although you could just run the curl command over and over, you could use a handy command called watch, which will run whatever command you specify every x number of seconds (by default 2). So if you wanted to keep an eye on the status page and have it refresh every 5 seconds on the command line, you would type
$ watch -n 5 'curl http://localhost/server-status?auto'
To exit out of watch, just hit Ctrl-C.
Solve Common Web Server Problems
Although it’s difficult to document how to solve any and all web server problems, you are likely to run into a few general classes of problems that have identifiable symptoms. This section will highlight some of the common types of problems you may find, their symptoms, and how to remedy them.
One common and relatively simple problem to identify in a web server is a configuration problem. Since web servers need to be reloaded to take on changes in their configuration, it can be tempting to make many changes without reloading the web server; however, if you do so, you are likely to find out during a server maintenance (or when you need to restart the server to load new SSL certificates) that there’s some syntax error in your configuration files and your server will refuse to start.
Both Apache and Nginx validate their configuration files when you start, restart, or reload the service, so that’s one way to find configuration errors—unfortunately, it also means that in the case of a problem, the server is down while you fix the errors. Fortunately, both web servers provide means to test configuration syntax and highlight any syntax errors while the server is still running.
In the case of Apache, the command is apache2ctl configtest. Be sure to run this command as a user who can read all of the configuration files (probably the root user). A successful run looks like this:
$ sudo apache2ctl configtest Syntax OK
When there is a syntax error, this command will identify the file and line number of the error so it’s easy to find:
$ sudo apache2ctl configtest apache2: Syntax error on line 233 of /etc/apache2/apache2.conf: Could not open configuration ↪file /etc/apache2/conf/: No such file or directory
In this case, the configuration file had a typo—the directory you wanted to include was /etc/apache2/conf.d.
Nginx also provides a syntax check by running nginx -t:
$ sudo nginx -t the configuration file /etc/nginx/nginx.conf syntax is ok configuration file /etc/nginx/nginx.conf test is successful
As with Apache, when Nginx detects an error, it tells you the file and line number:
$ sudo nginx -t [emerg]: unknown directive "included" in /etc/nginx/nginx.conf:13 configuration file /etc/nginx/nginx.conf test failed
A common headache, especially for new web server administrators, is permission problems on the web server. Although both Apache and Nginx’s initial processes run as root, all subprocesses that actually do the work of serving content run as a user with more restricted permissions—usually a user like www-data or apache. If you are, for instance, uploading web pages as a different user, you may initially run into permissions problems until you make sure that each file you want to serve is readable by the www-data or apache user.
So what does a permissions problem look like from the outside? This example takes a basic Nginx setup and changes the permissions on the main index.html file so that it is no longer readable by the world. Then it uses curl to attempt to load the page:
$ curl http://localhost <html> <head><title>403 Forbidden</title></head> <body bgcolor="white"> <center><h1>403 Forbidden</h1></center> <hr><center>nginx/0.7.65</center> </body> </html>
The output from the web page tells us the HTTP error even without having to tell curl to display it: a 403 Forbidden error. Unfortunately, although we can see the page is forbidden, from this output, we’re not yet sure why. At this point, though, we would turn to the Nginx error logs and see
2012/07/07 16:13:37 [error] 547#0: *2 open() "/var/www/nginx-default/index.html" failed ↪(13: Permission denied), client: 127.0.0.1, server: localhost, request: "GET / ↪HTTP/1.1", host: "localhost"
This error log lets us know that Nginx attempted to open /var/www/nginx-default/index.html, but permission was denied. At this point, we could check out the permissions of that file and confirm that it wasn’t readable by the www-data user Nginx runs as:
$ ps -ef | grep nginx root 545 1 0 15:19 ? 00:00:00 nginx: master process /usr/sbin/nginx www-data 547 545 0 15:19 ? 00:00:00 nginx: worker process $ ls -l /var/www/nginx-default/index.html -rw-r----- 1 root root 151 2006-08-30 03:39 /var/www/nginx-default/index.html
In this case, you could fix this permission problem with the chmod o+r command, which would add world read permissions to the file. Alternatively, you could also change the group ownership of the file so it was owned by the www-data group (or by a group that www-data was a member of).
Although some administrators may sidestep permissions problems by basically making all files readable and writeable by everyone, the security risks of doing so aren’t worth the easy fix. Instead, consider creating a group on the system whose membership includes both www-data or apache users (depending on what user your web server runs as) and the users you upload files as. If you do try the “chmod 777” method of making the file readable by everyone, use it only as a temporary sanity check to confirm that the problem truly is a permissions issue. Be sure after you have solved the problem to change permissions back to something more secure.
Kyle Rankin is a systems architect; and the author of DevOps Troubleshooting, The Official Ubuntu Server Book, Knoppix Hacks, Knoppix Pocket Reference, Linux Multimedia Hacks, and Ubuntu Hacks.
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?
|Designing Electronics with Linux||May 22, 2013|
|Dynamic DNS—an Object Lesson in Problem Solving||May 21, 2013|
|Using Salt Stack and Vagrant for Drupal Development||May 20, 2013|
|Making Linux and Android Get Along (It's Not as Hard as It Sounds)||May 16, 2013|
|Drupal Is a Framework: Why Everyone Needs to Understand This||May 15, 2013|
|Home, My Backup Data Center||May 13, 2013|
- Designing Electronics with Linux
- Making Linux and Android Get Along (It's Not as Hard as It Sounds)
- Dynamic DNS—an Object Lesson in Problem Solving
- Using Salt Stack and Vagrant for Drupal Development
- Build a Skype Server for Your Home Phone System
- New Products
- A Topic for Discussion - Open Source Feature-Richness?
- Why Python?
- Validate an E-Mail Address with PHP, the Right Way
- Tech Tip: Really Simple HTTP Server with Python
- Understanding the Linux Kernel
2 hours 1 min ago
4 hours 30 min ago
- Kernel Problem
14 hours 33 min ago
- BASH script to log IPs on public web server
19 hours 38 sec ago
22 hours 36 min ago
- Reply to comment | Linux Journal
23 hours 8 min ago
- All the articles you talked
1 day 1 hour ago
- All the articles you talked
1 day 1 hour ago
- All the articles you talked
1 day 1 hour ago
1 day 6 hours ago