We haven’t seen anyone explain the meaning of following Varnish error message found in the logs of your server in detail. So here it is for the curious:

Dec  5 12:31:31 server varnishd[28446]: Child (28447) not responding to CLI, killing it.
Dec  5 12:31:59 server varnishd[28446]: Child (28447) died signal=3
Dec  5 12:32:03 server varnishd[28446]: child (22431) Started
Dec  5 12:32:04 server varnishd[28446]: Child (22431) said Child starts
Dec  5 12:32:04 server varnishd[28446]: Child (22431) said SMF.s0 mmap'ed 268435456 bytes of 268435456

The message above comes from the Varnish Manager. The Manager is the Varnish process (there are two Varnish processes) that monitors and manages the Varnish caching process. Here it’s telling us that caching process has stopped responding back with a PONG to its PING. A ping from the Manager to the Cache process with its response look like this and happens every 3 seconds (at least in this configuration):

    0 CLI          - Rd ping
    0 CLI          - Wr 200 19 PONG 1323181633 1.0
    0 CLI          - Rd ping
    0 CLI          - Wr 200 19 PONG 1323181636 1.0

So going back to the error message, the Manager is telling us that it hasn’t received a “PONG” back from the Cache process in a while. A “while” is controlled by the variable cli_timeout which has a default of 10 seconds in Varnish 3.x and 20 seconds in Varnish 2.x. So the Manager waits for about 10 seconds before it fires a UNIX kill using signal SIGQUIT (signal=3). Next the Manager starts the child back up again afresh.

You may be asking yourself: why is the Cache process not responding? There are a few of possible causes. Either the cache process is deadlocked (deep code issue) or the server is under a very heavy disk IO load / memory shortage. So how do you determine which category you fall under? I recommend checking the load average and memory utilization levels on your server using Sysstat tools.

Did you know?

UNIXy is a fully managed server and cluster provider. What this means is we don’t expect you to know anything about servers or server management. The good news is it doesn’t cost you extra to have us manage your UNIXy server! Get in touch with us here to get the ball rolling!

 

That’s all folks.

The error below is typical in a new Nginx installation for a busy web site. Nginx apparently hits a user resource as well as configuration limitation:

2010/10/30 19:33:23 [alert] 5554#0: accept() failed (24: Too many open files) while accepting new connection on 0.0.0.0:80

This means that Nginx reached an imposed limit (read: artificial) on the number of files it can open simultaneously. Without further delay, here’s how to fix this error.

The fix is simple and consists of two lines worth of changes. The first change needs to be done in the startup script of Nginx (/etc/init.d/nginx for example). Add the following line at the top right after the shell script interpreter location (shebang):

ulimit -n 65535

Then open the nginx configuration file (default: /etc/nginx/nginx.conf) and the following line right after the “worker_processes” entry line

worker_rlimit_nofile 20480;

Once both changes are made, run the Nginx lint to make sure no errors have been introduced:

# nginx -t

2010/11/01 17:07:46 [info] 9520#0: the configuration file /etc/nginx/nginx.conf syntax is ok

2010/11/01 17:07:46 [info] 9520#0: the configuration file /etc/nginx/nginx.conf was tested successfully

/etc/init.d/nginx restart

Here is a workaround to the Drupal imagecache issue with Nginx running as a reverse proxy to Apache. From Drupal, ImageCache is:

ImageCache allows you to setup presets for image processing. If an ImageCache derivative doesn’t exist the web server’s rewrite rules will pass the request to Drupal which in turn hands it off to ImageCache to dynamically generate the file.

So Nginx shouldn’t try to serve static files from this directory unless they exist. Otherwise, it will generate a 404 and images will break. With this fix, we are catching the 404 generated by Nginx on static files. We are then forwarding the request to Apache (proxy_pass) so it processes it correctly. We are also caching all 404s for 1 minute to avoid slamming Apache. Be sure to “fix” all the other 404 errors so Apache does the least amount work. Most of the time, these 404 errors come from missing favicon.ico or non-manipulated images.

server {

access_log off;

error_log  logs/vhost-error_log warn;

listen    80;

server_name  www.unixy.net;

location @imagecache {

client_max_body_size    10m;

client_body_buffer_size 128k;

proxy_send_timeout   90;

proxy_read_timeout   90;

proxy_buffer_size    4k;

proxy_buffers     16 32k;

proxy_busy_buffers_size 64k;

proxy_temp_file_write_size 64k;

proxy_connect_timeout 30s;

proxy_redirect  http://www.unixy.net:81   http://www.unixy.net;

proxy_pass   http://1.1.1.1:81;

proxy_set_header   Host   $host;

proxy_set_header   X-Real-IP  $remote_addr;

proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;

}

location ~* \.(gif|jpg|jpeg|png|wmv|avi|mpg|mpeg|mp4|js|css|mp3|swf|ico|flv)$ {

root   /home/user/public_html;

error_page    404 = @imagecache;

proxy_cache my-cache;

proxy_cache_valid  200 302  10m;

proxy_cache_valid  404  1m;

}

location / {

client_max_body_size    10m;

client_body_buffer_size 128k;

proxy_send_timeout   90;

proxy_read_timeout   90;

proxy_buffer_size    4k;

proxy_buffers     16 32k;

proxy_busy_buffers_size 64k;

proxy_temp_file_write_size 64k;

proxy_connect_timeout 30s;

proxy_redirect  http://www.unixy.net:81   http://www.unixy.net;

proxy_pass   http://1.1.1.1:81/;

proxy_set_header   Host   $host;

proxy_set_header   X-Real-IP  $remote_addr;

proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;

}

}

That’s all folks. We hope this is useful to someone.

Depending on which version of Varnish you have running, there is a chance the code is still experimental and the whole Varnish daemon is prone to a fatal crash.

Here is a quick and dirty monitoring script for Varnish. It essentially runs as a crontab job and requests a PING response. Should it not receive a PONG from Varnish, it will attempt a restart. There is a second round of testing within the same script. This is a second level of PING request; just in case. Be sure to set the usual defaults like -T port and varnish runlevel script.

#!/bin/bash

result=$(echo -e "ping\n\r" | nc localhost 6082|grep PONG|wc -l);

if [ "${result}" -lt "1" ];
then
/etc/init.d/varnish stop;
sleep 5;
/etc/init.d/varnish start;
echo "$(hostname): Varnish restart" | mail -s "$(hostname): Restarting varnish" alerts@example.com;
fi

sleep 5;

results=$(echo -e "ping\n\r" | nc localhost 6082|grep PONG|wc -l);

if [ "${results}" -lt "1" ];
then
/etc/init.d/varnish stop;
sleep 5;
/etc/init.d/varnish start;
echo "$(hostname): Second Varnish restart" | mail -s "$(hostname): Restarting varnish second time" alerts@example.com;
fi

exit 0;

Finally load the job into crontab:

*/5 * * * * /root/varnish.sh

A cleaner way of accomplishing the same goal would be to establish a permanent connection to the Varnish admin port (6082 in this case) and issue a PING command every other second or so. The interval can be in the sub seconds as opposed to minutes as is the case with crontab.

Here is an example:

# telnet localhost 6082
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
ping
200 19
PONG 1274931430 1.0
ping
200 19
PONG 1274931434 1.0
ping
200 19
PONG 1274931437 1.0
ping
200 19
PONG 1274931443 1.0

We deploy Varnish and other tools on our clients’ server to get the most performance out of hardware. Our clients are happy with the performance boost and we are happy of the end result. Please do get in touch if you have any question or comment

That’s all folks!

Howdy!

The above error can come up quite often on a busy database-driven website or it could happen at one time or the other for different reasons. Today, we are going to cover the main two reasons for it happening. I shall break this down by architecture.

The Out of memory error indicates that the whole system has run out of memory at that point in time. For example, I happened to be loading a customer’s database that required more than 4GB of free RAM on the system. The system had 2GB of free RAM to work with so the MySQL process failed to import the database due to a lack of memory. The solution here is to add more memory into the system.

Here, the error indicates that either the system ran out of memory, just like in x86_64 above, or that the process cannot address more memory even if it is available. This is more peculiar to the architecture than it is a shortcoming of MySQL. 32-bit operating systems are not capable of letting a process address more than roughly 4GB of memory. The best approach here is to install a 64-bit OS that is capable of addressing that much RAM.

Keep in mind that UNIXy fully manages its customers’ servers. Should this particular problem occur while assisting a customer, we are always more than happy to provide a complimentary memory upgrade to complete the task.

That’s all folks!

phpBB is a open source forum software. The board’s index page lists the topics along with a listing of online members. While trouble shooting a customer report of a slower loading of the board “index” page, we found out that the fix in itself is simple and easy to implement. We would like to share this solution in the hope that it will benefit others. It turns out one of the hot spot table, phpbb_users, is not indexed. The performance fix is to simply add an index on column user_email of only 15 characters. Here is an example of the index creation:

create index phpbb_users_email on phpbb_users (user_email(15));

That should do it. Enjoy the quick performance bump! Keep in mind that UNIXy is a fully managed shop and is always happy to lend a hand with systems questions. This is part of our fully managed services guarantee that comes with our VPS and dedicated servers.

That’s all folks!