In this article, we shall outline the steps required to build a private Content Delivery or Distribution Network (CDN) using Varnish Cache and Nginx. The goal is to build a CDN using free, readily available software but most importantly spend the least amount of funds possible. To this end, all nodes participating in this network are going to be virtual machines (Xen, Virtuozzo, OpenVZ, etc). Should you have any questions or comments on the configuration of this CDN, please post them in this forum: http://www.varnish-cache.info/

Our global CDN can not only keep the latest copy of static files closer to our global visitors but can also cache the most used pages (dynamic or not) in memory on the edge nodes! This means less trips to the geographically distant and slower Dynamic Node (see below). This is similar to what Akamai and other well known firms do, only at a fraction of the cost. However, in this article, and to keep things simple, we will only be caching static files.

Some of you might be surprised to learn that we built this global CDN free of charge for one of our beloved customers. UNIXY offers truly fully managed dedicated servers and clusters with managed servers starting as low as $45. Our motto is simple: what you cannot do with a few mouse clicks, we will gladly do it for you! Please visit us online when you have a chance: http://www.unixy.net. Please do ask if you have any question or comment. No question is minor!

• The Big Picture

The illustration below presents a logical layout of our CDN. Edge nodes can be located just about anywhere in the world. One could also add more nodes at any location should there be a capacity need. The Dynamic Content Node will typically run a mixture of MySQL, Apache, and server-side software built using PHP, Ruby, Python, .Net, or any language for that matter.

Global CDN

Nginx is a lightweight high-performance Web server that is able to handle large traffic consistently. We are leveraging its proxy and caching capabilities. We shall compile Nginx and leverage the proxy module. This module allows us to cache data on the local disks of the remote edge locations.

As its name implies, Varnish Cache is a high-performance caching engine used to keep recently accessed content in memory for fastest access. Varnish is not a Web server. Hence our need to bundle it with Nginx, which is acting as a Web server at the edge nodes. We will cover Varnish in detail in our next installment.

And finally the glue that holds all of these components together: BIND. BIND is the DNS software used to map Internet host names to IP addresses. We shall patch Bind to add geographical filters support. In other words, BIND will serve each client the IP of closest edge node in the CDN. For example, an vistor from Africa will receive the edge node IP of South Africa or Morocco depending on the filters. We will touch on this later.

At a minimum, we will need two nodes to demo and build our private CDN. That’s one Dynamic Content Node and one Edge Location node. The Dynamic Content Node will run the full LAMP stack along with BIND and the geographical filters patch. The Edge Location node will run Nginx and Varnish. One could always run BIND+GeoIP on a separate node as it is good practice. We will assign the hostname dynamic_content to the Dynamic Content Node and edge_node to the Edge Location.

Download BIND from ISC: http://www.bind9.net/download

Download MaxMind’s C API: http://geolite.maxmind.com/download/geoip/api/c/

[root@dynamic_node /]# cd /usr/src/

[root@dynamic_node src]# wget http://mirrors.24-7-solutions.net/pub/isc/bind9/9.2.4/bind-9.2.4.tar.gz

[root@dynamic_node src]# wget http://geolite.maxmind.com/download/geoip/api/c/GeoIP-1.4.6.tar.gz

[root@dynamic_node src]# tar -xzvf bind-9.2.4.tar.gz

[root@dynamic_node src]# tar -xzvf GeoIP-1.4.6.tar.gz

[root@dynamic_node src]# cd GeoIP-1.4.6

[root@dynamic_node GeoIP-1.4.6]# ./configure –prefix=/usr/local/geoip

[root@dynamic_node GeoIP-1.4.6]# make

[root@dynamic_node GeoIP-1.4.6]# make install

[root@dynamic_node GeoIP-1.4.6]# cd ..

[root@dynamic_node src]# patch -p0 < bind-9.2.4-geodns-patch/patch.diff

[root@dynamic_node src]# cd bind-9.2.4

[root@dynamic_node bind-9.2.4]# CFLAGS=”-I/usr/local/geoip/include” LDFLAGS=”-L/usr/local/geoip/lib -lGeoIP” ./configure –prefix=/usr/local/bind

[root@dynamic_node bind-9.2.4]# make

[root@dynamic_node bind-9.2.4]# make install

[root@edge_node src]# wget http://nginx.org/download/nginx-0.8.45.tar.gz

[root@edge_node src]# tar -xzvf nginx-0.8.45.tar.gz

[root@edge_node src]# cd nginx-0.8.45

[root@edge_node nginx-0.8.45]# ./configure –prefix=/usr/local/nginx –with-http_realip_module

[root@edge_node nginx-0.8.45]# make

[root@edge_node nginx-0.8.45]# make install

http {

include       mime.types;

default_type  application/octet-stream;

sendfile        on;

keepalive_timeout  65;

upstream dynamic_node {

server 1.1.1.1:80; # 1.1.1.1 is the IP of the Dynamic Node

}

server {

listen       81;

server_name  cdn.unixy.net;

location ~* \.(gif|jpg|jpeg|png|wmv|avi|mpg|mpeg|mp4|htm|html|js|css|mp3|swf|ico|flv)$ {

proxy_set_header  X-Real-IP  $remote_addr;

proxy_pass http://dynamic_node;

proxy_store /var/www/cache$uri;

proxy_store_access user:rw group:rw all:r;

}

[root@edge_node src]# wget http://downloads.sourceforge.net/project/varnish/varnish/2.1.2/varnish-2.1.2.tar.gz?use_mirror=cdnetworks-us-1&ts=1279434397

[root@edge_node src]# tar -xzvf varnish-2.1.2.tar.gz

[root@edge_node src]# tar -xzvf varnish-2.1.2.tar.gz

[root@edge_node varnish-2.1.2]# ./configure –prefix=/usr/local/varnish

[root@edge_node varnish-2.1.2]# make

[root@edge_node varnish-2.1.2]# make install

• Wrap up

The instructions above can replicated across however many additional Edge Nodes you want to add. One could also add redundancy to the BIND+GeoIP setup by configuring secondary nodes. The illustration below shows the flow of a request from top to bottom.

CDN Request and Response Flow

That’s all folks! I hope you enjoyed this article.

The MySQL query cache is used to store data retrieved using SELECT statements. The advantage of the query cache is that subsequent and identical SELECT statement results are retrieved directly from the query cache, which resides in memory. So instead of executing the SELECT statements again and again, the stream of data is pulled from memory. The end result is lower CPU and disk hits, which can improve server load average and most importantly responsiveness.

The query cache functionality has enough smartness that once configured, it runs itself. MySQL will make sure to flush cached data if there’s an update that takes place. This ensures stale data is flushed out and replaced with fresh entries. There is, however, an extra step that is required. The cache tends to become fragmented over time. But nothing that cannot be addressed with a simple script (see below).

Stock MySQL configurations set the query cache to 32MB. This means that 32MB worth of server memory is pre-allocated to MySQL. If you have a relatively large database, you will most certainly gain from increasing that value. After configuring the query cache, it takes a while for MySQL to build the cache depending on how busy your MySQL server is.

Let’s get the query cache configured. Inside the [mysqld] section in the MySQL configuration file (ex: /etc/my.cnf), configure the options below. For a server with 8GB memory and a very busy database, set the cache to 1GB:

[mysqld]

query_cache_size = 1024M

Then restart MySQL for the new option to take effect. Run this command to verify that the new query cache value is in place and that the query cache feature is indeed enabled:

# mysql -u root -e “show variables like ‘%query%’”
+——————————+————+
| Variable_name                | Value      |
+——————————+————+
| ft_query_expansion_limit     | 20         |
| have_query_cache             | YES        |
| long_query_time              | 10         |
| query_alloc_block_size       | 8192       |
| query_cache_limit            | 1048576    |
| query_cache_min_res_unit     | 4096       |
| query_cache_size             | 1073741824 |
| query_cache_type             | ON         |
| query_cache_wlock_invalidate | OFF        |
| query_prealloc_size          | 8192       |
+——————————+————+

If the have_query_cache option is set to YES, it means that the query cache is indeed up and running. query_cache_size represents the cache size in bytes. You can also monitor the mysqld process using top and you will see its RES value increase over time to above 1gb. This would be a good sign asb MySQL fills up the query cache with result sets from SELECT. There is another way to check on cache usage and health. Run the following command:

# mysql -u root -e “show status like ‘qc%’”
+————————-+———–+
| Variable_name           | Value     |
+————————-+———–+
| Qcache_free_blocks      | 69776     |
| Qcache_free_memory      | 756433256 |
| Qcache_hits             | 9518567   |
| Qcache_inserts          | 3878394   |
| Qcache_lowmem_prunes    | 49447     |
| Qcache_not_cached       | 173233    |
| Qcache_queries_in_cache | 122754    |
| Qcache_total_blocks     | 315609    |
+————————-+———–+

If Qcache_free_memory stays high over a few days means that the query_cache_size value is set too high. A high Qcache_free_blocks means that memory is fragmented. So it is time to coalesce (make memory blocks contiguous) the region. This is done with the following command:

# mysql -u root -p -e “flush query cache”

Contrary to the command name, this does not flush the cache in the sense that all entries become invalid. It simply defragments the memory region that is assigned to the query cache. The above command should be put in a script that run every 6 hours or so. Create a root crontab entry as such:

* 6/* * * * /root/coalesce.sh

Where coalesce.sh is a shell script:

#!/bin/bash

mysql -u root -pyour_root_password -e "flush query cache";

The query cache can be further configured to match your database’s usage pattern. Be sure to read the MySQL documentation for the details. Hopefully, this post will come in handy for those that have unused memory and need to make the most out of their server. As our customer, we configure the query cache on your server from the get-go. Be sure to ask us how we can help with your online project!