We’ve just upgraded our cPanel Varnish plugin from Varnish 2.1.4 to 3.0.0. To kick off the new release I’ve performed a before and after benchmark to get a feel of the performance boost. The results are outstanding! I spun up one VPS, loaded it with our plugin version 1.4.5rc1, and performed a benchmark using ab. Then loaded the same VPS with plugin version 1.4.5rc2 to perform the same benchmark. Keep in mind that these results are relative and that’s what I’m focusing on here.

Here are the VPS specs (typical of medium sized virtual machine):

• 4-core Opteron Magny-Cours (vcpu allocation)
• 1gb ddr3
• thread_pools@2 and thread_pool_min@200
• Custom VCL (same across both experiments)

Here are the results from the first run against 2.1.4 (full ab results):

Concurrency Level:      1600
Time taken for tests:   1.514520 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      3764888 bytes
HTML transferred:       60078 bytes
Requests per second:    6602.75 [#/sec] (mean)
Time per request:       242.323 [ms] (mean)
Time per request:       0.151 [ms] (mean, across all concurrent requests)
Transfer rate:          2427.17 [Kbytes/sec] received

Here are the results from the first run against 3.0.0 (full ab results):

Concurrency Level:      1600
Time taken for tests:   1.371934 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      4001094 bytes
HTML transferred:       60318 bytes
Requests per second:    7288.98 [#/sec] (mean)
Time per request:       219.509 [ms] (mean)
Time per request:       0.137 [ms] (mean, across all concurrent requests)
Transfer rate:          2847.80 [Kbytes/sec] received

It’s important to note that the resources allocated to both experiments are the same and the goal is to stress test 2.1.4 and 3.0.0. Release 3.0.0 is over 600 requests per second or 10% faster than 2.1.4, which is great! Congratulations to the Varnish Cache dev team for job well done :)

That’s all folks!

About UNIXY

UNIXY is a long-time Varnish Cache user and evangelist. They have been offering Varnish acceleration to their clients for more than three years. They have released the first cPanel Varnish plugin as well as spun a new startup,Fastlayer, the on-demand HTTP accelerator for the cloud.

That’s all folks!

Here is a post detailing an interesting setup of Web servers made to go against one another in real time. This could prove to be entertaining for benchmarkers or perhaps decisive for those who need an ultimate killer proof that their favorite Web server can kick some behind.

Definition of server Web benchmarking (according to Wikipedia):

It is the process of estimating a web server performance in order to find if the server can serve sufficiently high workload [sic].

In this article, I’m going to butcher this definition and depart from the tradition – a tad bit. Here are a some interesting questions that come to mind: What if the end goal is not to obtain measurements but to take a knock-out approach? What if we were to put three Web servers all against one another is a threesome match? Is the outcome of this death match meaningful? How do we select the winner? Switching gears forward a bit, could this death match approach be maliciously leveraged in a botnet situation?

Let’s dive right into the code! But first, let’s discuss the logic necessary to trigger, enable, and maintain a match until death occurs or an apparent draw ensues. The death match in this post is triggered via a single GET action from a regular browser and can be sent to any Web server (or all) that is participating in the fight. The trigger calls a server-side python script called bench.py, which in turns forks a threaded python program, called benchlet.py, that perpetuates the  match. Each Web server participating in the death match receives a GET request and then resonates another similar multi-threaded request against the other Web server. This back and forth exchange escalates the match further and further…until death occurs.

The following video illustrates the trigger action and an excerpt from the match.

The mechanism used to trigger and maintain the match is written in python but it could be just about any server-side processing language like PHP or Ruby.

About UNIXy

UNIXy is a fully managed server and cluster services providers with a focus on high traffic Web sites. Server and cluster management is offered courtesy at not extra costs to their clients. They have been offering Varnish acceleration to their clients for more than three years. They have released the first cPanel Varnish plugin as well as spun a new startup, Fastlayer, the on-demand HTTP accelerator for the cloud.

That’s all folks!

If you haven’t had a chance to check on the new features in the next release of Apache’s HTTPD then here is a digest. The most notable changes have been made to the core and those are KeepAliveTimeout and Loadable MPMs. One interesting change is the addition of mod_lua; lua being an embeddable scripting language that is not so popular in the Web app world. Interesting nonetheless…

In Apache 2.2, the KeepAliveTimeout directive tells Apache how many seconds  it should wait for subsequent requests from the same client before it closes the connection. In Apache 2.4, KeepAliveTimeout can be specified in milliseconds. If you’ve ever had to fine tune Apache, you know this is good news. It means Apache can shed and/or recycle idle threads much faster providing more bandwidth to handle more connections.

Also in Apache 2.4, you now have the option to pre-build all MPMs (like prefork, event, and worker) and then switch back and forth amongst them as needed at runtime. This is more of a convenience feature than anything else.

That’s all folks!

Update (2011-03-14): UNIXy has released the first & only cPanel Varnish Plugin. The plugin installs, configures, and makes it very easy to manage Varnish via WHM. The plugin is fully integrated with cPanel EasyApache, cPanel URLs, and everything else that makes a functional cPanel server. Read about it more here: http://www.unixy.net/varnish

While we are in the business of providing managed server and cluster services, some folks still keep it a heart to manage their own dedicated server or vps nodes. Incontestably, and however much you cherish your web servers, family always takes priority. In this post, we are going to cover a peculiar server configuration that can adapt to abnormal traffic without the server dropping down to its knees or you getting involved… at 3AM on new year’s eve.

The goal is to configure such server to be autonomous for weeks or months at a time without requiring your involvement. Based on observatory experience, we decided to develop a 3-state throttle mechanism where each state is triggered based on server vitals like load average (run queue depth).

3-state throttle Web server

Let’s lay down the requirements and goals. We want our shiny adaptable server to be able to:

1. graciously handle C10K challenges
2. handle a Slashdot / Digg effect without dropping off the net
3. change operating state and adapt to the traffic load

Background

Before we delve into the specifics, we’ll need to cover some important aspects of high performance web servers. All web servers, at least the ones worthy to be called as such, must implement the standards defined in RFC2616. What differentiates these web servers, however, is their feature set and their ability to manage large sets of concurrent connections. More specifically, they need to be able to handle the C10K problem gracefully. At least that’s what matters the most to us in this article.

So what is this C10K problem? C10K is a condensed term coined to mean 10000 concurrent clients. A Web server that solves the C10K problem should be able to handle much more than 10000 concurrent connections transferring small-size files on a 1Gbps internet connection using 1GB of memory without causing any noticeable slow down

At a first glance, the C10K challenge appears to be easily achievable. The truth of the matter, the implementation not only requires efficient programming but also a mastery of low level system API and system calls. We’re not going to cover the fine details here. But implementors that focus too much on looking good in benchmarks end up building software that performs poorly in the real world. As a matter of fact, such systems have little practicality in this new hip Web world.

In this new hip online world, serving static files represent a very tiny fraction of the work put forth by a Web server. Server side processing accounts for the majority of the effort. So now not only do we expect a Web server to deliver relatively-simple static objects at sub milisecond speeds but also hold it accountable for the time it takes to complete server-side request. Server-side requests could be anything from PHP, Perl, to Python and Ruby. It doesn’t matter.

Implementation

How are we going to achieve this bullet proof Web server? We have been imposed the following requirement as far as Web server stack:

• Linux 2.6.x
• Apache 2.2.x
• MySQL 5.x
• PHP 5.2.x

While the requirement is reasonable we still need a secret sauce engine that will make a decision for us so we know how to throttle back and forth between the three states. We have very little flexibility in implementing this engine “inside” the LAMP stack as it means fiddling with the client’s environment. We needed a seamless dynamic way of throttling. That is no Web server restarts are permitted and the whole throttling action should take less than a 100ms.

We decided to leverage Varnish Cache’s ability to load and compile a new set of configuration instructions on the fly. This essential function is not the only feature that sold us; the main advantage is that Varnish is a very capable caching engine and satisfies all three of our initial requirements (adaptability, state machine, C10K). We will load three Varnish configuration and these are as follow:

• Light configuration:

This configuration of Varnish caches static files only. We’re still letting Apache do a lot of leg work still and that is fine considering we’re getting low traffic in this state and a low impact on perceivable performance.

• Moderate configuration:

In this state, we enable caching but we respect cache control headers coming out of Apache and PHP. In other words, we cache only what we are told to cache. Sessions do not get cached in this state. Having too many “logged in” users could overpower the server. Hence, the need for a third state.

• Heavy configuration:

Enable caching but still respecting cache headers. Additionally, we enable per-user session caching (cookie based caching). There needs to be plenty of memory to handle as many users as possible since Varnish will be creating a cache entry for each registered user and accessed object.

The above diagram illustrates the “gear switching” logic of our engine. The pieces that we need to implement this logic are the following:

1. The decision engine based on overall system vitals
2. The Varnish configuration loader
3. A daemon to tie it all together

We picked Python as as tool to accomplish the above. We decided to leverage the python-daemon package as well as the python-varnish manager. The former gives the daemon functionality as we intend to run the engine full time in the background. The latter is a simple wrapper around telnetlib and an interface to interact with Varnish’s admin port.

One of the tricky questions we needed answered revolves around the mechanism used to trigger a state transition. We have already decided on the load average being the metric we poll and base our decision upon. What remains to be determined is the value threshold that triggers the transition. A high load average does not always mean a stressed server nor does a low load average always mean an optimal server configuration.

As a rule of thumb, however, and based on experience, the load average is almost always reflective of resource capacity. We we are going to have to trust the kernel on this one. While we have covered the when, the how still needs some demonstration. As mentioned above, Varnish comes with a neat feature that allows it to load and use a new configuration file on the fly. It takes the new configuration, compiles it, and finally loads it up for execution. All of that without dropping a packet. Think of it as importing a python module at run-time. Here is a quick demo:

Here is now where the on-the-fly loading of our config happens. Of course, with our Python daemon program the state transitions are triggered depending on the load:

To recap, we wrote a program, python daemon, that monitors the vitals of a dedicated server and then determines the next step to take. In this case, we have preloaded our caching engine with three configurations: light, moderate, and heavy. These configurations anticipate and handle different levels of traffic and load. Our python program makes use of any of the three configurations by loading them into Varnish.

That’s all folks! We hope you enjoyed this article.

Here is a workaround to the Drupal imagecache issue with Nginx running as a reverse proxy to Apache. From Drupal, ImageCache is:

ImageCache allows you to setup presets for image processing. If an ImageCache derivative doesn’t exist the web server’s rewrite rules will pass the request to Drupal which in turn hands it off to ImageCache to dynamically generate the file.

So Nginx shouldn’t try to serve static files from this directory unless they exist. Otherwise, it will generate a 404 and images will break. With this fix, we are catching the 404 generated by Nginx on static files. We are then forwarding the request to Apache (proxy_pass) so it processes it correctly. We are also caching all 404s for 1 minute to avoid slamming Apache. Be sure to “fix” all the other 404 errors so Apache does the least amount work. Most of the time, these 404 errors come from missing favicon.ico or non-manipulated images.

server {

access_log off;

error_log  logs/vhost-error_log warn;

listen    80;

server_name  www.unixy.net;

location @imagecache {

client_max_body_size    10m;

client_body_buffer_size 128k;

proxy_send_timeout   90;

proxy_read_timeout   90;

proxy_buffer_size    4k;

proxy_buffers     16 32k;

proxy_busy_buffers_size 64k;

proxy_temp_file_write_size 64k;

proxy_connect_timeout 30s;

proxy_redirect  http://www.unixy.net:81   http://www.unixy.net;

proxy_pass   http://1.1.1.1:81;

proxy_set_header   Host   $host;

proxy_set_header   X-Real-IP  $remote_addr;

proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;

}

location ~* \.(gif|jpg|jpeg|png|wmv|avi|mpg|mpeg|mp4|js|css|mp3|swf|ico|flv)$ {

root   /home/user/public_html;

error_page    404 = @imagecache;

proxy_cache my-cache;

proxy_cache_valid  200 302  10m;

proxy_cache_valid  404  1m;

}

location / {

client_max_body_size    10m;

client_body_buffer_size 128k;

proxy_send_timeout   90;

proxy_read_timeout   90;

proxy_buffer_size    4k;

proxy_buffers     16 32k;

proxy_busy_buffers_size 64k;

proxy_temp_file_write_size 64k;

proxy_connect_timeout 30s;

proxy_redirect  http://www.unixy.net:81   http://www.unixy.net;

proxy_pass   http://1.1.1.1:81/;

proxy_set_header   Host   $host;

proxy_set_header   X-Real-IP  $remote_addr;

proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;

}

}

That’s all folks. We hope this is useful to someone.

Update (2011-03-24): In this article. we’re discussing how we leveraged the smart Russian-built Web server Nginx to stop a DDoS attack. We’ve also experimented quite a bit with Varnish, another fine half-Danish half-Norwegian open source software, as a DDoS mitigation tool. It can be made to form the same constellation that we used for Nginx but can perform much better in the face of a large DDoS. So it’s worth pursuing should you be interested (read: desperate). Shameful plug: check out our cPanel Varnish plugin: http://www.unixy.net/varnish

In this post we (UNIXY) are going to share our experience fending off a large Distributed Denial of Service (DDoS) attack for a client. Generally, Website owners deal with DDoS attacks on their own. There are equipment and solutions vendors cater to these owners and guarantee protection against these kind of attacks up to a certain threshold. The cost of hiring these vendors can range from thousands to hundreds of thousand or millions of dollars depending on the severity of the attack.

Our goal was to build a solution with the least amount of funds possible. This solution is scalable and can handle the worst attacks. The client’s dedicated server is not a special server but a simple quad core Xeon managed server running the LAMP stack. The DDoS riposte described in this article can scale to stop a 10Gbps attack or more. The good news is this solution does not require changing anything on the dedicated server itself. The server could be running just about any software stack. This configuration will work just fine with almost all cases effortlessly.

• Distributed Denial of Service – The Social

Before we delve into the glorious technical details, there is an important aspect of DDoS attacks that one should know about; that is the social dynamics that lead to the attack. The more one understands about the the social aspect of a DDoS attack the easier it becomes to prevent or stop it. Because once a DDoS has started, priorities shift quite dramatically and rational for making wise decisions becomes flawed.

DDoS comic

DDoS attacks do not occur randomly. They are targeted and come with a motive. The motive could be revenge but most of the time the motive is financial. The individual or groups that conduct the DDoS attacks are most of the time hired to complete the job. They have the resources and know-how to orchestrate the attack while hoping to avoid getting caught by the authorities. They have no emotional attachment to the DDoS attack itself; they have no hard feelings towards the victim. They just get paid for what they do and nonchalantly, but meticulously, execute.

As explained, DDoS attacks are preceded by an email, post, or phone call, from the individual or group with interest, to the victim. It is always recommended to treat strangers you meet online or offline professionally and politely. The smallest altercation can lead to a negative reaction, which can escalate actions. In the face of anonymous threats against your business or organization, remain calm and composed.

DDoS Offer in Forum

There are public markets online (please don’t ask for links) where wannabe DDoS perpetrators get to hire the attackers. Pricing varies from $5/hr to $10 for a simple non-distributed DoS attack. A DDoS, however, tends to be more expensive depending on the sheer amount of data or packets that needs to be delivered at the target. It can range from $20/hr to $100/hr. The word used to in the circles in lieu of DDoS is to “drop;” meaning to drop a certain Web site or network off the Internet. It really means to either overwhelm the target with enough traffic that the equipment fails or to force upstream providers to “null route” the destination IP at the network level. The end result is that the IP gets dropped from the routing tables and the server to stop responding to all requests.

The fact that DDoS is not cheap has got to be comforting to an extent. It means that it is only a matter of time before the DDoS “client” runs out of cash. This in itself is encouraging. Keep that in mind should you begin to lose patience. Perseverance is omnipotent. Denial of service attacks are considered a crime and are punishable by Federal law in the US and by the police in the UK. As we will explain in the technical part of this article, DDoS attacks are almost impossible to trace to back to the individual or group that are orchestrating the attack. Because of the distributed nature, it requires cooperation from several network engineers that work for upstream providers.

Distributed Denial of Service – The Technicals

First things first, What is a DoS? what is the difference between a DoS and DDoS? A Denial of Service (DoS) is an attack originating from one source or one system that results in the service in question being unavailable to its legitimate users. It denies its very users access either because the service runs out of available resources or has been tricked to deny access to legitimate users. For example, a DoS attack on a Web server can cause it to run out of resources and stop responding to requests. A DDoS, on the other hand, is a more sophisticated attack since the attack originates from hundreds or thousands or nodes.

A DDoS attack is almost impossible to trace back to the source due to its distributed nature. DDoS orchestrators call the nodes and controller system a “bot.” With a few commands, the bot owner can instruct infected nodes from around the world to attack a target. The bot systems are hosted and controlled via the Internet Relay Chat (IRC) system or via a direct connection port connection. The nodes used to attack the target are made of compromised Windows and Linux nodes from around the world.

Before we present our solution, we need to discuss the two types of DDoS attacks that exist. On one hand you have attacks are bandwidth-based and seek to saturate the connectivity link. On the other hand, you have attacks that are packet-based and seek to saturate the processing capability of the equipment. In other words, they seek to overwhelm the processing power of the CPU and memory  or fabric of the routers or switches. All equipment has hard limits when it comes to their ability to handle a certain number of packets per second. Routers and switches are no exception.

Capacity of networking equipment - Mbps vs pps

For example, take the above specification for a Cisco 6500 firewall. Each module is able to handle 5Gbps or 2.8 million pps. This firewall sure looks like it can handle a 5Gbps attack. Great! However, should there be a packet-based DDoS attack, one would only need a 1.5Gbps payload to saturate it. That’s 2.8 million pps * 64 Bytes = 1.5Gbps. So bandwidth capacity means nothing by itself and small packets can cause havoc.

Our client was facing a 2Gbps DDoS attack that is packet based. It sought to force routing equipment along the way to start dropping legitimate packets. This caused the upstream to null route the IP to alleviate the burden on other customers that are behind the link. This is the typical reaction from all upstreams as they seek to protect their many other customers from feeling the pinch of the attack. We were given one last chance to “fix” things before the IP could be routed back in. Here is how we were able to fend off the attack and keep the server running.

We have deployed what we call a “constellation” of reverse proxy VM or VPS nodes running the high performance Web server Nginx. The VM nodes were purchased from several providers given they are located at separate facilities. Essentially, we are off-loading and “splitting” both packet processing and bandwidth consumption across several data center facilities (physical routers & carriers).

Nginx constellation

The configuration of the Nginx nodes is a typical reverse proxy configuration with the usual extra kernel security configuration. So for a 2Gbps attack and with 20 VM nodes, the bandwidth consumption per node is a maximum of 2GBps / 20 = 100Mbps. That’s a 100Mbps load per VM node, which is reasonable enough and is below the threshold for getting one’s IP null routed by the provider. One could add more and more Nginx nodes to the constellation without issues.

So how is 20 VM nodes going to be affordable? VM prices have dropped dramatically over the last year. For the above configuration, a VM can cost between $5/mo and $10/mo. That’s an average of $8*20 = $160/Mo. Knowing that most DDoS attackers have the attention span of a gold fish, the $160 is all you need to send your attacker and his accomplice packing.

Let’s talk more about the Nginx constellation configuration. The Nginx front-end nodes will run in proxy mode caching static files and requests. The more aggressive the DDoS the higher the time-to-live for cache objects should be. This prevents the Nginx nodes from proxy-passing requests to the quad core node. Although, if the main node has idle CPU and plenty of memory it wouldn’t hurt to put it to good use to alleviate the burden on the Nginx front nodes. Your domain’s A records is going to be the IP of the Nginx front nodes configured in round robin fashion. DNS round robin has its shortcomings in terms of not having control over how long (bad) records get cached by resolvers around the world. But in this case, it does not matter much. Just be sure to set high TTL for the records so your DNS server does not collapse under the enormous volume.

Nginx DDoS Constellation

There are tons of online tutorials that go over the installation of Nginx as a reverse proxy so be sure to read up on it. But we will list some of the peculiar settings that are needed to handle a large scale DDoS. Of importance is the number of Nginx worker processes and worker connections. Those values will need to adjusted gradually and higher to handle different kind of attacks depending the VM resource allocation. But you should set them at least as high as the following:

worker_processes 8;
events {
.
.
worker_connections 4096; # Be sure to set ulimit -n 4096 or more
.
.
}

Keep in mind that one still needs to gear up for the event by setting kernel and system variables on the Nginx nodes. Simple things like per-IP rate limiting, flooding rate limits, and syn cookies should be enabled without a question. Here are some measures you can implement:

net.ipv4.tcp_syncookies = 1
# source validation / reversed path
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
kernel.pid_max = 65536
net.ipv4.ip_local_port_range = 9000 65000

Recap

In brief, here are the elements that constitute our solution:

• Nginx reverse proxy constellation
• DNS round robin records
• Security at the Nginx front end level
• Know the social and technical dynamics behind DDoS attacks

That’s all folks. We hope you enjoyed this article. Should you have any question or comment, don’t hesitate to get in touch! No question is minor and we are always looking for feedback.