In this post we (UNIXY) are going to share our experience fending off a large Distributed Denial of Service (DDoS) attack for a client. Generally, Website owners deal with DDoS attacks on their own. There are equipment and solutions vendors cater to these owners and guarantee protection against these kind of attacks up to a certain threshold. The cost of hiring these vendors can range from thousands to hundreds of thousand or millions of dollars depending on the severity of the attack.

Our goal was to build a solution with the least amount of funds possible. This solution is scalable and can handle the worst attacks. The client’s dedicated server is not a special server but a simple quad core Xeon managed server running the LAMP stack. The DDoS riposte described in this article can scale to stop a 10Gbps attack or more. The good news is this solution does not require changing anything on the dedicated server itself. The server could be running just about any software stack. This configuration will work just fine with almost all cases effortlessly.

• Distributed Denial of Service – The Social

Before we delve into the glorious technical details, there is an important aspect of DDoS attacks that one should know about; that is the social dynamics that lead to the attack. The more one understands about the the social aspect of a DDoS attack the easier it becomes to prevent or stop it. Because once a DDoS has started, priorities shift quite dramatically and rational for making wise decisions becomes flawed.

DDoS comic

DDoS attacks do not occur randomly. They are targeted and come with a motive. The motive could be revenge but most of the time the motive is financial. The individual or groups that conduct the DDoS attacks are most of the time hired to complete the job. They have the resources and know-how to orchestrate the attack while hoping to avoid getting caught by the authorities. They have no emotional attachment to the DDoS attack itself; they have no hard feelings towards the victim. They just get paid for what they do and nonchalantly, but meticulously, execute.

As explained, DDoS attacks are preceded by an email, post, or phone call, from the individual or group with interest, to the victim. It is always recommended to treat strangers you meet online or offline professionally and politely. The smallest altercation can lead to a negative reaction, which can escalate actions. In the face of anonymous threats against your business or organization, remain calm and composed.

DDoS Offer in Forum

There are public markets online (please don’t ask for links) where wannabe DDoS perpetrators get to hire the attackers. Pricing varies from $5/hr to $10 for a simple non-distributed DoS attack. A DDoS, however, tends to be more expensive depending on the sheer amount of data or packets that needs to be delivered at the target. It can range from $20/hr to $100/hr. The word used to in the circles in lieu of DDoS is to “drop;” meaning to drop a certain Web site or network off the Internet. It really means to either overwhelm the target with enough traffic that the equipment fails or to force upstream providers to “null route” the destination IP at the network level. The end result is that the IP gets dropped from the routing tables and the server to stop responding to all requests.

The fact that DDoS is not cheap has got to be comforting to an extent. It means that it is only a matter of time before the DDoS “client” runs out of cash. This in itself is encouraging. Keep that in mind should you begin to lose patience. Perseverance is omnipotent. Denial of service attacks are considered a crime and are punishable by Federal law in the US and by the police in the UK. As we will explain in the technical part of this article, DDoS attacks are almost impossible to trace to back to the individual or group that are orchestrating the attack. Because of the distributed nature, it requires cooperation from several network engineers that work for upstream providers.

Distributed Denial of Service – The Technicals

First things first, What is a DoS? what is the difference between a DoS and DDoS? A Denial of Service (DoS) is an attack originating from one source or one system that results in the service in question being unavailable to its legitimate users. It denies its very users access either because the service runs out of available resources or has been tricked to deny access to legitimate users. For example, a DoS attack on a Web server can cause it to run out of resources and stop responding to requests. A DDoS, on the other hand, is a more sophisticated attack since the attack originates from hundreds or thousands or nodes.

A DDoS attack is almost impossible to trace back to the source due to its distributed nature. DDoS orchestrators call the nodes and controller system a “bot.” With a few commands, the bot owner can instruct infected nodes from around the world to attack a target. The bot systems are hosted and controlled via the Internet Relay Chat (IRC) system or via a direct connection port connection. The nodes used to attack the target are made of compromised Windows and Linux nodes from around the world.

Before we present our solution, we need to discuss the two types of DDoS attacks that exist. On one hand you have attacks are bandwidth-based and seek to saturate the connectivity link. On the other hand, you have attacks that are packet-based and seek to saturate the processing capability of the equipment. In other words, they seek to overwhelm the processing power of the CPU and memory  or fabric of the routers or switches. All equipment has hard limits when it comes to their ability to handle a certain number of packets per second. Routers and switches are no exception.

Capacity of networking equipment - Mbps vs pps

For example, take the above specification for a Cisco 6500 firewall. Each module is able to handle 5Gbps or 2.8 million pps. This firewall sure looks like it can handle a 5Gbps attack. Great! However, should there be a packet-based DDoS attack, one would only need a 1.5Gbps payload to saturate it. That’s 2.8 million pps * 64 Bytes = 1.5Gbps. So bandwidth capacity means nothing by itself and small packets can cause havoc.

Our client was facing a 2Gbps DDoS attack that is packet based. It sought to force routing equipment along the way to start dropping legitimate packets. This caused the upstream to null route the IP to alleviate the burden on other customers that are behind the link. This is the typical reaction from all upstreams as they seek to protect their many other customers from feeling the pinch of the attack. We were given one last chance to “fix” things before the IP could be routed back in. Here is how we were able to fend off the attack and keep the server running.

We have deployed what we call a “constellation” of reverse proxy VM or VPS nodes running the high performance Web server Nginx. The VM nodes were purchased from several providers given they are located at separate facilities. Essentially, we are off-loading and “splitting” both packet processing and bandwidth consumption across several data center facilities (physical routers & carriers).

Nginx constellation

The configuration of the Nginx nodes is a typical reverse proxy configuration with the usual extra kernel security configuration. So for a 2Gbps attack and with 20 VM nodes, the bandwidth consumption per node is a maximum of 2GBps / 20 = 100Mbps. That’s a 100Mbps load per VM node, which is reasonable enough and is below the threshold for getting one’s IP null routed by the provider. One could add more and more Nginx nodes to the constellation without issues.

So how is 20 VM nodes going to be affordable? VM prices have dropped dramatically over the last year. For the above configuration, a VM can cost between $5/mo and $10/mo. That’s an average of $8*20 = $160/Mo. Knowing that most DDoS attackers have the attention span of a gold fish, the $160 is all you need to send your attacker and his accomplice packing.

Let’s talk more about the Nginx constellation configuration. The Nginx front-end nodes will run in proxy mode caching static files and requests. The more aggressive the DDoS the higher the time-to-live for cache objects should be. This prevents the Nginx nodes from proxy-passing requests to the quad core node. Although, if the main node has idle CPU and plenty of memory it wouldn’t hurt to put it to good use to alleviate the burden on the Nginx front nodes. Your domain’s A records is going to be the IP of the Nginx front nodes configured in round robin fashion. DNS round robin has its shortcomings in terms of not having control over how long (bad) records get cached by resolvers around the world. But in this case, it does not matter much. Just be sure to set high TTL for the records so your DNS server does not collapse under the enormous volume.

Nginx DDoS Constellation

There are tons of online tutorials that go over the installation of Nginx as a reverse proxy so be sure to read up on it. But we will list some of the peculiar settings that are needed to handle a large scale DDoS. Of importance is the number of Nginx worker processes and worker connections. Those values will need to adjusted gradually and higher to handle different kind of attacks depending the VM resource allocation. But you should set them at least as high as the following:

worker_processes 8;
events {
.
.
worker_connections 4096; # Be sure to set ulimit -n 4096 or more
.
.
}

Keep in mind that one still needs to gear up for the event by setting kernel and system variables on the Nginx nodes. Simple things like per-IP rate limiting, flooding rate limits, and syn cookies should be enabled without a question. Here are some measures you can implement:

net.ipv4.tcp_syncookies = 1
# source validation / reversed path
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
kernel.pid_max = 65536
net.ipv4.ip_local_port_range = 9000 65000

Recap

In brief, here are the elements that constitute our solution:

• Nginx reverse proxy constellation
• DNS round robin records
• Security at the Nginx front end level
• Know the social and technical dynamics behind DDoS attacks

That’s all folks. We hope you enjoyed this article. Should you have any question or comment, don’t hesitate to get in touch! No question is minor and we are always looking for feedback.

The benchmarks are in! Find out how Apache smoked Litespeed with the help of Varnish: http://www.unixy.net/apache-vs-litespeed/

Cheers!

Varnish is an excellent Web accelerator that can be made to proxy requests in and out of a cluster of somewhat more fully fledged Web servers like Apache or Litespeed. It has some great features like its compiled language, called VCL, and C-like programming API.

Large vBulletin deployments tend to be heavy on CPU and memory due to PHP script processing. For a large vBulletin forum, we recommend a cluster of 5 physical servers with three of those running Xen virtualization. One of those servers will be dedicated to the MySQL master database. Three to be setup as “headless” PHP nodes and Varnish load balancing and failover. And finally one as the NFS file store. The three headless servers need to run Varnish in their own VM and Litespeed or Apache in their own VM similarly.

The varnish backend director functionality makes it ideal to balance incoming traffic across all PHP headless nodes. It makes the configuration scalable and plug and play especially when needing to scale out within hours. The challenge in this setup is in making Varnish work correctly with vBulletin. Otherwise, session problems will occur.

We have a lot to share on this implementation so keep checking this blog as we will post it all. In the next installment, we’ll go through our deployment of a large vBulletin forum for a customer. In the mean time, feel free to get in touch should you have a question or comment. If you are interested in us helping you accelerate your server, we have a page explaining the different technologies we deploy on our clients’ dedicated servers. Read up here: http://www.unixy.net/accelerate-your-server

Update: we are offering a Varnish configuration for vBulletin (3 & 4) for a one-time fee. We can also configure it free of charge should you decide to rent your fully managed dedicated server from UNIXY (http://www.unixy.net). Please contact us today to get your forum running with a blazing fast speed!

That’s all folks!

If you receive one of the following errors, it’s very possible the Apache process is listening on port 443 but is not responding using the SSL protocol routines. In brief it is a protocol mismatch. Here’s a list of possible errors:

Firefox error: SSL received a record that exceeded the maximum permissible length

Apache error log: \x80d\x01\x03\x01

Curl CLI error: curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

The fastest way to recover from such error is to restore the most recent Apache configuration file from backup. If you do not have a backup copy, you will need to look at your virtualhost section associated with the domain and SSL port combination. Chances are it does not exist. In cPanel, this issue can be corrected with a simple run of the script rebuildhttpdconf as such:

/scripts/rebuildhttpdconf

After making the necessary changes, and whether you are running cPanel or not, you will need to stop Apache then restart it.

That’s all folks!

suPHP is a module that enhances overall server security. It forces the system to execute PHP scripts with user privileges as opposed to Web server privileges. suPHP is a good security measure especially when you expect to host multiple unrelated websites or hosting accounts. It keeps rotten apples away from the good ones! Keep in mind that UNIXY is a truly fully managed dedicated server company. We are always happy to assist our customers in accomplishing tasks such as installing suPHP on their dedicated server or cluster.

This small guide will go over the installation of suPHP on a cPanel server. cPanel has a peculiar way of setting up suPHP. The good news is that cPanel simplifies the installation! Here’s an overview of the required steps:

1. Install suPHP Using EasyApache
2. Configure suPHP
3. Verify The Configuration

It’s important to pick a maintenance window that is less intrusive on your users or customers. The rebuild of Apache to support suPHP can cause down time. So be sure to send out a friendly email to your customers.

1. Install suPHP Using EasyApache

There are two ways to launch the EasyApache program. The first one is through SSH and the other via WHM. I prefer SSH so the rest of this guide will be based on that method. So go ahead and remote into the server as user root. Once logged in, run the easyapache script as such

/scripts/easyapache

The first textual screen that pops up will say the following: “Please choose a profile to load.” Simply hit the Tab key twice. The textual box “Start customizing based on profile” will become highlighted. At which point will hit the Enter key on your keyboard. Once you hit Enter, you will be presented with another screen. This time the screen says: “Please choose which apache to build.” Simply hit the Tab key once and then hit Enter again.

There are two more screen to go and we’re done. After hitting the Enter key above, another screen will pop up. This time it says: Please choose which main PHP versions (if any) to build.” Hit the Tab key once and then the Enter key. Follow the same instructions for the screen with title “Please choose which specific PHP version(s) to build.” The next screen, however, is very important. This is where we get to pick the suPHP module that we are building. Once you hit Enter in the previous step, the screen “Short Options List” comes up. Go ahead and hit the Tab key a few times until “Exhaustive Options List” is highlighted then press Enter. Scroll down the list of options until you cursor is on the entry “Mod SuPHP…”.  Hit the Space key once to check the box. Hit the Tab key once then Enter, then select “Save and Build”. Answer Yes to all questions

At this point the build has started. Wait for the build to complete. It’s important that no one uses WHM while the build is in progress. The build takes about 20 minutes to complete depending on server resources. Once the build is finished, proceed to step 2 below.

2. Configure suPHP

This is an important step because it formally enables suPHP. Simply run the following command

/usr/local/cpanel/bin/rebuild_phpconf 5 none suphp 1

We’re essentially telling it that we want PHP version 5 running on suPHP and SUEXEC. To verify that command has taken effect run the following command. The output should be indentical:

# /usr/local/cpanel/bin/rebuild_phpconf –current
Available handlers: suphp dso cgi none
DEFAULT PHP: 5
PHP4 SAPI: none
PHP5 SAPI: suphp
SUEXEC: enabled

So far so good! Now go ahead and restart Apache by running the following command:

/scripts/restartsrv_httpd

3. Verify The Configuration

At this point all should be working fine. But it’s prudent to keep an eye on the suPHP log file as that’s where errors and warnings show up. The suPHP log file is located here:

/usr/local/apache/logs/suphp_log

Most errors are related to permissions on PHP files. suPHP is very picky about permission and / or file ownership so be sure your PHP files have permission 755 and are owned by the same user account. You can change permission on any file using the following command:

chmod 755 <filename>

You can also change file ownership using the chown command:

chown user.user <filename>

I hope this is useful to anyone setting to install suPHP on a cPanel server.