In this post we (UNIXY) are going to share our experience fending off a large Distributed Denial of Service (DDoS) attack for a client. Generally, Website owners deal with DDoS attacks on their own. There are equipment and solutions vendors cater to these owners and guarantee protection against these kind of attacks up to a certain threshold. The cost of hiring these vendors can range from thousands to hundreds of thousand or millions of dollars depending on the severity of the attack.
Our goal was to build a solution with the least amount of funds possible. This solution is scalable and can handle the worst attacks. The client’s dedicated server is not a special server but a simple quad core Xeon managed server running the LAMP stack. The DDoS riposte described in this article can scale to stop a 10Gbps attack or more. The good news is this solution does not require changing anything on the dedicated server itself. The server could be running just about any software stack. This configuration will work just fine with almost all cases effortlessly.
Before we delve into the glorious technical details, there is an important aspect of DDoS attacks that one should know about; that is the social dynamics that lead to the attack. The more one understands about the the social aspect of a DDoS attack the easier it becomes to prevent or stop it. Because once a DDoS has started, priorities shift quite dramatically and rational for making wise decisions becomes flawed.
DDoS comic
DDoS attacks do not occur randomly. They are targeted and come with a motive. The motive could be revenge but most of the time the motive is financial. The individual or groups that conduct the DDoS attacks are most of the time hired to complete the job. They have the resources and know-how to orchestrate the attack while hoping to avoid getting caught by the authorities. They have no emotional attachment to the DDoS attack itself; they have no hard feelings towards the victim. They just get paid for what they do and nonchalantly, but meticulously, execute.
As explained, DDoS attacks are preceded by an email, post, or phone call, from the individual or group with interest, to the victim. It is always recommended to treat strangers you meet online or offline professionally and politely. The smallest altercation can lead to a negative reaction, which can escalate actions. In the face of anonymous threats against your business or organization, remain calm and composed.
DDoS Offer in Forum
There are public markets online (please don’t ask for links) where wannabe DDoS perpetrators get to hire the attackers. Pricing varies from $5/hr to $10 for a simple non-distributed DoS attack. A DDoS, however, tends to be more expensive depending on the sheer amount of data or packets that needs to be delivered at the target. It can range from $20/hr to $100/hr. The word used to in the circles in lieu of DDoS is to “drop;” meaning to drop a certain Web site or network off the Internet. It really means to either overwhelm the target with enough traffic that the equipment fails or to force upstream providers to “null route” the destination IP at the network level. The end result is that the IP gets dropped from the routing tables and the server to stop responding to all requests.
The fact that DDoS is not cheap has got to be comforting to an extent. It means that it is only a matter of time before the DDoS “client” runs out of cash. This in itself is encouraging. Keep that in mind should you begin to lose patience. Perseverance is omnipotent. Denial of service attacks are considered a crime and are punishable by Federal law in the US and by the police in the UK. As we will explain in the technical part of this article, DDoS attacks are almost impossible to trace to back to the individual or group that are orchestrating the attack. Because of the distributed nature, it requires cooperation from several network engineers that work for upstream providers.
Distributed Denial of Service – The Technicals
First things first, What is a DoS? what is the difference between a DoS and DDoS? A Denial of Service (DoS) is an attack originating from one source or one system that results in the service in question being unavailable to its legitimate users. It denies its very users access either because the service runs out of available resources or has been tricked to deny access to legitimate users. For example, a DoS attack on a Web server can cause it to run out of resources and stop responding to requests. A DDoS, on the other hand, is a more sophisticated attack since the attack originates from hundreds or thousands or nodes.
A DDoS attack is almost impossible to trace back to the source due to its distributed nature. DDoS orchestrators call the nodes and controller system a “bot.” With a few commands, the bot owner can instruct infected nodes from around the world to attack a target. The bot systems are hosted and controlled via the Internet Relay Chat (IRC) system or via a direct connection port connection. The nodes used to attack the target are made of compromised Windows and Linux nodes from around the world.
Before we present our solution, we need to discuss the two types of DDoS attacks that exist. On one hand you have attacks are bandwidth-based and seek to saturate the connectivity link. On the other hand, you have attacks that are packet-based and seek to saturate the processing capability of the equipment. In other words, they seek to overwhelm the processing power of the CPU and memory or fabric of the routers or switches. All equipment has hard limits when it comes to their ability to handle a certain number of packets per second. Routers and switches are no exception.
Capacity of networking equipment - Mbps vs pps
For example, take the above specification for a Cisco 6500 firewall. Each module is able to handle 5Gbps or 2.8 million pps. This firewall sure looks like it can handle a 5Gbps attack. Great! However, should there be a packet-based DDoS attack, one would only need a 1.5Gbps payload to saturate it. That’s 2.8 million pps * 64 Bytes = 1.5Gbps. So bandwidth capacity means nothing by itself and small packets can cause havoc.
Our client was facing a 2Gbps DDoS attack that is packet based. It sought to force routing equipment along the way to start dropping legitimate packets. This caused the upstream to null route the IP to alleviate the burden on other customers that are behind the link. This is the typical reaction from all upstreams as they seek to protect their many other customers from feeling the pinch of the attack. We were given one last chance to “fix” things before the IP could be routed back in. Here is how we were able to fend off the attack and keep the server running.
We have deployed what we call a “constellation” of reverse proxy VM or VPS nodes running the high performance Web server Nginx. The VM nodes were purchased from several providers given they are located at separate facilities. Essentially, we are off-loading and “splitting” both packet processing and bandwidth consumption across several data center facilities (physical routers & carriers).
Nginx constellation
The configuration of the Nginx nodes is a typical reverse proxy configuration with the usual extra kernel security configuration. So for a 2Gbps attack and with 20 VM nodes, the bandwidth consumption per node is a maximum of 2GBps / 20 = 100Mbps. That’s a 100Mbps load per VM node, which is reasonable enough and is below the threshold for getting one’s IP null routed by the provider. One could add more and more Nginx nodes to the constellation without issues.
So how is 20 VM nodes going to be affordable? VM prices have dropped dramatically over the last year. For the above configuration, a VM can cost between $5/mo and $10/mo. That’s an average of $8*20 = $160/Mo. Knowing that most DDoS attackers have the attention span of a gold fish, the $160 is all you need to send your attacker and his accomplice packing.
Let’s talk more about the Nginx constellation configuration. The Nginx front-end nodes will run in proxy mode caching static files and requests. The more aggressive the DDoS the higher the time-to-live for cache objects should be. This prevents the Nginx nodes from proxy-passing requests to the quad core node. Although, if the main node has idle CPU and plenty of memory it wouldn’t hurt to put it to good use to alleviate the burden on the Nginx front nodes. Your domain’s A records is going to be the IP of the Nginx front nodes configured in round robin fashion. DNS round robin has its shortcomings in terms of not having control over how long (bad) records get cached by resolvers around the world. But in this case, it does not matter much. Just be sure to set high TTL for the records so your DNS server does not collapse under the enormous volume.
Nginx DDoS Constellation
There are tons of online tutorials that go over the installation of Nginx as a reverse proxy so be sure to read up on it. But we will list some of the peculiar settings that are needed to handle a large scale DDoS. Of importance is the number of Nginx worker processes and worker connections. Those values will need to adjusted gradually and higher to handle different kind of attacks depending the VM resource allocation. But you should set them at least as high as the following:
worker_processes 8;
events {
.
.
worker_connections 4096; # Be sure to set ulimit -n 4096 or more
.
.
}
Keep in mind that one still needs to gear up for the event by setting kernel and system variables on the Nginx nodes. Simple things like per-IP rate limiting, flooding rate limits, and syn cookies should be enabled without a question. Here are some measures you can implement:
net.ipv4.tcp_syncookies = 1
# source validation / reversed path
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
kernel.pid_max = 65536
net.ipv4.ip_local_port_range = 9000 65000
Recap
In brief, here are the elements that constitute our solution:
That’s all folks. We hope you enjoyed this article. Should you have any question or comment, don’t hesitate to get in touch! No question is minor and we are always looking for feedback.
In this article, we are going to cover one trick that is used in Web social circles to weather short-lived traffic storms. In most cases, servers collapse due to the server-side processing that needs to occur to produce the HTML that sent to the browser. Whether the server-side script is written in PHP, Ruby, Python, or Perl with a database back-end, this trick is effective and does the job. It gives you much needed breathing room to plan ahead should the spike continue for weeks.
In the case of a WordPress blog, say for example http://blog.unixy.net/2010/03/bargain-bin-fully-managed-dedicated-servers/, upon visiting the Bargain Bin blog link, the server where blog.unixy.net is hosted will need to perform several tasks to render the page and send it the HTML results back. The tasks consist of running PHP scripts, querying the database, and retrieving files from disk amongst other tasks. Under extended peak traffic periods, these tasks when repeated continuously overwhelm the server and cause it to slow down and sometimes to crash.
So, if all the server does is produce the HTML code and send it back to the browser, we might as well just generate the HTML ourselves once and instruct the server to send it instead, bypassing all of that heavy processing. So without further delay, here is how one can generate the HTML:
GET http://blog.unixy.net/2010/03/bargain-bin-fully-managed-dedicated-servers/ &> /tmp/bargain-bin.html
The GET command comes as part of the libwww-perl package, which is available for both Linux and Windows. Now that we obtained the HTML, we are going to add a .htaccess rule to temporarily redirect all requests to the article /2010/03/bargain-bin-fully-managed-dedicated-servers to bargain-bin.html. Be sure to move the file my-page-dugg.html to the HTML root folder. Here is the .htaccess rule:
RewriteEngine on
Redirect 302 /2010/03/bargain-bin-fully-managed-dedicated-servers /bargain-bin.html
Be sure to put the above rules before the WordPress rules. The 302 rule means that it is a temporary redirect. This is so it is SEO friendly and does not break your links as search engines see them.
I hope you enjoyed this short article. The above instructions can work for about any CMS or portal. Keep in mind that should you be one of our customers, we will be very happy to complete this workaround on your server for no extra charge.
The benchmarks are in! Find out how Apache smoked Litespeed with the help of Varnish: http://www.unixy.net/apache-vs-litespeed/
Cheers!
In this post, we will explain what a dedicated server is and the differences between a managed dedicated server and a fully managed dedicated server. At the expense of confusing you, a managed server being offered at Company A might not feature the same service level as a managed server offered at Company B. Why so? Read on to find out.
Before we delve into the differences, know that UNIXY (http://www.unixy.net) is a fully managed dedicated server hosting provider. We clearly state that our Linux server management service covers everything one would ever need. Our management philosophy is unambiguous: if you cannot complete a task with a few mouse clicks, UNIXY will gladly do. We secure, optimize, and monitor your server at all times throughout the year. We are always happy to help our clients and give them the support they need to thrive and stay worry-free.
First things first, a server is a specialized computer that is installed at a private facility called a data center. The lingo used in the hosting business for “installed” is racked up. A data center is a private building built to withstand harsh environmental conditions, secured to fend off any kind of physical intrusion, configured with redundancy at the network and power grid level, and staffed with technicians, engineers, and security officers. Data centers are generally boring from the outside but very interesting in the inside. They sport state-of-the-art equipment and technology. The picture below is of a Google data center.
Outside View of a Data Center
A dedicated server is a computer server that is racked up and configured to handle the requirements of an individual or an entity. It is said to be dedicated because its resources are not shared with any other entity. They are dedicated. At a bare minimum, the provider is required to install an operating system like Linux and replace any hardware component that fails while you are renting the dedicated server. This means that if your hard drive has a defect, you are entitled to a speedy replacement at no charge to you. The same applies to the network. Should you notice network connectivity issues, your provider is required to investigate and identify the root cause, again, at no charge to you.
Server Cabinet Inside a Data Center
It is common to install a Web control panel, which is a software application that one can access via the browser to manage and control the server, add websites and email accounts, and just about anything one can imagine. There are several control panels in the market each with its advantages and learning curve. Some are free and others cost money. The cost of such control panel can range from $15 to $40 per month. cPanel (http://www.cpanel.net) is a one such control panel for Linux. DirectAdmin (http://www.directadmin.com) is another.
Keep in mind that while a control panel is convenient and practical, one still needs trained engineers and technicians to maintain the back-end of the system. This is where server management comes in. At a bare minimum, server management should include system patching (software fixes and upgrades), system security enforcement to prevent compromises, and trouble shooting. So a control panel is a step forward in the right direction but is not all encompassing.
A managed dedicated server is a dedicated server that comes with a certain level of support from the provider. Software upgrades and patching is provided for free. Software crash root cause remediation is also covered. The security of the server is assumed by the provider. Should you need help with installing common software on the server, that will also get taken care of. That includes a Web server software anddatabase software. The provider expects you to have a certain technical knowledge and understanding so you can navigate around the system. For example, they expect you to know how to add email accounts, add Web sites, transfer your content via FTP, and generally have a basic understanding of Web hosting.
This is the creme of the crop of all management plans. If you rent a fully managed dedicated server, you, as a customer, are not to be concerned with anything beyond updating your Web content and running your business. Full management means that regardless of the request, the provider will complete it for you. Keep in mind that software development or software bug fix requests are not covered under this level. Those requests are meant to be handled by a software development firm.
Server management tasks are not industry standard so there is ambiguity as to what plan covers what exactly. Some managed providers refuse to complete a certain request because the software is not covered. Others charge for the labor required to complete the task. As a consumer, it is always recommended to approach the provider, before signing up, and ask for what is or what is not covered by the managed service. Keeping a record of such conversation can protect you should you be in a such situation.
That’s all folks! This concludes our article on server management. I hope you enjoyed it.
There are rumors going around that ThePlanet is in acquisition talks with Softlayer. It will be very interesting to see what comes out of this.
Stay tuned!
Update: Yes, it’s official! Softlayer has entered an acquisition deal. It’s still unclear as to who holds the majority. Read it all here: http://softlayer.com/press/release/281/gi-partners-and-management-acquire-softlayer
In this article, we shall outline the steps required to build a private Content Delivery or Distribution Network (CDN) using Varnish Cache and Nginx. The goal is to build a CDN using free, readily available software but most importantly spend the least amount of funds possible. To this end, all nodes participating in this network are going to be virtual machines (Xen, Virtuozzo, OpenVZ, etc). Should you have any questions or comments on the configuration of this CDN, please post them in this forum: http://www.varnish-cache.info/
Our global CDN can not only keep the latest copy of static files closer to our global visitors but can also cache the most used pages (dynamic or not) in memory on the edge nodes! This means less trips to the geographically distant and slower Dynamic Node (see below). This is similar to what Akamai and other well known firms do, only at a fraction of the cost. However, in this article, and to keep things simple, we will only be caching static files.
Some of you might be surprised to learn that we built this global CDN free of charge for one of our beloved customers. UNIXY offers truly fully managed dedicated servers and clusters with managed servers starting as low as $45. Our motto is simple: what you cannot do with a few mouse clicks, we will gladly do it for you! Please visit us online when you have a chance: http://www.unixy.net. Please do ask if you have any question or comment. No question is minor!
The illustration below presents a logical layout of our CDN. Edge nodes can be located just about anywhere in the world. One could also add more nodes at any location should there be a capacity need. The Dynamic Content Node will typically run a mixture of MySQL, Apache, and server-side software built using PHP, Ruby, Python, .Net, or any language for that matter.
Global CDN
Nginx is a lightweight high-performance Web server that is able to handle large traffic consistently. We are leveraging its proxy and caching capabilities. We shall compile Nginx and leverage the proxy module. This module allows us to cache data on the local disks of the remote edge locations.
As its name implies, Varnish Cache is a high-performance caching engine used to keep recently accessed content in memory for fastest access. Varnish is not a Web server. Hence our need to bundle it with Nginx, which is acting as a Web server at the edge nodes. We will cover Varnish in detail in our next installment.
And finally the glue that holds all of these components together: BIND. BIND is the DNS software used to map Internet host names to IP addresses. We shall patch Bind to add geographical filters support. In other words, BIND will serve each client the IP of closest edge node in the CDN. For example, an vistor from Africa will receive the edge node IP of South Africa or Morocco depending on the filters. We will touch on this later.
At a minimum, we will need two nodes to demo and build our private CDN. That’s one Dynamic Content Node and one Edge Location node. The Dynamic Content Node will run the full LAMP stack along with BIND and the geographical filters patch. The Edge Location node will run Nginx and Varnish. One could always run BIND+GeoIP on a separate node as it is good practice. We will assign the hostname dynamic_content to the Dynamic Content Node and edge_node to the Edge Location.
Download BIND from ISC: http://www.bind9.net/download
Download MaxMind’s C API: http://geolite.maxmind.com/download/geoip/api/c/
[root@dynamic_node /]# cd /usr/src/[root@dynamic_node src]# wget http://mirrors.24-7-solutions.net/pub/isc/bind9/9.2.4/bind-9.2.4.tar.gz[root@dynamic_node src]# wget http://geolite.maxmind.com/download/geoip/api/c/GeoIP-1.4.6.tar.gz[root@dynamic_node src]# tar -xzvf bind-9.2.4.tar.gz[root@dynamic_node src]# tar -xzvf GeoIP-1.4.6.tar.gz[root@dynamic_node src]# cd GeoIP-1.4.6[root@dynamic_node GeoIP-1.4.6]# ./configure –prefix=/usr/local/geoip[root@dynamic_node GeoIP-1.4.6]# make[root@dynamic_node GeoIP-1.4.6]# make install[root@dynamic_node GeoIP-1.4.6]# cd ..[root@dynamic_node src]# patch -p0 < bind-9.2.4-geodns-patch/patch.diff[root@dynamic_node src]# cd bind-9.2.4[root@dynamic_node bind-9.2.4]# CFLAGS=”-I/usr/local/geoip/include” LDFLAGS=”-L/usr/local/geoip/lib -lGeoIP” ./configure –prefix=/usr/local/bind[root@dynamic_node bind-9.2.4]# make[root@dynamic_node bind-9.2.4]# make install
view “us” {// Match clients from US & Canadamatch-clients { country_US; country_CA; };// Provide recursive service to internal clients only.recursion no;zone “cdn.unixy.net” {type master;file “pri/unixy-us.db”;};zone “.” IN {type hint;file “named.ca”;};};view “latin” {// Match from Argentina, Chile and Brazilmatch-clients { country_AR; country_CL; country_BR; };// Provide recursive service to internal clients only.recursion no;zone “cdn.unixy.net” {type master;file “pri/unixy-latin.db”;};zone “.” IN {type hint;file “named.ca”;};};
[root@edge_node src]# wget http://nginx.org/download/nginx-0.8.45.tar.gz[root@edge_node src]# tar -xzvf nginx-0.8.45.tar.gz[root@edge_node src]# cd nginx-0.8.45[root@edge_node nginx-0.8.45]# ./configure –prefix=/usr/local/nginx –with-http_realip_module[root@edge_node nginx-0.8.45]# make[root@edge_node nginx-0.8.45]# make install
http {include mime.types;default_type application/octet-stream;sendfile on;keepalive_timeout 65;upstream dynamic_node {server 1.1.1.1:80; # 1.1.1.1 is the IP of the Dynamic Node}server {listen 81;server_name cdn.unixy.net;location ~* \.(gif|jpg|jpeg|png|wmv|avi|mpg|mpeg|mp4|htm|html|js|css|mp3|swf|ico|flv)$ {proxy_set_header X-Real-IP $remote_addr;proxy_pass http://dynamic_node;proxy_store /var/www/cache$uri;proxy_store_access user:rw group:rw all:r;}
A few seconds of browsing and the disk cache is already populating:
[root@edge_node /]# ls -al /var/www/cache
contact-unixy css images index.html javascript js
[root@edge_node /]#
[root@edge_node src]# wget http://downloads.sourceforge.net/project/varnish/varnish/2.1.2/varnish-2.1.2.tar.gz?use_mirror=cdnetworks-us-1&ts=1279434397[root@edge_node src]# tar -xzvf varnish-2.1.2.tar.gz[root@edge_node src]# tar -xzvf varnish-2.1.2.tar.gz[root@edge_node varnish-2.1.2]# ./configure –prefix=/usr/local/varnish[root@edge_node varnish-2.1.2]# make[root@edge_node varnish-2.1.2]# make install
backend default {.host = “127.0.0.1″;.port = “81″;}
sub vcl_recv {.
.
.
if (req.url ~ “\.(js|css|jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf)$”) {return (lookup);}
.
.
.}
sub vcl_fetch {.
.
.
if (req.url ~ “\.(js|css|jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf)$”) {unset obj.http.set-cookie;}
.
.
.}
The instructions above can replicated across however many additional Edge Nodes you want to add. One could also add redundancy to the BIND+GeoIP setup by configuring secondary nodes. The illustration below shows the flow of a request from top to bottom.
CDN Request and Response Flow
That’s all folks! I hope you enjoyed this article.
