My interest in NUMA (Non-Uniform Memory Access) has picked up quite a bit as of late. This technology has a huge potential for application in large scale Web hosting and Internet solutions. In brief, NUMA is a technology that was conceived to get around bus contention in general. What this means is that, with NUMA, programs are no longer limited by memory size or disk IOPS on the same host node and can, therefore, harness the power of other nearby servers as if they were part of the same chassis. In fact, IO scales up as a result!

The problem with uniform memory access architectures is that in a single chassis multi-processor system with a large amount of memory the bus tends to saturate very quickly. Processor cores addressing the memory bus are thus contending for the same resource, which becomes a hot spot. Take a 32-core server with 128GB of DDR3 memory. This server will under perform should it run a parallel memory hungry application. The issue being that each task on a core will at some point need to address memory via the bus (DDR3 is a three-frequency or three-channel bus). So out of 32 tasks that are all attempting to access the bus, there will be (32 -3 = 29/3 ~ 9) 9 tasks that contend for the same channel. The end result is a saturated bus and poor performance.

NUMA solves this bus contention problem by creating a virtual network of processor nodes and giving each node a certain weight based on its locality. The most interesting aspect is its true scalability potential. Most importantly, NUMA is the only technology that can scale up. Amazon’s EC2 and most other technologies in the market nowadays can only scale out. In other words, one has to implement load balancing, replication, and data synchronization.

Here’s where it gets interesting. IBM has already comoditised NUMA with its x86 offering: the IBM x440. Although one can only chain two x440 chassis together to form one NUMA network, what was once a concept is now practical! The x440′s can be chained together via their SMP expansion module using remote IO cables. Now imagine having the ability to chain 5, 10, 100 16GB dual quad core boxes! Not only does one take advantage of of a few terrabytes of memory but also scale core compute and IO.

A challenge remains. The remote IO cables that allow formation of a NUMA node cluster is based on proprietary technology. The research and development effort required to deliver such technology is colossal for a smaller firm. Or maybe not!

We were able to install DirectAdmin (a Web hosting control panel) on Amazon’s Elastic Compute Cloud, branded as EC2. We would like to share the steps required to build a working instance of DirectAdmin on an Amazon server. First, I would like to personally thank Mark from DirectAdmin for being so accommodating and granting us a trial license. His help is very much appreciated! Keep in mind that UNIXY‘s truly fully managed service goes above and beyond. We take business dreams and make them a reality.

Before we list the requirements, it is important to note that EC2 instances are Xen Virtual Machines. We encountered some issues running DA on the “Small” Amazon instances. The issues are related to 64-bit mode library mismatches. The only instances to be able to run DirectAdmin out of the box are of type “Large” and better. They are more expensive but ideal for this configuration.

Here are the requirements:

• Instance of type Large
• Linux CentOS 5 image
• One elastic (which is another word for static) IP address

Go ahead and bring up the node as you would normally do. Be sure to remove all extra software that comes pre-installed with the CentOS 5 image. Otherwise, it will break DirectAdmin. If in doubt, follow the instructions on the DirectAdmin Install page:

http://www.directadmin.com/install.html

Once logged in as root, go ahead and download the DirectAdmin install tarball:

# wget http://www.directadmin.com/setup.sh

By default, the licensing scheme of DirectAdmin makes it that setup.sh binds to the licensed IP address. Here’s how the license verification command looks like:

$BIN_DIR/wget $WGET_OPTION -O $DA_PATH/update.tar.gz –bind-address=$IP https://www.directadmin.com/cgi-bin/daupdate?uid=$CID\&lid=$LID

But as you know, EC2 instances are 1:1 private:public NAT. Which means that the license check step will fail and the DA install won’t start. To avoid this chicken-and-egg problem, we are going to tell setup.sh to not bind to any IP address but to simply connect to the DA licensing server using the elastic external IP address, which it will do by default. Go ahead and run this command against the setup.sh file:

# sed -i 's/--bind-address=\$IP//;' setup.sh

Before we run setup.sh, we need to trick the setup script to think that the external IP address is “attached” to the server. Here’s how:

# ifconfig eth0:0 inet 1.1.1.1 netmask 255.255.255.255 up

Be sure to replace the IP address 1.1.1.1 with your elastic IP. Then simply run setup.sh:

# cd /usr/local/directadmin/scripts; while [ true ]; do sed -i 's/--bind-address=\$IP//; s/--bind-address=\${3}//;' *.sh > /dev/null > 2&>1; sleep 10; done & sh setup.sh 11111 22222 vpslux.com eth0:0 1.1.1.1

Woaah! Wait a minute! What is that!? OK, let’s break it down. The DA scripts directory gets unpacked into the server after you run setup.sh. It dumps a list of scripts that are used by DA to setup accounts, domains, etc. Some scripts use the same –bind-address flag. Having that flag in those scripts breaks DA. the one-line script “while [ true ]; do sed -i ‘s/–bind-address=\$IP//; s/–bind-address=\${3}//;’ *.sh; sleep 10; done” simply goes to that directory and removes the flag.

The trick being DA attempts to run some of those scripts while setup.sh is running. So it’s important to have the sed script run in parallel during the installation. Once the setup.sh is done running, we’ll kill the job (or reboot per the final step). The arguments supplied to setup.sh are in this format

# setup.sh <ClientID> <LicenseID> <Hostname> <Interface> <ExternalIP>

Be sure to replace those values accordingly. Once the install finishes, simply reboot the VM:

# reboot

That’s all folks. Enjoy!