UNIXy

We were able to install DirectAdmin (a Web hosting control panel) on Amazon’s Elastic Compute Cloud, branded as EC2. We would like to share the steps required to build a working instance of DirectAdmin on an Amazon server. First, I would like to personally thank Mark from DirectAdmin for being so accommodating and granting us a trial license. His help is very much appreciated!

Before we list the requirements, it is important to note that EC2 instances are Xen Virtual Machines. We encountered some issues running DA on the “Small” Amazon instances. The issues are related to 64-bit mode library mismatches. The only instances to be able to run DirectAdmin out of the box are of type “Large” and better. They are more expensive but ideal for this configuration.

Here are the requirements:

• Instance of type Large
• Linux CentOS 5 image
• One elastic (which is another word for static) IP address

Go ahead and bring up the node as you would normally do. Be sure to remove all extra software that comes pre-installed with the CentOS 5 image. Otherwise, it will break DirectAdmin. If in doubt, follow the instructions on the DirectAdmin Install page:

http://www.directadmin.com/install.html

Once logged in as root, go ahead and download the DirectAdmin install tarball:

# wget http://www.directadmin.com/setup.sh

By default, the licensing scheme of DirectAdmin makes it that setup.sh binds to the licensed IP address. Here’s how the license verification command looks like:

$BIN_DIR/wget $WGET_OPTION -O $DA_PATH/update.tar.gz –bind-address=$IP https://www.directadmin.com/cgi-bin/daupdate?uid=$CID\&lid=$LID

But as you know, EC2 instances are 1:1 private:public NAT. Which means that the license check step will fail and the DA install won’t start. To avoid this chicken-and-egg problem, we are going to tell setup.sh to not bind to any IP address but to simply connect to the DA licensing server using the elastic external IP address, which it will do by default. Go ahead and run this command against the setup.sh file:

# sed -i 's/--bind-address=\$IP//;' setup.sh

Before we run setup.sh, we need to trick the setup script to think that the external IP address is “attached” to the server. Here’s how:

# ifconfig eth0:0 inet 1.1.1.1 netmask 255.255.255.255 up

Be sure to replace the IP address 1.1.1.1 with your elastic IP. Then simply run setup.sh:

# cd /usr/local/directadmin/scripts; while [ true ]; do sed -i 's/--bind-address=\$IP//; s/--bind-address=\${3}//;' *.sh > /dev/null > 2&>1; sleep 10; done & sh setup.sh 11111 22222 vpslux.com eth0:0 1.1.1.1

Woaah! Wait a minute! What is that!? OK, let's break it down. The DA scripts directory gets unpacked into the server after you run setup.sh. It dumps a list of scripts that are used by DA to setup accounts, domains, etc. Some scripts use the same --bind-address flag. Having that flag in those scripts breaks DA. the one-line script "while [ true ]; do sed -i 's/--bind-address=\$IP//; s/--bind-address=\${3}//;' *.sh; sleep 10; done" simply goes to that directory and removes the flag.

The trick being DA attempts to run some of those scripts while setup.sh is running. So it's important to have the sed script run in parallel during the installation. Once the setup.sh is done running, we'll kill the job (or reboot per the final step). The arguments supplied to setup.sh are in this format

# setup.sh <ClientID> <LicenseID> <Hostname> <Interface> <ExternalIP>

Be sure to replace those values accordingly. Once the install finishes, simply reboot the VM:

# reboot

That's all folks. Enjoy!

UNIXy Uncategorized Add new tag, Amazon, DA, DirectAdmin, EC2, linux

There are times when one needs to automate some of the mundate tasks associated with browsing a certain Web page or accomplishing a certain interactive task online. If the task is repetitive, then there has to be a way to automate it using a computer! Browser tasks are no exception. In this brief guide I’m going to show you how to automate some browsing actions on a website called http://www.vpslux.com. More specifically, I’m going to show you how to automate browsing a page called “explained.” Keep in mind that the action could be anything from authenticating to uploading a file!

Lynx is a text browser that runs from the command line, which is also known as “shell.” It runs on most operating systems notably Linux. It comes with a valuable feature that will help us automate actions that are normally performed interactively via a browser (Ex: Firefox, Safari, or Internet Explorer). The two most important features in our case are command logging and command scripting. Per the manual of Lynx:

-cmd_log=FILENAME
write keystroke commands and related information to the specified file.

-cmd_script=FILENAME
read keystroke commands from the specified file.  You can use the data written using the  -cmd_log  option.

Let’s record our actions by browsing http://www.vpslux.com using Lynx. From the command line or shell, launch Lynx with the following arguments and flags:

lynx -cmd_log=/tmp/www.vpslux.com http://www.vpslux.com

The above command will not only take me to the Website in question but also record every keystroke and action I perform and save it inside the file /tmp/www.vpslux.com. It is so simple and so powerful at the same time. Once I’m done with navigating to page “explained” by clicking on the “VPS Explained” located at the top of the first page (http://www.vpslux.com/?page=explained), I hit the letter ‘q’ on my keyboard to exit from Lynx. Inspect the file /tmp/www.vpslux.com to make sure all actions have been recorded. You should see something similar to this depending on the actions you have performed (I cut down quite a bit to keep it short):

# Command logfile created by Lynx 2.8.6rel.4 (15 Nov 2006)
# Arg0 = lynx
# Arg1 = -cmd_log=/tmp/vpslux.com
# Arg2 = http://vpslux.com
key Down Arrow
key Down Arrow
key Down Arrow
key Down Arrow
key Down Arrow
key ^J
key Down Arrow
key Down Arrow
key Down Arrow
key Down Arrow
key Down Arrow
key Down Arrow
key Down Arrow
key Down Arrow
key Down Arrow
key Down Arrow

The hard part is done! All we need to do now is create a small script that will use our www.vpslux.com file as the source or script. You could create a small Bash script called browseWeb.sh with the following content:

#!/bin/bash

lynx -cmd_script=/tmp/www.vpslux.com http://www.vpslux.com/

An extra step would to create a crontab entry to run the above script say every Monday morning at 4am:

crontab -e

0 4 * * 1 /home/user/browseWeb.sh

That’s all folks. I hope this is useful.

UNIXy Crash Course automate, automate Links, automate Lynx, browser actions, Links, Lynx

The MySQL query cache is used to store data retrieved using SELECT statements. The advantage of the query cache is that subsequent and identical SELECT statement results are retrieved directly from the query cache, which resides in memory. So instead of executing the SELECT statements again and again, the stream of data is pulled from memory. The end result is lower CPU and disk hits, which can improve server load average and most importantly responsiveness.

The query cache functionality has enough smartness that once configured, it runs itself. MySQL will make sure to flush cached data if there’s an update that takes place. This ensures stale data is flushed out and replaced with fresh entries. There is, however, an extra step that is required. The cache tends to become fragmented over time. But nothing that cannot be addressed with a simple script (see below).

Stock MySQL configurations set the query cache to 32MB. This means that 32MB worth of server memory is pre-allocated to MySQL. If you have a relatively large database, you will most certainly gain from increasing that value. After configuring the query cache, it takes a while for MySQL to build the cache depending on how busy your MySQL server is.

Let’s get the query cache configured. Inside the [mysqld] section in the MySQL configuration file (ex: /etc/my.cnf), configure the options below. For a server with 8GB memory and a very busy database, set the cache to 1GB:

[mysqld]

query_cache_size = 1024M

Then restart MySQL for the new option to take effect. Run this command to verify that the new query cache value is in place and that the query cache feature is indeed enabled:

# mysql -u root -e “show variables like ‘%query%’”
+——————————+————+
| Variable_name                | Value      |
+——————————+————+
| ft_query_expansion_limit     | 20         |
| have_query_cache             | YES        |
| long_query_time              | 10         |
| query_alloc_block_size       | 8192       |
| query_cache_limit            | 1048576    |
| query_cache_min_res_unit     | 4096       |
| query_cache_size             | 1073741824 |
| query_cache_type             | ON         |
| query_cache_wlock_invalidate | OFF        |
| query_prealloc_size          | 8192       |
+——————————+————+

If the have_query_cache option is set to YES, it means that the query cache is indeed up and running. query_cache_size represents the cache size in bytes. You can also monitor the mysqld process using top and you will see its RES value increase over time to above 1gb. This would be a good sign asb MySQL fills up the query cache with result sets from SELECT. There is another way to check on cache usage and health. Run the following command:

# mysql -u root -e “show status like ‘qc%’”
+————————-+———–+
| Variable_name           | Value     |
+————————-+———–+
| Qcache_free_blocks      | 69776     |
| Qcache_free_memory      | 756433256 |
| Qcache_hits             | 9518567   |
| Qcache_inserts          | 3878394   |
| Qcache_lowmem_prunes    | 49447     |
| Qcache_not_cached       | 173233    |
| Qcache_queries_in_cache | 122754    |
| Qcache_total_blocks     | 315609    |
+————————-+———–+

If Qcache_free_memory stays high over a few days means that the query_cache_size value is set too high. A high Qcache_free_blocks means that memory is fragmented. So it is time to coalesce (make memory blocks contiguous) the region. This is done with the following command:

# mysql -u root -p -e “flush query cache”

Contrary to the command name, this does not flush the cache in the sense that all entries become invalid. It simply defragments the memory region that is assigned to the query cache. The above command should be put in a script that run every 6 hours or so. Create a root crontab entry as such:

* 6/* * * * /root/coalesce.sh

Where coalesce.sh is a shell script:

#!/bin/bash

mysql -u root -pyour_root_password -e "flush query cache";

The query cache can be further configured to match your database’s usage pattern. Be sure to read the MySQL documentation for the details. Hopefully, this post will come in handy for those that have unused memory and need to make the most out of their server. As our customer, we configure the query cache on your server from the get-go. Be sure to ask us how we can help with your online project!

UNIXy Crash Course cache, coalesce, high CPU, memory usage, MySQL, performance improvement, query cache, query_cache_size, speed up mysql