I am proud to pre-announce our strategic expansion of Web hosting operations in Europe and in North Africa. Taking on this initiative is consistent with our vision to become a world-wide player in the managed server arena. More information will become available soon.

Disclaimer: this post covers Linux only. The installation steps will also only work with Linux Redhat and Debian derivatives. They can be made to work with other Linux and Unix flavors as well.

What Is This Post About

There are times when one can’t tell what is it that’s causing their server (as in VPS or dedicated server) to perform poorly. This post will hopefully help someone make an educated decision on whether to proceed with an upgrade or not. At least, it will help someone find the culprit and act accordingly.

The Basics First

There are mainly four components that can make or break server responsiveness. Those are hard disk, memory, CPU, and bandwidth. It only takes one weak component to cause a server to be unresponsive. Linux provides tools that help visualize and pinpoint the lowest performing component(s). The tools are somewhat analogically similar to a car dashboard. You can tell whether the car is running out of gas or if the engine temperature is higher than usual. etcetera. Ultimately, one can correlate component weakness with server responsiveness.

So, what makes a server sluggish? Is it the lack of memory or the lack of CPU cycles? It could be either, neither, or both. It’s essential to realize that there are so many jobs a CPU can run before it’s pegged. Hard disks can only take so many requests before it saturates. As the components are fed more work than they can handle, the system starts queueing jobs. The queuing of jobs is one of the earliest signs of a server that has reached capacity.

Introducing The Dashboard

How do we catch the early signs of a busy server? Unfortunately, and for a non-maintained server, user complaints is one of the earliest signs. There’s no doubt that we need to be proactive and on the lookout. There’s the great Top tool but it doesn’t provide a holistic view of what’s cooking on the srever. The good news is that there’s a tool that will do all the mundane work for us, all day long unattended. And from the comfort of your home once a day or once a week, you can go over the collected information.

My favorite tool is SAR, which stands for System Activity Reports. SAR is a collection of programs that run in the background on minimal resources and silently collect vital system data. It collects hard disk, CPU, memory, network information, and much more. While the SAR command line tools are all you need to display the data, there’s a bit of a learning curve to it. But don’t be discouraged; there are other ways, besides the command line, to display the data. We’ll go over an example later on.

So, let’s get SAR up and running. For those that have skipped the disclaimer, this tutorial deals with Linux Redhat and Debian compatible installs only. Therefore, it may or may not work with all Linux distributions. The following instructions require that you remote into the server. You should have a shell / command line open and ready for your input.

Installing SAR

ON DEBIAN

As root, install the sysstat package by executing the following command as root:

Edit the file /etc/default/sysstat and change the line:

To:

No action is required on your part if it’s already set to true. Also, edit the file /etc/sysstat/sysstat and set the variable HISTORY to however many days you want SAR to keep historical data:

To

And last, kick off the process:

ON REDHAT

Install the sysstat tools

Edit the /etc/sysconfig/sysstat file and change the variable HISTORY as such:

To

Then restart sysstat:

That’s all you have to do on the server side. We’ll have to let SAR run for a few hours and let it collect data.

The Fun Part

This is the part where we get to see the data in pretty graphs and charts! A tool that goes with the name kSar, and that you can download for free, will itself remote into your server, download the SAR-generated files, and neatly graph them for you. This is the best tool I have found so far. Simply point your browser to http://sourceforge.net/project/showf…ease_id=645912 and hit the download link. Unzip the file, and double click on the JAR file. Once the application has launched, select the “Data” option in the menu, then “Launch SSH Command”. Input your server parameters (Ex: root@vpslux.com:22). Finally, select “Yes” when prompted for running the command “sar -A”. I’ve attached some screenshots of kSar in action. kSar has a shortcoming, however. It only displays one day worth of data at a time. But you can always save the daily reports as a PDF file and view them at a later date.

Back To Work!

Let’s make some sense out of those graphs. Of interest are the following panel tabs: CPU all, Swapping, Memory Usage, Load, Processes, Paging Activity, Sockets, and finally Interface traffic under the Interface tab. We can group Swapping, Memory Usage, and Paging Activity together under the same category as they’re correlated to an extent. CPU all, Sockets and Interface traffic can be regarded as two independent indicators.

About 99% of the time, Sockets and Interface traffic are leading indicators in the sense that the number of open sockets and traffic increases as soon as the node in question sees more traffic as a result of user visits. CPU all shows the amount of CPU used over time. You can ignore short, occasional spikes. But sustained usage of 90% and above for periods of time is usually a sign of pegged CPUs. It’s most likely time to upgrade the CPUs or dig further into what’s hogging CPU. Swapping is the next most interesting indicator to look at. Any spikes here are usually a sign of lack of free memory. Let’s select Memory Usage to confirm. If Memory Usage is anywhere less than roughly 4MB over extended periods of time, then the box is begging for a memory upgrade (in conjunction with Swapping indicator). Paging, and in particular a spike in page faults, can further confirm the lack of memory.

To recap, and in general, interface traffic and therefore number of sockets increase, which leads to resources being consumed in order to serve data. The latter then causes either memory, CPU, or both to spike up, which then might cause swapping (disk I/O). One might suggest improving disk I/O but swapping is only a symptom of the issue. In this scenario, the lack of memory is definitely the root cause of the issue. But keep in mind that there are scenarios when disk I/O spikes up while memory is plenty available on the system. This is an easy case to resolve as most likely a rogue process is hitting the disks more than it should. One last scenario is when CPU, memory, and disk I/O are all almost flat but responsiveness is sluggish. In this case, pay close attention to the Interface traffic and Interface errors graphs for either saturation of your allocated port speed or errors.

That’s all folks. Feel free to reply back with questions, comments, corrections.

UNIXy has experience with migrating server applications in and out for customers. This is one of the value-add services we provide to our customers as they switch over to using our services. We are happy to share our ways of performing these tasks. Migrations are done differently in the industry and this is our proven and preferred method. We would like to hear from you about your ways so don’t hesitate to share them with us.

This is part one of a series of guides on how to perform a proper migration of a Web application. Proper because the goal is to seamlessly and preparedly switch over to a new server or provider. We’ll set the bar high enough to only allow room for a 5-10 minutes of cumulative downtime. Note that our experience is limited to Web applications residing on UNIX and/or Linux servers. Also, we are not covering the migration of Web applications that are part of a control panel such as cPanel, Plesk, or DirectAdmin. Control panels have their own migration tools. This guide, however, can assist in migrating a cPanel account over to a non-cPanel server and vice-versa.

While this proposed migration method might seem like a one-time process, it’s imperative to perform dry runs prior to the actual cut-over. Dry runs do not involve downtime at all and can thus help gauge the terrain for potential issues that may arise on go-live day. We call them exercises!

Web Application Migration Part I – Email suction

Email is one the most dreaded services to migrate because of the complexity involved. Not only is it our goal to have the new server send and receive email but also mirror the users’ email inboxes and folders as they were prior to the cutoff. Post-migration, users should wake up to their good old familiar inbox in the undergarment of the new server.

There are tools that one needs to equip herself to carry out an email migration; also commonly called suction in the industry. Our preferred tools are Imapcopy, the venerable Imapsync, and a multi-purpose Linux workstation with the GNU tools; but most importantly, a well seasoned systems administrator. As the saying goes: A cowl does not make a monk.

First things first: gather as much information on the current server as possible. Does the server allow direct IMAP access to users? Is it Webmail only? What about POP access? What is the mailbox / user count on the server? Is there a quota for disk space usage? What is the largest mailbox? Suction and IMAP go hand in hand. It’s only through the IMAP protocol that one can fully interact with a mailbox and its folders. This writeup will not over the nuts and bolts of IMAP or other protocols so be sure to read up on those before continuing.

Imapcopy allows one to copy a mailbox from one server to another using the IMAP protocol. This tool is useful as part of our drill exercises. Before we demo these tools, ensure that an IMAP daemon is listening on both the source and destination.Also, make sure that ports 143 on both the source and destination are open (telnet server 143). Then, download Imapcopy on your workstation. It’s also time to pick an existing mailbox (preferably one with multiple folders) for our trials.

The Imapcopy utility comes packaged with a sample configuration file. For the purpose of this demo, the changes are good enough for a successful run. We’re using our Linux workstation as the actual destination server. Here’s a diff of the changes we made (the symbols < and > and / or spaces were added to avoid getting spam on these inboxes once this page gets crawled):

— ImapCopy.cfg.orig 2008-07-12 23:39:12.000000000 -0500
+++ ImapCopy.cfg 2008-07-13 09:50:52.000000000 -0500
@@ -1,7 +1,7 @@
##############
# Sourceserver
##############
-SourceServer localhost
+SourceServer unixy.net
SourcePort 143

###################
@@ -51,6 +51,6 @@
# List of users and passwords
#############################
# SourceUser SourcePassword DestinationUser DestinationPassword
-Copy “foo” “foosrcpw” “foo” “foodestpw”
-Copy “bar” “barsrcpw” “bar” “test”
+Copy “null <@>unixy.net” “XXXXX” “null” “XXXXX”
+# Copy “bar” “barsrcpw” “bar” “test”

From the above sample configuration file, we are instructing Imapcopy to suction mailbox null <@>unixy.net. But before running the suction, we need to test connectivity and authentication between the “SourceServer” and “DestinationServer”. The flag “-i” of imapcopy achieves this connectivity test:

$ imapcopy -i
IMAPCopy 1.03 – 2005/04/20 [compiled with FreePascal] (c) 2001-2006 Armin Diehl <armin@freepascal.org>
Running on Linux

—————————————————————————
Login on sourceserver as null <@>unixy.net OK
Login on destinationserver as null OK
Sourceserver:
=============
Server-Info : [CAPABILITY IMAP4rev1 UIDPLUS
CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT
THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION
STARTTLS] Courier-IMAP ready. Copyright 1998-2005 Double
Precision, Inc. See COPYING for distribution information.
Capabilities : IMAP4REV1 UIDPLUS CHILDREN
NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION
Personal Namespace : INBOX
Folder sperator : .
other Users Namespace:
Public Namespace : #shared
Folders to copy : ALL
Skip this folders : NONE

Destinationserver:
==================
Server-Info : [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT
THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION]
Courier-IMAP ready. Copyright 1998-2005 Double Precision,
Inc. See COPYING for distribution information.
Capabilities : IMAP4REV1 UIDPLUS CHILDREN
NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT
QUOTA IDLE ACL ACL2=UNION
Personal Namespace : INBOX
Folder sperator : .
other Users Namespace:
Public Namespace : #shared

So far so good. Now let’s have it copy the null <@>unixy. net mailbox in its entirety. But before we do that, Imapcopy requires the default layout of a Maildir exists as such:

$ mkdir -p /home/null/Maildir/{cur,tmp,new}

And now we launch Imapcopy for the full copying of the mailbox:

$ imapcopy
IMAPCopy 1.03 – 2005/04/20 [compiled with FreePascal] (c) 2001-2006 Armin Diehl <armin@freepascal.org>
Running on Linux

Login on sourceserver as null <@>unixy.net OK
Login on destinationserver as null OK
Getting folderlist on sourceserver OK, found 1 folder
Getting List of messages in “INBOX” OK, 2 Messages found
Processing Folder INBOX
S:A0004 SELECT INBOX\r\n
R:* FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)\r\n
R:* OK [PERMANENTFLAGS (\* \Draft \Answered \Flagged \Deleted \Seen)] Limited\r\n
R:* 0 EXISTS\r\n
R:* 0 RECENT\r\n
R:* OK [UIDVALIDITY 1215995880] Ok\r\n
R:* OK [MYRIGHTS "acdilrsw"] ACL\r\n
R:A0004 OK [READ-WRITE] Ok\r\n
S:A0005 APPEND INBOX (\Seen) “14-Jul-2008 21:46:39 -0500″ {903}\r\n
R:+ OK\r\n
C: Command exit because of + (“+ OK”)
S: Message not shown (903 Bytes)
R:A0005 OK [APPENDUID 1215995880 9] APPEND Ok.\r\n
S:A0006 APPEND INBOX (\Seen) “14-Jul-2008 21:47:25 -0500″ {716}\r\n
R:+ OK\r\n
C: Command exit because of + (“+ OK”)
S: Message not shown (716 Bytes)
R:A0006 OK [APPENDUID 1215995880 10] APPEND Ok.\r\n
2 Messages copied, 0 Errors
S:A0007 LOGOUT\r\n
R:* BYE Courier-IMAP server shutting down\r\n
R:A0007 OK LOGOUT completed\r\n
S:A0008 LOGOUT\r\n
C: Command status 1

1 User processed, 2 Messages copied, 0 Error(s)
0 Folder(s) created, 0 Folder create errors, 0 Folder not copied

All good! As you can see from the above log, Imapcopy suctioned 2 messages for one user, null <@>unixy.net in this case. One extra step:

$ ls -al /home/null/Maildir/cur/
total 16
drwxr-xr-x 2 null null 4096 2008-07-14 21:48 .
drwxrwxrwx 6 null null 4096 2008-07-13 19:38 ..
-rw-r–r– 1 null null 695 2008-07-14 21:47 1216090136.M168971P8436V0000000000000305I00175CCF_1.MUX,S=695:2,S
-rw-r–r– 1 null null 874 2008-07-14 21:46 1216090136.M30165P8436V0000000000000305I00175CCD_0.MUX,S=874:2,S

Impeccable! While Imapcopy could potentially be used to carry out the migration, it is not the prefered method of suction. If it were to be used as part of a migration strategy, one would have to batch all the user mailboxes in one fell swoop. It makes the process time consuming and causes downtime especially when you have to transfer very large mailboxes. Imapcopy’s copying mechanism is also not incremental or retriable and that is a valid concern. Every time Imapcopy runs, it copies the mailbox in its entirety regardless. Here’s an example of a re-run. For brevity’s sake, we’re only showcasing the statistics of the run:

$ imapcopy
Login on sourceserver as null@unixy.net OK
Login on destinationserver as null OK
.
.
.
1 User processed, <b>2 Messages copied</b>, 0 Error(s)
0 Folder(s) created, 0 Folder create errors, 0 Folder not copied

Indeed, Imapcopy has no notion of incremental copying. You could see how it can get out of control with a large mailbox.

The goal here is to suction email messages per-user as they login into the new server for the first time. This is where Imapsync comes in. You supply the source and destination server, the mailbox to sync, and the password of the user’s mailbox for both the source and destination; imapsync does the rest.

First, let’s sync our IMAP mail box manually to test the waters:

$ imapsync –host1 unixy.net –user1 null <@>unixy.net –password1 XXXXX –host2 localhost –user2 null –password2 XXXXX
$RCSfile: imapsync,v $ $Revision: 1.219 $ $Date: 2007/04/04 09:32:20 $
Here is a linux system Linux MUX 2.6.20-16-server #2 SMP Tue Dec 18 05:52:19 UTC 2007 i686)
with perl 5.8.8
Mail::IMAPClient version used here is 2.2.9
Command line used :
/usr/bin/imapsync –host1 unixy.net –user1 null <@>unixy.net –password1 XXXXX –host2 localhost –user2 null –password2 XXXXX
will try to use CRAM-MD5 authentication on host1
will try to use CRAM-MD5 authentication on host2
From imap server [unixy.net] port [143] user [null <@>unixy.net]
To imap server [localhost] port [143] user [null]
Banner : * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN
NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT
QUOTA IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready.
Copyright 1998-2005 Double Precision, Inc. See COPYING
for distribution information.
Host unixy.net says it has NO CAPABILITY for AUTHENTICATE CRAM-MD5
Error login : [unixy.net] with user [null <@>unixy.net] auth [CRAM-MD5]: 3 NO Login failed.

Trying LOGIN Auth mechanism on [unixy.net] with user [null <@>unixy.net]
Success login on [unixy.net] with user [null <@>unixy.net] auth [CRAM-MD5]
Banner : * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN
NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT
QUOTA IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready.
Copyright 1998-2005 Double Precision, Inc. See COPYING
for distribution information.
Host localhost says it has NO CAPABILITY for AUTHENTICATE CRAM-MD5
Error login : [localhost] with user [null] auth [CRAM-MD5]: 3 NO Login failed.

Trying LOGIN Auth mechanism on [localhost] with user [null]
Success login on [localhost] with user [null] auth [CRAM-MD5]
From capability : QUOTA STARTTLS NAMESPACE IDLE
THREAD=ORDEREDSUBJECT ACL SORT UIDPLUS CHILDREN
ACL2=UNION IMAP4REV1 THREAD=REFERENCES
To capability : QUOTA STARTTLS NAMESPACE IDLE
THREAD=ORDEREDSUBJECT ACL SORT UIDPLUS CHILDREN
ACL2=UNION IMAP4REV1 THREAD=REFERENCES
From state Authenticated
To state Authenticated
From separator and prefix : [.][INBOX.]
To separator and prefix : [.][INBOX.]
++++ Calculating sizes ++++
From Folder [INBOX] Size: 1623 Messages: 2
Total size: 1623
Total messages: 2
Time : 12 s
++++ Calculating sizes ++++
To Folder [INBOX] Size: 1619 Messages: 2
Total size: 1619
Total messages: 2
Time : 0 s
++++ Listing folders ++++
From folders list : [INBOX]
To folders list : [INBOX]
++++ Looping on each folder ++++
From Folder [INBOX]
To Folder [INBOX]
++++ From [INBOX] Parse 1 ++++
++++ To [INBOX] Parse 1 ++++
++++ Verifying [INBOX] -> [INBOX] ++++
+ NO msg #5 [imWT88lfpdVUriyDdDcAOA:905] in INBOX
+ Copying msg #5:905 to folder INBOX
flags from : [\Seen][]
Copied msg id [5] to folder INBOX msg id [11]
+ NO msg #6 [lOzMkdoLOuFH7qheMK4NkQ:718] in INBOX
+ Copying msg #6:718 to folder INBOX
flags from : [\Seen][]
Copied msg id [6] to folder INBOX msg id [12]
Time : 1 s
++++ Statistics ++++
Time : 13 sec
Messages transferred : 2
Messages skipped : 0
Total bytes transferred: 1623
Total bytes skipped : 0
Total bytes error : 0
Detected 0 errors

At this point in the game, there’s no difference between imapcopy and imapsync. They both appear to fulfill the same role; copy the messages from one inbox to another. However, imapsync is much more capable than it appears on a first glimpse. As we’ll see in this next run, imapsync has more capabilities packed in.

++++ Statistics ++++
Time : 12 sec
<b>Messages transferred : 0 </b>
Messages skipped : 2
Total bytes transferred: 0
Total bytes skipped : 1623
Total bytes error : 0

Detected 0 errors

In the above snippet, we only display the compte rendu of the second run. As you can see, no messages were transfered over to the destination inbox. Impasync is an incremental tool. Big deal!

At this point, all that is needed is a source and destination host, a list of users and respective passwords. But our goal is not to batch the whole operation. The reason being we want to avoid long running jobs with higher latency that could affect end-user experience. Therefore we have to come up with a per-user at-logon solution.

It is common during migrations to ask users to visit a “migration” link in order to get the conversion or mailbox sync started. This is an OK approach and it works fine with Imapsync. The webmail page in question launches Imapsync on the server-side code of the page; quite simple. There’s an alternative, which is to hijack the IMAP software running on the legacy server. The latter method involves as creating a wrapper around the binary so that not only authentication is performed (as usual) but also an Imapsync session is started in the foreground or background. We won’t go over this here for now as we’ll stick with the migration link approach.

In the next installment, we’ll go over Web migration and then follow it up with database migration.