IOPS – Continued

Be sure to read through the previous post for a quick introduction of IOPS.

Suppose that you’re copying a 10GB file from one disk to another. Let us assume that the only bottleneck is the disks’ own IOPS, which in case is 120. We are also only focusing on the disk where the data is copied to and assume that the other disk has no IOPS limitation. Because we are writing about 10GB, the majority of the IOPS are going to be writes and not reads.

IOPS File Transfer

We will also assume that the disk has a block size of 64KB. In other words we are copying a maxium of 64KB of data in one operation (120 operations per second). That gives us a sustained throughput of about 7MB/s (120 * 64 * 1024). For a 10GB file, it would take about 1462 (10GB/7MB) seconds to copy the 10GB file.

Flush Buffers And cache On Linux

Many users tend to worry when the free memory on a Linux system is very low. Especially when the system is barely active. What many don’t know is that this is normal behavior for Linux. It simply takes your free memory and caches it. This caching behavior makes subsequent memory allocations much faster. So it improves performance overall.

However, if you really really need to flush the cache and buffers, you can run the following two commands on the shell as root:

sync; echo 3 > /proc/sys/vm/drop_caches

Here’s the output of free before and after running the above command.

Before

# free -m
             total       used       free     shared    buffers     cached
Mem:           502        487         15          0         15        170
-/+ buffers/cache:        301        201
Swap:         3153        375       2777

After

# free -m
             total       used       free     shared    buffers     cached
Mem:           502        345        157          0          1         44
-/+ buffers/cache:        300        202

As you can see the highlighted numbers, the buffers and cached values have dropped significantly after I ran the command. Again, this is not necessary. In fact, it’s not recommended at all but there it is if you’re curious to know.

That’s all folks. I hope you enjoyed this one.

Web application migration

UNIXy has experience with migrating server applications in and out for customers. This is one of the value-add services we provide to our customers as they switch over to using our services. We are happy to share our ways of performing these tasks. Migrations are done differently in the industry and this is our proven and preferred method. We would like to hear from you about your ways so don’t hesitate to share them with us.

This is part one of a series of guides on how to perform a proper migration of a Web application. Proper because the goal is to seamlessly and preparedly switch over to a new server or provider. We’ll set the bar high enough to only allow room for a 5-10 minutes of cumulative downtime. Note that our experience is limited to Web applications residing on UNIX and/or Linux servers. Also, we are not covering the migration of Web applications that are part of a control panel such as cPanel, Plesk, or DirectAdmin. Control panels have their own migration tools. This guide, however, can assist in migrating a cPanel account over to a non-cPanel server and vice-versa.

While this proposed migration method might seem like a one-time process, it’s imperative to perform dry runs prior to the actual cut-over. Dry runs do not involve downtime at all and can thus help gauge the terrain for potential issues that may arise on go-live day. We call them exercises!

Web Application Migration Part I – Email suction

Email is one the most dreaded services to migrate because of the complexity involved. Not only is it our goal to have the new server send and receive email but also mirror the users’ email inboxes and folders as they were prior to the cutoff. Post-migration, users should wake up to their good old familiar inbox in the undergarment of the new server.

There are tools that one needs to equip herself to carry out an email migration; also commonly called suction in the industry. Our preferred tools are Imapcopy, the venerable Imapsync, and a multi-purpose Linux workstation with the GNU tools; but most importantly, a well seasoned systems administrator. As the saying goes: A cowl does not make a monk.

First things first: gather as much information on the current server as possible. Does the server allow direct IMAP access to users? Is it Webmail only? What about POP access? What is the mailbox / user count on the server? Is there a quota for disk space usage? What is the largest mailbox? Suction and IMAP go hand in hand. It’s only through the IMAP protocol that one can fully interact with a mailbox and its folders. This writeup will not over the nuts and bolts of IMAP or other protocols so be sure to read up on those before continuing.

Imapcopy allows one to copy a mailbox from one server to another using the IMAP protocol. This tool is useful as part of our drill exercises. Before we demo these tools, ensure that an IMAP daemon is listening on both the source and destination.Also, make sure that ports 143 on both the source and destination are open (telnet server 143). Then, download Imapcopy on your workstation. It’s also time to pick an existing mailbox (preferably one with multiple folders) for our trials.

The Imapcopy utility comes packaged with a sample configuration file. For the purpose of this demo, the changes are good enough for a successful run. We’re using our Linux workstation as the actual destination server. Here’s a diff of the changes we made (the symbols < and > and / or spaces were added to avoid getting spam on these inboxes once this page gets crawled):

— ImapCopy.cfg.orig 2008-07-12 23:39:12.000000000 -0500
+++ ImapCopy.cfg 2008-07-13 09:50:52.000000000 -0500
@@ -1,7 +1,7 @@
##############
# Sourceserver
##############
-SourceServer localhost
+SourceServer unixy.net
SourcePort 143

###################
@@ -51,6 +51,6 @@
# List of users and passwords
#############################
# SourceUser SourcePassword DestinationUser DestinationPassword
-Copy “foo” “foosrcpw” “foo” “foodestpw”
-Copy “bar” “barsrcpw” “bar” “test”
+Copy “null <@>unixy.net” “XXXXX” “null” “XXXXX”
+# Copy “bar” “barsrcpw” “bar” “test”

From the above sample configuration file, we are instructing Imapcopy to suction mailbox null <@>unixy.net. But before running the suction, we need to test connectivity and authentication between the “SourceServer” and “DestinationServer”. The flag “-i” of imapcopy achieves this connectivity test:

$ imapcopy -i
IMAPCopy 1.03 – 2005/04/20 [compiled with FreePascal] (c) 2001-2006 Armin Diehl <armin@freepascal.org>
Running on Linux

—————————————————————————
Login on sourceserver as null <@>unixy.net OK
Login on destinationserver as null OK
Sourceserver:
=============
Server-Info : [CAPABILITY IMAP4rev1 UIDPLUS
CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT
THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION
STARTTLS] Courier-IMAP ready. Copyright 1998-2005 Double
Precision, Inc. See COPYING for distribution information.
Capabilities : IMAP4REV1 UIDPLUS CHILDREN
NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION
Personal Namespace : INBOX
Folder sperator : .
other Users Namespace:
Public Namespace : #shared
Folders to copy : ALL
Skip this folders : NONE

Destinationserver:
==================
Server-Info : [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT
THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION]
Courier-IMAP ready. Copyright 1998-2005 Double Precision,
Inc. See COPYING for distribution information.
Capabilities : IMAP4REV1 UIDPLUS CHILDREN
NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT
QUOTA IDLE ACL ACL2=UNION
Personal Namespace : INBOX
Folder sperator : .
other Users Namespace:
Public Namespace : #shared

So far so good. Now let’s have it copy the null <@>unixy. net mailbox in its entirety. But before we do that, Imapcopy requires the default layout of a Maildir exists as such:

$ mkdir -p /home/null/Maildir/{cur,tmp,new}

And now we launch Imapcopy for the full copying of the mailbox:

$ imapcopy
IMAPCopy 1.03 – 2005/04/20 [compiled with FreePascal] (c) 2001-2006 Armin Diehl <armin@freepascal.org>
Running on Linux

Login on sourceserver as null <@>unixy.net OK
Login on destinationserver as null OK
Getting folderlist on sourceserver OK, found 1 folder
Getting List of messages in “INBOX” OK, 2 Messages found
Processing Folder INBOX
S:A0004 SELECT INBOX\r\n
R:* FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)\r\n
R:* OK [PERMANENTFLAGS (\* \Draft \Answered \Flagged \Deleted \Seen)] Limited\r\n
R:* 0 EXISTS\r\n
R:* 0 RECENT\r\n
R:* OK [UIDVALIDITY 1215995880] Ok\r\n
R:* OK [MYRIGHTS “acdilrsw”] ACL\r\n
R:A0004 OK [READ-WRITE] Ok\r\n
S:A0005 APPEND INBOX (\Seen) “14-Jul-2008 21:46:39 -0500” {903}\r\n
R:+ OK\r\n
C: Command exit because of + (“+ OK”)
S: Message not shown (903 Bytes)
R:A0005 OK [APPENDUID 1215995880 9] APPEND Ok.\r\n
S:A0006 APPEND INBOX (\Seen) “14-Jul-2008 21:47:25 -0500” {716}\r\n
R:+ OK\r\n
C: Command exit because of + (“+ OK”)
S: Message not shown (716 Bytes)
R:A0006 OK [APPENDUID 1215995880 10] APPEND Ok.\r\n
2 Messages copied, 0 Errors
S:A0007 LOGOUT\r\n
R:* BYE Courier-IMAP server shutting down\r\n
R:A0007 OK LOGOUT completed\r\n
S:A0008 LOGOUT\r\n
C: Command status 1

1 User processed, 2 Messages copied, 0 Error(s)
0 Folder(s) created, 0 Folder create errors, 0 Folder not copied

All good! As you can see from the above log, Imapcopy suctioned 2 messages for one user, null <@>unixy.net in this case. One extra step:

$ ls -al /home/null/Maildir/cur/
total 16
drwxr-xr-x 2 null null 4096 2008-07-14 21:48 .
drwxrwxrwx 6 null null 4096 2008-07-13 19:38 ..
-rw-r–r– 1 null null 695 2008-07-14 21:47 1216090136.M168971P8436V0000000000000305I00175CCF_1.MUX,S=695:2,S
-rw-r–r– 1 null null 874 2008-07-14 21:46 1216090136.M30165P8436V0000000000000305I00175CCD_0.MUX,S=874:2,S

Impeccable! While Imapcopy could potentially be used to carry out the migration, it is not the prefered method of suction. If it were to be used as part of a migration strategy, one would have to batch all the user mailboxes in one fell swoop. It makes the process time consuming and causes downtime especially when you have to transfer very large mailboxes. Imapcopy’s copying mechanism is also not incremental or retriable and that is a valid concern. Every time Imapcopy runs, it copies the mailbox in its entirety regardless. Here’s an example of a re-run. For brevity’s sake, we’re only showcasing the statistics of the run:

$ imapcopy
Login on sourceserver as null@unixy.net OK
Login on destinationserver as null OK
.
.
.
1 User processed, <b>2 Messages copied</b>, 0 Error(s)
0 Folder(s) created, 0 Folder create errors, 0 Folder not copied

Indeed, Imapcopy has no notion of incremental copying. You could see how it can get out of control with a large mailbox.

The goal here is to suction email messages per-user as they login into the new server for the first time. This is where Imapsync comes in. You supply the source and destination server, the mailbox to sync, and the password of the user’s mailbox for both the source and destination; imapsync does the rest.

First, let’s sync our IMAP mail box manually to test the waters:

$ imapsync –host1 unixy.net –user1 null <@>unixy.net –password1 XXXXX –host2 localhost –user2 null –password2 XXXXX
$RCSfile: imapsync,v $ $Revision: 1.219 $ $Date: 2007/04/04 09:32:20 $
Here is a linux system Linux MUX 2.6.20-16-server #2 SMP Tue Dec 18 05:52:19 UTC 2007 i686)
with perl 5.8.8
Mail::IMAPClient version used here is 2.2.9
Command line used :
/usr/bin/imapsync –host1 unixy.net –user1 null <@>unixy.net –password1 XXXXX –host2 localhost –user2 null –password2 XXXXX
will try to use CRAM-MD5 authentication on host1
will try to use CRAM-MD5 authentication on host2
From imap server [unixy.net] port [143] user [null <@>unixy.net]
To imap server [localhost] port [143] user [null]
Banner : * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN
NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT
QUOTA IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready.
Copyright 1998-2005 Double Precision, Inc. See COPYING
for distribution information.
Host unixy.net says it has NO CAPABILITY for AUTHENTICATE CRAM-MD5
Error login : [unixy.net] with user [null <@>unixy.net] auth [CRAM-MD5]: 3 NO Login failed.

Trying LOGIN Auth mechanism on [unixy.net] with user [null <@>unixy.net]
Success login on [unixy.net] with user [null <@>unixy.net] auth [CRAM-MD5]
Banner : * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN
NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT
QUOTA IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready.
Copyright 1998-2005 Double Precision, Inc. See COPYING
for distribution information.
Host localhost says it has NO CAPABILITY for AUTHENTICATE CRAM-MD5
Error login : [localhost] with user [null] auth [CRAM-MD5]: 3 NO Login failed.

Trying LOGIN Auth mechanism on [localhost] with user [null]
Success login on [localhost] with user [null] auth [CRAM-MD5]
From capability : QUOTA STARTTLS NAMESPACE IDLE
THREAD=ORDEREDSUBJECT ACL SORT UIDPLUS CHILDREN
ACL2=UNION IMAP4REV1 THREAD=REFERENCES
To capability : QUOTA STARTTLS NAMESPACE IDLE
THREAD=ORDEREDSUBJECT ACL SORT UIDPLUS CHILDREN
ACL2=UNION IMAP4REV1 THREAD=REFERENCES
From state Authenticated
To state Authenticated
From separator and prefix : [.][INBOX.]
To separator and prefix : [.][INBOX.]
++++ Calculating sizes ++++
From Folder [INBOX] Size: 1623 Messages: 2
Total size: 1623
Total messages: 2
Time : 12 s
++++ Calculating sizes ++++
To Folder [INBOX] Size: 1619 Messages: 2
Total size: 1619
Total messages: 2
Time : 0 s
++++ Listing folders ++++
From folders list : [INBOX]
To folders list : [INBOX]
++++ Looping on each folder ++++
From Folder [INBOX]
To Folder [INBOX]
++++ From [INBOX] Parse 1 ++++
++++ To [INBOX] Parse 1 ++++
++++ Verifying [INBOX] -> [INBOX] ++++
+ NO msg #5 [imWT88lfpdVUriyDdDcAOA:905] in INBOX
+ Copying msg #5:905 to folder INBOX
flags from : [\Seen][]
Copied msg id [5] to folder INBOX msg id [11]
+ NO msg #6 [lOzMkdoLOuFH7qheMK4NkQ:718] in INBOX
+ Copying msg #6:718 to folder INBOX
flags from : [\Seen][]
Copied msg id [6] to folder INBOX msg id [12]
Time : 1 s
++++ Statistics ++++
Time : 13 sec
Messages transferred : 2
Messages skipped : 0
Total bytes transferred: 1623
Total bytes skipped : 0
Total bytes error : 0
Detected 0 errors

At this point in the game, there’s no difference between imapcopy and imapsync. They both appear to fulfill the same role; copy the messages from one inbox to another. However, imapsync is much more capable than it appears on a first glimpse. As we’ll see in this next run, imapsync has more capabilities packed in.

++++ Statistics ++++
Time : 12 sec
<b>Messages transferred : 0 </b>
Messages skipped : 2
Total bytes transferred: 0
Total bytes skipped : 1623
Total bytes error : 0

Detected 0 errors

In the above snippet, we only display the compte rendu of the second run. As you can see, no messages were transfered over to the destination inbox. Impasync is an incremental tool. Big deal!

At this point, all that is needed is a source and destination host, a list of users and respective passwords. But our goal is not to batch the whole operation. The reason being we want to avoid long running jobs with higher latency that could affect end-user experience. Therefore we have to come up with a per-user at-logon solution.

It is common during migrations to ask users to visit a “migration” link in order to get the conversion or mailbox sync started. This is an OK approach and it works fine with Imapsync. The webmail page in question launches Imapsync on the server-side code of the page; quite simple. There’s an alternative, which is to hijack the IMAP software running on the legacy server. The latter method involves as creating a wrapper around the binary so that not only authentication is performed (as usual) but also an Imapsync session is started in the foreground or background. We won’t go over this here for now as we’ll stick with the migration link approach.

In the next installment, we’ll go over Web migration and then follow it up with database migration.