UNIXy

In this short post, I would like to share a technique that will protect your confidential data even if your backup store were to be compromised. We shall leverage the powerful open source Encryption Filesystem. I’ll go through all steps required to install the software, use it, and finally integrate it with your back up strategy. This is a one-time configuration that doesn’t require much maintenance to keep it going and is well worth it in my opinion.

Off-server or off-network backup procedures are essential to any disaster recovery strategy. Current trends, however, show that little effort is directed at securing the backup node(s) and / or strategy. By storing plain text copies of your confidential databases, accounts, emails, and passwords on remote systems you’re exposing yourself to a host of issues. In light of the incident that affected WHT, if a capable intruder were to compromise your backup store (VPS, FTP, NFS, or server), it won’t take long before the intruder gains access to your production system. The consequences are material and the loss of productivity and revenue can break a business.

Installing EncFS

While I’m only covering installation of EncFS on Debian and Redhat derivatives, it’s relatively easy to install it on other Linux distributions. Special instructions are required to install the tools on OpenVZ. See http://wiki.openvz.org/FUSE

ON DEBIAN DISTRIBUTIONS

Let’s install EncFS and libraries. As root inside the shell prompt, execute the following two commands:

ON REDHAT DISTRIBUTIONS

First you have to add a yum application repository. Create a file called rpmforge.repo under /etc/yum.repos.d/rpmforge.repo and, with a text editor, copy / paste the following in it (this is for centos 5 / redhat 5. Checkout DAG for other versions):

Save and exit. Then run the following commands as root:

Using EncFS

EncFS is a set of tools that allow the creation of a filesystem that is by default encrypted. The encrypted filesystem can be mounted similarly to a hard drive. With EncFS, however, the encrypted filesystem is protected by a password. And this is where it’s useful. When you transfer your backup files from your production server to an off-server backup store, you’re transferring and storing clear text files and information. So, how do we use these tools to secure our backup store?

In brief, here are the steps we’re setting to accomplish

A) Initialize a folder on the production server as an EncFS volume and mount it
B) Point our backup scripts to the encrypted volume to store the generated backups
C) Seal the encrypted volume
D) Finally, transfer the encrypted files over to the backup store

A) First of all, we need to initialize the backup filesystem. Here’s are the steps:

At this point in the steps, we have created an encrypted and a decrypted folder. Plain text backups should always be copied in the /decrypted folder. Once copied, we unmounted the decrypted folder and leave all as is. Make sure you remember the Encfs password as it’s the only way to decrypt your backup files.

As a quick demo, let’s copy a random file in /decrypted to see all of this in action

Let’s pick a random file

Mount the encrypted filesystem:

B) If you have custom backup scripts, all you have to do in this step is the following

1) Before we modify the backup scripts, we need to store the encryption password in a file under the folder /root. Call it file /root/enc.txt and on the first line type in the password after running the below chmod command.

Add this command at the top of the backup script:

What this does is “feed” the encryption password to the command “encfs” so it runs unattended. Otherwise, encfs is interactive and might hand waiting for you to enter the password. Remember, we want to set this up and let it run itself.

Add this command at the end of the backup script:

For cPanel users, you can put include the above two steps in script files called /scripts/precpbackup and /scripts/postcpbackup as such:

Inside file /scripts/precpbackup

# In file /scripts/postcpbackup

Finally, make sure the two scripts are executable:

From WHM, in backup configuration, put /decrypted as the backup folder. And we’re done!

C) Let’s unmount the unencrypted filesystem since we’re done copying our files.

Sweet!

D) Transfer the encrypted backup files to the destination backup store

Now your backup files are secure. You can simply SCP or rsync the encrypted files from the encrypted FS /encrypted. Make sure to copy the .encfs5 file located inside the /encrypted directory. Without this file, the encrypted file are NOT recoverable!

UNIXy Security backup, cPanel, cpbackup, DR, EncFS, Security, WHM, WHT

Disclaimer: this post covers Linux only. The installation steps will also only work with Linux Redhat and Debian derivatives. They can be made to work with other Linux and Unix flavors as well.

What Is This Post About

There are times when one can’t tell what is it that’s causing their server (as in VPS or dedicated server) to perform poorly. This post will hopefully help someone make an educated decision on whether to proceed with an upgrade or not. At least, it will help someone find the culprit and act accordingly.

The Basics First

There are mainly four components that can make or break server responsiveness. Those are hard disk, memory, CPU, and bandwidth. It only takes one weak component to cause a server to be unresponsive. Linux provides tools that help visualize and pinpoint the lowest performing component(s). The tools are somewhat analogically similar to a car dashboard. You can tell whether the car is running out of gas or if the engine temperature is higher than usual. etcetera. Ultimately, one can correlate component weakness with server responsiveness.

So, what makes a server sluggish? Is it the lack of memory or the lack of CPU cycles? It could be either, neither, or both. It’s essential to realize that there are so many jobs a CPU can run before it’s pegged. Hard disks can only take so many requests before it saturates. As the components are fed more work than they can handle, the system starts queueing jobs. The queuing of jobs is one of the earliest signs of a server that has reached capacity.

Introducing The Dashboard

How do we catch the early signs of a busy server? Unfortunately, and for a non-maintained server, user complaints is one of the earliest signs. There’s no doubt that we need to be proactive and on the lookout. There’s the great Top tool but it doesn’t provide a holistic view of what’s cooking on the srever. The good news is that there’s a tool that will do all the mundane work for us, all day long unattended. And from the comfort of your home once a day or once a week, you can go over the collected information.

My favorite tool is SAR, which stands for System Activity Reports. SAR is a collection of programs that run in the background on minimal resources and silently collect vital system data. It collects hard disk, CPU, memory, network information, and much more. While the SAR command line tools are all you need to display the data, there’s a bit of a learning curve to it. But don’t be discouraged; there are other ways, besides the command line, to display the data. We’ll go over an example later on.

So, let’s get SAR up and running. For those that have skipped the disclaimer, this tutorial deals with Linux Redhat and Debian compatible installs only. Therefore, it may or may not work with all Linux distributions. The following instructions require that you remote into the server. You should have a shell / command line open and ready for your input.

Installing SAR

ON DEBIAN

As root, install the sysstat package by executing the following command as root:

Edit the file /etc/default/sysstat and change the line:

To:

No action is required on your part if it’s already set to true. Also, edit the file /etc/sysstat/sysstat and set the variable HISTORY to however many days you want SAR to keep historical data:

To

And last, kick off the process:

ON REDHAT

Install the sysstat tools

Edit the /etc/sysconfig/sysstat file and change the variable HISTORY as such:

To

Then restart sysstat:

That’s all you have to do on the server side. We’ll have to let SAR run for a few hours and let it collect data.

The Fun Part

This is the part where we get to see the data in pretty graphs and charts! A tool that goes with the name kSar, and that you can download for free, will itself remote into your server, download the SAR-generated files, and neatly graph them for you. This is the best tool I have found so far. Simply point your browser to http://sourceforge.net/project/showf…ease_id=645912 and hit the download link. Unzip the file, and double click on the JAR file. Once the application has launched, select the “Data” option in the menu, then “Launch SSH Command”. Input your server parameters (Ex: root@vpslux.com:22). Finally, select “Yes” when prompted for running the command “sar -A”. I’ve attached some screenshots of kSar in action. kSar has a shortcoming, however. It only displays one day worth of data at a time. But you can always save the daily reports as a PDF file and view them at a later date.

Back To Work!

Let’s make some sense out of those graphs. Of interest are the following panel tabs: CPU all, Swapping, Memory Usage, Load, Processes, Paging Activity, Sockets, and finally Interface traffic under the Interface tab. We can group Swapping, Memory Usage, and Paging Activity together under the same category as they’re correlated to an extent. CPU all, Sockets and Interface traffic can be regarded as two independent indicators.

About 99% of the time, Sockets and Interface traffic are leading indicators in the sense that the number of open sockets and traffic increases as soon as the node in question sees more traffic as a result of user visits. CPU all shows the amount of CPU used over time. You can ignore short, occasional spikes. But sustained usage of 90% and above for periods of time is usually a sign of pegged CPUs. It’s most likely time to upgrade the CPUs or dig further into what’s hogging CPU. Swapping is the next most interesting indicator to look at. Any spikes here are usually a sign of lack of free memory. Let’s select Memory Usage to confirm. If Memory Usage is anywhere less than roughly 4MB over extended periods of time, then the box is begging for a memory upgrade (in conjunction with Swapping indicator). Paging, and in particular a spike in page faults, can further confirm the lack of memory.

To recap, and in general, interface traffic and therefore number of sockets increase, which leads to resources being consumed in order to serve data. The latter then causes either memory, CPU, or both to spike up, which then might cause swapping (disk I/O). One might suggest improving disk I/O but swapping is only a symptom of the issue. In this scenario, the lack of memory is definitely the root cause of the issue. But keep in mind that there are scenarios when disk I/O spikes up while memory is plenty available on the system. This is an easy case to resolve as most likely a rogue process is hitting the disks more than it should. One last scenario is when CPU, memory, and disk I/O are all almost flat but responsiveness is sluggish. In this case, pay close attention to the Interface traffic and Interface errors graphs for either saturation of your allocated port speed or errors.

That’s all folks. Feel free to reply back with questions, comments, corrections.

UNIXy Crash Course CPU, graph, hosting, kSar, memory, mpstat, SAR, swap, sysstat, top, tutorial, vmstat, VPS