Heartbleed SSL vulnerability and how UNIXy handled the crisis on behalf of our clients

Starting two days ago, we went through every one of our clients’ servers and updated OpenSSL to the latest patched version. Unbeknownst to us, the updates did not actually stop and start Apache and/or Nginx, which is a must-do step (so the vulnerable OpenSSL libraries are fully unloaded). So another round of service restarts was planned out and executed shortly afterwards.

heartbleed

If you have a concern or question regarding this vulnerability or its impact on your services don’t hesitate to get in touch via email or the support system. We’re more than happy to answer questions, comments, or help in any way we can.

WHMCS and the AES_ENCRYPT vulnerability / exploit. Hooks to stop AES_ENCRYPT account registration

In case you’ve been out for a while, an WHMCS exploit surfaced about a week or so ago. It’s a SQL injection attack that leads to a full compromise of a WHMCS installation. First things first, head out to WHMCS to get the patch and apply it right away. This hook will NOT protect you against the exploit. This hook will merely prevent some sign ups that attempt to exploit the vulnerability. It’s essentially there so you don’t waste too much time deleting fake AES_ENCRYPT sign up accounts.

WHMCS hooks, to be nice, aren’t easy to work with. Be sure to change the email suffix $values[“email”]  to your own (just in case).

 

function hook_validate_aesencrypt($vars) {

if (!isset($vars)) {
return;
}

$vars_str = implode($vars);

if (strpos($vars_str, “AES_ENCRYPT”) !== False) {
$command = “updateclient”;
$adminuser = “whmcs_admin”;
$values[“email”] = $vars[’email’] . “no_soup_for_you”;
$values[“clientid”] = $vars[‘userid’];
$results = localAPI($command, $values, $adminuser);
return;
}

}

function hook_validate_aesencrypt_pre($vars) {

if (!isset($_POST)) {
return;
}

$post_str = implode($_POST);

if (strpos($post_str, “AES_ENCRYPT”) !== False) {
$error_msg = “Registration denied”;
return $error_msg;
}

}

add_hook(“ClientDetailsValidation”, 1,”hook_validate_aesencrypt_pre”);
add_hook(“ClientAreaRegister”, 1,”hook_validate_aesencrypt”);
add_hook(“ClientEdit”, 1,”hook_validate_aesencrypt”);
add_hook(“ClientAdd”, 1,”hook_validate_aesencrypt”);

add_hook(“ClientChangePassword”, 1,”hook_validate_aesencrypt”);

cPanel Varnish Plugin Security Release – Important

It has been brought to our attention that a security vulnerability exists in the cPanel Varnish script. It affects Varnish releases 1.8.0-4. 1.2.2b, and older. We urge our subscribers to upgrade to the latest available release. Releases 1.8.0-5 and 1.2.4b have been listed as CRITICAL.

The vulnerability can exploited when you have explicitly granted your reseller(s) root PHP access in WHM or if an insecure plugin has disabled this security feature. More information has been made available in the announcement section at https://www.unixy.net/secure/announcements/12/cPanel-Varnish-Plugin-Security-Release—Important.html

cPanel licensing issue when using multiple subnets / networks on the same server. Fix Inside.

With providers running out of IPv4 allocations around the world and people resorting to using multiple IP blocks/subnets on the same dedicated server, there’s a chance someone will run into this problem. If you have multiple subnets on the same box, chances are cPanel will get confused and use one of the other subnets as the outgoing IP for license verification purposes. This causes issues because now the cPanel licensing server thinks you’re not licensed (outgoing IP doesn’t match on-file IP). NB: we aren’t circumventing any licensing mechanism (unethical). This issue can happen to a wide array of software that depends on the outgoing email address (ex: email).

Here’s how your eth0:X layout might look like:

eth0     -> 1.1.1.1 / 255.255.255.0

eth0:0 -> 2.2.2.2 / 255.255.255.0

eth0:1  -> 3.3.3.3 /255.255.255.0

So let’s move on to the fix. Normally a cPanel server is licensed against its main IP/interface. In Linux that’s eth0 (0 for zero). So we’re going to tell the server to give eth0’s route a preference over its other interfaces. From the above example, we will increase the metric of eth0:0 and eth0:1 by 1. In metric speak, 0 has higher preference than 1. And 0 happens to be the default metric of interfaces on Linux boxes. All said and done, eth0 will keep its metric of 0 and will the de facto preferred route.

So open /etc/sysconfig/network-scripts/ifcfg-eth0:0 and add this line at the end of the file:

METRIC=1

And here’s a visual:

METRIC

METRIC option in /etc/sysconfig/network-scripts/ifcfg-eth0

Do the same for /etc/sysconfig/network-scripts/ifcfg-eth0:0 or any other interfaces you have. To effect this change on your system, you’ll need to run this next command but be warned this could knock your server offline if you end up doing things precariously and especially if you’re remoted in via SSH. Here’s the command:

nohup /etc/init.d/network restart &

The above command will freeze your terminal (understandably because you’ve restarted the network subsystem). Wait a minute or two, then open a new SSH/console and get back in as you would normally. It’s preferable to complete this work at the console or via KVMoIP.

That’s all folks!

wp-app.php/service 401 403 crawl error webmaster tools – Resolved!

You’ve searched all over the net but can’t find a fix for this issue? This blog post will show you how to disable this wp-app.php/service so it doesn’t lead to crawling errors from Google and other search engines. You should NOT proceed with this fix if you’re posting from a remote application outside of your WordPress site (in other words using wp-admin). Otherwise you won’t be able to remote publish anymore.

We came up with this fix because our client needed help with his server. We provide fully managed servers and clusters in four of our locations around the world and are very happy to go above and beyond to make a client’s day. We LOVE helping our clients!

wp-app.php is part of the AtomPub module, which is used for remote publishing. Some folks find it more comfortable to publish their posts from a desktop application. That’s understandable but for the majority of people out there’s /wp-admin. If you’re one of those people that prefer posting via /wp-admin, then you can safely disable this module by navigating to Dashboard -> Settings -> Writing. Scroll down to the Remote Publishing section and uncheck Atom Publishing Protocol / Enable the Atom Publishing Protocol. Be sure to hit Save. Make sure to go back to webmaster tools -> Health -> Crawl Errors -> Check the box next to wp-app.php/service -> MARK AS FIXED -> OK.

If you’re still seeing a 401/403 HTTP status, I recommend adding this line to your blog’s .htaccess file:

redirect 410 /wp-app.php/service

It tells search engines this URL is gone for good.

That’s all folks! I hope you enjoyed this entry.

Varnish “not responding to CLI, killing it died signal=3” Explained

We haven’t seen anyone explain the meaning of following Varnish error message found in the logs of your server in detail. So here it is for the curious:

Dec  5 12:31:31 server varnishd[28446]: Child (28447) not responding to CLI, killing it.
Dec  5 12:31:59 server varnishd[28446]: Child (28447) died signal=3
Dec  5 12:32:03 server varnishd[28446]: child (22431) Started
Dec  5 12:32:04 server varnishd[28446]: Child (22431) said Child starts
Dec  5 12:32:04 server varnishd[28446]: Child (22431) said SMF.s0 mmap'ed 268435456 bytes of 268435456

The message above comes from the Varnish Manager. The Manager is the Varnish process (there are two Varnish processes) that monitors and manages the Varnish caching process. Here it’s telling us that caching process has stopped responding back with a PONG to its PING. A ping from the Manager to the Cache process with its response look like this and happens every 3 seconds (at least in this configuration):

    0 CLI          - Rd ping
    0 CLI          - Wr 200 19 PONG 1323181633 1.0
    0 CLI          - Rd ping
    0 CLI          - Wr 200 19 PONG 1323181636 1.0

So going back to the error message, the Manager is telling us that it hasn’t received a “PONG” back from the Cache process in a while. A “while” is controlled by the variable cli_timeout which has a default of 10 seconds in Varnish 3.x and 20 seconds in Varnish 2.x. So the Manager waits for about 10 seconds before it fires a UNIX kill using signal SIGQUIT (signal=3). Next the Manager starts the child back up again afresh.

You may be asking yourself: why is the Cache process not responding? There are a few of possible causes. Either the cache process is deadlocked (deep code issue) or the server is under a very heavy disk IO load / memory shortage. So how do you determine which category you fall under? I recommend checking the load average and memory utilization levels on your server using Sysstat tools.

 

Did you know?

UNIXy is a fully managed server and cluster provider. What this means is we don’t expect you to know anything about servers or server management. The good news is it doesn’t cost you extra to have us manage your UNIXy server! Get in touch with us here to get the ball rolling!

 

That’s all folks.