I ran into this problem with WordPress running on a Mac OS X server. I suppose that this may happen on other servers running Apache as well. Attempting to access the WordPress site threw this error at you:
The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there was an error in a CGI script.
If you think this is a server error, please contact the webmaster.
Turned out that the .htaccess file in the root of the WordPress install had some gibberish in it. It looks like this gibberish replaced some of the actual contents of the file. I don’t know how it got there, but fixing the .htaccess file solved the problem. So, that’s something to check if you are having this problem as well!
Update, June 6, 2014: This post seems to generate more spam than normal so I am disabling comments.