Google is weird how it crawls websites...
I recently have been working on a side project hosting dedicated game servers at HostedGameServers.com and have been hitting a lot of snags.
Basically, the site used to be at Hostedd.com and was doing ‘ok’ for Google rank. It was steadily climbing the ladder toward page one. I decided since the domain hostedd.com had only been around for a month or two, I would re-brand the entire company to Hosted Game Servers so relevant things were in the URL. I also went ahead and added a keyword or two in my page URLs. I set my prior domain hostedd.com to be a 301 redirect to hostedgameservers.com and figured all was well. I setup the old URLs to redirect to the new slightly longer URLs with keywords in them in case anyone hit the old URLs.
I generated a new sitemap, added the new site to google analytics and google webmaster tools. A strange thing happened. Google appeared to crawl one of the old non-existent pages that I had a redirect for… detected it missing (thats how it handles RewriteRule?!!?) and left my site promptly! It didn’t return for days. Considering I had a current sitemap which mentioned NOTHING of these files; its rather weird it even tried to visit them.
Wow I thought… what a setback. I’m sure this kind of thing is on my record with Google for awhile….
So I went and requested removal of he old URLs from google at both the new and old domain names. Google denied them 24 hours later because they were not restricted by my robots.txt. In the mean time google crawled my website again, ignored what was listed in my html and xml sitemaps, hit a redirect and reported it as a couple 404 missing pages. WTF? My web host is wasn’t even having outages! I did all this sitemap and immigration work and google tagged my domain with a big red F for having what it thinks are 404 errors. Total crap!
So anyway, I requested removal again after adding the old url’s to the robots.txt file and they were accepted. Google crawled my site again, indexed two pages, and left. My site is ranking horribly, even if i search for all the words in the new domain name and all I can do is wait.
It’s common practice o use rewrite rules and rename pages. I’m really surprised this caused complications with the internet giant. It’s really frustrating and for an internet business can be deadly. Maybe google has become scored and petrified of google rank stealing tricks?!