Tight reigns on development servers you would think would be an absolute priority for mid-market retailers. As new commerce functionality is being developed, development pathways should be blocked off from any type of web crawler. Sounds pretty basic right?
Well not always. The problem is that development teams, and the individuals that comprise them, are not aligned with best practices in search engine optimization and are unaware of the ramifications of what can happen if these development sub-domains get indexed by search engines.
Recently we were working with a larger retailer who was using an outside development firm for all construction of their platform modifications. A new enhancement was being tested on the site and a development server instance was created. For example purposes, let’s just say it is cms.yoursite.com. This domain was created with no password protection as the developer thought there was no way that a user/bot would discover the URL.
Search engine crawlers did find the site and indexed it. From an algorithmic standpoint, page rank was divided in half across the site layers, and overall Google penetration took a big hit. See the chart/data below showing organic traffic decreases close to 50%.
When made aware of the situation of significant organic declines (the business was unaware the dev server was open) and after reviewing the data, we knew it was a site-wide issue. Due to the steep and sudden decline we were able to decipher that the decline was not parallel to any algorithm update and knew it had to be index oriented.
After running “site:” operators in Google, and by using Copyscape for duplicate content analysis, we were able to discover the development server instance and instruct the working development firm to robots.txt exclude and 301 re-direct all pages.
Soon after the re-directs were instituted, organic traffic spiked and returned to normal levels.
Moral of the story? Be sure to safeguard any development work as it relates to pages being locked down. We have seen $50mm plus retailers have development pages that are open due to programmer oversight. Don’t let this happen to your business, or your organic traffic could be significantly impacted.