As I mentioned in a previous post about the numerous reasons for organic search engine ranking discrepancies, there are a wide variety of tools that can help simplify the process of checking a site’s rankings for multiple keywords across the primary search engines. In this post I will explore four methods for tracking rankings and compare the features, costs, and results of each.

Methodology

I tested the same set of 78 keywords on Google, Yahoo!, and MSN using each of the four methods listed below. The destination domain was AmericanAqua.com, a Michigan-based small business that provides a variety of services including bottled water delivery, salt delivery, water treatment, and water analysis. The keywords selected were a mix of branded terms, product/service terms, and geographic terms. As a control, several broad, generic keywords were chosen where the company has no supporting content on their site and thus would not expect to be ranking. Only the first listing for the destination domain was counted, so if the site was listed in positions 4 and 12 for the same keyword on the same engine, only the position #4 ranking was counted in the results.

In order to control for as many variables as possible, the data was collected in a 4-hour window from the same computer, using the same browser. I was not logged in or cookied by any of the engines at the beginning of the data collection period. The automated rank checkers were set to query no faster than once every 5 seconds to avoid triggering the engines’ anti-scraping mechanisms. Only the first 30 organic results (3 pages) were scanned because let’s face it, if your site doesn’t appear in the first 3 pages you might as well be invisible.

The Contenders

• Web Position 4 is a feature-rich software package that was launched by WebTrends in 2005. Judging by the age of the content and “latest” press releases on their site, there hasn’t been much activity or improvement of the software since then. The software is available in “Standard” and “Professional” packages. Only results from the Professional version are included below.
• SEOBook’s Rank Checker Firefox Extension is a free product offered by Aaron Wall. Once downloaded, the tool itself lives within your Firefox browser as an extension and does not require any additional software.
• SEOMoz’s Rank Checker is another web-based tool offered by the SEOMoz.org team. Unregistered users can run a limited number of queries per day for free, but premium members get unlimited usage for a monthly fee.
• Manual Rank Checking is done the old fashioned way…with a keyboard, mouse, and a spreadsheet. No bells and whistles, just you and a mild case of Carpal Tunnel Syndrome.

¹Web Position 4 Standard edition is $189 and performs all of the same organic rank checking functions. The Professional edition has a few useless add-ons that are not useful for experienced SEO’s or no longer considered useful, i.e. search engine submissions.
²SEOMoz allows registered, non-Premium members to check up to 5 keywords a day for free. Unlimited usage of their Rank Checker tool requires a $49 per month Premium membership, although discounts are given for 6 and 12-month subscriptions.
³For some reason there is an arbitrary 45 minute delay in running Web Position reports on Yahoo after about 50 keywords. Total running time was 84 minutes, but 45 of those were just waiting around.
⁴This probably would have taken much longer, but I didn’t bother checking the ranks of about 150 keywords (50 KWs * 3 engines) that did not show up in any of the other tools.

The Results

Despite the various data collection methods and reporting tools, all 4 data collection methods returned the exact same rankings nearly 72% of the time. Where they were off, the average standard deviation was only 0.84, meaning the difference between collection methods was off by less than 1 position on average. That indicates that the tools are consistently pulling very similar rankings.

However, it should be noted that of the 78 potential rankings on each engine, at most we saw 29 keywords ranking at one time. When the non-ranking keywords are removed from the previous calculations, we only see the same results from each tool 35% of the time with a much higher standard deviation of 1.95.

The complete set of data and calculations are stored in this Google Spreadsheet for you to examine.

Interesting Findings

The following tidbits stuck out while collecting and analyzing the data:

• The four rank checkers agreed most often on MSN rankings. Of the keywords that ranked, the standard deviation on MSN was only 1.08, compared to 1.99 on Google and 2.77 on Yahoo.
• Yahoo’s results were the most inconsistently reported by the four rank checkers.
• SEOBook’s Firefox Rank Checker was the fastest, but missed a few rankings than the other tools found.
• SEOMoz’s tool only allows one query at a time, which slows down the data collection considerably for larger keyword sets.
• SEOBook’s Firefox Rank Checker managed to delete one word from the file export, which caused a few minutes of working backwards to determine what was dropped. I ran that one again and it was fine.
• SEOMoz’s tool reports results as “Found on page X, position #Y”, which slows down the process even more to manually convert to an exact position out of 30.
• Many of Web Position’s additional features are no longer selling points and are practically worthless, such as their search engine submission tool. The only compelling features are the ability to run multiple keywords across many engines and the data trending and export functions.

Conclusions

Given the similarities in results between all four methods, I recommend SEOBook’s Rank Checker Firefox Extension because it is free, fast, can handle multiple keywords across the top engines, and has sufficient data export capabilities. It won’t trend your results over time, but a simple Excel spreadsheet or database can make some great charts and graphs from the raw data. Each rank checking method has their own pros and cons, but Aaron’s tool has the right combination of cost (free) and features (just enough to get the job done, without too many bells and whistles). Other tools have more robust feature sets but are too expensive, or are free but too time-consuming and limited in functionality.

None of the organic search ranking tracking methods proved to be dramatically more or less accurate than the others. There are many external variables that could influence rankings by a few positions or more, and these are likely the main cause for the discrepancies between the various tools.

These data only represent rankings at one point in time. The best way to determine the effectiveness of your SEO campaigns is to measure rankings across multiple engines at regular intervals so you can examine the trends. Do not put too much emphasis on any one data point, since they are very likely to fluctuate over the short term. Instead, pay attention to the longer-term trends to look for signs of improvement in your rankings.

As with any software decision, you must draw your own conclusions based on what is important to you. What features do you think are lacking from today’s rank checkers? How should they be extended to be more useful? Which do you use, and which are your favorites?

One of the challenges of any SEO’s job is to accurately report the results of his or her work. If you can measure it, you can improve it, right? Traffic, site usage, and online conversion metrics are easy. My 15 year old cousin with Google Analytics can handle that before breakfast and still have time left over to sign in to Facebook and play his bingo on Scrabulous (I still win…usually).

What about organic search rankings? Those are a little harder to nail down. But as SEO’s, isn’t that what we you contracted to improve? How can you show off your SEO prowess without a basic rankings report? Consider the increasing number of variables that are factored into search engine results pages (SERPs) and the frequency of algorithm updates and you might begin to understand why my results might look different than yours for the same keyword on the same engine. Just off the top of my head, my results could be different than yours due to one or more of these reasons:

• Geographic differences in our IP addresses
• Whether or not one of us is logged in to a search engine account
• Different search histories
• Search preferences such as # of results per page, profanity filters, etc
• We hit different data centers (server farms) that are in different stages up nearly constant updates
• Random SERP and algorithm testing conducted by the engines

As if that wasn’t enough, reporting a ranking for one keyword on one engine at one particular point in time is misleading because it represents just one data point without much context. Who knows if that particular ranking is better or worse than an average ranking? What we really like to see are trends and rolling averages. It’s the only clear way to assess an SEO campaign’s impact on rankings over time.

Automated Rank Checker Basics

So, how does a smart SEO repeatedly track rankings for multiple keywords across many different search engines without manually searching, counting, and storing the data in a spreadsheet? I prefer to use automated tracking software. There are several different products on the market, but they all basically do the same thing: simulate a search and parse the source code of the resulting SERP to look for your site’s rank. This is usually called “scraping” or “data mining” and is typically frowned upon or blocked by the search engines because it consumes server capacity in much greater quantities than a human would. Consider that a human usually spends several seconds scanning a single SERP and analyzing the results. A scraper can search several pages per second, which significantly adds to the search engine’s server load. Enough scrapers running at the same time could slow the server to a crawl and degrade human users’ experience.

A smart SEO should be responsible and set their scraper to crawl at more reasonable rates (1 request every 3-10 seconds) and from different IP addresses if possible. Proxy servers work well for this if you have access to them. This will lessen the load on the search engine and reduce the risk of your IP address being flagged and potentially banned from the engine for a while.

Interpreting the Results

Once you have selected a rank checker, run the same keywords on the same engines at least once a month. Plot your keyword rankings (y-axis) against time (x-axis) on a graph if your program doesn’t do this for you. Do you see rankings generally improving over time? Congratulations. Your SEO efforts might be doing the trick.

If your rankings are flatlined or headed downhill, re-evaluate your strategies and tactics. Are your keywords too competitive? Are you focusing too much on on-site or external factors? Are your tactics in line with the engines’ guidelines? Most of all, give it time. Depending on how frequently your site is crawled, updates or new links can take weeks or months to impact your rankings.

Lastly, check your site analytics and referral logs to see if there is a corresponding fluctuation in traffic from your targeted keywords. It may be helpful to overlay your SEO tactics on the x-axis to look for patterns. Remember, correlation doesn’t necessarily indicate causality, but if you see repeated instances of rankings changing after you make changes to the site, it might be a good indicator that you are doing something right. There might also be some lag time as previously mentioned.

Getting the Client on Board

Educating a client to understand that rankings can (and frequently do) change overnight is just one of the things that makes SEO such a hard sell sometimes. Just like other metrics, it is wise to review rankings data as a trendline rather than fixed data points. Compare the data to last week, last month, or last year to look for improvements. Overlay your SEO campaign history on the graph to isolate the impact of your tactics.

Most of all, learn from your past successes and failures. Learning what DOESN’T work is just as important as learning what DOES work.

As reported on TechCrunch and ZDNet, Adobe has created a standardized format for its Flash development suite that will allow search engines to index dynamically generated content, which was not previously possible. What does this mean for the industry? Frankly, it means that “existing and future” Flash sites will be more accessible to search engine crawlers which will result in many previously invisible websites being indexed and ranked in search results. Google is already incorporating the new capabilities into their search platform, and Yahoo! reportedly still “has some work to do.”

This is good news for designers that prefer Flash for its flexibility and animation capabilities. However, it also opens up a whole new series of questions that will have to be explored and tested by the SEO community:

• Will this newly indexable content start out with zero site history? As we know, the age of a domain plays a role in calculating its authority and credibility. If a Flash site has been around for years but is just now becoming “visible” to a crawler, can its age be accurately determined?
• Without a hierarchical markup system, how will crawlers treat text and images of varying importance? For example, in HTML we can use the <h1> and <strong> tags to emphasize particular pieces of content. What is the Flash equivalent, and will those tactics be more or less impactful than their HTML counterparts?
• How will crawlers treat Flash sites that have a properly optimized HTML framework? As an SEO, I know how to create a crawler-friendly HTML version of Flash content to aid with crawler accessibility. Will Google and Yahoo treat sites with both formats differently or prefer one over the other?
• Will sites built entire of Flash be treated differently than HTML sites with Flash components? Good SEO and user experience (accessibility) dictate that Flash should not be used to create an entire site, but rather certain interactive elements within it. Will two sites with similar content be treated differently if one is built entirely in Flash and the other is a Flash element wrapped in standard HTML?
• How will the use of Flash adjust to incorporate deep links? Most purely Flash sites have no unique URLs for each “page” because all of the content exists within one .swf file. Therefore, most inbound links would have to point to the top-level URL, regardless of where the desired content resides within the rich media application. This will help the domain-level rankings but not individual pages.
• Will Flash designers adapt their use of analytics to properly measure the traffic and conversions from newfound organic search traffic? There is a little extra work required to properly track Flash site usage statistics using traditional site analytics packages. I’m guessing relatively few Flash sites have this capability built in, and even fewer are going to retrofit their sites with action tags.
• How will non-text digital assets (images, video, audio, etc) be optimized within Flash? Will the same rules apply, and if so, will digital assets embedded in Flash be more, less, or equally weighted compared to their HTML counterparts?

I’m sure this list of questions will grow over time and many smart, talented SEOs and Flash designers will figure out the answers. It’s certainly a game changer for the industry because it expands the playing field significantly. We are now competing against millions of “new” sites that were once thought of as unfortunate victims of emerging SEO best practices.

All marketers know that their jobs are part art, part science. Some companies lean more towards the analytical science of quantifiable marketing, and others are more comfortable with producing beautiful works of art. Is one necessarily better than the other? Not necessarily, depending on your target audience and your business objectives.

According to Google CEO Eric Schmidt, the marketing industry is heading down the path of producing and analyzing quantifiable metrics for as many aspects of a campaign as possible, similar to the financial industry’s transition in the 1970’s:

“There is every reason to believe marketing will go through a similar transition, but the principles of marketing–which are around storytelling, entertainment, targeting and selling–will be augmented by analytical tools,” Schmidt said.

This sounds good on paper. There are very few marketers who want to know less about how well their campaigns are working. But it is easy to fall victim to the hype and start thinking that all of our analytical problems will be solved by throwing more CPU cycles and clever code at the problem.

From my days as a Media Planner at a very analytical advertising agency and a Web Marketing Manager a very analytical automotive retailer, I can tell you that the current crop of advertising metrics and analytics packages are nowhere near where they need to be to replace a human being’s experience and strategic capabilities. As Schmidt said, analytical tools will only augment the principles of marketing, not replace them.

The Holy Grail of Marketing Analytics (doesn’t exist)

To really satisfy all the needs of all marketers, a full-fledged analytical marketing program would have to satisfy my newly-created (and partly tongue-in-cheek) tenets of the “Golden Rectangle“; balance the aesthetically pleasing with mathematical accountability.

Tenets of the Golden Rectangle of Marketing

1. Analytical marketers must be able to correlate sales to marketing initiatives. Every sale must be attributed to one or more marketing channels for a period of time. This includes online-to-offline conversions and purchase decisions that involve multiple decision makers and/or long periods of time.
2. Analytical marketers must take all known (and unknown) variables into account when drawing conclusions. This includes macroeconomic factors, competitive strategies, seasonality, media mix, creative execution, brand recognition, solar flares, word of mouth, personal beliefs, and customer’s internal thought processes among other things.
3. Analytical marketers must drive real-time decisions about optimizing across every marketing campaign a client is running, regardless of medium. Hindsight is always 20/20, but it doesn’t tell advertisers what to do with their next marketing dollar when the variables change.
4. Analytical marketers need to understand that some of the success of an advertising campaign will always depend on the human input, whether it is creative or strategic. Machines won’t be able to replicate those traits for a long time to come.

So while it’s great to think that one day all marketing will be entirely analytical and based on some algorithm somewhere in the computing cloud, we have to remember that we live and work in the real world of human wants, needs, desires, psychology, sociology and physiology. If Google can learn to manipulate those variables, they may very well create the next golden rectangle. Until then, people can still serve as the cogs in the marketing wheel.

An article in today’s Wall Street Journal made me very happy, both as a former media planner and current search engine marketing consultant. TV measurement is finally moving towards leaving the dark ages of Nielsen’s audience panel and its on the verge of entering the age of enlightenment. This is a step in the right direction where offline media is as accountable and transparent as online media. A quote by Tracey Scheppach at Starcom Worldwide says it best, “We shouldn’t continue to justify doing $70 billion of business off of 12,000 homes when the data exist. So let’s explore it.”

TNS Media Research has partnered with DirecTV in order to measure the TV viewing habits of 100,000 households, which is more than 7 times larger than Nielsen’s 14,000 household panel. This leads to more stable results and increased precision when it comes to measuring smaller channels.

All of this is made possible because of the set-top box that DirecTV users have in their house. Once subscribers have opted in to the panel, the box reports on their second-by-second viewing habits and demographics. Similar to most online tracking cookies, all personally identifiable information is removed and only the usage and demographic data are shared with advertisers and media buying agencies.

There are some potential kinks in TNS’ plans though. For example, Nielsen’s system requires viewers to regularly confirm that they are still watching TV and haven’t left it on in the background. TNS doesn’t offer that yet, and thus may overstate some metrics such as time spent watching a particular program or channel.

What does this mean for online marketers and publishers? Well, it could show that TV viewers are even less engaged with the medium than previously thought, which would be even more bad news for the ad-supported networks and more money flowing to channels that demonstrate quantifiable results, i.e. online. Or, it could bolster the ratings of some programs or networks which might lead to a slow-down in the amount of marketing dollars that are transitioning from offline to online channels. Either way, more data tends to open up markets, level playing fields and create competition and innovation. This will lead to even more insistence from advertisers that their marketing efforts are accountable and productive.

Personally, I think both data sets will co-exist for quite some time before one is chosen (or unseated) as the industry standard. By that time, every set-top box company (Tivo, Comcast, AT&T, DISH, etc) will be trying to break into the lucrative and influential data and measurement market. Increased competition is a good thing, especially because the Nielsen data that haven’t upgraded their tracking methodologies fast enough to keep up with the pace of technology.