Amazon's philosophy for Web site success: Data Trumps Intuition.
Even the most experienced Amazon employees came up with lots of cool ideas that failed to show significant improvement in conversion. The company now finds it is easier to build and test a prototype than build a mathematical model to predict user response. So it stopped guessing and started testing.
Amazon's home page is prime real-estate, and all category VPs wanted a link to their content top-center of the page. Every Friday, the VPs would meet, and the meetings would be long, loud and lacking in performance data, so they automated the selection decision process: format your content for the appropriate slot, drop it into the hopper and let it wait for its turn. Content that underperforms is replaced by the next item in the queue, and the politics are removed from the process. This, Amazon discovered, is the road to success.
In Part 1, we began to highlight the experience of leading experts from the 2004 Emetrics Summits held in London and Santa Barbara. In Part 2, we continue with the details of this lesson from Amazon and others.
The wrap-up of the London sessions starts with a look at RICS—that's the Royal Institution of Chartered Surveyors. RICS is the world's "leading source of land, property, construction and related environmental knowledge…. Chartered surveyors cover all aspects of property: from conserving and restoring historic buildings; residential and commercial; industrial and retail to planning home extensions, homebuyer surveys and valuations, dilapidations, boundary disputes, energy efficiency and party walls."
If you ran a site that was primarily content and you boasted 110,000 members worldwide, how would you measure the success of your site? Stephan Mitchell, RICS's New Media Executive, had the answer right from the start: you measure the success of your various constituents.
Stephan listed seven visitor types that he is constantly trying to delight:
- Members
- Press
- Public
- Business
- Government
- Education
- Job seekers
Stephan shared his Key Performance Indicators, which include unique users, repeat users, stickiness (pages viewed per visit/time spent per visit), most popular pages and user origin. In a nutshell, he wants to know where people come from and which sources generate the most repeat visits. Because its audience is found in 46 countries, RICS is also very keen to track access speeds and monitor resolution—not everybody has fancy monitors or broadband yet.
Stephan finished up with some solid advice, of interest to all, including ecommerce sites:
- Remember to cater to all of your audiences.
- Narrow it down to less than 50 key reports.
- Measure who uses the analytics tools and how much.
- Only show graphical reports to upper management.
Peter Nolan is the Group Marketing Director at William Hill. If you live in the UK, 'nough said. If not, William Hill is a gaming company that has a very successful Web site with more than a quarter of a million gamblers signed up to play.
Here's a company with a successful business model: encourage your average Tom, Dick and Harriet to plunk down hard-earned cash for a chance to win more. Simple. So how do you improve on that?
William Hill does it by clearly stating specific objectives. Peter walked us through its goal-setting, hypothesis creating, testing methodology. Very, very clear goals here: Increase William Hill's revenues and make more profits by
- Improving the registration process
- Converting more registrants to active customers
- Developing higher-value customers
Creating and testing a new registration process was the obvious start. But how do you contend with visitors who sign up but fail to wager? The hypothesis was that the frequency and speed of email communications might tip the balance in the company's favor. How much is enough? How much is too much? Segmenting the customer base into four cells revealed the creative that worked best and how often to send it—with improvements in conversion of 500%. Not bad.
The next test took a bit more sleuthing. Peter wanted to know the value of some expensive content: expensive to create as well as maintain. Very little traffic (only 6% of all visitors) looked at this particular content. Naturally, some wanted to drop it from the site, but a bit of segmentation analysis stopped them in their tracks.
Bettors who looked at this content area visited the site twice as frequently as bettors who didn't—and those who did, generated over 30% of all revenues. Rather than killing the content, the William Hill folks did what they could to get more people to look at it. They truly found a way to use their data to get more customers and get them to wager more, more often.
Amazon.com rarely shows up in public to divulge the secrets of their success, but when the people of Amazon do show up, it's worth the wait. We were lucky enough to have two Amazonians—one in the UK and one in the US. Matt Round was the Director of Data Mining and Personalization at Amazon in Seattle, but he moved back to the UK to work on the software development side of the house. Ronny Kohavi took Matt's place in the Pacific Northwest. They both gave the same presentation, and those of us lucky enough to be in both cities were not a bit sad to see it twice.
What makes Amazon the perennial poster-child of ecommerce? Several things:
- Testing
- A willingness to experiment
- Testing
- A culture of trial and error
- More testing
Here's the process with Web site changes: format your content for the appropriate slot, drop it into the hopper and let it wait for its turn. Content that underperforms is replaced by the next item in the queue, and the politics are removed from the process.
The scariest part of the Amazon story is how it goes about bidding on pay-per-click keywords from the search engines. Sponsored links are initially written by humans, but then the automation takes over to determine keywords, write the ads (yes, the computers write the ads), determine the landing pages, manage the bids, measure the conversion rates, calculate the profit per converted visitor and update the bids. For those of you selling products that Amazon sells, I apologize for making it harder for you to sleep. I warned you this was scary.
Troy Skabelund, VP of revenue operations at the Walt Disney Internet Group, was on hand in the US to present on behalf of the Interactive Advertising Bureau (IAB). The IAB is cut from the same cloth as the ABCE, with an eye on standardization and auditability. It recently published its Interactive Audience Measurement and Advertising Campaign Reporting and Audit Guidelines, which are well worth a look if you're in the ad biz.
You may recall that the London conference included Mathew Berk who had gone from practitioner to analyst to practitioner. He left a hole at Jupiter Research that was very capably filled by Eric Petersen. Eric had been helping clients with implementation at WebSideStory for several years, seeing the inside of a wide variety of companies. He took the stage to advance the cause of key performance indicators (KPIs) for Web analytics.
What makes a good KPI? Four things:
- Define your terms. What do your different metrics mean? What are the discrete relationships among various data elements?
- Set the proper level of expectation. Establish specific targets for improvement in a given time frame.
- Make it look good. Pay attention to data presentation, and highlight changes for easy identification.
- Take action. Use your reports to drive decisions about areas of the Web site that need additional work and to direct additional study.
Eric then came up with a list of example KPIs. Here are just a handful:
- A retailer might measure Order/Buyer Conversion Rates in order to measure the likelihood of successfully driving a visitor to purchase.
- A Web site designed for lead generation might use Percentage of Visits by Entry Page to measure the efficacy of marketing messages for driving visitors to the site.
- A publisher with a content site might track Heavy User Share to measure the percentage of overall visitors who consume the largest volume of content.
- A customer service site might use Percent of Visits Under 90 Seconds to understand the proportion of the visiting audience unlikely to have found information successfully.
Barry Peters, VP of customer relationship management at Carat Interactive, offered up a quick look at the evolution of e-metrics:
| Web analysis | Web intelligence | |
| Yesterday | How many hits am I getting? | What do my customers look like? |
| Today | What types of users are coming to my sites, from where are they coming, what are they doing? | How are visitors to my site transacting and converting based on their source and profile? |
| Tomorrow | What content performs best to various segments of visitors/customers? | How are various customers using my sites, and what are the important factors to drive conversion? |
Then he gave us a disturbingly familiar list of Web analytics roadblocks:
- Lack of access to data
- Stakeholder resistance
- Process issues
- Technical constraints
- Channel conflict
If you've conquered these, please let me know, as I am building the speaker's roster for 2005. Barry suggested avoiding analysis paralysis through a process of constant exploration and making sure your organization stays focused on those metrics. Your goal is to identify which variables affect those metrics.
***
Want a unique way to organize around analytics? That's what we learned from Shubhra Srivastava, Senior Analytics Consultant E-Commerce, at InterContinental Hotels, where she works in the Decision Sciences department. That's an independent group, responsible for crunching numbers: the business intelligence people. So when somebody mentioned Web logs to them, their reaction was, "Oh good, another datastream!" They live for fresh data
Shubhra went on to verify Terry Lund's "Do a little, learn a lot" mantra by describing an effort to do too much. They redesigned their entire Web site. They tested all the changes well before implementation. They put the site through a massive traditional usability review. They addressed 90% of the issues identified in a comprehensive user satisfaction survey. Unfortunately, their conversion rate remained the same, satisfaction scores were unchanged and there was no increase in revenue.
They went back to the drawing board and started testing distinct elements of the room reservation process. One tweak—listing the whole range of room rates rather than just the lowest "and up"—was credited with an increase in annual revenue of $20 million. Web analytics at work.
Jim Novo, author of The Drilling Down Project, gave us a little nugget that is useful for every Web site: the most recent visitors are the most likely to ______. You can fill in that blank with any activity your site is intended to evoke:
- Return
- Register
- Subscribe
- Download
- Purchase
- Purchase again
Jim showed a wonderful chart, mapping out recency against propensity to interact, using data gleaned from multiple clients. It's one of those insights that is somewhat obvious, somewhat pedantic and seldom used. It can, however, be put to excellent use by any Web site. Simple, tactical and profitable. Are you utilizing the recency of your visitors?
As we get to the last-but-not-least speakers, more and more of what they had to impart has been heard before. So the following are not the only points made in their presentations, but they are the unique points.
The Director of Online Relationship Marketing at software giant SAP is Crispin Sheridan. Crispin blasted the typical sales funnel that almost every speaker had PowerPointed. To Crispin, the funnel is linear, and customer relationships are spiral. He offered up a graphic, worthy of another article all by itself, that depicted awareness on the outside of the circle (television advertising and the like) and purchase at the center. There are many different spiral paths your prospects might take, through many different touchpoints. If you map out those potential paths, you are better prepared to meet your prospects information and relationship needs and turn them into customers.
Crispin also went into some detail about the plague of spiders and robots that can seriously skew your Web activity statistics. To avoid optimizing your site for non-humans, you must combine the use of known spider lists and heuristic measurements. Most web analytics vendors and a couple of associations (Like the IAB) publish blacklists of IP addresses where evil automatons dwell, and your analytics data can be purged of traffic from these addresses. Many companies simply block requests from those addresses in order to minimize the impact on the servers.
Heuristic measurements watch for clicking patterns. It's unreasonable to think a human clicked on 20 links in two seconds. It's unlikely that a human would click on links in order of appearance at exact intervals. Robots don't render graphics or execute javascript, and those are exclusion clues as well.
Probably the most well-known speaker at the Emetrics Summit was Jared Spool from User Interface Engineering. He got everybody's attention with a PowerPoint slide that said: "The Fastest Way to Increase Conversion Rates: Stop Marketing Immediately!"
Jared railed against using conversion rates rather than revenue as a measuring stick, as it is a ratio between browsers and buyers. If you want that ratio to improve (more buyers per browser), then simply stop inviting new people to your site. Those who know and love you will come back, and eventually you'll have only buyers on your site. Of course, your sales will begin a death spiral from which there is no escape, but you will have made Jared's point.
His suggestion, then, is to use usability measurements to find lost revenues, to locate those places where people get bewitched, bothered and bewildered and simply leave your site before leaving any of their money.
Our final speaker was Dylan Lewis of SmartDraw.com. Dylan surprised us all by giving a presentation very similar to that given by Seth Romanow from HP. Dylan talked about how SmartDraw, a provider of business graphics software, is a metrics-driven company. It believes in benchmarking, measuring success, measuring failure and having "Big Fat Hairy Goals." The only real difference between SmartDraw and HP is scale.
While HP has tens of thousands of employees, SmartDraw has 30. While HP uses wildly sophisticated datawarehouse systems with terabytes of data, Dylan manages his data with Microsoft Access.
When asked what works really well, Dylan's responses were the same as Seth's:
- Continuous Improvement: Do a little, learn a lot.
- Pilot testing: It's easier to try it than model it.
- A/B Testing: Try things head to head to see which is the most effective.
- Asking the Customer: What they do does not give you a clear picture of how they feel.
Those were tips that should be printed out and pinned next to your monitor as wisdom for the ages.
And that, my friends, is what I learned on my summer vacation.
