Web site performance is a critical and underappreciated component of Web marketing and e-commerce.
It doesn't matter how well you design and market your site unless it delivers. An analysis of performance data from Keynote Systems and Symphoniq Corporation showed that performance problems could be costing almost $1 billion in lost e-commerce revenues.
Furthermore, a comparison of the statistics from these two companies reveals that the current standard of performance measurement, "synthetic" transaction monitoring, may be understating the impact of performance issues.
Synthetic transaction monitoring—the process of using scripted transactions to periodically simulate end-users performing business transactions, such as logging in, searching and purchasing—creates a false sense of security for leading online merchants.
The Magnitude of Soft Downtime
Downtime gets a lot of attention. As one of my clients commented, "My job is to keep our company's name out of the papers. When we go down, we make the front page of the Wall Street Journal."
Yet, despite the publicity that "hard" downtime receives, its impact may be dwarfed by the "soft" downtime of poor performance. Who hasn't abandoned a Web site because of slow page loads? Yet, according to standard measures of availability, those incidents don't count.
In 2001, Zona Research reported that 30% of surfers would abandon a Web site if it took more than eight seconds to load, creating the widely quoted "eight-second" standard.
Four years have cut that safety margin in half.
"The wait becomes frustrating for most Internet users after about four seconds," said Mark Ehr, an analyst with Enterprise Management Associates, according to an article in Internet Retailer. "Patience is waning as more shoppers become accustomed to quicker broadband connections."
Let's assume that 30% of surfers abandon a site that takes more than four seconds to load. This gives the following formula for the impact of site performance problems on revenues:
Impact = [(%Downtime) + 0.3*(%Page Load Times >4 seconds)] * Sales
To calculate the impact of performance problems, I turned to the abovementioned two companies that provide Web performance reports:
- Keynote Systems publishes weekly performance indices for various sectors, ranging from travel and hospitality to financial services. I looked at the following sectors, totaling 63 Web sites: travel and hospitality, broker, e-commerce, and news media.
- I also looked at site performance data collected by Symphoniq Corporation during the 2004 holiday shopping season. From Thanksgiving through December 26, Symphoniq audited the performance of 60 e-commerce Web sites, ranging from Alaska Airlines to Walmart.com.
I cross-referenced the two lists to generate a list of 25 common sites.
I began by looking at Keynote's measure of availability, which is considered the industry standard. Keynote uses a global network of reference sites that monitor Web sites using simulated user transactions. The performance indices studied measure availability from the 10 largest metropolitan areas in the United States. The average success rate for completing transactions across all those sectors was 98.59%, which means that the percent downtime was 1.41%.
If you input this into the Impact equation, you find that the impact of downtime is $223 million (1.41% of $15.8 billion). To measure the impact of sluggish Web site load times, I next turned to Symphoniq's data. Symphoniq monitored a broad set of pages per site (at least 100 URLs) and reported on the percentage of pages that took over four seconds to load.
Across the 25 sites in common, the average proportion of Web pages that took longer than four seconds to load was 13.8%. The individual Web site figures ranged from 0.56% to 52.09%.
Inputting the average into the Impact equation, you find that the impact of slow page load times is $654 million (0.3(13.8%) * $15.8 billion).
By combining these two measures, we can estimate that the total impact of Web performance problems on the 2004 holiday shopping season was $877 million.
A False Sense of Security?
Even though downtime and outages receive the bulk of the press, the model showed that the soft downtime of slow Web sites constituted the majority of the real business impact. Slow pages cost e-tailers three times as much as unplanned outages, making up nearly 74% of the billion-dollar shortfall.
What this indicates is that the current standard of performance measurement—single-transaction synthetic monitoring—while useful for benchmarking and trending, fails to capture the true impact of Web performance problems. In fact, the synthetic transaction approach may be giving e-tailers a false sense of security.
To explore this issue further, I examined the performance results for each of the 25 common sites.
In 92% of cases, Keynote's single-transaction availability approach resulted in a lower estimate of problems. In 76% of cases, Keynote underestimated user-visible problems by more than 80%. Overall, slow pages occurred nearly 10 times as often as outages: 13.8% versus 1.41%.
To some extent, this is comparing apples and oranges; Keynote measures availability, while Symphoniq measures Web site speed. Nonetheless, the analysis underscores the point that looking at availability figures for a small number of URLs fails to capture the reality of the customer experience. If customers become frustrated with a Web site's performance and leave without completing a transaction, the effect is the same as if the site went dark—the e-tailer loses revenue and customers.
Conclusion: The Billion-Dollar Cost of Poor Site Performance
Web site performance is a billion-dollar problem.
Downtime and slow-loading pages cost e-tailers $877 million between Thanksgiving and December 26, 2004, and probably cost hundreds of millions more during the rest of the year. The majority of the revenue impact comes not from Web site outages but from the prevalence of slow Web pages and the impatience of Web customers in a broadband world.
While the standard "hard" measures of availability are still important, their importance is being eclipsed by the importance of measuring page-load times and other "soft" performance issues.
I recommend that Web operations adopt tools that allow them to measure the experience of real users across all URLs to understand the real business impact of poor performance.
