When American Airlines sued Farechase, Inc. in Federal District Court in Texas earlier this year, claiming that Farechase's “screen-scraping” of AA's flight information from AA.com was illegal, it was only the most recent in a series of cases challenging unauthorized data collection from Web sites.

What practices constitute “screen-scraping”? Is “scraping” really illegal? What does this line of cases mean for your business?

What Is ‘screen-Scraping'?

Despite its pejorative title, screen-scraping software simply gathers and aggregates data from other Internet Web sites for use by the gathering party. Usually, the purpose is to reform the data and display it for the benefit of the gathering party's customers.

Examples of data aggregation range from sites that collect prices on retail sites to companies that aggregate personal financial data on mutual fund and banking Web sites, permitting registered users to access information about multiple accounts on a single Web site.

The software that performs this function—often referred to as a robot (or “bot”), “spider” or “crawler”—searches Web sites for specific information.

The Farechase case provides one example of how this technology works. Farechase's customers are travel agencies. When a customer uses Farechase to research a particular airline, hotel or auto rental fare, Farechase's software searches different airline sites and collects the “Web fares” offered. A popular site such as AA.com might be searched thousands of times a day in response to queries initiated by Farechase customers.

Farechase's real-time search technology is an advance over more traditional data mining, in which companies search sites on a regular basis and maintain a separate database that may be queried by users. In the case of sites selling books or music, for example, a real-time search may not be essential as long as the database is updated frequently.

Farechase took this concept one step further by permitting its users to search for fares offered at the very moment the search is conducted, thus guaranteeing that the results would be current.

Needless to say, one's views on this type of data mining depend largely on whether one is the scraper or scrapee!

The targets of this practice, such as American Airlines, complain that the constant traffic resulting from scraping puts an extra burden on their Internet servers, slowing down their response times for legitimate users.

In the Farechase case, American Airlines claimed that if left unstopped, Farechase would be performing over 200,000 daily searches by the end of 2003. Moreover, it argued that as a result of Farechase's permitting of customers to access Web fares by going directly to the American booking pages at AA.com, American is unable to establish the relationship with its customers that it otherwise might—that is, if customers were required to navigate through AA.com's preliminary pages—thereby costing American customer goodwill.

On the other hand, companies like Farechase argue that their service encourages comparison-shopping. Companies that resist it are afraid of the resulting competition (and lower prices), Farechase says.

Technological Defenses

Before discussing the legalities of screen-scraping, it is worth pointing out that companies that are targeted by this practice and object to it often undertake a measure of “self-help” before authorizing their lawyers to file suit. Such self-help sometimes leads to a technological battle worthy of a William Gibson novel.

The defenders attempt to identify and block the Internet Protocol (IP) addresses of the attackers. The attackers respond by hiding or disguising their scrapers' identities by using fake IP addresses, thereby evading the blocking firewalls.

The attackers, not easily discouraged, seem to have a limitless supply of disguises, perpetuating this high-tech cat-and-mouse game. In several cases, the attackers have prevailed. As a result, several of these disputes have ended up in the courts.

Subscribe today...it's free!

MarketingProfs provides thousands of marketing resources, entirely free!

Simply subscribe to our newsletter and get instant access to how-to articles, guides, webinars and more for nada, nothing, zip, zilch, on the house...delivered right to your inbox! MarketingProfs is the largest marketing community in the world, and we are here to help you be a better marketer.

Already a member? Sign in now.

Sign in with your preferred account, below.

Did you like this article?
Know someone who would enjoy it too? Share with your friends, free of charge, no sign up required! Simply share this link, and they will get instant access…
  • Copy Link

  • Email

  • Twitter

  • Facebook

  • Pinterest

  • Linkedin


ABOUT THE AUTHOR

Lee Gesmer Mr. Gesmer is a partner and founder of Lucash, Gesmer & Updegrove LLC, a Boston law firm that concentrates in all aspects of the representation of high technology companies. He can be reached at gesmer@lgu.com