Businesses are constantly on the prowl for better information that will reveal what their customers truly want and feel about their products and services. Current tools, however, limit a company's abilities to accurately discern sentiment and, in turn, close the gap between its operations and its customers.

Surveys can help to collect sampled customer insights, but the biases inherent in sampling and directed questioning and the question/answer structure can limit natural responses and freely submitted opinions.

And although Web pageviews, abandoned pages, and sales transactions can be quantified and analyzed, doing so fails to provide a company with prospective information about customer intent, sentiment toward products and services, and underlying motivations or needs driving customer behavior.

In fact, customers already communicate their opinions, issues, and perceptions very clearly, but the information is sitting mostly idle both inside and outside the enterprise.

Customers dial call centers, submit emails and send letters to make their opinions known. Even more customers and prospects voice their opinions on newsgroups, blogs, customer and product forums, and on Web 2.0 sites.

The typical approach used by most companies—if they do any analysis of this content—is to pay a roomful of people to manually scour and react to the incoming information on an ad-hoc basis.

Ideally, corporations should be able to efficiently collect all of the online product reviews, call-center notes, survey verbatims, customer-relationship management (CRM) text fields, press releases, and other textual information available across and beyond an enterprise and quickly convert them into useful business intelligence.

There is a better way to shorten the distance between a company and its customers.

"Text mining" enables a company to automatically harvest and analyze the array of unstructured textual information available to it. Early-adopter organizations in industries such as consumer goods, healthcare/pharmaceutical, retail, hospitality, and government are already successfully leveraging the practice to get closer to their customers—and add distance between themselves and their competition.

Internet-based consumer-generated content offers an especially rich source of insight. Consumers are increasingly turning to blogs, social networking sites, wikis, and folksonomies to vent grievances, opinions, desires, and expectations. The benefits of mining these sources of information are considerable.

Gathering information from the Web represents the shortest distance between a company and its customers. It enables companies to spot trends and adverse effects and prepare or respond proactively

The best source of customer data is the customers themselves.

Imagine the potential impact on a product launch. A product manager could benchmark buzz for an offering not only before and after but also during the launch. That information could be used to assess the effectiveness of marketing and public-relations campaigns as they are going on.

Whereas in the past a product manager could only measure transactions and compare sales figures before and after a campaign, now a product manager could use text mining to quantify flux in consumer sentiment throughout a launch and tune messages or tactics dynamically to ensure that key audiences are reached and directly impacted.

Companies until now have found the Internet to be a mostly frustrating, impossible-to-harness source of intelligence—the information being at once too sprawling to grasp and too narrow to dependably gauge sentiment across the customer base.

It wasn't until the very recent maturation of key underlying technologies that manage variations in language and structure of data that a comprehensive approach to text mining was even possible. Today, mining the Internet has become an "operationalized" capability; even general business users (as opposed to only Information Technology power users) can quickly convert consumer-generated content into useful business intelligence.

Also, the Web is becoming steadily mainstream. As it does, its consumer-generated content is growing steadily into a more representative sample of a company's entire customer base. And second-generation Internet sources such as blogs and social-networking sites stand to become only more crucial as more mainstream consumers start using them more frequently.

Companies will require easy-to-use tools for scouring and leveraging business intelligence from the Internet—untamed and largely untapped until now—if they are to remain competitive in the Web 2.0 and emerging 3.0 eras.

Of course, the sources of valuable textual information are not limited to the Internet, and the impact of mining customer data is not limited to marketing departments. Unlocking the huge and growing array of unstructured information available both inside and outside the enterprise through text mining enables companies to enhance operations in multiple ways:

• CRM data, call-center notes, email, and instant messages can be mined to illuminate quality problems and cut warranty claims.
  
  
• Consumer-generated content can be mined to bring a product more in line with what the market wants, enhance customer experience, and boost customer retention rates.
  
  
• Publicly available government filings can be mined to identify fraudulent activity and reduce investigation risk.

Both structured data (which quantifies existing customer interaction and sales activity) and unstructured textual information (which indicate customer perception, sentiment and intention) have important roles to play in business intelligence.

Integrating and analyzing structured data and unstructured information side by side enables a company to dramatically enhance its competitive positioning through varied, differentiating improvements in business performance.

Text mining can bring a company closer to its customers than ever previously possible.