I found the topic of this article purely by chance, or at least by being open to it.

While researching online, I stumbled upon a Web site for a marketing analysis project. It said that something called "Chance Discovery" was a central to its efforts.

Huh? Chance Discovery?

I was not quite sure what was meant by that odd phrase, so I looked it up with a Google query. From there, I soon found myself immersed in a series of obscure Japanese Web sites that discuss an emerging approach to marketing decision analysis called Chance Discovery (チャンス発見).

Despite being little known in the North America and Europe, Chance Discovery has groundbreaking implications for Western marketing analytics. It endeavors to solve a longstanding paradox of standard quantitative marketing analysis: how to find opportunities in our data that have yet to be realized.

In other words, Chance Discovery moves our analysis of marketing data from standard description or modeling into a formal approach for seeking inspiration from within the data.

What Is Chance Discovery?

Chance Discovery strives to open its practitioners to the chance opportunities and risks that swirl all around us every day in our work.

The Chance Discovery movement was started by Yukio Ohsawa of the University of Tsukuba in the late 1990s. Frustrated with the limits of standard quantitative analysis to predict low frequency and unrealized events, Ohsawa began to develop a new analytic approach to finding decision inspiration.

Ohsawa describes the process of Chance Discovery visually as a double helix with humans and computers working in tandem as the two rails of a ladder structure. Computers are used to mine data for subsequent visual analysis by the human eye, which spawns new rounds of computer driven analysis and human visualization in an ongoing process of machine/human iteration. The idea is not look at the data for statistically significant patterns but to explore it for patterns of significant opportunity.

This vision is reminiscent of the works of statistician John Tukey and information design expert Edward Tufte, who have long argued for the power of visual representations of complex data. After all, the human eye has an unmatched capacity for pattern recognition, which no computer can equal.

In many ways, Chance Discovery turns the ideas behind traditional statistics and data mining on their heads. These standard analyses strive to extract generalized understandings from data, while limiting the biases of inherent data noise. Chance Discovery seeks patterns within that normally discarded noise as opportunistic moments to gain insights for future efforts; it sees noise as a laboratory of potentiality and a font of creativity.

This process can work in much the same way as dog breeders identifying advantageous chance mutations in their breeding stock to pass selectively on to the next generation of a dog breed while removing the unwanted traits from the same breeding pool.

The future opportunity presented by these chance discoveries is more important than their present frequency in the data.

The above words probably beg the question in many reader's minds as to why we need to abandon the rigors of our existing quantitative methods. The answer lies in complexity.

Reality tends to play havoc with our marketing decisions, since all of these decisions are cast into uncertainty by the overwhelming dynamism of our world. To make matters worse, we rarely are allowed the controls of true scientific methods, which makes our decision making more akin to that of a mystic oracle than labcoat-wearing scientist. That is the heart of the thesis in Malcolm Gladwell's latest book, Blink (2005), in which he argues that snap judgment is often as effective as more involved decision-making strategies.

The prediction of future buzz trends shows well the central paradox within the existing marketing decision process. On the one hand, we want hard-number assurances that our efforts will yield results. Yet, on the other hand, we are unable to understand statistically events that have yet to occur.

Using traditional statistics, we can only really understand buzz after it materializes. We can use surveys to gauge interest, but these marketing tools are only as good as their assumptions and stand at best as proxy measures of true future interest in an idea.

Chance Discovery in Action

So how is Chance Discovery put into practice? To be truthful, Chance Discovery is still in a nascent phase, and its early practitioners are still finding out what works and what does not. That said, there are a growing number of examples of Chance Discovery in action.

The most complete survey of the state of the field in English can be found within Yukio Ohsawa and Peter McBurney's book Chance Discovery (2003) and at the Chance Discovery Consortium Web site.

In these early works of Chance Discovery, there are three main type of marketing analyses represented:

• Analysis of shopping carts at the point of sale—to understand natural clusters of products purchases, which are then examined to find the chance products that act as bridges between those clusters
  
  
• Exploration of online communities—to find chance opportunities involving ephemeral consumer insights or emerging opinion leaders
  
  
• Examination of product usage data—to uncover latent desires, such as the connection between infrequently consumed food and the weather

Chance Discovery is most often methodologically wedded to quantitative methods coming from linguistics and natural language processing (NLP). This fact is readily seen in the wide use the KeyGraph software for Chance Discovery analysis. KeyGraph is a data visualization tool originally created to explore word usage but has subsequently found more diverse applications.

The KeyGraph tool examines a series of sets (a mathematical term for a grouping of things) for the co-occurrence of set members. The end result is a graph showing natural groupings of commonly co-occurring members, and members that act as bridges between these clusters. These sets can be understood as composed of shopping cart items: purchases done by a single consumer, words used in a sentence, any other measure that can be represented as a set.

One can examine the key graphs by looking for chance opportunities within these clusters, embodied as low-frequency bridges between clusters. The hope is to recognize emerging data patterns that can then be intentionally replicated for gain.

For instance, Ohsawa observed in one analysis of point-of-sale data that two product clusters could be created for supermarket purchases—a cluster of food items and a cluster of other products, which were bridged by the presence of potato chips or toys in the shopping cart. This pattern suggested to him an experiment with making these bridging products more prominent in the store to cross-sell consumers between the normally independent shopping cart clusters—thus increasing average cart size.

Here's another example, a little closer to home. Earlier this summer, I had a Chance Discovery breakthrough during a purchase intent study I was doing for a consumer package goods client.

I was working with CHAID decision tree analysis to prospect for rules that govern purchase intent within a set of 100-plus survey questions. Normally where I would have ended the project, I did one more step.

I took the top 10 most predictive rules for purchase intent and returned to raw individual data that I used to derive these rules. Using the top 10 rules, I scored all individuals in the data set for their cosine similarity in the vector space to the score of a hypothetical "perfect" participant whose dimensional scores were totally predictive of the highest level of purchase intent.

Doing this, I found that there was one clear and fascinating outlier... a woman in Texas who was the absolute most dissimilar individual to the perfect individual, but yet she still had the highest level of purchase intent.

This outlier can be understood to be one of two things. She is either a coding error, or she represents a chance opportunity to gather new consumer insights, since she might be connecting to the product in a manner that was not predicted by the creators of the study.

I ended the analysis by talking my client into reconnecting with this woman to ask her in a more freeform manner why she desires the product. It was a wonderful chance opportunity to open our marketing minds to new ways of seeing the product—a fleeting moment that needed to be seized.