(Revised 17/10/1999)

Large Size of Data Sets (millions/billions of records, thousands of variables);

Even with large memory capacities of modern computers data on this scale has to be analyzed adaptively or sequentially. Typically, the information to be analyzed is not stored as a single flat file but in several files that may be linked in some relational database. "As a consequence of the structured way in which data are necessarily stored, it might be the case that straightforward statistical methods cannot be applied ...". The usual procedures for drawing statistical inferences may not be appropriate because with very large data sets even very tiny effects will be of statistical significance. "In place of statistical significance, we need to consider more carefully substantive significance: is the effect important or valuable or not?"

Contaminated Data:

"... when data sets are large, it is practically certain that some of the data will be invalid in some way." Yet the amount of data and its origin as part of a complex database structure means that checking source data may not be feasible. As it is often not be possible to identify and correct invalid data, prior to DM, some form of imputation may be required and/or the DM methods themselves must be robust to missing and incomplete data.

Selection bias and dependent observations:

"In general, very large data sets are likely to have been subjected to selection bias of various kinds - they are likely to be convenience or opportunity samples rather than statisticians' idealized random samples." For example, "... of people offered a bank loan, comprehensive data is available only for those who take up the offer. If these are used to construct the models, to make inferences about the behaviour of future applicants, then errors are likely to introduced."

Finding 'useful' patterns:

To recognise a pattern one must have some idea what one is looking for and what constitutes a 'useful' pattern. "The essence of data mining is that one does not know precisely what sort of structure one is seeking, so a fairly general definition will be appropriate". "Familiarity with the problem domain and a willingness to try ad hoc approaches seems essential …".

Statistical Fishing:

The pejorative terms 'Statistical fishing' or 'data dredging' are used to describe the practice of repeatedly fitting models to data in the hope of finding some 'interesting' results. If this is done often enough some interesting results will turn up by chance alone. This is a recognised problem in conventional statistics but in DM "… with huge data sets it is possible to model small idiosyncrasies which are of little practical import." How do we know our useful patterns are not spurious especially given the possibly biased nature of our data?

Although the references given below discuss the popular DM techniques such as; decision trees, neural networks, memory-based reasoning, cluster analysis, logistic regression, genetic algorithm, etc. They also contain many case studies and examples of DM in areas such as database marketing, fraud detection, credit scoring, market basket analysis, segmentation, astrophysics! and more. It is via these case studies and examples that DM is really understood. Understanding the techniques (with reference to " statistics, database technology, pattern recognition, machine learning", etc.) is secondary to understanding why you are using them in the first place. And that at least requires an understanding of the cycle of data acquisition, analysis, decision making, … , whether it is the field of 'data and business process', TQM, scientific research, etc.

To find out more about DM: