by Graham Williams
Knowledge leads to wisdom and better understanding. Data mining builds knowledge from information, adding value to the tremendous stores of data that abound today--stores that are ever increasing in size and availability. Emerging from the database community in the late 1980's the discipline of data mining grew quickly to encompass researchers from Machine Learning, High Performance Computing, Visualisation, and Statistics, recognising the growing opportunity to add value to data. Today, this multi-disciplinary effort continues to deliver new techniques and tools for the analysis of very large collections of data. Searching through databases measuring in gigabytes and terabytes data mining delivers discoveries that can change the way an organisation does business. It can enable companies to remain competitive in this modern data rich, knowledge hungry, wisdom scarce world. Data mining delivers knowledge to drive wisdom.
For a long time, Statisticians and more recently Machine Learning researchers, have sought to add value to data by building models from data samples. From a statistics point of view, the aim is generally to build accurate models. From a machine learning point of view, the aim is generally to gain understanding that can be turned into actionable knowledge. Irrespective the models can help better understand the general behaviour of systems and even predict outcomes for new cases.
Statistical and symbolic techniques have often been hamstrung by their computational and memory requirements, leading to long waits for models to be built over very large datasets. Alternatively sampling of the data is required in order to generate models in a reasonable time. Traditionally we might also characterise the statistical approach as apriori hypothesis testing rather than data exploration.
Data mining strives to discover new knowledge (new hypotheses) from data, effectively letting the data speak for itself. Previously unknown patterns in very large databases are searched for, presenting discoveries in a human accessible form.