Evolutionary principal direction divisive partitioning

Abstract

While data clustering has a long history and a large amount of research has been devoted to the development of clustering algorithms, significant challenges still remain. One of the most important challenges in the field is dealing with high dimensional datasets. The class of clustering algorithms that utilises information from Principal Component Analysis has proven very successful in such datasets. Unlike previous approaches employing principal components, in this paper we propose a technique that uses a quality criterion to select the most important dimension (projection). This criterion permits us to formulate the problem as an optimisation task over the space of projections. However, in high dimensional spaces this problem is hard to solve and analytic solutions are not available. Thus, we tackle this problem through the use of an evolutionary algorithm. The experimental results indicate that the proposed techniques are effective in both simulated and real data scenarios. © 2010 IEEE.