Independent Component Analysis and Its Applications By Qing Xue, 10/15/2004 Outline ?Motivation of ICA ?Applications of ICA ?Principles of ICA estimation ?Algorithms for ICA ?Extensions of basic ICA framework Motivation of ICA ?Cocktail-party problem Motivation of ICA Motivation of ICA ?The problem: How to separate the voice from the music using recordings of several microphones in the room. ?Solution: ?aijare known: the linear equation can be solved by classical methods. ?aijare unknown: ICA can be used to estimate the aijand separate the original source signals from their mixtures. Applications of ICA ?Blind source separation (classical application of ICA) ?Cocktail party problem Fig1: An illustration of blind source separation. This figure shows four source signals or independent components. Applications of ICA Fig2: Due to some external circumstances, only linear mixtures of the source signals in Fig1 can be observed. Fig3: The estimates of the source signals using only the linear mixtures in Fig2. Estimates are accurate up to multiplying factors. Applications of ICA Separation of artifacts in MEG data ?Magnetoencephalography (MEG): ?The measurement of the magnetic activity of the brain. ?Provides very good temporal resolution and moderate spatial resolution. ?Problem when using a MEG record: ?Extracting the essential features of the neuromagnetic signals in the presence of artifacts. ?An ICA approach to separate brain activity from artifacts Applications of ICA Fig4: Samples of MEG signals, showing artifacts produced by blinking, saccades, biting and cardiac cycle. For each of the 6 positions shown, the two orthogonal directions of the sensors are plotted. Applications of ICA Fig5: Nine independent components found from the MEG data Application of ICA ?Finding hidden factors in financial data Financial data with parallel time series (e.g. currency exchange rates, daily returns of stocks) ?May have some common underlying factors. ?ICA decomposes the data into independent components to give an insight to the structure of the data set. Application of ICA ?Problem: ?Suppose a cashflow of several stores belonging to the same retail chain ?Find the fundamental factors common to all stores that affect the cashflow data. ?Input data ?The weekly cash flow in 40 stores in the same retail chain; ?The cash flow measurements cover 140 weeks. ?Pre-whitened to project the original signal vectors to the subspace spanned by their first five principal components. Applications of ICA Fig6: Five samples of the original cashflow time series. Horizontal axis: time in weeks. Applications of ICA Fig7: Five independent components or fundamental factors found from the cashflow data Application of ICA ?Interpretations of the factors ?Factor 1 and 2 follow the sudden changes caused by holidays, the most prominent is the Christmas time. ?Factor 3 could represent a still slower variation, such as a trend ?Factor 4 might follows the relative competitive position of the retail chain with respect to its competitors. ?Factor 5 reflects the slower seasonal variation, with the summer holidays clearly visible. Application of ICA ?Feature extraction The columns of A represent features, and siis the coefficient of the i-th feature in an observed data vector x. ICA is used to find features that are as independent as possible. Linear Representation of Multivariate Data ?A suitable representation of the multivariate data can facilitate the subsequent analysis of the data. ?Linear transformation of the original data: computational and conceptual simplicity. The problem can be rephrased as x: random vector whose elements are the mixtures x1, ,xm, y: random vector with elements y1, ,yn. W: the matrix to be determined by the statistical properties components yi. Wxy tx tx tx W ty ty ty mn = = )( )( )( )( )( )( 2 1 2 1 MM Dimension Reduction Methods ?Principal component analysis (PCA) ?Choose W to limit the number of components yito be quite small. ?To determine W so that each component contains as much information on the data as possible. Independence as a Guiding Principle ?Determine W by finding statistically independent components of yi ?Any one of these components gives no information on the other ones. ?The starting point of ICA Find statistically independent components in the general case where the data is nongaussian. Definition of Linear ICA ?Definition ?Given a set of observations of random variables (x1(t),x2(t),xn(t) (t is the time or sample index) ?Assume that they are generated by a linear mixture of independent components (s1(t),s2(t),sm(t) ?ICA consists of estimating both the unknown matrix A and the si(t), only by observe the xi(t). = )( )( )( )( )( )( 2 1 2 1 ts ts ts A tx tx tx mn MM Definition of Linear ICA ?Alternative definition Find a linear transformation given by a matrix W, so that the random variabl