# Karhunen-Loeve变换

## 正交基的选取

• 编码
• 系数的稀疏性
• 系数集中了原信号的功率
• 滤波
• 变换后统计不相关
• 降低滤波器的复杂度
• 提高信噪比
• 最小均方误差

## 与主成分分析的异同

"The above expansion into uncorrelated random variables is also known as the Karhunen–Loève expansion or Karhunen–Loève decomposition. The empirical version (i.e., with the coefficients computed from a sample) is known as the Karhunen–Loève transform (KLT), principal component analysis, proper orthogonal decomposition (POD), Empirical orthogonal functions (a term used in meteorology and geophysics), or the Hotelling transform."

• PCA分析协方差矩阵

• KLT分析相关矩阵

PCA depend on the scaling of the variables and applicability of PCA is limited by certain assumptions made in its derivation. The claim that the PCA used for dimensionality reduction preserves most of the information of the data is misleading. Indeed, without any assumption on the signal model, PCA cannot help to reduce the amount of information lost during dimensionality reduction, where information was measured using Shannon entropy.

The coefficients in the KLT are random variables and the expansion basis depends on the process. In fact, the orthogonal basis functions used in this representation are determined by the covariance function of the process. KLT adapts to the process in order to produce the best possible basis for its expansion.it reduces the total mean-square error resulting of its truncation. Because of this property, it is often said that the KL transform optimally compacts the energy. The main implication and difficulty of the KL transformation is computing the eigenvectors of the linear operator associated to the covariance function, which are given by the solutions to the integral equation.

The integral equation thus reduces to a simple matrix eigenvalue problem, which explains why the PCA has such a broad domain of applications.

• KLT是一种对于连续或离散的随机过程都可进行的变换

• PCA则是KLT处理离散情况的算法

• 定义上KLT比PCA广泛

• 而实际上PCA比KLT实用

## Karhunen-Loeve定理

$X_t$为概率空间$(\Omega,F,\mathbf P)$上的零均值且平方可积的随机过程，在闭的有界区间$[a,b]$内，具有连续的协方差函数$K_X(s,t)$，令$\boldsymbol e_k$是平方可积空间$L^2([a,b])$上由线性算子$T_{K_X}$的特征函数构成的正交基，对应的特征值记为$\lambda_k$，则

• $K_X(s,t)$是一个Mercer核函数：$K_X(s,t)=\sum_{k=1}^\infty \lambda_k \boldsymbol e_k(s) \boldsymbol e_k(t)$
• $X_t$可由特征函数$\boldsymbol e_k(t)$展开表示$X_t=\sum_{k=1}^\infty Z_k \boldsymbol e_k(t)$
• 无穷级数关于$t$$L^2$收敛$S_N=\sum_{k=1}^N Z_k \boldsymbol e_k(t) \to 0,t\to0$
• 变量则可表示为$X_t$$\boldsymbol e_k(t)$上的投影$Z_k=\int_a^b X_t\boldsymbol e_k(t) dt$
• 零均值性$E(Z_k)=0,\forall k\in \mathbb N$
• 不相关性$E(Z_i Z_j)=\delta_{ij}\lambda_j,\forall i,j\in \mathbb N$

## 小结

• Karhunen-Loeve变换与小波变换、傅里叶变换不一样的地方在于自适应的正交基函数。
• 信号处理领域对应Karhunen-Loeve变换，机器学习领域对应主成分分析。
• 线性Karhunen-Loeve变换可以看作离散Karhunen-Loeve变换的连续版本，但是非线性Karhunen-Loeve变换就看不太懂了。
• wiki上的东西非常赞，推荐有数学功底的人看，后面的例子与应用都具有工科特色，后续还会继续更新。

## References

1. 张贤达. 矩阵分析与应用(第二版)[M]. 清华大学出版社, 2013. ↩︎
2. Kamisetty Ramamohan Rao and Pat Yip. 2000. The Transform and Data Compression Handbook. CRC Press, Inc., USA. ↩︎