# 信息论

## 信息的度量

### 信息熵

${\displaystyle H(X)=\mathbb {E} _{X}[I(x)]=\sum _{x\in {\mathcal {X}}}^{}p(x)\log _{2}({\frac {1}{p(x)}})}$

${\displaystyle S(X)=k_{B}H(X)}$

#### 例子

P(面一)=1/5,

P(面二)=2/5,

P(面三)=2/5

${\displaystyle H(X)={\frac {1}{5}}\log _{2}(5)+{\frac {2}{5}}\log _{2}\left({\frac {5}{2}}\right)+{\frac {2}{5}}\log _{2}\left({\frac {5}{2}}\right)}$

### 聯合熵與條件熵

${\displaystyle H(X,Y)=\sum _{x\in {\mathcal {X}}}\sum _{y\in {\mathcal {Y}}}^{}p(x,y)\log({\frac {1}{p(x,y)}})}$

${\displaystyle H(Y|X)=\sum _{x\in {\mathcal {X}}}\sum _{y\in {\mathcal {Y}}}^{}p(x,y)\log({\frac {1}{p(y|x)}})}$

${\displaystyle H(X,Y)=H(X)+H(Y|X)=H(Y)+H(X|Y)=H(Y,X)}$

#### 链式法則

{\displaystyle {\begin{aligned}H(X_{1},X_{2},...,X_{n})&=H(X_{1})+H(X_{2},...,X_{n}|X_{1})=H(X_{1})+H(X_{2}|X_{1})+H(X_{3},...,X_{n}|X_{1},X_{2})\\&=H(X_{1})+\sum _{i=2}^{n}H(X_{i}|X_{1},...,X_{i-1})\end{aligned}}}

### 互信息

${\displaystyle I(X;Y)=H(X)-H(X|Y)=H(X)+H(Y)-H(X,Y)=H(Y)-H(Y|X)=I(Y;X)}$

${\displaystyle I(X;Y)\leq \min(H(X),H(Y))}$，其中等號成立條件為Y=g(X)，g是一個雙射函數

## 参考文献

1. ^ F. Rieke, D. Warland, R Ruyter van Steveninck, W Bialek. Spikes: Exploring the Neural Code. The MIT press. 1997. ISBN 978-0262681087.
2. ^ cf. Huelsenbeck, J. P., F. Ronquist, R. Nielsen and J. P. Bollback (2001) Bayesian inference of phylogeny and its impact on evolutionary biology, Science 294:2310-2314
3. ^ Rando Allikmets, Wyeth W. Wasserman, Amy Hutchinson, Philip Smallwood, Jeremy Nathans, Peter K. Rogan, Thomas D. Schneider 互联网档案馆存檔，存档日期2008-08-21., Michael Dean (1998) Organization of the ABCR gene: analysis of promoter and splice junction sequences, Gene 215:1, 111-122
4. ^ Burnham, K. P. and Anderson D. R. (2002) Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach, Second Edition (Springer Science, New York) ISBN 978-0-387-95364-9.
5. ^ Jaynes, E. T. (1957) Information Theory and Statistical Mechanics, Phys. Rev. 106:620
6. ^ Charles H. Bennett, Ming Li, and Bin Ma (2003) Chain Letters and Evolutionary Histories, Scientific American 288:6, 76-81
7. ^ David R. Anderson. Some background on why people in the empirical sciences may want to better understand the information-theoretic methods (PDF). November 1, 2003 [2010-06-23]. （原始内容 (pdf)存档于2011年7月23日）.