Bayesian Classifier

Bayesian decision theory

	定义	最小化分类错误率
loss	$\lambda_{ij}$	$[i=j]$
Expected loss	$R(c_i\vert x)=\sum_{j=1}^N\lambda_{ij}P(c_j\vert x)$	$1-P(c\vert x)$
Bayes optimal classifier	$h^*(x)=\arg\min_{c\in Y}R(c\vert x)$	$\arg\max_{c\in Y}P(c\vert x)$
decison loss	$R(h)=E_x(R(h(x)\vert x))$	$P(h^*(x)\vert x)$
Bayes risk	$1-R(h^*)$	$1-P(h^*(x)\vert x)$

$P(c|x)=\frac{P(x,c)}{P(x)}=\frac{P(c)P(x|c)}{\int p(c)P(x|c)dc}$

prior: $P(c)$
evidence: $P(x)$
class-conditional probability/likelihood: $P(x|c)$ $P (x ∣ c)$
- class-conditional probability: $x$
- likelihood: $\theta, P(x|c)(\theta)$ $θ, P (x ∣ c) (θ)$
  - $P(D_c|\theta_c)=\prod_{x\in D_c}P(x|\theta_c)$
  - $\text{LL}(\theta_c)=\log P(D_c|\theta_c)=\sum_{x\in D_c}\log P(x|\theta_c)$

属性条件独立性假设: $P(x|c)=\prod_{i=1}^{d}P(x_i|c)$
$h_{nb}=\arg\max_{c\in Y}P(c)\prod_{i=1}^dP(x_i|c)$ $h_{nb} = ar g max_{c \in Y} P (c) \prod_{i = 1}^{d} P (x_{i} ∣ c)$
- $P(c)=\frac{|D_c|}{|D|}$
- $P(x_i|c)=\frac{|D_{c,x_i}|}{|D_c|}$
拉普拉斯修正
- $\hat P(c)=\frac{|D_c|+1}{|D|+N}$ $\hat{P} (c) = \frac{∣ D _{c} ∣ + 1}{∣ D ∣ + N}$
  - $N$ 为 $D$ 中可能的类别
- $\hat P(x_i|c)=\frac{|D_{c,x_i}|+1}{|D_c|+N_i}$ $\hat{P} (x_{i} ∣ c) = \frac{∣ D _{c, x_{i}} ∣ + 1}{∣ D _{c} ∣ + N _{i}}$
  - $N_i$ 为第 $i$ 个属性可能取值数
连续属性
- $p(x_i|c)\sim N(\mu_{c,i},\sigma_{c,i}^2)$

仅保留了强相关属性间的依赖性
基于最大带权生成树
算法
- conditional mutual information: $I(x_i,x_j|y)=\sum_{x_i,x_j,c}P(x_i,x_j|c)\log\frac{P(x_i,x_j|c)}{P(x_i|c)P(x_j|c)}$ $I (x_{i}, x_{j} ∣ y) = \sum_{x_{i}, x_{j}, c} P (x_{i}, x_{j} ∣ c) lo g \frac{P ( x _{i} , x _{j} ∣ c )}{P ( x _{i} ∣ c ) P ( x _{j} ∣ c )}$
  - 在已知类别情况下的相关性
- 在以属性为节点，互信息为边建完全图上构造最大带权生成树，挑选根节点，边置为有向
- 加入类别节点 y，增加 y 到每个属性的边