Bayesian Technique

Here we will explore the relationship between maximizing likelihood p.d.f - $p(\vec t|\vec w)$, maximizing posterior p.d.f - $p(\vec w|\vec t)$, minimization of the sum-of-squares error function - $E_D(\vec w)$ and the regularization technique.

When we maximise the posterior probability density function w.r.t the parameter vector using the “bayesian technique” - we need both the likelihood function and the prior. (The denominator in the bayes theorem is just a normalization constant so that doesn’t really matter.)

$$p(\vec w|\vec t) \propto p(\vec t|\vec w)p(\vec w)$$

The model

We have a set of inputs, $\vec x= [x_1, . . . , x_n]^T$ with corresponding target values $\vec t =[t_1, .... t_n]^T$.

We assume that there exists some deterministic function $y$ such that we can model the relationship between these two as the sum of $y(x,w)$ with additive gaussian noise,

$$t_i = y(x_i,\vec w) + \mathcal{N}(0,\beta^{-1})$$

$β$ is the precision (inverse variance) of the additive univariate Gaussian noise.

We define $y$ as the linear combination of basis functions,
$$y(x_i,\vec w) = w_1\phi_1(x_i)+w_2\phi_2(x_i)+...+w_p\phi_p(x_i) = \vec w^T\vec \phi(x_i)$$
We define the parameter vector as $\vec w_{p \times 1}=[w_1,w_2,...,w_p]^T$ and basis vector as $\vec \phi(x_i)=[\phi_1(x_i),\phi_2(x_i),...,\phi_p(x_i)]^T$.

This parameter vector is very important as the posterior p.d.f is the updated probability of $\vec w$ given some training data. which is found from the prior data of $\vec w$. While the likelihood p.d.f of getting that training data given $\vec w$.

We usually choose $\phi_1(x)=1$ because we need a bias term in the model. (to control the extent of the shift in $y$ itself - check this answer out)

For the data set as a whole we can write the set of model outputs as a vector $\vec y_{n \times 1}$,

$$\vec y(\vec x,\vec w) = \Phi\vec w$$

Here the basis matrix $\Phi_{n \times p}$ is a function of $\vec x$ and is defined with its $i$th-row being = $[\phi_1(x_i),\phi_2(x_i),...,\phi_p(x_i)]$ for $n$ such rows.

Likelihood function

We assume that these data points $(x_i,t_i)$ are drawn independently from the distribution we would have to multiply the individual data point’s p.d.f - which are gaussian.

$$p(\vec t|\vec x,\vec w, β) =\prod^n_{i=1}\mathcal{N}(t_i|\vec w^T\vec \phi(x_i), β^{-1})$$

Note that the $i$th data points p.d.f is centered around $\vec w^T\vec \phi(x_i)$ as the mean.

Does the product of $n$ univariate gaussians forms a multivariate distribution in {$t_i$}?? I say this because we choose a gaussian prior, thus the likelihood should also be gaussian right?

Prior

We choose the corresponding conjugate prior, as we have a likelihood function which is the exponential of a quadratic function of $\vec w$.

No clue why but for now for this to make sense let’s say that the likelihood function is also gaussian - product of all those gaussians.

Thus the prior p.d.f is a normal distribution - $\mathcal{N}(m_o,S_o)$

Posterior

The posterior p.d.f is a $\mathcal{N}(m_N,S_N)$ (as we choose a conjugate prior)

After solving for $m_N$ and $S_N$ we get,

(The complete derivation is available in Bishop - (2.116)) - coming soon

$$m_N = S_N(S^{-1}_0 m_0 + βΦ^T\vec t)$$
$$S^{-1}_N = S^{-1}_0 + βΦ^TΦ$$

The sizes are,
The mean vectors, $m_N$ and $m_o$ are both $p \times 1$ and they can be thought of as the optimal parameter vector and pseudo observations respectively.
The covariance matrices, $S_N$ and $S_o$ are both $p \times p$

We shall consider a particular form of Gaussian prior in order to simplify the treatment. Specifically, we assume a zero-mean isotropic Gaussian governed by a single precision parameter $α$,
$$\mathcal{N}(\vec 0,α^{-1}I)$$
So we basically take $m_o = \vec 0$ and $S_o = α^{-1}I_{p \times p}$

Thus if we use this prior we can simplify the mean vector and the covariance matrix of posterior p.d.f to,

$$m_N = βS_NΦ^T\vec t$$
$$S^{-1}_N = αI + βΦ^TΦ$$

Now if we take log of the posterior pdf - $\mathcal{N}(m_N,S_N)$ , in order to maximize it with respect to w, we find that what we obtain is equivalent to the minimization of the sum-of-squares error function with the addition of a quadratic regularization term, corresponding to $λ = \frac{α}{β}$.

$$\ln p(\vec w|\vec t) = \frac{-\beta}{2}\sum_{i=1}^N(t_i - \vec w^T\vec \phi(x_i))^2 - \frac{α}{β}\vec w^T\vec w = -βE_D(\vec w)-\frac{α}{β}\vec w^T\vec w$$

Thus we conclude that while maximising likelihood function is equivalent to the minimization of the sum-of-squares error function, maximising posterior p.d.f is equivalent to the regularization technique.

The regularization technique is used to control the over-fitting phenomenon by adding a penalty term to the error function in order to discourage the coefficients from reaching large values.

This penalty term arises naturally when we maximize posterior p.d.f w.r.t $\vec w$

Here the minimization of the sum-of-squares error function - $E_D(\vec w)$ is also same as Maximization of the likelihood p.d.f. Taking log of $\ln p(t|\vec w, β)$ we get,

$$\ln p(t|w, β) = \frac{N}{2}\ln \beta - \frac{N}{2}\ln(2π) - \beta E_D(\vec w)$$

thus maximizing likelihood is equal to maximizing $- E_D(\vec w)$ (rest are all constants w.r.t $\vec w$)