Mathematical Toolkit

A convex function lies above each of its supporting hyperplanes: at the point \(\bm{\mu}=\E[X]\) there is a vector \(\mathbf{c}\) (the gradient if \(f\) is differentiable, a subgradient otherwise) with \(f(\mathbf{x})\)\({}\ge f(\bm{\mu})\)\({}+\mathbf{c}^\T(\mathbf{x}-\bm{\mu})\) for all \(\mathbf{x}\). Replace \(\mathbf{x}\) by \(X\) and take expectations: the linear term has mean zero, so \(\E[f(X)]\ge f(\bm{\mu})\).

If \(f\) is strictly convex the supporting inequality is strict for \(\mathbf{x}\ne\bm{\mu}\), so \(f(X)-f(\bm{\mu})-\mathbf{c}^\T(X-\bm{\mu})\) is a non-negative random variable that is positive whenever \(X\ne\bm{\mu}\). Its mean is zero only if \(P(X\ne\bm\mu)=0\). For concave \(f\), apply the result to \(-f\). ∎

Use \(\ln t\le t-1\) for \(t>0\), with equality only at \(t=1\). On the set \(S=\{q>0\}\),

Equality needs \(p=q\) almost everywhere on \(S\) (equality in \(\ln t\le t-1\)) and \(\int_S p=1\), so \(p\) has no mass outside \(S\); together, \(p=q\) almost everywhere. Asymmetry: for \(q=\N(0,1)\) and \(p=\N(0,4)\), the formula of Result T.16 gives \(\KL(q\|p)\)\({}=\tfrac12(\tfrac14-1+\ln 4)\)\({}\approx0.318\) and \(\KL(p\|q)\)\({}=\tfrac12(4-1-\ln 4)\)\({}\approx0.807\). ∎

Write \(p(\mathbf{X},\mathbf{Z})\)\({}=p(\mathbf{Z}\given\mathbf{X})p(\mathbf{X})\) inside the logarithm: \(\LB(q)\)\({}=\int q\ln\frac{p(\mathbf{Z}\given\mathbf{X})}{q(\mathbf{Z})}\)\({}+\int q\ln p(\mathbf{X})\)\({}=-\KL(q\|p(\cdot\given\mathbf{X}))\)\({}+\ln p(\mathbf{X})\), since \(\int q=1\). Rearrange, and apply Result T.2. ∎

Moving from \(\mathbf{x}^\star\) along any direction \(\mathbf{v}\) tangent to the constraint surface (\(\nabla g_j^\T\mathbf{v}=0\) for all \(j\)) is possible to first order by the implicit function theorem, and cannot change \(f\) to first order at an extremum, so \(\nabla f^\T\mathbf{v}=0\). Thus \(\nabla f\) is orthogonal to the tangent space, which is the orthogonal complement of \(\mathrm{span}\{\nabla g_j\}\); hence \(\nabla f\) lies in that span. ∎

Lagrange route (how the book finds it). Stationarity of \(\sum_kN_k\ln\pi_k+\lambda(\sum_k\pi_k-1)\) gives \(N_k/\pi_k+\lambda=0\), so \(\pi_k=-N_k/\lambda\). Summing over \(k\) and using the constraint gives \(\lambda=-N\), hence \(\pi_k=N_k/N\). The positivity constraints are inactive because \(\ln\pi_k\to-\infty\) as \(\pi_k\to0\) whenever \(N_k>0\).

Why it is the global maximum. With \(\hat\pi_k=N_k/N\), for any \(\bm\pi\) in the simplex, \(F(\bm\pi)-F(\hat{\bm\pi})\)\({}=N\sum_k\hat\pi_k\ln(\pi_k/\hat\pi_k)\)\({}=-N\,\KL(\hat{\bm\pi}\|\bm\pi)\)\({}\le0\) by Result T.2, with equality only at \(\bm\pi=\hat{\bm\pi}\). ∎

Without calculus of variations. \(F[q]\)\({}=\int q\ln\frac{\tilde p/Z}{q}+\ln Z\)\({}=\ln Z-\KL(q\|\tilde p/Z)\)\({}\le\ln Z\), with equality if and only if \(q=\tilde p/Z\) (Result T.2).

With a functional derivative (Appendix D). Add a multiplier for normalisation: \(G[q]\)\({}=\int q\ln\tilde p-\int q\ln q\)\({}+\lambda(\int q-1)\). Perturbing \(q\to q+\epsilon\eta\) and differentiating at \(\epsilon=0\) gives \(\int\eta(\mathbf{z})\{\ln\tilde p(\mathbf{z})\)\({}-\ln q(\mathbf{z})\)\({}-1+\lambda\}\d\mathbf{z}\)\({}=0\) for every \(\eta\), so the bracket vanishes: \(q\propto\tilde p\). The first route also proves that this stationary point is the maximum. ∎

(a) By definition \(g(\eta)\ge\eta x-f(x)\) for each \(x\). (b) If \(f\) is convex and differentiable, \(f(y)\ge f(x)+f'(x)(y-x)\) for all \(y\), that is \(f'(x)y-f(y)\le f'(x)x-f(x)\), so the supremum defining \(g(f'(x))\) is attained at \(y=x\): \(g(f'(x))=f'(x)x-f(x)\), and the bound in (a) is an equality. The concave case follows by applying this to \(-f\).

Example (Figure 10.10). For \(f(x)=e^{-x}\) and \(\eta\lt0\), the supremum of \(\eta x-e^{-x}\) is where \(\eta+e^{-x}=0\), i.e. \(x=-\ln(-\eta)\), which gives \(g(\eta)=\eta-\eta\ln(-\eta)\) (10.130). The tangent at \(x=\xi\) has \(\eta=-e^{-\xi}\) and is the line \(e^{-\xi}(1+\xi-x)\) (10.125). ∎

(a) \(\Tr(\mathbf{AB})\)\({}=\sum_{i,j}A_{ij}B_{ji}\)\({}=\sum_{j,i}B_{ji}A_{ij}\)\({}=\Tr(\mathbf{BA})\). (b) A scalar equals its trace; then use (a) with \(\mathbf{x}^\T\) and \(\mathbf{A}\mathbf{x}\). (c) By (b) and linearity, \(\E[\mathbf{x}^\T\mathbf{A}\mathbf{x}]\)\({}=\Tr(\mathbf{A}\E[\mathbf{x}\mathbf{x}^\T])\) and \(\E[\mathbf{x}\mathbf{x}^\T]\)\({}=\bm\Sigma+\bm\mu\bm\mu^\T\). (d) Write \(\mathbf{x}_n-\bm\mu\)\({}=(\mathbf{x}_n-\bar{\mathbf{x}})\)\({}+(\bar{\mathbf{x}}-\bm\mu)\) and expand; the cross terms sum to zero because \(\sum_n(\mathbf{x}_n-\bar{\mathbf{x}})\)\({}=\mathbf{0}\); the first sum is \(\Tr(\bm\Lambda\sum_n(\mathbf{x}_n-\bar{\mathbf{x}})(\mathbf{x}_n-\bar{\mathbf{x}})^\T)\) by (b). ∎

1. \(\d(\mathbf{x}^\T\mathbf{A}\mathbf{x})\)\({}=\d\mathbf{x}^\T\mathbf{A}\mathbf{x}\)\({}+\mathbf{x}^\T\mathbf{A}\d\mathbf{x}\)\({}=\mathbf{x}^\T(\mathbf{A}^\T+\mathbf{A})\d\mathbf{x}\).

2. \(\d\Tr(\mathbf{AB})\)\({}=\Tr(\d\mathbf{A}\,\mathbf{B})\)\({}=\Tr(\mathbf{B}\,\d\mathbf{A})\), so \(\mathbf{G}^\T=\mathbf{B}\). For the second, \(\d\Tr(\mathbf{ABA}^\T)\)\({}=\Tr(\d\mathbf{A}\,\mathbf{BA}^\T)\)\({}+\Tr(\mathbf{AB}\,\d\mathbf{A}^\T)\)\({}=\Tr(\mathbf{BA}^\T\d\mathbf{A})\)\({}+\Tr(\mathbf{B}^\T\mathbf{A}^\T\d\mathbf{A})\), using \(\Tr(\mathbf{M}^\T)=\Tr(\mathbf{M})\) on the second term; so \(\mathbf{G}^\T\)\({}=(\mathbf{B}+\mathbf{B}^\T)\mathbf{A}^\T\).

3. Differentiate \(\mathbf{A}\mathbf{A}^{-1}\)\({}=\mathbf{I}\): \(\d\mathbf{A}\,\mathbf{A}^{-1}\)\({}+\mathbf{A}\,\d(\mathbf{A}^{-1})\)\({}=\mathbf{0}\). Then \(\d\Tr(\mathbf{A}^{-1}\mathbf{B})\)\({}=-\Tr(\mathbf{A}^{-1}\d\mathbf{A}\,\mathbf{A}^{-1}\mathbf{B})\)\({}=-\Tr(\mathbf{A}^{-1}\mathbf{B}\mathbf{A}^{-1}\d\mathbf{A})\); the quadratic form is the case \(\mathbf{B}=\mathbf{x}\mathbf{x}^\T\) (Result T.8b).

4. For a small \(\epsilon\) and any \(\mathbf{E}\), \(|\mathbf{A}+\epsilon\mathbf{E}|\)\({}=|\mathbf{A}|\,|\mathbf{I}\)\({}+\epsilon\mathbf{A}^{-1}\mathbf{E}|\), and \(|\mathbf{I}+\epsilon\mathbf{M}|\)\({}=\prod_i(1+\epsilon\mu_i)\)\({}=1+\epsilon\Tr\mathbf{M}+O(\epsilon^2)\) where \(\mu_i\) are the (possibly complex) eigenvalues of \(\mathbf{M}\). So \(\d\ln|\mathbf{A}|\)\({}=\Tr(\mathbf{A}^{-1}\d\mathbf{A})\) and \(\mathbf{G}=\mathbf{A}^{-\T}\). ∎

(a) Multiply \(\mathbf{A}+\mathbf{BDC}\) by the right-hand side and write \(\mathbf{K}\)\({}=(\mathbf{D}^{-1}+\mathbf{CA}^{-1}\mathbf{B})^{-1}\):

(b) Compute the determinant of \(\begin{pmatrix}\mathbf{A}&-\mathbf{U}\\ \mathbf{V}^\T&\mathbf{I}\end{pmatrix}\) in two ways. Eliminating the lower-left block with the first block row gives \(|\mathbf{A}|\,|\mathbf{I}\)\({}+\mathbf{V}^\T\mathbf{A}^{-1}\mathbf{U}|\); eliminating the upper-right block with the second block row gives \(|\mathbf{A}\)\({}+\mathbf{U}\mathbf{V}^\T|\,|\mathbf{I}|\).

(c) Take \(\mathbf{A}=\sigma^2\mathbf{I}\), \(\mathbf{B}=\mathbf{W}\), \(\mathbf{D}=\mathbf{I}\), \(\mathbf{C}=\mathbf{W}^\T\) in (a): \(\mathbf{C}^{-1}\)\({}=\sigma^{-2}\mathbf{I}\)\({}-\sigma^{-4}\mathbf{W}(\mathbf{I}+\sigma^{-2}\mathbf{W}^\T\mathbf{W})^{-1}\mathbf{W}^\T\)\({}=\sigma^{-2}\mathbf{I}\)\({}-\sigma^{-2}\mathbf{W}\mathbf{M}^{-1}\mathbf{W}^\T\). For the determinant use (b) with \(\mathbf{U}=\mathbf{V}=\mathbf{W}\): \(|\mathbf{C}|\)\({}=\sigma^{2D}|\mathbf{I}\)\({}+\sigma^{-2}\mathbf{W}^\T\mathbf{W}|\)\({}=\sigma^{2D}\sigma^{-2M}|\mathbf{M}|\). ∎

The spectral theorem is standard linear algebra (Appendix C). For (b), write \(p_i\)\({}=\|\mathbf{V}^\T\mathbf{u}_i\|^2\), the squared length of the projection of \(\mathbf{u}_i\) onto the column space of \(\mathbf{V}\). Then

(\(p_i\le1\) because projection does not lengthen a unit vector.) Hence, using \(\lambda_i\ge\lambda_M\) and \(p_i-1\le0\) for \(i\le M\), and \(\lambda_i\le\lambda_M\) for \(i>M\),

The value \(\sum_{i\le M}\lambda_i\) is attained when \(p_1=\dots=p_M=1\), i.e. when \(\mathbf{u}_1,\dots,\mathbf{u}_M\) lie in the column space. If \(\lambda_M>\lambda_{M+1}\), equality forces \(p_i=0\) for every \(i>M\), hence \(p_1=\dots=p_M=1\), so the column space must be \(\mathrm{span}\{\mathbf{u}_1,\dots,\mathbf{u}_M\}\). Part (a) is the case \(M=1\). ∎

\(\mathbf{X}^\T\mathbf{X}\) is symmetric and positive semi-definite; let \(\mathbf{q}_i\) be orthonormal eigenvectors with eigenvalues \(\sigma_i^2>0\) (\(i\le r\)), and put \(\mathbf{p}_i\)\({}=\mathbf{X}\mathbf{q}_i/\sigma_i\). These are orthonormal: \(\mathbf{p}_i^\T\mathbf{p}_j\)\({}=\mathbf{q}_i^\T\mathbf{X}^\T\mathbf{X}\mathbf{q}_j/(\sigma_i\sigma_j)\)\({}=\delta_{ij}\). Then \(\mathbf{X}\mathbf{q}_i\)\({}=\sigma_i\mathbf{p}_i\) and \(\mathbf{X}\) vanishes on the orthogonal complement of the \(\mathbf{q}_i\) (there \(\|\mathbf{X}\mathbf{q}\|^2\)\({}=\mathbf{q}^\T\mathbf{X}^\T\mathbf{X}\mathbf{q}\)\({}=0\)), which gives \(\mathbf{X}\)\({}=\sum_i\sigma_i\mathbf{p}_i\mathbf{q}_i^\T\). The covariance statements follow by substitution. For the last formula, \(\mathbf{X}^\T\mathbf{v}_i\) is an eigenvector of \(\mathbf{X}^\T\mathbf{X}\) with eigenvalue \(N\lambda_i\), and its squared length is \(\mathbf{v}_i^\T\mathbf{X}\mathbf{X}^\T\mathbf{v}_i\)\({}=N\lambda_i\) for a unit \(\mathbf{v}_i\). ∎

Expand the right-hand side: \(-\tfrac12\mathbf{x}^\T\mathbf{A}\mathbf{x}\)\({}+\tfrac12\mathbf{x}^\T\mathbf{b}\)\({}+\tfrac12\mathbf{b}^\T\mathbf{x}\)\({}-\tfrac12\mathbf{b}^\T\mathbf{A}^{-1}\mathbf{b}\)\({}+\tfrac12\mathbf{b}^\T\mathbf{A}^{-1}\mathbf{b}\), using \(\mathbf{A}^{-1}\) symmetric; the middle terms are both \(\mathbf{b}^\T\mathbf{x}\)/2. The constant does not depend on \(\mathbf{x}\), so it is absorbed by normalisation. ∎

Conditional. As a function of \(\mathbf{x}_a\), \(-\tfrac12(\mathbf{x}-\bm\mu)^\T\bm\Lambda(\mathbf{x}-\bm\mu)\)\({}=-\tfrac12\mathbf{x}_a^\T\bm\Lambda_{aa}\mathbf{x}_a\)\({}+\mathbf{x}_a^\T\{\bm\Lambda_{aa}\bm\mu_a\)\({}-\bm\Lambda_{ab}(\mathbf{x}_b-\bm\mu_b)\}\)\({}+\const\), using \(\bm\Lambda_{ba}=\bm\Lambda_{ab}^\T\). Result T.13 gives the stated mean and covariance.

Block inverse. Multiplying out shows that \(\bm\Lambda\bm\Sigma=\mathbf{I}\) holds blockwise iff \(\bm\Lambda_{aa}\)\({}=(\bm\Sigma_{aa}-\bm\Sigma_{ab}\bm\Sigma_{bb}^{-1}\bm\Sigma_{ba})^{-1}\) and \(\bm\Lambda_{ab}\)\({}=-\bm\Lambda_{aa}\bm\Sigma_{ab}\bm\Sigma_{bb}^{-1}\) (the Schur-complement formula (2.76), (2.77)); substituting gives the covariance form.

Marginal. \(\mathbf{x}_b\) is a linear map of a Gaussian vector, and a linear map of a Gaussian vector is Gaussian (its characteristic function \(\E e^{i\mathbf{t}^\T\mathbf{B}\mathbf{x}}\)\({}=e^{i\mathbf{t}^\T\mathbf{B}\bm\mu-\frac12\mathbf{t}^\T\mathbf{B}\bm\Sigma\mathbf{B}^\T\mathbf{t}}\) is Gaussian), with mean \(\bm\mu_b\) and covariance \(\bm\Sigma_{bb}\). ∎

Posterior. As a function of \(\mathbf{x}\), \(\ln p(\mathbf{x})\)\({}+\ln p(\mathbf{y}\given\mathbf{x})\)\({}=-\tfrac12\mathbf{x}^\T(\bm\Lambda+\mathbf{A}^\T\mathbf{L}\mathbf{A})\mathbf{x}\)\({}+\mathbf{x}^\T\{\bm\Lambda\bm\mu\)\({}+\mathbf{A}^\T\mathbf{L}(\mathbf{y}-\mathbf{b})\}\)\({}+\const\); apply Result T.13.

Marginal. Write \(\mathbf{y}\)\({}=\mathbf{A}\mathbf{x}+\mathbf{b}+\bm\epsilon\) with \(\bm\epsilon\sim\N(\mathbf{0},\mathbf{L}^{-1})\) independent of \(\mathbf{x}\). The pair \((\mathbf{x},\bm\epsilon)\) is jointly Gaussian, so \(\mathbf{y}\) is Gaussian (a linear map, as above) with mean \(\mathbf{A}\bm\mu+\mathbf{b}\) and covariance \(\mathbf{A}\bm\Lambda^{-1}\mathbf{A}^\T\)\({}+\mathbf{L}^{-1}\) (independence kills the cross-covariance). ∎

(a) Write \(\mathbf{x}\)\({}=\bm\mu+\mathbf{L}\mathbf{u}\) with \(\mathbf{L}\mathbf{L}^\T=\bm\Sigma\) and \(\mathbf{u}\) standard normal: \(\E\mathbf{u}=\mathbf{0}\) and \(\E\mathbf{u}\mathbf{u}^\T\)\({}=\mathbf{I}\). (b) \(-\E\ln\N\)\({}=\tfrac D2\ln2\pi+\tfrac12\ln|\bm\Sigma|\)\({}+\tfrac12\E[(\mathbf{x}-\bm\mu)^\T\bm\Sigma^{-1}(\mathbf{x}-\bm\mu)]\) and the last expectation is \(\Tr(\bm\Sigma^{-1}\bm\Sigma)=D\) (Result T.8c). (c) \(\KL=\E_0[\ln\N_0-\ln\N_1]\); the \(\ln\N_0\) part is minus (b), and \(\E_0[(\mathbf{x}-\bm\mu_1)^\T\bm\Sigma_1^{-1}(\mathbf{x}-\bm\mu_1)]\)\({}=\Tr(\bm\Sigma_1^{-1}\bm\Sigma_0)\)\({}+(\bm\mu_0-\bm\mu_1)^\T\bm\Sigma_1^{-1}(\bm\mu_0-\bm\mu_1)\) by T.8c; collect terms. (d) Add (or subtract) the exponents \(-\tfrac12\mathbf{x}^\T\bm\Lambda_i\mathbf{x}\)\({}+\mathbf{x}^\T\bm\Lambda_i\bm\mu_i\) and apply Result T.13. ∎

(a) Differentiate \(g(\bm\eta)\int h\,e^{\bm\eta^\T\mathbf{u}}\d\mathbf{x}\)\({}=1\) (differentiation under the integral sign is allowed in the interior of the natural parameter space): \(\nabla g\int h e^{\bm\eta^\T\mathbf{u}}\)\({}+g\int h e^{\bm\eta^\T\mathbf{u}}\mathbf{u}\)\({}=\mathbf{0}\); since \(\int he^{\bm\eta^\T\mathbf{u}}\d\mathbf{x}\)\({}=1/g\), the first term is \(\nabla g/g=\nabla\ln g\) and the second is \(\E[\mathbf{u}]\), so \(\nabla\ln g+\E[\mathbf{u}]\)\({}=\mathbf{0}\). Differentiating \(\E_{\bm\eta}[\mathbf{u}]\)\({}=\int\mathbf{u}\,q\) once more gives \(\E[\mathbf{u}\mathbf{u}^\T]\)\({}-\E[\mathbf{u}]\E[\mathbf{u}]^\T\).

(b) \(\KL(p\|q_{\bm\eta})\)\({}=\int p\ln p-\int p\ln h-\ln g(\bm\eta)\)\({}-\bm\eta^\T\E_p[\mathbf{u}]\). Only the last two terms depend on \(\bm\eta\). By (a) the gradient is \(\E_{q_{\bm\eta}}[\mathbf{u}]-\E_p[\mathbf{u}]\) and the Hessian is \(\cov_{q_{\bm\eta}}[\mathbf{u}]\succeq0\), so the function is convex and any stationary point is a global minimum. ∎

\(\E[\tau^k]\)\({}=\frac{b^a}{\Gamma(a)}\int\tau^{a+k-1}e^{-b\tau}\d\tau\)\({}=\frac{\Gamma(a+k)}{\Gamma(a)b^k}\), giving the mean and \(\E\tau^2=a(a+1)/b^2\). For the log moment, differentiate \(\int\tau^{a-1}e^{-b\tau}\d\tau\)\({}=\Gamma(a)b^{-a}\) with respect to \(a\): \(\int\ln\tau\,\tau^{a-1}e^{-b\tau}\d\tau\)\({}=\Gamma(a)b^{-a}\{\psi(a)-\ln b\}\); divide by \(\Gamma(a)b^{-a}\). Finally \(\mathrm{H}\)\({}=-\E\ln\Gam\)\({}=-a\ln b+\ln\Gamma(a)-(a-1)\E[\ln\tau]\)\({}+b\E[\tau]\); substitute. ∎

The normaliser means \(\int\prod_k\mu_k^{\alpha_k-1}\d\bm\mu\)\({}=1/C(\bm\alpha)\) for every \(\bm\alpha\). Mean: \(\E[\mu_k]\)\({}=C(\bm\alpha)/C(\bm\alpha+\mathbf{e}_k)\)\({}=\frac{\Gamma(\hat\alpha)\Gamma(\alpha_k+1)}{\Gamma(\hat\alpha+1)\Gamma(\alpha_k)}\)\({}=\alpha_k/\hat\alpha\). Log moment: differentiate \(\ln\int\prod_j\mu_j^{\alpha_j-1}\d\bm\mu\)\({}=-\ln C(\bm\alpha)\) with respect to \(\alpha_k\): the left side gives \(\E[\ln\mu_k]\) and the right side gives \(\psi(\alpha_k)-\psi(\hat\alpha)\). Entropy: \(-\E\ln\Dir\)\({}=-\ln C-\sum_k(\alpha_k-1)\E\ln\mu_k\). Conjugacy: \(\prod_k\mu_k^{N_k}\prod_k\mu_k^{\alpha_k-1}\)\({}=\prod_k\mu_k^{\alpha_k+N_k-1}\). ∎

The density integrates to one for every \((\mathbf{W},\nu)\): \(\int|\bm\Lambda|^{(\nu-D-1)/2}e^{-\frac12\Tr(\mathbf{V}\bm\Lambda)}\d\bm\Lambda\)\({}=1/B\), where \(\mathbf{V}=\mathbf{W}^{-1}\), and \(\ln B\)\({}=\tfrac\nu2\ln|\mathbf{V}|-\tfrac{\nu D}2\ln2\)\({}-\tfrac{D(D-1)}4\ln\pi\)\({}-\sum_i\ln\Gamma(\tfrac{\nu+1-i}2)\).

(a) Differentiate \(\ln\int(\cdot)=-\ln B\) with respect to \(\mathbf{V}\) (Result T.9, items 2 and 4): the left side gives \(-\tfrac12\E[\bm\Lambda]\) and the right side gives \(-\tfrac\nu2\mathbf{V}^{-1}\), so \(\E[\bm\Lambda]=\nu\mathbf{W}\).

(b) Differentiate with respect to \(\nu\): the left side gives \(\tfrac12\E[\ln|\bm\Lambda|]\), the right side gives \(-\partial\ln B/\partial\nu\)\({}=-\tfrac12\ln|\mathbf{V}|+\tfrac D2\ln2\)\({}+\tfrac12\sum_i\psi(\tfrac{\nu+1-i}2)\). Multiply by 2 and use \(-\ln|\mathbf{V}|=\ln|\mathbf{W}|\).

(c) \(\mathrm{H}\)\({}=-\E\ln\Wish\)\({}=-\ln B-\tfrac{\nu-D-1}2\E\ln|\bm\Lambda|\)\({}+\tfrac12\Tr(\mathbf{W}^{-1}\E\bm\Lambda)\), and the last term is \(\tfrac12\Tr(\nu\mathbf{I})=\nu D/2\) by (a).

(d) Condition on \(\bm\Lambda\): \(\bm\mu\) has mean \(\mathbf{m}\) and covariance \((\beta\bm\Lambda)^{-1}\), so by Result T.8c \(\E_{\bm\mu}[(\mathbf{x}-\bm\mu)^\T\bm\Lambda(\mathbf{x}-\bm\mu)]\)\({}=(\mathbf{x}-\mathbf{m})^\T\bm\Lambda(\mathbf{x}-\mathbf{m})\)\({}+\Tr(\bm\Lambda(\beta\bm\Lambda)^{-1})\)\({}=(\mathbf{x}-\mathbf{m})^\T\bm\Lambda(\mathbf{x}-\mathbf{m})\)\({}+D/\beta\). Now average over \(\bm\Lambda\) using (a). ∎

(a) \(\int_0^\infty\frac{\eta^{D/2}|\bm\Lambda|^{1/2}}{(2\pi)^{D/2}}e^{-\eta\Delta^2/2}\,\frac{(\nu/2)^{\nu/2}}{\Gamma(\nu/2)}\eta^{\nu/2-1}e^{-\nu\eta/2}\d\eta\) is a Gamma integral with shape \((\nu+D)/2\) and rate \((\nu+\Delta^2)/2\); it equals \(\frac{|\bm\Lambda|^{1/2}(\nu/2)^{\nu/2}}{(2\pi)^{D/2}\Gamma(\nu/2)}\Gamma(\tfrac{\nu+D}2)\big(\tfrac{\nu+\Delta^2}2\big)^{-(\nu+D)/2}\), which simplifies to the stated density. The mean is \(\bm\mu\) by symmetry; the covariance is \(\E[\eta^{-1}]\bm\Lambda^{-1}\) with \(\E[\eta^{-1}]\)\({}=\frac{\nu/2}{\nu/2-1}\). As \(\nu\to\infty\), \(\eta\to1\) in probability.

(b) Integrating out \(\bm\mu\) with Result T.15 gives \(\N(\mathbf{x}\given\mathbf{m},(1+\beta^{-1})\bm\Lambda^{-1})\)\({}=\N(\mathbf{x}\given\mathbf{m},(c\bm\Lambda)^{-1})\) with \(c=\beta/(1+\beta)\). With \(\mathbf{d}=\mathbf{x}-\mathbf{m}\),

where \(\mathbf{W}'^{-1}\)\({}=\mathbf{W}^{-1}+c\,\mathbf{d}\mathbf{d}^\T\). As a function of \(\mathbf{x}\), \(1/B(\mathbf{W}',\nu+1)\)\({}\propto|\mathbf{W}'|^{(\nu+1)/2}\)\({}=|\mathbf{W}^{-1}\)\({}+c\mathbf{d}\mathbf{d}^\T|^{-(\nu+1)/2}\)\({}\propto(1+c\,\mathbf{d}^\T\mathbf{W}\mathbf{d})^{-(\nu+1)/2}\) by Result T.10b. Matching with \((1+\mathbf{d}^\T\mathbf{L}\mathbf{d}/\nu')^{-(\nu'+D)/2}\) gives \(\nu'=\nu+1-D\) and \(\mathbf{L}=\nu'c\,\mathbf{W}\). ∎

Factor \(e^m\) out of the sum. On the right every exponent is at most 0 and one equals 0, so the sum lies in \([1,K]\): nothing overflows, and nothing underflows to \(\ln0\). Differentiating gives \(\partial/\partial a_k=\gamma_k\) and \(\partial^2/\partial a_k\partial a_j\)\({}=\gamma_k\delta_{kj}-\gamma_k\gamma_j\), the covariance matrix of a one-hot vector with probabilities \(\bm\gamma\), which is positive semi-definite. ∎