Where Jensen's inequality is tight

The logarithm is strictly concave, so $\log\E[Y]=\E[\log Y]$ only when $Y$ is constant.

Write $Y=p(\vec x,\vec z)/q(\vec z\mid\vec x)$ with $\vec z\sim q$. Its mean is $\int p(\vec x,\vec z)\,d\vec z=p(\vec x)$, so by Jensen’s inequality (Jensen's inequality · 0.IT.05)

Because $\log$ is strictly concave, equality holds exactly when $Y$ is the same number for every $\vec z$ that $q$ can draw. Then $q(\vec z\mid\vec x)\propto p(\vec x,\vec z)$, and normalising gives $q=p(\vec x,\vec z)/p(\vec x)$, the posterior. The constant is $p(\vec x)$.

For the linear model, with $q$ the posterior, $Y$ is $p(\vec x,z)/p(z\mid\vec x)$, which by Bayes’ rule is $p(\vec x)$ at every $z$. The script evaluates $\log p(z)+\log p(\vec x\mid z)-\log\mathcal N(z;0.866025,0.25)$ at $z=-1$, $0$, $0.5$ and $2$ and gets $-3.531024$ each time, the log-evidence.

For any other $q$ the ratio varies with $z$, and the spread of $\log Y$ is what Jensen’s inequality charges. The first derivation in proposition 2 shows that the charge is exactly $\KL(q\,\|\,p(\vec z\mid\vec x))$.