Ambiguous marginal likelihood in a Non-stationary GP

[patel-zeel]

I am trying to implement a Non-stationary variant of Gibbs kernel. I agree that the following equation (For the Global GP) seems correct as per the common marginal likelihood equations in GPs (at the end of page 6)

But, why the marginal likelihood for local/latent GP (given below and on page 7 beginning) is not in the same format (Missing the first term compared to above)?

According to the above equation, this question is relevant because if I directly use f_post(X_bar).logpdf(l_bar) for latent GP, it would be an incorrect thing to do. I would greatly appreciate an explanation on this problem.

[wesselb]

Hey @patel-zeel,

After taking a quick look at the paper, I'm left slightly confused. In the last paragraph of Section 3 and below equation (5), the authors state that they set $\ell$ equal to the "mean predictions of $\mathcal{GP}_\ell$", but they don't explain how these mean predictions are produced. (Clearly, those are not the true posterior means.) From what I can infer, perhaps they mean the predictions by conditioning $\mathcal{GP}_\ell$ on the "support values" $\overline{\ell}$. However, in that case, the expression for $p(\ell | ...)$ appears incorrect, because, as you point out, it would need to also include the missing term. The one resolution I can imagine is that the authors define the prior for $\ell$ as the posterior obtained by conditioning on the support values. However, that also does not line up with the presentation, because $K_{\overline{x}, \overline{x}}$ appears to be a prior covariance (last equations of Section 4.1). Is this clear to you?

[patel-zeel]

Thank you very much for your detailed comments @wesselb.

Your resolution exactly matches with the code of one paper (paper, code snippet) that builds over this paper. Can you please elaborate on how could you come to this resolution? Is there any standard method to follow here? if Yes, where can I find resources to learn that in detail?

[wesselb]

Hmm, yes, if I read the MATLAB code correctly, that indeed seems to line up!

Is there any standard method to follow here? if Yes, where can I find resources to learn that in detail?

I'm not sure, to be honest! It was more of a wild guess. 😅 Perhaps it is a well-established way of doing things, but I haven't come across this construction before.

If you wanted to implement such a thing, I think something along the following lines would work:

from stheno import Measure, GP, EQ
import lab as B

B.set_random_seed(0)

# Support values for the length scale process:
ell_support_noise = 1e-2
ell_support_x = B.randn(20)
ell_support_y = B.randn(20)

# Define prior with a posterior.
ell = GP(EQ())
ell = ell | (ell(ell_support_x, ell_support_noise), ell_support_y)

# Sample some data.
x = B.randn(50)
y = B.randn(50)

# This needs some noise too, to stabilise the numerics.
fdd = ell(ell_support_noise, 1e-8)  

# Compute log-determinant term directly how they describe it in the paper:
# evaluate the prior (posterior conditional on support values) at the
# predictive mean conditional on the support values.
print(fdd.logpdf(fdd.mean))

# Compute log-determinant term (more efficient).
print(-0.5 * B.logdet(2 * B.pi * fdd.var))

2.3074910011357743
2.3074910014025383