Reading Notes

Personal reading notes. Original paper: arXiv:2606.11057v1.

Paper Reading Note: Flexible Kernels for Protein Property Prediction

Paper ID: 2606.11057v1
Title: Flexible Kernels for Protein Property Prediction
Authors: Martin Jankowiak, Yerdos Ordabayev, Rudraksh Tuwani, Henry N. Ward, Hunter Nisonoff, James M. McFarland, Gevorg Grigoryan
Affiliation: Generate Biomedicines, Somerville, MA, USA

1. Background and Motivation

In the field of protein design, accurately predicting protein properties (such as binding affinity and thermal stability) is a crucial tool. However, predicting these properties from sparse experimental data remains a significant challenge. The authors note that while much recent attention has focused on all-atom generative models, the utility of high-fidelity discriminators based on structure predictors (such as AlphaFold2) is equally evident.

The paper argues that the entire “generative to discriminative” continuum is an essential tool for well-equipped protein designers. Motivated by this, the authors are dedicated to developing accurate protein property landscape models. The authors specifically focus on methods that meet the following three criteria:

1. Data-efficiency;
2. High-quality uncertainty estimates;
3. Well-suited to multi-task learning across multiple landscapes.

Through theoretical argumentation and empirical demonstration, the authors propose that Gaussian Processes (GPs) with appropriately constructed kernel functions meet all the above criteria.

2. Methodological Foundations: Gaussian Processes and Sequence Kernels

2.1 Problem Formulation and Gaussian Processes

The paper defines the empirical protein property landscape as $N$ protein sequences $\{x_n\}$, each associated with a scalar property $t_n$. Sequences are assumed to have a fixed length $L$ and a fixed alphabet size $A$, represented using one-hot encoding. The goal is to learn a predictive model capable of predicting the property $t$ for any sequence $x$.

The authors review Gaussian Processes (GPs) as powerful non-parametric function priors, which typically perform well in data-scarce regimes. A GP is specified by a covariance function or kernel function $k: X \times X \to \mathbb{R}$. For scalar regression, the joint density form is known, the marginal likelihood can be computed in closed form, and is maximized via gradient methods to fit kernel hyperparameters. The posterior predictive distribution is normal, providing mean and variance estimates.

2.2 Basic Sequence Kernels

The paper introduces two of the simplest kernels defined on the one-hot sequence space:

• Linear Kernel: Corresponds to Bayesian linear regression, assuming no prior structure between coefficients. For example, observing a mutation $8A \to 8V$ does not inform the effect of other mutations such as $8A \to 8I$.
• RBF Kernel: Has a product structure and decays exponentially with increasing Hamming distance. In the context of one-hot sequences, the RBF kernel can be viewed as a product of covariance matrices.

3. Core Method: Flexible Protein Sequence Kernels

The authors propose a class of sequence kernels that utilize evolutionary substitution matrices and local linearity.

3.1 Substitution Matrices and Infinite Divisibility

The authors first discuss substitution matrices, which are classic bioinformatics tools for quantifying the probability of amino acid substitutions during evolution. For convenience, the authors use the correlation matrix form of substitution matrices.

The paper points out an interesting observation: while many substitution matrices (such as most PAM matrices) are positive semi-definite (PSD), certain substitution matrices (particularly most of the BLOSUM family) possess infinite divisibility. This means that element-wise exponentiation (Hadamard power) of these matrices preserves positive semi-definiteness.

This characteristic makes BLOSUM matrices very suitable as components of protein sequence kernels. By using a learnable exponent as a hyperparameter of the GP kernel, a powerful and parsimonious mechanism is provided to regulate the covariance structure of the kernel. Powers greater than 1 weaken off-diagonal similarities, while powers less than 1 amplify them. Therefore, by learning landscape-specific exponents, the biophysical similarities encoded by these substitution matrices can be adapted to the current landscape.

Figure 1: BLOSUM50 substitution matrix as a correlation matrix. The matrix on the right has been raised to the power of 0.03.

3.2 Local Linearity

The authors propose explicitly building local linearity into the modeling method. It has long been recognized that in many cases, protein properties are roughly additive with respect to single mutations, and pairwise epistatic effects are typically small.

The authors point out that any kernel of the form $k' k_{lin}$ (where $k'$ is an arbitrary kernel and $k_{lin}$ is a linear kernel) is naturally viewed as describing a locally linear function class. If $k'$ is chosen such that the coefficients $\beta(x)$ vary slowly in sequence space, then the function $f(x)$ is locally linear near $x_0$.

3.3 LOCK: Locally Linear Correlation Kernels

Combining the above elements, the authors propose LOCK (Locally Linear Correlation Kernels). Although the specific formulas are not fully displayed in the excerpt, the core idea is to combine a correlation kernel based on substitution matrices with a linear kernel to capture evolutionary constraints and local linear effects within protein sequences.

3.4 Structure-Conditional Kernels

The authors further demonstrate that these kernels can easily incorporate structural information from foundation models. By learning what are effectively structure-aware substitution matrices, these kernels can utilize structural embeddings provided by foundation models (such as AlphaFold, etc.).

Figure 6: Learned structure-conditional amino acid correlations.

4. Experimental Results and Multi-Task Learning

4.1 Benchmarking

The authors conducted comprehensive benchmarks, evaluating over 30 protein property predictors on 21 datasets across three different regimes. The results show that sequence-based GP models frequently outperform more complex (structure-conditional) predictors that rely on foundation models with millions of parameters.

4.2 Multi-Task Learning

The paper demonstrates that structure-conditional kernels are well-suited for multi-task learning across multiple protein property landscapes. The authors propose a simple but powerful recipe for training multi-task GPs, utilizing foundation model embeddings to form zero-shot structure-conditional kernels.

Figure 5: Depiction of multi-task model performance variation.

Experiments show that these multi-task models can significantly outperform local supervised learning methods. By sharing structural information across tasks, the models can utilize sparse data more effectively.

5. Summary

This paper proposes a flexible kernel method based on Gaussian Processes for protein property prediction. The main contributions include:

1. Data-efficiency: Sequence kernel GPs perform excellently under sparse data conditions, often outperforming methods relying on large foundation models.
2. Uncertainty Estimation: Leverages GP characteristics to provide high-quality uncertainty quantification.
3. Multi-Task Learning: Achieves effective multi-task learning across multiple property landscapes through structure-conditional kernels.

The authors believe that this method, combining evolutionary information (substitution matrices) and local linearity assumptions, provides a powerful discriminator tool for protein design, complementing existing generative models.