Conditional Linear Layers

This technical lecture clarifies the concept of Conditional Linear Layers. In geometric deep learning, obtaining the weights of a layer often depends on some geometric attribute, like the relative position of neighbors.

1. Standard vs. Conditional Layers

• Standard Linear Layer: $y = Wx$. The weights $W$ are fixed parameters learned during training.
• Conditional Linear Layer: $y = W(e)x$. The weights are dynamically generated by a function of some attribute $e$ (e.g., relative position).

2. Convolution as Conditional Linear Layer

Use the relative position $\Delta x = x_j - x_i$ as the attribute $e$. Then the operation becomes:

$$ y_i = \sum_j W(\Delta x_{ij}) x_j $$

This is exactly the definition of a convolution! The kernel $W(\cdot)$ is simply a weight-generating function conditioned on the spatial offset.

3. The Tensor Product Perspective

If we parameterize the weight function $W(e)$ by expanding it in a basis $\{Y_J(e)\}$ (e.g., Spherical Harmonics), the operation becomes:

$$ y = \left( \sum_J w_J Y_J(e) \right) x = \sum_J w_J \underbrace{(Y_J(e) \otimes x)}_{\text{Tensor Product}} $$

This reveals a fundamental insight: Conditional Linear Layers are equivalent to weighted Tensor Products.

In 3D Steerable GNNs, we use the Clebsch-Gordan Tensor Product to perform this operation equivariantly, mixing the geometric information from $e$ with the feature information from $x$.