Supervised, Fully-connected, strict-hierarchy, explicit Predictive Coding with precision weighting of prediction errors

Supervised predictive coding with analytic expressions of the gradients of the energy with respect to representations and learnable parameters, including precision weighting of prediction errors.

precision_modulated_supervised_explicit_pc.infer(weights, precisions, data, ir=0.05, T=40, f=<function relu>, df=<function relu_derivate>, predictions_flow_upward=False, target_shape=None, noise=0.0, forward_pass_initialize=True, initialization_bias=<tf.Tensor: shape=(), dtype=float32, numpy=0.0>)

Implements the following logic:

Initialize representations
do T times
    e = r - W * r
    r += ir * (-P * e + tranpose(W) * P * e)
return r
Parameters
  • weights (list of 2d tf.Tensor of float32) – list of weight matrices, can be generated e.g. using tf_utils.mlp()

  • precisions – list of precision matrices, can be generated e.g. using tf_utils.mlp()

  • data (3d tf.Tensor of float32) – inuput data batch

  • ir (float, optional) – inference rate, defaults to 0.05

  • T (int, optional) – number of inference steps, defaults to 40

  • f (function, optional) – activation function, defaults to tf.nn.relu

  • df (function, optional) – derivate of the activation function, defaults to relu_derivate

  • predictions_flow_upward (bool, optional) – direction of prediction flow, defaults to False

  • target_shape (1d tf.Tensor of int32, optional) – shape of target minibatch, defaults to None

  • noise (float, optional) – standard deviation of an eventual noise mask in the sensory layer, defaults to 0 (no noise mask)

  • forward_pass_initialize (bool, optional) – controls weither we initialize the hidden and top representations to the predictions or zero for models with predictions flowing upwards, defaults to True (intiialize to predictions)

  • initialization_bias (float) – If forward_pass_initialize is False, controls the value at which representations are initialized, defaults to 0.

Returns

latent representations

Return type

list of 3d tf.Tensor of float32

precision_modulated_supervised_explicit_pc.learn(weights, precisions, data, target, ir=0.05, lr=0.001, pr=0.001, T=40, f=<function relu>, df=<function relu_derivate>, predictions_flow_upward=False, diagonal=False, noise=0.0, learn_precision_indices=None, gamma=<tf.Tensor: shape=(), dtype=float32, numpy=0.05>)

Implements the following logic:

Initialize representations
do T times
    e = r - W * r
    r += ir * (- P * e + tranpose(W) * P * e)
W += lr * (P * e * tranpose(r))
P += pr * (I - P * e * transpose(e))
Parameters
  • weights (list of 2d variable tf.Tensor of float32) – list of weight matrices, can be generated e.g. using tf_utils.mlp()

  • precisions – list of precision matrices, can be generated e.g. using tf_utils.mlp()

  • data (3d tf.Tensor of float32) – inuput data batch

  • target (3d tf.Tensor of float32) – output target batch

  • ir (float, optional) – inference rate, defaults to 0.05

  • lr (float, optional) – learning rate for weights, defaults to 0.001

  • pr (float, optional) – learning rate for precision, defaults to 0.001

  • T (int, optional) – number of inference steps, defaults to 40

  • f (function, optional) – activation function, defaults to tf.nn.relu

  • df (function, optional) – derivate of the activation function, defaults to tf_utils.relu_derivate

  • predictions_flow_upward (bool, optional) – direction of prediction flow, defaults to False

  • diagonal (bool, optional) – controls weither we use a diagonal approximation of the precision, defaults to False

  • noise (float, optional) – standard deviation of an eventual noise mask in the sensory layer, defaults to 0 (no noise mask)

  • learn_precision_indices (list of int, optional) – list of indices of layers in which we want to learn precision matrices, defaults to None (all layers)