Predictive processing: action and action control.

In our exploration of the Predictive Processing (PP) framework, it is time to complete the overall theoretical sketch by discussing how action and action control fit in the overall picture. This will allow to finally appreciate the astonishing explanatory power that precision weighting is proposed to carry.

Tasks that appear to be simple, such as walking, are not simple at all.
Image by Vanillase  [CC BY-SA 3.0].

[Note: this series of posts concentrates on Clark’s “Surfing Uncertainty” book. For a general introduction to the idea, see this post and the literature cited therein. The concept of precision weighting and how it fits in the PP framework is discussed in the previous post of this series.]

So far, we’ve seen that sensory input is processed by trying to anticipate it across a series of hierarchical layers which compare mini top-down predictions with the bottom up signal coming from sensory pathways. One concept that I find important to fully grasp is that, when a sensory organ transduces a stimulus into a nervous signal, only the first PP layer will actually receive what we can easily consider as the nervous representation of the original stimulus (probably including the expected precision of the signal itself), the next level up will receive only the prediction error, meaning that if the prediction was spot-on, no further signal will be sent to the higher levels at all. The absence of an error signal then must be considered as a signal in itself, meaning: “prediction was correct”. In terms of action and action control, this special quality of PP signalling pattern will play a crucial role, which we are about to explore.

Clark discusses the problem of action control and the solution proposed by PP in a biological-centric way, he does not ignore the engineering perspective (i.e.  action-control of manufactured robots and effectors), but doesn’t quite put it into the centre stage. Clark’s approach makes a lot of sense, of course. However, I found that in order to appreciate it in full one needs to be armed with a large amount of multidisciplinary knowledge, which I wouldn’t be able to summarise here. For this post, I will try to explore the same topic starting from an engineering point of view, which I hope will make the subject easier to follow, even for non-specialists.

As we saw for the case of measuring instruments, also action control is a problem that has been extensively studied by engineers. It turns out that allowing mechanical artefacts to autonomously act on the world is a hard problem to solve, especially if high precision is a requirement. Since the world is noisy, even in a highly controlled environment (such as an automated factory), noise, in the form of random deviations from the perfect “action” (as idealised in the engineers’ “plan”) will interfere with the movements enacted by a given robot/effector. This poses the problem of detecting such deviations and correcting them in real-time. The intuitively sensible way to allow machines to interact with the world with high precision is to allow feedback loops, where the robot finely readjusts its movements according to the aforementioned “plan”. This strategy is potentially very powerful, but it is extremely difficult to implement in practice, as it requires to design complex control systems: these define how the robot will  detect each possible deviation and how it should dynamically readjust its actions while they are already occurring. The standard way to tackle this problem would be to have a long sequence of logic “if/then” steps. In the real world, this approach becomes quickly impractical as it entails an explosion of interacting possibilities; it is really hard to produce robots that are able to run the program quickly enough to intervene on their actions in a timely fashion. Moreover, the situation becomes unmanageable once one realises that changing the action plan while it is executed inherently changes what should count as new anomalies. If the “plan” itself keeps changing, also the systems used to detect deviations need to dynamically readjust accordingly, while what would be an appropriate reaction to further departures from an already changing plan would also change at the same time! If you sensed the dangers posed by bottom-less recursion, I’d say that you have grasped the computational difficulty that is inherent in action-control.

Realising how hard this problem is has a direct consequence in our context: it is self-evident that animals in general and humans in particular are very good at the what we have just found to be computationally difficult (to put it mildly). The questions that should therefore puzzle more or less every neuroscientist interested in action control are:

How can nervous systems achieve what seems to be almost impossible?

Or:

How does it happen that extremely complex dynamic actions such as walking along a hiking trail (where the surface is uneven and each step requires different and fine adjustments) are normally fluid and feel effortless?

As you’re probably guessing, PP promises to solve this particular conundrum. Let’s see how.

It is well known that proprioception (the ensemble of sensory signals that report about position of our movable parts along with forces applied to them) follows its own sensory pathways, which, in somewhat surprising ways, are still hard to fully understand. In PP, sensory “prediction-based” architecture is expected to apply also to proprioception, with the added expectation that proprioception error signals are also used to control effectors (muscles). In this context, predictions represent, as before, the best guess the organism can produce for what a given sensory signal should be in the current context. Importantly, the last sentence implicitly contains a major twist in our story: in the proprioceptive arena, the context necessarily includes what the body is doing, or, if you prefer, it includes action. Better still, action (how the sensed body is moving) is inevitably a major ingredient of what signals are produced by proprioceptive organs. This means that context-dependent predictions have to be heavily influenced by what the organism is doing; it is a clearly strict requirement for the PP model to even apply to proprioception as a whole. Thus, according to PP, at any given level in a proprioceptive pathway, a higher PP layer would produce a prediction of what the proprioceptive signals would be if the body was moving in the expected way.

As a consequence, if PP does apply to proprioception, the relevant prediction error signals become concise descriptions of what isn’t moving according to the “original plan” (the prediction). PP theorists therefore propose that the prediction error, besides participating in the usual PP pattern, can also be recycled to control muscles. The key element here is that error signals are the “distilled” representations of the deviations from the expected action plan: they are inherently the exact kind of information that is required to readjust and can therefore be used more or less directly to control muscles **[Update 01/07/2017: following Clark’s kind feedback, please see note below for two important addenda]. Moreover, because the same error signals also participate in the multilayered PP pathway, large deviations will get a chance to travel upwards to higher level layers, and will thus be able to influence the overall plan and/or trigger a radical re-evaluation of the current high-level hypotheses. In this way, the overall PP architecture is able to directly explain how finely tuned control is even possible, as well as the role that proprioception is expected to have in our ability to understand what is going on in the real world. Depending on the strength and amount of prediction errors, error signals may trigger fine movement readjustments, and/or a change of plan, and/or force the organism to realise that the current best hypothesis about the state of the world was wrong and needs to be re-evaluated.

Naturally, real-time control must be supported, and this is inherently included, for the lower layers will be able to produce quick and small adjustments (with minimal impact on the overall plan), while big prediction errors will fail to be ironed out by the lower layers and will keep travelling upwards, where, if necessary, the original plan itself might change in more significant ways (which would, unsurprisingly, require more time). If even major action plan changes would fail to minimise proprioceptive prediction errors, the overall increase of error signals would force a re-evaluation of the context itself, as this condition inevitably occurs if/when the current state of affairs is likely to be quite different from the currently active “best explanation” computed by the overall PP system.

Going back to our engineering perspective, it is worth noting that, for control problems that include more that one linear degree of freedom (applies to virtually all action-control issues encountered by complex organisms), common artificial controllers end up being error-minimising feedback circuits (see for example Proportional-Integral-Derivative controllers / PID-controllers), which are at the very least analogous to a single PP layer.

For single PP action-controlling (proprioceptive) layers (as well as PID-controllers), if the computed error signal is entirely cancelled, it means that “everything is proceeding according to plan” and therefore the effectors receive no new control signal and can continue operating as planned. This chimes with the observation I’ve reported above: the absence of an error signal becomes a signal than means “all is well, no adjustment is needed”.

To complete the “action-control” picture I’ve tried to summarise, one element is still missing: the role of precision weighting. As per “passive” sensory pathways, proprioception organs will also have their inherent precision, thus, the initial sensory stimulus will still have an associated precision which can be weighted against top-down confidence. In terms of action planning and control (fine tuning of an action plan can be conceptualised as action planning and control at a high spatio-temporal resolution), the confidence that we have on a given action plan would be a direct function of how confident we are on our assessment of the current situation, as well as “how robust” the current plan seems to be. In PP, this confidence measure can be obtained by recycling the residual error produced by whichever PP layer is issuing the relevant “prediction for action”. Since all PP layers are expected to report a prediction error along with its precision/confidence weighting, this information is always available, making it theoretically possible for any PP layer to control action. This is important, but requires a long digression which I plan to follow separately. For now, I will concentrate on the proposed function of the precision weighting signal in action control.

At one level, it is obvious: low confidence in a given action plan (justified either by low confidence on our current evaluation of the external state of affairs, or by a low confidence on the effectiveness of the plan itself), means that deviations from the plan will have higher relative importance. Thus, error signals will have a bigger chance of travelling towards higher level PP layers and less propensity of being “explained away” by adjustments to the action itself. This mechanism follows the general PP architecture without any ad-hoc change and seems entirely appropriate: the lower the confidence on our action plan, the higher our propensity of radically changing the plan should be. Moreover, one interesting case is what happens when confidence is minimal (I don’t think it can be zero*). In this case, the possibility that such “extremely low confidence action predictions” will have of actually controlling action will be minimal, perhaps to the point of having no chance of initiating and/or influencing any movement at all. Thus, such predictions will remain output-less: they should be understood as action-plans that are not expected to be acted upon.

… !!! …

Yes, what you are thinking is what I meant! Adding precision weighting to the proposed PP action control mechanism immediately explains how brains may become able to produce and evaluate alternative action plans and, by extension, allows to start building an explanation of how imagination and day-dreaming can be implemented. Along the way, the basic mechanism underpinning actual dreams is also implied. QED, if you are reading this, I hope you are starting to understand the huge explanatory potential of PP in general and of precision weighting in particular.

Bibliography and notes

* In mainstream PP implementations, precision weighting is encoded as the gain of a given signal (irrespective of its direction). Thus, a prediction issued with zero confidence would be implemented as a signal with zero gain, which means “no signal at all”.

** [Update] Andy Clark has very kindly (thanks!) made me realise that it may be useful to make two additional points explicit.
1. In the special case when action is being initiated, the error signal will be maximal (the prediction would be entirely wrong, as the expected movement isn’t happening at all). In this situation, the error signal itself would contain precisely the information needed to get the planned action started. To be translated into actual movement, precision weighting must, in this case, markedly favour the prediction itself. In this way, PP becomes a unified framework which may be able to encompass perception, action selection (issuing the prediction in question), action control, and learning (see below).
2. Importantly, the proposed architecture is also able to learn. The whole idea is that error signals that can’t be cancelled by issuing more accurate predictions will ignite additional mechanisms dedicated to finding new and better predictions. I confess that I don’t have a clear idea of how such mechanisms are expected to operate (in terms of precise neurophysiological mechanisms, I might tackle this point in a later post), but in this context, it is important to note that the multilayered architecture allows for a concurrent “search” of more apt predictions across the whole stack, from perception to action control, passing through action planning/initiation. This allows to dynamically accommodate deviations that are due to noise, as well as bigger changes (say, in the case of a damaged limb, extreme tiredness, or a change of situation – swimming, for example).
The proposed architecture actually (/theoretically) allows to bootstrap action control itself: in fact, this view directly affects how we might interpret the uncoordinated movements of newborns. The main purpose of such relatively random (or apparently aimless) movements might in fact be to allow the whole stack of layers to search for and select appropriate predictions, based on the feedback signals that are triggered by the movements themselves.

ResearchBlogging.org

Clark, A (2016). Surfing Uncertainty: Prediction, Action, and the Embodied Mind Oxford Scholarship DOI: 10.1093/acprof:oso/9780190217013.003.0011

Tagged with: , , , , , , ,
Posted in Neuroscience, Philosophy, Psychology
7 comments on “Predictive processing: action and action control.
  1. […] from layers that deal with sensory information, all the way to layers that are concerned with action selection and control. In this latter case, the ES will (or might) also be used to control muscles/effectors. Reducing […]

  2. Patrick Kenny says:

    Sergio,

    I am responding to your comment on the conscious entities blog where you suggest that from an engineering perspective, it would be better to steer away from the Free Energy Principle and stick to the PP account of motor control that you have given here.

    My own view is that engineers interested only in motor control already have a good handle on the problem and so would not be motivated to look into PP. But if neuroscience could give an integrated account of action and perception they would sit up and pay attention. A piecemeal approach to robotics that seeks to deal with perception and action independently is unlikely to work: my guess is that it would run into exactly the sort of “almost impossible” problems that you allude to here. So an integrated approach such at the FEP purports to offer sounds very appealing.

    Friston’s idea of casting motor control as a problem of Bayesian inference is, on the face of it, weird — engineers and neuroscientists would generally agree about that. It only makes sense in the context of his larger ambitions such as to give a coherent account of “epistemic foraging” (how eye saccades serve to minimize the mind’s uncertainty about its explanations of visual percepts).

    Engineers will only buy it
    if somebody can suggest a plausible mathematical form for the probabilistic generative model of sensory data (visual + proprioceptive) that is required evaluate free energies. Free energy in Friston’s sense is actually a mathematical, not a physical, quantity and the right mathematical model of what is going on upstairs in the association cortex seems to be anybody’s guess. In the absence of any guidance from neuroscience a naive and foolhardy engineer might try some sort of recurrent neural network …

    http://www.consciousentities.com/2015/02/minimising-free-energy/#comment-656960

  3. Sergio Graziosi says:

    Hello Patrick, thanks for this!
    I must apologise for my sloppiness: I didn’t mean to suggest that you could find concrete proposals here!
    I find little to disagree in your comment. Integrating action and perception is indeed one reason why PP (note: I am staying away from FEP, for my own reasons – at the very least, because I’m not sure I fully understand it, and/or that we understand one another, see below) is promising.
    One source of confusion is, I think, that PP started off as an account of perception, and then ventured into cognition and eventually action planning and control – the terminology it uses follows it’s history, which doesn’t always help. I think this is confusing, because it’s likely that the evolutionary story proceeded in the other direction, from action control (requires sensory organs and some form of feedback), ending up to serendipitously “discover” cognition. This is one subject I’m hoping to cover in upcoming posts (I find it hard to get my criticism and propositional phase started as there is so much to say!). The paradigmatic example of the thermostat might help: observed with a PP lens, the temperature threshold (TT) acts as an embodied hyperprior (with “hyper” here I mean one that the PP circuitry itself cannot change – does this help with your original question over on CE?), the systems expects the temperature to be higher than TT, if it isn’t, an error signal remains, and this signal becomes action by turning on the heating. In PP parlance, this mechanism would be labelled “active inference” which is, as you rightly point out, weird, in this case. However, simple organisms do “act” in similar ways (turning on or off some mechanism in response of a specific stimulus), thus, they already are equipped (or better: they embody) something that can be interpreted in PP terms. My hunch is that this is the correct attack point to reverse engineer biological systems, which is my main interest.

    [In PP terms, you would say that the system “predicts” or “expects” the temperature to be higher than TT, which is confusing, to say the least. A little less confusing way of expressing the same concept (hoping it is still correct) is to say that the system “embodies the expectation” – you may notice here a rarefied promise to help tackling long standing philosophical problems such as intentionality and semantics…]

    OTOH, I’ve used the engineering lens as a tool to try to make it clear why the PP theoretical stance appears to be very powerful and promising, I wasn’t trying to suggest how it can be used to solve existing engineering problems – I’m afraid I might have misled you – my apologies if that’s the case. If you ask me (again, hopefully upcoming, in new posts!) I think that PP is still incomplete in reverse engineering terms, some (possibly multiple) fundamental ingredient(s) are still missing (for example: how exactly predictions are generated and how the search for better predictions is guided by the error signal).
    Hence my suggestion on CE: if the aim is to copy biology to solve engineering problems, an incomplete framework is unlikely to work; however, looking at the detail of some proposed elements might still provide useful hints, nothing more.

    On Free Energy itself: I myself struggle with the concept. I have my own mental interpretations, but I’m pretty sure Friston wouldn’t endorse them, so I’m too cowardly to voice them in public (for now: I still want to think about the subject, knowing/hoping that I can improve my understanding). Talking about understanding, may I ask you what you mean with “Free energy in Friston’s sense is actually a mathematical, not a physical, quantity“? We may be talking past each-other, as I’m pretty sure that I don’t really understand what you mean there.

  4. Patrick Kenny says:

    I would say that PP is only half the story: it seems to be flexible enough to explore questions about individual sensory modalities but by itself it cannot explain how the various modalities cooperate with each other. I don’t see that there is any hope of answering that sort of question without a thoroughgoing Bayesian formulation — to wit, the FEP.

    So to answer the question about “Free energy in Friston’s sense”: he is talking about variational free energy (math), not thermodynamic free energy (physics). In Bayesian machine learning, variational free energy is a measure of how well a probability model has succeeded in explaining a given set of data (the better the explanation, the lower the free energy).

    If that is Greek to you but you suspect that it might still be worth your while trying to figure out what Friston is up to, get back to me. This touches on what seems to be a major problem: there seems to be a lot of confusion about what the Bayesian brain is supposed to be doing.

  5. Sergio Graziosi says:

    🙂
    Patrick, we’re slowly getting to understand one another. I’m forcing us to make things explicit (probably too much) also because I’ve witnessed too many conversations go sour when people meant slightly different things but were using the same words. In this multidisciplinary territory, the danger is always present.

    Never mind, I now know what you meant with “[variational] free energy (maths)”. It’s not Greek to me, more like Latin: something I did study, but still requires plenty of effort to parse!
    [Thanks for your offer! But beware: I might accept it, someday.]

    [All: a good explainer, with added useful criticism, comes from Wolfgang Schwarz and can be found here.]

    I also think I’m finally getting your main focus, it’s not that you didn’t explain it, it’s that I’ve been slow to move away from mine.
    Specifically: when you talk about FEP, what you have in mind is the computational problem it promises to solve (or might help solving). This is because you’re trying to see if and how one could use it to design new things. It took me some time, but I think I finally see why (namely: it is one of the things that FEP does promise to do, duh!). In my case, I’ve been exclusively looking at how and if FEP might be useful to figure out what neurons/brains do.

    Thus, when thinking of the engineering perspective, the path I’ve always assumed is:
    Some theory (not necessarily FEP!) ->
    -> (a new) look at actual physiological neural network (the wet stuff) ->
    -> figure out what it does ->
    -> model it computationally ->
    -> see if the model contains stuff that can be recycled for engineering purposes.

    If I’m getting it right, your interest is quite different:
    FEP (a theory that promises computational tractability) ->
    -> Engineering application of FEP.
    From my perspective, it may be a viable “shortcut”, but I am not surprised it isn’t an easy one.
    Does this clarify where I am?

    there seems to be a lot of confusion about what the Bayesian brain is supposed to be doing

    Hmm, if this is a polite way of saying that I am making a mess of it, please leave politeness aside and unpack. Last thing I want to do is add to the confusion. I’ve been trying to express in plain language the ideas behind PP, there are reasons why I don’t mention Bayes very often, but it’s quite possible that I’m getting stuff wrong.

    Finally: thanks so much for prodding me. Right now it seems that you’re helping me to get over the current obstacles and to start putting together the next chapters. Highly appreciated.

    • Patrick Kenny says:

      Sergio, that’s right, what caught my attention was the question of whether the FEP might conceivably be of practical use to hard-engineering types.

      I am glad to see that you mention Schwartz’s post — that’s the sort of robust criticism that the FEP needs. Now there is somebody who understands the Bayesian Brain hypothesis 🙂

      But of course I wouldn’t have made an effort to understand the FEP unless I suspected that it might actually be correct neurobiologically, that everything the brain does really is explicable in terms of prior beliefs and free energy minimization. (Clark calls this the “desert landscape” story: Hohwy seems to embrace the idea, Clark to seems to reject it.) I have done some evangelizing for the FEP on quora.com recently e.g.

      https://www.quora.com/Does-consciousness-serve-any-evolutionary-purpose-Is-it-even-necessary-for-biological-systems-to-work/answer/Patrick-Kenny-5?srid=OQSj

  6. […] The list above is incomplete. It took me a very long time to write this post also because I had to find a way to organise my thoughts and establish some reasonable criteria to decide what could be left out. The biggest omission is about the Free Energy Principle. This is because criticising FEP requires a full book, cannot be done in a few lines. Secondarily, such criticism might be aimed at a too broad target, and thus fail to be constructive. [For the gluttons: I’ve covered the brightest side of FEP here, while some hints of criticism are in this discussion.] […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow me on Twitter

All original content published on this blog is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Creative Commons Licence
Please feel free to re-use and adapt. I would appreciate if you'll let me know about any reuse, you may do so via twitter or the comments section. Thanks!

%d bloggers like this: