This is my notes after reading “Perspectives and Problems in Motor Learning” – Daniel Wolpert, 2001.

Summary

The paper is intended to explore the mechanics that humans and computers use to “learn” motor skills. This includes comparing how humans learn motor skills with how different fields of artificial intelligence including supervised, unsupervised, and reinforcement learning may accomplish the same thing. It also suggests some of the problems and barriers that existing technology exist to truly scale this type of learning.

Notes (representing what I surmise is suggested in the paper – not necessarily what I agree with):

• The brain’s whole purpose in life is to produce movement, since movement is the only way to truly interact with the world (even speech is a result of movement).
• We are born with some innate motor controls, and this is demonstrated via sensory deprived babies (blind and deaf) that don’t need to learn to smile. They just do it.
• However, despite the innate motor skills, learning is still clearly required, as teh world is non-stationary. The world clearly changes, as do our bodies (not just kids turning into adults, and people gaining weight, but people losing teach, growing fingernails, etc).
• Motor learning could deal with the brain learning how to send control signals, or the body (muscles) learning how to evolve. Most of motor learning deals with the former
• Motor learning is really the mapping between motor command, to sensory consequences, and the inverse.
  • The forward view (motor command => sensory consequences) creates a model predicting what senses will be felt if the agent behaves in a certain way. This is where supervised learning is most obvious. Taking an input and mapping it to an output as the self generated training data.
  • The inverse view (sensory consequences => motor command) are what allows the agent to decide which action to perform to achieve a desired goal. This clearly uses reinforcement learning since there are many different paths to a goal but the optimal one is difficult to find.
  • Unsupervised learning seems to be what allows motor primitives to be found. With these primitives, other more sophisticated actions can be learned to generate the desired consequence.
• There appears to be evidence that this model (supervised + unsupervised + reinforcement learning) is similar to how the brain works using dopamine and the cerebellum.
• But there are significant problems with this model. One such example is the rate at which visual data is processed doesn’t seem to be fast enough to make these foreground decisions. Furthermore, representations are needed, and given the massive state action space (600 muscles could either be flexed or not flexed), 2exp600 is a massive number.

Further reading and thoughts

• It’s pretty fascinating to consider optimizing physical action as being the whole purpose of the brain’s purpose. What does this mean for dreaming and planning? Are those prolonged brain activities designed simply to optimize future physical interactions? There’s too much buddhist in me to completely accept this 🙂
• I really like the idea of developing a set of primitive predictors which can interact to build more abstract predictions. Which is a good segue to future readings …
• Horde: A Scalable Real-time Architecture for Learning Knowledge from Unsupervised Sensorimotor Interaction
• Continual Learning in Reinforcement Learning