POMDP

In working on my Feeling Abandoned robot tool box I ran into what I thought was a conceptual problem with (Partially Observable) Markov Decision Process modeling of embodied agents. Around that time I went to an information theory talk at SFI by Daniel Polani. He had a really nice diagram of the system of interest that (seemed to) illustrate my conceptual lapse. As I thought about blogging my misgivings I searched the interwebs for the diagram and came up with this:

http://www.mis.mpg.de/ay/index.html?c=projects/embodiedai/embodiedai.html

As it turns out this image was made by Keyan Zahedi for my friend Dr. Nihat Ay's research group, where "...the presence or absence of arrows in the diagrams resulted from many discussions that we had in my group.." (personal communication, Dr. Ay).

My issue was with the right-hand illustration, so I should (try to) explain just WTF...
In that diagram:

• 'W' stands for a state of the external world and the top black lines show progression through time;
• 'S' is an agent's sensory sample taken of the world state at each time step;
• 'C' is the internal computation or cognition performed by the agent, where red lines indicate sense input and green lines indicate command outputs to actuators. I presume the lower black lines represent the agent's internal state which may change on each step as well;
• 'A' stands for the agents actuators taking some kind of action and the yellow lines indicate that the action has some effect on the world state as it advances.
• Lather. Rinse. Repeat. From T0→T∞

The specific problem was that my sensors are over-sensitive and prone to getting the wrong ideas depending on what the robot happens to be doing at the time. Two examples: I sense motor current in order to determine if the robot is stalled -- the current (i.e., power usage) will go up when the wheels can't spin. But there is a current spike when the motor starts, so I need to ignore that input for a short period after each motion change. I also have an accelerometer to detect if the robot is bumped or lifted (I wanted to detect if it is moving in the right direction, but for the most part the noise is greater than the signal). This sensor wiggles greatly when the motors start or stop and thus also needs to be ignored at certain times, including when the grippers are being operated.

In a more real-worldish example, think of running. You pretty much ignore the bang-bang-bang jarring of your footfalls because you expect them, but bumping into a telephone pole gets telegraphed pretty quickly.

The general problem is that one needs to have knowledge of the agent's behavior in order to make sense of its senses. Biological systems have sensory inhibitors that focus the input data to the particular tasks being performed. This isn't clearly represented in the right-hand diagram.

I thought, ah-ha, I've caught those theory guys over-simplifying. Then I looked more closely at the left-hand diagram and noticed the black arrow between Actuators and Sensors labeled Internal stimulation... Dang it, maybe not. So I wrote to Nihat for clarification and got:

Whenever the system is expected to maximise an information-theoretic quantity, such as the predictive information, and it has these internal links, it does it by decoupling from its environment. In other words, it starts to dream.  Our solution: simply consider everything "physical" as being part of the world W. This also includes the body of the agent! Comparing this with the diagram on the left, the internal and external stimulation will go through the world W, which is the purple part of the diagram on the left (including the dashed line).

Which contains (at least) two interesting ideas. The first is that, given enough control, an agent might preferentially use internal over external senses -- decoupling and dreaming. And the second is that we can eliminate that by treating knowledge of agent actions as 'senses' in themselves. Sort of Extra-Sensory (pun intended) Perception. While it is theoretically possible to get all the body-state information indirectly through the environment, it is a very noisy and computationally intensive task. But by treating some of the bodily functions as external senses we can short-circuit the noise and avoid the decoupling. In theory.

Now I have to mull over the whole mind-body problem thing again though...