I’ll put this out with my standard disclaimer, I’m not @steve and as such, I cannot guarantee the accuracy of my interpretation but I’ll try not to make any egregious errors and I apologize in advance for any inaccuracies.

First, a few definitions

YIN/Bottom up/Afferent – for the purposes of vision we will define this as any visual or auditory stimulus (the deeper into the brain network that you explore, the more abstract YIN gets but every map in the brain still uses the incoming signal in the same way. YIN gets processed it as if it were sensory data, the same way as from eyes and ears.)

YANG/Top Down/Efferent – This is the signal which leaves from a map or organ and tells the lower level maps what they ought to be doing (in practice, there are usually many YANG signals which converge from different maps and compete with any given map’s own INTRINSIC goals)

INTRINSIC – every map wants to be in it’s most comfortable state which essentially amounts to balancing YIN and YANG, but there can be restrictions to their ability to do this (e.g. we can only turn our eyes so far before we have to move our head. this situation causes a YANG signal to pass to the head map as an EXTRINSIC goal.)

EXTRINSIC – essentially any goal passed from another map or organ

LOCAL SPACE – The immediate area around a creature as represented by a 3×3 grid, each square on the grid is exactly the same width as the creature’s visual field and represent absolute coordinates within a larger world grid.

BODY SPACE – the relative positions of stimuli compared to our body, this is distinguished from LOCAL SPACE by the relative positions of objects/stimuli in relation to a creature

HEAD SPACE – The relative position of a stimulus compared to the current location of our head. moving the head will cause the stimulus to appear to move in relation

RETINOPTIC SPACE – the total visual field of the creature, everything within a cone of vision emanating from a point centered between the eyes as represented by a 2d map of X and Y coordinates (technically only one eye exists, the appearance of two eyes is merely cosmetic) in terms of YIN passed from the body, ears, or memory a stimulus can be outside of the visual field thereby requiring the head and possibly the body to turn in order to fixate on it with the eyes, depending on how far out of range and how SALIENT it is.

RECO – the object recognition and learning map

SALIENT/SALIENCE – how significant a stimulus is

SACCADE – tiny darting movements of the eyes. Whatever is most SALIENT will cause our focus to to fixate for a moment until the next SACCADE shifts it elsewhere

With that out of the way, lets first direct our attention to local space. As you can hopefuly see from my terrible diagram, the creatures exist in the center of a 3×3 mental grid. The size of the grid is determined by the size of a creature’s visual field, each square of the grid is exactly the same width as the widest point in the creature’s visual range so as long as the creature remains within the central zone of the grid, it can never see beyond it. Any objects or other creatures within local space will be remembered even if they are behind it or otherwise out of visual range. A creature can wander freely in the central zone of the grid, but as soon as it crosses the boundary line, the grid gets shifted by one space in order to keep them within the center, this has the additional effect of causing objects that they move away from to be forgotten (not truly forgotten as deeper parts of the brain still remember them with less precision, more like “out of sight, out of mind.”)

The reason that all of this is important is because a few compromises have had to be made when it comes to the visual recognition systems of our creatures. No one owns a computer capable of pixel perfect vision or ray-casting for multiple instances of creatures within the same world all at different perspectives, so Steve has figured out a way to simplify it without cheating too much.

Okay, so the cheaty bit goes like this. All objects within the 3×3 local space are plotted out as points on the grid and each one is assigned a SALIENCE regardless of it’s position or visibility. These points aren’t the actual objects themselves, just an assortment of facts about them. That is to say, when a creature looks at a carrot it will not see the same carrot we do, it will receive information about it (a sense of orangeness, a sense of plantness, a sense of longness, skinnyness and several other identifying characteristics represented as numerical values which can effectively be plotted onto other maps.)

This information is passed through a series of “daemons”, these daemons do basic calculations on each object to determine how important it is. Essentially all of the objects in the local space are competing for attention and the winner decides where the eyes will land at any time. once the eyes saccade to the object with the most salience, they will remain fixated for a moment and it’s salience will begin to drop. This in turn allows for another stimulus to become most salient and start the saccade process over again.

Each saccade stores this information about the target in local space and helps the brain make sense of where it should look next.

All potential targets within the visual cone will have a steadily increasing salience so that hopefully our creature will be compelled to look at and examine each element of its surroundings.

First we have the VisibilityDaemon, this decides which objects within local space are valid saccade targets (we cant decide to look at the details something we cant actually see, first we need to bring it into view before we can decide where precisely to lay our eyes.) For example, we can be aware of an object behind us, as we remember seeing it there previously, and when we hear it make a noise, we can assume it was that object, but it’s not until we can lay eyes on it that it actually becomes a target to look at and we can fixate on it precisely in space.

We have a FeatureDaemon which is called upon to tell the eyes where on a given object it should fixate. For example, buttons on a machine should be more enticing than the machine itself, just like a player or another creature’s face should be a more likely target to fixate on than their feet. This daemon steadily adds salience to these kinds of significant features (these features are actually coded directly into the objects themselves and the daemon interprets them.)

Next is the SpatialDaemon, this one calculates the distance and location of visible objects, closer objects are more likely to be salient than distant ones, then it stores this info for other daemons to use. I think of this like depth perception, it’s a kind of cheat for monocular vision.

Spatial awareness is followed by the KindDaemon which assigns salience based on the type of object. Plants and people tend to be more salient than furniture, utensils, devices and animals.

We have a MotionDaemon because moving objects should be more likely to grab our attention than stationary ones, and we have a SignalDaemon which increases the salience of certain objects in local space anytime they make a noise, a flash or otherwise broadcast sensory data like bumping into us.

The DeedDaemon is crucial to learning from others and learning the consequences of ones own actions. This daemon assigns a lot of salience to any object or person that is acting on another person or object in the world, it should allow for creatures to learn how to interact with new objects simply by watching another individual do it first. It is also activated when an individual does something themselves, essentially to alert them to the consequences of actions, be they good or bad.

The SeekDaemon is used to attach salience to objects which a creature is currently thinking about. if an object is similar enough to the top down signal coming from RECO, it will tend to be more salient than other objects. The more similar the object a creature has in mind is to the object in it’s field of vision, the more likely it is to draw focus.

Last, we have the FinalDaemon which takes stock of all of the salience calculations made by the earlier daemons and any top down goals from elsewhere in the brain (bumps, noises, navigation waypoints). Ultimately, this daemon has the final say on where the eyes land and whether to fixate or saccade to a new target.

So now that we know why the eye/brain decides where to look, we should examine how it makes this happen on a physiological level.

What I am about to discuss is what Steve calls the Gaze Chain. This is an excellent model system for understanding the larger processes at work within a creature’s brain. As you are likely aware, the brains of our creatures are very compartmentalized; each creature’s mind is made up of around 100 different maps which can represent sensory data, a literal (well figurative technically) map of the environment, or other more abstract concepts. It’s probably best to just to describe the maps within the Gaze Chain and their functions rather than generalize too much.

We have a system of three maps GAZA, GAZB, and GAZC, visual stimuli incoming from the eye organ enters through GAZA into retinoptic space which if you remember from our definitions can be represented as a 2d grid. this isn’t the same 3×3 grid of local space, it maps 1:1 onto the relative location of objects in the visual field. the intrinsic goal of GAZA is to orient itself so that the salient visual stimulus is centered on the retina (the creatures don’t actually have retinas, but it’s a useful analogy.) if a salient object is offset from the retina, then GAZA sends a yang signal to the muscles which control eye movement and this essentially attempts to center the signal. If a salient signal is coming from a higher map or sensory organ such as ears or skin and it’s too far out of range for a creature to swivel their eyes, this triggers a yang signal to GAZB which represents head space.

GAZB takes this bottom up signal and reads it as yin, this represents the senses telling the head “I’d really like to look over there but it’s further than I can rotate my eyes, could you turn to face it please?” The eyes tend to get tired easily, if an object is salient enough to keep their attention, they will keep nagging GAZB with steadily increasing urgency to turn the head in order to face the stimulus so that the eyes can relax. In this way, the intrinsic goal of GAZA becomes the extrinsic goal of GAZB.

If GAZA and GAZB can’t bring the object in front of the retina on their own, then GAZB sends a yang signal to GAZC which compels the body to turn and face our salient stimulus.  GAZC represents body space and is used by other maps to orient the creature towards salient signals so that the visual daemons can work their magic. GAZB also gets tired and nags GAZC to turn the body, our creatures don’t like to have their necks craned to the side for long periods any more than you or I. This creates an automatic reflex which orients the eyes, the head, and the body to any sufficiently salient targets. 

So now we have seen how an object is translated first from a relative eye position, to a relative head position and finally to a relative body position. Once a object has been located in body space like this, it gets translated into absolute coordinates as represented by the local space grid. This information keeps getting translated and handed higher up the hierarchy into memory and executive processes in increasingly abstract ways.

The process can also work in reverse. A creature can consider where it last saw an object and send a top down signal to GAZC, translating the absolute (remembered) location in the world into a position relative to it’s body. This signal gets passed down to GAZB and translated again into head space and ultimately through to GAZA and into retinoptic space providing the information needed to orient a creature’s eyes toward the object it would like to attend to.

I’ve tried to make this all as clear as possible, but I’m so deep into my research that it’s hard to remember what parts of this might be common knowledge and if I missed anything foundational that would be helpful to others. My journey will continue and I’ll try to keep posting informative documents as my understanding evolves over time. If I’ve left anything out or anyone has questions, feel free to ask!