REWIRING NEUROSCIENCE

2010-05-12T11:43:00.000-07:00

Chapter 13
How does visual memory work?
Photo courtesy of Ann Cantelow. The multichannel neuron model ascribes numbers to channels. The channel numbers store and communicate analog data. They can also be used, in a distinct addressing system, to sequentially query the twigs of visual memory.

Addressing and retrieval
For retrieval, the model requires two types of neurons: 1) an address generating neuron, which drives 2) a data storage neuron. To activate a memory stored as "a thing in a place," a stored datapoint must be addressed at precisely that place. In the specific case of a stored pattern of three bleached disks imported from a photoreceptor, a trio of associated datapoints, twigs, must be addressed, one right after the other.

We have a mechanism for generating sequential addresses. The principle is inherent in the multichannel neuron model. The address generator can be the commutator we have postulated at the axon hillock.

To stimulate the first 9 twigs of memory, #1 through #9, each in turn, requires this sort of circuit. The output lines of the axon driven by the addressing commutator are telodendrions, each corresponding to a channel. In this illustration of this model, telodendrions are numbered in order of their firing. Each individual channel synapses to a dendrite. Each dendrite will be stimulated in its turn, in accordance with the ascending circular order of the addressing commutator.

Each dendrite is a “twig memory”. It stores a channel number that stipulates which channel shall be fired in response to the addressing signal. The effect can be tabulated:

The dendrites, which comprise the twigs of memory in this simple model, are each stimulated in turn. The pattern of bleached disks that each twig has memorized is fired back into the nervous system – precisely replicating the pattern originally dispatched from a single photoreceptor’s outer segment at some time and day in the past. In the table, 9 upticks of the address counter’s commutator correspond to a trio of 3D pixels and 3 frames of a film strip. [A slicker model might use just one address tick to elicit all three datapoints, characterizing intensity, wavelength, phase -- but the point is, visual memory is sampled and read out by the ticking of a sequential address counter. It is probably written in the same way.]

All pixels recorded from the retina at the same time, stored in twigs on other photoreceptor antipodal "trees" will have identically the same time stamp in their address. So simultaneously, synchronously, one pixel from every other “tree” or photoreceptor antipode in the retina of memory is being triggered.

The effect is to pump out of memory a stream of past images -- each image made up of millions of 3D pixels. The system is massively parallel and, therefore, moves whole images all at once. It is lightning fast.

Why don't we see these torrents of images from the past? Why aren't we drowning in images? Because these are not literal images. They are images of the Fourier plane. Fourier images are invisible to us, except perhaps in the special case of LSD users. Literal images may impinge on the consciousness as, in effect, search products, but the search itself is conducted as a Fourier process and is unconscious -- offstage and out of sight.

Numbered synapses -- new evidence, old idea
The idea there might be some sort of detectable ordering or sequencing of synapses on the dendrites is attributed to Wilfrid Rall, who suggested it in 1964 in support of a wholly different and unrelated model of the nervous system. In the 24 September 2010 issue of Science there is a featured report that reinforces the notion there exists some sort of sequentially ordered input pattern in the dendrites.

In these experiments, a programmed series of successive stimuli is made to “walk” from synapse to synapse along the dendrite. If the stimulus series progresses toward the cell body it is more likely to trigger off action potentials than a programmed series of stimuli that walks the other way, away from the soma, toward the tips of the dendrites.

The front half of this experiment consists of the selective stimulation of a row of individual dendritic spines, one after another, using a laser to precisely localize release of glutamate. The basic technology was outlined here. The back half of the experiment is conventional, and consists of electronic monitoring and tabulation of the axon’s response.

In terms of the multichannel model electrophysiology is difficult to interpret. However, a significant feature of the model is a staircase of firing thresholds. One might speculate that as the stimulus is made to approach the soma, it is finding or ultimately directing a pointer to lower and lower firing thresholds, which is to say, lower channel numbers. These low numbered channels would be more easily triggered than higher numbered channels.

Unfortunately there is easy no way to directly measure or guess the channel number associated with an action potential in passage, if indeed multiple channels exist. Again in terms of the model, a plot of channel numbers versus synapse position on dendrites (or, using different techniques, on the teledendrions) would produce a fascinating picture. In any event it is interesting that even conventional electrophysiology suggests there may be some kind sequential ordering, progression, or directional structuring that underlies a map of dendritic spines.

The model
In modeling this visual memory system I think it would be best to use automated rotating or looping machinery, just as you would in many familiar recording and playback devices. The rotating machine is the commutator. At each addressing tree, let the loftiest addressing commutators walk forward through time automatically, incrementing higher channel by channel. Rough synchronization among trees should suffice. Now, instead of hardwiring and broadcasting addresses in detail, the retrieval system can simply be given a start date/time and triggered off. A string of retrieval instructions will ensue. The system will, in effect, read itself out like a disk drive.

As a practical matter, the model of a retina of memory should probably be constructed in software. Each tree of memory can be modeled as a disk drive storing analog numbers representing 3D pixels, stacked in serial order, that is, the order or sequence in which they were originally captured from the eye. Millions of disk drives, then, each of relatively modest capacity, comprise a retina of memory. In a primitive animal one would expect to find a single retina of memory. In a sophisticated animal, many.

Let’s say the memory trees pre-exist in a newborn animal and that their twigs are unwritten. Each branch is a point in a commutator sequence, and identifies time (that is, sequence) ranges.

From the point of view of addressing the visual memory, reading and writing are, as in a disk drive, similar processes. The writing commutator walks forward through the present moments, guiding incoming 3D pixels from the eye to a series of novel addresses. To elicit a visual memory a reading commutator, which could be the self-same machine, walks forward through addresses denoting a film strip of past moments.

In effect, the pointer of the base commutator on the address generator, as it ticks ahead, is the pointer of the second hand of a system clock. Although the images are recorded at a stately and regular rate, such as one per second -- the recall can be made to happen as fast as the commutator is made to sweep. And it could scan backwards as well as forwards.

How is a pixel memory deployed?
This is an unsolved problem in the model. We have to assume it happens but the answer isn't easy or obvious.

We have stipulated what a 3D pixel memory is: Three numbers -- integers -- that represent a pattern of light recorded from three disks in a single photoreceptor at a particular moment in time. The three numbers are sufficient to specify the instantaneous wavelength, intensity and phase of the incoming light, as read out of a standing wave in the outer segment of the photoreceptor.

We are suggesting these three numbers are configured and stored in the brain as an addressable twig of memory -- three dendritic launch pads for three action potentials to be fired down three specific, numbered axon channels. It is nicely set up, this memory, but how did it happen?

The commutator figures prominently in the problem.

The operation of an initial readout commutator in the addressing neuron seems clear. It simply counts up or down. Other commutators fan out from the initial or system counter. At the upper tier of the addressing tree, the commutators, once toggled, can tick forward “on automatic.”

But what about the commutator in the proposed memory neuron?

In the most basic model of the multichannel neuron, developed in Chapter 2, the neuron is functioning as a sensory transducer. The commutator pointer rotates up to a specific numbered channel in proportion to an input voltage or graded stimulus.

But in the memory neuron, we want the pointer to go, first, straight to a remembered channel. Then, second, to another remembered channel. Then, third, to another remembered channel. Hop hop hop. From the address neuron the memory neuron receives three signals in a sequence, via telodendrions 1, 2, 3. The data neuron fires channels corresponding to three remembered photoreceptor disk positions: 2, 7, 34.

Instead of responding proportionately to an input voltage, as in a sensory neuron, the commutator in the memory neuron is responding discontinuously to a memorized set of three channel firing instructions. So the needle of this commutator must swing, not in response to an analog voltage input, but in response to a pixel memory.

In the multichannel model synapses connect individual channels, rather than individual neurons. This suggests some other possible solutions.

It could be that the commutator is simply bypassed, so that the appropriate axon channels are hardwired to the dendritic twigs of memory. Synapses at the soma could suggest a short cut past or a way to overrule the inherent commutator.

Maybe there is some rewiring or cross wiring at the level of the dendritic synapses. To borrow a term of art from the conventional playbook of memory biochemistry, maybe the synapses are subject to "tagging." Maybe biochemical markers delivered into the dendrites when the memory was originally recorded are specifying in some way the channel numbers to be fired.

This model suggests a Y-convergence of three neurons, not just two. One delivers addresses. One stores the data. A third neuron delivers original data from the retinal photoreceptor – data to be written in sequential order into the dendrites of the memory neuron.

Whatever specific mechanism one might choose or invent, the model requires that pixel memory arriving from a photoreceptor in the eye be stored in an antipodal neuron as a trio or linkage of three distinct channel numbers.

Experiment
One interesting aspect of this memory model is that it suggests an experiment. We are guessing that the individual channels of an addressing axon are, in effect, split out and made accessible as numbered telodendrions. If there is indeed a numerical succession – a sequential firing order – of the telodendrions, then this should be detectable. We were taught that the telodendrions must fire simultaneously. Is this always true? I bet not.

Superimposed networks
Note that we have assumed there exists a double network. Above the information tree there is a second tree, a replica of the first, used to individually address each memory "twig".

The principle of two superimposed networks, one for content and the other for control, is a technical commonplace. An early application was the superimposition of a telegraph network as a control system for the railway network. The egregious present day example is the digital computer, with its superimposed but distinct networks for information storage and addressing.
We are long in the habit of dividing the nervous system into afferent and efferent, sensory and motor, but surely there must be other ways to split it, e.g., into an information network and a addressing network. It is typically biological that one network should be a near replica of the other. Evolution proceeds through replication and modification.

Arborization and addressing capacity
The first anatomist who isolated a big nerve, maybe the sciatic, probably thought it was an integral structure – in essence, one wire. Closer scrutiny revealed that the nerve was a bundle of individual neurons. We are proposing here yet another zoom-down in perspective, this time to the sub-microscopic level . We suspect that each neuron within a nerve bundle is itself a bundle of individual channels.

It follows that the functional wiring of the nervous system is at the level of channels. Synapses connect channels, not neurons. This is why one might count 10,000 synaptic boutons on a single neuron’s soma. The boutons were not put there, absurdly, to “make better contact” nor to follow the textbook model of signal integration. They are specific channel connectors, each with a specific channel number.

The neuroanatomical feature that most interests us at this point is axon branching. This is because branching is of paramount importance in familiar digital technologies for addressing – search trees and other data structures. We have proposed a treelike addressing system for the visual memory in the brain. It is reasonable to ask -- where are the nodes?

Not at the branch points.

Photo courtesy of Ann Cantelow
Branching in a nerve axon is just a teasing apart and re-routing of the underlying channels. It is not a branching marked by nodes or connections in the sense of an T or Y connected electrical branch, or a logical branch in a binary tree.

For an axon that addresses a dendritic twig of memory, all functional branching occurs at the commutator.

Any anatomical branching downstream of the commutator, such as the sprouting from the axon of telodendrions , simply marks a diverging pathway – an unwinding or unraveling, rather than a distinct node or connection. In other words, the tree is a circular data store. The datapoints are stored at twigs mounted on the periphery of a circle. The twigs are accessible through a circular array of addresses. It is analogous to a disk drive in which the disk holds still and the read-write head rotates.

Photo courtesy of Ann Cantelow

Summary of the technology to this point
The tree in this photograph is a metaphor for the brain structure which corresponds to, and is antipodal to, a single photoreceptor of the eye. It is one single photoreceptor cell's remote memory warehouse -- a tree of memory.

Each twig is a destination with an address, a neuronal process narrowed down to just two or three channels. For example channels 3, 7 and 29, only, might constitute a given twig. Each twig is a 3D pixel frozen in time. The tree will store as many unique picture elements from the photoreceptor’s past as it has twigs.

As many as 125 million of these trees will constitute a retina of memory. We will look for ways to hack down this number, but for the moment let it stand. The point is, we are talking about millions of trees.

All these trees must be queried simultaneously with a particular numerical address, probably associated with a time of storage, to elicit firing from all the right twigs -- just one twig per tree. Properly addressed, a forest of these trees will recreate, almost instantly, a whole-retina image from memory.

In a primitive animal, it would be sufficient to remember 300 images from the recent past. This could be accomplished with a single addressing neuron, a single commutator. But in a modern mammal, it will be necessary to stack the commutators. A bottom commutator can point to any of 300 other commutators. And each of these can, in turn, point to 300 more commutators. With a simple tree of neurons, which is to say, a logical tree built with commutators, one can very quickly generate an astronomical number of unique addresses. We require one unique address for each twig of the data trees.

Are there enough addresses available in this system to organize a mammalian lifetime of visual memories? Yes. Easily. Are there enough memory neurons to match the addressing capacity of the addressing neurons. Probably not. The neuronal brain that lights up our scanners is probably running its memory neurons as a scratchpad memory. It seems likely there is a deeper store.

But will it work?
The memory mechanism we have sketched is probably adequate as a place to start. It would work for a directional eye in which changes in wavelength are highly significant cues to the position and movement of a target. It is a visual memory for retaining the "just now," a film strip comprising a few recent frames.

For an imaging eye, or a human visual memory, this memory system is not yet practical because, in its present form, it is a hog for time and resources.

Bear in mind that this model is extremely fast in comparison with any conventional model of the brain based on single channel all-or-none neurons. Two reasons: 1) It is an analog memory, and 2) it is massively parallel.

But a persistent difficulty with this model is serial recall. It appears this memory has to scroll back through all history to find relevant past images. And each image to be tested for a "hit" is composed from as many as 125 million 3D pixels. This is a huge array to deploy and compare, even though the pixels pop up in parallel. This is why van Heerden's memory seems to be such a dream system -- comparison and recall are instantaneous.

The van Heerden memory has a limitation, however, which a film strip memory does not. If a single face is presented to the van Heerden memory system, it can respond with a class picture in which the face appeared. This supplies context -- a surround of useful information associated in the past with this particular face. However, the system does not automatically position the memory in time.

In a film strip memory, in contrast, progression through time is built in. If the input is an image of a shark, the film strip memory will (like the van Heerden memory) turn up an image that puts the shark in a momentary context from the past. Maybe the shark of memory is freeze-framed in the middle of a school of fish.

But in the film strip memory, the film strip can progress forward through past time. The remembered shark turns, sees you, comes toward you, looms large, opens its mouth. In other words the film strip memory provides not only context and associations -- but also shows cause and effect. Here is a shark. Fast forward. Here are rows of teeth.

Fourier patterns courtesy of Kevin Cowtan.

It is reasonable to imagine that the memory we have today evolved from a simple (possibly Cambrian) film strip memory of the type we have described. For a simple animal in fixed surroundings, a film strip memory is fairly easy to model and easy to evolve -- to a certain point. But the film strip model soon becomes oppressively slow and heavy with data.

How could this model be speeded up and expanded, that is, modernized? First, by taking full advantage of the Fourier plane. Second, by introducing a metamemory, in effect, a hit parade. Third, by multitasking.

The Fourier Flashlight
Let's take a moment to orient ourselves, using the central, red DC spot as a point of reference. The red spot marks, in effect, the center of the Fourier plane. It also marks, probably, the position of the fovea.

Fourier transform of a literal image, courtesy of Kevin Cowtan.

We are looking at a structure inside a brain -- the retina's memory -- and the red spot marks that part of the memory antipodal to the foveal cones. The fovea is a wonderful thing but we can ignore it in this discussion. It is the hole in the doughnut in terms of Fourier processing and filtering. In terms of natural history, it can't tell us much. The fovea is a rare and special feature, a splendid particularity of primates, birds, and a few other smart and lucky vertebrates.

But the visual memory evolved in vertebrates that had no fovea. It seems a reasonable guess that the visual memory is grounded on Fourier processing. We have speculated in Chapters 5 and 6 that Fourier processing evolved in vertebrates as means of clarifying a blurry picture of the world obscured by glia, neurons, and vascular tissue because vertebrate photoreceptors are wired from the front.

One could pump a time-series of addresses into the whole forest of memory, but it makes more sense to address the memory very selectively, so that only part of it responds. This approach is indicated in the photo above with a yellow disk -- effectively, a Fourier flashlight. (The red spot marks the position of an antipodal fovea.)

Recall that any part of the Fourier plane can be transformed into the whole of a literal, spatial image from the world. By selecting such a small part of the whole retinal output, illuminated by the flashlight, we have reduced the data storage and processing problem to a tiny fraction of that associated with the original 125 million neuron source. It is because of this holograph like effect -- the whole contained in each of its parts -- that Karl Lashley was able to physically demolish so much of the visual cortex with lesions without producing a significant loss in the animals' visual memory.

Let me emphasize that the Fourier flashlight is a metaphor. It draws a convenient circle around a small population of neurons. In the model this is accomplished by addressing that small population, rather than the whole retina of memory. As the population of neurons and, thus, the flashlight spot gets smaller, the resolution of the literal image that can be recreated by Fourier transformation deteriorates. For rapid scanning and quicker retrieval, one would favor the smallest practical spot. Say a "hit" occurs, that is, a Fourier pattern scanned up from memory is found to match, more or less, a Fourier pattern at play on the retina.

At this point, the spot we are calling the Fourier Flashlight could be expanded in diameter to improve the resolution of the remembered image, broaden the range of spatial frequencies to be included, and perhaps pick up some additional and finer detail.

Where on the retina of memory should we shine the flashlight? Spotlight addressing of the memory map gives us a means to accomplish Fourier filtering. For edge detection, we should select a circle of trees at the outermost rim of the Fourier pattern (and retina), where the highest spatial frequencies are stored. For low spatial frequencies, position a circle near the red DC spot.

For most animals most of the time, an enhancement of high spatial frequencies has significant survival value. Here are two images from the Georgia Tech database, the first literal, the second with high spatial frequencies enhanced. Remark the sharp definition of the edges of the mirror frame, the clown's arm, and of the edges of the makeup brushes and pencils. In effect, an image in which edges are soft and ill defined has been turned into a cartoon, with heavy outlines emphasizing the edges of objects. This is the information an animal needs immediately. If the animal were looking at a shark in the shadows, filtering for high spatial frequencies would make the shark's shape unmistakable.

Edge enhancement in visual image processing and visual memory has an additional advantage, which is that it creates a very spare, parsimonious image consisting of a few crucial outlines. This important data of high spatial frequency needs to be surfaced quickly for survival purposes.

This yellow band indicates an address map for the outer regions of the Fourier plane, where high spatial frequency information crucial to edge detection is concentrated. These are neurons antipodal to rod cells at the outer periphery of the retina. It is interesting that this neglected outer frontier of the retina might have such a critical survival benefit for the animal. The animal's concept of "an object" arises from edge detection. The uncanny ability to distinguish the integrity of an object even though other objects may intervene is probably rooted in high spatial frequency detection in this part of the retina.

Chopping for speed
A memory that is capable of storing a film strip of images is valuable but slow. We can speed it up by massively cropping the images to be scanned for recall, using the Fourier flashlight technique described above. But one must still scan the images accumulated for "all time" to identify the objects currently in focus on the retina. If the object is, in fact, a shark, one doesn’t have time to scan through a lifetime of accumulated memories, in serial order, in order to recognize it.

One solution is to chop the serial film strip into, for instance, one hundred short film strips. Or one thousand. Or ten thousand.

In effect we are now playing multiple Fourier flashlights upon the retina of memory. In this way one could make multiple simultaneous scans and comparisons with the incoming retinal image from the eye.
Maybe the image to be matched is an old automobile. Figuratively, one Fourier flashlight can scan for the cars of the 60s, one for the cars of the 70s, one for cars of the 80s, and one for the cars of the 90s.

This works because each flashlight is addressing a spot in the Fourier plane -- where any and every spot that might be addressed contains all the information needed to match or reconstruct a whole image.

Memory anticipates reality
By multitasking the scans, we are breaking past the cumbersome requirement for serial, linear scanning and recall. It is possible now to see an advantage in importing from the eye a huge retinal Fourier plane. Because of its large area, the incoming Fourier pattern is open and accessible to thousands of simultaneous memory scans.

Think of each Fourier "flashlight" as a projector running, from memory, a short, looped film strip. Looping is easy because the addressing mechanism is a commutator.

Each projected frame in this little movie is a Fourier pattern that corresponds to (and is transformable into) some remembered object.

All these projectors run constantly. In this metaphor, the essential comparator, which is derived from the idea originally conceived by Pieter van Heerden, is a screen. The Fourier flashlights play constantly at spots on one side of the comparator screen. The Fourier plane imported from the eye plays on the other side of the comparator screen. Say the comparator has sensitivity to the sum of the juxtaposed signals on either side of the thin screen. Thus, the comparator will develop a high amplitude signal -- the "Voila!" -- wherever and whenever there is good agreement between a projected pattern from a memory flashlight and an imported pattern from the eye.

There are of course no literal flashlights projecting Fourier patterns onto comparator screens in the brain. The lights and projections are metaphors for processes that are carried out numerically in the model, using channel numbers for addressing, for pixel memory including phase conservation, and for summation.

If this model is viable then the photo above polka dotted with an array of Fourier flashlights is a significant illustration. It explains how we can glance at an object from a bygone époque and immediately identify it. It also explains how that same object can be pictured in different visual contexts captured at several different past moments.

The visual memory is not an image retrieved by combing through a serial archive of old, static, stored images. The memory is "live," fully deployed in an enormous array, waiting in anticipation for reality to arrive from the retina of the eye.

What does this suggest about the performance of the system? Neuroanatomy has identified, so far, about 30 representations of the retina in the cerebral cortex. By replicating the incoming Fourier plane, one can multiply the area available for the deployment of Fourier flashlights, increasing the number and variety of the arrayed memories that wait in anticipation of an incoming image.

So many flashlights. It has a Darwinian quality. Thousands of memories are on offer all the time. Upon the arrival of a new image on the retina, one or more memorized images shall be selected. It is never necessary to discover an exact fit to the incoming image. The signal from a Van Heerden detector ascends with and reports similarity, so it naturally finds a “best fit.”

The memory of memories
The chopped, short, looped film strip memories for images can be further edited or recombined in such a way as to store, in a serial format, only useful images. By this we mean images that have been very frequently matched to incoming images from the retina. Such images are, in effect, played back again and again. This requires a memory for memories. A hit parade of useful associations. We can call such a pared down memory projector a metamemory.

The comparator's criterion for a "hit" is likeness. Similarity. Over time, objects that happen to be alike would steadily accumulate in a metamemory. The notion is roughly analogous to the conventional concept of priming.

A metamemory based on like-kind associations will outscore the serial memories and produce the quickest recalls. In other words, it will succeed and grow. With "likeness" as the selection criterion, one would expect to see the whole system evolve as the animal matures -- growing steadily away from serial recall, which is simply based on recording order, and toward recall based on association and analogy.

For example, a sea animal’s immediate surroundings – a rock, a coral head, a neighbor who is a Grouper, a sprinkled pattern of featherduster worms… all this could be stored as a readily accessible library of constantly recurring rims, edges and patterns and colors. These valuable memory strips could be allocated extra space on the screen, for more detail and higher resolution.

One purpose of this more compact and efficient strip of memory would be to help the animal notice novelty in its immediate environment – anything not familiar gliding into the quotidian scene. A second advantage is a quick read of edible prey and dangerous predators -- and visual elements associated with them -- lines, curves, colors, patterns and textures.

It would be helpful to have a metamemory that excerpts the recent past, the "just now." A metamemory could be written to exclude all the trivial steps that intervene between cause and effect.

A modern creature – a Grand Prix racing driver for example – will have developed a metamemory that strings together the downshift points and apexes of every successive corner and straightaway in Monaco or the Nürburgring.

But note that it is no longer necessary to imagine a stream of memories that arrive and are stored in a linear time sequence like a film strip of a racetrack. An anthropologist might have a metamemory for the totems and icons of every culture she has studied. A biochemistry student will have a metamemory for amino acids, for sugars, for the ox-phos pathway, and branching, non-linear alternative pathways like the phosphogluconate shunt.

One can probably regress this principle, so that there come to be metamemories of metamemories, using key images as tabs, or points of entry. The comparators all operate in the Fourier plane, which is offstage and invisible to the conscious intelligence.

Replacing the retinal image
Finally, instead of simply scanning against the incoming image on the retina of the eye, seeking something similar from past experience -- the metamemories might scan against each other’s products, which are images extracted from the past.

An image from a memory associated with a former boyfriend from the 1980s might include a Chinese restaurant from the 1980s. A bright red menu from that restaurant might be matched by a bright red pair of boots that seemed fashionable in 2005.

The requirement for serial recall is now exploded. We can jump from visual fragment to visual fragment, and match memories not just with the eye’s reality of the moment but also with past realities glimpsed and recorded in past moments. The fragments are serial recordings, but they are short, selected, and looped.

Conclusions
The past and present coexist in the visual memory of the brain.

It seems that a visual memory model structured in this way could mix present and past imagery into something new. In the frequency domain images can be combined and recombined and subsequently Fourier transformed into literal images never before seen. In this way the system can do more than remember. It can invent.

By arraying thousands of Fourier flashlights simultaneously and in parallel, memory recall can be made sudden.

This is a brain model that can actually function using our absurdly slow-moving nerve impulses, which have typical speeds ranging from just 60 mph to 265 mph.

The model is able to do so much work in parallel and simultaneously because of a peculiar property of recordings made in the frequency domain that conserve spatial phase information: Each tiny part one might isolate encodes and can be used to reproduce the whole of a spatial image.

Thousands of elements of an incoming Fourier plane from the retina can be separately and simultaneously compared with past Fourier plane images pumped out of memory. The model is massively parallel, incremental-analog, and massively, simultaneously multitasking.

It requires a multichannel neuron and the conservation of spatial phase. Its playground and operating system is the Fourier plane of the brain’s retina of memory. This crucial Fourier plane of the brain is antipodal to, and is a representation of, the back focal plane of the lens of the human eye.

2010-01-27T05:01:00.000-08:00

photo courtesy of Guy Meagher
Let's suppose the brain devolved from and still works rather like an eye. What structure in the brain could upload and store the information contained in a single photoreceptor? What is the meaning of arborization?

Chapter 12
The mind as an eye

The natural history of visual memory
It has been at least 50 years since anyone seriously thought about human visual memory as a store of images, rather than as a repository for extracted features. So in asking how images might be stored and remembered, we are moving sharply away from the standard story. The issue was argued in Chapter 11.

But there were eyes long before there were images, and probably there was a visual memory as well.

The eye matured before the brain, and let's guess that in the early going, the eye alone was functioning as a brain sufficient unto itself. The eye had some built-in capacity to sample, shift and hold an input, maybe by storing just one single frame snatch, in order to notice, through comparison or overlay, what had changed in the state of the world in the interval before the next frame snatch arrived. We are using the word "input" rather than "image", since for a directional eye, the input would simply be a ray of light.

There was not enough hardware in the retina to store a long succession of past impressions, but memory proved useful, and so over time the brain evolved as a storage warehouse -- a separate place to stack the incoming streams of 3-dimensional snapshots from the eyes. This does not necessarily mean 3-D images, although it could. In every case it means data captured in depth along the z-axis of each photoreceptor's outer segment.

In this version of history, the brain evolved as a child of the eye, rather than the other way around. This did not have to happen – this offloading of visual memory and visual logic to a brain. An alternative path would be an extreme local elaboration of the eye.

Photo by Jean-Michel Peers

Birds, with their remarkable intelligence, may have taken this other path, for in birds it is the eyes, rather than the cerebral hemispheres, that have become exaggerated structures that fill the skull.

What is novel here? The brain and the eye are the same thing, and the tradeoff between these two distinct yet perfectly unified components has long been remarked. The eye is typically presented as "part of the brain," but one can just as easily say the brain is part of the eye. The intelligent eye, notably in the frog, was the subject of one of the most cited papers in history, "What the frog's eye tells the frog's brain." by Jerome Littvin et al. (Slow download). There is a well worn classroom wise crack about the retina vs. the brain, to wit, the dumber the animal the smarter its retina. The idea that the eye has an intelligence -- the equivalent of logic circuits -- is long accepted.

What's new here is the idea that the eye has or once had a memory -- and that the photoreceptors were the first memory organs.

Granted this supposition, one can try to guess the rest: A retinal photoreceptor memory starts out as a feature of an evolving photoreceptor, but it quickly runs out of storage space. When this memory is made remote from the photoreceptor, off-loaded into the brain, the photoreceptor mechanism still specifies in detail the structure and mechanics of visual memory. This is because the visual memory in the brain must mimic the retina of the eye in order to surface a remembered image into our conscious thoughts.

In other words, the visual memory organ of a modern brain is functionally analogous to a photoreceptor. Moreover, the integration of a memory is analogous to the integration of an image formed by light on the retina.

Early memory
How is visual memory recorded? It is usually assumed that the eye transmits to the brain streams of nerve impulses that report samplings of the present moment. It is the brain’s job to make a memory from these real-time signals.

But let’s suppose the eye has its own separate and distinct memory. If the eye is a recording machine, then it is actually a memory – an instant replay rather than a live broadcast -- that is being delivered via the optic nerve to the brain.

This type of memory is called sensory memory. It is basically a buffer memory associated with a sensory organ or, to narrow it down for this hypothesis, a memory stored within a sensory cell. Its purpose in a photoreceptor would be to compare two moments in time – the Now vs. the Just Now. Suppose the image presented for Now is a shark. If you could compare it with a remembered image snapped Just Now of the same shark, you would know whether the shark were coming toward you or going away. For a primitive animal, the ability to compare two visual impressions slightly separated in time would confer a huge survival benefit.

It is easy to see that a photoreceptor cell might serve wonderfully as a recording machine, because light induces decisive biochemical changes in a disk’s cache of rhodopsin. These changes linger – it takes time for rhodopsin to regenerate, or re-cock.

So inside a rod or cone there are indeed persistent biochemical changes in response to a change in the world. This is a familiar textbook definition of memory – a biochemical change in response to experience. We are trained to ascribe such changes (in the very next breath) to the synapse. But let’s stay clear of the synapse for once and experiment with the idea that the earliest visual memory was written as a chemical and conformational change in the rhodopsin arrayed in a photoreceptor’s disks. Note that a rod might comprise 1500 disks, so there is plenty of excess disk capacity that might be used for visual memory storage.

A photoreceptor memory would be “early memory” in two senses. In a modern animal it could be the first record of an image from the world. In a very primitive but sighted animal it could constitute the whole memory – evolved long before the rest of the brain augmented the eye.

The role of standing waves in photoreceptor memory
The "thing to be remembered" is not just a flat pixel -- not just the presence or absence of light at the outer segment of a photoreceptor. It is the presence or absence of light at specific points along the z-axis of the photoreceptor, captured at two different times, that could make these samplings truly meaningful.

Beads of light are stacked inside the outer segment. They stretch out and contract in response to changes in the world. It is the pattern of beads, or the information that can be extracted from this pattern, that constitutes a useful memory.

Because of the way information is detected and recorded from a standing wave, standing wave theorists have proposed that the photoreceptor might capture color as a peak-to-peak or null-to-null or peak-to-null distance along the z-axis of the photoreceptor. To review the concept of photoreceptors as wave detectors, see Chapter 8, Rods and Cones as Wave Detectors. The chapter includes references to the unexpected discoveries of action potentials in human rods and, subsequently, in human cones.

The structure of a photoreceptor memory
This photoreceptor is an example of the multichannel neuron first presented in Chapter 2 on The Corduroy Neuron.

Below is a schematic representation of a multichannel photoreceptor’s outer segment. It could be a rod or a cone. A single helical output channel is marked in orange on the cell membrane – one of hundreds. These 300 odd channels wrapped around the outer segment are roughly analogous to the dendrites of a conventional neuron. Illustrated below is the same outer segment with the membrane subtracted. The single channel is left in view to indicate how the disks relate to an output channel. How to connect the helix to the disk or disks? One could imagine the assignment of a single channel to a single disk. One might also visualize an array of disks, positioned periodically to capture a particular wavelength -- reporting that wavelength via a single output channel. An initial exposure to a standing wave will seriously photobleach only a few disks marking the intensity peaks of the standing wave. In this figure, photobleached disks located at intensity peaks are indicated in yellow.
The next, novel exposure to light may bring a signal of a different wavelength, and thus make an impression on a different set of unbleached disks.

Therefore a linear peak-to-peak measurement could also be made along the z-axis between a newly impressed wavelength pattern (in green, below) and a persistently bleached set of disks that were excited by light received 1 second ago – leftover hot points.

In effect, inside a rod or cone, one can make a linear, z-axis measurement between the light of the present and the light of the past. In the simplest case, this measured distance would tell the creature about a change in wavelength (color). Recall in detail the radar eye suggested in Chapter 9 on Eye Evolution. A continuous change in color tracks the passage of a target object across the animal’s field of view. The delta wavelength, frame to frame, tells the animal which way the target is moving. If the change is clocked (if indeed the primitive eye had a built-in clock) the animal could sense how fast the target is moving.

A z-axis measurement could also be made between present and past intensity peaks out at the proximal end of the outer segment. The primitive animal observes the change in a standing wave pattern and mutters to itself: getting redder, getting brighter, and getting closer.

It would certainly take a clock to make this little memory machine facile. Interestingly the retinal ganglion cells have been shown to produce circadian signals, so a clock is a perfectly plausible component inside the retina. Note too that the pineal gland, which is a biological clock, essentially uses benighted photoreceptor cells as its escapement or, if you like, its ticker.

In a truly primitive system, however, the clocking of a frame shift could simply be accomplished by a change in the incoming light.

This is a most rudimentary sort of memory machine, in which rhodopsin is the recording medium. There are more elaborate ways to imagine such a device. For example, one could segment the rod into compartments, longitudinally, and assign to each compartment in rotation a captured moment in time -- creating, in effect, a film strip.

But whatever sort of mechanism might be spun from the idea, the basic principle is constant. Disks are bleached by light of a particular wavelength at z-axis positions that are defined by that wavelength.

A change in these z-axis positions induced by a change in the incoming light (e.g., in its position or wavelength) can be detected. This is because the initial wavelength is preserved -- marked and bracketed in effect -- by persisting biochemical changes in the rhodopsin of the originally affected disks.

Transducin reads rhodopsin
There is a nice biochemical piste or trail of clues to follow here, since transducin, a G protein, is next in line after rhodopsin in the cone, and in the nocturnal biochemistry of the rod. If indeed the rod operates in broad daylight as a wave detector, it seems likely transducin or a related G protein is next to tumble after rhodopsin. In other words, transducin "reads" the light induced change in rhodopsin, day or night. One must ask where that biochemical trail might lead in a photoreceptor -- rod or cone --that exhibits action potentials.

Rhodopsin is embedded in disk membrane. Transducin is shown anchored below it. From a Wikipedia illustration created by Devon Ryan.

The alpha subunit of rod transducin (in red) is only 79% homologous to that of transducin found in cones, which are certainly daylight receptors. But the rod and cone alpha subunits work the same way and are in fact interchangeable. The alpha subunit is the business end of transducin and the place to start. Downstream components in this cascade could mark a distinction between day and night chemistry, and between the lumped analog response described in texts and an action potential reporting on specific standing wave peak positions.

Transducin is intriguing for another reason. There is no obvious complement or antipodes to rhodopsin in the brain, in the visual cortex for example. But one might hope to discover, as a hint at the memory process, an analogy between the G protein step in the photoreceptor and a G protein coupled step far downstream, in a destination neuron deep in the visual memory of the brain.

G proteins are implicated in certain memory hypotheses I find impossible to credit -- but it certainly makes sense that G proteins could be involved in remembering. Memory is hidden. G proteins are ubiquitous. From the movies, we have learned the best place to hide is in a huge crowd.

Why a brain?
Doubtless evolution could do a lot with the simple standing wave principle proposed for photoreceptor memory, and perhaps our own eyes can remember more than just one or two frames. But there are clear physical limits to the storage capacity of a photoreceptor memory based on rhodopsin. Hence, perhaps, the evolutionary pressure to create a brain memory as an annex to the eye.

In any event, the trick to modeling an eye memory, the essential quality that makes it possible to even conceive of such a thing -- is multiple output channels that can report on light intensity, or pick a peak, at each photoreceptor disk. These multiple channels constitute the numbered increments of a z-axis yardstick.

At this point we can imagine the form of a visual memory to be remembered in the brain. It is a representation in the brain of a pattern of bleached disks -- transmitted via the multichannel neurons of the optic nerve from not just one photoreceptor, but from 125 million parallel photoreceptors comprising a whole retina. This is a reprise of Jerome K. Jerome's cherished 1889 notion of the retina of memory.

Before we venture into a guess at how this might work, suppose we jump from the photoreceptor all the way to the other end of the memory process, and ask what constitutes a memory stored in the brain in a multichannel nervous system.

Memory is a thing in a place
The late E. Roy John neatly defined in his book Mechanisms of Memory (Academic Press, 1967), two different ways to model the spatial distribution of memory. Memory might be understood as:

1) “a thing in a place” or
2) “a process in a population”

In 1967, I think many neuroscientists viewed memory as “a thing in a place,” that is, a single neuron or pathway – a so-called labeled-line memory. Hebb's 1948 model of memory as a grooved-in pathway was still taken literally. Hubel and Wiesel were in their heyday. One could single out with a fine probe a neuron in the visual cortex -- and attribute to it the power to distinguish a specific angle of orientation for an object in the animal's field of view.

If it were actually true, as it then appeared, that a neuron could be somehow pre-configured to watch for a specific edge angle, such as 41 degrees, then labeled-line memory must have seemed to make excellent sense. The concept of memory as a “reverberating circuit,” roughly analogous to the recirculating memory loops of early digital computers, was still an accepted idea. Another model represented memory as a specific line selected by a telephone switchboard. Connectionism was not yet a movement. It would be 15 years before Hopfield’s paper describing a neural network as a content addressable memory appeared in PNAS.

In a conversation in the late 1980s, E. Roy John referred to the believers in memory as a thing in a place as “the labeled-line guys.” He thought they were utterly wrong. He emphatically preferred to model the memory as a “process in a population.” By “process” he meant a statistical process.

E. Roy John was a brilliant iconoclast, and at the time he wrote that book he was way out in front -- and largely alone. But today, 43 years along, cognitive science absolutely agrees with his notion of memory as “a process in a population” as exemplified by neural nets. The idea that vertebrate memory might be modeled as “a thing in a place” seems to have receded with the ascent of neural nets, but the conflict between "place" and "process" still has echoes today in models of sensory systems. Here's one example pertaining to taste, and another on itching.

The prevalent view of vertebrate memory is as a process in a population, but it is believed that changes at the synapses (which are places) influence the pattern of that process.

The atom of memory
Alors. Maybe it’s time for another swing of the pendulum, another total reversal.

In a nervous system constructed with multichannel neurons, memory turns out to be “a thing in a place.” It is not remotely “a process in a population.”

The irreducible minimum kernel of memory is a single channel number, 17 for example. This is a comfort. The atom of memory is a channel number. It is embodied as a numbered slit, a longitudinal channel the length of an axon -- one of maybe 300 such longitudinal slits.

How are the slits assigned a "number?" Either by position in a sequence or by a biochemical marker, and ideally by both. We have to belabor this because, after all, the nerve has no real number sense. It computes by analogy, for example, by comparing analog levels. An analogy must be made to something, some physical property. In other words there must be a physical basis for a numberline, and for marking and manipulating "numbers", something like a child stacking cookies to learn how to count.

If a number is assigned to a channel by its relative position, then this means the number follows from sequence. You can arrive at such a number by performing a count. Sequence is excellent, a natural way to create linear order -- an arrangement. In biology, and especially in proteins and nucleic acids, sequence is crucial. A sequence (and the shape that follows from it) is the way in which biological information is usually stored. One channel follows another counting around the cylinder of the axon -- and is therefore higher or lower in number.

We have used sequencing to associate a certain input stimulus magnitude (say, 5 mV) with a particular channel, using a model mechanism analogous to a rotating commutator arm. The sequence is directional, ascending or descending, depending on the direction of the arm’s movement, clockwise or counterclockwise.

If a channel number is associated with a biochemical marker, then on what basis? Angular position on the axon, as indicated by a commutator arm? That’s one possibility.

Another possibility is physical length – a linear distance around the axon cylinder from channel 0 to channel 21, for example. Or, if the channels are helical, the distance to a particular channel can be captured with a measurement taken linearly along the axon’s longitudinal axis.

How to create biochemical channel markers
If the ascending scale of channel numbers is linear, then there is no clear difference in channel spacing -- only a sequence -- so the distance from one channel to another is best read as a distance from zero in a particular direction. The distance from channel to channel tells us nothing.

However, suppose the channels are arrayed logarithmically, as in a shell. Visualize this section through a conch as a section through the neuron at the commutator. Note that each ascending channel is spaced further apart than the one before, and the sequence has a built-in direction. In this case the physical interval between channels increases with channel number, so each channel can be uniquely identified by its distance to an adjacent channel.

The first two channels, 1 and 2, are very close together. The last two channels, 299 and 300, are spaced wide apart The interchannel distance makes each channel physically unique.

In modeling these systems you could probably create or cleave a peptide to length, in order to precisely match each interchannel distance with a biochemical marker molecule. In other words, the biochemical channel markers could be derived from, or replicates of, strutlike and cytoskeletal proteins bridging the channels at the axon terminal. This would be a way to produce 299 unique chemical identifiers, each corresponding to and in some way associated with a specific channel.

When an action potential arrives at the terminus of a particular numbered channel, let its unique, numbered peptide curl up, snap small -- and package it in or on a synaptic vesicle.

Peptides are the handiest and perhaps least obnoxious candidate markers, but there are other possiblities, including I should imagine coding and non-coding RNAs. One thinks of RNA because the passage of the contents of synaptic vesicles from cell to cell is oddly reminiscent of viral infection.

In any event, in this model, pools of synaptic vesicles (like nerve impulses) appear to be identical from one synapse to another, but they might not be. If markers are being used, then the vesicles in a given pool are unique to a specific channel -- marked and numbered. It is the number, not the vesicles, that matters. The intercellular transmission medium, which is a neurotransmitter, is not the message. It is just a carrier. The message is a channel number.

What is the value to the model of unique biochemical markers arising at and tagging each channel with a number? If each synapse represents a numbered channel connection, then the distinction between channels is kept distinct physically, at least at the nerve's terminals. So why have markers?

The possibility that biochemical channel markers exist is basically a modeling convenience. If they do exist, then they are a chemical form of memory. If the biochemical markers are embodied as short RNAs, then their promise grows larger.

Instant memory
Note that in this model, there is no conversion step involved in making a memory. In other words, streams of nerve impulses are not somehow decoded or processed in order extract a useful fact to be converted into a memory. The channel number exists from the instant a stimulus is first sensed. Sensing and memorizing are the same thing. Every action potential in motion is, in essence, a memory in passage.

The smallest element of a memory in storage is a single numbered channel. In effect a memory is a number, 17 for example, that is just sitting there, waiting to be fired back into the nervous system. But where is "there"?

Let's quickly preview a visual memory model based on the multichannel neuron. In the brain, the memory store for a single photoreceptor can be drawn as a treelike structure. Each memorized channel number has an address, and therefore “a place.” In this metaphor, the place is a twig in a treetop.

Twig memory
A single number in storage is just one element of one 3D pixel. We have estimated that a 3D pixel, that is, a bleached disk pattern from a photoreceptor, could be fully characterized with as few as 3 datapoints.

Click to enlarge, Back to return
Three datapoints, which are bleached disks, should be enough to capture the color, intensity and spatial phase of incoming light at a point on the retina -- at a moment in time.

This photoreceptor disk pattern is represented in the brain by channel numbers and stored as twigs of memory. The twigs comprise 3 associated channels, or 3 numbers in storage, unfired. In this example the channels are 2, 8 and 51.

A nearby twig in the same treetop stores another, different 3D pixel pattern recorded by the same photoreceptor. But that pixel (e.g. 2, 14, 75) will have been captured at a different time, perhaps months earlier or days later.

Memory is not a process in a population. It is a thing in a place – an analog number (or in this case, a trio of numbers) parked at a unique address.

The retina of memory
If the memory warehouse for a single photoreceptor is a treeful of twigs, then the brain's retina of memory is a forest.The retina of the eye maps to the retina of memory. In other words, each photoreceptor of the eye's retina maps to a tree in the forest. This forest is the brain's recording medium, its film.

A stored snapshot selectively extracted from the forest comprises as many as 125 million twigs, or 3D pixels. It depicts both a literal snapshot image (from the central fovea, in the heart of the forest) and a back focal plane interference pattern (from the surround).

Here is a literal image:

And here is the interference pattern from the surround -- a different way to encode the same image.

Refer to Chapter 6, What Does a Memory Look Like. Any tiny patch of the interference pattern can be used to recreate the whole of the literal image (a duck in this example) by performing a Fourier transform of the patch. In this sense, the visual memory of an image is stored "everywhere," per Karl Lashley, even though the individual 3D pixels that comprise the interference pattern are stored as discrete numbers with clearly defined addresses.

Stored Memory is static, a thing in a place, and a "labeled line" in an entirely new sense: The line is not a neuron or pathway, but a numbered channel. The number is denoted by a position in a sequence and/or by a biochemical marker.

We have suggested how visual memories might be stored, but we have not yet addressed the problems of how these memories are written and positioned in time. However, it can be anticipated at this point that storage and retrieval -- especially retrieval -- will put memories into motion. If you were to watch storage or retrieval with an instrument, you would probably characterize them as "processes in a population." But the memory itself is a thing in a place and that thing is, amazingly, a number.

Content addressable memory
At this point we have chosen a “thing to be remembered,” which is a pattern of bleached disks; and we have a means of storing a numerical representation of this pattern of disks, as a trio of channel numbers associated with three isolated channels in a “twig” of memory.

To help determine what sort of read/write machinery is required for this visual memory model, we should establish here that what I hope to emulate is a content addressable memory in the manner of Pieter van Heerden’s optical memory invention.

The “content” is an incoming image from the retina. The job of the memory is to match it with a remembered image or images. For example, suppose the incoming image is a face. The memory will be ransacked for scenes – images – in which that particular face has appeared in the past – the class picture, for example. The effect is to place the incoming image into a physical context supplied by memory.

Van Heerden’s approach calls for a comparator/detector. Think of the comparator as a frame in which the incoming image from the eye is mounted. Against this standard image, the memory will pump up and superimpose past images until it achieves a match. In the van Heerden machine, memorized images matching the input image would be displayed on a screen. In an analogous brain, the matching images or "hits" would be surfaced into conscious thought.

All these cut and try operations are carried out in the frequency domain, that is, using images from the Fourier plane of the eye and stored fragments of images pumped out of memory – also in the Fourier plane.

Two fourier patterns overlaid for comparison. Upper image is a cat, lower image is a duck. Images courtesy of Kevin Cowtan. The two patterns are roughly aligned on their DC spots. In the model, the red spot corresponds anatomically to the position of the fovea. One image, let's say the duck, arrives from the retina. Each of its pixels is "live", generated by a photoreceptor cell. The cat image is one of a series of images -- a film strip -- pumped from the visual memory for comparison.

The Similarity Detector
Van Heerden’s detector was styled as a similarity detector. Two superimposed Fourier patterns that happened to be identical would produce a maximal additive response from this detector – a hit.

Less similar images would generate responses of lesser amplitude. One could set the sensitivity of the detector to select for only a few images of the very highest similarity. Or one could dial down the sensitivity so that it would register many sort-of-similar images.

In the initial discussion of the van Heerden memory at the conclusion of Chapter 4 we used the example of a racehorse presented to the comparator/detector. The literal, photographic image of the racehorse is automatically registered in the retina of the eye as a Fourier pattern.

If the similarity detector were maxed, it would only record a hit for the image of a specific horse – War Admiral, for example. Dialed down, the detector might report any racehorse, a stableful of them. Further down, it might report a hit for any image of a horse or horselike animal, including not only racehorses but ploughhorses, donkeys and burros.
At a still lower level of sensitivity, the detector might record hits for many, many images of four-legged animals: zebras, pigs, giraffes and rats. This simple grouping of four legged animals has the germ of abstract reasoning, accomplished without resort to logic circuitry, made possible by filtering and comparing Fourier planes.

There are various other good reasons for feeding Fourier images, rather than literal images, into the comparator/detector from both the eye and the memory. Perhaps the first is that all this activity is “invisible” to the human consciousness, which is only aware of spatial (literal) images. For this reason we are not subjected to a distracting bombardment of past images as the brain seeks a fit between the present image and a library of past images. One is made aware of the results, but the search is carried out behind the scenes.

The structure identified as the “retina of memory” is film to record incoming images from the eye. It could also serve as the picture frame for a comparator/detector. Each pixel of a recalled Fourier pattern – a candidate for a match with the incoming Fourier pattern on the eye’s retina -- is generated by a twiggish anatomical structure in the brain antipodal to the photoreceptor. For these remembered individual pixels, we are using the metaphor of twigs in a tree in a forest. The twigs are memorized pixels. The tree trunk is the brain's photoreceptor analog. The forest is the store of human visual memory.

Film strip memory
Are we really on our way to a content addressable memory? Not in the sense of van Heerden’s. In the original holographic memory he invented the incoming image was projected through a solid block of recording material. The image essentially “found” or detected its own likenesses at z-axis positions within that block. The query and discovery process was almost instantaneous because the system operated at the speed of light. Coherent light at that.

In the multichannel nervous system, given the specific bits of machinery we have accorded ourselves for model building in this speculation -- a true van Heerden memory seems out of reach. It may well be biologically possible, but I am not seeing a way to reproduce it from the parts list now in hand.

The type of memory we can most easily create using multichannel neurons is a stack memory. Each incoming image is time stamped – given a position in a sequence of images, a film strip. Retrieval consists of running the film strip of memory backwards and forwards to discover a match, from visual memory, for the incoming image on the retina.

The content of the image on the retina cannot be said to “address” the memory. It can only be said to establish a standard of similarity for images addressed by another means – specifically, counting and commutation.

It seems to me that because of its technical simplicity, a stack memory would be plausible as a primitive animal memory. For a creature chiefly interested in the Now versus the Just-Now, the purpose of memory is to notice changes. This includes changes in the apparent size and direction of movement of a predator or prey. Stack memory, a short film strip essentially -- is adequate for these purposes. A slightly deeper stack memory could help the animal mark the all important distinction between predator and prey.

<<PREVIOUS CHAPTER NEXT CHAPTER>>

2008-08-24T07:27:00.000-07:00

#11
The retina of memory

WE caught a breeze, after lunch, which took us gently up past Wargrave and Shiplake. Mellowed in the drowsy sunlight of a summer's afternoon, Wargrave, nestling where the river bends, makes a sweet old picture as you pass it, and one that lingers long upon the retina of memory.
Three Men in a Boat (Fiction, 1889, 197 pages)
-- Jerome K. Jerome

In invoking the "retina of memory" in 1889, Jerome K. Jerome was expressing a very Victorian idea. He may have read it in a newspaper, for in 1888 the mapping of a "cortical retina" was first reported in Sweden. It seemed reasonable then that the mind’s eye should have a retina. To the retina of the eye, there could correspond conjugate points on a retina of memory, somewhere deep in the brain.

We will attempt in this chapter and the next to move from Jerome’s retina of memory, which was a literary conceit, to an anatomical and technically plausible version based on multichannel neurons. If there exists in the brain at least one retina of memory, then where could it be? What does it look like? How does it work?

Jerome K. Jerome was a popular Victorian humorist. Three men in a boat, a comic travelogue and his most successful book, has been reprinted again and again. The 1911 lantern slide from the Oxfordshire photographic archive, above, approximates the riverside scene Jerome’s narrator (and Jerome himself) must have remarked, drifting by on his boat. On the right is the St. George & Dragon, a landmark inn and pub. It is still there.

In the 1880s, when Jerome K. Jerome went gliding down the Thames past Wargrave “in the drowsy sunlight of a summer’s afternoon,” the prevalent metaphor for human memory was the photograph. It was thought something in the brain analogous to the grains of a photograph must be the substrate of memory -- that the brain was taking pictures constantly through aperture of the eye. The retina of the eye was also (and often still is) presented as analogous to photographic film.

The cortical retina
Every point of light reflected into Jerome K. Jerome’s eye from the St. George & Dragon registered as a pixel on his retina.

These pixels were thought to find their way through the optic nerve to re-form an inner picture, to be recorded somewhere in the unlit parts of the brain, framed perhaps on a structure corresponding to the retina – a “retina of memory.” From this inner retina, the same picture, a frozen snapshot image, could be elicited and recalled to consciousness years later.

The idea was grounded in the anatomy and physiology of the late 19th century. By 1870, a reasonably good picture of the visual pathway was already on the drawing board, though it was not yet fully detailed or accepted. They knew the pathway began at the retina, traversed the optic nerves and optic chiasma. In the 1890s it became clear that the pathway turned at the lateral geniculate nucleus, and then radiated back toward the occipital lobe, shown here in pink.

So they had the origin of an image, the retina, and they had followed the wires all the way back to their apparent destination: the occipital lobe of the brain – our visual cortex.

In 1888, Salomen Henschen, a Swedish neurologist and Professor of Medicine at Uppsala, published a study of lesions in the human occipital lobe. He had collected and sifted data on such lesions in 160 patients. Each lesion produced blindness in a different and distinct part of the visual field.

Serious damage restricted to the right side of the visual cortex produced blindness on the right halves of both eyes. The phenomenon, called hemianopia, is discussed at the Lighthouse International site. Similarly, important damage to the left side of the visual cortex produced blindness on the left half of both eyes.

In effect, each eye contains two half-retinas. Although the retinas are vertically split (in terms of their wiring) we are completely unaware of the junctures. Henschen concluded that fibers from the two right half-retinas converge and connect to the right visual cortex. Fibers from the two left half-retinas converge and connect to the left visual cortex.
The half-retina to half-brain mnemonic is rights-to-the-right, lefts-to-the-left. This wiring plan, which is now familiar, was first confirmed by Henschen.

Downshift in scale
He then looked at the problem at a much smaller scale, to see if he could generalize from the documented effects of tiny lesions. A nick in the left visual cortex created a small blind spot at corresponding points in the two left-side half-retinas of both eyes. An adjacent tiny lesion in the brain should, Henschen asserted, produce in the visual fields adjacent blind spots. It followed that one should be able to map, point for point, the retinas onto the occipital lobes.

Henschen proceeded to draw a map of the retinas' projections onto the occipital lobes of the brain. In retrospect it seems he got the map backwards, in that the foveal and peripheral fields were flipped, but his concept mattered vastly more than any mistaken detail. (Kort öfversigt af lären om lokalisation I hejernbarken. Uppsala, LäkFören. Forh 1888; 27: 507 and On the visual path and centre. Brain 1893; 16: 170–180).
Henschen was the first scientist to map the retina onto the brain. He thought it was the one and only such map, and that it perfectly replicated in the brain the distribution of receptors in the eye. He named the structure he had mapped "the cortical retina."

There are now more than 30 such maps and representations and re-representations of the retinal field (and the visual field, which is not exactly the same thing) at various sites in the primate brain. In the twentieth century, mapping the visual system in the brain became a core theme -- and from 1950 arguably the core theme -- in vertebrate brain physiology. Here is a thorough 2005 review of the mapping aspects of this work, including recent brain maps and references.

But somehow, Henschen's conception of a "cortical retina" was never quite fulfilled. An image appears on the retina, yes. But does a 1-for-1, pixel-for-pixel map of that retinal image make it back to the cortex? The conventional answer is an emphatic no.

So exactly what happened to the beautiful idea of images mapped into the brain, and of visual memory as a record of these images?

The hopeless bottleneck
The first difficulty with the notion of 1:1 mapping from the photoreceptors to the brain is the bottleneck of the optic nerve.
The retina has about 125 million photoreceptors. The axons of the retinal ganglion cells comprise the optic nerve, but there are only 1.2 to 1.5 million of them. The cell counts are imprecise but it is clearly quite a funnel. It would seem that if one did try to reconstruct an image of the whole retina in the brain in real time, it would have to be at a 100-fold reduction in resolution from that detected by the photoreceptor set.

The bottleneck does not exist for a multichannel neuron. In an optic nerve made up of multichannel neurons, there could be a half-billion distinct channels. More than enough.

But in the conventional view of the nervous system, the bottleneck is indeed an issue. With a 100-to-1 information loss or step-down between the photoreceptors and the visual cortex, how do we manage to see so clearly?

One solution is to remark that although the overall ratio of receptors to optic nerve output lines is 100:1 -- it is just an average. It doesn’t mean the ratio is fixed at 100:1 at all points in the retina. The distribution of retinal ganglion cells is not uniform in the retina. The ganglion cells are densely concentrated near the center of the retina, the area of acute vision, and spread out at the periphery. It can also be urged that the receptive fields of the ganglion cells are smallest at the center, and trend larger as one scans toward the periphery.

So at the center of the primate retina, in the fovea, there could indeed be a 1:1 relationship between cone receptors and output lines. It is therefore possible, in this scheme of things, to transmit a high resolution picture from the fovea, only, to the visual cortex.

Moving out along the radius line of the retina into the peripheral mix of rods and cones, the input to a given ganglion cell expands to include a collection of encircling photo receptors, so resolution drops. Well out in the periphery, single ganglion cells, each serving great clusters of rod receptors -- thousands -- are thought to report huge, blobby pixels. These big pixels might be more accurately characterized as large targets for occasional photons in the dead of night.

In effect this system suggests a privileged cable link inside of the optic nerve, running straight from the foveal cones to the brain. About 50 percent of the optic nerve axons are thought to be pipelining information from the fovea, so there’s 625,000 axons and, thus, picture elements.

Given its typical assumptions about the neuron and the nerve impulse, this story seems plausible enough, at first, but there is a petite problem, or a seed of conflict, at the periphery. If the retina captures the Fourier plane, then the most interesting part of it -- where the most finely grained detail is encoded -- must be detected at the periphery. The textbook approach makes it seem impossible to record this wonderful detail in the fringes at the outermost reaches of the retina.

The idea of a privileged foveal channel encounters deeper problems when you start carving into the channel capacity to allow for parallel transmission of multiple worldviews. In trichromats, for example, color vision can require the parallel transmission, via the optic nerve, of three distinct worldviews, one each from the red, green and blue cone sets. Color is not the only type of information you might want to transmit in parallel via the optic nerve. There are 20 different identifiable types of ganglion cells in the primate retina. If receptive fields are grouped by ganglion type, each type forms a regular lattice, and so it appears there exist 20 parallel systems in the retina capable of originating 20 distinct worldviews for the brain to parse and process. The privileged channel in the optic nerve begins to look extremely cramped for space.

In short, the bottleneck of the optic nerve is not a solved problem. In my view it will never be solved because it is a problem that doesn't exist.

There is by the way another, more recently conceived take on the problem of the bottleneck, which is that it could be overcome by a compression/decompression system that strongly resembles that used in digital television transmission and reception. The idea was probably inspired, in fact, by digital TV. Using compression and decompression maybe it would be possible, even within the conventional model of the nervous system, to resurrect Henschen's idea of 1:1 image mapping of the whole retina -- ever so slowly painted from the retina into the brain.

Hubel and Wiesel
Historically, however, the notion of a cortical retina was pretty much set aside in the 1960s by the celebrated neurophysiologists Hubel and Wiesel. The 19th century idea of using the brain to memorize a snapshot of a literal image on the retina was scrapped and supplanted. The new and very different idea was to let the brain extract and memorize the features of an image.

Feature detection quickly became the basis of a new model of how the visual cortex works – a model that is still widely accepted today. It contains such admirable ideas, and it is based on such extensive and beautifully described experimental work, that it went almost unchallenged for most of the rest of the 20th century. The model is problematical – to be candid I think it is dead wrong. However, it holds such an important place in neuroscience that one cannot spin alternative hypotheses about how the brain might work without first addressing it. Herewith.

“A roar of Impulses…”
In the 1880s Salomen Henschen was what we might call today a data miner. He collected in one place, his office, data that had been observed over time by many neurologists and pathologists on the effects of small lesions in the visual cortex. He drew his map of the “cortical retina” based on his precious collection of facts -- facts originally reported, for the most part, from clinical observations and autopsies performed by many different people. His work was brilliant and meticulous, but far removed from direct experimentation.

In the 20th century, it became possible to actually make electrical measurements from the visual pathway of animals, using probes planted near or in individual neurons, including neurons in the visual cortex. The new instruments were based on fine probes, electronic amplification and display. Often the experimenters used a loudspeaker to broadcast the crackle of nerve impulses as they worked. This made it unnecessary to constantly watch the screen of an oscilloscope.

This thread of research begins with Hartline’s studies of retinal ganglion cells in the late 1930s, and was picked up after the war by Steven Kuffler. Kuffler discovered the target-like geometry of inhibition and stimulation in receptive fields of the retinal ganglion cells of cats. David Hubel and Torsten Wiesel were working in Kuffler’s lab at Johns Hopkins. The Kuffler lab moved to Harvard, and Hubel and Wiesel ultimately became – with Eric Kandel – the most famous neuroscientists of their century. All of these men won Nobel prizes.

Hubel and Wiesel actually did the direct experimental work Henschen must have wished or dreamed he could have done. Torsten Wiesel was born in Uppsala, but it is not clear whether the two youthful collaborators were specifically aware of Henschen’s publications in the 1880s on the “cortical retina.”

A famous, rather incidental discovery
Their original experimental concept was to stimulate the retina of a cat with points of light (or points of darkness) while probing with an electrode in the visual cortex for a response – a burst of spikes. It was basically the same experimental approach Kuffler had taken in exploring the receptive fields of retinal ganglion cells, and they started out using Kuffler’s apparatus, a multibeam ophthalmoscope which included two extra beams for stimulating the retina.

But Kuffler had looked at spike trains on ganglion cells, which are the output neurons from the retina. Hubel and Wiesel were probing for responses far, far down the line of the visual pathway, at the back of the cat’s brain in the visual cortex.

Kuffler’s apparatus included a light beam projected through a small hole in a brass plate. The plate was the size and shape of a microscope slide, and was secured in a slot. To deliver the obverse stimulus, a black spot in a light field, the experimenter could remove the brass plate and replace it with a glass microscope slide, onto which an opaque black dot had been pasted.

It was the microscope slide that produced the most interesting response from a neuron in the visual cortex – but curiously, the black dot had nothing to do with it. The neuron’s response was quite specifically associated with the act of installing the slide in its slot. The experimenters eventually guessed that the edge of the moving slide was casting a "shadow" onto the retina. As the slide’s edge moved, the neuron responded. It emerged, moreover, that the neuron was “orientation selective.” The neuron gave a maximal response only when the edge of the slide was oriented at a certain, just-right angle. When the edge of the slide was oriented at precisely this angle and moved across the retina, the neuron would emit “a roar of impulses.”

Here is a video on YouTube that shows a re-enactment of the experiment. Be sure to watch it with the sound turned on, so you can hear the spike trains on the loudspeaker. Another YouTube video shows a similar experiment, using a bar of light in various angular orientations. The video shows the cat's point of view. Fast forward to the middle of the video. Note that the eye is immobilized, so the bar must be in motion to elicit a response.

In later experiments Hubel and Wiesel discarded the multibeam opthalmoscope and simply projected onto a screen images of lines, edges, bars, and the like. These figures could be moved and rotated. It was simplicity itself. Basically they were showing pictures to a cat while monitoring the reaction of neurons in its visual cortex. The first video is probably taken from this later period, and the image of the edge of the microscope slide was intercut to help retell the story.

There is a good account of this discovery in the widely used undergraduate textbook, Neuroscience: Exploring the Brain . You can find the same anecdote in several other places: In Hubel and Wiesel’s Brain and Visual Perception it appears on page 60. In this version the “cell seemed to come to life and fired impulses like a machine gun.” There is a helpful detail about the angular response. For example, if the edge is oriented at right angles to the optimal position, the signal actually goes dead. The Journal of Physiology paper (1959) 148, 574-591 that first mentions this work is reproduced from page 67. There is another account beginning on page 69 of Hubel’s Eye, Brain and Vision. Finally, the story is told in the context of subsequent research in Hubel’s Nobel acceptance speech of 1981.

The Edge Detector
It is the angular orientation of the slide’s leading edge that matters. The line or edge presented to the retina can be moved in translation any which way without affecting the pulse train response of the monitored cell. But as the angle of the edge is varied, the pulse stream from the neuron in the visual cortex speeds up or slows down. At one specific angle of orientation, the neuron’s firing rate will go wild.

In this animation the important response occurs at angles near 0 and 180 degrees. This is just a metaphor at this point in the narrative, but it suggests physical optical effects might underlie the experimental results.

A vigorous high frequency pulse train response from the cortical neuron was interpreted as a response to a high intensity stimulus of an orientation-selective cell. In effect the cortical neuron was thought to be shouting, "Yes -- that's 41 degrees exactly!" A mild response, or low frequency pulse train was read as a response to a low intensity stimulus -- that is, a neuron reporting that the detected line, or edge, is not yet turned to the just-right angle.

Different cells were found in the visual cortex that responded to different edge angles. It appeared in fact that these cells, now sometimes styled as “edge detectors”, had each been pre-tuned to trigger on an edge presented to the retina at a specific angle.

It was suggested that the ability of a neuron to discern a specific angle was conferred by a configuration of the intricate patterns of center-surround inhibition and sensitivity originally discovered by Kuffler.

Once the angle of maximum response had been determined for a given neuron, Hubel and Wiesel could change the stimulus input to the eye by rotating the line or edge projected onto the retina. As the line was rotated away from the ideal angle, the pulse stream on the responding neuron would die down. But if they changed neurons, by advancing the needle of the probe through the tissue of the cortex, they could find another neuron nearby, maximally responsive to the new angular position of the edge stimulator.

The generalization to feature detectors
The discovery of "orientation selective" neurons in the visual cortex might seem obscure, but it led very quickly to a completely new view of how the brain works -- and indeed it became the dominant view of how the brain works.

From the discovery it seemed to follow, in general, that the visual cortex contained cells designed to detect the angular orientation of lines or edges associated with objects in the animal’s field of view. These experiments with orientation selective neurons engendered the concept of a “feature detector.” From the image projected on the retina, the brain was, per this line of research and thinking, extracting information about the edges of objects, their angular orientation. Differential responses to their direction of motion were also noted.

In other words, it appeared the image on the retina was not being recorded frame by frame, like a movie of the world, in the visual cortex. Instead, the retinal image was being speedily dissected into a set of useful abstractions. Where does the object begin? Where does it stop? How fast is it going, and in which direction?

Since the abstract information provided by the edge detector nerve cell was pretty primitive, it followed that there must be an ascending series of brain components, assembling the incoming abstractions into a useful set of specifications that could indeed be memorized and, subsequently, recognized. Think about an egg. Such an object can be assembled – integrated, in effect – from many short line segments, each segment characterized by a specific angular orientation. A collection of orientation sensitive neurons, linked perhaps with a logic rather like an array of AND gates, could recognize and report to higher centers the quality I suppose we might call Eggness.

The idea of a hierarchy of abstractors, each recognizing an important feature and pushing it upstairs to a higher stratum of the brain to form an element of some higher abstraction – became enormously influential. The experiments were simple and the results had verisimilitude. We expect our brains to manufacture abstractions. It is what the brain is supposed to do.

The ineluctable grandmother cell
At the apex of this hierarchy in the brain, it was ultimately proposed that there should be a “grandmother cell,” that is, a cell which would fire only when the image of your grandmother’s face is projected onto the retina

Per this brain model the actual, literal image of your grandmother, pixel for pixel, was never preserved. In fact it was left far, far behind – a momentary visual episode that had flashed upon the retina, and probably never made it past the lateral geniculate nucleus.

What was conserved and memorized instead was a collection of abstracted facts and qualities: an assembly of edges, a yellowness of teeth perhaps, a grayness of hairs, a mannerism. When the actual grandmother reappeared as a live image impressed upon the retina, all these “features” would be recognized by individual feature detecting neurons. Their outputs would converge, as though via AND gates, to the grandmother cell. And the grandmother cell would signal recognition (to something – the CPU one imagines) with an energetic firing of impulses.

Here is a link to a quick history of grandmother cell by Charles Gross at Princeton. The author explains how it was elaborated as a concept, taking Hubel and Wiesel's work as a starting point.

The grandmother cell was probably the high watermark of this line of thought about how the brain works. David Rose outlined the main difficulties with the idea in an excellent short essay, here.

Many people got off the train because of the grandmother cell. One problem was that it sounds like a joke, and it is impossible to resist making more jokes when talking about it. The principle could not be decisively confirmed experimentally, though not for want of trying. Yet another problem was the rising promise, in the 1980s and 1990s, of neural nets. The idea that any one cell could do anything in isolation, at the top of a logical pyramid or anywhere else, let alone pick out your grandmother, doesn’t nicely fit the neural net model, which requires ensembles of connected cells to form a memory or decision.

Although the grandmother cell seems to have lost favor, the basic notions of feature detection, abstraction from imagery, and orientation selectivity by cortical neurons are still well accepted and studied today using (in the brain) rather sophisticated techniques. There is a helpful review by Robert Shapley and Dario Ringach in Chapter 17 of The New Cognitive Neurosciences. For a more contemporary overview of work in this field, search on these authors and read their recent papers.

The idea that the brain deconstructs the retinal image followed from the early experiments of Hubel and Wiesel, and so we should concentrate here on their experimental methods and assumptions.

The Sparse Code
It is often repeated in texts and in historical accounts of neuroscience that diffuse light cannot stimulate the retina in such a way as to produce significant signals in the visual cortex. This apparent lack of response greatly frustrated early researchers.

Richard Jung, in Germany in the 1950s, devised precise instrumentation to measure the responses of nerves in the visual cortex, but he used diffuse light to stimulate the eye. He found little or nothing. Before Hubel and Wiesel, many other experimenters reported the same difficulty.

By “no important response” to diffuse light, the historians mean that diffuse light produced no vigorous spike streams. The difficulty is invariably explained by pointing out that diffuse light produces simultaneous inhibition and stimulation. The net effect, when measured as the firing rate of a cortical neuron, is a negligible response.

But today, knowing what we know now, we should probably ask -- just how negligible was it? In the 1950s experimenters were not much interested in the passage of just one or two spikes. Not until the early 1990s did we learn that one or two spikes can convey an important signal. It now appears a “sparse code” – just one or two spikes -- is able to convey identically the same meaning as a long, loud and vigorous spike stream.

When should you look for a sparse code? When should you look for a rate code? Is it like shorthand vs. longhand? If these codes are equally able to convey meaning, then why should the nervous system use one rather than another? Or why should it use one code sometimes, and other codes at other times? In fact, given the energy cost, why should it ever use a rate code?

Interesting questions.

In Chapter 2 it is suggested that a sparse code indicates the neuron is reporting on a stimulus that falls within its normal operating range. The rapid-fire rate code appears only when the neuron is overdriven, that is, hit with a stimulus that exceeds its normal operating range. A runaway spike stream has the same significance as a pinned meter.

In this model system there are two different types of spike streams. These firing patterns are different in kind. We do not possess an instrument that can readily distinguish between them, but it is a fairly easy judgement call. If the stimulus is extreme, it will elicit a rate code. If the stimulus is typical, it will be tracked by a sparse code.

In a sparse coded spike stream, if the stimulus is not changing, or is slowly drifting, an occasional spike or two will be enough to characterize the magnitude of the stimulus. If the stimulus changes markedly, a spike stream will precisely describe the change as it happens, point for point, but the stream will last only until the stimulus stabilizes at some new value. The last spike in a stream always describes the most recent state of the stimulus.

A burst of impulses that suddenly stops could be interpreted in terms of the rate code model as an adaptation. In terms of the multichannel model, it means the stimulus has stopped changing.

In an Adrian or rate coded spike stream, the stimulus exceeds the normal (adapted) operating range of the neuron. The neuron, in effect, breaks into oscillation, and produces "machine gun" firing. Until 1995, this type of vigorous spike stream was an unquestioned goal for experimenters, because it seems to signify a recognition or response. But in the context of the model developed in Chapter 2, it means the nerve has ceased to communicate.

This is a speculative model, but I think most people can agree that an extremely rapid-fire pulse stream would be produced by a neuron that is sensing a very strong stimulus, and that the neuron has not yet managed to adapt.

50 years later...
What do Hubel and Wiesel’s feature detectors mean in 2011? The results of their experiments are by now woven through the fabric of theoretical neuroscience. The received wisdom about how the visual pathway in the brain works is still grounded on these results: The brain does not store and recall images. It stores and recognizes the abstracted features of images. The nervous system is hardwired to perform the necessary abstractions.

Or so it appeared. Hubel and Wiesel’s results are not as clear cut today as they were in the 60s, 70s and 80s. The main issue is the failure, around 1995, of Adrian’s rate code. Secondary issues arise from the physical optics of their experiment.

Note that the experimenters were using their ears, relying heavily on the crackling loudspeaker output you can hear on the video. They were looking for big responses and they were elated to find them: A “roar of impulses,” or “machine gun” fire.

They will have perforce ignored what could be the normal traffic in impulses from the retina – a pulse or two in passage every now and then. We will surmise here that what they found were signals that represented a response to a stimulus that exceeded the normal operating range of the neuron -- that a “roar of impulses” is the extreme response of an outrageously overdriven neuron.

What makes a neuron roar?
What sort of light pattern on the retina would be predicted to make a neuron roar?

An interference band. Simple edge diffraction projects alternating bands of bright light and deep darkness. These alternating bands, in turn, can play heavily upon the peculiar contrast sensitivities of the retina.
The bands are an almost ideal stimulus: The brightest of the bright alternating with the blackest of the black. If light interference bands were projected onto and moved across the inhibitory and stimulatory sensitivities which, as Kuffler discovered, are built into the retina -- the cortical neuron should respond with a perfect storm of impulses.

So possibly a storm of impulses produced by a potent and exceptional stimulus -- a moving interference band projected by edge diffraction -- was assumed to be the typical, normal, everyday, response from a neuron in the visual cortex.

Orientation selection by means of physical optics
Is there also a light pattern that could produce the appearance of orientation selectivity? It would require a light pattern that would change in a detectable way if the angle of presentation of some object were changed. Are there such patterns? Yes. Superimposed interference bands, one a rotor and the other a stator -- a moiré on the retina – would certainly do this.

The principle is the basis of one type of angular position detector, in which the position of a bright spot moves in translation when the angle between the rotor and stator bands changes. Visualize the bright spot as the point of intersection of a pair of scissors. When one blade is rotated and the other held fixed, the bright spot shifts in position.

By analogy, if a projected interference band associated with diffraction from a line or bar or slot or slit were turned relative to a static interference band in the eye, then a bright spot where the bands intersect would move in translation across the retina. But instead of scissors with two blades, we have many bands, many intersections, a constellation of shifting bright spots.

This could give the appearance that a single neuron with a receptive field localized at a particular point in retina was able to selectively detect a specific angle.

Here is a photo of sunspots. Let’s guess that the orientation-selective neuron was, in effect, looking straight at a sunspot on the retinal surface. Very bright spots could arise at the intersections of interference bands. There would be a constellation of spots forming runnels of light. The receptive field of an instrumented cortical neuron, however, will include only one or a few of these hotspots.

By turning the microscope slide to a certain angle, and then advancing the edge of the slide in translation, a runnel of sunspots could be brought into register with the receptive field of the neuron under study. If the slide is then turned just a little, the signal on the neuron will increase or decrease as a spot moves in and out of register with the RF. If the slide is turned dramatically, the sunspot will be shifted in translation to the receptive field of a different cortical neuron.

In this model, the cartesian position on the retina of the receptive field of the cortical neuron, and the specific patterns of the rotor and stator -- predestine that neuron to respond most strongly to a particular angle of the microscope slide. This is one way it could work, so it is a place to start, but the scissor geometry is too simple.

Bear in mind that for a neuron in the visual cortex, a hotspot or sunspot -- a source of extreme stimulus -- is likely to be a moving, banded pattern of light and dark, not just a central bright spot. Note too that with interference bands that are converging or diverging, curved and straight, underlying the moiré pattern, directional effects will also be observed.

Here are several examples of interactive moirés. In this selection I am not finding a rotor/stator pattern that seems right for the cat's eye. I would favor a straight or slightly curved rotor and a more dramatically curved stator. But everyone has a favorite animation and here's mine. This one uses two identical zone plates. I think it shows in a succinct way what an amazing bag of optical tricks could confound the experimenter in this seemingly straightforward system. Here is yet another pattern, a real beauty. Click and drag.

In this interpretation, the experimental apparatus supplies the interference bands that constitute the rotor. The stator is a fixed pattern produced in the open, immobilized eye. Or, since a cat's eye contains a mirror, there might be more than one rotor at work.

When the experimental apparatus is taken away, the cortical neuron might lose its talent for roaring at the sight of a specific angle. It is the "roar" of course that makes it seem likely the stimulus is abnormal, outside the typical operating range.

Pro and con
One could conjure with the idea that physical optics might help explain, rather than controvert, the notion of orientation selective neurons. Have at it. But note that the optical system is more stabile and precise in the lab than it is in the world. The eye is completely immobilized and dilated with atropine in the experiment. In a free ranging animal, the eye and optical system would be in motion constantly. Thus, the "stator" pattern would also be mobilized. It is not easy to see how a precise angular measuring system could made to work with both the rotor and the stator components in constant flux.

On the other hand, diffraction effects are inescapable, and double diffraction is the basis of imaging in the eye. The experimental set up would probably exaggerate edge diffraction effects, but there is not enough information in hand to state that the experiment produced effects inside the eye that are not physiological.

Still, the physical optical basis for detecting the angular orientation of a line, edge or bar is, according to the scissors model for example, nothing more than a bright spot or hotspot on the retina that moves in translation when the bar is rotated.

Would the brain be well advised to attribute -- to a moving hotspot -- a rotation in the presentation angle of some object in the world? Such as an oncoming truck? Lots of other events in the imaged world could shift a hotspot from point A to point B. Trains go by. Headlights sparkle on the highway. The world is filled with moving points of light, and banded diffraction patterns of light and darkness.

An optical Illusion?
The moiré pattern interpretation suggests the prototypical “feature detector” might have been, in effect, an optical illusion. In other words it is reasonable to suspect it was the physical optics of the experiment -- and not a cortical neuron specialized for the task of discerning, say, a 41-degree angle -- that produced this famous result.

Light in the system is not coherent, but it is partially coherent, so it produces banding that is instantaneous rather than visible. But the bands are in there.

To sort it out one might as a first step try subtracting the cat and its brain from the experiment, just to see if the results might be reproduced physically. One could use coherent light, along with an instrumented mock-up or preparation of the eye and the rest of the optical system. To visualize the system, one could use Rhino 3D and write plug-ins for the coherent light optics.

The physical optics of the eye in this experimental system are anything but simple. The cat has a concentric multifocal lens and a slit pupil, though the pupil is dilated. In the cat experiment light is projected past a barrier (edge diffracton), through an aperture containing the multifocal lens, onto the interior surface of an ellipsoid. The backwall of a cat’s eye is a curved mirror. The retina is a fiberoptic bundle. A lot can happen. In addition to optical effects arising from the tapetal mirror built into cat’s eye, and mirroring effects in the eyes of primates that lack a tapetum, there may be singularities, modes, and other complexities.

There are experimentally confirmed Fourier optical effects. It was suggested at the time that the neurons in visual cortex were responding to features in the frequency domain, rather than to the literal images of lines and bars. This research was published by the late Russell De Valois and Karen De Valois. The De Valois lab measured responses in the brain to the presentation of more sophisticated stimuli than lines and edges. Notably they used gratings and checkerboard patterns of varying spatial frequency. The effort is recounted in their book, Spatial Vision, published by Oxford in 1990. They probably came closer than anyone else at that time to understanding the effects Hubel and Wiesel made famous.

However, this work was also done before 1995, and it too assumed the validity of Adrian’s code. Experimentally this means they were, like Hubel and Wiesel, selecting for and giving weight to responses to extreme stimuli, and not picking up on the sparse coded responses that were unknown at the time.

It is still common, in 2011, to observe large neuroscientific enterprises grounded upon on the very energetic firing of some one particular neuron, and of collections or arrays of neurons. In Edgar Adrian's world, a high frequency pulse stream was easy to interpret, but what does it mean today?

Nobody knows.

How would a neuron say: "I remember that."
In my view, sustained rapid firing means that the neuron has been hit by a novel stimulus that is outside its normal operating range. The stimulus could be huge or it could be tiny or very negative, but it is extreme.

The machine gun neuron is not remembering or recognizing anything. It is encountering something new and excessive. A stimulus that had been encountered before should fall within the neuron's normal operating range, because the range will have been expanded to accommodate it. Nerves adapt.

Then what sort of firing pattern could signify a "memory", or recognition response, to a stimulus seen before? A faithfully accurate report on the magnitude of the same stimulus, when it is encountered a second time by the same neuron, could be conveyed by one or two spikes.

Now how would you detect with a simple probe a cell that "recognizes" some familiar stimulus? It appears to me to be impossible. There is nothing exceptional or egregious about the passage of one or two spikes. If indeed a pair of spikes is required, the two might not even be on the same neuron. And if one spike is enough, then which one? They all look exactly alike to a probe.

Conclusion: The movie of life...
Are there orientation selective neurons in the visual cortex? It is still possible of course. A hierarchy or network-based set of feature detectors? We probably lost feature detectors when we lost Adrian’s code. Grandmother cells? No takers here.

The idea that the brain deconstructs into a set of abstractions the salient features of an image, and shrugs off the image itself -- seems less and less convincing.

Even if the physical optical problem were set aside, the model should be questioned. One cannot model a brain using feature-extracting neurons or nets without a basic understanding of how a neuron communicates. When we lost that understanding in the early 90s, the model went soft.

Today we are back where we started. We are free to reset the rules of the game. Pictures are okay again. We are no longer constrained by a 1960s brain model that trivialized images and imagery in favor of abstractions.

Henschen’s cortical retina, that marvelous relic of 19th century neurology, has come down to us intact in the 21st . We are once again free to imagine, as did Salomen Henschen and Jerome K. Jerome, that the brain retains and operates upon retinal imagery – a scrapbook or a movie of life. The movie could be recorded in the frequency domain or the spatial domain or both, but it is a procession of images in any case.

Most of the jobs (like edge detection) that were supposed for decades to be done piecemeal by feature detectors can probably be accomplished all at once by Fourier filtering. Fourier filtering is also an excellent engine for a thinking and remembering machine, as van Heerden observed. Such a machine is fueled by images flowing in from the world and by images flowing out of memory.

But to process an image by any means, you must have an image to operate upon. How, then, might an image be retained in the brain?

<<PREVIOUS CHAPTER NEXT CHAPTER>>

2008-01-01T13:13:00.000-08:00

#8
Rods and cones as wave detectors

The photomultiplier tube was invented in the summer of 1930 in the Soviet Union by L.A. Kubetsky. It is easy to see from the photo why an analogy between the photoreceptor cell and this splendid antique technology is so often drawn. However, the analogy is not very strong, and it tends to make the photoreceptor story seem shorter and more succinct than it is.

Particles, rays and waves
A quantum of blue-green (500 nm) light packs quite a lot of chemical energy -- 46 kcal/mole. Incoming photons can thus be conceived as vigorous potential reactants with the pigments: chlorophyll, phytochrome, the flavins and retinal. This physical biochemical perspective -- in which light is styled as an energetic particle that might or might not strike and react with a pigment molecule target -- underlies the commonplace view of the photoreceptor as a photon detector.

There are two other ways biologists regard light: as rays and as waves. The ray approach is typical and serviceable, but the wave description of light is essential in understanding diffraction, double diffraction image formation, natural mirrors based on interference, and structural color.

In the specific study of photoreceptors, the light-as-a-wave approach has chiefly been taken by investigators of the waveguide (fiberoptic) properties of these cells, and by the standing wave theorists. Let’s consider here what it would take to make the photoreceptors operate as standing wave detectors.

Particle detector versus wave detector
In the conventional view the photoreceptor is a one-channel analog device. It is a particle detector -- it triggers on photons.

Here is how, in the retinal rod, the particle detector is thought to work. In darkness, the plasma membrane sodium channels are open, the membrane is depolarized, and at the output synapse the cell drips neurotransmitter like a constantly dripping faucet. Now admit a little light. When photons hit the pigment in disks, a biochemical cascade ensues that ends by closing the photoreceptor cell membrane’s ion channels. (These ion channels are for the most part ligand activated, rather than voltage activated.)

Thus, in response to light, the plasma membrane hyperpolarizes, that is, the transmembrane potential increases. Typical membrane potentials recorded from photoreceptors in the dark are about -30 mV. In response to bright light the cell may hyperpolarize by 20 to 30 mV to about -60 mV. Note that this is a value close to the resting potential of most neurons.

The transmembrane voltage change at the outer segment is “felt” all the way out at the ribbon synapse at the end of the inner segment, and at this synapse the dripping neurotransmitter faucet is tightened up. In understanding the system it is helpful to regard the neurotranmitter as inhibitory: Reduced inhibition => stimulation. In fact it isn't consistently inhibitory, but it helps to think of it that way in a first pass.

Note that this is an utterly analog system. There are analog degrees of membrane hyperpolarization, dialed up and down by the intensity of the impinging light. It follows that while the system is within its operating range, not all of the ion channels are closed. In light, the synaptic faucet continues to drip neurotransmitter (glutamate), but it drips much less than it did in darkness. As the light gradually strengthens, more channels close, the transmembrane potential rises, and the synaptic output of dispensed neurotransmitter gradually decreases. Where that neurotransmitter is inhibitory, downstream cells are un-inhibited --stimulated. More photons, fewer drips, less inhibition, more downstream stimulus.

How could we possibly detect waves with this thing?
As a first step, the system needs a mirror to reflect incoming light waves back on themselves, and thus create a measurable standing wave. Several candidate mirrors have been suggested in the literature. In cone cells a mirror effect can be physically demonstrated. It seems probable that a rod cell mirror exists as well. Types of biological mirrors will be discussed in subsequent paragraphs.

The second, more challenging problem is one of keeping individual disks' output signals straight. The photoreceptor response is, in textbooks, completely melded and generalized – a whole-cell response to impinging photons. The response is amplified in rod cells by a biochemical multiplication effect somewhat analogous (I would say slightly analogous) to the electronic avalanche in a photomultiplier tube. The voltage change at the plasma membrane is felt at the synapse, but there is no way to dissect the summed voltage signal. By this I mean there is no way to parse and trace the source of a signal back to any single disk. The individual disks' signals have been magnified and pooled, and so the signal source is essentially the whole outer segment.

In this variably hyperpolarized system, how could we arrange to “wire” each disk in order to detect localized light intensities, e.g., standing wave peaks?

And how could we signal the detected result to the ribbon synapse? Where can we run the necessary cable?

There are two conduits available to send discrete biochemical signals from the disks toward the ribbon synapse: internal and external. The internal pathways could run through aligned incisures in the disks. I would be inclined to reserve these for communication or processing within the outer segment, but this is mainly because I lean toward the solution offered by the external pathways.

The external pathways are, theoretically, longitudinal channels in the plasma membrane, those of a multichannel neuron based on linked transmembrane sodium channels, and they are essentially what this blog is about.

In this micrograph, the "pm" signifies the plasma membrane.
To localize the response to each disk, it will be necessary as a first step to draw a line linking each disk to a corresponding point on the plasma membrane. In the cone there is no problem. Disk and wall are continuous. In the rod, given the proximity of the rim of each disk and the plasma membrane, some link or localized biochemical bridge is a realistic first assumption.

But how can we get a signal from a local point on the plasma membrane, opposite a disk, all the way out to the synapse?

Suppose we stick very closely to the linked sodium channel hypothesis described in Chapter 2, the Corduroy Neuron. Recall that in the model of the multichannel axon, the sodium channels are re-conceived as 300 or 400 linked continuous, longitudinal channels. The linked sodium channels were conceived as a model device to convey, and impart meaning to, action potentials. In the context of an analog photoreceptor neuron, about all we can say is that the hypothetical linkage of sodium channels provides a way to draw a continuous line, or channel, from a point on the plasma membrane opposite the rim of a disk -- to a point as distant as the ribbon synapse.

Action potentials in rods and cones. But why?
Drawing a line isn’t enough. The channels must operate. The simplest and most typical operation would produce an action potential. There is another way to make the system work, discussed below, but let’s think about action potentials for a moment.

There is actually a nice body of literature on the subject of action potentials detected with patch clamps in vertebrate photoreceptors. It is thought that in lower vertebrates these action potentials depend upon calcium channels, but it was discovered in 2001 that in the human retinal rod, action potentials can arise quite conventionally from the operation of voltage-gated sodium channels. (In light of this discovery it was wondered whether in lower vertebrates the involvement of Na channels, as opposed to Ca channels, in the production of action potentials might have been missed.)

In humans, the voltage-gated Na channels and action potentials have now been demonstrated in both rods and cones.

Amacrine cells have also been shown to fire action potentials.

In other words, superimposed on the neat textbook story about the photoreceptor cell as an analog transduction system and photomultiplier, there is this other, isn’t-it-curious sidebar story, in which action potentials might figure in some obscure way in the operation of photoreceptor cells.

Night and day
Possibly what we are looking at here is a day/night divide. The rod photoreceptor could be like a computer with two different operating systems. Or like a photosynthetic organism, equipped with both a day biochemistry and night biochemistry. There would be many pathways in common, useful in either mode, but let's guess it would take a major biochemical switch to get from day mode to night mode. A light switch.

The textbook, analog photoreceptor story is essentially a description of, and extrapolation from, the nocturnal rod's operation as a photon detector and photomultiplier. This is an excellent system in dim light. None of the standing wave receptor theories is going to work in the dead of night. In darkness, a depolarized rod membrane could not possibly support an action potential.

At the other extreme, in broad daylight you would expect photomultipliers to saturate and ultimately shut down, and standing wave sensors to wake up and come into their own. Along with a daytime detection system, you might expect to see a distinct daytime rod biochemistry. Photoreceptor membranes would be hyperpolarized to the level of typical neurons' resting potentials. And action potentials could thus become the favored mode of communication.

On a multichannel photoreceptor, the action potentials would rarely need to appear. Maybe once every time the eye shakes. But they would give us a way to get standing wave information out of the photoreceptor disks. Assuming some internal pre-processing of the photoreceptor (e.g., peak picking) it should be possible to communicate intensity and wavelength with just two or three spikes. A photoreceptor of this type is modeled in Chapter 12, in the discussion of the role of photoreceptors in memory.

How to do without with action potentials

The outer segment of a photoreceptor is analogous to the dendrites of a less exotic neuron. If action potentials are to appear here, they would have to be called dendritic spikes. It would seem more realistic to postpone the appearance of action potentials to a downstream site, the photoreceptor’s nominal axon. However, the model requires some means to convey signals, via multiple channels, from each disk in the outer segment.

Na+ channels as a row of dominos.

In Chapter 2, in the discussion of myelinated nerves, we introduced the concept of a signal that moves along a neuron simply through the successive displacement of Na+ channel voltage sensors. The signal can be characterized as a wavefront of conformational change.

In the myelinated internode the sodium channels are inaccessible to sodium ions, and so the signal is perforce unaccompanied by action potentials. This type of signal might be detectable in principle, but in 2012 it could probably not be detected with any common instrument.

The signal is initially enabled by a sharp positive shift in the normally negative interior of the axon. The voltage sensors of the sodium channels, which are positively charged protein, are impelled outward (normal to the axolemma) by this enabling positive charge.

In the model, the voltage sensors are restrained from moving in unison by latches. The latch of each voltage sensor can only be released by the movement of the preceding voltage sensor. Accordingly, once a signal has been launched, the displacement of voltage sensors proceeds from the first unit Na+ channel in line to the next. The wavefront proceeds down the line of voltage sensors as though they were a row of falling dominos or, perhaps more aptly, popping corks.

In myelinated nerves, the initializing positive voltage shift is provided by the firing of action potentials at the nodes of Ranvier. Suppose we were to try to overlay this process on the outer segment of a photoreceptor. What could provide the essential enabling positive charge? Possibly the disks.

Disks form at the base of the outer segment by an outpouching of the cell membrane, as shown at a, b and c in the drawing. (Adopted from Steinberg et al, 1980). The green lines indicate the future interior of the disks. Suppose there were sodium pumps in the cell membrane. As the cell membrane is transformed into disk membrane, the sodium pumps are, in effect, flipped. In other words, they will actively pump any available Na+ into the disk instead of out of the cell. The rims of each disk could, by this means, be given a strong positive charge, positioned directly underneath the Na+ channel voltage sensors.

This is electronically similar to the enabling positive voltage found in the internode of a myelinated nerve, but there is an important difference.

In the internode, the voltage declines in a gradient as we move futher and further along the axon from its source: the positive ion current injection accompanying action potentials at the node.

In the photoreceptor, the positive charge is produced in each disk at the expense of ATP and it is solidly positive all along the length of the disk stack in the outer segment.

Here is a means to silently transmit a signal from each disk. The signal is "silent" in the sense that it does not create action potentials. In Chapter 2, we characterized this signal as a stealth impulse. The signal would be carried and energized, as in an internode, by a wave of successively ejecting voltage sensors. To avoid producing an action potential, which seems like a waste of time and energy in this context, the sodium channels would have to be inactivated in some way, since these cells, unlike myelinated internodes, do have access to extracellular sodium ions.

So there‘s a sketch of the basic idea: A line of successively unlatching voltage sensors carries the signal without inducing action potentials.

There are lots of questions here, including how (in detail) to launch a signal from a specific disk. Note too that the positive charges are close to the voltage sensors. If these charges are unrelenting, then how do the sensors recover after a channel is fired? The sensors must re-latch to recover. This suggests a periodic wave of negativity may pass down the outer segment to re-cock the sensors en masse.

Beyond the disk stack, the stealth impulse could no longer borrow its power from the positively charged disks. To create the necessary internal positive charge in the neuron, it would be necessary to proceed in the familiar way, that is, open the sodium channels and fire an action potential.

The ribbon synapse
At the output end of the photoreceptor is a ribbon synapse. This special type of synapse is found rods, cones, and bipolar cells. It is also found in the ear, in the cochlear hair cells and in the vestibular organ receptors. (The ear is more recently evolved than the eye. The resemblance of the two sensory systems, eye and ear, has often encouraged brain theorists to guess that the auditory system evolved following a pattern already established with the eye -- and that whatever it is that the eye does, the ear probably does it too.)

Photoreceptor cells equipped with ribbon synapses release transmitter constantly, and alter the rate of release in response to small graded changes in potential, rather than in response to action potentials. There could be a rather more complicated story here, as noted above, but this is our present understanding of this complex device.

A ribbon synapse, after Dieck and Brandstätter. Ribbon synapses have different exocytotic machinery from that of conventional synapses. The synapse exhibits a dense bar or “ribbon” which is anchored like a balloon on a string to the presynaptic membrane. The bar is associated with synaptic vesicles as shown.

It has been proposed that the ribbon synapse operates in such a way as to shuttle synaptic vesicles to exocytotic sites. Here is another helpful review of ribbon synapse research.

The ribbon synapse is the logical centerpiece of the whole problem of the retina. To make sense of the ribbon synapse in terms of a multichannel nervous system, one would as a first step interpret the ribbon synapse as a cable connector.

Where we are.
We don't really have a model for a wave detector photoreceptor here. We have enough components of a model to make it seem wave detection in photoreceptors is a possible mode of operation.

An easy tranche would be to declare the cones are wave detectors while the rods are particle detectors. Maybe a little too easy. The particle detector is a specialized adaptation to a nocturnal life style -- nature's gift to bush babies. The photon detector probably evolved as an extreme specialization of a wave detector. In other words, it appears the wave detector came first, and that its machinery is inherent in both rods and cones. Hence the notion of a day/night mode switch.

What has changed, conceptually? We know rods work in daylight. Our periferal vision depends on them. We have proposed here a particular daytime mode of operation, as a wave detector, to the rod cells. We are regarding cone cells as wave detectors rather than particle detectors. The cones are still restricted, as in the conventional view, to daylight operation. And the rods, working as particle detectors, can still count solitary photons on a moonless night.

We are also suggesting a day/night knife switch or biochemical shunt. It could alter the operation of a rod, or it could imply the existence of day-rods and night-rods. The day rod would be a wave detector. The night-rod would be a particle detector.

We have arrived at the day/night dichotomy in a roundabout way, tinkering with this and that -- but it is a standalone idea, and depends on none of these theories. We have the commonplace example of plant photochemistry, which is radically different in the dark and in the light. In this scheme of things, it might turn out that some of our interpretations of ganglion spikes could reflect the experimenters' unintended manipulation of day/night switches.

I am grateful to Gerald Huth for emphasizing and drawing my attention, on his site, to the distinctions between particle detectors and wave detectors in photoreceptors. Dr. Huth invented the silicon avalanche detector, a solid state technology that has supplanted the photomultiplier tube in many applications.

The essential mirror
From the abundance and variety of standing wave photoreceptor hypotheses, it seems clear that we might indeed be looking at the parts and pieces of a standing wave measuring system, an electronic wavelength or perhaps even phase detector. But one must begin at the beginning, and the gate question immediately arises -- are standing waves even possible in this system?
To create a standing wave inside a photoreceptor the incoming wave must be reflected back on its path, and this requires a biological mirror. Fortunately mirrors are ubiquitous in nature, and they are especially common in eyes. Examples of natural mirrors include the mirrored flanks of fish, cats eyes in the headlights, and the iridescent wings of butterflies, moths, peacocks, and birds of paradise.

Eyes that form images using mirrors instead of lenses also exist, for example in the scallop. Just as we might elect to make a telescope from a mirror rather than a lens, so too can nature choose to build an eye using reflecting, rather than refracting optics.

Some observers think our own eye probably evolved from a more primitive eye that used a mirror, rather than a lens, to form images.

A striking example of a natural mirror is the pupa of the butterfly Euploea core. It is impressive to us because its appearance is so like that of a man-made mirror. The surface seems metallic, like a gold or silvered glass Christmas ornament. In forest light the pupa so faithfully mirrors the leaf from which it depends and the surrounding foliage that it essentially disappears. The mechanism of this mirror was studied by R. A. Steinbrecht, W. Mohren, H. K. Pulker, D. Schneider and reported in a classic paper in Proceedings of the Royal Society of London.

Photo by Viren Vaz.
The pupa of the danaid butterfly Euploea core.

The most strongly reflective natural mirrors work according to the principle of thin film interference, and are made up of multiple fine layers spaced at precisely specified intervals. However, some natural reflectors are just polished optical surfaces, offering simple specular reflection, like Bruch’s membrane or the surfaces of the lens or the cornea. We know that a reflection can occur at the interface between two materials with different refractive indices. For example, a reflection can occur at the interface between a photoreceptor cell and its extracellular surround. Similarly, a reflection can occur within the photoreceptor cell – that is, reflection can occur at the interfaces between the disks and the intracellular fluid.

As we begin to itemize reflective surfaces within the eye and, perhaps especially, within the individual photoreceptors, things start to get overcomplicated. To order the problem, let’s sort the mirrors by the two major planes in which they can reverse light. We will begin with the longitudinal mirrors and then consider transverse mirrors.

In the hall of mirrors...
In the various standing wave color vision hypotheses reviewed and discussed above, the effect of their essential orthogonal mirror would be to reflect (reverse) light waves arriving along the longitudinal or optical axis of the photoreceptor.

In their 1992 book, "Standing Wave Analysis: A new vision of color," Jörg Krumeich and Alfred Knülle-Wenzel review the historical literature of standing wave color hypotheses, critique trichromacy, and present their own concept of color perception based on standing wave analysis. I am indebted to these authors for their deep and extensive literature search. Their hypothesis is also published in a 2002 paper, in Optica Applicata.

Drs. Krumeich and Knülle include a discussion of potential longitudinal mirrors in the eye. To their list I have added a mirror or two suggested in a different body of literature. Investigators of the Stiles Crawford effect have remarked that cone cells bounce back some of the light that enters. This implies the existence of a mirror, and so these researchers have also been interested in identifying candidate reflectors in the eye.

There appear to be at least six candidate mirrors.

Bruch’s membrane
the tapetum lucidum (although not in primates)
disks or ensembles of disks within the photoreceptors
lamellar inclusions clustered beneath the cones in the pigment epithelium. (in primates)
The interface between the inner and outer segments

The end of the outer segment

In the eye, the anatomically most evident reflective surfaces like the tapetum or Bruch’s membrane are positioned as planes orthogonal to, and skewered by, the optical or z-axis of the eye.

A less obvious mirror in this same plane, orthogonal to the optical axis of the eye, is formed by the photoreceptors’ disks, which can be conceived as reflectors working in concert. This concept was detailed by van Der Kratz in the 1996.

In a standing wave system it is also possible the disks may consititute internal mirrors positioned at wavelength intervals along the optical axis, since the disks’ "bleaching" necessarily alters their properies of transmittance and reflectance. The basic idea that the disks might form a mirror is a very old one. Krumeich and Knülle point out that Zenker, in the mid-1800s, seized upon this idea as soon as the disks were discovered. Zenker incorporated the disks as reflectors in his original hypothesis of standing wave color detection.

Krumeich and Knülle make a convincing case for Bruch's membrane as the essential mirror. I will quote one of their points here: "Retinal detachments from Bruch's membrane ceases immediately any function of the retina in this area. Traditionallly this blindness is put down to the interruption of nutrition to the cones, which should follow from the choroid. It is unknown why this blindness occurs immediately together with the detachment and why the reattachment of the retina onto Bruch's membrane leads to an instant restoration of sight." [italics added].

Upon detachment of the photoreceptors from the mirror one would surmise the standing wave would cease to exist. Upon reattachment, the standing wave would be instantly restored.

Krumeich and Knülle offer another fascinating conjecture: If Bruch's membrane is the mirror for a standing wave, then the backwall of the eye is, in effect, the origin of the standing wave signal we would wish to detect. This offers, at long last, a physiological reason for inverted structure of the retina, in which the photoreceptors are mounted against, can be said to "look backwards" toward -- the mirror.

In 1974, Niels Bülow reported his discovery in a monkey eye of strongly reflective inclusions clustered beneath the cone cells where the outer segments are seated in the pigment epithelium. These inclusions were not found, however, beneath rod cells. He remarked that "…the rods were inserted deeper into the pigment epithelium than the cones and passsed by the cluster of melanin granules at the end of the cones." He reported five types of inclusions. The most interesting of these were lamellar, showing the characteristic structure of a multilayer natural mirror. Bülow first discovered the inclusions using light microscopy, and then went back after them with an EM.

Neils Bülow is the author of the standing wave color detection hypotheses in which the cone photoreceptors are seen as tuned cavity resonators. He has published two papers detailing his view of this possibility. Of the inclusions he remarks: "As regards the physical effect, it has been suggested that light scattered by the melanin granules (Bülow, 1968) may pass backward through the outer segments of the photoreceptors, where standing light waves may be produced under certain circumstances (Bülow, 1968). The present observation, that melanin granules are situated at the end of the cones but not at the end of the rods, suggests that passage of light from the melanin granules, backwards through the outersegments of the photoreceptors is possible only in the cones, not in the rods."

The state of play:
All of the orthogonal mirrors remarked here have been incorporated in standing wave models or hypotheses. In addition, mathematical models of standing waves in photoreceptors have been created choosing as reflectors (essentially by executive fiat) the interface between the inner and outer segment at the ellipsoid, and the "end" or closure of the photoreceptor cell where the outer segment abuts the pigment epithelium.
The basic idea that standing waves should be found within cone cells is not widely celebrated --- nor is it strongly contested or resisted in the literature. From my reading my impression is, it sort of gets a nod. The six or seven standing wave color detection theories are largely forgotten or discounted or still unheard of, so let’s guess the existence of a standing wave in a photoreceptor is not viewed by many people as a dangerous, insidious threat to Fort Trichromacy. It might in fact represent a challenge to trichromacy, or it might alter in some way our understanding of trichromacy, but this is not an issue many people are aware of or hurrying to controvert. This is because almost no one with an orthodox view of how the eye works -- imagines that standing waves in photoreceptors could be detected.

We do know that cones cells are directional waveguides. When light is aimed straight into the cones – some of it comes straight back out again. This is not hypothetical -- it can be demonstrated physically. It appears that along the axis of a cone cell light is being reflected back on upon itself by some mirror or some system of virtual or pseudo or quasi mirrors – wherever and whatever they are. For this reason a longitudinal standing wave inside the photoreceptor seems likely and reasonable -- in cones at least.

A pivotal paper
Rods are far less directional than cones in their response to incoming light, so one cannot remark that light goes straight in and comes straight back out again, as we could for a cone. Accordingly, in rods one should probably remain agnostic about longitudinal standing waves. However, there is a well known and well regarded computer simulation of standing waves set up within the outer segment of a rod cell. This work was a technological tour de force, and you will find it cited again and again: M. Piket-May, A. Taflove, J. Troy, "Electrodynamics of visible light interactions with the vertebrate retinal rod", Optics Letters, vol. 18, pp. 568-570, 1993.

The basic idea is that the bulk structure of the photoreceptor has the physics of a waveguide, and that the internal disk stack adds to the system the physics of an optical interferometer. The authors conclude: "These effects combine to generate a complex optical standing wave within the rod, thereby creating a pattern of local intensifications of the optical field."
The splendid centerpiece picture in this paper's Figure 1 shows the standing wave pattern computed at three different wavelengths: blue, green and red. To compute just one of these three standing waves required 1.6 hours of grinding away by a Cray Y-MP supercomputer. The paper demonstrates the power of a computational technique called FDTD, for finite difference time-domain. It is a way to find numerical solutions to the Maxwell equations. Taflove, an electrical engineer at Northwestern with a renaissance range of interests, coined the term FDTD. He has developed and applied the FDTD technique to problems ranging from geophysics to oncology to neuroscience. Piket-May, who is now at Colorado, is also a high profile scientist both within and beyond the field of electrical engineering. Troy is a neuroscientist at Northwestern whose laboratory studies the activity patterns of the retina's ganglion cells.

The photoreceptor electrodynamics paper was published in 1993, but it apparently relied upon some dimensional data from the biological literature of a decade or two earlier: Specifically, the interdisk interval used in the calculations is different from the width accorded the disk. Today these two dimensions would be probably be set up as identical, with each disk 15 nanometers wide and the space separating the disks also 15 nm.

In strong multilayer natural mirrors, identical layer thicknesses and intervals are a trademark structural feature. The strongest natural mirrors exhibit this morphology. Whether newer, more fashionable rod cell dimensions, showing the periodicity of the disk stack, would alter the ultimate results and conclusions, if the computations were to be repeated today, is difficult to guess. There is a more recent (2005) paper by A.M. Pozo et al in which FDTD was used to model the cone (rather than the rod) photoreceptor.

We are by now familiar with the idea of longitudinal standing waves in photoreceptors. But note that the long-pondering Cray supercomputer’s three standing waves, which look like Navajo carpets in Piket-May’s wonderful Figure 1, are not simply depictions of longitudinal standing waves: rather, they show more complex waveguided structures, also incorporating transverse standing waves.

Adding a transverse dimension
The inside of a rod or cone cell has a higher refractive index than the surround: The interface, or wall, is therefore a transverse reflector. The reflective cylindrical walls of the photoreceptors figure prominently in contemporary models of the photoreceptor as a waveguide. According to these models, the rods and cones resemble fiberoptic transmission lines -- made more complicated of course by their internal disks and external anatomical shapes, which tend to flare or taper.

Waveguiding adds another dimension to the story -- a transverse dimension which cannot be neglected. In the real world a mirror set up at a right angle to the optical axis of the photoreceptor can produce a longitudinal standing wave -- but we must also consider as reflective/refractive surfaces the tubular walls of the photoreceptors. We must also consider the tiny diameter of the photoreceptor, which can only accomodate a few wavelengths of light. The diameter of frog rod photoreceptor, which is regarded as oversized, is still only 6 microns, or about 10 wavelengths of red light, wall to wall.

So, to the simple and familiar musical instrument models reviewed above, in which the standing waves are treated as longitudinal only, we must add transverse standing waves bookended by the walls of the waveguide.

We are still talking about standing waves in a biological organelle. However, light is an electromagnetic wave and per Maxwell this is how EM waves act. They propagate straight ahead, with transverse electric and magnetic field components positioned at right angles to each other and to the axis of propagation. Confined inside a photoreceptor by end and sidewall mirrors, this wave will produce a three-dimensional standing wave pattern. If the photoreceptor cylinder is cut, so that the 3D distribution of light can be viewed in section, certain characteristic 2-dimensional light patterns will appear.

These visible waveguide effects, first observed under a microscope in the early 19th century, are both intricate and spectacular.

This is a frog retina. The curious, detailed kaliedoscopic patterns seen in the rod cell sections were drawn by hand (A. Hannover, Vid. Sel Naturv. Og Math. Sk. X, 1843). They look like targets, benzene rings, starbursts, dots and circles. They are also uncannily reminiscent of Venetian glass millefiori patterns. In the 1840s, as he sketched, Hannover thought these patterns must represent anatomical structures. Today, in an era of fiberoptics, we recognize the designs as typical, changeable light patterns called waveguide modes.

Here is a sampling of twelve different waveguide modes produced in modern fiberoptic waveguides.

In the photoreceptors of humans and monkeys, at least 10 waveguide modes can be readily observed. Waveguide modes change rather abruptly, shifting from mode to mode with changes in wavelength.

What interests us here is detection. Within the photoreceptors, there exist resonant patterns of light intensity, waveguide modes, that are known to change decisively as a function of light wavelength. For a multichannel neuron, perhaps detection or recognition of these modal patterns is possible. Note that the amphibian rod photoreceptor "disks" are not really disks. They are cut through by radial incisures into as many as 18 distinct lobes -- pieces of pie. The disks' incisures are aligned. As you climb the ladder from amphibians, in higher vertebrates the disks have fewer lobes, anatomically. But at the level of biochemical domains cordoned off within the disks, who knows? In short, yes, it is possible that the waveguide modes, and therefore a staircase of about 10 distinct wavelengths, could be recognized and detected. In a multichannel photoreceptor, it could be accomplished by wiring and deploying, as sensors, sections of disks rather than whole disks. One might also look for sensor domains arrayed in concentric rings or bands on the surface of each disk.

There is an extensive literature developed over several decades on photoreceptors as waveguides, much of it focused on the two Stiles-Crawford effects. An accessible resource and a good place to begin is the Optical Society of America’s thick Handbook of Optics, available in many libraries. Part 2 of the Handbook is devoted to Vision Optics, and was edited by Jay M. Enoch, who pioneered the study of photoreceptors as waveguides. With lead author Vasudevan Lakshminarayanan he is also the co-author of the Handbook’s Chapter 9, “Biological Waveguides.” The wonderful 1843 Hannover drawing was perhaps unearthed by Lakshminarayanan and is reproduced in this chapter. He also includes fascinating photos of waveguide modes in human and monkey photoreceptors.

It would be nice if the waveguide modes simply clicked between higher and lower orders according to some well behaved and well understood clockwork, changing uniformly and predictably as a pure function of input power, wavelength and dimensions. But they do not. Along the light path within the photoreceptor energy is being bunched, reflected out, refracted out – lost and absorbed. There are often observable colors associated with certain modes, and the colors may be seen to change along with wavelength along the photoreceptor. Importantly, signals in one waveguide may influence the signals in adjacent photoreceptors, an intriguing phenomenon perhaps if you are looking for effects that might be helpful in phase or edge detection.

For experimentalists, the photoreceptors are difficult to study intact. If the cells are sectioned, as in the Hannover slice, the act of sectioning will change the optical dimensions of the structure under study and may, as Krumeich and Knülle point out, amputate an essential mirror.

For modelers and theoreticians, however, the field is rich. Do a quick scan for papers that cite Snyder and Pask (Snyder, A. W., and Pask, C.: The Stiles-Crawford effect—explanation and consequences, Vision Res. 13: 1115, 1973.). This paper was among the first theoretical characterizations of the cone as a waveguide. There are by now many waveguide theories (and alternative-to-waveguide theories), but at this point none is regarded as definitive.

Conclusion
The photoreceptor is an interface where input signals that move at the speed of light are transduced into output signals that move at the speeds of trucks and trains. The colossal signal speed transition in photoreceptors is rarely remarked, but it is the basic reason standing wave theories are so attractive. As a first step, standing waves in photoreceptors freeze incoming light signals and thus shift them into a physical regime that can be effectively measured and reported by biological sensors connected to slow neurons.

All of the information borne by incoming light waves could be conserved and captured by detectors sensitive to standing waves in photoreceptors: color, brightness, spatial phase and polarization.

The central idea we are playing with here is that the retina could read signals sampled by the disks at points throughout its depth. As many as 1500 distinct sensors – the disks -- are arrayed in a row along the optical axis of each photoreceptor cell. If each disk is separately wired, then the retina can be conceived as a 3-dimensional sensor, a sensitive solid, rather than a flat 2-dimensional sensor like photographic film. In order to work, this “sensitive solid” or 3D sensor array depends upon a multichannel neuron and, thus, a multichannel photoreceptor.

From a sensor array of disks aligned along the longitudinal axes of the photoreceptor cells, one can spin theoretical models in several different ways. Many of them involve or require the detection of standing wave patterns.

 The sensor array can be used to detect chromatic aberration, and thus used as the basis for a corrector or even as a color separator.

 The array can be used to separately detect the image plane and the Fourier plane of the eye’s converging lens.

 It can be used to detect and measure basic properties exhibited by longitudinal standing waves -- wavelength and intensity.

 By segmenting the sensors at each disk, and with a little further elaboration, the multichannel photoreceptor can be used to detect different waveguide modes and polarization.

 By integrating or simply memorizing signals from multichannel photoreceptors one can create a retina that detects and conserves spatial phase.

Standing waves in photoreceptors probably exist. This is not a very controversial proposition. The question is -- could the brain detect them? Given a 1-channel photoreceptor, no. Given a multichannel photoreceptor, yes, there would be several different ways to make it work.

<<PREVIOUS CHAPTER NEXT CHAPTER>>

2007-04-22T10:49:00.000-07:00

#9
Eye Evolution

Animation courtesy of
Daniel Cordier, ENSCR, France

Iridescence: The observed color changes as a function of viewing angle. The colors of this Morpho butterfly range in wavelength from blue through purple to russet and walnut. The physical principle underlying this relationship between viewing angle and colors can be used to create an angular position detector. Such a device “reads-out” the angle to a target object as a function of the color impinging on a photosensor. One can construct such a detector using a thin film étalon. This essay explores the possibility that primitive directional eyes might have exploited this simple principle. The resemblance of the Morpho animation to a radar antenna is coincidental but in some sense, quite apt as a metaphor.

A walk in the park
Suppose one fine day on a walk in the park you find yourself paced by and then suddenly engaged in an argument with a madman. A day later, in reviewing the experience, you may discover that your own sensible, well reasoned arguments were somehow shaped and made strange by the surreal arguments of that madman. This is because either side of any argument is, in effect, a template that lends its shape to the opposite side of the argument.

If we practical, sensible everyday partisans of biology formulate arguments against the Creationist template, we are basically arguing with a madman in the park. We should not be surprised to find that our own reasoning becomes, in this encounter, oddly distorted and not a little strange. Case in point. The wars of perfection, as follows:

The Wars of Perfection
Although the curious frontal wiring and backward looking orientation of the vertebrate retina is univerally accepted -- the "why" of it has long since become a football in arguments between creationists and biologists. The argument seems to be framed around the question of Perfection.

Biologists have observed that if the vertebrate eye were in fact intelligently designed, the vertebrate retina would be wired from behind. That way we would not have to peer at the world through the screen of the retina's wiring system. There would be no blind spot where the optic nerve makes its exit passage through the retina.

It is urged that what we have here instead is in fact an example of "stupid design," that is, a design which is not a very good one -- imperfect in fact -- and therefore cannot be defended as evidence of intelligent (Perfect) handiwork.

The Imperfect Eye argument was conceived as an elbow jab at the Creationists. Evolution, which is understood to be a rather accidental process, can produce imperfect results. This is okay, expectable, how it works. But to the Creationists, who do not accept evolution, the imperfection of the vertebrate eye raises a problem, because in the biblical version of events, there were no mistakes.

Naturally, technically inclined Creationists have countered with an earnest effort to show that the inversion of the retina is both Intelligent and Perfected after all.

This conflict over Perfection, like any other argument about the existence or specific intervention of God, is just a pastime –a very human pastime. It is also a complete waste of breath.

If we are going to insist on Imperfection, just to keep up our side of the argument against the Creationists, we will end by obscuring an important fact: The level of structural precision evolution can produce is simply stunning.

The vivid blue of this Morpho butterfly is not pigment. It is a precision mirror that, at certain viewing angles, beams blue light into our eyes. The mirror is formed as a carpet of scales on the wings of the butterfly. Each scale is a multilayer mirror. Its multilayer structure is so fine that it is below the resolution limit of a light microscope. It can be made evident only with the help of an SEM.

In general, biological mirrors are formed from a stack of typically twenty reflective layers dimensioned and spaced – precisely -- at intervals of a quarter-wavelength of light or at an odd multiple of a quarter-wavelength of light.

In multi-layer mirrors the evolutionary pressure is supplied by the physical principles of thin film interference.

This is an effect that comes into play when light waves are reflected from interfaces spaced apart by tiny, tiny distances -- less than a few wavelengths of light. If you have ever spilled oil on water, or blown bubbles, you've seen this familiar effect.

High light reflectance is a result of constructive interference. Light reflected from the top surface must interfere constructively with light reflected from the bottom surface, so the two waves must be in phase. The effect depends on the angle and wavelength of the impinging light, and on the exact dimensions and refractive indices of the structures the light penetrates.

In the butterfly's wing scales constructive interference occurs only for odd multiples of a quarter wavelength (1,3,5) of light. Low reflectance, a result of destructive interference, occurs for even multiples. The principle of thin film interference is discussed in detail at the BU site, and its role in nature is well presented at this Arizona State site.

The diagram above shows just two layers separated by a thin film. This structure can reflect only a few percent of the incoming light. Reflection owing to constructive interference can be enhanced to nearly 100% by stacking the mirror in multiple layers. In nature, we are talking about a biological structure with about twenty layers and gaps, dimensioned and positioned, accurately and repeatedly, at intervals on the order of an eighth of a micron.

Precision multilayer mirrors and gratings occur in the scales of butterflies and moths. Multilayer mirrors can also be found in the scales of other insects and fish, and in the tapetum lucidum of the eye. Insect scales in particular are hugely important in the history of science: Ernst Abbe's theory of image formation is based on his insight, in 1872, into the diffraction effects of the multiple layers (multiple slits, viewed end-on) on light entering the imaging system of a light microscope through a butterfly's scale. We owe our understanding of how images form -- in cameras, in microscopes, in our own eyes -- to an insect's scale.

Here are some papers on the subject of structural color and natural photonics, on the site of the University of Exeter's fascinating and impressive program on Natural Photonics. The best treatment I have seen of physics of the Morpho butterfly is on the site of Daniel Cordier in Rennes.

Natural multilayer mirrors are typically reflective of colors, since they are dimensionally tuned to specific wavelengths. Note too that they will reflect different colors at different viewing angles. They are iridescent.

White light mirrors in nature, like fish scales (in the Herring, for example), are even more sophisticated. In the fish scale structure, three multilayer mirrors are overlaid. Each has a different natural wavelength. A third of the mirror reflects blue-green, a third reflects red purple, and a third reflects orange yellow. The net reflection of the stack is white light.

In sum these biological optical structures are almost crystalline in their, well -- perfection. Examples of amazing structural precision – and movement -- also abound at the level of molecular biochemistry and include nanomachinery like hemoglobin (the "molecule that breathes") and the active site mechanisms underlying enzymatic reactions.

We can freely delight in our amazement, our scientific astonishment. Amazement does not require us to fall on our knees. Nor is it embarrassing to be impressed by biological precision – as though we would somehow, by being impressed, give away points to the Creationists on this absurd question of Perfection.

The conflict between reason and faith antedates Darwin by about 15 centuries, and it cannot be "won" with Darwinian reasoning or any sort of reasoning at all. It is not a debate. It is a power struggle. Reason requires questioning -- open, frequent, often defiant questioning. Faith is unquestioning acceptance and obedience. There is a good analysis of the origin and politics of this peculiar issue, which is a real quirk of Western civilization, in a 2004 book by the historian, Charles Freeman.

A short history of the vertebrate eye
In any event and in the meantime, the real question, which is not about Perfection, has been pretty much left behind the door. Here is the question:

What is the evolutionary reasoning or narrative that accounts for the inverted structure of the vertebrate retina?

The question is typically answered in texts on the basis of ontogeny, and this is a safe and solid answer. But as usual there are riskier, more imaginative approaches with more style and promise. One proposal I have admired, because it is based on function, is that the modern vertebrate eye was once and in some sense still is a mirror eye.

This type of eye uses a backwall mirror rather than a lens, or in combination with a lens, to form an image. The scallop eye is one modern example of a mirror eye. We may or may not have an interesting ancestor in common with the scallops, but in this hypothesis our eyes are at least analogous.

And the primitive chordate eye had no lens at all. The photoreceptors were positioned, aimed and wired to capture an image formed by the eye's backwall mirror. This explains why the retina "looks backward" toward the mirror.

In a subsequent step, a lens evolved, perhaps to correct the optics of the backwall mirror. (This is thought to be the purpose of the lens in the eye of the modern scallop. It is shaped to eliminate the mirror's spherical aberration.) Eventually the vertebrate lens took over the whole job of imaging, and the role of the backwall mirror shifted to other tasks. According to this hypothesis, the mirror on the backwall of the eye survives today in structures like the tapetum lucidum -- the eye mirror that accounts for the shining eyes of cats by night. The cat's eye shine helps its night vision, but a modern backwall mirror may be useful in other ways and in other creatures, including primates.

One can, of course, rewrite this narrative in many different ways. The one crucial element is the primitive chordate eye that worked -- formed images -- without a lens. The idea of a mirror eye as an ancestral eye is appealing because the thing evolves a retina and a lens in two distinct, successive steps. It is thus a plausible solution to the problem of simultaneity.

If we leave out the primitive mirror as a step in the evolution of the vertebrate camera eye it might seem necessary to evolve two very sophisticated and optically precise structures, the lens and the retina, one suspended above the other, each functionally dependent upon the other -- simultaneously. The difficulty of imagining, let alone accomplishing a simultaneous evolution of the retina and the lens was first remarked by Darwin. The shortcut solution offered to us by the scallop is that it never happened. The retina appeared first. The lens evolved much later.

The computer model
There is a well accepted model for eye evolution which depicts the beeline evolution of a camera eye. (There is no evolutionary detour through a mirror eye or compound eye to arrive, much later, at a camera eye.) Although Darwin was stumped by the problem of simultaneity, it has proved possible after all to model with a computer a stepwise, linear process of evolving a camera eye with a lens positioned over a retina.

The starting point is a flat, trilayer sandwich comprising an outer protective layer (which is transparent), a layer of photoreceptive cells, and an underlying layer of pigment cells.

This is a minimum eye -- a photosensor which, owing to the pigment backing, has directional sensitivity. Light can only reach it from one direction. When a shadow appears in that direction, the animal knows something's coming, and from which way, and it can react directionally. Or, underwater, a directional eye can simply indicate which way is up by detecting the light of the sun.

From this starting point, the computer generates random deformations and selects for improving visual acuity. It turns out the structure evolves by first dishing to form a pigment eye cup. Then, ultimately, gradually, a lens is formed over the aperture.

While the eye is still an open cup, it functions as a directional sensor. Light impinging on a particular point inside the cup can only have come from certain directions. (A cup is more sensitive to direction than a simple flat sensor blocked, in one direction only, by a pigment layer.) To take full advantage of the directional quality of the cup, the interior becomes populated with photoreceptors. As with the mirror eye, one could remark that a retina emerges before the lens.

However, cupping does not really produce a nice imaging eye until the cup has become a sphere with a hole in it and that hole -- now understood as an aperture -- starts to close.

Two alternative paths to a camera eye
This is the second half of the eye's story -- after it has cupped beyond the hemisphere and begun to close to form an eyeball. The word camera means "room", and a room implies enclosure.

In the limit of cupping, you would get a sphere with a pinhole aperture that produces a sharp but rather dim image on the retina, as in the eye of the modern nautilus.

One could imagine that in life, there would be at least two different possible sequences of evolutionary events as the aperture closes.

Perhaps the eye might close all the way to a pinhole aperture -- and in this way discover the miracle of an image. And then invent the lens as a way to open the aperture up again -- i.e, to capture more light without losing focus.

Or it could be the cupping process would not run to completion -- that the closing aperture would be capped by a lens. An evolving lens would arrest the cupping process and produce an image before the aperture ever contracted to the diameter of a pinhole.

In any event: According to the model, using what we might call brute force evolution, it takes only about 400,000 generations, or perhaps a half-million years, to arrive at what we would regard as a modern camera eye. A half-million years is just a moment in geological time. In fact, over the eons, camera eyes have turned up many different times in creatures as diverse as spiders, squids and ourselves. From the short time frame it is easy to conclude that the camera eye has evolved several different times in several different species, each time starting from scratch. But this has turned out to be an incomplete or simplistic idea. The more we learn about development, and how to think about development, the more intricate this story seems to become.

The original computer model is a classical work (Nilsson D-E, Pelger S (1994) A pessimistic estimate of the time required for an eye to evolve. Proc R Soc Lond B 256: 53-58). I think what the model describes best is the most direct way to achieve a camera eye.

Colorized SEM of a jumping spider, courtesy of Tina Weatherby Carvalho, Biological Electron Microscope Facility, U. Hawaii

The spider's camera eye started out in a different direction -- it was probably a compound eye once, and then its multiple, faceted lenses coalesced. Spiders have lots of eyes, primary and secondary, and their secondary eyes sometimes exhibit lens/mirror combinations. So the mathematical model does not really attempt to tell the spider's complicated story. Nor was the model designed to be specific to vertebrate evolution. It does not offer any special explanation for the backward-looking retina of the vertebrate eye.

And we are left with some questions about the pull and tug between the evolving camera eye's two distinctly different functions. Initially the modeled eye is a directional sensor. Ultimately it is an imaging eye. The transitional steps or logical bridges between these two different types of eyes -- two different physiologies, really -- are not clear to me. This is not a criticism of this model. It is a criticism of the way we think about the evolution of vision.

The imaging eye is the superstar. The directional eye is marginalized -- we tend to fast forward past it on our way to the heroic, climactic formation of an image. This is perhaps an error in emphasis. Many of the parts and pieces of the imaging eye must have evolved in the service of a machine which could not and never did form an image, i.e., the directional eye. Among the parts and pieces I have in mind are the color photoreceptors and the backwall mirror.

Good enough now.
The hypothetical vertebrate mirror eye would have given the ancestral vertebrates a huge jumpstart on vision -- a good, working imaging system mounted early in the game.

They probably needed a jumpstart. The invertebrate (arthropod) compound eye -- was already advanced, perhaps perfected, in the lower Cambrian. There were chordates in Cambrian, but they are thought to have been sightless. The vertebrate camera eye was a come-from-behind effort.

The earliest vertebrate camera eye for which there is fossil evidence appears to have been installed and working in a fish that lived about 430 millon years ago. By that late date the arthropods, with their compound imaging eyes, had already been looking around (for things to eat, such as the small, supposedly blind chordates) for about 100 million years.

A mirror eye is optically inferior to a modern eye with a fine converging lens. But it works and, more to the point, in evolutionary terms it works almost immediately. As soon as the primitive eye is cupped, the structural basis for a mirror is in place.

Parabolic reflecting antennae, Imperial War Museum.

In 1940-41, in the urgency of the Battle of Britain, English radar scientists had a peculiar motto, intended to help restrain themselves from perfecting their instrument: "Good enough now." They coined this little slogan because it was far more important to make radar work and put it into service immediately, even as a makeshift, than it was to make it perfect. "Good enough now" limns an evolutionary principle for the mirror eye.

When the lens finally evolved, optical excellence became possible. The mirror eye hypothesis clearly explains the backward-looking vertebrate retina: In the old days, there was something to see back there: The world was reflected in and focused by the backwall mirror. For our remote ancestor, the mirror eye would have been a timely solution and great thing. But it necessarily left the vertebrates a legacy of -- and a problem with -- light scattering by the retina's own intervening wiring system.

Why a mirror? The color radar hypothesis.
To get a better sense of the prehistory of the vertebrate eye, we should reconsider the directional eye.
This early eye is informally conceived by most of us as a “real” eyeball that is still under construction. (Informally, that is, because we relax the usual teleological objections.) The primitive eye might be flat, it might be dished, it might be half an eye, a hemispherical cup – but it could not yet form an image. So in what sense was the directional eye a finished product? What could it actually do?

The possibility exists that the directional eye was once a very finished product: a passive radar system based on the detection of color. Its structural basis would be a monolayer -- and then later a multilayer biological mirror. With this type of directional eye, color is not perceived by the animal as painted onto a literal image, as it is with a modern camera eye. Instead color (wavelength) is read by the animal as a signal that reveals the solid angle to another, targeted animal – prey or predator. When the target animal moves, the detected color changes, and it changes in such a way as to suggest, to the beholder, the speed and direction of the targeted animal’s path.

We cannot know if a color-based radar eye existed 500 million years ago – or ever existed. It will always be a pure speculation. Probably best to just regard it as an eye that never was – an imaginative exercise. But if it did exist in the world, perhaps by the early Cambrian, it could have had a fairly long run.

Photo of a Wiwaxia model courtesy of the Royal Tyrrell Museum in Alberta. The iridescence of the blades decorating this hatlike Cambrian creature is owing to a natural diffraction grating. The iridescent effect and its underlying physical structure were discovered by the Oxford zoologist, Andrew Parker, who recounted the discovery in his book, "In the Blink of an Eye." Note that the blades, in effect, shoot colors in every direction. Why did this little Cambrian animal find it useful to bombard its immediate surroundings with iridescent colors? One possibility is that Wiwaxia’s blades evolved as a very effective radar jamming strategy.

In the end a color based radar eye could have been defeated by color itself -- by radar jamming. Iridescent displays, which were popular in the Cambrian and later, could have flummoxed a color radar eye. We know of no modern examples of a radar eye, so if it ever existed, it was in fact defeated.

It could be an interesting high school science project to test the possibilities of this hypothetical but intriguing form of vision by actually fabricating a color-based radar eye using analogous optical, electronic and logic hardware. Better, it could be readily modeled or simulated with a computer program, perhaps Mathematica (TM) or a modified drawing program. In a virtual world one could try out the device and perhaps gradually re-invent it (as evolution would) to suit several different underwater settings, since wavelength and light intensity and directionality all change as a function of depth.

The solid angle problem
To see the basic problem facing a primitive directional vertebrate eye a little more clearly, let’s consider briefly a different, perhaps rival solution -- a type of anatomical structure favored by bugs and other arthropods. Here is a compound eye. Notice that the lens/receptor units are arranged and packed into a convex hemispherical array. Each photoreceptor is aligned, rather like a telescope aimed at the sky, along a specific and unique solid viewing angle. In the most primitive, no-neck animals, the eye could not be quickly aimed at a target. The head did not move relative to the body. The eye did not move within a socket. The retina did not move within the eye. It was a question of aiming the whole animal. Not a problem for a modern, expertly piloted high speed insect like a housefly, for example -- but not so easy for a soft, slow, primitive underwater creature, crouching on the seabed.

For a primitive fixed eye, one good solution is to collect information from an array of receptors arranged as a convex hemisphere, parsing the world into a global array of solid angles. Suppose one of the many photoreceptors reports a target – reflected light or shadow. The identity of that receptor, because of the direction in which it is structurally fixed and aimed, automatically defines and reports the solid angle to the target.

The arthropod’s convex hemispherical eye structure shows us a literal minded engineering solution to the problem of parsing the world into an array of solid viewing angles. The hypothetical color-based radar eye does the same job: It determines solid angles to a target – a point of light or darkness. It uses much less equipment, and it need not be hemispherical. It can be perfectly flat. But it has the same technical objective, which is to detect a change in the light at some specific solid angle drawn from the center of the eye.

How does a radar eye work?
Suppose we start with the familiar concept of “the minimum eye,” the primitive structure posited in 1994 by Dan-Erik Nilsson and Suzanne Pelger as a reasonable launch point for mathematically modeling camera eye evolution.

The minimum eye is a flat, trilayer sandwich comprising an outer protective layer (which is transparent), a layer of photoreceptive cells, and an underlying layer of pigment cells. This is a minimum eye just by definition – a photo sensor which, owing to the pigment backing, has directional sensitivity. Light enters from the front, but not from the back, where it is blocked by pigment. A binary device. Yes, no. Up, down. Nothing to it.

But there is a wonderful built-in complication. As noted, the minimum eye is fitted with a transparent outer layer and across this layer, which is essentially an étalon, we must expect to observe the effects of thin film interference. The important physical property of the thin film sandwich structure, from the standpoint of building a directional eye, is not color per se. It is iridescence, that is, perceived color change as a function of viewing angle or, more importantly for our purposes -- angle of illumination. The position and therefore the viewpoint of the photoreceptors is fixed by the anatomical structure of the eye. What can vary is the angle of the incoming light reflected from (or interrupted by) an object.

In the discussion of biological mirrors here, we have repeatedly remarked on their typical property of iridescence. Iridescence is a natural phenomenon in which a surface exhibits different colors depending on the observer’s viewing angle. A good example is the surface of a Morpho’s wing. If the Morpho has alighted and is sitting perfectly still -- and you edge sidewise to shift your point of view a bit, the color reflected into your eye by the wing will be seen to change from blue to a russet or walnut color.

Slightly different angles produce distinctly different colors, in this instance, russet vs blue. Iridescent color varies with viewing angle and with the angle of illumination. Photo courtesy of Daniel Cordier, ENSCR, France

The optics are reversible. If your point of view is held fixed, and the butterfly moves, the color you see will change. The color you see will depend upon the solid angle that one might draw between your eye and the reflective surface.

The Iridescent Eye

The hypothetical radar eye of the predator (if you could be transported back to the pre-Cambrian world with your own imaging eye in order to see it) would itself be iridescent. This is the eye of a modern bottom dwelling coffinfish, from the collection of the Yale Peabody Museum. The eyes have iridescent flecks. If an iridescent eye did once exist, perhaps vestiges of its structure could still be identified.

In the simplest, earliest circumstance, the predator's target would not be iridescent. It would simply be an object, possibly a living object, illuminated by ambient daylight underwater. Underwater lighting is a complicated story. For our purposes this object, a prey animal for example, can be imagined as either reflecting or interrupting a ray of light impinging on the eye. I find it easier to visualize the ray interrupted by a black object.

Suddenly, the watchful predator notices, blue light has disappeared. Gone soft or gone out completely. This means: an object exists somewhere in a sharply defined cone of possible solid angles, let’s say 35 degrees. But 35 degrees right or left? Fore or aft? And how far away? One eye can distinguish the presence of an intruder, but it can only report its solid angular position as a point existing on a conical plane of possibilities.

So it would take at least two flat, simple radar eyes to construct, in effect, a triangulating radar system – each eye, on the alert, perceiving the instantaneous weakening and strengthening of different colors.

As the object moves across the predator’s “field of view,” the spectrum of each eye trills colors, like running a finger up a keyboard. The spectral sequence shows direction. The rate and rate-of-change of the trill show speed and acceleration. Stereoscopic logic triangulates the instantaneous position.

But note clearly that the animal never "sees" color as we do. For this creature, color does not exist. We can say "trills colors" but the animal does not mutter to itself a litany of "red-orange-yellow-blue-indigo-violet" as an object traverses its field of view. Its photosensors simply detect the intensification of light at certain wavelengths at certain points. Detected bands of intensified light correspond, in turn, to certain solid angular positions.

The AND logic that puts the radar eye’s color information together would not be a bad prototype for the AND logic that is thought to support trichromacy in a modern, imaging eye.

Parts and pieces
What would it take to build a radar eye, structurally? Not much. It can be flat. In a pre-Cambrian and lower Cambrian world of flat creatures, so recently descended of flatworms – floating doormats, some of them -- flatness is a style motif. The detection system needs a zero, that is, it must be oriented vertically, pointed "up."

Modern coffinfish, photo by the Yale Peabody Museum.

Modern fish do this with amazing accuracy, and it does seem that "up", or "toward the light" is a concept in the repertoire of the most primitive sorts of creatures.

The minimum eye requires a light sensitive cell capable of color discrimination. It requires a doubly reflective layer, an étalon. The power of the reflector and, thus, the range of the eye could be improved by adding layers, producing a multilayer mirror, but a simple monolayer would do for a starter eye.

The photoreceptive cells are mounted in such a way that they “look at” the thin film reflector. Photosensitive elements could be opposed to the surface. They could be positioned in front of the étalon, or behind it, or they could penetrate into or interleave with transparent multilayers.

How can a primitive photoreceptor detect color? Color sensitivity can be provided to a monochromatic photoreceptor physically, by varying or dialing or oscillating the critical distance between the interfaces of the thin film. Chemical tuning to particular center wavelengths, as in a modern color photoreceptor, could evolve in due time.

When we look at a modern cone or rod cell, we immediately observe the orientation and spacing of the multiple disks. It could be that this bag-of-cookies structure is a relic of a photoreceptive system that was once embedded in, and integral with, the stratified mirror of an ancient radar eye.

In the Stiles Crawford Effect of the second kind, the color of monochromatic light is perceived to change as a function of its angle of incidence on the retina. It might be that this effect had a place or purpose in the prehistoric directional eye.

The radar eye, as an element we might wish to introduce into the dialog of eye evolution -- is a splendid parts bin for the modern, imaging eye that would have rather abruptly succeeded it. The radar eye supplies a retina populated with color photoreceptors, retinal circuitry, a species of stereoscopic logic and – my personal favorite – a backwall mirror.

The end of an era
Why an abrupt shift to the imaging eye? Imagine an ocean full of edible and abundant iridescent creatures, beaming bright colors in every direction, colors evolved to jam the radar eye. It becomes more and more difficult for animals equipped with the radar eye to do well. The radar eye makes mistakes. The predator lunges in the wrong direction. The prey flees straight into a predator’s mouth. The directional eye isn’t really workable anymore.

Then imagine inserting into this environment a new kind of predator equipped with an imaging eye that can actually perceive, rather than simply detect -- colors. An imaging eye is vastly superior to a radar eye.

Maybe she has a mirror eye, maybe she has a faceted compound eye. (We are guessing we are in the pre-Cambrian or lower Cambrian, but we don't really know where we are in geologic time.)

We do know she is hungry. She looks around. So many brightly colored snacks. So easy to see, so easy to track. A massacre ensues. She thrives on easily-won protein. Before long the iridescent colors fade. If there are any fish around, then fish scales will tend to overlay, evolving from iridescent color to white reflecting invisibility. Camouflage becomes fashionable. And in the end everyone who is left who has eyes -- has imaging eyes.

Eight steps
Let’s pull together these several threads of speculative ideas and see if we can synthesize a single narrative line -- a sequence of evolutionary steps leading to the modern vertebrate eye.

Here again is our friend the scallop: hardly a vertebrate yet not very far along, perhaps, from the split between invertebrates and protochordates. Suppose we regard it as an eye in mid-passage, a snapshot exhibit of the technical features that may have succeeded each other in the course of the evolution of our own modern eye.

It was first discerned by Michael F. Land, and reported in 1965, that the scallop eye was forming an image with its mirror, rather than with its lens. (Land, M.F. 1965. Image formation by a concave reflector in the eye of the scallop, Pecten maximus. J. Physiol. (London) 179: 138-53.)

Mirror eyes are also described from page 104 in Animal Eyes, which the definitive book on the whole subject, written by Michael Land and Dan-Erik Nilsson.

The scallop eye has been presented here as a good example of a mirror eye, but we have been developing a novel sense of where that backwall mirror, and the retinal photoreceptors nested against it, might have come from.

1) Specifically, we can now guess that the mirror first arose as a single transparent layer, an étalon, originally used as a component of a solid angle detector. Its purpose was to convert the angle or slant of an incoming light ray into a detectable color.

2) To multiply the sensitivity of the étalon, and add range to the detector, the single layer evolved into a multilayer, a strong reflector of light into the photoreceptors and – thus -- a pretty good mirror.

3) In the service of color detection, color sensitive photoreceptors and the retina evolved, firmly mounted against the surface of the mirror.

4) To improve the directionality of this purely directional eye, the tissue of the eye cupped and the backwall mirror curved. This helped in triangulating targets between or among two or more eyes, essential for rangefinding. Note that cupping the backwall changes the interference effects one would expect of a flat eye. The curvature of the backwall alters the incoming ray’s path length between layers as measured at different points along the backwall. This is an effect to compensate for or try to to use productively in modeling the system, perhaps as a means to create and detect rings.

The curved mirror and retina were already in place, according to this narrative, long before the evolving eye ever captured its first image. The animal could not "see," in the sense of imaging, but it had a rather precise sense of its own position relative to predator or prey. This is the type of positional sense that one could get from radar set: An accurate detector reporting the target’s direction, speed, acceleration, and range. And perhaps size.

Then what?

To defeat this type of eye, there followed an age of iridescence, possibly in the Cambrian. Surrounded by color, drowning in color -- the radar begins to jam. The animal makes costly mistakes.

5) A happy accident occurs. The retina actually doubles. A duplication, one of evolution’s favorite stunts. As in modern animals like the turtle, a layered retina appeared. Imaging is now possible, very close.

6) All it takes is a bubbling, a physical separation of the two retinas, in the manner of the scallop. And suddenly, voila, we have an imaging mirror eye. One retina, the new retina bubbled up away from the mirror, will capture an image focused by the backwall mirror. Another retina, the original backwall retina, remains stuck in place, plastered against the backwall. The old retina is not well defined in the 1906 Lankester drawing, but the retinal split it is described by Schwab in some detail.

There ensues a transitional period when the eye has two visual senses to process: a true image, from the upper retina; and the original color radar readout, from the lower retina. Ultimately the outboard, imaging retina triumphs, takes over, and we can have a true mirror eye.

7) A lens evolves to correct the spherical aberration of the backwall mirror. Now that the mirror is focusing an image, the aberration suddenly matters.

8) The lens can do more. The lens ultimately takes over the job of focusing an image on a retina.

But which retina? This is a guess I cannot make, but the end product is evident. Ultimately the retina in the middle of the eye migrated back "home" to the eye’s backwall. The evolutionary pressure to do this would be to improve and perfect the optics of a camera eye. In the end, the optics of the original mirror eye no longer matter. The mirror finds other work, as the tapetum lucidum.

Conclusion
I should probably re-emphasize the obvious -- that this is a speculation. There is no other way to get at the problem. Fossils, however abundant, will never show us how eyes evolved. Eyes are soft tissue structures, rarely preserved. Comparative anatomy, engineering imagination, molecular clues, computer modeling and simulation are our best tools, and our best product will always be a good guess.

A footnote about Step 5, the doubling of the primitive retina to create a tiered retina. This is not a novelty in nature. Tiered retinas can be discovered in modern animals, usually where there exists no mechanical means for accommodation. Notice in the drawing, however, that in the scallop there are not only two retinas – there are actually two anatomically distinct optic nerves. When the two retinas re-married, as the lens took over from the mirror, the retinal structure on the backwall would still, for a while at least, represent a bipartite retina.

The bipartite retina is a concept we have repeatedly advanced, in this essay, as one likely means for separately and simultaneously detecting both the Fourier plane and the imaging planes of the lens in the vertebrate eye. It has been suggested that the Fourier plane or "back focal plane" is also the plane of memory.

<<PREVIOUS CHAPTER NEXT CHAPTER>>

2007-02-14T06:38:00.000-08:00

#7
Standing Waves in Photoreceptors

Animation courtesy of Dr. Dan Russell, Kettering University.

One cannot, ruler in hand, race alongside a light wave in hopes of measuring its wavelength or its phase relative to an adjacent light wave. The great appeal of standing waves is precisely that they are standing still, and therefore susceptible to sensing and measurement, specifically at the backwall of the eye. In the animation, the blue wave pumps up and down but remains stationary on the x-axis -- hence the name, "standing" wave.

Let's start with a quick review of how standing waves form. The two upper light waves are identical, moving at lightspeed in opposite directions. They are separated here for clarity but one should imagine both waves moving in opposite directions along the same axis.

This pattern would occur if an incoming wave (let's say the uppermost, rightbound wave) encountered a mirror and was reflected back along its original path. The reflected wave is represented in the animation as the middle wave. The bottom wave, that is, the blue standing wave, results from the superposition and algebraic addition of the upper two.

A measurement or detection of the peak-to-peak distance of the stationary wave would tell us the wavelength of the incoming light, that is to say, its color.

As one mentally accelerates these ultraslow motion depictions of waves, the standing wave acquires the fixed-in-place appearance of beads on a string.
Because the wave is sitting still, it is easy to take measurements from it. If these were radio waves, for example in the amateur 2-meter band, one could simply walk along the radiation pattern with a handheld sensor and detect the succession of peaks and nulls (antinodes and nodes). Various electronic frequency measuring instruments are based on the principle of standing wave detection and measurement.

For visible light, however, the wavelength is much shorter, on the order of 400 nanometers to 700 nanometers.

It is extraordinary – a triumph, actually -- to capture an image of a standing wave at this tiny dimensional level. Conventional optical microscopy is not helpful. This is because the microscope’s optical resolution approaches its ultimate limit, which is half the wavelength of the very lightwave we would wish to observe.

It is possible to break past this conventional resolution limit, however, using the novel technique of scanning near field optical microscopy (SNOM). Solid state physicists working with W1 photonic crystal waveguides have made standing waves in this regime a focus of study. One remarkable result is the standing wave image reproduced below. The work is described in detail in a 2005 paper in Applied Physics Letters by Robert Wüest et al at ETH in Zurich.

Dr. Wüest kindly contributed the following description of the image he obtained using scanning near field microscopy. This is a
standing wave produced in a photonic crystal waveguide using a laser source at wavelengths ranging from 1480 nm to 1495 nm:

In scanning near-field optical microscopy (SNOM) a subwavelength fiber tip is used to pick up the light at the surface of a structure. The tip is scanned at a constant height of 10-20nm above the surface and thus, the topography and the optical-field information is acquired simultaneously with a resolution of roughly 100nm. This method was used to characterize photonic crystal based integrated optics waveguides at telecom wavelengths. The SNOM images show the standing wave inside of a terminated W1 waveguide, in a very intuitive and at the same time quantitatively accurate way. Such measurements were used to characterize the photonic crystal waveguides concerning their dispersion relation and the propagation loss with a high degree of accuracy.

Incidentally, the SNOM technique has been applied with interesting results in scrutinizing neurons. Here is a report in a book on nano-optics. It takes a bit of navigating. Click the "Search Books" button at the very top of the page, and then click the search results for page 183. Several other pages are accessible on line that discuss scanning biological material. Also, here is a paper that may be helpful as an entry point.

Standing waves in the eye?
The physiologist W. Zenker first remarked on the possibilities for color sensing based on the detection of standing waves in retinal photoreceptors in 1867. (Zenker, W. Versuch einer Theorie der Farbrezeption, Arch. F. mikrosk. Anatomie 3, 248 ff, 1867). The idea has persisted ever since, in a sort of shadow literature on color detection in the eye. Zenker never sensed a conflict between trichromacy and his own standing wave theories. But after Zenker the literature becomes divergent in the sense that it explores ideas which are, for the most part, politically positioned by their authors as conflicting with, or even as revolutionary alternatives to, the almost universally accepted Young-Helmholtz or trichromacy model of color detection.

I am not sure that the two models could not be rather easily reconciled or in some way quite logically enfolded into each other but, in any event, the standing wave color theorists typically present their ideas as alternatives to trichromacy. The early papers are in German and the most recent papers are in English, but the locus of interest in these ideas, and the research that has devolved from them, seems to be centered in northern Europe, in Germany and Denmark.
3D illustrations of rod and cone cells courtesy of Dr. Roger C. Wagner, Professor of Biology, University of Delaware.

One idea is that a cone or rod can be modeled as a dimensionally tuned cavity, resonant at a specific color wavelength. This is a second cousin of the trichromatic model, in that each photoreceptor is wavelength specific. But instead of just three major cone types, each with a maximum sensitivity set for one of three different colors, the resonant cavity hypothesis offers a much wider variety of dimensionally diverse photoreceptors, each attuned to a different specific color.

The charming quality of tuned cavity color vision is its shared physics with tuned musical instruments. In this particular standing wave hypothesis, the retina is analogous a cathedral organ. Lots of pipes of different diameters and lengths. When you try to visualize how this pipe organ metaphor might apply in practice, in the eye, it turns into a real puzzler -- but the basic resonant cavity idea seems solid nevertheless.

The tuned dimensions of a rod or cone are diameter and length. (Incidentally foveal cones, which is to say, most cones, do not actually taper. They are cylindrical rather than conical. There is subtantial statistical variation across the retina in cone diameters and lengths.) To get a sense of how sensitive standing waves can be to dimensional parameters, click to this string animation. Slide to wavelength multiples of 25, i.e., 25, 50, 75, 100 and 125, hit the start button, and watch the resonance effects.

The self-tuning trombone
In a slightly more mechanized and, in my opinion, more readily workable standing wave photoreceptor hypothesis, physically, the resonant cavity is seen as self-tuning, so that each photoreceptor can respond across a spectrum of color.

It was first observed in 1943 (by Detwiler) that retinal rods elongate and contract in light and darkness, respectively. The effect was studied carefully in the 60s and 70s, and it was shown by Wolken that in vitro, a severed rod outer segment can actually double in length when illuminated -- without a change in diameter. In darkness, if ATP is added back, the outer segment contracts, returning to its normal length. These are not passive changes. Photoreceptors are descended of cilia. They contain contractile and structural proteins that enable active mechanical changes and movement

In life and in theory and in daylight, the length changes in the outer segment need not be so extreme.

If the outer segment is viewed as a dimensionally tuned resonator for standing waves, then tiny incremental changes in cylinder length could bring the rod into resonance with an incoming light wave -- thus maximizing the intensity of the standing wave. The proposed self-tuning effect works like sliding a trombone -- or simply nudging the slider in this animation. Turn on the ruler, and try values approaching and at 100 MHz.

To make the tuning process automatic in the eye, i.e., self-tuning, requires a feedback loop. Or the receptor could simply oscillate in length a bit, continually scanning for, and passing through, resonances.

Another hypothetical self-tuning mechanism, sort of a calmer, more sophisticated, solid state alternative, does not require any mechanical movement. In this concept self-tuning results from the alteration, by incoming light, of the transmittance/reflectance property of each disk. This is a chemical change which is accompanied by changes in optical properties and dimensions. It was originally characterized as "bleaching." It can be modeled as the positioning or re-positioning -- at wavelength intervals -- of multiple mirrors.

One receptor, all colors
In the most optimistic view of these self-tuning models, each photoreceptor can report the reception of any color. From a purely engineering standpoint -- setting aside for the moment questions of spectral range and of biological plausibilty -- this is a very appealing approach.

In the eye, each photoreceptor is a point of convergence. When light hits a point on an object in the world, it scatters in all directions. The lens of the eye differentially refracts some of the rays originating from that specific point in space and focuses them back into a single spot on the retina – a photoreceptor.

Thus, each photorecepter positioned in the image plane of the lens responds, electronically, to a specifc point in the world. It would be ideal if the brain could somehow directly and instantly read, at each photoreceptor, the color of the light arriving from the conjugate point on the object we happen to be looking at. The payoff would be the best possible resolution that can be obtained from a given population of photoreceptors.

In the textbook trichromacy model, cones are "color blind," in that they respond to many different light wavelengths across a spectrum centered on an absorption peak. Mentally pull a horizonal line across this graph at a y-axis value of 50, and regard it as an intensity reading from a red cone. Notice where the line would intersect the red absorption curve. A red cone intensity readout of 50 could mean the impinging light has a wavelength of 500 nm, which is blue-green -- or it could mean with equal authority that the impinging light has a wavelength of 625 nm, which is red. The intensity reading from a cone is perfectly ambiguous.

Therefore a comparision of the intensity output of one cone with that of another cone -- one with a different absorption peak -- is absolutely required to produce a meaningful determination of color. A specific color wavelength is not directly measured by any single cone. It must be deduced.

Thus, in the trichomatic model it takes at least two photoreceptive cones, not just one, to sense the color of incoming light. It also takes additional circuitry to perform the logical evaluation of cones' output signals.

The self-tuning standing wave models are much simpler. Each single cell can directly identify many colors -- and the detector works across a whole spectrum of wavelengths. No output comparison or additional processing steps are required. A photoreceptor simply looks at a color and reports it.

However, when we say "a whole spectrum of wavelengths," we should also say, to be realistic, something like "within a spectrum constrained by the wavelength range which is possible for that particular photoreceptor." For example, a red cone could report any received wavelength within its range -- a range defined and constrained by that cone's photosensitive pigment. Each type of cone, Low, Medium and High wavelength, would cover an appropriate segment of the total spectrum.

In this view, the evolutionary point of using different photopigments is simply to extend the wavelength range of the eye. Within each of the three (or four) bracketed ranges, the specific wavelength of the incoming light, whatever it is, can be precisely sensed and reported using standing wave detection. The total responsive range is the same as that for the conventional model.

In the following paragraphs we will consider how a standing wave detector might work, briefly glance at two more standing wave models, and then circle back to look at the main historical objection to any and all of these standing wave photoreceptor hypotheses -- their Achilles' heel.

The detection system
The length of the outer segment of the retinal rod is usually taken to be about 25 microns, or 25,000 nanometers. If one were to align within the cylinder of the outer segment of a rod cell an incoming blue-green light wave of 500 nm wavelength, then the cylinder would encapsulate as many as 50 cycles. Rod outer segments may reach 45 nm in length, and cone outer segments can be over 50 nm long. And in rods, as in the trombone model, the cylinder length might actively vary from moment to moment. So for our purposes, let’s just say the outer segment of a photoreceptor could encapsulate 50 to 100 cycles of a standing light wave.

3D illustrations of rod and cone cells courtesy of Dr. Roger C. Wagner, Professor of Biology, University of Delaware.

The photosensitive disks wrapped like cookies in the outer segment are positioned at center-to-center intervals of 30 nanometers. Each disk is 15 nm thick. Each inter-disk space is also 15 nm. (In older literature you will find numbers like 20 nm center-to-center and a 10 nm disk thickness, but the 30 nm center-to-center and 15 nm thickness reflect the current view.)

Resolution of the sensor
Suppose we regard the disks as increments inscribed along a transparent ruler. To measure wavelength one might mentally overlay, on a standing wave cycle with a wavelength of, say, 500 nm, about 17 photosensitive disks upon each standing wave cycle, or "bead of light" appearing within the rod cylinder.
.
The 30-nanometer center-to-center interdisk interval is common to most photoreceptors and is often characterized in the literature as "almost crystalline" in its periodicity. Here are 17 disks viewed edgewise in a photomicrograph of the outer segment of a rod cell. The 30 nm “center to center” distance between disks implies a distance measured between idealized, one dimensional geometric detector planes. In fact the disks are rather meaty objects, 15 nm thick. Of this thickness the two closely stacked photoresponsive membranes of each disk have a combined thickness of perhaps 12-13 nn. The two membranes are so close together they appear as one thick line in this photomicrograph. Separating the two membranes there is just a 1.5-2.5 nm aqueous gap -- a gap barely visible in this picture, except as a white dot near the disks' rims.

Bear in mind that the rod structure depicted in the EM is dead as a doornail. In life, these membranes are surely a little plumper and juicier. This doubled-up membrane thickness necessarily fuzzes the resolution of the disk – it is not a perfect, one dimensional sensor plane. In other words we have a been handed a ruler on which each single increment is drawn-in as a closely spaced double line every 30 nm. Yet the rod’s resolution is still pretty impressive. As an approximation, I have superimposed the photomicrograph on a single bead of light. As noted, in the example of a 500 nm blue-green light wave, the rod cell positions 17 photosensitive disks, or increments, in each bead of light. The resolution is comparable and perhaps superior to the resolution of the SNOM device used to take the photo of the standing wave.

How to build a detector:
How can we make a color detector out of this? A simple model works like calipers. Suppose we wish to measure – on this string of beads of light -- the peak-to-peak distances. The photoreceptive disks are each responding, as a function of light intensity, at finely graded (30 nm) increments along the Z-axis of the rod cell. Along the standing wave, some disks will register bright light -- some disks will register light levels near darkness.By observing any two light intensity maxima (antinodes) of the standing wave, and then remarking the distance between these two peaks, we can call out a number which varies with the color of the incoming light. It would work equally well, in theory, to pick out a pair of light intensity minima, and jot down a number representing the distance between these two nodes. One could also measure the peak-to-node distance. So there are three ways to detect color using photoreceptive disks as though they were electronic calipers.

One could also simply count-up to or accumulate a signal proportional to beads per micron, or peaks per micron, or nodes per micron, since any such ratio would vary as a function of the incoming light's wavelength. This approach has the advantages of averaging -- it is a little less dependent on exact dimensions.

Intensity is amplitude squared, so the negative going excursions disappear. We know the nervous system typically triggers at preset thresholds of sensitivity. A threshold detector associated with each disk or, in the vernacular of analog electronics, a "peak picker", could be set up to identify, that is, selectively trigger off of -- light intensity peaks.

One receptor. Three data points

We are exploring here the possibilities for a multichannel neuron. A photoreceptor cell is a specialized neuron, and we are proposing to use our hypothetical multiple channels to separately wire each single disk. Thus, we are regarding each single disk as a distinct light intensity sensor. In this way we can sample light intensity at, for example, 500 different points arrayed along the optical axis of the photoreceptor cell. Any level of sensing sophistication is possible, ranging from binary or threshold detectors to fully analog devices. One approach is to regard the disk sensors as producing countable peak detector outputs.

With a system of this type we can detect three essential properties conveyed by the incoming light: brightness, color, and depth. As we have noted, a count of peaks appearing within a fixed unit length along the z-axis of the photoreceptor will vary with wavelength. This "count" could be detected, represented and communicated in various different ways.

An absolute count of the total number of peaks along the z-axis exceeding the threshold setting of the peak picker -- will vary with intensity. This count, too, could be detected or represented in various ways.

A system to remark the position of the peaks along the z-axis, relative to the positions of adjacent standing wave peaks formed along the z-axes of the adjoining photoreceptors -- indicates spatial phase. This requires a comparative reading of at least two adjacent photoreceptor cells. One likely candidate phase detection system would depend upon the horizontal cells. To capture the 3D structure of a plane wave at play on the whole retina, one should perhaps imagine a scanned or massively parallel array output.

In general, this seems like one plausible and intuitive basis for a photoreceptor mechanism, perhaps because it so strongly resembles radio frequency and phase shift measuring instruments based on the creation and observation of standing waves.

Let’s elaborate a little on the phase detection system. Visualize a section through the retina as a trainyard viewed from above. On hundreds of parallel tracks (photoreceptors) are parked hundreds of parallel freight trains (standing waves). Each freight car represents a standing wave peak, or “bead of light.” Notice that the trains and freight cars are not perfectly lined up. Each car is offset somewhat – phase shifted – from the freight car on the adjacent track.

If you painted bright red, say, the fifth freight car behind each and every locomotive, you could create a red line snaking across the trainyard, -- representing a line through the phase-shifting plane light wave we are interested in detecting. At the level of adjacent photoreceptors, the local phase shift can be detected and measured as an incremental shift along the z-axis between any max peak located at one disk of one photoreceptor -- and a neighboring max peak located at one disk of an adjacent photoreceptor. To rely on the analogy, one is measuring an incremental spatial shift along the z-axis between two red freight cars parked side by side.

Note that in a standing wave detector, color (wavelength) dictates the coupling distance between the freight cars. Therefore the simultaneous detection of color and phase is going to be complicated by changes in the color of the light spread across the plane. Sorting phase from color is a problem the organism, or the standing wave theorist, must take into account. However, the information needed to color-compensate the phase data is ready to hand, since color wavelength at each photoreceptor is separately detected and known.

There are several other interesting hypothetical standing wave mechanisms. One imputes to the system reading the disks a pattern recognition ability, able to notice multiple harmonics and complex intensity patterns associated with mixed frequency inputs. This theory takes into account the fact of partial reflection.

Another hypothesis is based on the detection, at the disks, of distinct waveguide modes -- unique geometric patterns that change rather abruptly as a function of wavelength. We will come back to this one.

The insurmountable 1-channel output problem.
In this review I have been using terms like "jot down" or "call out" or "count up" in order to avoid, up to this point, the tricky problem of how to export wavelength information from a photoreceptor as a nerve signal. Over the past 140 years, the standing wave hypotheses of color perception have all hit the wall on this problem.

A lot of interesting physics can happen when light hits a rod or cone, no question about it -- but news of these intricate events is restricted to what can be transmitted via the rod or cone cell's 1-channel output line.

Consider for example the cone cell. In the trichromacy theory, the cone cell is basically just a light meter. For a given center wavelength (one of three) the cone will produce a voltage output proportional to light intensity. The output signal will fade on either side of the absorption peak, and in this sense one could say that the voltage output varies with the input wavelength. But the point is, the textbook cone has no separate and distinct output signal that varies with light wavelength. There exists only one information channel out of the cone, and the cone uses it to signal its perception of light intensity -- nothing more.

Reporting to the brain both intensity and wavelength (color), separately and simultaneously, could be accomplished pretty easily with a multichannel neuron. But given the 1-channel output, the standing wave theorists have been obliged to invoke some sort of yet-to-be-discovered serial signalling or other time-multiplexing scheme -- in order to transfer wavelength information out of the photoreceptors.

This is probably why these theories have not, to date, gained much traction. Several of the proposed standing wave detectors make a lot of sense -- as detectors, viewed in isolation -- but there is no very direct way to plug them into the nervous system.

The photoreceptor, bipolar, amacrine and horizontal cells are analog type processors, so it is possible within the retina to communicate and process graded analog signals rather than action potentials. But as one tries to shift information from the eye toward the brain the problems become even more daunting, because the output lines – the retinal ganglion cells, the optic nerve – transmit all-or-none spikes. Now we are plunged into the familiar connundrum, and hopeless bottleneck, of neural encoding and decoding.

The idea that standing waves probably exist in photoreceptors is reasonable enough and is not strongly contested. Published mathematical models of photoreceptors commonly take into account both longitudinal and transverse standing waves. But the brain is believed to be completely oblivious to these standing waves.

In short, the concept of standing wave detection in photoreceptors has been brushed aside, historically, repeatedly, because in order to come up with a physiologically workable scheme in which the retina detects standing waves -- one is almost obliged to reinvent the nervous system.

It is of course the premise of this blog that -- exactly, yes -- the nervous system should indeed be reinvented, and that a multichannel neuron exists. We take it as a given, as the basis of our thought experiment. So within the logical framework of this blog, we have a degree of freedom that was not available to the original standing wave theorists, and we hope to see what can be done with it.

<<PREVIOUS CHAPTER NEXT CHAPTER>>

2006-12-05T07:35:00.000-08:00

Chapter six
What does a memory
look like?

Fourier transform of a literal image of a duck, courtesy of Kevin Cowtan. The Fourier Duck originally appeared in a book of optical transforms (Taylor, C. A. & Lipson, H., Optical Transforms, Bell, London 1964). An optical transform is a Fourier transform performed using a simple optical apparatus. In the memory hypothesis to be presented here, the "simple optical apparatus" is the lens of the human eye.

Images from waves
It is surprising that so few of us are ever exposed to a wave-optical version of how images form. Even students of physics don’t get much of a glimpse, in basic texts, of the physical basis of image formation. Optics presented in general biology texts is largely based on ray tracing. Diffraction is mentioned in a sidebar, essentially as a nuisance effect which puts a limit on resolution. There is rarely a hint that the whole process of imaging is based on diffraction.

Here is the main element that is missing from the ray-tracing approach and story. Notice in the following illustration the blob marked "diffraction pattern" hovering behind the lens (figure adapted from Elementary Wave Optics, by Robert H. Webb, Dover, 1997).

How and why should a diffraction pattern appear in the focal plane (at the distance f) behind the lens? What produced it? What is it good for?

It turns out the diffraction pattern is the product of an extra optical device which has been inserted into the system as a theoretical convenience: This system is acting as though a diffraction grating had been mounted in front of the lens, although in fact no grating is literally present.

It is quite a story. In the early 1870s, Ernst Abbe, the genius behind Zeiss, noticed that certain specimens studied under the microscope were highly periodic in structure. An insect scale is a good example.

Colorized SEM of a butterfly’s wing scale. Photo courtesy of Tina Weatherby Carvalho, Biological Electron Microscope Facility, U. Hawaii

It occurred to Abbe that an insect scale could be understood as, or simply declared to be -- not just a tissue specimen but an optical device: a diffraction grating. By thinking about an insect scale in this novel and very special way, and by assuming coherent light conditions, Abbe realized he could freely apply to the mystery of image formation the known mathematics of diffraction gratings.

The straightforward geometric math worked out by Thomas Young in the early 19th century to describe the results of his two-slit experiment works equally well in describing the interference effects of light passed through multiple slits – diffraction gratings. Abbe applied Young’s formulation. What emerged was a description of image formation as an effect produced by light wave interference.

It is now called the double diffraction theory of image formation. Light wave interference occurs in two places: The first interference occurs behind the lens in the diffraction plane (the "blob" remarked in the figure). The second interference occurs at the image plane, and produces bright points of light -- Airy circles. An image plane populated with these bright points of light produces an image.

So our understanding of image formation is based on a rather startling assumption – that all sorts of objects on display might be successfully modeled as diffraction gratings intervening between a coherent light source and a converging lens.

This assumption was useful because it put into the hands of Ernst Abbe a huge mathematical short cut provided by Young’s handiwork on diffraction at slits. The Abbe assumption is clearly based on a very special case. Most objects are not, after all, insect scales or other types of diffraction gratings, and most light is not coherent for long. We do not view the world through a venetian blind. But Ernst Abbe was on the right track.

Any sort of object can be said to diffract light -- diffraction is universal, not unique to diffraction gratings.

Although the light of the world is not coherent, it is partially coherent, and one can describe the system in terms of instantaneous fringes with continuously changing positions. Nice sharp interference bands are best demonstrated by beaming, at parallel slits, coherent monochromatic light from a laser. In our messy, partially coherent, polychrome world the bright and dark bands do appear, but their appearance at a given point is merely instantaneous. For this reason the bands are imperceptible to us. But they exist.

Abbe's treatment of image formation has stayed glued for over 135 years.

In a paper published in 1906, A.B. Porter built upon Abbe’s idea. Porter proposed that Abbe’s two planes, the diffraction plane and the image plane, could be interpreted using the Fourier series: It turns out that the image on the image plane is the Fourier transform of the diffraction pattern on the diffraction plane. The diffraction plane is now often called the "Fourier plane." It is also called the "back focal plane" of the lens.

The double diffraction theory was refined and given its name by the Dutchman Frits Zernike in 1935. Zernike ultimately won the Nobel for inventing phase contrast microscopy, but his earlier work on double diffraction image formation seems more fundamental.

In the end Abbe’s theory was successfully generalized to include non-periodic objects and partially coherent light – the objects and lighting of the real world.

The theory is still introduced by showing how an image is formed by a very special and specific type of object, a diffraction grating, illuminated by coherent light. Here is what Abbe came up with – an iconic picture you will find again and again in explanations of image formation.

Thanks to the diffraction grating, or "object," light waves interfere at the back focal plane of the lens. Wave interference produces bright and dark interference bands there. The bright bands act as point sources of light. Rays drawn from point sources in the back focal plane of the lens will converge with rays drawn from other point sources that arise simultaneously at the back focal plane.

These rays converge at points on the retina. At these points a second wave interference occurs, resulting in bright points of light imprinted at the retina. It is the picture dotted-in by these secondary bright points on the retina, the Airy circles, which we perceive as an image.

An Aside: "The Focal Plane"
as slang physics

Photographers and physics teachers may tell you that for distant objects, the image behind a converging lens forms very simply on a single plane called “the focal plane.” This is wrong.

In fact there are always two planes behind the lens, the image plane and the back focal plane, and the image always forms on the image plane. An image never forms on the focal plane

When students get a first look at the wave optical model of image formation, they often wonder why a diffraction pattern should suddenly appear on the focal plane of the lens. They have been taught an image is supposed to appear on the focal plane. But this is just physics slang.

Practical logic supports the slang treatment of “the focal plane” as though it were the image plane, and this is probably why the misconception is so common. To see the reasoning, start with this animation of a converging lens system. If you push the slider to the left, you can see the object recede toward the western horizon. As it does so the image plane advances toward the back focal plane of the lens. Conversely, as the object is brought nearer to the lens, the image plane retreats further behind the focal point, so that the focal plane and image plane become well separated. (The planes are represented here as points, which lie on the respective planes.)

As an object recedes into the distance, according to the thin lens equation, the image plane and the focal plane get closer and closer together until finally, when the object is "essentially at infinity," the two planes can be regarded for all practical purposes as one single plane called, very unfortunately, "the focal plane."

In detail, d sub o is the distance to the object; d sub i is the distance to the image; and f is the back focal length. As d sub o approaches infinity, the denominator explodes and the term becomes, practically speaking, zero. Negligible.

This makes the distance to the image plane identical to the distance to the focal plane. So now you can get away with saying the image forms on the focal plane. This is the formal logic behind the lazy vernacular of photographers and physics teachers and, to be candid, almost everyone else including me, that would put the image on "the focal plane."

But it never happens. An image never forms on focal plane, and the two distinct planes always exist. As long as an object is this side of infinity (and they always are), its image will always form on the image plane, and both the focal plane and the image plane will always exist.

Note too from the animation that as the object heads toward infinity, the image on the image plane shrinks. An infinitely distant object would produce an infinitessimal image, yes? This is just another way to say the same thing, which is that an image never appears on the focal plane.

In wave optics, it matters. This is because you need both planes, the back focal plane and the image plane, to make the wave-optical machinery of imaging work.

To add further confusion, the "back focal plane" has two more names, "diffraction plane" and "Fourier plane." You just have to be alert to all this terminology.

Finally, bear in mind that the focal plane is not flat. It is curved in 3-space like the backwall of the eye, and this is essential. To create an image that is actually flattened, as it must be on camera film, requires serious optical design work.

Optics is so full of tricks and traps that physics educators have done studies in an effort to try to figure out why this particular body of knowledge is so difficult to convey from teacher to student. One reason is that the nomenclature of optics is not precise.

The distance between the two planes
It is clear from the double diffraction diagram that the back focal plane and the image plane are well separated in Abbe’s Zeiss microscope. The thin lens equation tells us to expect a substantial separation between the two planes when very near objects are imaged. In fact in a microscope there is sufficient space between the two planes that it is possible to insert additional hardware to get a look at the diffraction pattern at the back focal plane. This can be accomplished by positioning a half-silvered mirror at the microscope’s back focal plane.

For the human eyeball, unlike a microscope, the object to be imaged is relatively distant. For this reason the focal plane and the image plane are very close together. The close proximity of the two planes is maintained, in the optical system of the eye, by accommodation.

Here the rays are shown converging to a single point on the image plane. Lots of points like this one would constitute an image. The back focal plane is not even shown -- it has been merged, as usual, and for everyone's convenience, into "the focal plane."

We know the real back focal plane must lie somewhere between the image plane and the lens, but where exactly? How close are the two planes?

This picture is accurate in the sense that the two planes (points) are so very close together they cannot be separately drawn by an artist with a thick pencil. Both critical planes, that is, back focal plane and the image plane, stack up on the retina.

Two planes, two pictures and
the point of it all.
The retina is about as thick as of a sheet of paper, which is pretty thick at the working scale of a light wave. The outer segment of a human retinal rod is nominally about 25 microns long, so we can take this as the thickness of the photosensitive layer of the retina. Within the z-axis boundaries of the retina, we have positioned two planes, the diffraction plane and the image plane.

Let’s guess the retina senses the play of light on both of these two distinct planes, separately and simultaneously.

The literal picture on the image plane is what we see. The interference pattern on the back focal plane (i.e., the diffraction plane) is what we store.

How to model a Fourier Transformer?
On the planet where I live, each neuron has 300 or more numbered channels.

The brain receives, instantly recognizes and transmits analog levels we can readily represent as numbers. It is an immensely fast and competent analog computer. It reads the senses, calculates, and issues exact motor commands, all with enormous speed and facility.

The retina detects spatial phase. And each cerebral hemisphere is a glorified, stratified stack of quasi-retinas.

This is a very different hypothetical machine from the nervous system as it was understood in 1963 or as it is accepted today.

This brain is analog-incremental, not binary. It has the calculating speed of an analog computer and the clean data transmission of a digital computer. Each single spike represents a number, an integer, and the number does not have to be 1 or 0. It could be 247.

The first place to look for an interference pattern in the human brain is in the eye, in the diffraction plane (back focal plane) of the lens.

It is this interference pattern that is ultimately recorded in the brain, in distributed fashion -- as distributed memory. Remembering consists of Fourier transforming the interference pattern back into a sharp literal image.

The whole process depends on phase conservation, but we have provided for this, on our planet, by wiring every single light sensitive disk in each photoreceptor of the retina.

What is a memory?
If you can agree with a brilliant hunch, then I agree fundamentally with the basic intuition of Karl Lashley and the holographic memory theorists: Memory is a stored interference pattern corresponding to a literal image.

But it is not an interference pattern made by mixing a scattered wave with a pure, coherent reference wave, as in benchtop Holography. Nor is it produced in the nervous system by electrical wave interference on the dendrites.

It is simply a light interference pattern produced in an Abbe imaging system at the Fourier plane, or back focal plane, of the lens in the eye.

So on my planet, this is a memory: The interference pattern corresponding to a literal image (of a duck in this case) is the thing to be remembered.

The memory could retain both of these two pictures readily enough, and it might well do so in the short term or even indefinitely. But it is the interference pattern which can constitute a durable long term memory resistant to injury and insult. This is because the interference pattern encodes the duck in distributed fashion. The interference pattern is also susceptible to filtering and processing steps for feature extraction, edge detection, and image cleanup. Finally it could be the basis of a content addressable retrieval system in the manner of Van Heerden's.

In this hypothesis, both the interference pattern and the literal image fall within the thin depth of the retina, on the focal plane and the image plane of the lens, respectively -- one plane stacked close behind the other.

The focal plane of the retina is the memory's doorstep and precisely here, on this doorstep, nature has delivered an interference pattern, ready to store and process.

What is new here?
The idea that the brain processes or creates interference patterns as information has been kicking around, in and out of vogue, for at least 65 years, and you can find in the literature quite a range and variety of brain hypotheses rooted in Karl Lashley’s basic intuition of 1942. Fourier transformation seems to figure into all of these streams of ideas at some point.

In the visual system, the usual assumption is that the eye captures a literal image, and the brain, subsequently, transforms it by calculation into the frequency domain by performing a Fourier transform. The retina is conceived conventionally, and is capable of detecting only one plane -- the image plane. The Fourier plane in the eye is ignored. The brain recreates or approximates the interference pattern of the Fourier plane by calculation. The calculated interference pattern, once arrived at, is then stored as a memory.

In the present hypothesis, things happen much faster. The interference pattern is not calculated -- it is simply detected and stored.

The Fourier transformation is accomplished optically, by the lens of the eye, at the speed of light. The transformation operates on the interference pattern on the Fourier plane, and creates (optically) the literal image on the image plane. You would expect this of any Abbe imaging system.

However, it is proposed that the human retina can detect both the literal image and its Fourier transform simultaneously and in real time. This is new. It implies a bipartite retina.

As noted, its utility depends upon spatial phase conservation and this talent, in turn, depends on a multichannel neuron. In my view it would be impossible to accomplish this work with one-channel neurons.

Note that any subsequent interconversion, in the brain, of a literal image and its Fourier transform doesn't require much time or technology. This is because both pictures are being captured and, probably, recorded simultaneously and in tandem. The brain could flip back and forth between a literal image and its Fourier transform without any calculation or analog processing at all.

Realtime access to the Fourier plane could be quite useful in the short term (in cueing an associative memory for example), but for Fourier filtering and processing steps (thinking, imagining, remembering) analog calculation would be required of the brain.

Parenthetically, note that the Fourier plane and the image plane are just useful geometric constructs. The two exist at the opposite ends of a continuum -- they are not necessarily sensed or stored separately.

So there are several new things here. Lightspeed, a different starting point and order of Fourier transformation, a bipartite retina – and, perhaps mainly, a memory process that is essentially biological in its nature, grounded in eye anatomy, physiology, and evolution.

In lower vertebrates....
Primates and birds and a few other vertebrates, including chameleons, are lucky enough to have evolved a clear-sighted fovea, the special, exceptional and superb apparatus for seeing literal, high resolution images. More is required than a concentration or packing of photoreceptors at the point where the optical axis intercepts the backwall of the eye. There must also be a rearrangement of nerve and vascular tissue to expose the outer segments of the photoreceptors.

About 50 percent of bird species have two foveas per eye and some birds have three foveas per eye, a trick to widen angular range of vision without requiring the animal to move its eye or head. Birds are smart. Parrots are very smart. Interestingly, the exaggerated structures in a bird’s head are its eyes -- the cerebral hemispheres don’t add up to much.

In vertebrates without any fovea -- most vertebrates -- I would guess the essential signal sought from the retina by the brain is in the frequency domain, on the Fourier plane, although it would be helpful to sample the blurry image plane as well.

These animals must extract a useful image from a murky fuzzy scene owing to the intervening retinal tissue that obscures their photoreceptors and wildly diffuses the incoming light. For these animals, we can guess that the incoming signal from the Fourier plane on the retina would be filtered and transformed, by analog calculation in the brain, into a sharper image of the world. Because the intervening goo is a fixed feature of the animal's own retina, the filter specification should be constant in every scene, so the delivery of a crisp, useful image could be accomplished very quickly. The whole process could be, in essence, hardwired.

Modern image enhancement software and de-blurring software works on similar principles. One speculates that Fourier de-blurring machinery in vertebrates is truly ancient. It probably evolved before the lens did, when the eye had to function with some other, less wonderful means of manipulating light: e.g. a pinhole, a sieve, a zone plate or a mirror, or some combination of these.

Why are we so intelligent?
At this point it becomes easy to speculate that our intelligence evolved from de-blurring machinery, that is, from Fourier processing hardware in the brain. When the clear sighted fovea finally appeared and made it unnecessary to run the de-blurring machinery all day every day, perhaps this left the Fourier processor of the brain some leisure time to start thinking about things. Mixing and matching, filtering and remembering.

The idea reads like a parody of the walking-upright hypothesis, wherein the evolution of our 2-legged stance freed our hands to make tools, throw things, etc., and the big brain followed.

But anyway, there it is. Maybe we are so smart because we evolved an open fovea, and thus freed for other tasks our already fully-evolved Fourier processors. The fovea provided the innate Fourier processor with the leisure to do things other than de-blurring our vision. This could account for the special intelligence of primates and birds, but one would have to make a more complicated case to explain the intelligence of dolphins. And the question also arises -- just how smart is a chameleon?

Parsing the retina
There are at least two ways to split the retina into two distinct detectors for the Fourier and image planes, respectively. One would be a detector split into two overlaid tiers, stacked up along the optical axis. This approach would seem to work best for lower vertebrates.

Turtles, for example, have a very thick, multi layered retina. This is understood to be the turtle's solution to the problem of accomodation without a squeezable lens, but perhaps one could also allocate tiers of the turtle retina to a Fourier plane and an image plane.

For humans, the more logical split of the retina is probably into its center, which is the fovea, and its surround. Essentially, this center-surround split also suggests a functional split between cones and rods.

I have come across one recent hypothesis – other than the one discussed here -- in which the retina is able to report to the brain directly from the Fourier plane. It is on the remarkable web site of Gerald Huth.

I think he is on a parallel track, but there are a couple of things that I would have to do differently.

He is detecting the Fourier plane on the fovea. In the human eye, I would prefer to nominate the surround – the rods, essentially -- as the detector for the Fourier plane.

In this colorized Fourier transform of the duck, notice the prominent red block of color at the center. In terms of information content, this central red spot is the least interesting part of a Fourier transformed image. It is almost a throwaway. In terms of spatial frequency, that is, detail encoded in the pattern, it is essentially DC– it tells you only the brightness of the image.

The central spot is often called the DC component. It is so strong that film exposed to record a Fourier pattern is often processed into a logarithmic image to reveal the finer, weaker bands that surround the dominant central DC spot.

Suppose we mentally organize a Fourier transformed image by overlaying it with a target pattern.

You can say that the DC component at center of the target is the least informative part of the pattern. As you work outward in rings from the center of the target, you get more and more information encoded about finer and finer details in the original image. The bigger the target, the more interference fringe bands you can accommodate – and the better the detail resolution of a literal image you could recover from the pattern by reverse transforming it.

Where should we put the fovea?
In short, the center of the target, the DC spot, which happens to coincide in the eye with the location of the fovea, is the perfect place to put the fovea. Inserting it there subtracts almost nothing from the information encoded in the interference pattern on the back focal plane. The bigger the target pattern, the better the detail that is retained. The finest, weakest fringes at the outer periphery of the target pattern contain the most finely resolved information to be recovered when the image is recreated.

The machine drawing of the human retina that emerges is a bit like a flashlight seen head-on. At the center the fovea, just 1.5 mm in diameter, is the receptor set for literal images. It senses the image plane. Surrounding it, extending all way out to the edge of the retina, is the much larger (in area) receptor set sensitive to the Fourier plane. Notice the essential 3-dimensional curvature of the Fourier plane, corresponding to the curvature of the retina in the eyeball.

In an idealized version of this system, the two concentric detectors would be offset along the optical axis. That is, the image plane detector would be slightly deeper in the eye, along the z-axis, than the focal plane.

In fact the fovea resides in "the foveal pit." The pit morphology is produced by peeling back the intervening nerves so that the outer segments of the foveal cones get a clearer view of the incoming light. Whether there is also a tiny additional optically precise offset corresponding to the distance between the focal plane and the image plane, is an interesting anatomical question.

In any event, the human bipartite retina looks very much like the textbook picture of the human retina -- with different captions. The fovea is still the fovea, and it is also the image plane detector.

The concentric surround of rods -- most of the back of the eyeball -- is conventionally allocated for black and white and, especially, night vision. We have newly labeled this part of the retina as the focal plane detector or, more precisely, as the Fourier plane detector. The exquisite sensitivity of the rods is put to work detecting the finest and most subtly varying outer fringes of the interference pattern on the Fourier plane -- corresponding to the finest details, or highest spatial frequencies, in the picture being received by the eye.

The memory’s own eye?
In summary, according to the bipartite retina model, the tiny fovea is the sourcepoint of our crisp, conscious view of the world. The foveal surround – all the rest of the retina – is memory’s own sense organ.

Within the foveal surround, from a Fourier plane positioned at a certain depth in the retina, the memory can directly record a densely informational interference pattern. This interference pattern, which constitutes in this hypothesis the substance of human long term visual memory -- plays all day on the foveal surround, the unconscious and unsuspected eye of memory.

How plausible is this hypothesis? There is no question a light interference pattern lies in the back focal plane of the lens of the eye. The Fourier plane of the lens exists. Without it, there could be no image. This is just physics. But has a neuroanatomical structure ever evolved that could, in effect, peel the Fourier plane away from the image plane and separately read it out to the brain?

To decipher the interference pattern, the brain must receive both phase and intensity data from the photoreceptor set. A one-channel output neuron cannot handle this job. It would be like trying to play a symphony with a telegraph key. However, we are exploring in this essay the possibilities for a multichannel neuron. The photoreceptor is a specialized neuron. A multi-channel rod cell, ultimately served by a multichannel output line, could indeed detect the Fourier plane effortlessly in real time. It might detect the image plane as well, simultaneously.

In sum, given a one-channel, all or none neuron, nothing works here. The Fourier plane exists -- part of the landscape of light in the eye -- but it cannot be detected or memorized. On the other hand, given the multichannel neuron assumption, yes, the hypothesis is suddenly plausible and something to conjure with.

I think Karl Lashley was probably right. The visual long-term memory is stored in the brain, in distributed fashion, quite literally in the form of an interference pattern. It is an interference pattern imported into the brain directly from the retina, from the Fourier plane of the lens of the eye.

An aside:
Glimpses of the Fourier plane?

What drug users report as effects of LSD could be accounted for as a curious, essentially pathological melange of signals from the image plane and the Fourier plane. The hallucinatory shapes and colors famously induced by the drug are pretty much what one might expect to see if the Fourier plane were somehow translated or projected onto, or bleeding into, the image plane. (I should add another possibility -- LSD produces a loss or distortion of the spatial phase information we are not supposed to have conserved).

Here from Wikipedia is a summary description of effects produced by the drug:

"…sensory changes [induced by LSD] include basic "high-level" distortions such as the appearance of moving geometric patterns, new textures on objects, blurred vision, image trailing, shape suggestibility and color variations. Users commonly report that the inanimate world appears to animate in an unexplained way. Higher doses often bring about shifts at a lower cognitive level, causing intense, fundamental shifts in perception such as synesthesia."

The effect of synesthesia from LSD is interesting in that it suggests the mechanism of memory may indeed be closely involved with the machinery we are exploring here. Flashbacks -- unbidden visual images that may recur for weeks or months -- suggest a link to visual memory. (Metaphorically, it is almost as though these images did not fit or pack properly in the memory, and kept popping back out of it. The images tend to be symmetrical, characterized as bugs and scorpions and reptiles.)

The best known psychedelic drugs are LSD, mescaline (from peyote) and psilocybin. At this site is an account of the discovery of LSD. It presents juxtaposed molecular structures for these three drugs. The recurring theme is phenethylamine. This is evident for mescaline but the phenethylamine structure can also be discerned in the more complex rings of the ergoline system of LSD. Other psychedelics include the substituted phenethylamines: 2,5-dimethoxy-4-ethylthiophenethylamine; 2,5-dimethoxy-4-(n)-propylthiophenethylamine, and 4-bromo-2,5-dimethoxyphenethylamine.

The uncanny relationship between psychedelic drugs and the perception of diffraction patterns will be developed in a later entry, but there is one optical fact that belongs here, in the discussion of the eye.

The power of mescaline (3,4,5-trimethoxy-phenethylamine) to alter imagery is much diminished for distant objects.

Aldous Huxley, in his long, meticulous essay on his experiments with mescaline, reported that the color and other effects (essentially diffraction pattern effects, one might guess) were induced only when viewing nearby objects.

He rhapsodizes over the intricate details of a flower in a pot.

But a chauffeured evening trip to the top of Mulholland Drive, to view the sunset over Los Angeles under the influence of the drug -- was a big disappointment to Huxley. It looked like any other sunset over LA. Beautiful, yes, but not transcendent.

He writes: "…we drove on, and as long as we remained in the hills, with view succeeding distant view, significance was at its everyday level, well below transfiguration point." But with closer objects, sharp edges and periodicity (a cruise past the repetitive, nearly identical homes in a "hideous" 1950s housing development) the power of the drug re-asserted itself.

It suggests that for near vision, some drug-induced problem with accomodation or adaptation (aperture) -- something in Huxley's eye -- was distorting the normal geometrical relationship between the Fourier plane and the image plane. With far vision, the long view, the drug lost its power (we hypothesize) to paint or spill the Fourier plane onto the image plane detector. This seems to be something happening in the eye, rather than in the unlit parts of the brain.

We actually know a little about Aldous Huxley's eyes. They were damaged by a keratitis in his youth. He was in his late 50s in the early 1950s, when he was working on the Doors of Perception, so his eyes' ability to accommodate would have been gone. This means that for near objects, one would expect to see some distance open up at the retina between the image plane and the Fourier plane. In the extreme case, for a very near object, the image plane might retreat behind (that is, fall off of) the retinal receptors. If both planes were retreating for some optomechanical reason, the Fourier plane might fall onto the image plane detector, and thus become visible in full-spectrum technicolor.

The thin lens equation is not enough to account for the difference in mescaline's near/far effects, but maybe it hints at a mechanism, or a way to guess at a mechanism. Withal, a fascinating problem.

<<PREVIOUS CHAPTER NEXT CHAPTER>>

2006-11-18T09:10:00.000-08:00

PREVIEW OF THE BOOK:

In the early 1990s, our long accepted (cc 1926) understanding of how a nerve encodes and conveys information was unexpectedly overturned by experiments on fast flying bats and insects. Around 1995, we began to realize we no longer knew what neurons actually do.

In the seventeen years since, many competing hypotheses have been advanced, suggesting various alternate neural encoding schemes. But the question of how a nerve communicates remains unanswered. It is a huge, gaping hole, at the most basic level, in our understanding of how the nervous system works.

This book is about what would happen if, as a thought experiment, we were to rewire the human nervous system using a multichannel neuron. It explores the impact of this hypothetical "smarter" neuron on vision, memory and the brain.

The ancient anatomist who first isolated a big nerve probably thought it was an integral structure – one wire. Closer scrutiny revealed the nerve comprises a bundle of individual neurons. We are proposing yet another zoom-down in perspective, this time to the molecular level: In this model, each axon within a nerve bundle is itself composed of about 300 longitudinal channels created by linking adjacent sodium channels.

Multichannel neuron membrane. A single channel marked in orange is firing. The device is an analog-digital hybrid.

A nerve impulse traveling along a multichannel axon is digital on its face. It is an all-or-none spike. To a voltmeter, each nerve impulse looks like every other nerve impulse. Its analog information content is unsuspected and undetected. But to the brain, each arriving nerve impulse communicates an analog gradation denoted by a specific channel number -- an integer such as 2, 3, 4, 17, or 131.

The theoretical payoff is enormous. Suddenly the brain, which operates on impulses moving at velocities barely better than highway speeds – becomes in theory a dazzlingly fast and competent thinking machine. Which is, of course, exactly what the brain is in real life.
A single longitudinal channel modeled by linking adjacent sodium channels. Each sodium channel is represented by its four homologous transmembrane domains and is linked to its neighbor by a protein moiety. Click to enlarge, back to Return.

The new neuron model requires that we imagine adding an additional fillip – a protein moiety -- to the well-understood biochemistry of the sodium channels. Small and subtle changes at the level of the neuron have billionfold consequences, and so one quickly arrives at a very different neuroscience.

Thirteen chapters have been completed and posted. The fourteenth chapter is currently in progress.

1. One Spike is Enough. In 1995, our long accepted understanding of how nerves encode information was unexpectedly overturned by experiments on fast flying bats and insects. Seventeen years later, no alternative neural code has emerged as definitive. What if no code exists?

2. The Corduroy Neuron. A more sophisticated neuron is one possible solution. How it works. What it requires at the membrane level. The model is codeless. No encoding, no clocking, no processing, no decoding. A much faster way to move information. The model is able to explain why a nerve’s firing frequency will sometimes vary with stimulus intensity, as Adrian discovered in 1926. And why, under other circumstances, a single spike will suffice.

3. Eye-Memory. Why the retina was once regarded as a prototypical memory organ by computer designers, notably John von Neumann.

4. The retina conserves spatial phase. What is spatial phase? What would you see if the eye conserved it? Everyday reality.

5. Gems in a junkyard. Double diffraction imaging and the classical holographic memory theories of 1963-73.

6. What does a memory look like? The retina reconsidered as a bipartite system, able to capture both an image plane and its corresponding Fourier plane simultaneously. What we see and what we store.

7. Standing waves in photoreceptors: An obscure but fascinating literature suggests both rods and cones could detect color, intensity and spatial phase. And yes, they did say rods could detect color.

8. Rods and cones as wave detectors. To detect standing waves the photoreceptor cell, which is conventionally understood as a particle detector, must operate as a wave detector. It absolutely requires a mirror. How does the ribbon synapse function in this system? Sodium pump inversion. Nerve impulses that can travel without action potentials.

9. Eye evolution. Eyes before lenses. Why the retina looks backwards. Imagine two optical systems, a mirror and a lens, crammed into one eyeball. The mirror is now just a vestige of the original, but perhaps it set the course of vertebrate eye evolution.

10. Reverse transcriptase writes memories. Memory as a massive infection of the cerebral hemispheres. The 30 fossil retroviruses in the human genome, and the recording machine they brought to us.

11. The retina of memory. The Victorians saw the human visual memory as a photograph. They thought the brain was clicking away constantly though the aperture of the eye. This charming idea was obliterated in the mid-20th century by Hubel and Wiesel and the higher concept of “feature detector” neurons, but this idea, in turn, took a torpedo in 1990s. Maybe the Victorians were right.

12. The mind as an eye. The brain modeled as a functional replica of the eye. In this hypothetical, lightless eye of the mind, what structure corresponds to retinal photoreceptors? What does arborization accomplish?

13. How does visual memory work? In the familiar neural network model of memory, the memory is distributed and there are no addresses. In contrast, in a biological memory store modeled as "a thing in a place" that place has a distinct address. It is numerical.

14. (in progress) Problems and observations. How and why memory could end up in DNA. What the sequence might look like.

2006-11-18T08:52:00.000-08:00

#10
Reverse transcriptase writes human memories

The idea has been lofted, and is lofted again here, that reverse transcriptase literally writes memories: not just genetic memory, but also immune memory and memory in the brain. It goes very much against the grain of course but if you can suspend disbelief for a moment, you may discover the idea has a certain fascination.

The background news that makes this idea worth bringing up, again, is the sudden new importance of Alternative Splicing -- and the still rather obscure discovery that we humans contain the genetic fossils of 30-odd ancient retroviral infections.

The engine of each of these old viruses was, of course, reverse transcriptase. In some of the viral sequences the engine is now missing but in others, it is still in there.

When the draft of the human genome was published in June, 2000, it revealed at least two unexpected results. One is famous. Not enough genes. The other unexpected outcome was less remarked. Maybe it was less of a surprise, but it certainly surprised me. It shows that not quite half of our DNA was "written back" from RNA. All this retro DNA was written into the genome by the remarkable enzyme reverse transcriptase.

Reverse transcriptase has almost doubled the genome. It did most of this work a very long time ago. It was mainly an internal process of retrotransposons reproducing themselves -- but some of that retro DNA, to be exact 4.7 percent of the human genome -- was written into our chromosomes by reverse transcriptases inherent in retroviruses.

We are infected.
We know of only 4 modern, active human retroviruses: Two are HTLVs (human T-cell leukemia Virus) and two are HIVs. These four familiar viruses are exogenous, not endogenous – they initially arrive from outside the body.

Scans of the human genome have shown that our DNA contains the residues of about 30 ancient, now endogenous retroviruses – 30, that is, thus far identified. There are probably more. Anyway, there you have 4.7% of your own personal DNA: Retroviruses. Say yikes.

A virus exists to write and rewrite itself. Retroviruses can infect many different types of cells, but when they infect germ line cells, in the testes or ovaries, they can write themselves into the genome – to be passed from generation to generation. In this sense they have the power to violate Darwin’s dictum, ‘No inheritance of acquired characteristics."If the "characteristic" you acquired happened to be a retrovirus you could indeed pass its DNA sequence along to your offspring and theirs. In this way, a virus becomes endogenous. By infecting the germ line, it takes up permanent residence in the genome.

They are called HERVs, for Human Endogenous RetroViruses. Some are ancient indeed, on the order of 30 or 100 million years old. They infected us before we were human.But some HERVs are of relatively recent origin – which is to say, recent infection. One entered the human germ line just 200,000 years ago. This particular virus is lodged uniquely in the human genome – it is not to be found in the genomes of our cousins, the chimps and gorillas. Potentially it is dangerous because it is still, apparently, in good enough shape to be or become infectious.

Are we sick with prehistoric viruses? Could they kill us?
Logic and scrutiny of the sequences suggest that the older retroviruses have been pretty much detoxified, tamed. In some cases their most dangerous sequences have been pruned out altogether. In others, the sequence is too drifted and degenerate to work.

The systematic transcription of any really lethal retroviruses seems unlikely. (If these retroviruses were killers, that is to say, quick-killers, they themselves would not have survived in our genomes, since by killing their hosts they would have themselves committed suicide long ago.)

Probably some of the ancient retroviruses are in fact involved in disease processes. Its not clear how. It is not even clear in every case who’s side they are on, that of the host or that of the disease.

The younger the retrovirus, the more likely it is to be dangerous in the conventional way, since its infective machinery may be intact or restorable.

There is evidence that ancient retroviral genes are able to promote or enhance or alter the splicing of other genes. Considerable research is now focused on the effect, on normal gene transcription and splicing, of nearby or intrinsic viral sequences (promoters, enhancers, etc). In these models, the HERVs are involved in, maybe, launching disease processes that are not well understood: MS, diabetes, cancers, and autoimmune diseases are candidates.

There exists the possibility of another level of control, here. And another possible way to make trouble. Since we all contain retroviruses, and since they seem to be passive for the most part – it seems reasonable some biochemical machinery evolved to put them out of business and keep them out of business. A disease could start when the machinery who’s job it is to inhibit or inactivate the endogenous retroviruses – glitches.

Captured weapons
One fascinating idea is that our endogenous retroviruses have not merely been domesticated -- but actually turned into weapons against disease. For example, the expression of an endogenous viral protein (a coat protein) has been found associated with Multiple Sclerosis.

The first thing you might think of is that maybe MS follows from a revival of an ancient retroviral infection. A different explanation, however, is that the body is borrowing the old virus’s capacity to manufacture coat proteins – then using them to fight the disease by blocking cellular receptor sites.

The fundamental idea is that we have probably turned our endogenous retroviral code to advantage in defending ourselves against disease. This is a very good idea but it perhaps understates the possibilities for cooperative arrangements -- deals that may have been cut, over the eons, with our inborn HERVs.

Retroviruses have the power to break all the rules -- to both store and spread biological information within an organism. What couldn't we do with that?

What else could you make from these rusty retroviral parts?
Reverse transcriptase is one characteristic enzyme all active retroviruses encode. They use it to write themselves into the genome. It was discovered independently by Temin and by Baltimore in 1970. Reverse transcriptase is the only enzyme of which one could say: "It writes memory." Although this verbal construct makes some people deeply uneasy, it is unarguable.

In theory the retroviral mechanisms have the power to both write and broadcast memory, through the process of infection, from cell to cell. If our ancient endogenous retroviruses have indeed been tamed, de-toxed and put to work doing something useful in the cause of human physiology -- this would be an attractive and natural job for them: Writing memory.

Memory as an infective process
Reverse transcriptase has quite a history of high performance and remarkable feats. It was responsible for nearly doubling the genome. And it is still in us, however suppressed or motheaten or domesticated -- this strange and dangerous writing enzyme. Constrained to do strictly boring work, it is the engine of the enzyme telomerase. A waste of talent. Couldn't reverse trascriptase be made to do something much more interesting?

Perhaps we could revisit (knowing what we know now) Lineas Pauling’s long shelved notion of an "instructive" immune system; the discovery of "immune RNA"; and the swirl of ideas in the 1960s surrounding the notions of DNA and RNA as memory media for the brain.

There is a parallel stream of ideas to the effect that memory might be an infective process. It takes several days to consolidate memory in the brain, and it was once imagined that this long period of time was needed for a process of distribution, perhaps an "infection" of the brain tissue with new information.

In neurons, synaptic vesicles are formed, fuse with the cell wall at a specific pore site -- evert and release their contents into the synaptic cleft. A message is thus delivered to the neighboring nerve cell. The machinery of the synaptic vesicles is suggestive of the viral budding and viral invasion processes. Pockets of RNA have been found at the nerve endings. What's RNA doing out there, so far from the nucleus? It is thought to support local protein synthesis, but is that the whole story?

Maybe the clonal response isn’t one. Maybe it is instead a massive infection – or counter infection – that transfers and broadcasts the code for an appropriate antibody.

If you were going to invent memory…
In the mid 1960s, when the notion of biochemical memory enjoyed a brief vogue, they knew DNA was a stabile long term storage medium, and they had the most basic form of the Central Dogma to describe gene expression:

DNA => RNA => Protein

To turn a machine for expressing genes into a recording machine for sequences, it seemed reasonable, as a place to start, to visualize a reversal of the central dogma, so that somehow:

Protein => RNA => DNA

This was before reverse transcriptase had been discovered, so both of the arrowheads seemed hopeless -- there was no known way for the process to proceed. Most biochemical memory enthusiasts focused, therefore, on RNA, some on protein. In 1965, the whole pursuit ended in a failure-to-replicate paper, and scandal, and that ended all that.

To follow in the footsteps of the original line of reasoning, today, suppose we were to begin as they did, with an innocent eye and a reversal of the central dogma.

As a first step, re-sketch the central dogma to reflect the reality given us by the published genome. Instead of one gene => one protein, we must now make an adjustment to show that one gene => 2 or 3 proteins. Or, in the extreme case, one gene => tens of thousands of different proteins. So let’s just write:

DNA => mRNA => Protein 1 or Protein 2 or Protein 3

To show that one gene codes for a cluster of possible proteins, depending upon how you splice it. To make a biological recording machine you could reverse the central dogma, as follows:

Protein 1 or Protein 2 or Protein 3 => RNA => DNA

That second step, RNA => DNA, is what reverse transcriptase does. The first step, from protein back to RNA, still does not seem possible.

But what do you really want to record as a memory? Probably not the protein, per se, but its identity -- and its identity reflects a choice that has just been made between or among various alternative protein domains.

Notebook RNA
A useful immune memory, for example, could consist of a code that simply identifies or points to the protein sequence of an antibody -- an assembly of domains that works perfectly against a specific antigen. You don’t have to remember and reiterate the whole sequence. You just need to be able to call its name, loudly, in an emergency.

It is possible to imagine ways to keep an RNA record of what proteins are being made – as they are being made. That record, essentially an RNA notebook on protein production, could be written back into DNA by reverse transcriptase. The notebook would tell you which proteins had been made, how (i.e., how their mRNA was spliced), and in what order they came off the line.

How might such a notebook be kept?
The gene splicing process leaves specific introns behind, like a dressmakers cuttings on the floor. If you collect the cuttings, you can readily reproduce the shape of the dress.

From a sequential collection of introns, you can re-construct the program used to make a specific protein. And it would not be necessary to actually gather in all those introns. Tags, consisting of just enough sequence to uniquely identify each intron, would suffice. A notebook in shorthand RNA becomes the protein factory record -- the thing to be memorized.

As a final step, reverse transcriptase writes the RNA notebook into DNA, that is, into the stabile long term memory storage medium.

Cut and Try
If you integrate this reversed-dogma recording machine with Pauling’s old, outlawed idea of an instructive immune system, you can begin to imagine how a process of protein cut-and-try against the template of an invading antigen could lead to the construction of an appropriate, perfectly fitting antibody.

The missing element in the story thus far is a signal of success. Somehow the winning antibody has to be associated with the specific RNA notebook of splicing instructions that produced it. If you then reverse transcribe the RNA notebook on how to construct a successful antibody into DNA, which is to say, memorize it -- and broadcast it through the immune system using a process analogous to infection, voila: Immune response, immune memory.

Meanwhile, the considerable body of notebook RNA describing all the antibodies which did not work can be digested and, well, forgotten.

The story treads much the same path as the "Generation of Diversity" narrative familiar from immunology. Scrambling of sequences, cutting and trying. But it discards the Darwinian notion that one cell produces one antibody pre-destined to greet one antigen. All the cells in confrontation with a fresh antigen are scrambling their sequences all the time. And it replaces the clonal response with cell-to-cell communication -- a massive infection with the right answer, the ideal sequence.

We are just surfing along on new facts, here -- the wildly interesting possibilities inherent in the HERV machinery and in alternative splicing. But here is an old fact that pertains.

Immune RNA
An RNA that seemed to instruct the immune system was actually reported, in 1957, and became something of a cause célèbre. It was called immune RNA. It is largely a forgotten finding, now, but it was never successfully controverted. Only scoffed down.

The most telling criticism of immune RNA was that the molecule was too short, too small, to describe an antibody. Even though it apparently did exactly that. This criticism now resonates as a promising observation -- it is exactly what one would wish to hear about a notebook RNA, which only encodes a splicing program. It is also pertinent that it was discovered wandering from cell to cell, and could be used experimentally to coax appropriate antibodies from still other cells.

If one sifted through new found knowledge about spliceosomes and HERVs, perhaps all parts and pieces for a memory machine are actually in there. If it is RNA that must be memorized, then the final common path to memory has to be reverse transcriptase.

Reverse transcriptase makes a lot of mistakes. In some applications, perhaps in the immune system, this error prone work could be useful, a way to manufacture variety. In others inaccurate transcription would have to be policed, corrected.

Secrets of successful viruses?
Suppose that, by whatever means, the immune system were instructive, that is, creative. Viruses confronting such a system will have evolved ways to beat it. One way would be for the virus to change its identity (cell surface antigens) faster than a creative immune system can retool its antibody production. Not a new tactic, but a new way to explain it. The virus could monkey-wrench some step in the creative process.

Not incidentally, an attack on the immune system's "creativity" would automatically and completely elude detection in the lab, since immunology does not recognize the possibility that a creative immune system might exist. Straight under the radar.

There is an excellent long article on HERVs by Michael Specter in The New Yorker of December 3, 2007. Specter interviewed an eclectic selection of HERV researchers and interested commentators, including Thierry Heidman, John Coffin, Robin Weiss, Paul Bieniasz, Harmit Malik, Michael Emerman, Aris Katzourakis, Robert Belshaw, and Luis P. Villarreal. If you are interested in learning more about this field, or thinking about getting into it, there are a number of additional labs whose work should be searched, including the Eugene Sverdlov lab in Moscow and the Jack Lenz lab at Einstein. Sverdlov's recent book, Retroviruses and Primate Genome Evolution, is a helpful resource.

RNA is a far a more capable system than we once imagined. For new thinking on this broader subject, a number of starting places can be found in this overview piece in the March 28, 2008 Science: The Eukaryotic Genome as an RNA Machine by Paulo P. Amaral, Marcel E. Dinger, Tim R. Mercer, John S. Mattick

What about memory in the brain?
Biological information is typically stored as sequences and shapes. There is no reason to imagine that information in the brain should be stored in some entirely different way. But in the immune system, the thing to be memorized, which is the shape and sequence of a specific antibody, is clear.

To make some guesses about how reverse transcriptase could write memory in the brain, where the thing to be memorized remains a complete mystery -- it will first be necessary to think through the consequences of Radical Idea #2, the multichannel neuron.

One should probably start with the photoreceptors of the retina, which are specialized neurons.

The eye is the last place in the brain where anything can happen at the speed of light. What happens is image formation and everything that image formation entails, including Fourier transformation. It may turn out that the "thing to be memorized" is a transformed image, or perhaps a geometric and a transformed image simultaneously.

Ghosts in the machine
Some old, toothless and discredited ideas from the biochemistry of four, five and six decades ago, like the old, supposedly de-fanged retroviruses we still carry with us, may be – after all -- stirring restlessly in the museum by night.

Many of the potential talents and implications of HERVs are upsetting, in the sense of upsetting the applecart. The power to self-infect, to write and broadcast memory from cell to cell within a mature organism, even from generation to generation -- threatens bedrock notions of the clonal response or the synaptic memory or "no inheritance of acquired characteristics". The very idea of memory as an infective process is dangerous. In this sense, if you like revolutions or comeback stories, HERVs could turn out to be quite delightful.

<<PREVIOUS CHAPTER NEXT CHAPTER>>

2006-11-18T08:27:00.000-08:00

Chapter five
Gems in the junkyard

The idea that holography might be a useful metaphor for understanding memory in the human brain was first proposed by the Dutch optical physicist, Pieter Jacobus van Heerden, in April, 1963, in two back-to-back papers in the Journal of Applied Optics. At the time he was with Polaroid in Cambridge.

Van Heerden presents in the second paper a technology for the optical mass storage of holographic images in a solid block of photosensitive medium. In both papers he describes a technique for the nearly instantaneous retrieval, from the solid, of holographic images. There is no sorting, no indexing. It is a "content addressable" memory. Presented with one sample image, the van Heerden system can instantly identify any similar image in storage, and project it.

The image to be matched does not have to call up an exact copy – only a similar image. Degrees of similarity can be detected and sensed. A simple threshold detector can be set to retrieve only a few images of the very highest similarity. Or it could be set unselectively, to retrieve many sort-of-similar images.

The system will also work for a larger image containing a similar image – a single face in the class picture. Having invented hardware for an optical memory, van Heerden recognized and explained that the machinery he had conceived was able to do pretty much what the brain does. Here is a review of a modern optical storage and recognition system of a very similar type, designed to recognize fingerprints .

In as little as 400 milliseconds, a human being can recognize a face in a picture of a crowd and push a button to signal this recognition. Van Heerden was quite interested in the face-in-a-crowd feature of his invention. Seven years later, in 1970, van Heerden got into a little public skirmish about it with a connectionist in an exchange of letters published in Nature. The connectionist had written an article urging that neural nets could accomplish everything then claimed for holographic memories.

Van Heerden responded that a holographic memory could do something the neural nets could not do – recognize and pick faces out of a crowd. The connectionist replied that these were early times (1970) – too early to say what one system might do and what another might not do.

A holographic memory can indeed, in the hardware version, recognize a face like lightning. Of course both optical holographic hardware and neural net software operate, at the wafer level, at the speed of light.

In his original papers Van Heerden suggested some reservations about too literally projecting his optical memory onto the brain – and they are, in retrospect, familiar objections. "The quantitative theory points out the necessity of maintaining exact phase relationships in the waves over long distances. Any change in the speed of propagation in one part of the neuron network compared to the speed in another part would scramble the information in a hopeless way. It seems therefore necessary to assume a calibrating system of pulses in the brain, which at all times checks this and compensates, if necessary, the speed of propagation…"

"Maintaining exact phase relationships" is the core principle of holography. The difficulty of accomplishing this in the brain dogged the biological holographic memory theory from the beginning, and in the end it was probably this problem – inexact phase relationships in the brain -- that pegged it as an implausible idea.

Van Heerden and Stephen Benton were well acquainted, and there is a little bit of biographical material about van Heerden in an essay of Benton’s on the history of holography. For a thorough summary of van Heerden’s thinking about memory in the brain, and for the best overall history of the idea that a hologram might be analogous to human memory, see Chapter 7 of the wonderful book, Metaphors of Memory by Douwe Draaisma.

Karl Pribram, Karl Lashley, and Donald Hebb
The idea that memory is holographic was independently conceived and advanced a second time, in 1969, in an article in the Scientific American, by Karl Pribram, a researcher and neurosurgeon who was then at Stanford. The emphasis of the Scientific American article was on the holographic mode of storage, rather than retrieval. Pribram focused on the fact that holographic storage is distributed. Any tiny part of the film on which a hologram is recorded can be used to reproduce the whole image.

Karl Lashley had suggested in his famous essay, In Search of the Engram, that the brain must store a memory image everywhere, throughout its substance, rather than in localized areas or specific circuits. A brain can be wounded or physically damaged in various ways, sometimes massively damaged – and yet retain its power to remember images. Lashley concluded that "It is not possible to demonstrate the isolated localization of a memory trace anywhere within the nervous system. Limited regions may be essential for learning or retention of a particular activity, but within such regions the parts are functionally equivalent. The engram is represented throughout the region." [Italics added].

It was Karl Lashley who, in 1942, first suggested the idea that a memory might be stored in distributed fashion in the form of an interference pattern. He died in 1958.

Karl Pribram, who had worked with Lashley, recognized in the late 1960s that the holographic technique provided a physical model that could explain the apparent distribution of memory throughout the medium of brain tissue. In his long, intriguing retrospective essay, Brain and Mathematics, Pribram recalls the origin of the idea that memory might be stored as an interference pattern – and also, perhaps, the origin of the deepset conflict between holographic memory theory and the Hebb hypothesis.

Hebb's idea, which was the sourcepoint of the dominant view of memory today, was that the nervous system rewires itself (though facilitation at the synapses) in response to new experience, and that the "grooved in" or facilitated new pathway constitutes the memory. According to Pribram's very politely phrased account, Lashey could not buy it:

"Lashley (1942) had proposed that interference patterns among wave fronts in brain electrical activity could serve as the substrate of perception and memory as well. This suited my earlier intuitions, but Lashley and I had discussed this alternative repeatedly, without coming up with any idea what wave fronts would look like in the brain. Nor could we figure out how, if they were there, how they could account for anything at the behavioral level. These discussions taking place between 1946 and 1948 became somewhat uncomfortable in regard to Don Hebb's book (1948) that he was writing at the time we were all together in the Yerkes Laboratory for Primate Biology in Florida. Lashley didn't like Hebb's formulation but could not express his reasons for this opinion: 'Hebb is correct in all his details but he's just oh so wrong'. "

In subsequent decades Karl Pribram worked so long and faithfully to develop and make visible and plausible the idea of a holographic memory that most people associate this theory most strongly with Pribram. Until recently, when I read Metaphors of Memory, I did not know the theory had had an earlier protagonist, van Heerden.

Holographic memory: the hardware version
Holographic images on film are memories of a sort. One could say this of any type of photographic image on film, but holograms are special. They are 3-dimensional, but there is much more to the metaphor.

Let's suppose you record on film a holographic image of an ordinary object, perhaps a book on a tabletop.

The details of the setup are important.

Between your eyes and the book, you would place a film holder.

To illuminate the book, in order to take this hologram, you could scatter the light of a laser beam off of another object in this scene -- so let’s place a coffee cup next to the book. The laser can be mounted on the tabletop at your right hand, so you can aim it into the scene obliquely. Note that the laser should be aimed at the coffee cup, not directly at the film. Turn on the laser, to shoot the picture, then turn it off.

Now remove both the book and the coffee cup. Leave the laser in place. Develop the film; re-mount it in the same film holder. And then, finally, turn on the laser once again. The developed film is a hologram, but nothing appears when you project it.

Now pick up the book and return it to its original position in the scene, so that it reflects light back through the film. Immediately an image of the coffee cup appears. And it will appear "lifelike," in three dimensions, quite indistinguishable from the real coffee cup. If the coffee was steaming when you took the picture, the steam will be recreated too.
Through the physical mechanism of the hologram, one object (the book) has somehow conjured up the recorded image of another object (the coffee cup) that has been associated with it in the past.

When the real book is removed from the scene, the image of the cup will vanish. If we now carefully replace the holographic image of the cup with the real cup, an image of the book will appear.

This is a magic show. But the idea of bringing back, from storage on film, the fully realized image of one object -- simply by introducing into the scene some other object that has been associated with it in the past -- this seems an intuitively exact model of how our memory works. Think of an old stamp and your memory conjures up an album. Think of a collar and your memory fills in the dog.

So holography can be used to give a splendid demonstration of a content-addressable associative memory.

The book-and-cup demonstration depends on a coherent light beam and upon precise positioning and re-positioning of the objects.

In this system, however, coherent light that has been scattered from each object is acting as a phase reference beam for light scattered from the other object. In this sense it is a little more like the real world than most holographic setups. More typically a pure reference beam – never scattered -- is used as a phase yardstick. In the book and cup demonstration, the reference beams are impure, worldly. Each object "references" the other.

Distributed Memory understood as a lens
Suppose you snapped a holographic picture of a coffee cup. But this time, use a pair of scissors to snip the film into a hundred little squares.
Any tiny square of the film can be set up and illuminated with a coherent laser light to reproduce the image of the whole coffee cup. Somehow, the image of the cup has been stored on the film in distributed fashion, so that all the information needed to recreate the image of the cup is stored in each tiny square of the film.

It's less mystifying that it seems. Recall the insight of the physicist Gordon L. Rogers. A hologram can be thought of as a lens created within the substance of the film. This lens focuses the illuminating light of the laser to form the image we see as a coffee cup. In other words, a hologram is a form of lens that specifies a particular image.

Consider for a moment an ordinary glass lens that has been broken into several pieces: thin spectacles that have fallen on a sidewalk, for example. Any little piece of a lens you might pick up and peer through still acts as a perfectly good lens. In the same way, any little piece of a hologram will suitably focus a laser to recreate the image stored on the film.

The brain can be broken or cut to pieces by injuries or disease or surgical procedures. The memory survives intact. Whole segments of the cerebral hemispheres can be cut out and discarded. Memory still clicks.

Karl Pribram remarked in interviews that in some experiments, as much as 98% of an animal’s neuronal wiring could be severed in the visual pathway – yet the animal persisted in recognizing, from memory, an image.

How can this be? The holographic concept provides a model which explains such effects. If memory works like a lens, or hologram -- if it is stored as an inteference pattern -- then the memory is distributed. Each and every "part" of the brain contains the whole of the memory. Thus a memory cannot be extinguished until almost 100% of the tissue is cut away. In the theory as presented, incidentally, the "parts" of the brain are evidently understood to be neurons – not just cells in general.

Distributed memory storage is quite familiar as a biological concept. We know that the genetic memory can be divided and redivided by mitosis and yet remain intact. Each cell of every multicellular organism contains, in its DNA, all the information necessary to reproduce the entire organism.

A telling point for connectionism
Distributed memory was perhaps the strongest suite of the holographic memory theory. The same feature, however, can be accounted for more readily in neural nets, in which a specific memory can be said to be stored across multiple synapses. Moreover, it is possible to visualize and demonstrate how the neural net mechanism works.

With the holographic model, the distributed memory store is contained in an interference pattern. This idea works fine for light sensitive film -- but it is hard to see how it could work in the nervous system. Since no wave interference could occur on the digital part of the nerve, the axon, it was urged that the dendrites provided places where electrical wave interference might occur. The idea of wave interference in the nervous system is attributed to Karl Lashley but it was elaborated by R.L. Beurle, who published it in 1956 in the Philosophical Transactions of the Royal Society of London. But it must have seemed a nebulous argument or at best, impossible to prove.

In contrast, the principle of distributed storage in neural nets is easy to illustrate with a breadboard circuit, and to see and admire in action on the screen of your computer.

Historically, it is clear that the evident superiority of neural nets in explaining the distributed storage of memory kicked one of the strongest props out from under the classical version of holographic memory theory. From reading Douwe Draaisma's Metaphors of Memory I have formed the impression that this was an important turning point, or fork in the road, in this science. Neuroscientists he interviewed who lived through that era recalled that they were strongly impressed by the principle of distributed storage urged by the holographic memory theorists. When neural nets turned out to offer a distributed storage system based on a different, somewhat more comprehensible principle, they recognized the concept and crossed over to connectionism.

Let me emphasize that neural nets do not offer a superior solution to the problem of distributed memory. They do not. What connectionism provided, in the 1970s, was a more plausible explanation than the holographic model, given the textbook neuron. I think the holographic idea stalled out, around 1973, because a believable and biologically realistic model for a holographic memory could not be constructed using 1-channel, all-or-none neurons. The all-or-none nervous system was "not analog enough" and not synchronous enough and not fast enough to support holographic processing.

If we posit a multichannel neuron, however, the classical holographic memory concept, or at least the underlying principle of storing interference patterns as distributed memory, becomes plausible and accessible once again. We cannot reset our thinking to 1973, but we can see these shelved ideas about distributed memory storage in a new light, and see what we might cherry pick from this once very influential collection of concepts.

Memory storage and retreival
In addition to distributed storage, the holographic memory theory had several other intriguing features we should revisit. These advantages followed from the application, in the theory, of the Fourier transform.

We can be certain that the animal ability to perform a Fourier transformation evolved as soon as nature created the first lens. The transform is accomplished at light speed between the diffraction plane and the image plane of a lens. The lens of course does not calculate like we do. The Fourier transform is just a mathematical description of what the lens does to light waves.

One speculates that the power to do both the Fourier transform and to reverse its effect, by the inverse transform -- by other analog means, inside the brain -- is probably ancient as well, and may in fact antedate the lens.

In any event the Fourier transform, understood as a biological process and a recurring theme in natural history, is vastly more ancient than Fourier himself, the revered French mathematician who worked through the technique by hand for the first time in the 19th century.

The Fourier transform was the read-in/read-out machine for the theory of holographic memory. The transform converts a literal image into an interference pattern. Run it again, inversely, and it converts an interference pattern (a memory, per the theory) back into a literal image. Here is a delightful example, using a literal image of a duck:

And here is its Fourier Transform, an interference pattern that contains the same information as the literal image, but encodes the duck in a different way:

The Fourier duck, and the method of portraying the transform in color, is to be found at Kevin Cowtan's Book of Fourier, along with several other playful Fourier creatures and transforms.

The transform can be accomplished by calculation, but it can also be accomplished with hardware – by a lens.

In addition to transforming images for storage, and reverse transforming memories into images, the Fourier transform can be used to accomplish two other useful things: image cleanup, and feature recognition.

The problem of image cleanup
Light must pass though a tangled nest of the eye's own wiring on its way to the photoreceptors of the vertebrate retina. These structures are fairly transparent and do not completely block the view (we can see, after all) but because they are inhomogeneous they scatter incoming light.

For example, in the 200-micron thick turtle retina, scatter increases the diameter of an incoming parallel beam from 8 microns as it enters the retina to 50 microns at the level of the outer segments. In anatomical terms this means a neat incoming beam the diameter of a single photoreceptor will cone out, in its passage through the retina, to illuminate 5 photoreceptors. In the process the intensity at the center of the beam diminishes by tenfold.

The result is a degraded, blurry image. Turtles have an especially thick retina but all vertebrates eyes must solve, or live with, the problem of image degradation from light scattering from the tissues positioned ahead of the photoreceptors.

Click here for a really excellent SEM of the retina; This photo is on the U. Delaware's online histology site. It shows clearly the positioning of the photoreceptors near the bottom strata of the retina, the thick layer of jello salad overhead, and the photoreceptors' apparent orientation -- pointed at the backwall of the eye. It is commonly said, for this reason, that the vertebrate retina is "wired from the front."

The Shower Glass Effect
The neatest way to clarify the view obstructed by intervening nerves, vasculature and glia would involve an application of the Fourier transform. The idea relies on a principle called the Shower Glass Effect. Imagine a three dimensional hologram taken of a bather who is obscured by frosted or opaque glass, taking a shower. Because the hologram is 3-dimensional, the observer, or voyeur, can change points of view. By moving far enough to one side, the observer can simply look behind the glass panel. The clear view of the bather is obtained even though the holographic recording system was staring directly at an opaque, distortion inducing glass shower door.

What this tells us is, the bather is included in the information captured and stored in the hologram. The hologram can be processed – filtered – in such a way that the intervening visual obstruction is removed.

The mathematical solution of the Showerglass Effect is useful in improving the optical clarity of military intelligence photographs shot from space, through the opaque screen formed by miles of air intervening between the camera and its target. It is similarly useful in astronomy.

One argument for some sort of Fourier processing in the mind is that we all stare at the world through a showerglass all the time. The showerglass is the messy intervening layer of nerves and tissue that obscures all but a tiny spot on the retina from the rest of the world, and visa-versa. Somehow, the brain sees around all this goo, and successfully "subtracts it out" of every scene.

Fourier filtering, feature detection.
Photographic images can be converted into interference patterns, and then converted back into an image, using the Fourier transform. A transformed photo, which is an interference pattern, can be filtered in certain ways and then Fourier transformed back into a snapshot. The effects are fascinating. These techniques can be used to improve clarity, enhance fine detail, and to detect, in aerial surveillance photos, recurring patterns of high spatial frequency (e.g., straight lines not of natural origin, like gun barrels or fuel barrels).

One of the most interesting capabilities is edge detection. Using the transform, one can in effect turn a person into a cartoon – distinguishing a person from the photographic background with a clear outline.

Richard Gregory, in his classic book Eye and Brain, writes extensively about the brain’s hunger for objects. He shows with optical illusions that the brain will insist on discovering objects, even in scenes where none exist. The Fourier transform is a handy technique for sorting objects away from background.

An Aside:
FOURIER OPTICS LINKS AND RESOURCES

We are interested here in Fourier filtering because we suspect the brain does Fourier filtering routinely – and that it has been developing this skill for many millions of years. Mixing, matching, filtering and transforming images are, one speculates, the core brain processes that underly discerning, thinking, remembering, and imagining. Conscious thought may have evolved from the most basic Fourier filtering task -- cleaning up an image.

Where to start
As John Brayer remarks on his helpful site, the best way to get a feel for this technology is by looking at lots of images in tandem with their Fourier transforms. Scroll down to the photos of the lady in the hat.

Learning Fourier Optics
A detailed journal kept by a gifted student over a summer of studying Fourier optics and image formation. She identifies the most accessible book on Fourier Optics (Steward) and lists links to the most useful net resources on the subject.

If you decide to use Steward, I would suggest you start on pages 106-116, which should probably have been the introduction, and then back up to the beginning.

Fourier filtering sites
Most of the rest of these links show examples of Fourier Filtering. Some of the presentations are rather mathematical, but if you prefer you can scroll past the math -- concentrate on the photos.

To apply a Fourier filter, you start with a snapshot. This literal image is then Fourier transformed into an unintelligible but ordered pattern "in the frequency domain. " The Fourier transform can be accomplished optically or by a computer. Fourier Filtering consists of carefully modifying the (still unintelligible) image – and then Fourier transforming it back into a literal image of the original snapshot. When the second transform is completed the resulting image from the snapshot will look a little different – sometimes much better. The edges of the object or person in the photo may be strongly enhanced – the whole picture may appear much crisper and better resolved.

The Hypermedia Image Processing Resource is one of the best links for showing what can be accomplished with image filtering, etc. Be sure to click on the small pics.

Here is their discussion of frequency filtering. "Frequency filters process an image in the frequency domain. The image is Fourier transformed, multiplied with the filter function and then re-transformed into the spatial domain. Attenuating high frequencies results in a smoother image in the spatial domain, attenuating low frequencies enhances the edges."

The photos show what this means in practice. When they say "frequency" they are talking about spatial frequency, for example, the coarseness or fineness of a periodic object like a pocket comb or a diffraction grating. Again, scroll down. Be sure to click on the postage stamp sized photos to enlarge the images.

Finally here is an online short course on Image processing fundamentals.
Scroll past the mathematics to the images and transforms of the girl in the hat (a different girl, another hat). Notice the effect on these images of selectively isolating phase information and magnitude information.

Summary
What ideas or features can be salvaged from the old Holographic Theory of Memory?

Distributed storage of a memory as an interference pattern;

Memory storage and retreival by Fourier transformation back and forth between a stored image and its interference pattern;

Fourier filtering and image processing for object recognition, feature extraction and image cleanup.

And what have we lost?

It appears we lost Holography.

Because we have assumed the retina conserves spatial phase, we can dispense with the notion of a reference beam. We are now free to relax the requirements for a fixed wavelength and perfect coherence – laser properties. Let's see if we can now construct a memory that works, as vision does, for moving objects illuminated by white light of only partial coherence -- the objects and light of the everyday world.

<<PREVIOUS CHAPTER NEXT CHAPTER>>

2006-08-23T08:42:00.000-07:00

Chapter four
The conservation of spatial phase

Museum of Holography in New York. Photo courtesy of Paul D. Barefoot

Holograms are 3-dimensional photographs of uncanny realism. They are unlike a slide projected onto a screen. Objects reproduced holographically appear in mid air, and they appear exactly as they do in real life. If you shift your point of view, by taking a sidestep to the left or right for example, you can open up to scrutiny parts of the object that were previously masked. The only way to distinguish a well-made hologram from a real object is to put your hand through it.

In other words, you don’t need to actually see a hologram to know what one looks like. If you have moved through the world, you already know what a hologram looks like. It looks like reality.

When the technology of holography first came to be widely understood and appreciated in the 1960s and 1970s it was quickly seized as a metaphor for both human vision and human memory. What we see and what a holographic apparatus remembers and projects are virtually identical.

The holographic vision theories were quickly extinguished. They were clobbered in the 1970s with "insuperable objections" based on the eye's supposed inability to capture or conserve spatial phase. We will revisit this problem after we take a look at what spatial phase means, but I should point out here that I think the "insuperable" phase argument was completely specious. It is unfortunate the early holographic vision theorists were cut short. In essence, they were probably right.

Holographic memory theories have had a much longer run. One version, based on the curious and oddly popular notion that quantum effects must account for the workings of the brain, was launched around 1987, and still has adherents today. But the golden age of the original -- let’s say Classical -- holographic memory theory was probably 1963 to 1973.

The idea failed to stick for several reasons. It eluded most people -- the holographic memory theory was difficult to understand without studying Fourier optics. There were strong technical objections, arising from problems of coherence and phase reference, which made it seem biologically unrealistic.

And at some point the decline became sociological: The holographic memory idea, or analogy, lost credibility in proportion as neuroscientific enthusiasm tilted toward and then finally avalanched into neural networks.

At the end of the day, the holographic memory theory sort of flipped over and slid sideways into mysticism and quackery.

For example, it would be possible for you to attend a seminar in the Pacific Northwest on holographic memory. In this seminar you would learn that the body and the mind are the same thing, and so the body is included with the brain in the holographic memory. Each part of a hologram contains the whole, and the whole contains the parts. You are in the universe but the universe is also in you -- and so on in this vein.

Now, it seems that by squeezing the body in a certain way, pretty hard, one can elicit and purge various bad holographic memories. For several hundred dollars you can be squeezed to get rid of your own damaging memories and you can also, later, learn to do the squeezing, as a sort of profession. This knowledge will enable you to establish a profitable therapeutic practice in the business of purging holographic memories. Perhaps attaining oneness with the universe at the same time.

So there you have the wreck of an idea.

If you leaf through the early literature, you will find that holographic memory was once regarded as one more arguable but interesting and well reasoned scientific hypothesis about how the memory works. Today it is an eye-roller. I know I must be missing a chapter or two of this history. I don't understand how a simple scientific analogy turned into some sort of strange, quasi-religious ideology. But it did.

To see what might be salvaged, and to get a clearer look at the solid, unmystical and underlying technology, which is holography, let’s consider here the Museum of Holography.

The Museum of Holography
In lower Manhattan, in Soho, there existed for 15 years behind a storefront at 11 Mercer Street a museum of holography. A fine account of the museum, along with an excellent general history of the science and art of holography, is at the site of Holophile, Inc.

On the upper level of the Museum of Holography was an active art gallery displaying the artistry and novelty of holographic images, including some fine art. But the main attraction was a gallery of ominously realistic heads – holographic portraits -- of celebrities and atheletes including, for example, that of Mike Eruzione, captain of the 1980 U.S. Olympic Hockey Team. In fact, the heads of the whole team were on display.

The museum once had a show of holographic portraits of Famous New Yorkers. William Buckley and the young Tom Brokaw were included among the heads of the famous. No question, the gallery of celebrity heads was what most impressed visitors. You could not view the celebrities' heads without thinking of a wax museum and then, pretty quickly, of the French revolution, of Robespierre.

There was also some technological art, that is, pictures conceived as demonstrations of the strange power of holography to project reality. Or anyway, the reality perceived by human beings, the reality communicated to us by visible light.

On weekdays the museum filled with grade school children, who arrived in groups and toured the gallery in delighted, squealing clusters of threes and fives. They left with souvenirs: goofy looking diffracting sunglasses, tiny holographic charms and trinkets.

Down in the basement, precisely at the foot of the stairs, was positioned the full sized holographic image of a man, the Hungarian born physicist Dennis Gabor, seated as though he might have just then looked up from his desk in order to watch you come downstairs to visit him. Gabor invented the hologram. He was a small inquisitive looking man with a hooked nose and a walrus mustache. He wore glasses of distinctive design, with special thick arms that would block peripheral vision but shield the eye, perhaps from chance exposure to the blast of the laser beam used to create his image.

Dennis Gabor was a Nobel laureate and a stunningly gifted inventor and scientist. He was one of that famous group of expatriate Hungarian Jews who studied in Germany in the 1920s and then proceeded to astonish the world in the 1940s, mostly by inventing nuclear weapons. By coincidence, several of these geniuses were enrolled together in a course in statistical physics taught by Albert Einstein. The role call for this small class would sound, if we could hear it now, like a Who's Who of twentieth century physics: Leo Szilard, Eugen Wigner, John von Neumann, Edward Teller and Dennis Gabor.

These expatriate Hungarians were primarily theoreticians. Their lifeworks are inaccessible to most people. Perhaps for this reason they tend to be remembered their achievements in applied science. Szilard conceived the nuclear chain reaction, following the model of chain reactions published in the literature of organic chemistry. Wigner designed the first full scale nuclear reactors. Von Neumann – the air burst explosive, the implosion lens for the atom bomb dropped on Nagasaki, the hydrogen bomb, the digital computer, and the Game Theory. Teller, the hydrogen bomb. And Dennis Gabor, the hologram.

It is pretty unlikely, however, that Albert Einstein, who loathed schools and pedagogy, ever called the role.

In 1947, at the age of 47, Dennis Gabor created the first hologram. Notice that he did this work before the invention of the laser. But the techique begged for a laser, and when it finally appeared, in the 1960s, holography was enthusiastically re-discovered. Dennis Gabor died in 1979. In the basement of the holography museum in Soho, near beside the hologram of Dennis Gabor, was a hologram of the bound and embellished certificate proclaiming his Nobel Prize. When they switched off the power at night, both the man and his Nobel prize would of course vanish. Every morning they reappeared when the power was switched on again. Apparently this went on for years. Flickering immortality.

Under the hammer
Not long after I discovered the museum in the early 90s, it lost its benefactors and went bankrupt. Its entire collection, including Dennis Gabor's surviving image, and that of his Nobel prize, were sold at an auction. The auction was held on a snowy Saturday in January, 1993, at the order of the United States Bankruptcy Court of the Southern District of New York, in a tin building on the Saw Mill River Road in Hawthorne, New York. A tiny ad in the Times classifieds offered the following 19th century P.T Barnum style advertising copy:

"Very Extensive Collection of Some of The Finest And Most Important Holograms in The Country, Featuring The Nobel Prize of Dr. Dennis Gabor, Plus His Holographic Portrait And Other Supporting Papers. Also Works By: Yaacon Agam, S. Benton, J. Burns, Dr. Jeong, Claudius. And Many More Early and Prominent Creators. The Museum Also Includes Holographic Production Equipment, Lasers, Lighting, Etc."

I attended the auction, curious to know what might become of Dennis Gabor's immortal image. During the morning inspection period before the auction began I actually found it -- the glass holographic plate, face up on a cafeteria table.

The glass measured 18" by 24". A label explained that Gabor sat for his three dimensional portrait at the McDonnell Douglas Labs in St. Charles, Illinois, in October, 1971. That was year he won the Nobel Prize. The plate had been set out next to three liquor store boxes filled with faded newspaper clippings, correspondence, magazine articles and scientific papers. Here was a photo of Gabor in tails with the queen of Sweden upon his arm. The queen of Sweden was much bigger and taller than Dennis Gabor.

The whole collection, including the plate, the papers, and a paper copy of the Nobel Prize, was marked as lot #210. I picked up the holographic plate and held it at various angles against the fluorescent lights of the warehouse. There was no visual hint that this flat panel of glass contained a three dimensional image of a man. As with any hologram, it was impossible to discern any sort of literal image.

Abruptly the auctioneer turned on a microphone and declared, with an amplified Queens accent, that there would be no further viewing. His assistant popped up – a woman about 30, with bobbed red hair. She wore a bright white blazer over a peach blouse. Her fingernails were Chinese red. She wore rings on each hand and a butterfly broche on her lapel that wildly scattered light. Evidently her job was to attract and hold the attention of the crowd, which she certainly did. She picked up a hologram (bleached, nondescript, unintelligible) and held it high over her head. The auctioneer started his chant and the bidding commenced.

When they got around to lot #210, the initial bid was just $500, which doesn't seem like much for the memorabilia of a Nobel Prize winner, but this low bid was in the end brushed aside by much bigger money. The Media Lab of MIT was represented in Hawthorne by Professor Stephen A. Benton, the inventor of the white light holography technique. In the early afternoon it was announced that Dr. Benton had bought out the whole auction for MIT with a single bulk bid of $180,000.

In Cambridge, in 1994, they held an exhibition. Perhaps they set up the plate of Dennis Gabor and switched him on once again, good as new. Stephen Benton died in 2003. MIT has added substantially to the material bought in from the Museum of Holography, and has now the most extensive holography collection in the world. The Gabor plate is still in Cambridge, still safe, still sequestering the image of the man and still, somehow, ineffably sad.

What is spatial phase?
Light carries five kinds of information into the eye.

direction of propagation
polarization
frequency
amplitude
phase

Direction -- the sense of where the light ray came from -- is fundamental.

Scattered sunlight is polarized, and polarization is sensed by many insects, crustacea and cephalopods. Bees use this information to navigate. Amazingly, polarization effects can also be sensed by a very few humans (Haidinger's Brush Effect).

Let's concentrate here on three properties: Frequency, amplitude and spatial phase.

Frequency is perceived as color. Light amplitude is perceived as brightness. It is thought that spatial phase is not perceived at all. But what sort of information is carried by spatial phase?

Phase can be described only mathematically, but we can treat the subject in a casual narrative way in order to seed the basic idea, as follows:

Let’s say that light "takes an impression" of a three dimensional object as it bounces off of it. Light then carries that impression, as phase information, into a camera or into your eye.

Illustration reproduced courtesy of Prof. Jay Newman

For example, the upper panel of this illustration shows, edgewise, a plane wave of light inbound toward a reflective object -- a mirror with a channeled surface.

The lower panel shows the same wave outbound after reflection. The wave now bears the impression of the surface with a channel in it. It is as though the plane wave had "picked up" the impression of the channel (although it should be noted that the distortion introduced into the plane wave is double the depth of the channel.)

The "3D impression" is carried through space as a distortion of light’s plane wavefronts. Think of these wavefronts as a sheets of transparent plastic that have been instantaneously vacuformed around an object –maybe a toaster – and are now racing through space in rapid succession. Bubblepacks.

In a camera, this incoming 3D information about the shape of the object is lost -- squandered -- by photographic film. Phase information is lost when the precisely distorted light wavefronts plaster themselves, over the time span of an exposure, against the flat surface of the film. This is why a snapshot is just 2D. Flat.

We are taught that phase information is also squandered against the retina, exactly as it is against film, though I no longer accept this notion. (See for example the discussion of The Corduroy Neuron under the subhead, "Where can we go with this." I think the eye probably conserves spatial phase, and that this an important aspect of our excellent depth perception.)

It must be said immediately that spatial phase is not adequately described as a 3D impression. Spatial phase is a formal concept that we will come back to again and again, particularly in the discussion of photoreceptors in the following chapters. For a start, however, "3D impression" will do. For an accessible discussion of holography, I would refer you to this "How stuff works" essay by Tracy Wilson.

Dennis Gabor’s quest to save phase
Dennis Gabor did not set out to invent the hologram. His purpose was to try to conserve spatial phase and to improve electron microscopy by eliminating lenses. Here is a quote from Gabor:

"But an ordinary photograph loses the phase completely, it records only the intensities. No wonder we lose the phase, if there is nothing to compare it with! Let us see what happens if we add a standard to it, a 'coherent background".

To bench test his idea, Gabor used a high pressure mercury arc lamp, the most coherent source available to him at that time.

Gabor hoped to conserve the normally-lost spatial phase information by simultaneously recording, along with the distorted incoming wave fronts, a baseline reference – a yardstick wave. In practice he mixed two light waves at the face of the film: 1) the incoming, physically distorted wave bounced from the object to be filmed, plus 2) a pure, perfectly coherent, undistorted wave.

These two waves interfered, and the recording of their relationship consisted of a photograph of their interference pattern. This was the first hologram.

To project it, he developed the film and replaced it in front of the pure reference beam. A 3D image of the object of the hologram appeared, floating in space.

The principle of capturing spatial phase by remarking a wavefront’s position relative to a reference wave was clear in principle. However, it may be that Gabor did not have an understanding, in 1947, of what his hologram would turn out to mean physically. "Let us see what happens…." he said.

In 1950, a physicist at University College in Dundee, Scotland, Gordon L. Rogers, advanced the helpful idea that the finished hologram was actually focusing light by means of diffraction, rather than refraction. Using Rogers’ insight, the hologram could be understood as analogous to a lens created within the substance of the film. When coherent light was passed back through this "lens," the original object reappeared in 3-space. In effect, the holographic lens created within the film was both a recording medium and a focusing device.

Notice that holography is all about spatial phase – its capture, conservation, and reconstruction. The technology depends on coherent light, and upon precision achieved at the level of a quarter wavelength of light. It is a perfectionists’ medium.

The Holographic Memory machine
A holographic memory is an electro-optical machine based on holography that has the power to recognize a face in a crowd. It was first conceived in 1963. This machine’s ability to recognize a very specific object can also be, in effect, de-tuned; so that instead of recognizing a particular racehorse, for example, it can recognize any racehorse. Further de-tuned, it can recognize any four-legged creature. In this sense the holographic memory machine has the rudiments of abstract thinking.

This is certainly tantalizing -- but the idea that memory in the brain works like a holographic memory begins to falter when we try to extend the analogy to concrete parts and pieces. There is no laser or laserlike generator of coherent wave trains in our brain. Where is the essential reference beam? Where is the beam scattered from the object? Where and how might the two signals be mixed? Where is the extremely fine grained film?

It is possible to grapple with these many questions but ultimately, with whatever sort of analogous biological hardware one might propose -- the question becomes this: How could spatial phase be reconstructed in the brain? Because at bottom, that’s what holography really does. It saves and then reconstructs spatial phase.

The non-holographic hologram.
But back up just one step, historically, from that first hologram of 1947. One of Dennis Gabor’s objectives in inventing holography – the conservation of spatial phase – could indeed turn out to be an important concept in understanding both vision and the brain. If you could truly conserve spatial phase -- if you never lost it -- you would not need to reconstruct it. You would not need a hologram, nor a reference beam, nor a beam splitter, nor a laser, nor any of the rest of the holographic apparatus.

Without any sort of contraption, freestyle, you could see holograph-like images in 3-space, which is to say – you could see images which are indistinguishable from the reality we see all around us every day.

The possibility that the eye conserves phase has never been seriously considered. This is because the retina isn’t supposed to be up to the task – and the nervous system as presently understood certainly isn’t. If the subject of spatial phase comes up at all in the context of the biology of vision it is quickly dismissed as impossible. Too quickly, let’s guess.

The textbook retina is a 2-dimensional sensor. It is optically flat, like photographic film. It necessarily squanders spatial phase information because it has no way to detect the 3-dimensional contour of an incoming plane wave.

The conservation of spatial phase should actually be pretty simple. It requires a "yardstick" photoreceptor, that is, a rod or cone cell that is able to distinguish and report out light intensity impressed at points along its length, at specific disks.

Receptors in depth would comprise a retina that can act as sensitive solid -- able to detect the contours of an incoming plane wave as registered along the z-axes of each and every photoreceptor cell.

In the context of the multichannel neuron model, the notion of a photoreceptor that is sensitive along its z-axis is very important. It is the key to understanding what a visual memory is, and how it can be stored and remembered in the brain.

<<PREVIOUS CHAPTER NEXT CHAPTER>>

2006-07-27T13:46:00.000-07:00

Chapter three
Eye Memory

John von Neumann was a genius like no other. In 1938, in middle life, von Neumann transformed himself into a weapon designer, arguably the best one of his century. In the late 1940s and early 1950s, he was working fast to perfect the digital computer. He was fascinated by it but for him it was a necessary intermediate step – a tool or fixture. He needed the computer to run the monumental Monte Carlo calculations for the hydrogen bomb.

By coincidence, this was an heroic period for neurophysiology. Beginning in 1949 at Woods Hole and Plymouth, the newly invented voltage clamp, applied to the squid nerve’s large diameter axon, was revealing for the first time, in splendid detail, the workings of the axon membrane as a system to govern and gate the flow of ions. It seemed the problem of how nerves conduct impulses was tumbling.

Von Neumann hoped to discover any and all useful analogies between the human brain and nervous system and the electronic brain he was trying to perfect. He actively sought the opinions and expertise of neurophysiologists. He evidently believed most of what they told him – though it is interesting to notice what piqued his skepticism.

It was his strange sojourn in the land of neurophysiology – another country – that he later reported in his book, The Computer and the Brain. He wrote it in 1956-57, in the last year of his life. It was conceived as a series of lectures but published posthumously as a long essay.

What he did not say.
Perhaps the most famous and often quoted line in this book appears at the beginning of Part II, where von Neumann declares that "The most immediate observation regarding the nervous system is that its functioning is prima facie digital."

The "prima facie" modifier is commonly taken to mean von Neumann saw the brain as "obviously digital," or "patently digital," and that the brain therefore must resemble a digital computer. You tend to hear this quote from people who believe, or want to believe, exactly that.

But as you read The Computer and Brain you will quickly discover that this is not what John von Neumann intended. Von Neumann used words precisely and to him, "Prima facie" meant "on its face."

In 1956, the brain appeared, on its face, to be a digital machine. But von Neumann thought this impression might be misleading. He thought that deeper biological investigation might well demonstrate that the nervous system is not, in fact, digital, or not completely digital.

He suggests that perhaps some intermediate signaling mechanism, a hybrid between analog and digital, might be at work in the brain. For this and other reasons he actively resisted labeling the brain as a digital computer.

The nerve as a hybrid device
Hybrid computers – part analog, part digital – were transitional between the analog computers of the 1940s and the fully digital computers of the 1960s. When von Neuman wrote his essay on the brain in 1956 and 1957, analog/digital hybrid computers were part of his technical repertoire. They were still available as commercial products in the 1970s.

The analog component was fast and easy to use, but imprecise. The response of an analog electronic circuit is readily described mathematically; hence, analog electronic circuits can be used to describe or depict mathematics.

Analog computation is easy to visualize with an oscilloscope. An analog circuit can be made to draw a curve on the gridded screen, typically depicting voltage versus time. By dialing numeric values into variable components in the circuit under observation (variable resistors, capacitors, inductors) one can draw curves to represent mathematical functions. Straight lines, parabolas, hyperbolas, sine waves, circles, ellipses, Lissajous curves, 3-space objects, whatever you need. You can then pick numerical points off the desired curve -- graphic solutions.

To "hybridize" this analog computer, add a digital memory and processor – in today’s parlance, an A-to-D converter. Sample the analog voltage or voltages at a point in time, store the result in a digital register. You can then perform digital calculations, or go back to the analog system for another quick read of a point on a curve.

Not surprisingly, the idea of the hybrid analog/digital system was projected onto the nervous system.

The nerve is functionally and anatomically split. The nerve cell appears to be an analog device on its input side, that is, in the dendrites, where incoming voltages may be buffered, summed and subtracted – even seen to depict perhaps, a curve.

Beyond the axon hillock, the nerve must be viewed as a digital device. The axon either fires or does not fire. It conveys a purely digital signal.

Not digital in the sense of a binary code -- not a one or a zero. Digital in the sense of a finger sticking up -- the famous "all-or-none" voltage spike.

The nerve conserves analog input information encoded as the interval between the digital output spikes. Or so it seemed in 1956.

Possibly this is what Von Neumann meant by "a hybrid of analog and digital". The hybrid neuron idea identified the dendrites as an analog signal processor, and the axon as a digital transmission line.

A variable voltage level could be sampled by dendrites, somehow converted into a launching schedule for spikes on the axon, and then finally transmitted along the axon as a traveling time interval between two strong digital output signals. (We no longer believe much of this stuff, and haven't since about 1995 -- but in von Neumann's time it was bedrock.)

Incidentally, this idea that the dendrites could be an analog signal processor appealed very strongly, in later decades, to theorists who thought the brain’s memory might work like a hologram – mainly because they needed to locate, so very badly, an analog signal processor of some kind, somewhere.

A different kind of hybrid
There exist other types of hybrids of analog and digital signal characteristics. One such hybrid is presented in Chapter two: the multichannel neuron.

A single nerve impulse traveling along a multichannel axon is digital on its face – it is a spike -- but it communicates to the brain analog gradations denoted by channel numbers, 1, 2, 3… n.

We don’t know what n might be, but we have been using 300 as an anatomically plausible number of channels. It isn’t a purely analog system, because the steps between channels are finite – represented by integers in fact. It is an "incremental analog" system perhaps, with the specific increment denoted by the channel number. Because each pulse is all-or-nothing – always a digital "1" in effect -- it is a solidly reliable signal, indifferent to noise.

Notice the surprising computational possibilities for this type of hybrid system. It could use numbers almost like we do, to measure, compare, count and tally. It could be set up to add, subtract, multiply, divide -- perhaps even to convolve. It could also use sliding scale, rubber scale, and scale-combining operations to produce logarithmic and vernier results.

In the Corduroy Neuron model the dendrites, though superficially analog devices, have become rather mysterious – one must guess there is something very important going on back there.

What is the best "memory organ"?
Von Neumann was eloquent on the problem of selecting a "memory organ," for the brain. For his own electronic brain, he chose to use the eye as a model for a memory organ.

He thought the worst possible choice for a memory organ in the human brain would be a neuron.

As for the synapses, it is my impression that his hope for them, never fully expressed, was that they might turn out to work like AND and OR and NAND and NOR logic gates.

Could it be true in general, as von Neumann concluded in particular, that the eye might provide a superior model for a memory organ? It seems plausible. If we could understand the eye, we could probably understand the rest of the brain. It has been suggested from time to time that one might re-imagine the cerebral hemispheres as huge everted eyes, with each hemisphere a glorified, multiplied, stratified stack of retinas.

The von Neumann machine
When von Neumann began to work with the engineers who developed the digital computer, one of his first contributions was the suggestion that a memory store be allocated for programming.

This seems so astoundingly obvious now: That a computer should be able to remember what we want it to do. But before von Neumann suggested the concept of the stored program, each computer operation was laboriously set up in advance by the machine's attendants.

The first stored programs were committed to punched paper tape, one of the simplest forms of memory. It was at this point that the digital computer became the modern instrument we now call, generically, a von Neumann Machine.

A fast, compact working memory, such as the common desktop computer's reserve of DRAM chips, did not yet exist in 1945. The electrical working memory for ones and zeros consisted of devices that could assume and hold two states: these included manual toggle switches, electromechanical relays, and vacuum tube flip-flop circuits.

The early computerists knew they needed to invent a better memory, a better short-term storage medium, and they tried many different approaches before they arrived at the historically important (though now deeply obsolete) solution of ferrite cores.

Eye memory
Early computer memories resembled eyes. The designers borrowed the basic concept of the TV camera. A binary "bit" representing a 1 or 0 was stored as a charge induced by a glowing phosphor at a certain geometrically defined point on the screen of a cathode ray tube. A bit could be set, scanned out or refreshed with a fast moving electron beam. The glowing bit's "address" was its x,y coordinate position on the face of the tube.

At von Neumann's urging, RCA's labs tried to create a reliable computer memory from this Cartesian visual concept, but while they were still experimenting in Princeton an Englishman named Williams perfected the scheme. The Williams memory was quickly adopted by American computer makers, including IBM. It worked but it was not reliable.

A war story has it that when IBM introduced its first mainframe with a Williams eye-memory, at a formal news conference in New York, an eager photographer with a news camera walked up to the machine and popped a flashbulb. The flash blew the machine's memory to kingdom come.

Core memory, invented in Boston by An Wang, then 26, and independently invented by two other men, quickly replaced the Williams eye-memory. Core memory stored ones and zeros as tiny magnets, and it could be layered in three dimensions, rather than two. It should be noted that these early room-sized computers were each about as powerful as a modern pocket calculator.

It interests me that when the electronic computer was first developed, the designer very quickly visited the concept of an eye memory. When Nature got started on the same problem, brain building, it may well have started at the same place.

A primitive eye was perhaps the vertebrate brain’s earliest prototype. Such an eye might freeze an incoming image for a few milliseconds in order to extract useful information from it. A memory, however short-lived, is a memory.

It is easy to see that a photoreceptor might serve wonderfully as a recording machine, because light induces decisive biochemical changes in it’s cache of rhodopsin. These changes linger – it takes time for rhodopsin to regenerate, or re-cock. So within a photoreceptor there are indeed persistent biochemical changes in response to a change in the outside world. This is a familiar textbook definition of memory – a biochemical change in response to experience. We are trained to ascribe such changes (in the very next breath) to the synapse. But let’s stay clear of the synapse for a bit and posit the idea that the earliest visual memory was written as a chemical and conformational change in a visual pigment such as rhodopsin. The idea will be developed in Chapter 12.

The bizarre anatomy of the human eye:
The eye is the only part of the brain we can look at without surgery. It is the best-examined component of the brain, and it is the last place in the brain where anything can happen at the speed of light.

Von Neumann, who essentially used a television camera as a "memory organ" in the digital computer, took a particular interest in eye anatomy.

In the course of his biological studies von Neumann noticed and remarked to his brother on something curious about the eye. He was puzzled by an anatomical fact that most biologists, then and now, brush aside as a mere happenstance or oddity of evolution.

Like all vertebrate eyes, the human eye looks backward, toward the eyesocket. The photoreceptive cells of the retina are wired from the front, so that it would seem that the business end of each retinal photoreceptor is pointed at the back wall of the eye.

Incoming light from the pupil must pass through a web of blood vessels and a fine network of nerve fibers, including three layers of cell bodies and a host of supporting and glial cells -- before it reaches the eye's photoreceptors.

At the tiny fovea, the nerves are layered aside, a peephole, but what about the rest of the retina? What about animals (most vertebrates) who have no fovea? The brain must somehow subtract all the intervening goo out of the clear picture we see of the world. How could the brain accomplish this wonderfully clarifying step? With a calculation? Von Neumann wondered about it. We should too.

Holographic memory
John von Neumann’s interest in eye-memories and eye anatomy has a modern reprise in computer technology. After decades of trying, it finally appears that the next major computer mass storage memory medium may be optical -- holographic.

A holographic memory system can store colossal amounts of data in three-dimensional blocks or films of photosensitive material. Computer storage media of this type could dwarf the mass storage capacity of our present computers’ DVDs, though it remains to be seen whether holographic devices will become commercially successful, or could be made to supplant RAM as well as mass storage.

The use of holographic memory in digital computers, if it happens, might help rescue from contempt a physiological concept that peaked around 1973 and then, for various reasons, faded away: the idea that memory in the human brain works like an optical hologram.

It probably doesn’t work quite that way – but the holographic brain guys flew very close to the truth once or twice. My opinion of course.

The idea of holographic memory actually gains some evolutionary credence from the curious, backward-looking structure of the human eye, as we shall see.

Had he lived Von Neumann might have been quite interested in holographic memory, for the hologram itself was conceived by a classmate of his, Dennis Gabor: Another Budapester, another genius.

<<PREVIOUS CHAPTER NEXT CHAPTER>>