Abstract: This paper first reviews our present understanding, and misunderstanding, of the ‘binding’ problem of how phenomenal colour, shape, and movement are integrated within the visual system. The traditional theory that this is done by interactions between the separate colour, shape, and motion computational centres in the visual cortex has been undermined by experiments carried out by Semir Zeki that suggest that such interactions do not take place. This is linked to the hypothesis that the visual system contains two channels: (1) Ohyama’s hierarchical behavioural network and (2) the visual phenomenal network. The review presents a further linkage between the computational mechanisms (in particular information compression) used in vision with those used in digital television. Then a number of theories designed to explain how the ‘binding’ may operate are examined. Two theories have their basis within the identity theory and a third theory is based on a recent version of the psychoneural interaction theory that distinguishes phenomenal space (and its contents) from physical space (and its contents). The conclusion is made that the new formation presented in this review supports the latter theory.
1. Introduction: The Problem
Current research in neuroscience into all aspects of consciousness is based entirely on the psychoneural identity theory (IT). This assumes that brain events and their correlated phenomenal events are identical — in other words that Chalmers’ (1998) ‘hard problem’ has been solved. However, there are grave difficulties with this position (Smythies, 1994). Basically, phenomenal objects can be observed by introspection undeniably to have one set of properties (e.g. the boundary of a visual after-image forms a Jordan curve), whereas the neurons involved in creating the phenomenal object have quite a different topology. A Jordan curve is any curve that uniquely divides space into one ‘inside’ and one ‘outside’. This hypothesis thus violates Leibniz’s Law of the Identity of Indiscernibles that states that for objects, or events, to be identical they must share all properties, including spatio-temporal properties. In common sense terms it seems improbable that the expanded, clear, and detailed visual field that a person experiences can be literally identical with any part of the pink jelly that is a brain. As Crick (1994, p. 159) described it, ‘We have for example a vivid internal picture of the external world’. There is, however, no such vivid picture in the brain, nor any mechanism capable of constructing it. This fact has led most people to deny the obvious by denying that we have any such vivid internal pictures at all. The alternative solution, discussed in the next section, may be that we have such pictures, but they are not inside the brain in physical space but are outside the brain in phenomenal space.
Common sense deals with this dilemma by recourse to the theory of perception known as naïve realism in which phenomenal objects are held to be direct views of external objects. This, of course, is incompatible with the mountain of evidence from neuroscience that shows that phenomenal objects are not direct views of external objects but are constructs of the representative mechanisms of perception. Until recently the only alternative theory to explain brain–consciousness relations was Cartesian dualism, which is even less plausible as it holds that consciousness has no properties at all except the ability to think. Thus it is quite unable to explain the observable properties of phenomenal visual consciousness and its extended, coloured, and moving contents — phenomenal objects that constitute our sensations, i.e. our visual sensations.
An alternative realist theory (extended realism) has been proposed to explain the nature of consciousness and its relation to its brain. This is based on the simple proposition that phenomenal space and physical space are both real spaces, but are different from each other in that both are different subsections (slices) of a common higher-dimensional space. The brain is located in one section (called at present physical space) and a phenomenal consciousness is located in the other section (called phenomenal space). Both sections contain real events of two different kinds — physical events and phenomenal events — both of which have intrinsic and extrinsic geometrical and topological properties. They are linked by causal interactions. The theory was first suggested by the Cambridge philosopher C.D. Broad (1923) and the Oxford philosopher H.H. Price (1953). It was further developed by Bertrand Russell (1948). Since then it has been extended by the astrophysicist Bernard Carr (2008; 2015) and by myself (Smythies, 1994; 2003; 2014).
In essence the theory proposes that the cosmos has not three but four dimensions of space (or five of space-time) and that a person’s organism consists of two parts — both equally ‘material’ (but composed of different material) and equally ‘mechanistic’ — a physical body and a phenomenal consciousness. Thus the causal chain of visual perception does not end in the brain, but it extends to the phenomenal visual field that is created at this level by a system of causal relations that we can picture as a kind of televisual-like mechanism (TLM) located in higher-dimensional space (likened to the operations of Plato’s cave — Smythies, 1994). This mechanism scans the brain and uses this information to construct the visual field. Thus the brain, in this hypothesis, is merely an intermediate station in this causal chain that extensively receives, processes, and records the information on its way to consciousness, and provides a host of ancillary mechanisms for the smooth control of behaviour. In the context of this present paper we can suggest that the final binding discussed earlier that creates the phenomenal object in the visual field is done by the TLM. This can explain the apparent absence of cross-talk between the three channels in the visual input in the brain. A similar system may be used in audition, as well as in the somatosensory input during the construction of the body image.
To set this problem in perspective it will be helpful to review our present understanding of how the brain mechanisms involved in conscious visual perception actually operate. This understanding has been greatly extended by some very recent experiments.
2. How the Brain Mechanisms Concerned in Visual Perception Actually Operate
Currently much effort is being expended in neuroscience on trying to explain how visual mechanisms in the brain generate phenomenal objects in the visual field in consciousness. A good way to approach this topic is to consider the so-called ‘binding’ problem that relates to the fact that neurocomputations for colour, ‘shape’, and motion in vision are carried out in three anatomically different and widely separated areas. ‘Shape’ may be defined as that which the system produces when the stimulus is a stationary black-and-white image lacking colour and movement. This leaves it obscure how these activities are ‘bound’ together to produce a single phenomenal object in which colour, shape, and motion are united in a single entity. Traditionally it was believed that this ‘binding’ was effected in higher visual cortex. However, mainly on the basis of a series of psychophysical experiments, that paired colour with motion or colour with orientation, Zeki (2015) has proposed that the brain is a massively asynchronous organ and has no central (master) clock that resets the activity in each of its parallel systems. Moreover, he proposes, in the visual system that colour, form, and motion are processed independently, resulting in an asynchronous behavioural output from each independently. Apparently the proposed brain mechanism for binding does not exist.
Key points in his argument are as follows:
- Experiments on pairing colour with motion and colour with form suggest that we perceive colour 40 ms before we perceive form and 80 ms before we perceive motion. These experiments also suggest there are asynchronous individual behavioural outputs from each of the three systems.
- This perceptual asynchrony is likely due to differences in processing times in these three pathways needed to bring signals to a perceptual end point.
- These results suggest that the brain is a massively asynchronous organ, which possesses no central clock that resets the activity in each of its parallel systems.
- Parallel, asynchronous, processing operations make for increased efficiency in the brain because the fastest system does not have to ‘go on hold’ for the slowest one to complete its task.
Zeki says ‘Parallel and asynchronous outputs make the problem of interattribute binding even more emphatic. How to bind activity at so many different nodes, when processing speeds within and between them differ, as do the outputs from and return inputs to them?’ (ibid.).
Zeki further suggests that the visual system cannot integrate the colour, shape, and movement data transmitted to the higher visual brain without integrating the electrical rhythms carrying them with an external electrical rhythm (here he suggests an alpha or theta rhythm). This mechanism, he suggests, acts as a ‘clock’ so as to synchronize this activity. ‘It is simply a question of one system “tagging” onto another through a third’ (ibid.). However, no evidence for this mechanism is presented. An alternative suggestion has been made by Fingelkurts et al. (2003), who have suggested that this is effected by neuronal assemblies called the construction operational modules (OM). They state,
OM is the synchronized operations executed by spatially distributed neuronal assemblies (and in the EEG indexed by the quasi-stationary segments in different locations). OMs could coexist on different time-scales, over spatial patches ranging from a small number of brain areas to an entire hemisphere and eventually, the whole brain parallel processing is performed by individual neuronal assemblies, whereas serial processing emerges as a result of formation of OMs and their changes along with shifts in the process of actualization of objects in the physical or mental world. (ibid.)
However, OMs involve functional synchronized connections between the three pathways that, according to Zeki (2015), are apparently lacking. So how are the three incoming visual pathways integrated to form the single phenomenal object when the pathways apparently do not talk to each other? To solve this problem it may be necessary to look again at the fundamentals of what the visual field in consciousness — that contains the ‘bound’ visual phenomenal objects — is and how it is constructed. This is tackled in the next section.
3. The Ohyama Hierarchical Behavioural Network
So how can we answer Zeki’s question? It would appear that we are dealing with two parallel systems in conscious vision. The first behavioural component consists of Zeki’s three asynchronous largely independent pathways that carry colour, form, and motion information to the specialized multisensory areas in higher cortex that work according to the saliency activated hierarchical mechanism, that drives behaviour, described by Ohyama et al. (2015). These workers showed that integrating multisensory networks are arranged in a hierarchical order — mono-sensory first, followed in order by bi-, tri-, and quadri-sensory networks. Lastly, impulses are fed to the motor command centre and behaviour is initiated. Importantly, in addition, Ohyama et al. (2015) have discovered that action selection actually starts at the first-order level, not only at the highest level (at least in Drosophila). A particular salient input increases the probability of expression of the behaviour appropriate to that input (e.g. escape behaviour in response to a visual signal indicating a predator attack). The multiple stages of multimodal integration serve to enhance, or diminish, the probability of this selection being translated into action. For example, signals in the pathways of other senses may confirm, or fail to confirm, that a predator is attacking. A single lowest level input, if powerful and salient enough, may by itself trigger the appropriate behavioural response. This starts working as soon as the first message arrives in the visual brain provided it is salient enough.
4. The Visual Phenomenal Network
Then, in the visual system, there is the second phenomenal system. This is the mechanism that mechanically reconstructs the incoming tripartite colour, form, and motion visual information into the single clear-cut visual phenomenal object observed in the visual field in consciousness. Employing advanced multivoxel fMRI pattern analysis techniques, Erez et al. (2015) have produced strong evidence that the construction of visual phenomenal objects in higher visual cortex (especially perirhinal cortex) depends on an explicit conjunctive coding mechanism (based on Hubel and Wiesel’s, 1967, hierarchical hypothesis). However, this finding involves only the ‘form’ visual pathway, and not the ‘colour’ and ‘movement’ visual pathways. Thus it is really not relevant to the ‘binding’ problem, which involves the integration of these three purely visual pathways.
It is now known that the visual field in consciousness is built up partly using information transmitted from the retina (‘reality’) and, to a considerable extent, by information transmitted from cortical memory banks (‘virtual reality’). The former are focused in foveal vision and are augmented by attention. The latter are augmented in peripheral vision and are augmented in the default state. As Crick (1994) has said — we see not what is ‘out there’ but what the brain computes is most probably ‘out there’ (see Smythies, 2005; 2009, for details).
Visual objects can enter the behavioural network before they are seen consciously in the phenomenal network. Ramachandran and Seckel (2014) tested four projector synaesthetes, who saw printed letters of the alphabet as coloured. The stimulus presented consisted of such letters hidden in puzzle pictures. The subjects gave behavioural reports that they saw the colours before the conscious recognition of the letters. They were also able to see mirror-reversed letters in the same colours as non-reversed letters, which enabled them to read mirror-reversed text at three times the normal speed.
5. Information Compression Mechanisms in Vision
The representative theory of visual perception holds that visual phenomenological events are constructed by the representative mechanisms in the brain. Another system that receives light input, transmits information, and constructs visual images is digital television. It is therefore of the greatest interest that the mechanisms used by these two systems have been found to be very similar (Smythies and D’Oreye de Lantremange, 2016). In both the signal is divided into three streams (colour, ‘shape’ or ‘object image’, and motion), which are processed separately and at different speeds. The three streams are then united in the final stage finally to form the complete picture on the screen. In addition, and most significantly, both systems use the same mechanism for data compression, known as removal of temporal redundancy.
The TV picture to be transmitted is divided into a series of consecutive ‘frames’. The first frame is transmitted as a whole. In subsequent frames the whole picture is not transmitted but only the differences between one frame and the previous frame. This significantly reduces the informational load and results in cheaper and faster transmission. Computational mechanisms in the brain are also faced with the problem of overload by the massive amount of incoming signals. It is of interest therefore that experimental evidence has been reported that indicates that such a mechanism exists in the visual brain. Nortmann et al. (2013) conducted voltage-sensitive dye imaging experiments in the visual cortex. The presentation consisted of vertically and horizontally filtered natural images. The authors found that at 33 Hz the encoding represented the current input. In contrast at low frequencies (10 Hz) the encoding represented, not the currently exposed images, but the difference in orientation between consecutive images. The authors state,
When compared with the preceding image, the cortical activity patterns characterized exactly the difference in orientations. Consequently, large amounts of incoming data were relatively suppressed, reminiscent of differencing methods (Fowler et al. 1995) used for video data compression in communication technology. (ibid.)
In the visual brain the system works like this. The primary visual cortex V1 receives two major inputs. One comes from the retina and transmits information about what is out there now. The second comes back from higher visual cortex and transmits information about what the brain has computed was out there until a moment ago. In V1 the two inputs are arranged in the form of a simple template so that the ‘now’ picture is located functionally exactly over the ‘moment ago’ picture. If there has been no change in the picture at any particular location the message in that location is eliminated. If there has been a change, then that region is allowed to send its signal to the higher cortex. In this way a great deal of redundant information is eliminated with resulting increased speed, efficiency, and lower metabolic cost of the operation. This mechanism is sometimes called ‘predictive coding’ (Huang and Rao, 2011). However, this is somewhat misleading. The ‘moment ago’ picture can logically be regarded as a ‘prediction’. But a more accurate description of the process is ‘information compression’ or ‘redundancy reduction’.
It can be argued that the new experimental data presented here, in particular that the brain shares some basic mechanisms of operation with digital TV, implies that the relationship of the phenomenal visual field in consciousness to the physical eye is logically and functionally the same as the relationship of the TV screen with its attached TV camera. This is different from the idea as to how vision works held by contemporary ‘common sense’ (naïve realism) in which we are supposed to experience physical objects directly. The experimental evidence against that theory is very strong.
The experimental data are not incompatible with the hypothesis that a phenomenal consciousness is an organized system, which includes the functional equivalent of the TV screen, located in higher-dimensional space and connected to its brain (that contains only intermediate computational machinery) by causal relationships. The new data from neuroscience show that Crick’s ‘vivid internal picture of the external world’, that forms the visual field in consciousness, is built up of the same kind of information processing that is used to build up the pictures on the TV screen in digital television. As it is certain that no such pictures exist in the brain, this makes it plausible that they exist outside the brain. All that the brain contains are the neural correlates of consciousness (NCCs) composed of electrical activity in neurons that contain the same information but in a completely different format as detailed in this review. These NCCs correspond to the computational events inside the computers that prepare the TV picture, but not to the resulting events that are projected onto the TV screen itself.
Broad, C.D. (1923) Scientific Thought, London: Routledge and Kegan Paul.
Carr, B. (2008) Worlds apart? Can psychical research bridge the gulf between matter and mind?, Proceedings of the Society for Psychical Research, 59, pp. 1–96.
Carr, B. (2015) Hyperspatial models of matter and mind, in Kelly, E., Crabtree, A. & Marshal, P. (eds.) Beyond Physicalism, London: Rowman and Littlefield.
Chalmers, D.J. (1998) The problems of consciousness, Advances in Neurology, 77, pp. 7–16.
Crick, F. (1994) The Astonishing Hypothesis, New York: Scribner.
Erez, J., Cusack, R., Kendall, W. & Barense, M.D. (2015) Conjunctive coding of complex object features, Cerebral Cortex, pii: bhv081 [epub ahead of print].
Fingelkurts, A., Fingelkurts, A., Krause, C., Möttönen, R. & Sams, M. (2003) Cortical operational synchrony during audio–visual speech integration, Brain and Language, 85, pp. 297–312.
Fowler, G., Korn, D., North, S., Rao, H. & Vo, K. (1995) Vdelta: Differencing and compression, in Krishnamurthy, B. (ed.) Practical Reusable UNIX Software, New York: Wiley.
Huang, Y. & Rao, R.P. (2011) Predictive coding, Wiley Interdisciplinary Reviews: Cognitive Science, 2 (5), pp. 580–593.
Hubel, D.H. & Wiesel, T.N. (1967) Cortical and callosal connections concerned with the vertical meridian field on cat, Journal of Neurophysiology, 30, pp. 1561–1573.
Nortmann, N., Rekauzke, S., Onat, P., Koenig, P. & Jancke, D. (2013) Primary visual cortex represents the difference between the past and the present, Cerebral Cortex, 25 (6), pp. 1427–1440.
Ohyama, T., Schneider-Mizell, C.M., Fetter, R.D., Aleman, J.V., Franconville, R., Rivera-Alba, M., et al. (2015) A multilevel multimodal circuit enhances action selection in Drosophila, Nature, doi: 10.1038/nature14297.
Price, H.H. (1953) Survival and the idea of another world, Proceedings of the Society for Psychical Research, 50, pp. 1–25.
Ramachandran, V.S. & Seckel, E. (2014) Synesthetic colors induced by graphemes that have not been consciously perceived, Neurocase, 21 (2), pp. 216–219.
Russell, B. (1948) Human Knowledge: Its Scope and Limits, London: Allen and Unwin.
Smythies, J. (1994) The Walls of Plato’s Cave, Aldershot: Avebury Press.
Smythies, J. (2003) Space, time and consciousness, Journal of Consciousness Studies, 10 (3), pp. 47–56.
Smythies, J. (2005) How the brain decides what we see, Journal of the Royal Society of Medicine, 98, pp. 18–20.
Smythies, J. (2009) Reality and virtual reality mechanisms in the brain and their significance, Journal of Consciousness Studies, 16 (4), pp. 69–80.
Smythies, J. (2014) The nature of consciousness and its relation to the brain: The pith of a formidable problem, Journal of Consciousness Studies, 21 (1–2), pp. 183–202.
Smythies, J. & D’Oreye de Lantremange, M. (2016) The nature and function of digital information compression mechanisms in the brain and in digital television technology, Frontiers in Systems Neuroscience, 10, art. 40.
Zeki, S. (2015) A massively asynchronous, parallel brain, Philosophical Transactions of the Royal Society of London B, 370 (1668), doi: 10.1098/rstb.2014. 0174.
Paper received January 2016; revised March 2016.