The simple questions cracking the hard problem of consciousness

Philosophers have a long-standing fascination with the structure of experience. By structure, they mean how sensations relate to one another. “Structure will be at the core of the science of consciousness,” says David Chalmers at New York University, who coined the hard problem of consciousness in the 1990s. If structuralists are right, our experiential repertoire is holistic in the sense that every experience depends on every other experience. When we encounter red, we draw a contrast with other colours, not to mention entirely different perceptual categories. For instance, red is sort of like orange, but distant from blue, and nothing at all like pain or joy. “Any individual experience hinges on this totality,” says Holger Lyre, a philosopher at the University of Magdeburg in Germany.

The idea of using objective methods to study subjective experience led to the first structural studies in psychology in the late 19th century, and the experimental method has hardly changed since. Researchers show people multiple stimuli, such as pairs of colours, and ask them to rate how similar they are. The main innovations today are scale and scope. Indefatigable study participants classify thousands of colour combinations, shapes, moving patterns, musical tones, spoken syllables, emotions.

This bonanza of quantitative data has attracted a community of mostly young, mathematically inclined researchers from both inside and outside of neuroscience. Based on people’s judgements of these differences in qualia, they create classification schemes of experience, often couched as abstract geometric shapes. “Our approach is to categorise all possible relationships between qualia,” says psychologist Nao Tsuchiya at Monash University in Melbourne, Australia, and ATR Computational Neuroscience Laboratories in Kyoto, Japan, who heads one of the biggest experimental pushes, the Qualia Structure Project.

People’s judgements are fairly consistent, says Tsuchiya. Across age and culture, they make broadly similar assessments about colour, sounds and other sensory qualities. In research published last year, he and his colleagues queried 247 children aged 3 to 12 in Japan and 29 children aged 6 to 8 in China, as well as a contingent of 84 adults, about colour. The responses hardly varied. “The effect of language, culture and development seems rather small,” he says. Language, culture and environment determine the colour words we use, which can vary drastically: land-locked Indigenous Tsimané forager-farmers in the Bolivian Amazon, for instance, elide blue and green into a single concept, whereas the ancient Greeks had a whole lexicon of words to describe the nuanced shades of the sea. Although culturally specific labels shape our view of the world, says Tsuchiya, they don’t seem to alter our immediate experiences.

At the same time, people’s judgements often differ from how those qualities have historically been structured through musical scales and colour mixing rules. Most study participants don’t identify tones an octave apart as the same, as musicians in diverse cultures do. The colour comparisons they make often seem mutually inconsistent; they don’t line up neatly on the rainbow. In other words, their experienced colour space has more than the standard three dimensions , redness, greenness, blueness , in the widely used RGB colour system. Based on his experiments, Tsuchiya estimates that we experience at least seven dimensions of colour.

The project has turned some of the classic philosophical thought experiments into actual empirical experiments. For instance, is my red the same as your red? Studies of atypical colour vision have got at this. Following up an online survey last year, the project invited participants into the lab for intensive testing. Four were flagged as colour-blind on a standard screening test, four weren’t flagged but volunteered that their colour vision seemed different and three had typical colour vision. Within each group, people made the same judgements about the resemblance of colours, but between groups, these assessments diverged. By doing a geometric analysis, the researchers boiled the data down to a shape. That of the second group , the self-reported colour-blind people , had features of the shapes of the other groups. “It’s an intermediate shape,” says Tsuchiya. In the structural way of thinking, what we take to be red is defined by its position within this shape. So, people within a group were entitled to say, “My red is the same as your red.” But what was red for people in one group might effectively be green for those in another. Tsuchiya describes the intermediate, second group as a “bridge” between atypical and typical colour experience, able to grasp the colour experiences of both.

The project has been extending its methods to other forms of experience. To study emotion, it presents participants with pairs of videos intended to evoke certain emotions and asks them to compare how they felt. So far, says Tsuchiya, people with difficulty expressing emotions ,  known as alexithymia , make the same distinctions as others. So, just because someone can’t articulate their emotions doesn’t mean they are unfeeling.

The periodic table of experience

The 19th-century structural psychologists compared their analyses to chemistry. Just as you break chemical compounds into elements, you can parse complex mental states into basic units. Adopting the same metaphor, Tsuchiya seeks to place qualia into a periodic table, with boxes for colour, pitch and so forth. He admits it is an imperfect metaphor, since what gives chemistry’s periodic table its power , and its name , is the repeating pattern of properties. He sees nothing like that with qualia. On the contrary, they all have different structures, which, he thinks, is why sight and sound feel so different. But Tsuchiya speculates that, if experience is holistic, the various qualia should exhibit some common features. “Maybe there’s some kind of underlying similarity between the different modalities,” he says. “It must be the case, I think.”

Having put experience into these buckets, researchers have new ways to test theories of consciousness. Going beyond a consciousness detector, they can look for patterns of brain activity that match the mapped structure of experience. “We need to understand what exactly physically makes something feel red, or feel blue, or be painful, or be joyful,” says João Pedro Parreira Rodrigues at the Einstein Center for Neurosciences in Berlin.

For instance, neuroscientists have long used similarity judgements to identify what area of the brain does what, such as a part of the visual cortex involved in distinguishing colours. “The signals that come out of this part of the brain, when I measure those collections of stimuli, have the same similarity structure as the ones I got from behaviour,” says neuroscientist Brian Wandell at Stanford University in California. Tsuchiya’s project is now doing a larger study of this sort. The idea is to use the structure of qualia to identify relevant brain activity, measured with an fMRI, which can then be checked against the predictions of various consciousness theories.

Lucia Melloni, a neuroscientist at Ruhr University Bochum in Germany, says she likes the project in general, but has concerns. Tsuchiya and his colleagues gather data by showing people two colours or other stimuli, clearing the screen and asking participants how similar they were, on a scale of 1 to 8. People have to form a judgement, remember it and assign it a number. Melloni worries something might get lost in translation: “I wonder whether he’s not just testing memory.”

Working with Parreira and Zefan Zheng at Ruhr, Melloni is conducting a basic reality check on the structural approach. Whereas Tsuchiya asks participants to compare two stimuli they are conscious of, her team elicits comparisons of stimuli that participants aren’t conscious of.

That sounds impossible by definition. How can the brain compare things it isn’t even conscious of? So, I was eager to try the experiment for myself. Parreira walked me through installing the software. It was like a boring 1980s arcade game. A coloured circle flashed, then a coloured ring; sometimes there was just a ring. I pressed the left arrow if the ring was red and right for green. It went on like this, iterating through different colours, for 40 minutes.

Oh, but wait. I thought that a circle flashed only half the time, but Zheng and Parreira told me one always flashed. If I didn’t see it, that was because the ring appeared so quickly afterwards that it confused my visual cortex and kept the earlier stimulus from entering conscious awareness. By varying the timing, the experiment controlled what I did and didn’t consciously register, so as to compare my responses.

Even when the circle slipped under the radar, it still had a subtle effect: it primed my brain. When a green circle preceded a green ring, I consistently pressed “green” about 50 milliseconds faster than if the circle was red. So, my unconscious mind was making colour-similarity judgements of its own. I couldn’t articulate these judgements, but my reaction time betrayed them.

It turned out that my conscious and unconscious colour structures were very different. The conscious one put colours into a neat sequence from green to blue to purple to red; my unconscious one was like a toddler who dumped all the crayons onto the floor. “We found that there is no unconscious colour space at all,” says Zheng. That’s a win for structuralism. Although Zheng cautions that the experiment is rudimentary and needs to be validated, it seems that structure differentiates conscious from unconscious perception and can therefore be used to search for signatures of consciousness in brain activity. “The structuralist turn is a very exciting development,” he says.

The hard problem of consciousness

Ultimately, Tsuchiya isn’t only looking for evidence that could lend weight to one consciousness theory or another , his ambition is to tackle the hard problem, and he thinks qualia structure provides the germ of an answer. One way to phrase the hard problem is to say that experiences have an intrinsic or unanalysable quality, such as redness or joyfulness. Red looks like something to us independent of whatever associations it may conjure. “Think of red while just ignoring all its relations,” says philosopher Hedda Hassel Mørch at the University of Inland Norway. “There’s clearly something still left to think about.”

But science doesn’t do unanalysable qualities. Everything in science is ultimately a relation to something else. That’s what equations do: express relations. “Most scientific descriptions are framed in structural terms,” says Kristjan Loorits, a philosopher at the University of Helsinki in Finland.

Tsuchiya thinks that experiences, too, may be entirely structural; they may not, in fact, have any intrinsic qualities. To him, this is the lesson of Buddhism as well as integrated information theory, which identifies consciousness with the causal structure of information-processing networks. “Experience is all about the relationships,” he says. What seems like an intrinsic property may be nothing more than a dense thicket of these relationships. If so, you can capture it in an equation or other mathematical object; Tsuchiya has turned to a branch of mathematics called category theory for a descriptive language. Then science will be able to explain experience after all , and the hard problem ceases to be hard.

Most philosophers who are generally sympathetic to structuralism don’t go that far. But they agree that experience is, in some way, analysable and that the real question is why we feel that it isn’t. Lyre suggests our brains have some faculty that gives our experience the impression of having intrinsic qualities. Ron Chrisley, a cognitive scientist at the University of Sussex, UK, suggests a reason for such a mechanism. The brain is always monitoring itself and could easily fall down a rabbit hole of self-doubt. “Yes, I believe there’s a plate on the table, because it looks to me that there’s this oval thing, and there’s this shading here,” he says. “But why do you believe that there’s that oval shape and that shading? If it always has to give an answer, it’ll be caught in this infinite regress of justification.” At some point, the brain has to stop analysing and take its experiences as unanalysable.

For Loorits, the unanalysability of qualia is an important feature of human psychology. Great art often violates our expectations and leaves us in awe. “It’s beautiful, and wow, I can’t say why,” he says. But this is a temporary condition. We talk to our friends, read the critics and find a language to analyse our feelings. “Beauty that seems unanalysable to me now maybe becomes analysable by me,” he says. Experiences are purely structural from a third-person perspective, but we can perceive them as unanalysable from the first-person perspective, until we adopt some critical distance and come to view them structurally.

Whether sublime beauty is just one among innumerable qualia that are susceptible to Tsuchiya’s methods remains to be seen. Still, these attempts to map the connections among all kinds of experience will continue to alter how we see ourselves. If it is true that experiences are all relative to one another, then if you come to view one thing differently, it will transform how you view everything else, says Lyre. People with synaesthesia, who might taste words or smell colours, perceive these connections directly, but all of us make them. “To a certain extent, we are all synaesthetes,” he says.