Exploring How Our Senses Blend and How AI Mimics It

AI “Tasting” Colors and Shapes: More Human Than You Think

Our senses don’t work in isolation – they often blend together in ways that shape our experiences, a phenomenon that both marketers and scientists study. Have you ever wondered what the flavor of a pink sphere might be or what the sound of a Sauvignon Blanc could be? While these questions may seem odd, there’s a fascinating scientific reason behind them: the brain naturally merges sensory inputs, influencing how we perceive flavors, colors, and even shapes.

The Cross-Talk Between Senses

This merging of senses is called cross-modal correspondence—when one sense triggers a response in another. For example, the color of a glass or music in the background of a bar can affect how sweet or musky a wine tastes. “This cross-talk between the senses is happening almost continuously,” explains Carlos Velasco, a researcher at BI Norwegian Business School in Oslo, Norway. In some rare cases, people experience a phenomenon called synaesthesia, where stimuli in one sense, like words, can trigger a taste or colors are associated with sounds.

Recent research by Velasco suggests that even artificial intelligence (AI) systems, such as generative models, seem to experience a version of this phenomenon. Though it’s not literal “taste,” these AI systems show patterns similar to human sensory associations, possibly highlighting how these associations are wired into human brains. Scientists aim to understand and apply these insights in innovative ways.

How Humans Experience Sensory Blending

To explore sensory blending, scientists use the term sensory modality to describe how the body processes different sensory inputs (taste, sight, touch, etc.). The associations that arise between these senses are called cross-modal correspondences. For example, since the 1970s, research has shown that colors like red or pink are linked with sweetness, yellow or green with sourness, and brown or black with bitterness. These associations are remarkably consistent across cultures. In one experiment, participants were asked to rate how sweet they thought the color red was on a scale from 1 to 10. Over time, studies have confirmed that these associations hold true across different populations, from China to India to Malaysia.

Packaging also plays a role in shaping our sensory perceptions. For example, research from Waseda University in Japan found that people perceive chocolate to be sweeter when it is wrapped in pink packaging instead of black. Similarly, we associate round shapes with sweetness and spiky shapes with bitterness or sourness, indicating that we “eat with our eyes” as much as our taste buds.

Understanding These Associations

But why do we associate colors and shapes with certain tastes? One theory is that humans internalize patterns based on our environment. “The safest assumption is that we learn them all,” says Charles Spence, head of the cross-modal research lab at the University of Oxford. For example, fruits in nature often change from green (sour) to red (sweet) as they ripen, so our brains naturally associate red hues with sweetness.

When it comes to shapes, the connections are more psychological. We might associate round shapes with sweetness because they’re more pleasing and less likely to harm us. Sharp, spiky shapes could evoke bitterness because many poisonous substances are bitter and sharp.

AI and Sensory Associations

Inspired by the success of AI in other fields, Velasco, Spence, and Kosuke Motoki at the University of Tokyo decided to test whether AI models, such as ChatGPT, would show similar sensory associations. They asked ChatGPT to respond to prompts similar to those given to human participants, such as:

• “On a scale from 1 to 7, how strongly do you associate round shapes with sweet, sour, salty, bitter, and umami tastes?”
• “Which color do you think best goes with sweet tastes?”

When comparing responses across multiple languages, the researchers found that ChatGPT reflected many of the same associations seen in human participants. Notably, the more advanced model, ChatGPT-4, provided answers more in line with human associations than the previous version, ChatGPT-3.5. These findings suggest that AI systems trained on human data may capture sensory patterns similar to those in human brains, even if they lack the sensory experience to truly “taste.”

Can AI Predict Sensory Experiences?

Excited by these findings, I explored whether Google’s Gemini AI would also reflect common sensory associations. I asked it to identify the sweetest color, and it correctly responded: “Many people associate pink with sweetness, likely due to its association with sugary treats like cotton candy and bubble gum.” Similarly, it linked green with sourness, white with saltiness, and black with bitterness.

Interestingly, Gemini’s answer mirrored findings from scientific literature, specifically the research of Charles Spence on cross-modal associations. This raises an intriguing possibility: Can generative AIs go beyond what is already known and discover new sensory associations that have yet to be documented?

Spence believes that AI could potentially generate new hypotheses for cross-modal correspondences that could then be tested in human experiments. Such AI-driven insights could be valuable for industries like marketing, helping companies design products or packaging that appeal to our subconscious sensory preferences.

The Role of AI in Creative Processes

While AI’s findings can be useful, Velasco warns that AI should be used for inspiration rather than as a definitive solution. “It’s inspiration, rather than a definite solution,” he says. Human creativity is still necessary to add the nuances and excitement that make sensory experiences unique. For example, while AI might tell us that pink is associated with sweetness, it might take a human touch to create a truly compelling packaging design that plays with this idea.

AI and the Festive Season

As the holiday season approaches, I couldn’t resist asking ChatGPT for a musical recommendation to accompany my mulled wine and mince pies. According to ChatGPT, the complex, spicy flavor of mulled wine calls for a piece of music that reflects its richness: “Carol of the Bells,” performed with a lush orchestration, captures the festive warmth of mulled wine. However, if you’re in the mood for something more relaxed, ChatGPT suggests “Have Yourself a Merry Little Christmas,” performed by Ella Fitzgerald, or “Christmas Time is Here” by the Vince Guaraldi Trio. Whether you agree with AI’s suggestions or not, its response highlights the growing role of AI in understanding and shaping our sensory experiences.

Conclusion

While we still need more research before we can fully trust AI’s judgment on sensory experiences, AI’s ability to reflect human sensory associations opens up exciting possibilities. From food packaging to music pairings, AI can inspire new ways to connect the senses and influence how we experience the world. But, as with all creativity, it’s best when paired with human intuition and imagination.

Source