Robot Finger Feels in Color

Imagine running your fingertip over the surface of a U.S. penny. You would feel the ridges of the raised letters and numbers, Abe Lincoln’s bearded side profile, and, if it’s tails, the fluted columns of the Lincoln Memorial. Getting a robot to sense the same things is an imposing task, often requiring gathering data on pressure and force at many spatial locations at once. But, that’s just what a team of scientists in Europe has now managed to do, using an unusual, colorful robotic skin that provides high-resolution sensing in real time.

“To be honest, when they showed us this, we thought it was, and pardon my French, [expletive] cool, because it’s a distinctly different approach,” recalled Rich Walker, director of Shadow Robot, the UK’s longest-running robot company, which primarily focuses on robotic hands.

RELATED: “This DIY Bipedal Robot Used Pneumatic “Air-Muscles” Instead of Motors”

The research team, which hails from Queen Mary University of London, the University of Florence, the University of Trieste, and the University of Trento, designed a robotic fingertip with a synthetic skin that reflects different colors of light in response to mechanical deformation. By reading the light reflected off the skin, the fingertip generates maps of topology, strain, and contact pressure. The team has already used the sensor to generate maps of a human fingertip, a penny, and a leaf.

Giacomo Sasso, a postdoctoral research associate in the lab of Frederico Carpi at Queen Mary University of London, came up with the idea for the sensor. He had been researching optics when he stumbled upon an interesting paper published in the journal Nature. It described the “mechanochromic material” that would eventually make up the reflector in the skin.

Following the method described in Nature, Sasso exposed a light-sensitive film to a 5-megawatt, 635-nanometer (red) laser for seven minutes. The laser beam creates an interference pattern which causes the film to polymerize in alternating densities, creating layers with different refractive indices.

This structure is called a Bragg reflector. The alternating densities and refractive indices in the polymer cause specific wavelengths of light to be reflected. When the reflector is deformed by contact with an object, its layers are stretched, becoming thinner and reflecting light of a different wavelength.

It took Sasso less than a week to recreate the material in the lab. “From there, we started seeing how we could translate these color patterns into something that was useful for us,” he says. Soon, they realized that the color produced by the material was all they needed to be able to sense the topology of objects.

In the robotic finger, the Bragg reflector is sandwiched between a layer of silicone which protects it from the outside, and a transparent, fingertip-shaped silicone finger with a camera and LED light embedded inside of it.

The light from the LED shines through the clear polymer of the finger. When the fingertip is deformed by an object, the reflector bounces light back to the camera, with wavelengths depending on the level of deformation—red for least deformation, shifting to green and then to blue when most deformed.

The team also made adjustments to increase the sensitivity of the skin and help the camera to better read color differences. The silicone of the outer layer of the fingertip is colored black to increase the color contrast, allowing the camera to better translate color into the morphology. The rigidity of the camera inside the finger also enhances the deformation of the reflector, producing greater differences in reflected wavelengths.

After all that optimization, the finger provided 100-micrometer resolution with no computational latency, the researchers determined.

What robot fingers need

Human skin takes in a variety of tactile information in order to successfully move and manipulate objects, including temperature, texture, pressure, and vibration. But engineering a robot to do the same is challenging because of spatial constraints. There often isn’t enough room in a robotic fingertip to incorporate more than one type of sensor. The question then becomes: which type of sensor should be used?

“And the answer to that is… that’s a really hard question. No one knows yet,” says Walker. Carpi’s team’s robotic finger is exciting because it presents yet another option for roboticists to experiment with, Walker says.

Although Carpi’s team isn’t the first to use soft materials for tactile sensing, its technology is unique because it is able to extract quantitative information about depth and size from the topologic maps it generates. According to Walker, most sensors can only generate topological maps which reveal the relative sizes of object features.

To Sasso, another key advantage of this robotic finger is that it embeds tactile sensing directly into the material of the finger, rather than using taxels, or pixels that measure force or pressure at specific spatial points.

RELATED: Robot Hand Manipulates Complex Objects by Touch Alone

“The core aspect of the sensor is that we’re essentially [moving towards] having the sensing element at the material level,” he says. “The camera, which is a very highly optimized electronic component, is translating whatever the material is already doing directly into digital signals.”

Michael Wang, co-founder and chief scientist at Daimon Robotics, which, unlike Shadow Robot, primarily uses vision-based sensing, echoed Walker’s sentiment that it’s beneficial to explore new methods of sensing which may bring unique advantages. But, he also explained that soft materials often face challenges with durability and that the significance of the team’s work would be revealed when the finger is integrated into real robot hands.

“The practical and useful benefits, especially in the context of robot hands, remain to be tested and validated,” he says.

When the materials of soft sensors, like the silicone in Carpi’s team’s fingertip, become eroded or damaged after repeated use, the signals measured by the sensors may not reflect objects’ topography as well.

“Especially if you have the electronics embedded into the material layer itself, that becomes a very challenging engineering problem. And I haven’t seen [many] good soft electronics materials that really can undergo long periods of usage,” Wang says.

However, because the Bragg reflector isn’t in direct contact with objects itself, the outer layer of silicone material acts as a protective barrier, Sasso says. The silicone can also be made more durable using certain chemical coatings, according to Wang.

The team has already been talking to companies that could potentially employ the new sensor. They also hope to improve the sensor so that it can sense objects that don’t lie flat on surfaces. That could open up its use in surgical instruments that require precise contact mapping of tissues and organs, Carpi says.

“There are significant developments that we expect with a clear path toward transition to real world applications,” he says.