A human artist while painting an image on a canvas puts a paintstroke, then another and so on. It then pauses and looks at the full canvas and replans. Jean Oh’s Lab at Squirrel Hill, Pittsburg is researching with a robotic arm busy at painting . Dr. Oh is the Head of the robot Intelligence Group, Carnegie Mellon University. He has worked in areas of robot vision and autonomous aviation.
He focuses on what is known as the sim-to-real gap. In other words, how machines trained in a simulated environment act in the real world. Stable Diffusion tries to bridge the gap between simulation and physical works of art.
A project called FRIDA after Frida Kahlo has been initiated. FRIDA is an acronym for Framework and Robotics Initiative Dr. Oh believed that the process of moving from language prompts to pixelated images to brushstrokes is complicated, as the robot has to deal with the noise of the real world. Others insisted on developing FRIDA to improve the interface of humans and machines through art.
Image generating software responds to prompts. Such generated images exist only in the simulated world of computers.
In FRIDA’s experiment, they used a paper with 130 different brushstrokes. All these corresponded to a robot’s motion. These were programmed into a diffusion model. They took pictures of brushstrokes, model that interaction. They tried to get a really accurate simulation of brushstrokes grounded in what a robot can actually do. A prompt made the model to create an image of a frog ballerina in pixels. However, the image was in configurations possible for the robot to paint using those 130 brush strokes.
Researchers developed a way for the robot to step back a little occasionally from painting. This is to gauge how close it was to the desired goal in pixels. It then revised that pixelated goal. Every few dozen brush strokes, a photo was taken of the work so far. It paused and resumed work.
It is akin to how human artists work. Of course, it is mechanical, algorithmic and statistical.
Several such images painted by FRIDA robot are on display at the Lab. These images can be attributed to FRIDA.
Of course humans are involved in FRIDA’s painting — prompts to the machine, paint mixing, canvas set up, brushstroke number. There are paintings and photographs of other people to train FRIDA and other image generators. There is no point competing with human artists. The idea is to encourage human creativity.