Algorithms promise to expand the limits of the human visual system, and in a way that might transform industries from medical imaging to security.
For humans, seeing really is believing. That’s probably because sight is the most sophisticated of all the senses. So sophisticated, in fact, that 40 per cent of our brain is devoted to processing visual data.
If scientists could find a way to create artificial seeing systems with human-like capabilities, it would represent a major technological breakthrough. Computers would be able to replace human eyes, both in the performance of labour-intensive tasks and in more complex processes such as identifying individual cancer cells among normal tissue. Such systems could even help visually-impaired people to “see.”
When we see, our eyes detect only raw visual information, such as colours and shades. Our brain then processes and translates the data into meaningful and coherent pieces of information.
The problem, though, is that many of the mechanisms that enable a human brain to analyse visual data and make sense of it remain a mystery to brain researchers.
Even so, rapid progress is being made in the science of computer vision. At the Weizmann Institute of Science, we are developing mathematical algorithms that will eventually enable a computer to imitate a function of a human eye. This technology could potentially shed light on our understanding of the human brain. But even if it doesn’t, it may still be useful in building artificial seeing systems.
We have discovered that one of the basic principles of natural images is efficiency. Images are composed of lots of tiny repeating elements, and we have devised an algorithm which can make “intelligent guesses” to fill in whatever information gaps are missing from a picture, based on a series of tiny repeating elements in the surrounding background.
Our algorithm can reconstruct visual information about a scene that the camera has not been able to record clearly, either because of problems with resolution or because objects partly block what we’re looking at.
In the images below, the photograph on the far right is a computer-generated version of what the original photo – on the left – would have looked like without the bungee jumper.
The algorithm has essentially produced a realistic representation of the unknown background scenery based on what is statistically likely to be there. It does that by analysing the repeating elements contained within the photo at different visual scales.
This algorithm is already in commercial use. It forms the basis of Adobe Photoshop’s “Content Aware Fill” function. Using a system that fills the missing parts of a picture based on the available data in the surrounding area, users can remove objects from an image, restore a damaged part or expand the boundaries of a cropped image.
Artificial visualisation systems have great potential. We can apply our algorithm to much broader aspects of everyday life. They can be used, for example, to enable security cameras not only to record visual data but also to alert if an irregular activity is taking place. They can also provide manufacturers with computer-based means for identifying defects in faulty products.
Elsewhere, computer-based vision systems can enhance a medical diagnosis of a patient. In one of our projects, we are adapting the principle of repeating elements to increase the resolution of MRI and microscopy data. This line of research could potentially lead to improved medical imaging diagnostics, for the benefit of patients, doctors, and researchers worldwide.
French artist Paul Gauguin once said: “I shut my eyes in order to see.” In the not so distant future, we may be able to shut our eyes, leaving computers to do the seeing for us.
