It seems Apple released a new smartphone yesterday. This one has several reasons to make me want to upgrade, but the most drool-worthy is its power as a computational camera. I'll leave it to other to write the reviews and compare it to Google's efforts - a new iPhone isn't in my budget this year. But perhaps a few words on computational photography are relevant.
Computational photography has been around for over sixty years.1 In particle physics you slam particles into each other and carefully study the wreckage that emerges from the collision. About sixty years ago bubble chambers came into use. The idea was to record the wreckage in three dimensions. A chamber was filled with a transparent liquid like liquid hydrogen, kept just below its boiling point. Just before a fast moving particle entered the chamber a piston at the bottom of the chamber was dropped suddenly causing the liquid to superheat. The incoming particle collides with a particle in the liquid - usually a proton if the liquid is hydrogen - and charged particles from the collision leave a trail of tiny bubbles in the sensitized superheated liquid. Three cameras at right angles to each other simultaneously captured three views of the bubble tracks and the piston is moved back into position waiting for the next collision.
A large magnet surrounds the chamber causing the particles to bend one way or another depending on their charge. The slower a charged particle moves, the more it curls. The photographic sheets are developed and "girls" - it was considered women's work - would trace the tracks by hand with digitizers creating lists of coordinates for each track that a computer program could use to compute the fundamentals of each collision.2 This type of computing was the most demanding of the day. In the late sixties and beyond recording the collisions directly in digital form using a variety of "counters" took over and the scale of experiments and necessary computation exploded.
By the late 70s people were beginning to categorize medical x-rays and ultrasounds computationally and resonance imaging began to emerge. The computational requirements were stiff in the day, but your phone could easily handle the computing required for your MRI. It has been argued that this form of computational photography has had the largest impact on human health of any form of computation.
Playing with photographic images from telescopes and microscopes was necessary in science, but the techniques quickly migrated to computer science labs. Just as the Utah Teapot was the universal test for 3d imaging, a photo known as Lenna was the ubiquitous test for imaging algorithm research in male dominated CS departments during the 70s and 80s. It wasn't unusual to see her image appear in several papers in a single journal issue. Journal images today aren't as sexist.
Some of what went into academic image processing made its way into Photoshop in the late eighties. Photoshop went on to be a preferred platform for creative artists to do computational post-processing of their images. Take a photo on your camera and play with it on your computer.
But what if the camera and computer could be combined into something that could redefine imaging? I'd argue that took place in science some time ago as the imaging part of telescopes, microscopes and particle detectors had began to require built-in computation. It's been in the past few years we've seen the emergence of a new type of photography in handheld cameras. Smartphones, with vast development resources, are leading the way and we're on the verge of an explosion of new techniques and ideas.
At this point it would make sense to talk a bit about the basics of 2d fourier transforms to get a sense of how that's done. Here's a short non-technical introduction that gives a sense of a basic function.
The computational power of the latest smartphones is staggering and, at least in Apple's case, a good deal of it can be devoted to computational photography. Currently much of it is used to improve snapshots that would otherwise be unusable. Noise can be reduced while still keeping important detail - often using psychological models of human perception, the dynamic range of an image can be extended by taking a number of shots and bringing out detail in dark areas and suppressing burned out overexposed regions, realtime video can be overlayed with other information creating a computation synesthesia of sorts, machine learning is becoming useful in many ways, .. the list goes on and on.
I'll offer one that could appear in about a year with the next generation of smartphones. Time of flight image sensing is now sort of practical for a smartphone and will probably be a feature in high volume phones like the iPhone next year. Basically a pulse of infrared laser light is sent out to an object and the time it takes to go out and return is used to calculate the distance. This is done for a somewhat smaller image than your camera takes - a few hundred thousand to a million pixels. But these pixels now have distance information in addition to enough image information that it can be used to create a distance map for the full image from the main sensor. This could be very useful in augmented reality, but here's one I'd like to see. Take two or three phones on tripods and point them at a dancer, athlete or someone getting physical therapy. A very accurate motion capture video could be quickly made and you wouldn't have to spend $50k and up on specialized equipment. Coaches and physical therapists would be delighted.
A thousand other uses will emerge we haven't thought about. Independent of their platforms there are strong reasons why Apple needs a great computational camera. What will separate winners from losers will be usefulness, playfulness, and great user interfaces and experiences. That's the hard part.
For the best photos a good photographer is still required and the tiny lenses and sensors on smartphones are limited by basic physics. But always having a camera with you and one that can correct basic mistakes is a serious plus. That and there are emerging capabilities that will take photography in new directions.
__________
1 digital computational photography. I won't go into details here, but optical computation has been around even longer. If you're interested in the subject realize a lens is an optical computer capable of creating a Fourier transform.
2 I like this example as a neutrino slams into a protron bangnig the vacuum hard enough that a D+ (d plus meson) appears. It is composed of two quarks: charm and anti-down. I have a fondness for charm quarks:-) Also this was made during the twilight of bubble chambers in the late 70s.
Comments
You can follow this conversation by subscribing to the comment feed for this post.