Om recently asked for thoughts about the impact on retina displays will have on video on the web. I don't think it is a big thing - unless you are talking about smartphones and 3G or LTE. It reminded me of an old question we used to ask
what is the effective bandwidth of human senses?
The idea was to try and understand what it would take to provide a fully immersive experience to a user. After thinking about it for awhile you come to the conclusion that the question is just silly. Let me explain...
We live embedded in a sea of an enormous amount of information. Some of it triggers our senses. We usually think of the non-contract senses of vision and hearing first, but both of them are only a narrow slit on the reality of what surrounds us.
We only respond to a tiny segment of the electromagnetic spectrum - photons with wavelengths ranging from about 0.4 to 0.7 microns. It turns out this is a an artifact of light from the Sun that reaches the Earth and most organisms with vision respond to a roughly similar range and photosynthesis takes place in this range. Really useful, but not surprising.
Some creatures have the ability to "see" at much longer wavelengths - infrared radiation. Some snakes can sense warm blooded animals. They are looking at light emitted by the heat of the body. It turns out a person shines brightest at wavelengths about twenty times the wavelengths of visible light - about 9.5 microns.
These long wavelengths don't pack much energy, but there are a huge number. Our eyes are very selective. We are surrounded by photons of a variety of wavelengths - about the width of your palm for the wifi radio in your laptop. FM radio ranges from about 3 to 3.3 meters, AM radio from 175 to 550 meters and you can find manmade signals that are much longer and much shorter.1
In addition to not being able to see very much of the spectrum we can only sense a small range of possible brightness (although it is pretty impressive as animals go) and we live in a world where we time unfolds in a linear fashion. We don't perceive times shorter than about a thirtieth of a second and we have difficultly dealing with long periods of time and doing other forms of analysis. For example scientists and engineers find it useful to look at signals for some length of time and examine their frequency content rather than how they unfold with time. The time domain and frequency domain are two ways of viewing the same reality and our mind is embedded in the time domain.
Recently a video appeared that showed some very rapidly changing content superimposed on a normal speed event. It is a special effects trick, but gives a sense of what a scientist or engineer is doing when they are observing an event in the frequency domain. Take a look as it is superbly done!
I won't dwell on hearing or the other senses much other than noting hearing sorts out a very complex signal and sends it to the brain where it is reconstructed into a three dimensional soundscape. It is much less stable than vision and speech is processed in a different region than music. It brings to focus the old philosophy question about a tree falling in the woods. The right answer, of course, is "what do you mean by sound?" A neurologist or physicist would say that our ears detect compression waves in a fluid turning them into electrical signals that are processed in the brain where "sound" is produced. All that exists in the real world is compression waves. If nothing is around to "hear" the tree falling there is, in fact, no sound - just the energy propagation of the compression waves.2
Before moving on to a different way of thinking about Om's question an interesting question popped up in my mind that I will calculate.
Why is it that it is dark (at least mostly) when we close our eyes?
A crude answer is that our eyelids (or a blindfold) does a pretty good job of keeping out light our retina responds to, but why does our retina respond only to light in that range? To set up the problem a bit more dramatically let's ask another question
If I close my eyes what is the change in energy within my eye?
It turns out there will be energy from radio waves, visible light, and infrared light. Some of the radio waves are natural (the Earth, Sun and Jupiter generate radio waves on their own) and some are manmade. I'll ignore them here claiming usually the two big components will be infrared from body heat and visible light. So rather than an exact number I'll be calculation an upper limit on the contribution from visible light. That is good enough for the question I think.
Now is the part where you could let your mind wander or check your email if we were face to face as I'm going to my pad and pencil for about five minutes of thinking,
OK I'm back - here is the calculation....3
The energy in each eye from the infrared heat of your body and the room is about a thousandth of an erg and the energy from visible light is about four billionths of an erg - several hundred thousand times less! So close your eyes and it gets dark even though the energy in your eyes is only dropping by less than a factor of one hundred thousand! Why it gets dark - why your eye is so selective requires a bit of quantum mechanics to explain.4 It is neat that there are such terrific filters to sort out parts our brain can deal with. A neat example you can play with yourself is to build a crystal radio and make a mechanism that lets you vary the frequency the little circuit is sensitive to...5
[wow - I've been going for over an hour and I haven't managed to get to the real question I wanted to address. Such is life. I'm going to give myself more time, but I'll refrain from editing as per my personal requirement for these posts.]
So we have these sensors in our body sending electrical signals to the brain to assemble a good enough representation of the real world for us to make a bit of sense about it. It would be sweet to perceive in the frequency domain at will, but that would not make evolutionary sense. Likewise vision and hearing with response times in the minutes or hours would have been an evolutionary dead end. Our limited color sense, sound field reconstruction, and flavor (a combination of smell, touch and taste - and possibly with some contributions from sight and hearing!!) are all good enough, but don't fully represent the real world.
How would you calculate the bandwidth from the eye to the brain? It turns out HD TV is close to a limit and that can be done with good enough compression at about 3 megabits per second delivered to the eye. You can do better, but not much better. But that is only a signal sent to a display. Vision synthesis is a different beast entirely.
It turns out several researchers have asked the question. While approaches vary and there are a lot of estimates (the exact mechanism is not fully known by any means!), some hammer and tongs work and extrapolations by Koch et. al. (pdf) at UPenn and Princeton a few years ago using frogs and guinea pigs and extrapolating give about 10 megabits per second - about the capacity of a 10baseT ethernet cable. These numbers will refine with time and mechanisms will become clear, but the bottom line is the data rate is not terribly fierce - certainly much less than the internal data rate of a HD video camera (nominally about 30 Mb/s before compression).
We greatly enhance our senses with a combination of specialized sensors and computers to process the information. Some of this is very simple. I made a few trivially simple bat listening devices that shift the very high frequency sounds of bats (100s of kHz) into a region our ears can hear. It is amazing to hear them hunt in real time. Of course you can look at frequency patterns on a spectrum analyzer and learn much more about what they are doing - but sensing that in real time in a "natural" fashion is beyond our perception. It is certainly easy for amateurs and even kids to build aids that allow us to extend our range. Perhaps sensing magnetic fields like a migrating bird (some of them appear to visualize the fields in their eyes one species only in one eye!), or the low frequency electric signals of whistlers from lightings storms around the world. I have a simple night vision scope that lets me view images in the infrared with wavelengths up to about 1.5 microns and have used kit that allows me to easily see the 9.5 micron signature of a living human. You can hear the very quiet and see the very faint. These are tools that allow you to answer all of those questions and come up with deeper ones. They allow us to study the real world that lives beyond our tiny vista.
And a final thought that I remember from a walk with Juliette in Manhattan a few years ago - the brain's ability to deal with compressed signals is impressive. Garrison Keillor remarked that you "see much more listening to radio rather than watching TV..." And writing can be even more compressed. Consider the image compression here
Don't think about a pink elephant waltzing in Times Square on a hot summer day
So what is the most parsimonious path from the boundary of the body to the brain and the image? Reading the words on a page, listening to someone speak them, tapping morse code on someone's hand...?
Sorry for all of the words this time. One gets on a roll of sorts. The recipe follows the footnotes and is both delicious and mercifully short and easy.
1 One sign of technical life on Earth would be the 50 and 60 Hz signal from AC electrical lines. 60 Hz waves have a wavelength longer than the distance from New York to San Francisco.
2 Similarly color and scenes are constructed representations of the brain. Color is a very artificial concept although most people think in terms of a correspondence between wavelength and color. If that was true color and printing would be quite impossible. In fact we can get a reasonable approximation of what our eye/brain combination puts together with three wavelengths and very accurate reproductions can be had with six or seven.
3 This is in the shorthand that a lot of folks use. Other people expect a lot of equations (there are several integrals here with some complicated terms - the 4/c term to go from radiated power to energy density, for example would fill several pages of notebook to work out, but I've done that in grad school, so why bother?), but there is no need to write them down as their manipulation is obvious to the person writing them.
Many people seem disappointed when they see real blackboards or notebooks for the first time - usually not that many equations unless someone is doing some theoretical work, but rather a lot of intermediate steps and drawings as visualization is often critical (although not in this case) They aren't meant to be instruction, but rather to gather the significant steps and provide a path for backtracking and taking different approaches. This is a manifestation of play looks like:-)
The caricatures presented in the entertainment media are just that - caricatures. "Big Bang Theory" socially has much more in common with the residents of an MIT undergrad dorm (say Baker House), than anything you'll find at the practicing scientist level.
4 OK - I'll take a stab at it from what I know about the retina. Note this is not a detailed biological look at the process! When visible light strikes photoreceptor cells in the retina the energy in the photon is converted to chemical energy and sends a signal to the brain in the process. A bit deeper the photons are absorbed by a retinal which isomerizes in the process. OK - the two bit word isomerization is key. It is the change of structure; leaving the chemical formula for the molecule in place. It is similar to the energy levels in an atom - just at the molecular level. Molecules are picky about the energy of these states - they are quantized. So the infrared light from the 310°K blackbody radiation lacks the energy to change the form of the molecule. No matter how many photons - nothing happens. If some have enough energy (the energy of visible light in this case) the molecule isomerizes and the reaction goes. Sort of like the photoelectric effect.
The bottom line is this is a quantum effect! The photoreceptors have become outstanding filters keeping out the longer wavelength photons. So that nearly million to one energy density advantage for the infrared radiation suddenly doesn't mean anything. Even a 1 micron photon won't trigger the reaction.
5 There was a lot of progress in a thirty year period on radio sensitivity (how faint a signal it can detect) and selectivity (how accurately it can detect the signal you are instructing it to find while ignoring other signals. It turns out this was codified in radio specifications and in the past forty years technologies have appeared that allow us to move beyond radio of the twenties and thirties. If we were to build modern receivers the so-called spectrum shortage simply wouldn't exist and wireless companies would not have the ability to sell spectrum as if it was in short supply. Spectrum shortage is an artificial concept made real by legal convention and not by physics! Basically we have very crappy receivers compared to what is technically possible. I hope we move to modern receivers as quickly as possible so artificial constructs like bandwidth limits - at least for the portion sent over the air rather than backhaul from the towers - go away completely. It has to happen eventually and it will change the wireless landscape.
Vanilla Frozen Yogurt
This is easily as good as what you will find in the average frozen yogurt store and much less expensive. If you have an ice cream freezer it is trivial to make. You can easily scale the recipe - the yogurt/sugar ratio is 5:1. Also mix in fruit, nuts and chocolate!
This is reasonably low fat and very high protein.
500g Fage 2% plain greek yogurt. Full Fage will is amazingly rich and still less fat than ice cream if you want to go in that direction and 0% Fage is tolerable, but less satisfying.
100g white sugar (it is also fantastic with grade B maple syrup!)
1/2 tsp high quality vanilla extract
° Thoroughly mix the ingredients in a large enough mixing bowl
° Chill in a refrigerator if they aren't pre-chilled
° Churn freeze until it is the consistency of soft ice cream. About 10 minutes in my freezer.
° mix in whatever you want and enjoy!