And I don't mean Cylons.
Every now and again you notice a few trends coming together even though they hardly seem related at first glance. Someone asked a question today that turned out to be something I've thought a lot about over the years - a collision between tiny machine learning, privacy and an ocean of tappable ambient information.
Twenty years ago some of us were playing with sound. After spending a lot of time analyzing it we wondered what areas existed beyond voice recognition and music. There was a small "name that tune" excursion where we tried to match sung, hummed or whistled bars of music to - well - name the tune. We were trying to turn the sound into note patterns. It sort of worked, but the the problem of building a notation library was too difficult at the time. The easy way out would be a hammer and tongs approach - namely matching played music with a library of played music. We filed a patent on it, but never pushed hard as we didn't see the use case. Oh well...
Next I tried to recognize misbehaving car sounds.. Auto mechanics have great stories of people trying to make broken car sounds. Unfortunately the problem was not small enough and I shifted gears when a ventilation fan in my office started squealing.
Big heating, ventilation, and air conditions systems mostly run with boring industrial sounds, but a good HVAC mechanic can recognize when something's about to fail. They also tune these systems for efficiency by ear - for some systems the sweet spot isn't far from the point where problems happen and every installation behaves differently. With my trust Sony DAT recorder I made recordings of these systems in various stages of failure. Then some fancy pants signal processing and, lo and beware, you can detect and analyze failure modes. There was probably a lot of interesting work that could be done, but my organization evaporated around that time.
I was fascinated with low power computing a bit earlier than the sound work. How little energy does it take to do a computation? (you can make some interesting entropy arguments and get down to basic physics.) More practically what could be done with very low powered sensors that didn't need batteries or wires? The interest took several directions - all good as you slowly begin to learn.
Then about five years ago it struck me that the HVAC problem could be done more easily with machine learning. I've expressed reservations about the misuse of machine learning and regard some of it to be not very different from phrenology, but potentially very useful where you can understand it and its use.1 I supervised a senior thesis project that attacked the HVAC project using machine learning and a microphone. It showed some potential, but for me the learning was finally realizing how computationally cheap machine learning can be at the low end. I also ended up learning a fair amount about machine learning as the best way to learn is to do it yourself and then teach.
Given Bitcoin mining efficiency and the massive machine requirements at FaceBook and Google it isn't all together clear, but think about unconnected machine learning on your iPhone and perhaps your Apple Watch. In conventional computing the CPU deals with a large number of instructions compared to the type of processor used in machine learning. ML processing chips have a large number of tiny processors that work in parallel. Most ML is just matrix multiplication. While there can be a lot of steps, each is really simple .. generally just addition and multiplication. A properly designed ML element holds the numbers in the matrices its working memory cache. It's much more efficient in time and energy use to keep what you're computing nearby rather than having to constantly read and write it to memory. You should be able to do a specialized bit of silicon with enough ML horsepower to do some rather surprising tasks while using very little power. And you can do a lot on a smartphone if some of it's silicon was designed to support ML.
I think Apple understands this at a rather deep level. They're very careful about how to efficiently deal with graphics processors which is shorthand for saying they have optimized how to do huge volumes of very very simple computations. The chips in the current iPhones are optimized for the task. They've build machine learning machines you can hold in your hand and, in theory, run ML applications that don't connect to the net and divulge information. The new measuring app on iOS 12 is an excellent example of a local machine learning deployment. I don't know if it is being done on the watch, but there's no reason why they couldn't. Perhaps the biggest design advantage Apple has these days is silicon.
machine learning in your hand rather than the cloud.
I can think of a number of applications that use the sensors on the phone or watch -- from health, to motion analysis (think sports).. why you could imagine wearing a watch and using a couple of iPhones on tripods or held by friends to analyze your volleyball, gymnastic, or whatever moves. A personal coach or a serious tool is you already have a coach.
Exciting stuff, but I suspect it goes much deeper.
A back of the envelope calculation suggests you can do useful ML with micropower. I'm sometimes asked to come up with senior projects in engineering classes (dangerous as I'm not an engineer). I think you could do the HVAC work with an accelerometer or microphone and simulate the type of compute power that, if done in silicon, could run for months on a watch battery. Maybe some very simple chemical analysis. Or recognize a very small vocabulary -- perhaps five words - to control your toaster, thermostat or anything else. You could use a key word to make it listen or shine a laser pointer on it. You could also build a very simple image recognizer that would detect someone's gaze. Look at your toaster and say medium please.
Way safer than saying medium please to a Cylon.
A need for privacy may drive this even deeper and away from the Net. You can sit down and go through a few dozen applications in an afternoon. It seems very rich and very latent. Apple knows how to do silicon and low power.
So many possibilities.. perhaps even my dream "big data" project of listening to the woods.
1 I've written and talked a fair amount on problems. It can get technical and I won't spend more time on them here, but it's a serious problem as surveillance systems transition to judgement systems.