omenti

why is there an open can of tomato paste on the carpet?

 Yesterday Horace wrote with an interesting set of questions.  

 

Engineers typically are concerned with how, people generally with what.

Why may or may not be relevant.

Arguably concern about causality is a human condition and may be either an asset or liability.

So anyway, I really enjoyed this recently: Feynman on Why

Perhaps you might want to post something on this. Are scientists motivated by “Why” but prefer not to answer the question?

 

 Horace is a keen observer and deep thinker, so it's reasonable to take a crack at it.   I'll only address why.  Causality is interesting in it's own right.  I've written bits and piece but, it makes sense to eventually put together something a bit more formal.  An early part of his note has an implied question on the framework of stories.  While that's core to how I see the world - I maintain to how all of us see the world, I'll pass on it for now and just take on how and why questions.  

 

First watch the brief Feynman piece Horace mentions.  

 

He's in an interview situation answering off the cuff.  A slightly deeper view might say how questions are addressable and why questions never lead to a conclusion.  Scientists are fascinated by how something works,  but sometimes you're playing with ideas and asking why leads you into areas where you can ask how.  That's the common answer, but I don't think it's right.  Here's my take on it.

 

We live in a Laplacian universe where one instance follows the next according to physics, but there are facts that don't appear to have anything to explain them.  We just can't say there's a reason something happens because of the deep physics underlying it.  There seem to be contradictions. 

 

Definitions.  What do you mean by reason?  First off you have to recognize there are different kinds of facts.   Things that happen at specific times ..  like an open can of tomato paste sitting on the carpet (half of the contents were on the carpet and there were little footprints indicating some sort of celebratory dance occurred), and things that are specific features of the universe ..  the conservation of momentum,  a four dimensional space-time with three space dimensions and one of time, the conservation of energy, and so on...

 

The first category have a reason ..  a cause.  Here we can invoke physics and the prior configuration of the universe.  This is satisfying and works well if you're dealing with very simple things .. elementary particle interactions, nuclear decay, etc.  When we start talking about more complex things it isn't particularly satisfying to say "because physics."

 

As we move on the path from atoms to sociology (physics → chemistry → biochemistry → biology → neurology → sociology/anthropology ) the increasing levels of complexity allow new features to emerge.  We simply don't have the information and context to look at the appreciation of a Bach cello suite and go back to the laws of Nature.  We might deduce some of what is going on -enough to build models that predict , but it's mostly in the context of neurology and sociology. 

 

We have to accept why questions may or may not have an answer.   They don't exist by themselves.  You need to consider the context.  For example Spinor, one of my ferrets, is fascinated by pull tabs.  He'll grab a can half his weight and drag it all over the place.  It must be terribly exciting as he's completely pilo-errected and will fight for possession.  Our house has objects regularly appearing where they shouldn't.  They tend to weigh less than a kilogram and tend to have pull tabs.  If it was a flower pot and was near the window I'd suspect Y?-Y?, a small female ferret who likes to dig out flowers and push over the empty pots.

 

If you're playfully curious you're asking a lot of why questions.  They can be unconstrained and unexpectedly rich letting your mind soar, sometimes leading to interesting territory.  Maybe there's an answer, maybe not.  You need to know how far to go and generally they're the stuff of play.  How is a more direct path.  You're not guaranteed to find an answer, but it is usually bounded and that's a good thing as time and money are finite.  

 

Of course how questions don't always pay off.  In fact, at least in physics, individual attempts to probe deeply usually fail.  But failure has a different meaning than what the dictionary says and we celebrate as something was learned.  It is important to know when and why to stop.  Very few people have sorted out that skill. 

 

 

 

 

going beyond regular flights over the arctic

Two days ago a friend told me about a wonderful auroral display over Greenland and Canada as he flew from Paris to San Francisco.  We're used to the notion of great circle paths. The shortest distance between two points on a sphere is a geodesic - the fancy pants generalization of the concept of straight line to curved spaces.¹

 

I didn't think about this until a professor who I corresponded with as a kid sent a letter that told me to draw an equilateral triangle on a sphere, the seat of a saddle (I did have a Montana address and knew what a horse was), and a sheet of paper and measure the angles.  Next draw two parallel lines.  Finally I was  to draw a circle on each and tell him what π meant.  I won't offer a drawing. If you haven't thought about this, give it a try.  Euclidian geometry falls apart.

 

People started thinking about this seriously a few hundred years ago.  A specific geometry was developed for one class of curved spaces - a certain type of space where the equivalent of a plane would fit hug the surface of a saddle seat.  Interesting, but limited and mathematicians lust for generality.²  

 

Bernard Riemann went off to college to study theology - one of the few fields where a college degree offered reasoanble employment prospects.  He went to a lecture by Gauss and gave it all up to study math.  A few years later, as the Civil War neared in America, Riemann was going for Habilitation - a sort of  Ph.D.++ in Germany.   He suggested a few dissertation topics and Gauss told him to work on the one Riemann thought boring - namely push the foundation of geometry.  After a year and a considerable amount of time the heavens opened.  A short paper that allowed you to understand the geometry of a space of any number of dimensions with any curvature at any point in space.  My mentor said it was like Euclid had a few lines in the sand for counting, the first non-Euclidian geometry mathematicians moved to  a hand crank adding machine and Riemann developed a supercomputer that we'll have in a hundred years.  

 

People still wonder what kind of coffee he drank

 

He thought about the intrinsic geometry of space ..  not how it might look from the outside.  For him each point in the space would have information relevant to the space.  I won't go into the math, but tensors are perfect containers for the book keeping. But the real breakthrough (actually the first item - looking at space from the inside was a huge breakthrough) was realizing that everything you needed to know about the geometry of a space was encoded in the distance along any curve you might draw in the space and that distance could be written in the form of a tensor he called the metric.   So if you tell me any segment of a curve of a space you're fond of, I can use the metric and tell you how long it is.   From this you can build up and calculate areas, volumes, geodesics (like flying from Paris to San Francisco), angles and everything you'd like about geometry.  You could even talk about the path of a bug walking on an apple.  Geometry from the ground up.

 

It was beautiful math and for about 50 years mathematicians had it to themselves.  Then this guy named Einstein comes along with this crazy idea that gravity is really the bending of spacetime by a mass.  (This is where all of those lame demos with balls and rubber sheets come from.)   He had no idea how to approach the problem mathematically. After all, this space-time would have constantly changing curvatures.   Riemann's work was an almost tailor made for the task.  He worked with a people more conversant in the math for awhile until the training wheels came off. 

 

If you've played with gravity this is all old hat.  I've avoided the math attempting to describe the beauty of Riemann's work - the  notion that any kind of space can be understood by looking at it from the inside and using this metric thingamabobby.  Some may be too difficult to calculate, but in principal it's always possible.  It turns out to be just the tool for a revolution in physics.  Physics that is necessary if the GPS in your phone is to work properly and physics that still stands after over 100 years as our deepest understanding of gravity. 

 

There are no lone geniuses in math or science.  It's all connected.  And the future of technology depends on it.  That part scares me given the current anti-science sentiment in government.

 

Oh - the book.  That's the undergrad textbook I had for General Relativity.  Beautifully written with a sense of humor.  One of the authors was very young at the time.  In a few days Kip Thorne gets his Nobel Prize for his work detecting the sound of some of the most violent events in the universe... events that generate ripples in spacetime  as predicted by Einstein with a theory made possible by Riemann. 

 

_____

 

And one more thing.  I've made this point several times, but it bears repeating.  

 

People tend to talk about some theories as replacing others.  While that's often true, sometimes the "superior" theory is just more general and reduces to the old one under certain conditions.  If you're large - say over a millimeter in size - and moving slowly - say less than a tenth of a percent of the speed of light, Newtonian physics is just fine thank you.  It may not be a deep way of thinking about the world, but for some practical matters like engineering, driving a car and playing volleyball it's just dandy and is much easier to use calculate than relativity or quantum mechanics.  

The stories we tell are are equally valid to use if they're relevant to the domain we're using them for and if they don't contract physical evidence at the domain level. 

__________

¹ In physics you'd say it's a locally length minimizing curve ...  the path a moving particle follows if it isn't accelerating.

² only surfaces of constant negative curvature.

 

 

a kid redesigns the olympic symbol

I was in the basement with the brand new Motorola 21 inch color TV and a children's telephone.  The wires snaked out and up to the roof where my Dad was on the other toy phone braving the Montana Winter as we tried to get a usable signal.  It was my first year of high school and the Winter Olympics would be on in a day or two.   The Olympics and we had a COLOR TV!  

 

We sort of got it working by the time the broadcast began.  The Olympic symbol was displayed and the story given

 

... the six colors [including the flag's white background] combined in this way reproduce the colours of every country without exception. The blue and yellow of Sweden, the blue and white of Greece, the tricolor flags of France, England, the United States, Germany, Belgium, Italy and Hungary, and the yellow and red of Spain are included, as are the innovative flags of Brazil and Australia, and those of ancient Japan and modern China. This, truly, is an international emblem.

 

It's a very clean design, but it offended my sense of mathematics.

 

My sister had very long hair at the time.  Long enough to sit on.  She was always working on various braids.  The year before I was curious about knots and braids and learned that a simple three strand braid becomes very interesting if you connect the ends.  Take three strands here colored red, blue and green.  Wrap them in a standard braid and go through one cycle so the colors end up where they've started.  (I like to think of braids as musical rounds)

 

 

Rather than repeat the process like my sister, bend them back and connect red to red, blue to blue and green to green.  You'll be left with three loops that are fully linked, but if you remove one the whole works falls apart.  Here's a nice image.  Note that they're not really two dimensional circles as they have to pass over and under each other.  Making this forces you to use three dimensions.  

 

Technically these are a special case of brunnian links - a nontrivial link (a link is a grouping of the loops here) that becomes a set of trivial (unconnected) circles if you pull out one of the loops.  Cut any loop and the whole thing can easily be pulled apart.  Do it with five loops and you'd have powerful symbolism for the Olympic movement - or so I thought.   I knew you can make brunnian links for any odd number of loops so it was time to draw.

 

I sketched a five loop version.  Cut and remove any loop and it all comes apart.  All I had to do was find some way of making a nice drawing.  My sister is the arty one.  She had all of the right kit including some high quality colored pencils.  She also happened to be smart enough to not let me use them - even if I had noble intensions.  

 

Plan B.  Every kid has Crayola crayons.  I selected five and, after a couple of tries, I had it.  

 

(pretend the colors are accurate.  my drawing here is pretty bad, but so was the crayon version)

 

The drawing went into a special international airmail envelope with an explanation of the math and symbolism.  It was addressed to the Olympic organization in Lausanne..  I have yet to hear from them.  Apparently crayon drawings from 15 year olds don't make the cut.

 

I draw quite a bit.  It's an integral part of how I think.  It isn't pretty and I consider it an ephemeral part of thinking and trash it as soon as it's served it's purpose.  If you want meaningful design it's best to use a real designer.

 

I'm looking forward to the games!

 

managing time, choosing your path and hanging on for the ride

 Every November Pip Coburn and his co-workers instigate a gathering in Manhattan that brings about a hundred and fifty people together.  I've been going for a long time and have met a few wonderful people.   If you look at backgrounds, I'm sort of an outlier.  On the other hand, from my perspective most everyone else is an outlier, so it's a good time to listen.

 

This year I had a time constraint and, other than giving a talk, was only able to  attend a few sessions.  One on creativity, another on the future of food and finally a chat with a group of articulate high school students. Three of the students - all seniors - hung around after my talk (self driving vehicles).  That conversation triggered this post.   

 

I mumbled something about techno-solutionism and cloudy crystal balls and turned the table to ask them a few questions - something like:   "tell me about screen use, face to face interactions, time management and our plans for the future".  It's much more interesting to sit back an listen rather than to lead them towards your preconceived ideas.  The sample was tiny and non-representative - I'm guessing all were from affluent families. But it's always interesting to ask questions.

 

They place great value on face to face time with their friends, but find it hard to come by as all of them are heavily scheduled.  Two of them said they hang around school for about an hour - their best time of the week.  All of them were stressing on getting into the "right" colleges and all were worried about picking the "right" career paths. Two were very worried about pleasing their parents.  None seemed to be worried about the cost of their education, but worried about what jobs in the future would pay. One had an interest in the arts, but thought she couldn't take that path.  To them college was training for work.    Finally all of them talked about multitasking and told me how good they were at it compared to their parents and teachers.

 

None of these comments were surprising, but hearing them in person brought me back to issues all of us deal with.  Time management for example.   A few people are self-driven.  They prioritize and make time often making sacrifices along the way. Again nothing new, but it's nice to see clear articulations by people who are good at it.  I told them to check out a post by Sarah Pavan. Sarah is a world class professional beach and indoor volleyball player - perhaps the only one who does both sports at such a high level.  Growing up she studied the piano and was a biochem major with a 4.0 average in college along with being one of the standout student-athletes in the country (she's Canadian, but went to college in the US).  Her recommendations are straight-forward and solid.  Although they're aimed at student-athletes, anyone can use them.  They aren't easy to follow, but that kind of focus is necessary if you intend on excelling in many paths.  Recommended!

 

I'm concerned about external over-scheduling.  I'm a great believer in more unsupervised time that allows kids to find and explore their passions.  I did that as a kid and had time management problems with my normal high school work. I made some choices and fortunately my parents went along with them.  I wonder if I would survived in the current environment and, if I did, what I'd be doing.  

 

There were some threads I would have loved to have followed more deeply.  How they found information and vetted it seemed shallow.  That may be an artifact of the change that is currently taking place.   And then the comment on multi-tasking...

 

Our brains do an enormous amount of multitasking, but almost all of it isn't available to our conscious mind.  A large amount of research suggests it is difficult/impossible for us to do true conscious multi-tasking.  Instead we context switch,.  You're thinking about something and someone walks in talking to you.  You park the state of what you were thinking about and focus on your visitor.  When they leave you have to move out what you were talking about and recreate as much of the original thought process as you can. And that's a simple linear case.   It turns out context-switching/multitasking isn't very efficient.

 

People can train themselves to context switch more efficiently, but it turns out it's not close to the efficiency of finishing a task and moving on to the next.  Some of the most efficient thinking - though associated with Csikszentmihalyian flow (I love that word:-) - effectively blocks other conscious thoughts from taking center court.

 

A conjecture is people who have trained themselves to multi-task are better suited to complex situations where many shards of shallow thought are the norm.  That would be very interesting.

 

We're always rewiring some of our brain - an enormous amount when we're babies and toddlers and massive chunks extending up until our late teens. Learning second and third languages changes your brain structure. The same with learning music and playing a sport seriously.  Some of these may turn out to be useful outside of their domain.  A high percentage of scientists were serious about music as kids.  I wonder if Sarah's brilliance on the volleyball court and in biochemistry were aided by music? 

 

So the question:  Is high screen usage doing it's own wiring and is that orthogonal to some skills?   The first part of the question is clear -  yes it does.  The second part is  under serious study now and we won't know much for some time.¹  We're living in a huge experiment.  We've been through similar experiments in the past, but this one is massive by comparison.  Will work change to match newer wiring?  Will it have to?  Stay tuned...

 

It seems like the wrong place to stop, so on to something a bit more up-lifting.   I mentioned a session on creativity.  It seems I'm approaching it wrong.  On the other hand what I do works for me.  I'm approximately curious and manage my time in such a way that I have the time and space to light the fuse and hang on.   A lot of work is involved, but it really doesn't seem like work   I'm better at it now than when I was in my teens and twenties - much better.  I've been lucky enough to work in places and situations that don't penalize my rather lumpy productivity.  One of my main goals is finding questions I hadn't imagined.  Answers are just stepping stones,.

 

It turns out being very curious is a good way to keep your brain flexible and "plastic."  As you acquire new areas they begin to connect up in ways you hadn't imagined.   Old thoughts even from high school come back with a new freshness and it's just exciting even if most of it seems to be impractical at the time.² 

 

Anyone can do it.  As you get older a new type of connection emerges.  Part of the wiring of nerve cells is surrounded by myelin - an insulating layer.  As you age the myelin sheath breaks down a bit effectively slowing down communication.  While that sounds unfortunate, the upside is  it allows for easier communication between some parts of the brain that normally don't link.

 

We like to think of neural paths as highways between regions of the brain.  Highly creative thought at any age is different.  It's slower and tends to meander here and there making curious connections and ideas come together.  It can be very slow and insight can come at the most unexpected times.  It'd take too long to describe it, but some neurologists refer to transient hypofrontality being a mechanism associated with creativity.  It's accessible with practice and may be aided with age.

 

There's a down side.  A couple of you have had to take me by the arm when we're walking engaged in joint thinking.  I'm not very good at knowing where I am when I'm thinking.  Having to save me from traffic is a good sign you're engaging my mind.

__________

¹There are too many other explanations, but anecdotally young people taking their PhDs in the physical sciences in the past two years show extremely social media usages.  

² There are several types of creativity - I separate perceptual curiosity and epistemic curiosity ... they're so different that it is probably a mistake giving both of them the label 'curiosity'.  It's probably worth a post some day.  For what it's worth I value and am driven by epistemic curiosity. 

One of the time management issues I constantly fight is the balance between driven focus and novel connections.  You need both.

 

 

 

 

 

when machines dream of atmospheres

Fall and Winter is a good time to look for the Andromeda galaxy in the Northern Hemisphere.  Find a dark area on a moonless night and you should be able to easily find it.  At about two and a half million light years away it's the most distant object you can see with an unaided eye.¹  Twice as big as our own Milky Way and nearly a trillion stars, it takes up six times as much sky as the full moon.  The beautiful photos you see of it are usually several images stitched together to cover it's full extent.  And that illustrates a problem astronomers have had.

 

Most telescopes look at very small regions of the sky, but a lot of interesting things happen almost randomly and the pros often come upon them later.  For about one hundred and fifty years astronomers have had a friendly relation with amateur astronomers to get wind of something new events as well as working with data.  When you study something so big, you are grateful for enthusiastic help.²

 

In the past ten years astronomy has seen an enormous increase in data to study.   A few large scale sky surveys are recording far more information than the tiny community can study.  In the near future the Large Synoptic Survey Telescope comes on line looking at over 37 billion stars and galaxies every night making a huge video of part of the sky that will  continue for decades.  Several other projects are coming on line that generate data on the order of what comes out of CERN.  Astronomy is likely to go through another revolution, but while there have been some citizen-science projects have helped with real discoveries, the fundamental problem is new huge data firehoses are growing at a fast clip.   Experienced people don't scale.

 

The Dark Energy Survey has taken a crack at the problem.  It had a few dozen scientists looking carefully for about a half year at images from an area of the sky that covers about the same amount of area as a thousand full moons.  They were looking for warped shapes that mark strong gravitational lensing.³  About a thousand of these objects are known, but a very small number are lensed by supernovae and happen to be extremely interesting.  For that reason the astronomers don't want to miss anything.  

 

Machine Learning is good at recognizing shapes and it seemed to be the reasonable approach.  Once the scan had been done it was decided to see if larger areas could use neural networks.  The DeepLensing project's neural network is still under development .  At this point machine learning picks up about  90% of the known objects.  It may miss some important objects, but it's  about a million times faster than a person, so even with ten thousand good volunteers it will cover more sky and find more objects. 

 

Neural nets are being used to find odd-ball objects ... one project is looking at data from the now defunct Kepler spacecraft and organizing objects by "weirdness"- the mark of potential discoveries.  The idea is to flag unusual objects and events for further study. So far not much has been found, but a few existing very strange stars that weren't in the training data have been spotted.  

 

My favorite so far is the Robotic Exoplanet Recognition program. RobERt studies spectrums from the bit of light that makes it through an exoplanet's atmosphere.  There's not much data and it isn't terribly complex.  The problem is the hammer and tongs approach of using theoretical atmosphere models of these atmospheres that might match the observed spectral images is a difficult computational task.  It can take a week to get a result to tell the a space based telescope to have another look.  That's way too long.  RobERt  was trained on what water, carbon dioxide and TiO in the spectrum look like and makes a rapid determination.

 

Now the fun part.  There was a good deal of experimentation playing with the size and design of the network.   Once it was working it was decided to run it in reverse.  Using the output side as the input the label 'water' was entered.  This would be like using our number recognizer after it had been trained by putting 7 as an input and seeing what the new output - the 784 neuron first layer - would look like.  Ideally it would be something like a hand-drawn 7.

 

At first the water trial failed.  It turned out there were far too many neurons that weren't doing much other than making noise.  The network was redesigned with very few neurons - just enough to do a good job on regular classification.  Now when the input and output were reversed entering  the label "water" made a plausible spectrum.  No need to really understand it - just look at the similarities.  This helps people think about interesting atmospheres and that can lead to insight.

 

The reversal of input and output is referred to as dreaming.  The machine was dreaming novel potential spectrums for water (also carbon dioxide and titanium oxide)

 

There are many flavors of machine learning and "AI".  Several other experiments are underway, but astronomy is a physical science and you're supposed to know deep things about data processing and the provenance of your data.   You only use machine learning when you have to and you need to approach it with great caution.  Even so there is a huge need and enormous potential for improvement.  I wouldn't be surprised to see fundamental improvement come from machine learning users in fundamental research rather than industrial computer science.

 __________

¹ With the naked eye you mostly see the core of the galaxy - use binoculars to reveal a much larger extent.

² Some areas of observational biology - traditional field work - makes use of amateurs.  Ornithology in particular. Cornell's eBird program aimed at tapping birders worldwide as part of a amateur science extension.  Participation was light  until they realized they could make the program attractive to birders.  Some changes were many and now eBird has over 300,000 participants many of whom are heavily invested in the community,.  By making a useful birding program they've managed to create the largest source of good data ornithologists have ever dealt with and it's in usable digital form.  Big data.

 ³ General Relativity tells us light  traveling past a mass like a star gets bent. Unseen masses can be detected by looked for warped shapes.

 __________

 

 

 

much more artificial than intelligent

Look at the little group of numbers on the right.  They were handwritten and aren't exactly neat and tidy.   Take the first four on the third line.  Our brain says 1 , 7, 4 , 2.   We might bring it together as 1742 and some of you might think of a year - perhaps the year Handel's Messiah was first performed.  And we do it with a brain composed of nearly 90 billion neurons each with up to ten thousand connections to other neurons.  All of this  uses just 20 watts of power and weighs about a kilogram and a half.

 

The brain's neurons are simple computational objects.  There are input 'wires' called dendrites that connect to the cell body and outputs called axons.  Signals arrive on the dendrites and the neuron sums them.  If they exceed some critical value the neuron "fires", sending a signal out through the axons to other neurons.  There's a lot of detail to the nature of the wiring and how it comes to be, but for now think of the neurons as objects that sum inputs and fire if a critical value is exceeded...  more abstractly they're just mathematical functions.

 

The reason for starting off with some numbers is one of the basic programs - the 'hello world' if you will - for machine learning is to build a number recognizer.  A very large set of handwritten and printed numbers was scanned by a 28 by 28 pixel sensor giving a 784 pixel image.  Each of these pixels has a value depending on the brightness of the number it corresponds to.  For the sake of argument say it ranges from 0.00 for black to 1.00 for bright white.  A very crisply written 1 would be a line of dark values in a sea of mostly white. The edges would have intermediate grey values.  Less crispy numbers have lot of grey regions.  

 

To make our neural network (call it a NN) each one of these numbers is put into a corresponding neuron. The number in the neuron is called the activation.  You can think of the neuron as doing something when it's activation is a high number.

 

These 784 neurons are  going to be the first layer of our NN.  We'll have 784 neurons on one side representing the brightness of each pixel of the image.  On the far side is a small layer of 10 neurons on the other side the numbers 0 through 9.  Each of these contains the NN's best guess for how much the scanned number is one of the numbers.  For example you might have 0.02, 0.14, 0.38, 0.02, 0.98, 0.22, 0.13, 0.22, 0.33, 0.07 .. the guess  that it is the number 1 is 0.02, 2 is 0.14 and so on.  In this case 0.98 stands out, so NN is jumping up and down saying 'it's 5".

 

In between are other layers .. often called 'hidden layers.'  Rather than presenting the math that describes the network and waving our hands saying "it's all in the training", let's go in more deeply and see what's going on.  The diagram shows a very simple NN intended to recognize numbers with 784 input neurons on the left (it only shows a few... use your imagination), a single hidden larger of 15 neurons and an output layer with 10 neurons..   

 

Several years ago I wrote my first machine learning program to recognize numbers from this data set.  I used two hidden layers with 20 neurons each.  Their number and sizes are not hard and fast choices - there's a lot of room for experimentation and frankly a lot of this is more art rather than engineering.

 

Consider of the numbers from the pixels in the first layer.  These activations will determine the activations in the first hidden layer which in turn determine the activations in the second hidden layer which determine those in the output layer.  A pattern in the first determines a pattern in the second which determines a pattern in the third and the final pattern tells us the guesses for each number.

 

I like to think about the hidden layers doing something rather than just saying they're a black box.  It may be something very foreign to us, but in this case it is possible that one might, say, recognize components like edges and the next may be larger components like lines or roundish structures (a 9 is a line and a roundish structure, a 7 has two lines...).  It probably isn't doing this, but it's reasonable to think of a NN breaking down problems into layers of abstraction.  Trying to understand what's going on in these hidden layers is very difficult in large problems - perhaps impossible and that becomes an important issue when machine learning is used for purposes that impact us.¹  

 

At this point the post will diverge a bit.  I'll go into some but not detail that some folks may find useful.  I'll highlight it in blue.  Feel free to skip it if you're only interested in a high level view.  Up to now we've established a neural network is just a function that takes input on one side and spits out an answer on the other.  In this case the input is the brightness values of a 28 by 28 pixel scan of numbers and the output is the NN's best guess for each digit, from 0 to 9, of how likely the input was that number. 

 

Consider the simple, but artificial,  example where the second layer is trying to look for a some kind of feature of the number - say certain types of line segments.  Each of the 784 neurons is connected each of the 20 neurons in the second layer.    Each value in the left most layer is a pixel value.  As an exercise to make things clear take out a sheet of paper and make a vertical column of circles labeling them a₁, a₂ , a₃, .... up to a₇₈₄.  (only do a few of them  ^... are your friends)  On the right draw another column with a couple of circles.  Connect each circle on the left with each circle on the right.  We multiply each of these activations by a weight - just a number - and find the weighted sum  of all of the pixels ..  labeling the n^th neuron as a_n and the n^th weight as w_n we get:  w₁a₁ + w₂a₂ + w₃a₃ + ... + w₇₈₄a₇₈₄.   In the example of a trying to find a little line segment imagine making all of the weights 0 except for those in the region right around the little segment.  Then taking the weighted sum just focuses on the values of the pixels we care about.  If we really wanted to force the issue we could add add a negative number called the bias to the sum to require the weighted average be large.  The bias surpasses other bits of signal in the vicinity of what we're interested in.  The weighted sum going into the neuron on the second row is now (w₁a₁ + w₂a₂ + w₃a₃ + ... + w₇₈₄a₇₈₄ - b),where b is the bias.  

 

We're getting closer.  This new activation for the pixel in the second layer can have quite a range. If the weights are between -1 and 1, it can be between -784 and +784 before adding the bias.  It is common to scrunch everything to a range between 0 and 1 using a sigmoid function, call it s(x).²  Now the scrunched weighted sum is   s(w₁a₁ + w₂a₂ + w₃a₃ + ... + w₇₈₄a₇₈₄ - b) for each of the 20 neurons in the second layer.  This new activation that goes into the neuron in the second column is the crude equivalent of a synapse in biology.

 

You'll note the naming convention I've used isn't good enough.  We're just looking at what goes into a single neuron of the second layer. In this example there are  19 more, each with a connection to each of the 784 neurons of the first row and each connection with it's own weight.  And there's another bias.  Do this and you have the first two rows.  Next you consider the connections between the second and third layers (the two hidden layers in this example).  After you finish that you have the connections between the third and final layer. And this is just a simple example.   Fortunately computers are really good at this kind of math and there's a natural way to easily organize this.  If you get into this but  you'll need to be on good speaking terms with linear algebra.³

 

Before getting into the "magic"of how these weights are found take a look at the complexity of this example.  There are (784 * 20) + (20 * 20) + (20 * 10) weights and 20 + 20 + 10  biases.  That's 16,300 knobs to adjust correctly - yikes!  Imagine what it's like for problems with columns millions of elements long.  This approach, and there are many variations,  did't take off until computers became powerful enough and we had interesting data - big data - to play with.   Although there are a lot of calculations, each is very simple.  You don't need a big Intel processor with it's complex instruction set.  GPU - graphics processing units - are arrays of thousands of tiny and fairly slow simple processors.  Each is something like ten or twenty times slower than a modern Intel CPU, but all of them can run in parallel..  you end up being faster by a factor of a thousand or so.  Complex machine learning problems have moved Nvidia beyond computer graphics.  It extends to smartphones - Apple's machine learning core on it's latest processors can make this happen in your phone rather than having to rely on the cloud. 

 

Onward to machine learning because we're not daft enough to adjust this by hand.  

 

What we want to do is come up with a recipe - an algorithm - that adjusts those 16,300 knobs so we're likely to get the right answer when we give the machine a scanned number.  For that we'll use some big data.. the enormous set of  scanned and labeled (what the scan really is:  7 for example)  28 by 28 pixel numbers that's in the public domain.⁴   The data is used to train the network.  You start out running some training data, check what the results are, make some adjustments to the weights and biases and keep repeating the process until the result is good enough.  Then you test your NN by giving it another set of labeled training data it hasn't seen so you can see if it's still doing a good job. The hope is the layered structure generalizes what the recognizer does beyond the training data.  This is critical and whoever implements a NN needs to be aware of the domain(s) where it's valid.

 

So each neuron is connected to all of the neurons in the previous layer and it's activation is the weighted sum of all of the all of those neurons.  You might think of the weights as the strength of each connection and the bias is an indication if that neuron tends to be active or inactive.   To start off you set all of the weights and biases completely randomly.  It will give terrible results on the first set of training data. 

 

To make it learn you define a cost function.  We know what the number should be:  4 for example.  Now look at the differences between each final digit and what the machine guessed.  Consider the square of the difference between what you expect and what you get.  If the original number was 4 we expect the output list to behave like this:  0 to have the value 0, 1 to be 0, 2 to be 0, 3 to be 0, 4 to be 1, 5 to be 0,...

 

The first column is the number in question, the second column is the expected result of a perfect NN, the third is the square of the difference of the computed expectation and the expected result (the second column) which are shown in the forth column.

0  0        0.3       (0.3 - 0)²    0.09        

1  0        0.2       (0.2 - 0⁾²    0.04

2  0        0.1       (0.1 - 0)²    0.01

3  0        0.8       (0.8 -0)²    0.64

4  1        0.3       (0.3 - 1)²   0.49

5  0        0.7       (0.7 - 0)²    0.49

6  0        0.3       (0.3 - 0)²    0.09

7  0        0.8       (0.8 - 0)²    0.64

8  0        0.5       (0.5 -0)²    0.25

9  0        0.0       (0.0 - 0)²   0.00

 

The cost is the sum of the squares of the differences... in this case  2.74. Ideally we would get 0.00 for the cost...  

 

The cost is small when the image is accurately recognized, large otherwise. .  Now look at the average of all of the costs over all of the training data...  that gives a measure of how good our NN is for that set of data.  A single number for each set of settings.

 

The trick is just calculus.  Our AI is much more artificial than intelligent.  I'll just state a method exists to find out how to turn the knobs and when to stop.  I'll use a small bit of  blue so be ready to skip if you don't know about gradients, 

 

Imagine the case where we just have one variable.  There would be some kind of curve for the weight relating that weight to the average cost.  If it was very simple, and it's not likely to be, you could just find the minimum by taking the derivative and be done with it.  A better tactic is to start somewhere on the curve and figure out which direction to move to lower the output by measuring the slope .. move left if the slope is positive, right if it's negative. You'll end up at a local minimum.  Now imagine the function with many inputs - so you have many dimensions.  Now you're looking for the direction and steepness of steepest descent ..  just the negative of the gradient of the function.

 

So compute the gradient of the function, take a small step downhill and repeat over and over until you find a local minimum.  So put all of the 16,300 weights and biases into a tall column vector. The negative gradient of the cost function of that vector tells which corrections to all of those numbers gives the most rapid decrease of the cost function.  Minimizing it means better average performance across all of those training samples.

 

A useful way of thinking about the negative gradient of the input vector is each element tells us if a weight or bias should be increased or decreased and how important it is.  so one that starts out as 0.02, -1.29, 0.33 ...  would say w₁ should decrease a little and it is relatively unimportant, w₂ should increase a lot and it is important, w₃ should decrease a somewhat.

 

This setting of the weights and biases is called backpropagation.  We're just minimizing the cost function.  It's important that this function is smooth so we can find local minimums...  that's why the activation numbers of the neurons have a smooth range rather than something more discrete like ones and zeros.  Oddly enough this is where the brain is more digital than the neural network.

 

Playing with this example I was able to recognize a set of number images I hadn't trained the NN on at about 94% accuracy.  There are more sophisticated approaches.  It is important to stress that the innards get very messy and difficult to understand.  That's an unavoidable bug as it is very easy to fool ourselves.  There's bias in the building of the neural network, bias in the training data and bias on it's application.  It is great for some applications, but when it's used in science (I have some experience with it in astrophysics), you use it as a last resort and try very hard to keep the models simple and understandable.  The opposite of what happens in machine learning on social networks. 

 

I think they're a fantastic way to approach certain classes of otherwise intractable problems  and potentially misleading and even dangerous in other domains.

 __________

¹ Like associating aspects of our life and the games we play with our likelihood of us being a good engineer or a neo-Nazi.  These biases, some on purpose and others by accident are extremely important and worth taking up separately.  Machine learning tends to be good at identifying things very similar to known sets.  There be dragons and caution is required.  

² For example s(x) = 1/(1 + e^-x).   There are a variety of functions of this class.  This one is known as the logistic function - you pick what's appropriate for the problem at hand.   In this case large negative numbers go to 0 and large positive numbers go to 1.  

³ Organize the input activations as a column vector and the weights as a matrix where the row corresponds to the connections between one layer and a particular neuron in the next layer.

 ⁴ the MNIST Database from LeCun, Cortes, and Burges.

__________

 

The first good brussels sprouts are out.

 

Roasted Brussels Sprouts with Mustard

 

Ingredients 

° 1 pound brussels sprouts halved

° 6 tbl olive oil

° kosher salt

° 1 tbl dijon mustard

° 1 tbl coarse grain mustard 

° 1 tbl cider vinegar

 

Technique

° oven to 425°F

° put brussels sprouts on a baking sheet.  drizzle on half of the olive oil and some salt and stir them (I use my hands).  Roast on middle rack for about 35 - 45 min turning them a few times until softened and browned.

° whisk the mustards, olive oil and vinegar into a dressing and add a bit of salt to taste.

° put the baked brussels sprouts in a bowl and toss with the dressing.

 

 

 

plants that create blue light districts for bees

Plants have a problem when it comes to reproduction.  In most cases they need genetic material from another partner, but they can’t just walk around and find one.  About one hundred and thirty million years ago Nature came a neat solution to the problem - flowers.  Flowers attract insects in a variety of ways .. color, patterns, scent and so on.  Bees are an important intermediary and are attracted by color and shape. Apparently how they use color has been a bit of a mystery.

Dogs, ferrets and most carnivores have color receptors (cones) that are sensitive to two colors.  Humans and insects like bees are sensitive to three. Bee color vision peaks in the blue - their receptors are ultraviolet,  blue and a rather weak green. Not that many plants produce blue and even fewer drop down in the ultraviolet.  There’s shape, but when you control for it bees still go to blue-free flowers.  Something’s going on!

As a teenager I worked on the grounds crew at a hospital.  I spent most of my break time getting bored and just looking at things.  I noticed a daisy-like flower … yellow petals and a brown interior - that would form a faint blue circular halo coming out at about a twenty to twenty five degree angle.  I managed to convince myself that it was an effect like a rainbow where most of the action is at about a twenty three degree angle.  I’m sure many others had the same idea.  I never asked myself ‘why blue>?’  (it’s quite beautiful and difficult to focus on…  the human eye has real problems focusing on blue, fascinating, but another topic)

Now a bit of a side track.  You’ve seen fancy sunglasses with that produce iridescent colors or select for a certain color in sunlight.   And the colors that form on a thin film of oil on water,.  It's just light reflecting off surfaces and interfering in such a way that different wavelengths of light are made more or less  intense.  Butterfly wings and some plants do it, but usually it isn’t strong enough to attract bees.

There’s another trick.  The rainbow effect you get when you hold a CD or DVD up to white light is caused the geometric pattern of tiny dots.  You get the same thing with lines and spaces.  It’s called a diffraction grating and is a cheap way to make something like a prism.  (another aside - aske me how I managed draw in rainbows that floated on a piece of quartz)  All you need are microstructures roughly the size and spacing of a wavelength of light…  about a hundredth the diameter of thin human hair (blonde for example).

It turns out a lot of plants form microstructures like this…  linear ridges around a half micron wide.  They’d produce a rainbow, but often they’re flawed with a bit of disorder … their spacing might change or parts of them are missing.  The neat thing is they tend to scatter short wavelengths of light - blue light.


Here’s the amazing thing.  In a beautiful paper published in Nature, researchers  looked at a dozen flowers that are distant relatives — like they have common ancestors about fifty million years ago.¹  The disorder in their microstructures are mathematically similar,  It strongly scatters blue light and at about a twenty five degree angle.  Many of these plants form blue halos.  An early ancestor found it useful for bees and it was too useful to go away,

Armed with this knowledge researchers built fake plastic flowers with a variety of microstructures.  Bees will try and pollinate flowers that form these blue halos  and it’s been going on for tens of millions of years,.

So the next time you’re in a garden try looking carefully at some of the flowers.  I had luck with common daisies.  There is some neat physics, material science, genetics and animal behavior going on in your garden.  

___

¹ Disorder in convergent floral nanostructures enhances signalling to bees. Edwige Moyroud et. al.  26 OCTOBER 2017 | VOL 550 | NATURE | 469

The image is from the paper.