omenti

medical tests and probabilities

a mini post

 

There are many references in the news to how good viruses tests are and what conclusions you can draw from them.  Unfortunately some are wrong and I haven't seen any easy to understand explanations of how to figure these things out.  It's all conditional probability. The standard approach is to use Bayes' Theorem.  Unfortunately most people have never heard of it or vaguely recall a formula with P's and |'s they once memorized without knowing what it really did  

 

Here's an easy way to think about it in the form of an example.  Imagine a population, say a city, where 1% of the people have a viral disease (I'll call the disease 'virus').  There's a test with a 90% chance of returning a correct result.  So if you have the disease there's a 90% chance the test returns a positive, and if you don't have the test there's a 90% chance the test returns a negative.   So what's the probability that someone who has just been tested has the disease?

 

For simplicity imagine an unbiased group of 1000 people in the city.  Let's screen them and see what happens. Ten people will have the disease, 990 will be disease-free.Of the ten people with the disease the test will correctly identify nine of them and incorrectly identify one.   Looking at the people who do not have the disease 90% of 990 or 891 will correctly test negative, but 10% of the 990 or 99 will incorrectly test positive.

 

 

 

 

So the probability a person who tests positive actually has the disease is the number who test positive and have the disease divided by the total number who test positive:  9/108 or about about an 8% chance.

The example has nothing to do with the specifics of CV-19, but maybe will give you a way to think about how to sort out what the real tests are saying when an accuracy is published.   Going deeper really requires tools like Bayes' theorem, but hopefully this is clear.

 

Armed with this technique perhaps this otherwise well-written piece in the NY Times will make sense.  Also note this is for an unbiased initial sample.  If you're screening on existing conditions before testing (which is common outside of Iceland), additional complications appear. But at least this should give you an idea.

 

 

 

 

a dive into pizza pt.1

Sarah and I were talking and the subject of pizza came up.  It turns out she's a big fan - particularly of Hawaiian pizza.  Rather than getting mired in the cultural war of pizza types, let's go for the basics - the mathematics of pizza.  You'll only need a piece of paper, a ball (Sarah recommends a volleyball), and a marker. 

 

Take a piece of paper and lay it flat on something.  It's flat with zero curvature everywhere.  if you draw a triangle on it and sum the angles, you get 180°.  A circle will have a circumference of 2Pi times the radius - 2πr - and an area of πr². All is well. Now fold it along an axis - say the long one if it's a rectangular piece.  You've introduced some curvature.   Now grab it by a corner and let it flop. Again there's curvature, but notice which one is stronger and save that thought.

 

Here's where the math comes in.  Exploring what curvature means, Carl Friedrich Gauss wondered if there was anything intrinsic about the curvature of a surface.  The sheet of paper - what is the relation between its flat shape and when you fold it?  His approach was to consider the paths you can take along the surface.    Imagine you're tiny and walking on the sheet of paper.  You can take any path you want.  When it's folded state there are a lot of paths that are concave (he called these negative curvature) and others, along the axis of the fold, with no curvature.  Gauss defined the curvature at any point on the surface as the product of the two most extreme curvatures running through that point.  In this case the red one that runs along the axis has zero curvature and the most curved green path has some negative value.  The total curvature for that point is zero times a negative number or zero.  The beautiful part is the curvature for all points on the paper are zero if it's flat or folded in such a way that doesn't stretch or tear it.  

 

Now find a volleyball or some other sphere.  The surface is convex everywhere and the curvature is always positive in any direction.  The Gaussian curvature for any point - the product of the curvature of the two extreme paths - is always a positive number times positive number or positive. (for now just consider the sign of the curvature - the actual measure is a bit beyond this post).  This is why you can't perfectly wrap a sheet of paper - or pizza - around a ball.  The paper always has zero curvature and the ball is always positive.   This also explains why can't make an accurate flat map of the Earth and have to rely on projections with distortions.

 

Many surfaces are more complicated. For fun explore points on bagel or bananas, but for now back to the pizza.  If you let it flop it still has zero curvature, but no strength.  Putting a fold down an axis gives it rigidity.  You find this throughout nature and engineering. An empty Coke can should hold your weight if you put it on the ground upright and carefully stand on it.   Lay it on its side and will crush flat under your weight. The fold in a surface of zero curvature gives a lot of rigidity in one direction.   Next go somewhere that is easy to clean and try to break an egg by squeezing it in your hand.  The positive curvature everywhere makes it surprisingly strong.  If the shell has an imperfection or if you tap it with the edge of a knife or fork while squeezing, you'll have a mess. The positive curvature was distorted and the surface failed.

 

Assuming you didn't make a mess, go back to the ball.  Draw an equator and then mark a North pole. Drawing an arc at a right angle to the equator, you'll always intersect the North pole - it's a line of longitude.  Next pick another point on the equator and make another arc at a right angle.  You've made a spherical triangle. Sum up the angles.  If you started with the points close together the sum will be just over 180° and if you go all the way around it will be nearly 540°. Our old rules of trig don't work on this non-flat surface.¹  A neat thing you can do is measure the shape of the Universe in a way that is similar to making these trigonometric measurements.  So far it looks extremely flat and that's a deep result about the structure and evolution of the Universe.

 

So get some pizza to celebrate the Remarkable Theorem its explanatin of why we hold pizza the way we do. 

__________

¹ If you draw and measure some circles on the sphere you'll notice the circumference is less than 2πr and the area is less than πr².  If you try it on a surface with negative curvature - a saddle shape or the inner part of a bagel for example - the circumference is greater than 2πr and the area is greater than πr².

 

order! - a minipost

Apparently a math puzzler became something of a theme on the Internet..  a half dozen people asked me what I thought the answer was

8÷2(2+2)=?

Some came up with 1, others with 16.  Apparently there were near-religious arguments. A mathematician would tell you both are incorrect - the question posed is ambiguous and not an answerable bit of math ..  at least not generally.

 

The problem is the order in which one evaluates expressions.  It turns out the one you learned in grade school is a bit different from the one you may have had when you learned algebra. If you remembered grade school you probably get 16.. algebra people are more likely to get 1.¹  If your algebra teacher was more careful they would have said the expression is wrong - you need to add parenthesis to make it unambiguous. The exception would be if the only people you communicate with agree on representation. That can be a big if..

 

Taking it a bit further, the simple rules you learn in secondary education don't cover don't cover unary ±, which implicitly have lower precedence than exponents on one side and higher on the other!  For example

-x³ → -(x³)
x^-3 → x(^-3)

This is so well agreed upon that no one uses parenthesis in this case.  Historically juxtaposition was treated as implicit multiplication, but that led to ugliness like

1/3x →  1/3*x →  (1/3)*x
a^2x →  a^2*x →  (a²)*x

uck! 

People decided juxtaposition of numeric literals is like a unary operator ..  so the 2 in 2x is attached to the x like the - in  -x.  

 

Back to the original problem. The fact that people had different interpretations and mentioned rules from school meant a better rule was required.  The problem needs to be unambiguous to anyone who might encounter it.  This is an issue for computer languages - different languages can parse the same written expression differently and come to different results.  Agreement on standards is an issue too ..  the Mars Climate Orbiter was lost as  teams at JPL and Lockheed Martin had different takes on a single critical bit of data - one used metric units, the other english..   when the retrorockets on the spacecraft were fired to go into Martian orbit the spacecraft burned up in the Martian atmosphere - it was literally lost in translation.  And this was code that passed multiple very careful reviews - code written for space probes is some of the carefully crafted software period.  Just wait for complex coordination systems and things that move with people nearby or inside.

 

That said there are some areas were common agreement among practitioners creates a local set of standards..  Einstein summation notation in physics is confusing to outsiders, but all physicists know it. And then there's my favorite - the double factorial.  In math (n!)! is not n!! ...  (note that I also have the problem of people interpreting ! as an exclamation point:-)

 

This local agreement is probably true in most fields - even descriptions of an action in a sport are local to the sport.  We all use local shorthand - it's a form of jargon, but we need to know when we can use it and when we need to be more general even it it's awkward.

__________

¹ In one case 8÷2(2+2) → (8÷2)*(2+2) = 4*4 =16

   in the other 8÷2(2+2) → 8÷(2*(2+2)) = 8÷(8) = 1

 

 

is the drug test fair and the prosecutor's fallacy

a minipost

Some sports have doping issues and there is drug testing in many of these.  There's always an issue of how good the tests are, but there's also an issue of how fair they are and how good they have to be to be fair.   The best way to illustrate this is with an example.

 

Consider a sport with 1000 players - roughly the size of major league baseball in the US .  Suppose steroid abuse is an issue and there's a test with 95% accuracy.  And assume somehow it's known (or strongly suspected) that about 5% of the players are abusers.  These are artificial numbers, but it's just an illustration.

 

What if the players are tested?  We expect 50 to be abusers and 950 to be clean.  The test is 95% accurate.. so 48 (rounding up from 47.5) are correctly identified.  So far so good.  Now consider the 950 clean players..  5% or 48 (rounding up again) are incorrectly labeled as abusers.  So of 1000 players there would be 96 positives and only a fifty-fifty chance of being right. Careers can be be ruined ..  the stakes are high.

 

A way to sort out these conditional probabilities was described the The Reverend Thomas Bayes in the mid 1700s in a paper called Essay Towards Solving a Problem in the Doctrine of Chances.  The good Reverend described the relationship between the probability an athlete is an abuser given a failed test and the probability an abuser fails the test.   Consider these probabilities (apologies for notation)

P(A) = probability a drug test is positive

P(B) = probability a player is an abuser

P(A|B) =  probability an abusing player tests positive

P(B|A) = probability a player has taken drugs if their test is positive

Perhaps the most important takeaway is that P(A|B) and P(B|A) are not the same!  It's a very common prosecutor trick to fool the jury into thinking they are - so common that it's called the prosecutor's fallacy.  But to the problem at hand.

 

We want to know P(B|A).  We already know P(B) or at least we have the 5% estimate.  So P(B) = 0.05.  This also gives us  P(not B)  = (1 - 0.05) = 0.95 as P(B) and P(not B) must add to 1.0. We also have 47.5 (I'll go with the fractional athlete here) of 1000 drug free athletes tested positive, so P(A|not B) = 0.0475.

 

Bayes derived a formula showing how all of these relate.  Sticking with our notation:

 P(B|A) = {P(A|B) *  P(B)} / {P(A/B) * P(B) + P(A|not B) * P(not B)}

                         = {0.95 * 0.05} / {0.95 * 0.05 + 0.0475 * 0.95}

                         = 0.513

This is a very unacceptable test!  Although 95% accuracy sounds good on the surface, it only finds a bit better than half the abusers and incorrectly labels a lot of players.  For fun try a 99% probability and a perfect 100%  to see what's happening. 

 

You can have much more complex conditionals, but this is the general idea.  You really have to think things through a bit..

 

 

it's switches all the way down

The other day I saw something that a few of you might want to consider for a kid or even yourself - the Turing Tumble Mechanical Computer.  

 

It turns out digital computers are mostly switches.  A lot of switches these days - perhaps a trillion in your smartphone. As a teenager I built a very simple relay based digital computer based on a column in an old Scientific American. Later I built an updated version that substituted transistors for the relays using some (free) components from the high school physics teacher.  

 

Playing with these beasts is an excellent introduction to understanding how logic circuits work.  You'd be surprised how many computer programmers are hazy on the subject (of course they work at a level of abstraction far above the silicon - or what old timers called the iron).  The early machines of the 30s were very important and set the stage for the dramatic developments that followed.  Here's an outstanding non-technical short history of those early days. (recommended!)

 

But back to the Turing Tumble.. It's limited, but that's central to its beauty.  The instruction book has about sixty examples to play with and an aha! just might strike leading to out-of-the-book insights.  

It's also great to play around with extremely simple silicon digital computers - particularly those that you can hook up to sensors, lights and other interfaces to the real world.   But getting down to the basics is usually a paper or YouTube exercise or, if you want to do it yourself,  a frustrating experience involving a lot of soldering and a non-trivial amount of money.   The Turning Tumble might be just the ticket.  It is play and that is the best way I know of to learn. I didn't have much of a chance to play with it and can't do a thorough review, but I'll probably end up buying one to give to a kid.  The caveat is that you have to spend some time with it to learn.  If the user doesn't find it fascinating enough to devote time to there won't be any learning.  But the same can be said about so many things.

 

The education link on their site has the following in addition to a set of resources:

 

Appropriate Age Range

Turing Tumble is for ages 8 to adult - and that is not a stretch! We find that kids 8-12 are able to get through the first 20-30 puzzles. Adults get addicted by puzzle 27, and their minds are blown by puzzle 35. Younger kids enjoy the first 10 and building their own computers.

 

 

finding the right position for the slider between education and vocation

William Rowan Hamilton had become obsessed with the idea of a three dimensional number system - sort of analogous to complex numbers being a two dimensional system.  The story he later told was every morning his son would say 'papa, can you multiply triples?' to which Hamilton would sadly shake his head and say 'no, I can only add and take away'.   Then, on a walk on the 16^th of October, 1843, he  crossed the Broom Bridge in Dublin and suddenly knew how to do it.  Rather than adding a single dimension to the complex plane, he'd add two giving three imaginary dimensions and one real dimension perpendicular to them.  In his excitement he carved it into the bridge with his knife:  i² =  j² = k² = ijk = -1

 

He called them quaternions.  They had some use in the day, but when people invented linear algebra, the technique fell by the wayside ... that is until quantum mechanics  came along about eighty years later.  And then computer graphics further down the road.

 

It turns out they're a lovely way to think about quantum mechanics and perform real calculations. Erwin Schrödinger knew enough about forgotten math to realize it was a beautiful gem.  Decades later, trying to improve the speed of computer graphics, a physicist realized they could be a key to quickly calculating rotations in space -- and Tomb Raider became the first computer game with smooth graphics on using relatively primitive personal computers.

 

Hamilton had other accomplishments that deeply impacted physics and math, but the quaternion story has a certain romantic element. To this day physicists gather at the Broom Bridge on the 16^th of October to celebrate.

 

The story of physics is that it's extremely important to think broadly and to keep expanding your learning.  It is also is relevant to education and work across many fields today.  That's what I want to get to, but first this wonderful parody of a certain song..  perhaps the best math song ever.

 

What prompted all of this was an email with a link describing the most contrarian college in America.  It's laser focused on a vision of an education seemingly without much a nod to vocation ...  seemingly..   I'd argue that it's too focused on a narrow piece of education, but have no doubt its graduates have learned how to learn..  they have the beginnings of a real education.  Mark Roosevelt, its President, sums it up:

Education should prepare you for all of your life. It should make you a more thoughtful, reflective, self-possessed and authentic citizen, lover, partner, parent and member of the global economy.

It  takes a special kind of student to thrive in a school like that.  I wouldn't have survived - my interests, those that I was motivated to dive into, lay elsewhere.  I'm happy it exists and that there is spectrum of schools for those who look.

 

My belief is a deep education with at least a few focuses creates the second type of person Claude Shannon describes:

“There are some people if you shoot one idea into the brain, you will get a half an idea out. There are other people who are beyond this point at which they produce two ideas for each idea sent in.”

The ability to get multiple ideas is core to serious play and creativity.   I'd claim focusing on education rather than finding the minimally easy path helps create the neural connections that make this possible.

 

A worry is that many, if not most, schools and students tend more towards vocation.  Focus on what is important to get a degree that will get you employed to pay off that mountain of debt.  Many (not all) businesses, on the other hand, would do well to have broader employees who are still learning and growing - those who can think creatively.  There is a disconnect. 

 

I hope this focus on vocation changes.  The rate of real change is great and people will keep up better if they are broader and more adaptable.  When recommending colleges I tend to focus on education ..  often to the consternation of those asking.  Of course I may be a poor example.  My education was far too narrow, but I've been lucky and curious enough to have been able to expand it a bit.  The other note is you can get a wickedly broad experience at many more places than you might imagine.

 

 Finally there's this odd emphasis on over-scheduling many pre-college kids, but I've probably written on how it kills creativity enough.

 

a kid redesigns the olympic symbol

I was in the basement with the brand new Motorola 21 inch color TV and a children's telephone.  The wires snaked out and up to the roof where my Dad was on the other toy phone braving the Montana Winter as we tried to get a usable signal.  It was my first year of high school and the Winter Olympics would be on in a day or two.   The Olympics and we had a COLOR TV!  

 

We sort of got it working by the time the broadcast began.  The Olympic symbol was displayed and the story given

 

... the six colors [including the flag's white background] combined in this way reproduce the colours of every country without exception. The blue and yellow of Sweden, the blue and white of Greece, the tricolor flags of France, England, the United States, Germany, Belgium, Italy and Hungary, and the yellow and red of Spain are included, as are the innovative flags of Brazil and Australia, and those of ancient Japan and modern China. This, truly, is an international emblem.

 

It's a very clean design, but it offended my sense of mathematics.

 

My sister had very long hair at the time.  Long enough to sit on.  She was always working on various braids.  The year before I was curious about knots and braids and learned that a simple three strand braid becomes very interesting if you connect the ends.  Take three strands here colored red, blue and green.  Wrap them in a standard braid and go through one cycle so the colors end up where they've started.  (I like to think of braids as musical rounds)

 

 

Rather than repeat the process like my sister, bend them back and connect red to red, blue to blue and green to green.  You'll be left with three loops that are fully linked, but if you remove one the whole works falls apart.  Here's a nice image.  Note that they're not really two dimensional circles as they have to pass over and under each other.  Making this forces you to use three dimensions.  

 

Technically these are a special case of brunnian links - a nontrivial link (a link is a grouping of the loops here) that becomes a set of trivial (unconnected) circles if you pull out one of the loops.  Cut any loop and the whole thing can easily be pulled apart.  Do it with five loops and you'd have powerful symbolism for the Olympic movement - or so I thought.   I knew you can make brunnian links for any odd number of loops so it was time to draw.

 

I sketched a five loop version.  Cut and remove any loop and it all comes apart.  All I had to do was find some way of making a nice drawing.  My sister is the arty one.  She had all of the right kit including some high quality colored pencils.  She also happened to be smart enough to not let me use them - even if I had noble intensions.  

 

Plan B.  Every kid has Crayola crayons.  I selected five and, after a couple of tries, I had it.  

 

(pretend the colors are accurate.  my drawing here is pretty bad, but so was the crayon version)

 

The drawing went into a special international airmail envelope with an explanation of the math and symbolism.  It was addressed to the Olympic organization in Lausanne..  I have yet to hear from them.  Apparently crayon drawings from 15 year olds don't make the cut.

 

I draw quite a bit.  It's an integral part of how I think.  It isn't pretty and I consider it an ephemeral part of thinking and trash it as soon as it's served it's purpose.  If you want meaningful design it's best to use a real designer.

 

I'm looking forward to the games!

 

much more artificial than intelligent

Look at the little group of numbers on the right.  They were handwritten and aren't exactly neat and tidy.   Take the first four on the third line.  Our brain says 1 , 7, 4 , 2.   We might bring it together as 1742 and some of you might think of a year - perhaps the year Handel's Messiah was first performed.  And we do it with a brain composed of nearly 90 billion neurons each with up to ten thousand connections to other neurons.  All of this  uses just 20 watts of power and weighs about a kilogram and a half.

 

The brain's neurons are simple computational objects.  There are input 'wires' called dendrites that connect to the cell body and outputs called axons.  Signals arrive on the dendrites and the neuron sums them.  If they exceed some critical value the neuron "fires", sending a signal out through the axons to other neurons.  There's a lot of detail to the nature of the wiring and how it comes to be, but for now think of the neurons as objects that sum inputs and fire if a critical value is exceeded...  more abstractly they're just mathematical functions.

 

The reason for starting off with some numbers is one of the basic programs - the 'hello world' if you will - for machine learning is to build a number recognizer.  A very large set of handwritten and printed numbers was scanned by a 28 by 28 pixel sensor giving a 784 pixel image.  Each of these pixels has a value depending on the brightness of the number it corresponds to.  For the sake of argument say it ranges from 0.00 for black to 1.00 for bright white.  A very crisply written 1 would be a line of dark values in a sea of mostly white. The edges would have intermediate grey values.  Less crispy numbers have lot of grey regions.  

 

To make our neural network (call it a NN) each one of these numbers is put into a corresponding neuron. The number in the neuron is called the activation.  You can think of the neuron as doing something when it's activation is a high number.

 

These 784 neurons are  going to be the first layer of our NN.  We'll have 784 neurons on one side representing the brightness of each pixel of the image.  On the far side is a small layer of 10 neurons on the other side the numbers 0 through 9.  Each of these contains the NN's best guess for how much the scanned number is one of the numbers.  For example you might have 0.02, 0.14, 0.38, 0.02, 0.98, 0.22, 0.13, 0.22, 0.33, 0.07 .. the guess  that it is the number 1 is 0.02, 2 is 0.14 and so on.  In this case 0.98 stands out, so NN is jumping up and down saying 'it's 5".

 

In between are other layers .. often called 'hidden layers.'  Rather than presenting the math that describes the network and waving our hands saying "it's all in the training", let's go in more deeply and see what's going on.  The diagram shows a very simple NN intended to recognize numbers with 784 input neurons on the left (it only shows a few... use your imagination), a single hidden larger of 15 neurons and an output layer with 10 neurons..   

 

Several years ago I wrote my first machine learning program to recognize numbers from this data set.  I used two hidden layers with 20 neurons each.  Their number and sizes are not hard and fast choices - there's a lot of room for experimentation and frankly a lot of this is more art rather than engineering.

 

Consider of the numbers from the pixels in the first layer.  These activations will determine the activations in the first hidden layer which in turn determine the activations in the second hidden layer which determine those in the output layer.  A pattern in the first determines a pattern in the second which determines a pattern in the third and the final pattern tells us the guesses for each number.

 

I like to think about the hidden layers doing something rather than just saying they're a black box.  It may be something very foreign to us, but in this case it is possible that one might, say, recognize components like edges and the next may be larger components like lines or roundish structures (a 9 is a line and a roundish structure, a 7 has two lines...).  It probably isn't doing this, but it's reasonable to think of a NN breaking down problems into layers of abstraction.  Trying to understand what's going on in these hidden layers is very difficult in large problems - perhaps impossible and that becomes an important issue when machine learning is used for purposes that impact us.¹  

 

At this point the post will diverge a bit.  I'll go into some but not detail that some folks may find useful.  I'll highlight it in blue.  Feel free to skip it if you're only interested in a high level view.  Up to now we've established a neural network is just a function that takes input on one side and spits out an answer on the other.  In this case the input is the brightness values of a 28 by 28 pixel scan of numbers and the output is the NN's best guess for each digit, from 0 to 9, of how likely the input was that number. 

 

Consider the simple, but artificial,  example where the second layer is trying to look for a some kind of feature of the number - say certain types of line segments.  Each of the 784 neurons is connected each of the 20 neurons in the second layer.    Each value in the left most layer is a pixel value.  As an exercise to make things clear take out a sheet of paper and make a vertical column of circles labeling them a₁, a₂ , a₃, .... up to a₇₈₄.  (only do a few of them  ^... are your friends)  On the right draw another column with a couple of circles.  Connect each circle on the left with each circle on the right.  We multiply each of these activations by a weight - just a number - and find the weighted sum  of all of the pixels ..  labeling the n^th neuron as a_n and the n^th weight as w_n we get:  w₁a₁ + w₂a₂ + w₃a₃ + ... + w₇₈₄a₇₈₄.   In the example of a trying to find a little line segment imagine making all of the weights 0 except for those in the region right around the little segment.  Then taking the weighted sum just focuses on the values of the pixels we care about.  If we really wanted to force the issue we could add add a negative number called the bias to the sum to require the weighted average be large.  The bias surpasses other bits of signal in the vicinity of what we're interested in.  The weighted sum going into the neuron on the second row is now (w₁a₁ + w₂a₂ + w₃a₃ + ... + w₇₈₄a₇₈₄ - b),where b is the bias.  

 

We're getting closer.  This new activation for the pixel in the second layer can have quite a range. If the weights are between -1 and 1, it can be between -784 and +784 before adding the bias.  It is common to scrunch everything to a range between 0 and 1 using a sigmoid function, call it s(x).²  Now the scrunched weighted sum is   s(w₁a₁ + w₂a₂ + w₃a₃ + ... + w₇₈₄a₇₈₄ - b) for each of the 20 neurons in the second layer.  This new activation that goes into the neuron in the second column is the crude equivalent of a synapse in biology.

 

You'll note the naming convention I've used isn't good enough.  We're just looking at what goes into a single neuron of the second layer. In this example there are  19 more, each with a connection to each of the 784 neurons of the first row and each connection with it's own weight.  And there's another bias.  Do this and you have the first two rows.  Next you consider the connections between the second and third layers (the two hidden layers in this example).  After you finish that you have the connections between the third and final layer. And this is just a simple example.   Fortunately computers are really good at this kind of math and there's a natural way to easily organize this.  If you get into this but  you'll need to be on good speaking terms with linear algebra.³

 

Before getting into the "magic"of how these weights are found take a look at the complexity of this example.  There are (784 * 20) + (20 * 20) + (20 * 10) weights and 20 + 20 + 10  biases.  That's 16,300 knobs to adjust correctly - yikes!  Imagine what it's like for problems with columns millions of elements long.  This approach, and there are many variations,  did't take off until computers became powerful enough and we had interesting data - big data - to play with.   Although there are a lot of calculations, each is very simple.  You don't need a big Intel processor with it's complex instruction set.  GPU - graphics processing units - are arrays of thousands of tiny and fairly slow simple processors.  Each is something like ten or twenty times slower than a modern Intel CPU, but all of them can run in parallel..  you end up being faster by a factor of a thousand or so.  Complex machine learning problems have moved Nvidia beyond computer graphics.  It extends to smartphones - Apple's machine learning core on it's latest processors can make this happen in your phone rather than having to rely on the cloud. 

 

Onward to machine learning because we're not daft enough to adjust this by hand.  

 

What we want to do is come up with a recipe - an algorithm - that adjusts those 16,300 knobs so we're likely to get the right answer when we give the machine a scanned number.  For that we'll use some big data.. the enormous set of  scanned and labeled (what the scan really is:  7 for example)  28 by 28 pixel numbers that's in the public domain.⁴   The data is used to train the network.  You start out running some training data, check what the results are, make some adjustments to the weights and biases and keep repeating the process until the result is good enough.  Then you test your NN by giving it another set of labeled training data it hasn't seen so you can see if it's still doing a good job. The hope is the layered structure generalizes what the recognizer does beyond the training data.  This is critical and whoever implements a NN needs to be aware of the domain(s) where it's valid.

 

So each neuron is connected to all of the neurons in the previous layer and it's activation is the weighted sum of all of the all of those neurons.  You might think of the weights as the strength of each connection and the bias is an indication if that neuron tends to be active or inactive.   To start off you set all of the weights and biases completely randomly.  It will give terrible results on the first set of training data. 

 

To make it learn you define a cost function.  We know what the number should be:  4 for example.  Now look at the differences between each final digit and what the machine guessed.  Consider the square of the difference between what you expect and what you get.  If the original number was 4 we expect the output list to behave like this:  0 to have the value 0, 1 to be 0, 2 to be 0, 3 to be 0, 4 to be 1, 5 to be 0,...

 

The first column is the number in question, the second column is the expected result of a perfect NN, the third is the square of the difference of the computed expectation and the expected result (the second column) which are shown in the forth column.

0  0        0.3       (0.3 - 0)²    0.09        

1  0        0.2       (0.2 - 0⁾²    0.04

2  0        0.1       (0.1 - 0)²    0.01

3  0        0.8       (0.8 -0)²    0.64

4  1        0.3       (0.3 - 1)²   0.49

5  0        0.7       (0.7 - 0)²    0.49

6  0        0.3       (0.3 - 0)²    0.09

7  0        0.8       (0.8 - 0)²    0.64

8  0        0.5       (0.5 -0)²    0.25

9  0        0.0       (0.0 - 0)²   0.00

 

The cost is the sum of the squares of the differences... in this case  2.74. Ideally we would get 0.00 for the cost...  

 

The cost is small when the image is accurately recognized, large otherwise. .  Now look at the average of all of the costs over all of the training data...  that gives a measure of how good our NN is for that set of data.  A single number for each set of settings.

 

The trick is just calculus.  Our AI is much more artificial than intelligent.  I'll just state a method exists to find out how to turn the knobs and when to stop.  I'll use a small bit of  blue so be ready to skip if you don't know about gradients, 

 

Imagine the case where we just have one variable.  There would be some kind of curve for the weight relating that weight to the average cost.  If it was very simple, and it's not likely to be, you could just find the minimum by taking the derivative and be done with it.  A better tactic is to start somewhere on the curve and figure out which direction to move to lower the output by measuring the slope .. move left if the slope is positive, right if it's negative. You'll end up at a local minimum.  Now imagine the function with many inputs - so you have many dimensions.  Now you're looking for the direction and steepness of steepest descent ..  just the negative of the gradient of the function.

 

So compute the gradient of the function, take a small step downhill and repeat over and over until you find a local minimum.  So put all of the 16,300 weights and biases into a tall column vector. The negative gradient of the cost function of that vector tells which corrections to all of those numbers gives the most rapid decrease of the cost function.  Minimizing it means better average performance across all of those training samples.

 

A useful way of thinking about the negative gradient of the input vector is each element tells us if a weight or bias should be increased or decreased and how important it is.  so one that starts out as 0.02, -1.29, 0.33 ...  would say w₁ should decrease a little and it is relatively unimportant, w₂ should increase a lot and it is important, w₃ should decrease a somewhat.

 

This setting of the weights and biases is called backpropagation.  We're just minimizing the cost function.  It's important that this function is smooth so we can find local minimums...  that's why the activation numbers of the neurons have a smooth range rather than something more discrete like ones and zeros.  Oddly enough this is where the brain is more digital than the neural network.

 

Playing with this example I was able to recognize a set of number images I hadn't trained the NN on at about 94% accuracy.  There are more sophisticated approaches.  It is important to stress that the innards get very messy and difficult to understand.  That's an unavoidable bug as it is very easy to fool ourselves.  There's bias in the building of the neural network, bias in the training data and bias on it's application.  It is great for some applications, but when it's used in science (I have some experience with it in astrophysics), you use it as a last resort and try very hard to keep the models simple and understandable.  The opposite of what happens in machine learning on social networks. 

 

I think they're a fantastic way to approach certain classes of otherwise intractable problems  and potentially misleading and even dangerous in other domains.

 __________

¹ Like associating aspects of our life and the games we play with our likelihood of us being a good engineer or a neo-Nazi.  These biases, some on purpose and others by accident are extremely important and worth taking up separately.  Machine learning tends to be good at identifying things very similar to known sets.  There be dragons and caution is required.  

² For example s(x) = 1/(1 + e^-x).   There are a variety of functions of this class.  This one is known as the logistic function - you pick what's appropriate for the problem at hand.   In this case large negative numbers go to 0 and large positive numbers go to 1.  

³ Organize the input activations as a column vector and the weights as a matrix where the row corresponds to the connections between one layer and a particular neuron in the next layer.

 ⁴ the MNIST Database from LeCun, Cortes, and Burges.

__________

 

The first good brussels sprouts are out.

 

Roasted Brussels Sprouts with Mustard

 

Ingredients 

° 1 pound brussels sprouts halved

° 6 tbl olive oil

° kosher salt

° 1 tbl dijon mustard

° 1 tbl coarse grain mustard 

° 1 tbl cider vinegar

 

Technique

° oven to 425°F

° put brussels sprouts on a baking sheet.  drizzle on half of the olive oil and some salt and stir them (I use my hands).  Roast on middle rack for about 35 - 45 min turning them a few times until softened and browned.

° whisk the mustards, olive oil and vinegar into a dressing and add a bit of salt to taste.

° put the baked brussels sprouts in a bowl and toss with the dressing.

 

 

 

measuring coastlines to helping someone see with sound

 Late in May of the year I was in forth grade we were given a map of the United States and Canada with outlines of the states and provinces.  Our task was was to correctly label it with capital names and rough locations along with the names of states and provinces.  We had a few for practice and the shapes caused my mind to wander.

 

States like Kansas, Colorado and Wyoming, were boring rectangles.  Or were they?  The supposedly parallel lines on the East and West would converge on the North Pole.   My Dad had had this neat little mechanical map measuring tool.  You'd zero it and then mechanically trace out a path and write down how many millimeters it had gone (his was German and labeled in millimeters).  Then you'd look up the scale on your map and convert to miles or kilometers.  I quickly learned the school maps weren't very good and started playing with road maps from the gas station.  In the day you'd pull up and have someone pump your gas, clean your windows, check your oil and water and tire pressure and then pay about thirty cents a gallon for the gasoline.  They had a pile of maps for the asking. I had a nice collection.

 

These were better, but didn't answer my basic question.  I was starting to wonder what represented ground truth.  If you're admiring the  Slartibartfast's crinkly edges of Norway you have to wonder about scale.  From space there may not be as much detail as from an airplane and you will get somewhat different results.  Down to ground level, where you can easily measure in centimeters rather than meters, it's much more complicated and is now dynamic with forces like the tides, continental drift, erosion and so on.  Where you do draw the line, so to speak?

 

At about the same time I was getting boggled, Benoit Mandelbrot was investigating fractal geometries.  He was something of a polymath and the only person I've met whose personal style included a piece of fresh broccoli in his suit pocket.   He loved self similar shapes..  A lot of math is just drawing things to see what's going on - a neat little example is the Koch curve.   Draw an equilateral triangle and the add three a smaller triangle with a sides a third the length of the original.   Now place each midway on the original triangle with so they're pointing out and just keep the shape along the outside boundary (you may want to think of cutting them out of paper and just using the outer edge.)  Now it looks like a Star a David, but you've just started.  Add new even smaller triangle along each of the edges of the Star of David and you start to get a snow flake.  Keep going. (a computer may be useful:-)

 

The edges are much more complex that anything smooth like a line or a circle.  Mandelbrot and others came up with a way to assign a dimension to any shape.  A line has one dimension, a plane has two, the space we're used to has three and math is just getting warmed up.  The Koch curve has a dimension of 1.2619.  The definition give 1.0 for lines and 2.0 for planes, but is also a way to talk about complexity and repeating complexity.  

 

Just in case someone asks,  the dimensionality of the Southern Norwegian coastline is about 1.3.  The Southwestern coast of Britain is very close to 1.25. 

 

Drawing these curves by hand or computer could and should be taught in middle and high school because it's playful.  Understanding how to calculate dimensionality of these shapes would take too long for me to describe, but I think it could be done as part of a high school calculus class.  Playing with these concepts builds your tool kit.  A tool kit that allows you to stitch seemingly unrelated concepts together.  

 

Our brains are remarkably "plastic."   If you push them with new mental activities they rewire so as to become more efficient.  Profoundly deaf people use some of the brain normally associated with sound to increase their visual acuity.  Musicians enhance regions of their brains and the mental component of many sports leads to very measurable changes. If you work at several different things your wiring may become uniquely creative -  I suspect some of the most creative athletes spent a lot of time studying an instrument at some point. These enhancements are readily seen in brain scans even in people in their sixties. It's easy to measure the changes even in people who are in their sixties.  You can teach old dogs new tricks.  (this is such a rich area, but back to the equally rich math)

 

What if you wanted to help a blind person by encoding the information from a camera in sound?  How would you approach the prolbem?

 

To get an idea of what's going on I take a very simple picture..  in this case 256 pixels on a side.  I start with my avatar. (pretend it's 256 x 256...  this one is twice as large)  I make it black and white..  256 grey levels in this case.  I don't know what to do with more detailed information...  I'm just dinking around a bit with the problem.  

 

How to present this to the brain? Don't worry if this would actually work - just consider it a little thought experiment.  We could associated each pixel had a frequency?  The loudness could be associated with the brightness.  In this case white could be very quite and black the loudest.  You might play them all at once, but it might be too confusing..   Try adding a bit of order.  The question is what kind of order?  (again. . there are smarter ways to do this - you may be thinking of some now, but let's work on this one)

 

It could be some randomly chosen ordering.  After all, the brain is good at finding order in a mess.. as long as we use the same random ordering.  But that presents a problem..  what if we use a better camera..  like 3000 x 4000 pixels?  Then we need a new random ordering and the brain has to start all over again.,

 

There is raster scanning .. like an old CRT ... you move back and forth t across the image starting at the bottom and shifting up a pixel each time you reach an edge reading the brightness of each pixel you encounter until you've scanned the whole image.   Simple but it throws away some good intuition.  An image tends to have a lot of neighboring similarity.  It would be nice to make use of that. Hang onto that thought for awhile.  

 

A mathematician might suggest you consider a Hilbert curve -  more specifically a Pseudo-Hilbert curve. Rather than get impressed by the fancy name we'll drop the pseudo and just play around.   Take out a sheet of paper and draw a square.  (line 1 in the figure) Now divide the square into equal quadrants.  Starting in the lower left quadrant, draw a line up to the upper left quadrant, continue to the upper right quadrant and then down to the lower right quadrant.   This upside down U is an Order One Hilbert curve.  Now it's time  to make it work.  (line 2)   Draw another square with four quadrants.  Now draw an Order One curve in each of the quadrants.  Start traveling from the lower left and you find yourself making some unfortunate traverses.  Is there any way to rearrange these Order One curves to get something that doesn't intersect itself?

 

 

 

 

It turns out there is.  (line 3).  Rotate the curve in the lower left quadrant 90° counter clockwise and it's neighbor in the lower right quadrant 90* counter clockwise.  Now you can connect all four Order One curves into something that doesn't cross itself - an Order Two Hilbert curve. You can probably guess the algorithm from here.  To draw the next order draw your current order Hilbert curve in each of four quadrants of a square.  Rotate the bottom left curve 90° clockwise and the bottom right 90° counter clockwise.  Connect them and you're done.

 

Now that you've got the recipe you can draw Hilbert Curves of any order your patience allows.. Past four and it's best to use a computers.  Here are Orders One through Five

 

 

There are two things to notice:

 

First the curve is covering more points of the square as the order increases.  We can talk about it's dimension.  As the order goes to infinity, the curve's dimension is 2.0 ... in other words the curve hits every point inside the square.   How sweet - an infinite one dimensional object that covers two dimensions.  Our job is much easier - we're using pixels and only require the line to go through a pixel.

 

Second, the curve preserves localized information as the order of the curve goes up.  Look at the nose of my ferret. If I stretch out the curve the points associated with the nose are roughly in the same region of the line.  This is useful if our blind person is listening to the string of notes coming out.  Most of them around the nose have a similar value and are played at nearly the same time.  If we use a camera with more pixels that local nature is still preserved.  The raster scan that snakes up the image don't preserve locality.

 

There are many uses for these locality preserving mappings (as they're called) and space filling curves like the Hilbert Curve are just what the doctor ordered.  And you don't have to stop with one dimension. They may get hard to visualize, but the technique works with as many dimensions as your heart desires.

 

Math is useful for calculating, but more important is the fact it's a playground that offers insight and new ways of thinking.  It is one of the glues of creativity.

 

 

two goats, one car

I just heard that Monty Hall died at the age of 96.  He was a game show host famous for a simple probability problem that gets people riled up.  His show, Let's Make a Deal, had three doors. Behind one door was something very desirable - like a new sports car.  The other two doors had, well, something like a goat.  You were asked to pick door number one, door number two or door number three.  At that point your chances of worrying about the color of your new car were one in three.  After you made your choice Monty would open one of the other doors revealing a goat and would ask if you wanted to switch.

 

People tend to think it's a coin flip at that point, but that's not thinking it through.  Monte has perfect knowledge of the situation and has added some information to the puzzle with is reveal.  The right strategy is to switch - always.  Your chances will have improved to two out of three.

 

One way to think about it is if you picked, say, door number one, the chances of it wining are one in three and the chances of each of the other two are one third each.  This means the chance of it being in either two or three is two thirds.  Monte, by opening door three (for example), has increased the likelihood of it being behind door number two to two thirds.   He added information. 

 

always use extra ground truth when it is available!

 

You won't always win, but if you play a large number of times, you'll win about two thirds of the time.  You can write a little program to convince yourself of imagine a game where Monte has ten thousand doors.  One car, 9,999 goats.  You choose door number one.  Monte opens all of the doors except, say, 1,729 and asks if you want to switch.  My guess is your intuition will do the right thing:-)