omenti

order! - a minipost

Apparently a math puzzler became something of a theme on the Internet..  a half dozen people asked me what I thought the answer was

8÷2(2+2)=?

Some came up with 1, others with 16.  Apparently there were near-religious arguments. A mathematician would tell you both are incorrect - the question posed is ambiguous and not an answerable bit of math ..  at least not generally.

 

The problem is the order in which one evaluates expressions.  It turns out the one you learned in grade school is a bit different from the one you may have had when you learned algebra. If you remembered grade school you probably get 16.. algebra people are more likely to get 1.¹  If your algebra teacher was more careful they would have said the expression is wrong - you need to add parenthesis to make it unambiguous. The exception would be if the only people you communicate with agree on representation. That can be a big if..

 

Taking it a bit further, the simple rules you learn in secondary education don't cover don't cover unary ±, which implicitly have lower precedence than exponents on one side and higher on the other!  For example

-x³ → -(x³)
x^-3 → x(^-3)

This is so well agreed upon that no one uses parenthesis in this case.  Historically juxtaposition was treated as implicit multiplication, but that led to ugliness like

1/3x →  1/3*x →  (1/3)*x
a^2x →  a^2*x →  (a²)*x

uck! 

People decided juxtaposition of numeric literals is like a unary operator ..  so the 2 in 2x is attached to the x like the - in  -x.  

 

Back to the original problem. The fact that people had different interpretations and mentioned rules from school meant a better rule was required.  The problem needs to be unambiguous to anyone who might encounter it.  This is an issue for computer languages - different languages can parse the same written expression differently and come to different results.  Agreement on standards is an issue too ..  the Mars Climate Orbiter was lost as  teams at JPL and Lockheed Martin had different takes on a single critical bit of data - one used metric units, the other english..   when the retrorockets on the spacecraft were fired to go into Martian orbit the spacecraft burned up in the Martian atmosphere - it was literally lost in translation.  And this was code that passed multiple very careful reviews - code written for space probes is some of the carefully crafted software period.  Just wait for complex coordination systems and things that move with people nearby or inside.

 

That said there are some areas were common agreement among practitioners creates a local set of standards..  Einstein summation notation in physics is confusing to outsiders, but all physicists know it. And then there's my favorite - the double factorial.  In math (n!)! is not n!! ...  (note that I also have the problem of people interpreting ! as an exclamation point:-)

 

This local agreement is probably true in most fields - even descriptions of an action in a sport are local to the sport.  We all use local shorthand - it's a form of jargon, but we need to know when we can use it and when we need to be more general even it it's awkward.

__________

¹ In one case 8÷2(2+2) → (8÷2)*(2+2) = 4*4 =16

   in the other 8÷2(2+2) → 8÷(2*(2+2)) = 8÷(8) = 1

 

 

is the drug test fair and the prosecutor's fallacy

a minipost

Some sports have doping issues and there is drug testing in many of these.  There's always an issue of how good the tests are, but there's also an issue of how fair they are and how good they have to be to be fair.   The best way to illustrate this is with an example.

 

Consider a sport with 1000 players - roughly the size of major league baseball in the US .  Suppose steroid abuse is an issue and there's a test with 95% accuracy.  And assume somehow it's known (or strongly suspected) that about 5% of the players are abusers.  These are artificial numbers, but it's just an illustration.

 

What if the players are tested?  We expect 50 to be abusers and 950 to be clean.  The test is 95% accurate.. so 48 (rounding up from 47.5) are correctly identified.  So far so good.  Now consider the 950 clean players..  5% or 48 (rounding up again) are incorrectly labeled as abusers.  So of 1000 players there would be 96 positives and only a fifty-fifty chance of being right. Careers can be be ruined ..  the stakes are high.

 

A way to sort out these conditional probabilities was described the The Reverend Thomas Bayes in the mid 1700s in a paper called Essay Towards Solving a Problem in the Doctrine of Chances.  The good Reverend described the relationship between the probability an athlete is an abuser given a failed test and the probability an abuser fails the test.   Consider these probabilities (apologies for notation)

P(A) = probability a drug test is positive

P(B) = probability a player is an abuser

P(A|B) =  probability an abusing player tests positive

P(B|A) = probability a player has taken drugs if their test is positive

Perhaps the most important takeaway is that P(A|B) and P(B|A) are not the same!  It's a very common prosecutor trick to fool the jury into thinking they are - so common that it's called the prosecutor's fallacy.  But to the problem at hand.

 

We want to know P(B|A).  We already know P(B) or at least we have the 5% estimate.  So P(B) = 0.05.  This also gives us  P(not B)  = (1 - 0.05) = 0.95 as P(B) and P(not B) must add to 1.0. We also have 47.5 (I'll go with the fractional athlete here) of 1000 drug free athletes tested positive, so P(A|not B) = 0.0475.

 

Bayes derived a formula showing how all of these relate.  Sticking with our notation:

 P(B|A) = {P(A|B) *  P(B)} / {P(A/B) * P(B) + P(A|not B) * P(not B)}

                         = {0.95 * 0.05} / {0.95 * 0.05 + 0.0475 * 0.95}

                         = 0.513

This is a very unacceptable test!  Although 95% accuracy sounds good on the surface, it only finds a bit better than half the abusers and incorrectly labels a lot of players.  For fun try a 99% probability and a perfect 100%  to see what's happening. 

 

You can have much more complex conditionals, but this is the general idea.  You really have to think things through a bit..

 

 

it's switches all the way down

The other day I saw something that a few of you might want to consider for a kid or even yourself - the Turing Tumble Mechanical Computer.  

 

It turns out digital computers are mostly switches.  A lot of switches these days - perhaps a trillion in your smartphone. As a teenager I built a very simple relay based digital computer based on a column in an old Scientific American. Later I built an updated version that substituted transistors for the relays using some (free) components from the high school physics teacher.  

 

Playing with these beasts is an excellent introduction to understanding how logic circuits work.  You'd be surprised how many computer programmers are hazy on the subject (of course they work at a level of abstraction far above the silicon - or what old timers called the iron).  The early machines of the 30s were very important and set the stage for the dramatic developments that followed.  Here's an outstanding non-technical short history of those early days. (recommended!)

 

But back to the Turing Tumble.. It's limited, but that's central to its beauty.  The instruction book has about sixty examples to play with and an aha! just might strike leading to out-of-the-book insights.  

It's also great to play around with extremely simple silicon digital computers - particularly those that you can hook up to sensors, lights and other interfaces to the real world.   But getting down to the basics is usually a paper or YouTube exercise or, if you want to do it yourself,  a frustrating experience involving a lot of soldering and a non-trivial amount of money.   The Turning Tumble might be just the ticket.  It is play and that is the best way I know of to learn. I didn't have much of a chance to play with it and can't do a thorough review, but I'll probably end up buying one to give to a kid.  The caveat is that you have to spend some time with it to learn.  If the user doesn't find it fascinating enough to devote time to there won't be any learning.  But the same can be said about so many things.

 

The education link on their site has the following in addition to a set of resources:

 

Appropriate Age Range

Turing Tumble is for ages 8 to adult - and that is not a stretch! We find that kids 8-12 are able to get through the first 20-30 puzzles. Adults get addicted by puzzle 27, and their minds are blown by puzzle 35. Younger kids enjoy the first 10 and building their own computers.

 

 

finding the right position for the slider between education and vocation

William Rowan Hamilton had become obsessed with the idea of a three dimensional number system - sort of analogous to complex numbers being a two dimensional system.  The story he later told was every morning his son would say 'papa, can you multiply triples?' to which Hamilton would sadly shake his head and say 'no, I can only add and take away'.   Then, on a walk on the 16^th of October, 1843, he  crossed the Broom Bridge in Dublin and suddenly knew how to do it.  Rather than adding a single dimension to the complex plane, he'd add two giving three imaginary dimensions and one real dimension perpendicular to them.  In his excitement he carved it into the bridge with his knife:  i² =  j² = k² = ijk = -1

 

He called them quaternions.  They had some use in the day, but when people invented linear algebra, the technique fell by the wayside ... that is until quantum mechanics  came along about eighty years later.  And then computer graphics further down the road.

 

It turns out they're a lovely way to think about quantum mechanics and perform real calculations. Erwin Schrödinger knew enough about forgotten math to realize it was a beautiful gem.  Decades later, trying to improve the speed of computer graphics, a physicist realized they could be a key to quickly calculating rotations in space -- and Tomb Raider became the first computer game with smooth graphics on using relatively primitive personal computers.

 

Hamilton had other accomplishments that deeply impacted physics and math, but the quaternion story has a certain romantic element. To this day physicists gather at the Broom Bridge on the 16^th of October to celebrate.

 

The story of physics is that it's extremely important to think broadly and to keep expanding your learning.  It is also is relevant to education and work across many fields today.  That's what I want to get to, but first this wonderful parody of a certain song..  perhaps the best math song ever.

 

What prompted all of this was an email with a link describing the most contrarian college in America.  It's laser focused on a vision of an education seemingly without much a nod to vocation ...  seemingly..   I'd argue that it's too focused on a narrow piece of education, but have no doubt its graduates have learned how to learn..  they have the beginnings of a real education.  Mark Roosevelt, its President, sums it up:

Education should prepare you for all of your life. It should make you a more thoughtful, reflective, self-possessed and authentic citizen, lover, partner, parent and member of the global economy.

It  takes a special kind of student to thrive in a school like that.  I wouldn't have survived - my interests, those that I was motivated to dive into, lay elsewhere.  I'm happy it exists and that there is spectrum of schools for those who look.

 

My belief is a deep education with at least a few focuses creates the second type of person Claude Shannon describes:

“There are some people if you shoot one idea into the brain, you will get a half an idea out. There are other people who are beyond this point at which they produce two ideas for each idea sent in.”

The ability to get multiple ideas is core to serious play and creativity.   I'd claim focusing on education rather than finding the minimally easy path helps create the neural connections that make this possible.

 

A worry is that many, if not most, schools and students tend more towards vocation.  Focus on what is important to get a degree that will get you employed to pay off that mountain of debt.  Many (not all) businesses, on the other hand, would do well to have broader employees who are still learning and growing - those who can think creatively.  There is a disconnect. 

 

I hope this focus on vocation changes.  The rate of real change is great and people will keep up better if they are broader and more adaptable.  When recommending colleges I tend to focus on education ..  often to the consternation of those asking.  Of course I may be a poor example.  My education was far too narrow, but I've been lucky and curious enough to have been able to expand it a bit.  The other note is you can get a wickedly broad experience at many more places than you might imagine.

 

 Finally there's this odd emphasis on over-scheduling many pre-college kids, but I've probably written on how it kills creativity enough.

 

a kid redesigns the olympic symbol

I was in the basement with the brand new Motorola 21 inch color TV and a children's telephone.  The wires snaked out and up to the roof where my Dad was on the other toy phone braving the Montana Winter as we tried to get a usable signal.  It was my first year of high school and the Winter Olympics would be on in a day or two.   The Olympics and we had a COLOR TV!  

 

We sort of got it working by the time the broadcast began.  The Olympic symbol was displayed and the story given

 

... the six colors [including the flag's white background] combined in this way reproduce the colours of every country without exception. The blue and yellow of Sweden, the blue and white of Greece, the tricolor flags of France, England, the United States, Germany, Belgium, Italy and Hungary, and the yellow and red of Spain are included, as are the innovative flags of Brazil and Australia, and those of ancient Japan and modern China. This, truly, is an international emblem.

 

It's a very clean design, but it offended my sense of mathematics.

 

My sister had very long hair at the time.  Long enough to sit on.  She was always working on various braids.  The year before I was curious about knots and braids and learned that a simple three strand braid becomes very interesting if you connect the ends.  Take three strands here colored red, blue and green.  Wrap them in a standard braid and go through one cycle so the colors end up where they've started.  (I like to think of braids as musical rounds)

 

 

Rather than repeat the process like my sister, bend them back and connect red to red, blue to blue and green to green.  You'll be left with three loops that are fully linked, but if you remove one the whole works falls apart.  Here's a nice image.  Note that they're not really two dimensional circles as they have to pass over and under each other.  Making this forces you to use three dimensions.  

 

Technically these are a special case of brunnian links - a nontrivial link (a link is a grouping of the loops here) that becomes a set of trivial (unconnected) circles if you pull out one of the loops.  Cut any loop and the whole thing can easily be pulled apart.  Do it with five loops and you'd have powerful symbolism for the Olympic movement - or so I thought.   I knew you can make brunnian links for any odd number of loops so it was time to draw.

 

I sketched a five loop version.  Cut and remove any loop and it all comes apart.  All I had to do was find some way of making a nice drawing.  My sister is the arty one.  She had all of the right kit including some high quality colored pencils.  She also happened to be smart enough to not let me use them - even if I had noble intensions.  

 

Plan B.  Every kid has Crayola crayons.  I selected five and, after a couple of tries, I had it.  

 

(pretend the colors are accurate.  my drawing here is pretty bad, but so was the crayon version)

 

The drawing went into a special international airmail envelope with an explanation of the math and symbolism.  It was addressed to the Olympic organization in Lausanne..  I have yet to hear from them.  Apparently crayon drawings from 15 year olds don't make the cut.

 

I draw quite a bit.  It's an integral part of how I think.  It isn't pretty and I consider it an ephemeral part of thinking and trash it as soon as it's served it's purpose.  If you want meaningful design it's best to use a real designer.

 

I'm looking forward to the games!

 

much more artificial than intelligent

Look at the little group of numbers on the right.  They were handwritten and aren't exactly neat and tidy.   Take the first four on the third line.  Our brain says 1 , 7, 4 , 2.   We might bring it together as 1742 and some of you might think of a year - perhaps the year Handel's Messiah was first performed.  And we do it with a brain composed of nearly 90 billion neurons each with up to ten thousand connections to other neurons.  All of this  uses just 20 watts of power and weighs about a kilogram and a half.

 

The brain's neurons are simple computational objects.  There are input 'wires' called dendrites that connect to the cell body and outputs called axons.  Signals arrive on the dendrites and the neuron sums them.  If they exceed some critical value the neuron "fires", sending a signal out through the axons to other neurons.  There's a lot of detail to the nature of the wiring and how it comes to be, but for now think of the neurons as objects that sum inputs and fire if a critical value is exceeded...  more abstractly they're just mathematical functions.

 

The reason for starting off with some numbers is one of the basic programs - the 'hello world' if you will - for machine learning is to build a number recognizer.  A very large set of handwritten and printed numbers was scanned by a 28 by 28 pixel sensor giving a 784 pixel image.  Each of these pixels has a value depending on the brightness of the number it corresponds to.  For the sake of argument say it ranges from 0.00 for black to 1.00 for bright white.  A very crisply written 1 would be a line of dark values in a sea of mostly white. The edges would have intermediate grey values.  Less crispy numbers have lot of grey regions.  

 

To make our neural network (call it a NN) each one of these numbers is put into a corresponding neuron. The number in the neuron is called the activation.  You can think of the neuron as doing something when it's activation is a high number.

 

These 784 neurons are  going to be the first layer of our NN.  We'll have 784 neurons on one side representing the brightness of each pixel of the image.  On the far side is a small layer of 10 neurons on the other side the numbers 0 through 9.  Each of these contains the NN's best guess for how much the scanned number is one of the numbers.  For example you might have 0.02, 0.14, 0.38, 0.02, 0.98, 0.22, 0.13, 0.22, 0.33, 0.07 .. the guess  that it is the number 1 is 0.02, 2 is 0.14 and so on.  In this case 0.98 stands out, so NN is jumping up and down saying 'it's 5".

 

In between are other layers .. often called 'hidden layers.'  Rather than presenting the math that describes the network and waving our hands saying "it's all in the training", let's go in more deeply and see what's going on.  The diagram shows a very simple NN intended to recognize numbers with 784 input neurons on the left (it only shows a few... use your imagination), a single hidden larger of 15 neurons and an output layer with 10 neurons..   

 

Several years ago I wrote my first machine learning program to recognize numbers from this data set.  I used two hidden layers with 20 neurons each.  Their number and sizes are not hard and fast choices - there's a lot of room for experimentation and frankly a lot of this is more art rather than engineering.

 

Consider of the numbers from the pixels in the first layer.  These activations will determine the activations in the first hidden layer which in turn determine the activations in the second hidden layer which determine those in the output layer.  A pattern in the first determines a pattern in the second which determines a pattern in the third and the final pattern tells us the guesses for each number.

 

I like to think about the hidden layers doing something rather than just saying they're a black box.  It may be something very foreign to us, but in this case it is possible that one might, say, recognize components like edges and the next may be larger components like lines or roundish structures (a 9 is a line and a roundish structure, a 7 has two lines...).  It probably isn't doing this, but it's reasonable to think of a NN breaking down problems into layers of abstraction.  Trying to understand what's going on in these hidden layers is very difficult in large problems - perhaps impossible and that becomes an important issue when machine learning is used for purposes that impact us.¹  

 

At this point the post will diverge a bit.  I'll go into some but not detail that some folks may find useful.  I'll highlight it in blue.  Feel free to skip it if you're only interested in a high level view.  Up to now we've established a neural network is just a function that takes input on one side and spits out an answer on the other.  In this case the input is the brightness values of a 28 by 28 pixel scan of numbers and the output is the NN's best guess for each digit, from 0 to 9, of how likely the input was that number. 

 

Consider the simple, but artificial,  example where the second layer is trying to look for a some kind of feature of the number - say certain types of line segments.  Each of the 784 neurons is connected each of the 20 neurons in the second layer.    Each value in the left most layer is a pixel value.  As an exercise to make things clear take out a sheet of paper and make a vertical column of circles labeling them a₁, a₂ , a₃, .... up to a₇₈₄.  (only do a few of them  ^... are your friends)  On the right draw another column with a couple of circles.  Connect each circle on the left with each circle on the right.  We multiply each of these activations by a weight - just a number - and find the weighted sum  of all of the pixels ..  labeling the n^th neuron as a_n and the n^th weight as w_n we get:  w₁a₁ + w₂a₂ + w₃a₃ + ... + w₇₈₄a₇₈₄.   In the example of a trying to find a little line segment imagine making all of the weights 0 except for those in the region right around the little segment.  Then taking the weighted sum just focuses on the values of the pixels we care about.  If we really wanted to force the issue we could add add a negative number called the bias to the sum to require the weighted average be large.  The bias surpasses other bits of signal in the vicinity of what we're interested in.  The weighted sum going into the neuron on the second row is now (w₁a₁ + w₂a₂ + w₃a₃ + ... + w₇₈₄a₇₈₄ - b),where b is the bias.  

 

We're getting closer.  This new activation for the pixel in the second layer can have quite a range. If the weights are between -1 and 1, it can be between -784 and +784 before adding the bias.  It is common to scrunch everything to a range between 0 and 1 using a sigmoid function, call it s(x).²  Now the scrunched weighted sum is   s(w₁a₁ + w₂a₂ + w₃a₃ + ... + w₇₈₄a₇₈₄ - b) for each of the 20 neurons in the second layer.  This new activation that goes into the neuron in the second column is the crude equivalent of a synapse in biology.

 

You'll note the naming convention I've used isn't good enough.  We're just looking at what goes into a single neuron of the second layer. In this example there are  19 more, each with a connection to each of the 784 neurons of the first row and each connection with it's own weight.  And there's another bias.  Do this and you have the first two rows.  Next you consider the connections between the second and third layers (the two hidden layers in this example).  After you finish that you have the connections between the third and final layer. And this is just a simple example.   Fortunately computers are really good at this kind of math and there's a natural way to easily organize this.  If you get into this but  you'll need to be on good speaking terms with linear algebra.³

 

Before getting into the "magic"of how these weights are found take a look at the complexity of this example.  There are (784 * 20) + (20 * 20) + (20 * 10) weights and 20 + 20 + 10  biases.  That's 16,300 knobs to adjust correctly - yikes!  Imagine what it's like for problems with columns millions of elements long.  This approach, and there are many variations,  did't take off until computers became powerful enough and we had interesting data - big data - to play with.   Although there are a lot of calculations, each is very simple.  You don't need a big Intel processor with it's complex instruction set.  GPU - graphics processing units - are arrays of thousands of tiny and fairly slow simple processors.  Each is something like ten or twenty times slower than a modern Intel CPU, but all of them can run in parallel..  you end up being faster by a factor of a thousand or so.  Complex machine learning problems have moved Nvidia beyond computer graphics.  It extends to smartphones - Apple's machine learning core on it's latest processors can make this happen in your phone rather than having to rely on the cloud. 

 

Onward to machine learning because we're not daft enough to adjust this by hand.  

 

What we want to do is come up with a recipe - an algorithm - that adjusts those 16,300 knobs so we're likely to get the right answer when we give the machine a scanned number.  For that we'll use some big data.. the enormous set of  scanned and labeled (what the scan really is:  7 for example)  28 by 28 pixel numbers that's in the public domain.⁴   The data is used to train the network.  You start out running some training data, check what the results are, make some adjustments to the weights and biases and keep repeating the process until the result is good enough.  Then you test your NN by giving it another set of labeled training data it hasn't seen so you can see if it's still doing a good job. The hope is the layered structure generalizes what the recognizer does beyond the training data.  This is critical and whoever implements a NN needs to be aware of the domain(s) where it's valid.

 

So each neuron is connected to all of the neurons in the previous layer and it's activation is the weighted sum of all of the all of those neurons.  You might think of the weights as the strength of each connection and the bias is an indication if that neuron tends to be active or inactive.   To start off you set all of the weights and biases completely randomly.  It will give terrible results on the first set of training data. 

 

To make it learn you define a cost function.  We know what the number should be:  4 for example.  Now look at the differences between each final digit and what the machine guessed.  Consider the square of the difference between what you expect and what you get.  If the original number was 4 we expect the output list to behave like this:  0 to have the value 0, 1 to be 0, 2 to be 0, 3 to be 0, 4 to be 1, 5 to be 0,...

 

The first column is the number in question, the second column is the expected result of a perfect NN, the third is the square of the difference of the computed expectation and the expected result (the second column) which are shown in the forth column.

0  0        0.3       (0.3 - 0)²    0.09        

1  0        0.2       (0.2 - 0⁾²    0.04

2  0        0.1       (0.1 - 0)²    0.01

3  0        0.8       (0.8 -0)²    0.64

4  1        0.3       (0.3 - 1)²   0.49

5  0        0.7       (0.7 - 0)²    0.49

6  0        0.3       (0.3 - 0)²    0.09

7  0        0.8       (0.8 - 0)²    0.64

8  0        0.5       (0.5 -0)²    0.25

9  0        0.0       (0.0 - 0)²   0.00

 

The cost is the sum of the squares of the differences... in this case  2.74. Ideally we would get 0.00 for the cost...  

 

The cost is small when the image is accurately recognized, large otherwise. .  Now look at the average of all of the costs over all of the training data...  that gives a measure of how good our NN is for that set of data.  A single number for each set of settings.

 

The trick is just calculus.  Our AI is much more artificial than intelligent.  I'll just state a method exists to find out how to turn the knobs and when to stop.  I'll use a small bit of  blue so be ready to skip if you don't know about gradients, 

 

Imagine the case where we just have one variable.  There would be some kind of curve for the weight relating that weight to the average cost.  If it was very simple, and it's not likely to be, you could just find the minimum by taking the derivative and be done with it.  A better tactic is to start somewhere on the curve and figure out which direction to move to lower the output by measuring the slope .. move left if the slope is positive, right if it's negative. You'll end up at a local minimum.  Now imagine the function with many inputs - so you have many dimensions.  Now you're looking for the direction and steepness of steepest descent ..  just the negative of the gradient of the function.

 

So compute the gradient of the function, take a small step downhill and repeat over and over until you find a local minimum.  So put all of the 16,300 weights and biases into a tall column vector. The negative gradient of the cost function of that vector tells which corrections to all of those numbers gives the most rapid decrease of the cost function.  Minimizing it means better average performance across all of those training samples.

 

A useful way of thinking about the negative gradient of the input vector is each element tells us if a weight or bias should be increased or decreased and how important it is.  so one that starts out as 0.02, -1.29, 0.33 ...  would say w₁ should decrease a little and it is relatively unimportant, w₂ should increase a lot and it is important, w₃ should decrease a somewhat.

 

This setting of the weights and biases is called backpropagation.  We're just minimizing the cost function.  It's important that this function is smooth so we can find local minimums...  that's why the activation numbers of the neurons have a smooth range rather than something more discrete like ones and zeros.  Oddly enough this is where the brain is more digital than the neural network.

 

Playing with this example I was able to recognize a set of number images I hadn't trained the NN on at about 94% accuracy.  There are more sophisticated approaches.  It is important to stress that the innards get very messy and difficult to understand.  That's an unavoidable bug as it is very easy to fool ourselves.  There's bias in the building of the neural network, bias in the training data and bias on it's application.  It is great for some applications, but when it's used in science (I have some experience with it in astrophysics), you use it as a last resort and try very hard to keep the models simple and understandable.  The opposite of what happens in machine learning on social networks. 

 

I think they're a fantastic way to approach certain classes of otherwise intractable problems  and potentially misleading and even dangerous in other domains.

 __________

¹ Like associating aspects of our life and the games we play with our likelihood of us being a good engineer or a neo-Nazi.  These biases, some on purpose and others by accident are extremely important and worth taking up separately.  Machine learning tends to be good at identifying things very similar to known sets.  There be dragons and caution is required.  

² For example s(x) = 1/(1 + e^-x).   There are a variety of functions of this class.  This one is known as the logistic function - you pick what's appropriate for the problem at hand.   In this case large negative numbers go to 0 and large positive numbers go to 1.  

³ Organize the input activations as a column vector and the weights as a matrix where the row corresponds to the connections between one layer and a particular neuron in the next layer.

 ⁴ the MNIST Database from LeCun, Cortes, and Burges.

__________

 

The first good brussels sprouts are out.

 

Roasted Brussels Sprouts with Mustard

 

Ingredients 

° 1 pound brussels sprouts halved

° 6 tbl olive oil

° kosher salt

° 1 tbl dijon mustard

° 1 tbl coarse grain mustard 

° 1 tbl cider vinegar

 

Technique

° oven to 425°F

° put brussels sprouts on a baking sheet.  drizzle on half of the olive oil and some salt and stir them (I use my hands).  Roast on middle rack for about 35 - 45 min turning them a few times until softened and browned.

° whisk the mustards, olive oil and vinegar into a dressing and add a bit of salt to taste.

° put the baked brussels sprouts in a bowl and toss with the dressing.

 

 

 

measuring coastlines to helping someone see with sound

 Late in May of the year I was in forth grade we were given a map of the United States and Canada with outlines of the states and provinces.  Our task was was to correctly label it with capital names and rough locations along with the names of states and provinces.  We had a few for practice and the shapes caused my mind to wander.

 

States like Kansas, Colorado and Wyoming, were boring rectangles.  Or were they?  The supposedly parallel lines on the East and West would converge on the North Pole.   My Dad had had this neat little mechanical map measuring tool.  You'd zero it and then mechanically trace out a path and write down how many millimeters it had gone (his was German and labeled in millimeters).  Then you'd look up the scale on your map and convert to miles or kilometers.  I quickly learned the school maps weren't very good and started playing with road maps from the gas station.  In the day you'd pull up and have someone pump your gas, clean your windows, check your oil and water and tire pressure and then pay about thirty cents a gallon for the gasoline.  They had a pile of maps for the asking. I had a nice collection.

 

These were better, but didn't answer my basic question.  I was starting to wonder what represented ground truth.  If you're admiring the  Slartibartfast's crinkly edges of Norway you have to wonder about scale.  From space there may not be as much detail as from an airplane and you will get somewhat different results.  Down to ground level, where you can easily measure in centimeters rather than meters, it's much more complicated and is now dynamic with forces like the tides, continental drift, erosion and so on.  Where you do draw the line, so to speak?

 

At about the same time I was getting boggled, Benoit Mandelbrot was investigating fractal geometries.  He was something of a polymath and the only person I've met whose personal style included a piece of fresh broccoli in his suit pocket.   He loved self similar shapes..  A lot of math is just drawing things to see what's going on - a neat little example is the Koch curve.   Draw an equilateral triangle and the add three a smaller triangle with a sides a third the length of the original.   Now place each midway on the original triangle with so they're pointing out and just keep the shape along the outside boundary (you may want to think of cutting them out of paper and just using the outer edge.)  Now it looks like a Star a David, but you've just started.  Add new even smaller triangle along each of the edges of the Star of David and you start to get a snow flake.  Keep going. (a computer may be useful:-)

 

The edges are much more complex that anything smooth like a line or a circle.  Mandelbrot and others came up with a way to assign a dimension to any shape.  A line has one dimension, a plane has two, the space we're used to has three and math is just getting warmed up.  The Koch curve has a dimension of 1.2619.  The definition give 1.0 for lines and 2.0 for planes, but is also a way to talk about complexity and repeating complexity.  

 

Just in case someone asks,  the dimensionality of the Southern Norwegian coastline is about 1.3.  The Southwestern coast of Britain is very close to 1.25. 

 

Drawing these curves by hand or computer could and should be taught in middle and high school because it's playful.  Understanding how to calculate dimensionality of these shapes would take too long for me to describe, but I think it could be done as part of a high school calculus class.  Playing with these concepts builds your tool kit.  A tool kit that allows you to stitch seemingly unrelated concepts together.  

 

Our brains are remarkably "plastic."   If you push them with new mental activities they rewire so as to become more efficient.  Profoundly deaf people use some of the brain normally associated with sound to increase their visual acuity.  Musicians enhance regions of their brains and the mental component of many sports leads to very measurable changes. If you work at several different things your wiring may become uniquely creative -  I suspect some of the most creative athletes spent a lot of time studying an instrument at some point. These enhancements are readily seen in brain scans even in people in their sixties. It's easy to measure the changes even in people who are in their sixties.  You can teach old dogs new tricks.  (this is such a rich area, but back to the equally rich math)

 

What if you wanted to help a blind person by encoding the information from a camera in sound?  How would you approach the prolbem?

 

To get an idea of what's going on I take a very simple picture..  in this case 256 pixels on a side.  I start with my avatar. (pretend it's 256 x 256...  this one is twice as large)  I make it black and white..  256 grey levels in this case.  I don't know what to do with more detailed information...  I'm just dinking around a bit with the problem.  

 

How to present this to the brain? Don't worry if this would actually work - just consider it a little thought experiment.  We could associated each pixel had a frequency?  The loudness could be associated with the brightness.  In this case white could be very quite and black the loudest.  You might play them all at once, but it might be too confusing..   Try adding a bit of order.  The question is what kind of order?  (again. . there are smarter ways to do this - you may be thinking of some now, but let's work on this one)

 

It could be some randomly chosen ordering.  After all, the brain is good at finding order in a mess.. as long as we use the same random ordering.  But that presents a problem..  what if we use a better camera..  like 3000 x 4000 pixels?  Then we need a new random ordering and the brain has to start all over again.,

 

There is raster scanning .. like an old CRT ... you move back and forth t across the image starting at the bottom and shifting up a pixel each time you reach an edge reading the brightness of each pixel you encounter until you've scanned the whole image.   Simple but it throws away some good intuition.  An image tends to have a lot of neighboring similarity.  It would be nice to make use of that. Hang onto that thought for awhile.  

 

A mathematician might suggest you consider a Hilbert curve -  more specifically a Pseudo-Hilbert curve. Rather than get impressed by the fancy name we'll drop the pseudo and just play around.   Take out a sheet of paper and draw a square.  (line 1 in the figure) Now divide the square into equal quadrants.  Starting in the lower left quadrant, draw a line up to the upper left quadrant, continue to the upper right quadrant and then down to the lower right quadrant.   This upside down U is an Order One Hilbert curve.  Now it's time  to make it work.  (line 2)   Draw another square with four quadrants.  Now draw an Order One curve in each of the quadrants.  Start traveling from the lower left and you find yourself making some unfortunate traverses.  Is there any way to rearrange these Order One curves to get something that doesn't intersect itself?

 

 

 

 

It turns out there is.  (line 3).  Rotate the curve in the lower left quadrant 90° counter clockwise and it's neighbor in the lower right quadrant 90* counter clockwise.  Now you can connect all four Order One curves into something that doesn't cross itself - an Order Two Hilbert curve. You can probably guess the algorithm from here.  To draw the next order draw your current order Hilbert curve in each of four quadrants of a square.  Rotate the bottom left curve 90° clockwise and the bottom right 90° counter clockwise.  Connect them and you're done.

 

Now that you've got the recipe you can draw Hilbert Curves of any order your patience allows.. Past four and it's best to use a computers.  Here are Orders One through Five

 

 

There are two things to notice:

 

First the curve is covering more points of the square as the order increases.  We can talk about it's dimension.  As the order goes to infinity, the curve's dimension is 2.0 ... in other words the curve hits every point inside the square.   How sweet - an infinite one dimensional object that covers two dimensions.  Our job is much easier - we're using pixels and only require the line to go through a pixel.

 

Second, the curve preserves localized information as the order of the curve goes up.  Look at the nose of my ferret. If I stretch out the curve the points associated with the nose are roughly in the same region of the line.  This is useful if our blind person is listening to the string of notes coming out.  Most of them around the nose have a similar value and are played at nearly the same time.  If we use a camera with more pixels that local nature is still preserved.  The raster scan that snakes up the image don't preserve locality.

 

There are many uses for these locality preserving mappings (as they're called) and space filling curves like the Hilbert Curve are just what the doctor ordered.  And you don't have to stop with one dimension. They may get hard to visualize, but the technique works with as many dimensions as your heart desires.

 

Math is useful for calculating, but more important is the fact it's a playground that offers insight and new ways of thinking.  It is one of the glues of creativity.

 

 

two goats, one car

I just heard that Monty Hall died at the age of 96.  He was a game show host famous for a simple probability problem that gets people riled up.  His show, Let's Make a Deal, had three doors. Behind one door was something very desirable - like a new sports car.  The other two doors had, well, something like a goat.  You were asked to pick door number one, door number two or door number three.  At that point your chances of worrying about the color of your new car were one in three.  After you made your choice Monty would open one of the other doors revealing a goat and would ask if you wanted to switch.

 

People tend to think it's a coin flip at that point, but that's not thinking it through.  Monte has perfect knowledge of the situation and has added some information to the puzzle with is reveal.  The right strategy is to switch - always.  Your chances will have improved to two out of three.

 

One way to think about it is if you picked, say, door number one, the chances of it wining are one in three and the chances of each of the other two are one third each.  This means the chance of it being in either two or three is two thirds.  Monte, by opening door three (for example), has increased the likelihood of it being behind door number two to two thirds.   He added information. 

 

always use extra ground truth when it is available!

 

You won't always win, but if you play a large number of times, you'll win about two thirds of the time.  You can write a little program to convince yourself of imagine a game where Monte has ten thousand doors.  One car, 9,999 goats.  You choose door number one.  Monte opens all of the doors except, say, 1,729 and asks if you want to switch.  My guess is your intuition will do the right thing:-)

 

 

 

precision twinklers

Fifty years ago this month Jocelyn Bell was going through the chart recorder output from a new radio telescope she and a few others had built at Cambridge .  She had worked herself up to about a hundred feet of it a day looking for anything interesting in the squiggly line of ink. Very boring work.  Lots of noise and bits of the expected.  

 

But there was this "bit of scruff" when the telescope was pointed in a particular direction.  The scruff was repeating every 1.3 seconds.  Very regularly.¹ 

 

You always doubt your equipment.  Regular noise is usually something man-made or an artifact from the apparatus.  And there were those new fangled satellites beeping away,.  Carefully you try to eliminate the possibilities.  She and her advisor eliminated most possibilities, but they weren't that certain.  Then she found another bit of scruff.  That regular pulsing .. now at a different rate, but still very accurate.  Better than the clocks they had in the lab.  She had found something - something out there.  

 

There was a flurry of activity when they announced.  Astronomers around the world dropped what they were doing.  Theoretical physicists and astrophysicists conjectured.  A few of the conjectures advanced to hypotheses.  And in this swirl of activity  she found and third ... and then a forth.  And in theoryland of the hypotheses made sense.  It was called a pulsar.    

 

She had found the crack in the door to some of the deepest Nature yet encountered.

 

In 2017 something like two thousand have been found.  Shortly after the discovery it was suggested a star could collapse so dramatically that it's core was nothing but neutrons.  Imagine a mass greater than the Sun compressed down to something about a dozen kilometers in diameter.  Like a skater pulling in her arms it spins much faster than the star it started out as. The first went around every 1.3 seconds, but some spin more than a thousand times a second.  Their magnetic fields are trillions of times stronger than the Earth's and, combined with the rotation a beacon-like signal forms.  It sweeps through the sky like the light from a lighthouse as it rotates.  

 

She and her advisor opened up an entirely new and unexpected branch of astrophysics that has led to a much deeper understanding of both astronomy and physics.  Pulsars and their close relatives are hot areas of research.  And they can even be useful.  A GPS system is just a group of accurate clocks with transmitters orbiting the Earth.  Pulsars are as accurate as atomic clocks. You  could use them to build a galactic positioning system that would work anywhere in the Milky Way and not than just on Earth.   And you can make use of their regular beat to build another kind of gravity wave detector.  One that is complementary to the current interferometry technique.  

 

Some of the techniques developed since then have trickled down into important technology we use and many hundreds of astronomers and astrophysicists have spent some time in the private sector making their contributions to the economy.  Pure science is a very inexpensive mechanism for creating future value.  The "problem" is you can't predict where it might lead.

 

Hewish went on to receive a Nobel Prize in Physics for the discovery.  It is widely felt in the astronomy community that Jocelyn Bell should have shared in the discovery as her contributions were enormous.  But:    1967 and female.   She spoke of it ten years after the discovery:

 

"demarcation disputes between supervisor and student are always difficult, probably impossible to resolve. Secondly, it is the supervisor who has the final responsibility for the success or failure of the project. We hear of cases where a supervisor blames his student for a failure, but we know that it is largely the fault of the supervisor. It seems only fair to me that he should benefit from the successes, too. Thirdly, I believe it would demean Nobel Prizes if they were awarded to research students, except in very exceptional cases, and I do not believe this is one of them. Finally, I am not myself upset about it – after all, I am in good company, am I not!"

 

She's being far too generous.  This was one of those exceptional cases.  At least she's received other significant honors she's had throughout her career.

 

____ 

And something sad.  

 

I rarely use the term genius to describe anyone alive.  Maryam Mirzakhani was an exception.  A fearless mathematician, she was the only women to receive the Field Medal: math's highest recognition.  She died on Saturday at age forty.  She was just warming up.   Here's a well-written piece about her that appeared in Quanta a few years ago.

 

Excuse my political comment, but Mirzakhani was female, brilliant and from an Islamic county.   I doubt someone with these "liabilities" would be welcome here now ...

__________

¹ She recognized that pulsars are astronomical sources where others had failed because she noticed that the pulses in her data (Figure 6.1) didn’t look like other forms of interference and they reappeared exactly once per sidereal day, indicating an origin outside the Solar System.  She and Hewish, “decided initially not to computerize the output because until we were familiar with the behavior of our telescope and receivers we thought it better to inspect the data visually, and because a human can recognize signals of different character whereas it is difficult to program a computer to do so.”  Other people were using software to filter out noise and were throwing out the interesting signal in the process..   

 

 

  

norm

My dear friend Norm Schryer died unexpectedly this morning.  This past week hasn't been a kind one. 

 

Norm was one of those special beings you sometimes hear about and, if you're really lucky, you get to work and play with them.  He was a Bell Laboratories mathematician - he'd probably say the old Bell Labs - the Bell Telephone Laboratories.  The labs had a tendency to hire a large number of physicists and mathematicians - much larger than any other industrial lab.  People were granted enormous freedom, but HR was looking for people who played well together and who might consider AT&T a problem rich environment.  Given a few interesting constraints imposed by the government much of the future was invented. 

 

Norm made numerous contributions, but more impressively they covered several fields including those being invented.  He brought what appeared to be a near-reckless state of play.  Norm had the energy to plow into "short term" problems for weeks or months at a time.  Those of us who worked with him saw him as brilliant, generous almost to a fault, loud - very loud, and incredibly playful - think of an 8 year old wound on candy.  Norm was like that almost constantly.

 

We had a very small working group of three with a few other players coming and going.  We knew that brainstorming doesn't work (I have a great story about that - if you ever catch me with time ask about SewerNet and SantaDelivery ... perhaps my best three hour stint of working against the forces of corporate bureaucracy).  What does work is when you have a small group of people that know each other very well.  Diverse talents and interests are central - we had two physicists and a mathematician, but many others came in and out.  People would present ideas which would be torn to shreds and new paths tried.  On the fly estimates and calculations were central and we each  had different approaches.  

 

Norm would sit and listen as others opened up their ideas.  Mind you not quietly.  If there was something interesting he'd be up on the desk or couch whooping and hollering.  (once I had an office next to his..  my number was 664  and the sign read 'neighbor to the beast').  The blackboard would fill with drawings and calculations .. all of us were old hands at this sort of thing.  Everyone knew it was the idea under attack and not the person.  If I tried this approach in a group I didn't know I'd probably make enemies in short order.  We could move quickly.   It turns out to be an astonishing fun and productive path to creative collaborative thinking ... I've even helped some companies with it.  Friday afternoons were usually reserved for this - Friday from 1pm until 8 or 9 at night when we got too hungry.  

Back to Norm ..  his function was centrally important.  He was the great synthesizer and cheerleader.   I've never met anyone with such a powerful knack of synthesizing disparate ideas.  He'd listen to a few things and say .. "let me see if I'm understanding this properly.."  and then he'd launch into a beautiful description that was often much more interesting that what you had presented.  He'd find inspiration for bringing in ideas from all over - 13^th century Italy, a Norden bombsight and skip bombing from WWII, pure math, art, ...  I began to see the value of bringing in ideas and people from far outside your field and to offer your services to others because they might be curious.  

 

He helped me learn how to harness the power and joy of simple play.

 

This has stayed with me and helps define how I see the world.  He probably had no idea he was a powerful mentor. 

 

In two weeks time we would meet for a "ducking" to celebrate his birthday at the Peking Duck House in Chinatown.  Norm would have been 75.  The world seems too quiet, too sterile, too joyless tonight.  I only have a few very close friends and that group is now smaller by one.  One of the others suggested I howl at the moon inf frustration.  I may do just that ...

 

e^sad  

__________

No recipe corner other than noting Norm didn't exactly have a healthy diet.  Vegetables were "what meat eats..."  Shrimp was a matrix for holding melted butter.  In politics and food we were polar opposites.  That didn't get in the way of a long friendship.  So if you want to eat one last unhealthy thing this holiday season, Norm would probably applaud.  

 

Instead this is the shard that managed to get us interested in each other's mind. Bail now if you don't like math.  I'll just note it is semi-geekish.

 

I had only been with the Labs for two years when we first met.  I heard he had a great PDE solver and I wanted to do some numerically solve Maxwell's equations in the near field (if that seems foreign don't worry. ..  it's just trying to understand the behavior of light very close to the end of a hole much smaller than the light's wavelength.  It's messy and people usually ignore it as unimportant).  We were having such a fine time that one of us brought up Khinchin's Constant.  It surprised me he knew it's history (it was a fairly recent result) and apparently it surprised him that I knew about it in the first place as I wasn't a mathematician. It's so sweet it needs to be mentioned.    

 

If you take the geometric mean of the partial quotients of the simple continued fraction of a simple real number, the limit as n → ∞ the result is independent of the number (except for the set of measure 0) and is about 2.685452...    It isn't known if it is irrational, let alone transcendental.  It's enough to invoke wonder and that was enough to start a long friendship. 

 

(here it is more formally - you can't do this easily in html, so I'll use something else and paste an image)