It was the beginning of Spring in 2014 and there was electricity in the air. I've only seen scientific drama at that level a few times. First the leaks and then a well-known team made a dramatic announcement. These dramatic events can be confirmation of a radical hypothesis or sometimes it's Nature proclaiming how limited human imagination really is with something completely unexpected. This time it appeared to be confirmation of a beautiful idea. If it held up it would signal the potential of a new chapter in physics and cosmology. A few people thought about shopping for suits that would be appropriate at a formal event in Stockholm.
For about fifty years we've known the Big Bang couldn't be the primordial event. It was not the beginning of our Universe. There were a conjectures .. the most interesting came from Alan Guth who proposed an enormous mind-blowing-even-by-the-standards-of-cosmology inflation that took the Universe from something unimaginably small to something more the size of a baseball in a tiny sliver of time. Over the years it was modified and tweaked and, although unconfirmed, many of us fell back on it to think about the pre-Big Bang because it nicely predicated what followed. It had become a standard bias - one that people would abandon if proven wrong - but a bias nonetheless.
The conjecture was developed into a testable hypothesis by a few groups.1 The first report came from an experiment at the South Pole called BICEP2 in 2014. They saw a high quality signal that implied cosmic inflation. I noted my own excitement at the time.
During the year that followed the serious skepticism of science took hold. The team was known to be careful. There are several things you do in any physics experiment to check your apparatus, how you handle information, biases and errors. Ultimately you hope that someone else has a similar result with a different approach and apparatus.
There's a lot of preparatory work. You need to understand what processes might mimic what you might be looking for. Usually it's not a blind hunt.. you're guided by theory looking for something quite specific. Discovery is finding it or finding the unexpected. It's fun to make theorists go back to the drawing board. Most of them agree.
A good deal of thought and computer modeling goes into designing the experiment and understanding exactly what you should see - rediscovering well-known and agreed upon experiments. If you can't get this foundational piece right, no one will believe you. It turns out this step is where many of the data analysis paths are fleshed out. This part, by the way, is where playfulness is essential.
You need to understand what are known as "backgrounds" - events that mimic what you might be looking for. There are null tests where you put a detector in a shielded box and understand the noise it produces. These can get sophisticated. Then there are calibrations where you see if you can measure known signals with all parts of your experiment. Currently a hot topic in astrophysics is understanding what went on in the Universe when the first stars formed. Unfortunately there wasn't any light - it was a dark age. But there's a signal from hydrogen atoms that can be detected. The process is well-known in radio astronomy and regularly used, but tuning into the dark age is leading edge work. To sort out the background noise astronomers pointed a radio telescope at the Moon during the last lunar eclipse. The noise component from the Sun would be missing and they'd be able to calibrate their apparatus and computational approach. The signal was beautiful - the reflection of the rest of the Universe from the Moon. Most of it turned out to be the Milky Way and the animated gif shows the result.
There are also jack-knife tests to remove regular occurring biases. If you were studying car traffic flow you'd note rush hours are busy and weekends aren't. So you break up the data into chunks and examine them separately to see how they compare. The need for good enough random numbers often turns up here.
You constantly have to check the state of your apparatus and filters. In an experiment of any size there's always something broken and something malfunction. These have to be noted and accounted for. It's not uncommon to spend a significant amount of time looking for problems.
Then there's the human problem. How broad and deep is the team? In the case of BICEP2 there wasn't any communication with people who did observational galactic astronomy. When the results were published they noted there was a background that hadn't been accounted for. It seemed obscure to the team and to most of us, but in the end it was nearly fatal. The probability of discovery and confirmation dropped dramatically. About a year later another experiment drove the nail home. Inflation wasn't dead, but their experimental approach couldn't detect it.
And then there are dragons. If someone asks you what's in your driveway and you tell them a car, both of you are generally satisfied. But if you tell them it's a dragon ... well then you have some explaining to do. Reporting dragons can color fields. A classic example came during the 60s.
Joe Webber was an outstanding experimental physicist who thought he had come up with a way to detect what Einstein predicted, but said would never be detected ... gravity waves. He carefully eliminated many sources of noise and ended up seeing a signal. He didn't have much in the way of theoretical guidance to know how big the signal was and he thought he was careful enough, so he published.
The problem was no one else was able to find a signal. For the next few decades no one saw anything and the field was nearly dead quiet for about thirty years. There were theoretical hints that big advances in detector technology might find a signal and, through luck and enormous work, the Laser Interferometer Gravitational Wave Observatory - LIGO - was born. About twenty years later it bore fruit and an advance as important as Galileo's telescope was made. But people had to be careful... very careful. How do you deal with human sociology? What if people expect something to be there - or not - and these biases get built into data taking, algorithms and the final results.
They had a neat idea
The LIGO physicists, astrophysicists and engineers got the signal they were looking for. About four hundred people went to work trying to prove or disprove it. About six months later they knew they had something. The team leaders gave their thumbs up, champagne came out and several lengthy papers were written. As they were getting ready to send the papers for peer review a senior researcher held a town meeting. They had been working on a faked signal - a devilishly clever faked signal.
They needed to go back and understand what they were doing at a much deeper level. They had too many biases. Since then there have been several false alarms and about a half dozen real signals. They have developed one of the most careful cultures on the planet using their blind injection technique.
__________
Carl Sagan wrote extensively on critical thinking. Perhaps his best non-technical piece on the subject is a chapter in The Demon-Haunted World: Science as a Candle in the Dark. The specific chapter is The Fine Art of Baloney Detection. The type of skepticism is useful far beyond science... I'm thinking of advertising, social media, politics...
The kit is brought out as a matter of course whenever new ideas are offered for consideration. If the new idea survives examination by the tools in our kit, we grant it warm, although tentative, acceptance. If you’re so inclined, if you don’t want to buy baloney even when it’s reassuring to do so, there are precautions that can be taken; there’s a tried-and-true, consumer-tested method.
he offers nine tools - read the chapter for the detail
Wherever possible there must be independent confirmation of the “facts.”
Encourage substantive debate on the evidence by knowledgeable proponents of all points of view.
Arguments from authority carry little weight — “authorities” have made mistakes in the past. They will do so again in the future. Perhaps a better way to say it is that in science there are no authorities; at most, there are experts.
Spin more than one hypothesis. If there’s something to be explained, think of all the different ways in which it could be explained. Then think of tests by which you might systematically disprove each of the alternatives. What survives, the hypothesis that resists disproof in this Darwinian selection among “multiple working hypotheses,” has a much better chance of being the right answer than if you had simply run with the first idea that caught your fancy.
Try not to get overly attached to a hypothesis just because it’s yours. It’s only a way station in the pursuit of knowledge. Ask yourself why you like the idea. Compare it fairly with the alternatives. See if you can find reasons for rejecting it. If you don’t, others will.
Quantify. If whatever it is you’re explaining has some measure, some numerical quantity attached to it, you’ll be much better able to discriminate among competing hypotheses. What is vague and qualitative is open to many explanations. Of course there are truths to be sought in the many qualitative issues we are obliged to confront, but finding them is more challenging.
If there’s a chain of argument, every link in the chain must work (including the premise) — not just most of them.
Occam’s Razor. This convenient rule-of-thumb urges us when faced with two hypotheses that explain the data equally well to choose the simpler.
Always ask whether the hypothesis can be, at least in principle, falsified. Propositions that are untestable, unfalsifiable are not worth much. Consider the grand idea that our Universe and everything in it is just an elementary particle — an electron, say — in a much bigger Cosmos. But if we can never acquire information from outside our Universe, is not the idea incapable of disproof? You must be able to check assertions out. Inveterate skeptics must be given the chance to follow your reasoning, to duplicate your experiments and see if they get the same result.
He goes on and notes we are vulnerable to common pitfalls of common sense and lists twenty common ones... the list is excellent and exceptionally appropriate these days.
__________
1 In the physical science rough ideas are conjectures.. If you can test them they are elevated to hypothesis. Theory usually refers to something that is well understood and tested by several approaches. It is a fairly high bar. In particle physics it means being observed with an uncertainty lower than one part in three million - preferably by multiple approaches. This is one of those differences in definitions that causes confusion when communicating with the public where 'theory' usually means someone's idea -no matter how good or bad. I'll be a bit more specific as these ideas can be used to one degree or another outside of science.
I'll add that cosmology and astrophysics produce the most mind blowing graphs I've encountered. Math can be more dramatic, but these are trying to find a visual vocabulary to describe Nature.