overview

Science tries to understand the world.

We can just describe what we see: that is called ‘exploration’ (or descriptive studies). Exploratory science is about finding something novel, or looking at things we have never seen before: such as when the microscope was invented or when a new continent was discovered. But what we perceive is biased by what we expect and want to find.

What we also want is to discover general rules about how the world works (this is called induction). We cannot directly ‘see’ those rules; we need to infer them. The power of modern science lies in inference: we know the Earth moves around the sun even when we cannot directly ‘see’ this; and we , we know that climate depends on CO2 levels not because we somehow directly observed it, but because we inferred it.

Strong inference, which I refer to here as ‘The Scientific Method’, is to infer how the world works in a way that is robust to our biases and prejudices.

common misconceptions

Most common objection to the Scientific Method: scientists do a lot of other stuff. In particular, scientists also explore, describe, discover simply by observing. Indeed they do.

‘The Scientific Method’ is not a term that means ‘all the methods scientists use’. It is a term for a very specific approach that allows us to prove ourselves wrong, an approach that allows us to make inferences from observations, particularly when we cannot directly measure the claim in question.

Sometimes we distinguish between ‘descriptive’ or ‘exploratory’ studies on the one hand and ‘confirmatory’ or ‘hypothesis-testing’ studies on the other hand. The Scientific Method does the latter; the former is also necessary.

Step 1.

Ask a question.

The Scientific Method depends on defining the exact, specific question you want to answer before you start. Why? Because otherwise it is impossible to come up with a set of comprehensive, very specific answers (see step 2). The question can come from anywhere; maybe it was prompted by your earlier exploration & descriptive observations - or maybe you just dreamt it up in your head.

common misconceptions

Setting out without a clear question means you are doing exploratory research; this important when the area you are exploring is completely uncharted, but what you find is likely to be strongly affected by your expectations.

Many accounts, especially descriptions in K-12 educational materials, claim that you must start by observing - this is wrong. There is no reason for that except the practical one that if you can’t think of a question to study, it helps to do some exploration first. But where your question came from does not impact the rigor of your science.

Step 2.

Think of all possible answers.

You must define what possible answers (=hypotheses) you want to pit against each other before making any observations. Why? Because your goal is to demonstrate some of them are wrong; you can never demonstrate that any specific answer is right. All you will do is stick to the only answer that remains possible (after you have rejected all others).

If your set of answers wasn’t complete, this doesn’t work. If your answers don’t contradict each other, again this may not work since you may not reject any.

Again, it does not matter where these possible answers come from, but it helps to have done some exploration first, or to read all that is known about related subjects, to come up with a good set of hypotheses.

common misconceptions

It is NOT ok to come up with hypotheses after the fact, i.e. after doing your experiments, making measurements, doing observations etc. that are part of this study. If you did observations as part of your exploration to come up with a question and hypotheses, these observations cannot be used again below. Violating this rule is known as ‘HARKing’ (Hypothesize After Results Are Known) and is a form of scientific fraud (or at least malpractice).

Hypotheses should be answers to your previously defined question; and they should ideally also be interesting. This means they are not statements about your experiment, but statements about the world.

Hypotheses also help define what you are really asking.

Step 3.

Plan relevant measurements and then think about what each answer predicts.

This is the key step that locks you into a path, and prevents you from wiggling out of it if you don’t like the answer you are getting. To practice the Scientific Method, to get objective truth, you must define how you will interpret every possible result before you get it. In other words, you state what your measurements/observations will look like for each hypothesis - these statements are called predictions. Why? Because that way, when you get an answer you don’t like, you can’t say in hindsight ‘oh it was probably just X that changed the outcome, but my favorite hypothesis is still true’. No, if you previously said this outcome means hypothesis 2 is true not hypothesis 1, now you have to stick to it.

‘Prediction’, in everyday life, typically is a unique claim that means ‘what I think will happen to the best of my knowledge’. This is NOT what ‘prediction’ means here (it does not matter what you think is most likely to happen). What it means here is simply ‘what WILL happen if this hypothesis is true’.

common misconceptions

A lot of modern educational materials try to de-emphasize structure in science, and emphasize creativity and dynamics - but this leads to misunderstandings of how science works. Being a scientist, as a job, can be dynamic and fun; it requires creativity in designing questions and hypotheses but also in deciding what specific things to measure & make predictions about. Modern science happens in diverse teams, includes exploration and a host of actual activities in practice. But modern science does NOT mean we can weaken the structure of the Scientific Method. The key to being able to prove yourself wrong is to set up a process in which you, as a person, have zero wiggle room, not chance to dynamically change ‘what you meant’ or how you prefer to interpret things. Any such opportunities will be used to bias conclusions.

Because of this, ‘pre-registration’ is now becoming more popular in science: researchers actually publish what their question, hypotheses, and predictions are before they start collecting data.

A lot of educational materials misunderstand ‘predictions’. In the Scientific Method, it is not a person who predicts, it is each of the contradictory hypotheses (leading to at least two contradictory predictions). If this is not the case, no inference about the hypotheses can be made.

Step 4.

Do observations or measurements.

Now you can do your experiment, your measurements, whatever way you can get information about the world that you did not already know. It has to be precisely about the thing that your predictions are about. Why? Because you want to compare these measurements to the predictions of the two hypotheses; since the predictions are contradictory, no matter what your measurements are, they’ll agree with one and disagree with the other prediction.

Common Misconceptions

Many students are taught that science starts with observations; it may be that initial observations are what made people curious about the object they study, but the key to the superpower of science is that new, previously unknown and unanticipated data are used to actually test whether our ideas are correct. A hypothesis that has never predicted something we didn’t already know has not been confirmed by science.

Another misconception is that scientific ‘experiments’ are necessarily complicated or expensive, or need a lot of equipment. This is not necessarily true. Sure, studying DNA sequences or subatomic particles may be expensive; but many, even professional science studies require no more equipment than your eyes, or the internet, or a thermometer.

Step 5.

Interpretation: compare measurements to predictions.

That’s it. All the hypotheses that predicted something that did not happen are wrong (‘rejected’). Whatever hypothesis is left over is the true answer. You’re done, you have an answer to your question - no excuses.

Common Misconceptions

The most prominent misconception in K-12 teaching is that there are somehow several steps here of ‘analyzing’, ‘concluding’, etc. But it’s just one step, and it’s conceptually pretty trivial. Complication comes in because of noisy measurements and ‘confounding factors’, all of which is dealt with by using statistics. Indeed statistics is an important tool of any modern science, and can be complicated. But it’s important not to forget the forest for the trees - all we’re doing here is comparing the predictions to the measurements, and deciding which ones disagree.

Perhaps the most insidious misconception is that if you don’t get the result you want, you should start over from the beginning, perhaps tweaking your hypotheses or predictions to make them more likely to ‘work’, i.e. agree with data. Every time you do this, you weaken your inference, because you make it harder to prove yourself wrong. If there are reasons to think your method didn’t work, you should have decided to stop and re-do before getting results.

Another element recently introduced is the idea that you also have to communicate your results. Yes, I fully believe you should do so; but this is not part of the ‘Scientific Method’. Whether you communicate your results or not does not make them more or less true. Remember that the Scientific Method is not about what scientists do, or should do, all day - it is only about what allows strong inference, i.e. finding the truth without bias, being able to prove yourself wrong.