I have a (somewhat major) work event this week, and it’s my one-year wedding anniversary this weekend, so I won’t have as much time to write posts this week. I am still writing a post on Peter Millican’s assessment of Hume’s argument on miracles and will hopefully get that out in a couple of weeks. I decided to write this quick post (by my standards) because it’s less dependent on primary/secondary source material. Enjoy!
Earlier this week, Noah McKay wrote a nice, quick post saying that flexible hypotheses can be conclusively confirmed.
Oof. Conclusively Confirmed? That’s a strong claim! And it’s likely wrong or at least needs strong revision. Let’s dive in.
“Says Who?”
Before starting, I will just say that some of what I say here will come off as a line-by-line rebuttal and perhaps a little pedantic. I hate doing this! However, when it comes to forming beliefs and making arguments in philosophy (or science or history or sociology or…) word choice matters.
When we’re not specific about our wording, we can smuggle in a strawman argument or demand our intellectual opponents tell us when they last beat their wives (without evidence that they’ve ever done so). In his first few sentences McKay makes mistakes such as this. He starts out:
“Many people think that hypotheses can’t be strongly confirmed by observation unless they make strong, precise predictions. Flexible hypotheses — hypotheses that make no strong, precise predictions — are suspect, since they aren’t open to strong empirical confirmation, even in principle.”
To start,1 “many people” is doing a lot of heavy lifting here. Who exactly says that hypotheses can’t be strongly confirmed by observation unless they make strong, precise predictions?
As far as I can tell, philosophers of science who think and speak in the language of precise hypothesis predictions are usually Popperians or something similar, believing that empirical experience corroborates or falsifies hypotheses, but never confirming them. On that reading alone, I’m confident that this is a strawman.
I and other skeptics (including Popperians) would probably support the proposition that “one cannot be strongly confident in a hypothesis if it doesn’t make a precise, quantifiable prediction.” But that’s a different position than what McKay is arguing against.
The reason why skeptics or naturalists value precision in predictions is because it works as something of a bridge between the old Humean divide between logic and matters of fact. Math and numbers at least appear to be in the domain of logic, and so if we assign numerical values to the specific variables we are trying to test2 we can at least somewhat logically assess those theories.
If you have a scientific theory that makes a claim that x causes y, but 99% of the time (or more) x doesn’t conjunctively correlate with y, we can be as confident as possible that x probably doesn’t cause y. Similarly, when we run our tests, operationalizations, and so on and find that x is conjunctively correlated with y 20-30% of the time or so, we can be confident that there’s at least some causal effect going on, or that variable x is confounded with another variable z that causes y. People like to use examples of clear cut laws of physics to illustrate the scientific method, but the point of this paragraph is that most science is not this obvious.
To illustrate this point more bluntly and humorously, the laws of physics are easy, figuring out why I once gained 20 pounds in high school running cross country is hard. Laws of physics are usually pretty obvious, while the scientific and explanatory questions we’re trying to answer often have multiple causal variables influencing them.
Why Precision Is Important
The reason why I bring this up is because McKay is misunderstanding why the people he vaguely criticizes value precise hypotheses. At the risk of sounding like AI: It’s not for confirmation, it’s for confidence; it’s not for strength, but just precision.
Most hypotheses aren’t strong in the sense that they are bold and clear and monocausal, but rather that their effects are clear or at least measurable. They will create results that you can be confident in because they are repeatable and predictable. The reason why hypotheses that make no precise predictions are suspect is because they cannot be falsified or corroborated with any sort of confidence. To the extent that we are confident in imprecise hypotheses, it’s because we have a subconscious tally of evidence supporting them that comes from our familiar experience that informs our formulation.
None of this is to say that you can’t have a hypothesis that’s laced with uncertainty. You can have a hypothesis with bands of uncertainty, margin of error, and other tools that communicate uncertainty. But when you have no precision in your hypothesis, it’s arguably not even a hypothesis.3
A Sleight of Hand
I bring up the importance of precision in hypotheses to elaborate upon this more important mistake by McKay. That is, you can’t be “conclusively” confident of any hypothesis or data (to the point of declaring it conclusive) if the data itself is scarce or nonexistent.
This is a classic Humean argument, found in his Dialogues Concerning Natural Religion, in rebuttal to many natural theological arguments that rely on arguments from metaphor: We cannot be confident in arguments for matters of fact we are unfamiliar with, based on metaphor alone. First, a creative mind can come up with better analogies that supplant the metaphor with another metaphor that could itself be false.4 Second, as the differences in the two compared things become more pronounced, so too do analogies become increasingly disanalogous.
The Humean, as do most skeptics, maintains a position of agnosticism towards these unfamiliar causes and effects, until they gather high-quality data and evidence. For these skeptics, the scientific method (through precise measurement, operationalization, falsification, and other methods) is the way we capture that data, study it, and gain confidence in how causes and effects work beyond our intuitive familiarity.
So when McKay progresses from his strawman above to the quote below, keen eyes will notice the philosophical sleight of hand he has done here.5
“This common assumption is false, and provably so. Suppose that the following two hypotheses jointly exhaust the epistemic possibilities (that is, you know for certain that exactly one of of them is true)…”
The whole point of the criterion of precision, falsification, and so on is to generate data from which we can confidently formulate hypotheses in circumstances that we are not familiar with. When we’re doing this, we don’t know to what extent the epistemic space is exhausted.
That is to say, before we can get into the flaws of the content of McKay’s thought experiment, it’s already failed because it misunderstands why people support precise hypotheses in the first place! That is, precision, falsification, and other hypothesis criteria exist to narrow down an uncertain epistemic space. Closing the epistemic space changes the subject altogether.
The Roommate Analogy
Let’s look at his roommate thought experiment and explanation to understand further why it fails:
“Roommate
You’ve just checked into student housing. You know your roommate will arrive sometime this week, but you’re unsure exactly when. You know almost nothing about them, except that they have the only other key to your room. In the morning, you leave your room a mess. You return in the evening to find it thoroughly cleaned. You conclude with near-certainty that your roommate has arrived.
Your reaction in Roommate is perfectly rational, despite the fact that the hypothesis that your roommate has arrived in town makes no strong, precise predictions about the state of your room.
There’s a perfectly straightforward explanation for why these flexible hypotheses are conclusively confirmed by the evidence described: that evidence is much, much more likely given those hypotheses than given the alternatives. It’s not likely, given that the lottery is fair, that you’ve won; but it’s certain that you’ve lost unless the lottery is fair. Similarly, it’s not likely, given that your roommate has arrived, that your room has been thoroughly cleaned; but it’s near-certain that your room is a mess given that they haven’t arrived.”
Conflating Hypotheses And Conclusions
It seems that McKay is confusing a conclusion with a hypothesis.
A conclusion is the logical end or implication of two or more premises, while a hypothesis is an explanation based on limited evidence as the starting point for further investigation. It seems very odd to say that a few data points led one to form a conclusive hypothesis that didn’t make further precise predictions, as opposed to those few data points led one to infer a conclusion.
I mean, just re-read the excerpt and replace “conclusive hypothesis” and probabilistic language with logical and “conclusion” language. Heck, I’ll do it for you:
“Revised Roommate
You’ve just checked into student housing. You know your roommate will arrive sometime this week, but you’re unsure exactly when. You know almost nothing about them, except that they have the only other key to your room. In the morning, you leave your room a mess. You return in the evening to find it thoroughly cleaned. You conclude with near-certainty that your roommate has arrived.
Your reaction in Revised Roommate is perfectly rational, despite the fact that the conclusion that your roommate has arrived in town makes no strong, precise predictions about the state of your room.”
I just don’t see what McKay is doing here. He’s clearly conflating conclusions, which happen at the end of an investigation, and hypotheses, which are formulated at the beginning. Indeed, the idea that one’s roommate will come home this week is itself a precisely-defined predictive hypothesis! He could arrive after the heat death of the universe, after all.
Though one may not consciously formulate predictions of what a hypothesis entails, one can identify data points that conform to the hypothesis. That is, a hypothesis can make retrodictions, predictions of data we have already access to.
And so, under this thought experiment, the precise hypothesis is that the roommate will move in this week. The found clean room, absent a roommate, is a retrodiction of that hypothesis. The belief that he has moved in is a logical conclusion, based on inference to the best explanation, even if the probability is ambiguous.
The problem with this kind of reasoning is not that it’s generally unreliable, but because it’s vulnerable to error due to humans not knowing the full epistemic space. In Roommate, the hypothesis is not confirmed yet, because we need actual direct evidence that the roommate has moved in and arrived. As we’ll come to see a little later, the data McKay has demonstrated is compatible with alternative hypotheses.
McKay’s Strategy Renders His Point Trivial Or Irrelevant
And so, to the extent that McKay has “proven” that we can conclusively confirm a vague (“flexible”) hypothesis, it’s by closing off epistemic space and turning probabilistic reasoning into logical reasoning. There’s nothing inherently wrong with this strategy. If you’re trying to demonstrate one conclusion is necessarily more logical than another, and thus more rational to affirm it, this is the strategy one must take.
But such efforts render his point trivial. If hypothesis A has an 80% probability of being true, while hypothesis B has only 20% probability, of course you should affirm hypothesis A. That’s just basic math and logic. No one is disagreeing with that point. The disagreements exist over methodological disagreements and disagreements over specific relative probabilities.
The other problem is that, in most philosophical or religious disputes, we don’t know if the epistemic space is closed, and so proposing that epistemic space is closed changes the subject.
For instance, what if hypothesis A is 20% probable, but hypothesis B is 22% probable? It would be unwise to affirm hypothesis B merely because it’s relatively more probable than hypothesis A, especially given its absolute improbability. Admittedly, I don’t think McKay holds that position, which is why he is sure to emphasize the closure of epistemic space. But to make his thought experiment a generalizable principle to miracles, God, science, and philosophy of religion, the closure of epistemic space has to hold, which in many cases it doesn’t. So why propose this solution in the first place?
Possible Alternative Hypotheses
To illustrate why this reasoning is bad, let’s talk about the shortcomings of various probabilistic arguments. In Brian Blais’s book A Measure of Faith: Probability in Religious Thought, Blais points out that probabilistic arguments often fail because of a lack of imagination in constructing alternative explanations. This is exactly what we see in Roommate. I can think of two possible explanations for the retrodictive data that are compatible with the data but affirm that the roommate hasn’t arrived:
Rich Guy. It’s possible that the roommate is Bradley Uppercrust III, son of old money, who has a maid and patrician values. It’s possible that Bradley is a giant piece of work who has his maid pre-emptively and periodically clean his room before he moves into campus. Under this hypothesis he either copied his key or gave it to his maid to enter.
Maintenance. Maintenance enters rooms and moves things around all the time, sometimes even cleaning them, in order to accomplish a maintenance task. It’s possible that your room is clean for this very reason.
You Must Test The Hypotheses
I’ll admit that the Rich Guy hypothesis isn’t terribly likely just because someone like that being your roommate isn’t terribly likely, but the maintenance hypothesis is likely enough to be taken seriously.
In this situation, it’s not unreasonable or illogical to conclude that your roommate has arrived, but if we’re going to formulate this conclusion as a hypothesis, we have to test it.
To test the maintenance hypothesis, let’s assume that each roommate brings their own furniture when moving in. If you come back to your apartment and it’s cleaned, but there’s no furniture or belongings of anyone else in the apartment, the maintenance hypothesis (or something similar) is more likely than the roommate hypothesis.
To test the rich guy hypothesis, if you returned to your cleaned apartment, but looked around and saw additional expensive items like luxurious bags, fancy furniture, and a brand new big screen TV scattered around the room, and still no sign of a roommate, the rich guy hypothesis becomes increasingly more likely relative to the roommate hypothesis.
And finally, you can test the roommate hypothesis to see if he’s there, in the room or on campus. If there’s stereotypically college dorm furniture and a preponderance of personal items that scream “middle America,” you can be confident that your roommate is there, even though there may be other, less-likely-yet-plausible explanations.
All to say, the data you had (that the room was clean) was insufficient to infer a conclusive answer, absent additional data, even though you can infer a likely answer. When you were coming up with an explanation, you were drawing a conclusion, but when you were looking to test the explanation, it became a hypothesis. Let’s stop calling things hypotheses if we’re not actually going to test them.
You Can’t Prove or Demonstrate Something as Impossible or Unlikely Without Measuring It Somehow
McKay goes on to mathematically illustrate how one can conclusively prove his thought experiment using Bayes Theorem, then saying:
“A hypothesis can be conclusively confirmed by observation just in case it’s possible for empirical evidence to support a very high likelihood ratio in its favor. (How high the likelihood ratio must be depends on the prior ratio, italicized above, which quantifies how plausible the hypothesis is independently of the evidence in question.) This is just a technical way of saying that a hypothesis can be conclusively confirmed by observation so long as it makes some observations much, much more likely than alternative hypotheses.”
Once again, the key point of disagreement in this discussion is over the nature of epistemic space and its closure. To know that one hypothesis is much more likely than all others (a prerequisite to being near certain of it), you need to know the probability of all other alternative hypotheses, while also having the data that allows you to eliminate or greatly reduce confidence in all alternative hypotheses.
In this way, if you assume a closed epistemic space with only two explanations, and one is more likely than the other by an order of magnitude, that’s the closest thing to a conclusive proof we can get. As I’ve already demonstrated, this is problematic.
More to this point, sometimes the most likely explanation, given the information available to optimally rational agents, is still wrong. This is why philosophers like Popper proposed that we could never confirm hypotheses, so much as we corroborate or falsify them. We can be confident that a hypothesis/theory is true or false based on the quality and consistency of the evidence, and how predictions of that hypothesis/theory are consistent and repeatable, even though we can never be certain of the hypothesis/theory.
It would appear that McKay disagrees, attempting to blend Bayesian conditional probability and deductive reasoning in a way that just doesn’t work. It reminds me of Sherlock Holmes saying that “When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth.” That is, you don’t need precision in your hypotheses, you just need the best explanation, and you get there by showing other explanations are inferior to the point of impossibility.
The problem with this, though, is that you can’t really prove an alternative hypothesis as impossible or near impossible without using precise terms!
To illustrate this, consider the sentence “It’s impossible for humans to fly at 10,000 feet.” Whether or not it’s true or false is predicated on facts of the matter, and what we mean by “fly.” We need a precise definition of flight to be confident in that sentence. It may be impossible for humans to flap their arms and defy gravity to an altitude of 10,000 feet, but it’s certainly not impossible for humans to pilot or ride in planes or helicopters. Admittedly, we know the former hypothesis intuitively because we are familiar with human anatomy and gravity. But to evaluate the latter hypothesis, we need to know more about air resistance, physics, and if planes are possible. You can’t really get a strong understanding of any of those things without using precise definitions and measurements.
It would be quite silly to say that humans flying to high altitudes was impossible because we’ve erroneously closed off epistemic space and said we conclusively proved it, and this seems to be the tactic McKay is endorsing.
Why This Matters: Theist Cheat Codes
McKay continues on:
“Notably, hypotheses that say a particular person has acted are often both flexible and easily confirmed. They’re flexible because people are complex and unpredictable. Even if you know someone very well, it’s nearly impossible to predict precisely what they’ll do. But hypotheses that say a particular person has acted are often easily confirmed, since they nevertheless make certain observations much, much likelier than alternative hypotheses. The probability that I’d leave my finest watch at the scene of the crime, given that I’m guilty, is vanishingly small. (I am, after all, quite attached to my watch.) But the probability that I’d leave my finest watch at the scene of the crime is far smaller given that I’m innocent.”
Intentionality And Analogous Reasoning
This reasoning illustrates a “cheat code” with theistic hypothesis testing that I first noticed watching Matt Dillahunty and Matthew Adelstein debate earlier this year.6
That is, any hypothesis that proposes an intentional agent with the capability of creating that outcome will always appear more likely than alternatives. If you come home and your bed is made, that’s expected on the hypothesis that your mother made it or that there were bed-making gremlins who made it. Either hypothesis on its surface is more likely than your bed made itself spontaneously.
But we know that the bed-making gremlins aren’t real (because I made them up), so it doesn’t make sense to postulate them as the most likely or plausible hypothesis after we eliminate your mother as a candidate explanation. And so, the fact that the data conforms to this hypothesis shouldn’t be taken as a point in favor of it; rather, we need to examine how plausible such an agent’s existence is in the first place.
In philosophy of religion debates, analogous reasoning is intuitive when trying to infer aspects of God’s character because it feels familiar and compelling, even though we don’t have a firm experience of divine psychology. Going back to Hume and his Dialogues, whether you’re a skeptic or an orthodox believer, God is sufficiently disanalogous to us that analogous reasoning that McKay uses simply falls apart.
I agree that Mckay’s reasoning for Roommate works because we have abundant experience with roommates and the behaviors of individual humans. But we don’t have the same abundant experience of God, at the very least not to the same level of confidence. And so, trying to use probabilities in this manner is a misuse of the tools. We can’t form probabilities of datasets with no data.
This digression on intentionality and analogous reasoning may seem like an unnecessary tangent, but the reason I bring it up is because McKay is trying to nudge you in the direction of accepting this logic without telling you of its shortcomings. If you accept this logic for anticipating the actions of people, why not God?
My point in response is that the reasoning in Roommate is poorly formulated and insufficiently rational, and so we shouldn’t affirm it within the thought experiment, let alone for reasoning about God. And even if the reasoning was sufficiently rational, it would be an inappropriate method to evaluate God hypotheses, because God as an intentional agent would be sufficiently disanalogous to any intentional agent we’re familiar with.
What Are The Numbers For The Theistic Hypothesis?
Though McKay doesn’t do this in his post, it’s worth flagging more sleights of hand theists and Christian apologists engage in when they write posts such as these that tear down precision yet build up the ethos of hypothesis testing.
Namely, when they use this language of probability, hypothesis testing, and likelihood ratios, they don’t use precise measurements for their own hypotheses, and when they do give us these measurements, they are often completely absurd. For example, when Christians are trying to demonstrate the likelihood of the resurrection via Bayes Theorem, they will think they are being charitable to the nonbeliever by assigning a prior probability to Jesus being resurrected as being one in a million. This is so silly, not least of which because it implies that there are more than 7,000 people globally (including over 300 in the United States) who were dead for three days and were bodily resurrected walking around today. A more accurate prior would generously be one in 20 billion.
Anyway, these theists rely heavily on the improbability of non-theistic explanations to carry the weight of their argument of relative “conclusiveness”, hoping you won’t notice the poverty of data for their specific hypotheses and the unsound methods that they came to them. They hope that you will discard scientific hypotheses merely because they appear improbable relative to theistic hypotheses that don’t even pretend to formulate a probability.
And so, theists punish naturalistic hypotheses for attempting to make an explanation with high uncertainty and imperfect data, while at the same time rewarding theistic hypotheses for just existing, even though they lack any quality measurement data. It’s a clear and obvious double standard.
Final Thoughts
At the end of the day, I think McKay’s mistake is that he’s blending the tools of logic and probability to refute ill-defined opponents.
I think his implicit goal is admirable. As far as I can tell, he’s trying to come up with practical reasoning to believe particular theories (with high confidence) over other theories, with limited data and limited time to examine that data.
The good news is that we already have the vocabulary to do this without fancy math or Bayes theorem (and it’s not incompatible with Bayes Theorem). Namely, we can look to the concept of moral certainty, which is the certainty to act, even if there’s the possibility of data revealing itself giving us reasonable doubt in the future. What’s more, we can merely choose to assent to the theories that have affirmative evidence while ignoring those that have none, instead of pretending to evaluate every hypothesis.
Using these methods, in the case of Roommate, one can be morally certain that the roommate has arrived because we have no evidence supporting any other hypothesis. We have moral certainty not because we’ve arbitrarily limited the epistemic space (as McKay does) or because we’ve eliminated every other possible hypothesis, but because no other data is available to invalidate the roommate hypothesis or suggest any alternative.
In this regard, the limits of the data are the limits of the epistemic space. The Rich Guy, Maintenance, or some other hypothesis (maybe there’s a bandit who breaks into apartments and cleans them) could be the case, but until we have evidence that supports them or undermines Roommate, we don’t need to give them any attention. What’s more, we should always be a little inquisitive, looking for data that further confirms these hypotheses. In Roommate, I would look around to see if my roommate was in the apartment, knocking on his door, and so on. I would check my email or phone to see if maintenance came into my room. And so on.
As this relates to evaluating miracles, spectacular religious claims, or scientific claims outside of our experience, we still need precise hypotheses and evaluative metrics because those events are sufficiently unfamiliar to us and analogous reasoning is unreliable. Scientific reasoning that links causal events to measurable variables is more useful as it’s not analogous reasoning, but causal.
And so, much as McKay errs by confusing the language of logic and probability, we will also err if we confuse the reasoning we use for unfamiliar circumstances with the reasoning we use for familiar circumstances.
On a fundamental level, these methodological and epistemic differences are a key divide between theists and nontheists. Theists are more sympathetic to reasoning about God using analogy, while nontheists object that analogous reasoning is invalid for unfamiliar events, opting instead for scientific methods of evaluation. The response by theists like McKay is to cast doubt on nontheistic formulations of the scientific methods and to demonstrate how these familiar methods are compatible with unfamiliar circumstances. As I believe I’ve shown, these attempts fail.
I appreciate Noah for writing his post, because it gave me the opportunity to highlight these important differences, something that had been poking in the back of my brain for a while. Thanks to him!
And again, I understand that this is pedantic! But I wouldn’t be this way if it weren’t relevant.
This is called “operationalization” and I’ll talk about it more in a future post.
I’ll talk about why in a future post.
I love Hume’s illustration of this point when he argues that the universe is more comparable to a vegetable than a designed machine.
To be clear, I don’t think it’s intentional or malicious, this all stems from his original strawman he is rebutting.
Inadvisable watch FWIW
The Roommate thought experiment is literally a locked-room mystery. It’s the exact type of scenario famous for narrowing the epistemic space enough (at least in fiction!) to allow unusually large deductive leaps. Most types of evidence will be much more open-ended, including anything miracle-related.
This perfectly encapsulates the problem I flagged yesterday! Great read.