Inadequate Equilibria: Where and How Civilizations Get Stuck
Eliezer Yudkowsky
Only Eliezer Yudkowsky could write an entire book about how great he is for buying some lightbulbs for his girlfriend. Just kidding…sort of.
Seriously, I think this book has a really interesting premise, but isn’t executed well. Yudkowsky starts from a generalization of the efficient markets principle: to paraphrase, if something can be done that is valuable and cost-effective, someone else will have already figured out how to do it. Taken to the extreme, however, this idea would result in the conclusion that you shouldn’t really bother doing anything. We know, based on unscientific observation, that people do achieve good things. So given these two points that are apparently in tension, how can we discern when there may be opportunities that are both worthwhile and feasible? This is a very important question for a person trying to plan their life.
A lot of this book is spent going over the Econ 201 concepts that often explain why a thing appears to work suboptimally: principal-agent problems, asymmetric information, and collective action problems. I don’t think Yudkowsky adds much to the reams that have already been written on these topics, but it’s a reasonable enough treatment. Basically in this section, Yudkowsky establishes that we can often easily identify “inadequate” situations that are nonetheless not exploitable–the inadequacy doesn’t come from people being dumb or irrational, just from unfortunate information and/or incentive structures. In this part, Yudkowsky talks a lot about how he concluded that the Bank of Japan’s monetary policy was flawed, which was later borne out by a policy change that improved things. I’m willing to believe he actually did reach a correct conclusion, but it’s sort of odd to me that he spends so much time talking about it. Yudkowsky is a big proponent of “rationality-as-winning,” and I can hardly think of something that matters less, in terms of winning, than a non-Japanese non-central-banker’s view on Japanese monetary policy.
The other anecdote Yudkowsky talks a lot about, as I mentioned above, is his purchase of lightbulbs for his girlfriend. I’ll spell it out a little more, since I was being a little cheeky before. As Yudkowsky recounts it, his girlfriend suffered severely from seasonal affective disorder, and the standard light-box treatments weren’t working. He got the bright idea (ha) to just buy a ton more lightbulbs to increase the amount of light, and lo and behold, his girlfriend was cured. His conclusion is basically, I couldn’t find any academic studies about this, but I’m not super surprised because academia is full of bad incentives–so when you get an idea like this that seems so obvious it must have been tried, don’t be too quick to assume that.
I actually think this is good advice, although not necessarily reached for the right reasons. One thing Yudkowsky doesn’t mention at all is the placebo effect, which could easily explain this “prescription” working in an individual case while not being provable in a scientific sense. (Heck, it could even be that his girlfriend was so touched by her boyfriend coming up with crazy schemes just to try to help her, that it helped to cure her.) Another is simply that sometimes things that don’t work on average in statistically detectable ways can still work for individuals! Individual outcomes are determined by whole hosts of interacting factors that we can’t understand fully. But these don’t invalidate the point–they’re just further evidence that, even if you don’t think you’re able to come up with generally applicable innovative ideas, it’s still worth trying different stuff in individual cases.
A related and very topical discussion arose recently around face mask effectiveness, and whether public health bodies should recommend their use as protection from SARS-CoV2. As you may recall, there were initially clear communications in the US that in general, people should not wear masks; only later was this reversed to the current state where masks are generally recommended and indeed required in many public places. Scott Alexander had an extensive discussion of this issue on his blog (https://slatestarcodex.com/2020/03/23...). His conclusion is basically that the initial non-recommendation of masks was based on the evidence for their efficacy not reaching a sufficiently scientific burden of proof–“not proven effective beyond a reasonable doubt."
I think these discussions add up to some good advice that Yudkowsky sort of communicates in the book, but that I think could have been outlined more clearly. When it comes to individual judgments, a rational person should be weighing evidence in a Bayesian way, such that our determination of the “right” course of action always comes down to a degree of belief. This is a significantly different standard than is used in most of our society’s information-generating institutions–science and medicine may pose stricter rules that filter out more false positives but also some true positives; conversely, academic publishing may reward p-hacking, which results in a lot of non-replicable results being published. Or on the other hand, journalism works based on representativeness and availability. So it’s not responsible for us to outsource our decisionmaking entirely to any of these information-generating institutions, although we should of course consider their outputs in our own Bayesian decision process. And the corollary is that there may be opportunities for achievement that have not been ratified by our society’s information-generating structures. We can begin to identify these by thinking about the biases inherent in these structures.
We should also impose a heavy filter on “what to try” based on cost–it’s obviously not a good idea to start taking unproven medications based on the above reasoning, because they could have huge and/or irreversible drawbacks. However, something like “exposing myself to more lightbulbs” seems pretty low-cost and easily reversible. Yudkowsky does communicate this conclusion well–just try stuff if it’s low-cost, and don’t worry too much about whether you are second-guessing the (generalized) market.
The back end of the book seems like a lot of inside baseball relating to the specific circles Yudkowsky interacts with. The discussion focuses on the responsible use of “the outside view”–for those not familiar with the concept, taking the outside view on a problem means trying to abstract from your individual circumstances and reason by analogy with a related class of instances, whereas taking the inside view means trying to reason through the specifics of your circumstances. The relevant literature identifies that using the outside view can reduce bias, for example helping us to avoid the “planning fallacy” where we assume a project we are working on will go smoothly because we imagine the best possible outcome, rather than thinking of things that could go wrong–which will more naturally come to mind if we think about similar things that happened in the past.
Basically Yudkowsky argues that people use the outside view excessively, resulting in excessive modesty and suboptimal achievement. He cites, for example, conversations with startup founders who use the outside view to estimate their potential market rather than thinking about how it could be a thousand times bigger, and thus end up not shooting for anything more ambitious than others have already achieved. I think this must be a very specific problem with people in the circles Yudkowsky runs in, because I rarely if ever hear people reasoning based on the outside view. (To be fair, only a very specific subset of people are going to read an Eliezer Yudkowsky book either, and he points this out.) It’s true that the outside view has a serious drawback, in that it’s impossible to definitively identify the correct reference class. But I’ve never understood the outside view to be taken as a unique planning tool; rather, it’s useful to get a “second opinion” on whether you’re being too optimistic about something. It also shouldn’t prevent you for trying to achieve something greater than your reference class; it should just dissuade you from making plans that will have very bad outcomes if you don’t (for example, overleveraging a company)–which I think remains good advice. I feel like the people Yudkowsky is talking about are using the outside view in an idiosyncratically bad way.
Ultimately, I would boil down the message of this book to: think for yourself, and be willing to try unorthodox stuff if the cost is pretty low. Both very sound pieces of advice, just reached in what felt like a roundabout way.