IHME projections are absurdly optimistic

The state-specific projections from IHME have been widely disseminated and used for federal, state, and hospital planning. For example, here’s an interview with one of the UW professors on the project describing how their projections are being used by the feds. They have done yeoman’s work in collecting appropriate data, but their actual model has some serious flaws.

The details of the model itself can’t be found on their projection webpage; you have to go the medarxiv article. Their introduction gives a brief criticism of the SIR model that I’ve been using as a base for my estimates. In particular, SIR models generally don’t include the effect of changes in social behavior in response to the epidemic, they assume panmixia (random mixing among the population) and generally model using reported case numbers, which are distorted by variance in testing rates and testing strategies.

These criticisms are all pretty reasonable. It’s a good idea to look at the effect of controls (i.e., reducing R_0), which many simple models haven’t done. Panmixia isn’t accurate, and likely overestimates the speed of the epidemic, especially when the modeling is done on the scale of the whole country. And death rates are likely more accurate than case numbers, due to limited testing.

Let’s see what they’re suggesting as an alternative. Here’s how they’re doing the projections of total deaths in each state:

  • Gather state-specific data on imposed controls, hospital capacity and utilization, local age structure and death rates.
  • Estimate death rates for infection after correcting for local age structure.
  • Code controls from level 0-4 for # of the following: school closures, non-essential business closures (incl. bars/restaurants), stay-at-home recommendations, and travel restrictions.
  • Model the impact of controls on maximum death rate and time to reach the inflection point using published data from Wuhan, scaled based on the coded controls for each state, with Wuhan = 4.
  • Fit cumulative death rate (i.e., deaths per capita) to a Gaussian error function with three parameters: a location-specific inflection time where deaths stop accelerating, a location-specific growth rate, and a global (age-normalized) maximum death rate. Either the inflection time or the maximum death rate was assumed to depend on the delay between initial deaths and imposition of controls.
  • Assume that states without full controls impose full controls a week after the current date.
  • Project out the fits, incorporating variance in the parameter fits to generate a set of projections.

There are some really serious problems with this model, and all of them push the results towards underestimating the problem in the US. Let’s go through the problems.

Unqualified acceptance of CCP data

The data for deaths in Wuhan come from the provincial Hubei government, straight from the Chinese Communist Party. There are very good reasons to be skeptical of these numbers. The CCP lies constantly, and it is very hard for outsiders to verify these numbers. However, there is technical data that suggests strongly that the CCP is undercounting. A late-Feburary analysis looks at activity of crematory facilities in Wuhan as well as temporarily-leaked numbers on Tencent. This data suggests a 5-10 fold higher rate of deaths than was reported by the CCP in late February. There have also been recent reports of very high numbers of funerary urns needed in Wuhan, in excess of what would be required by the officially reported deaths.

This is a huge problem for the model, as it is highly parameterized by the assumed ending of the epidemic in Wuhan in short time, by the imposition of social distancing controls.

Undercounting of deaths elsewhere

There is good reason to believe that official COVID19 deaths are undercounted in most areas. In particular, total deaths per month for several hard-hit Italian towns went up 3-5 times compared to the same week last year, with only a fraction of that excess officially counted as COVID19 deaths.

In this model, current data on deaths in other polities are used to fit total deaths in the future, so a significant undercounting of current deaths would undercount the future projections.

Assumption of control imposition

They assume that states that do not have infection controls impose full controls within a week. This is unlikely at best.

Bad model choices

They choose to model the cumulative death rate as a Gaussian error function based on, AFAICT, no theoretical reason, only that it fit “the data” better. Given that the only data in their set that has run to completion is Wuhan’s, and that the Wuhan dataset is compromised, this is an extremely foolish decision.

Further, this model assumes that the acceleration of deaths as the epidemic grows is the same as the projected deceleration of deaths after infection controls are imposed. There is absolutely no reason to expect this. As the acceleration is related to R_0, this is equivalent to saying that R_0 before controls is equal to 1/R_{ctrl}. Even if controls are effective at bringing the growth rate negative, there is no reason why the deceleration can’t be much slower than the initial growth rate (e.g., if R_0 before controls is ~3 and after controls is ~0.95).

A better approach would be a modified SIR model using European, US, and non-CCP Asian data to estimate the impact of various infection control measures.

Overestimation of control effectiveness

They assume that state-wide infection controls are equivalent in effect with the controls the CCP imposed in Wuhan.

This is crazy. I was watching media reports and social media in January and February keeping an eye on what was happening in Wuhan. I saw officials welding apartment doors shut, abducting individuals with fevers from their homes or off the street, mass-disinfecting whole cities, setting up walls outside of villages to keep the infection out, people not wearing facemasks being harassed and assaulted, running internal checkpoints to identify people with fevers.

I live in an area with fairly strong infection controls by US standards. We are nowhere close to that level of infection control. I went out for the first time in two weeks to the grocery store to stock up. There was one other person wearing a face mask, and they were wearing it wrong (not covering their nose). I repeatedly had to dodge back out of aisles to avoid people coming straight at me. This assumption is ridiculous and crazy, and obviously so to anyone paying attention. They’re in Washington state, they should know better.

Chinese reports suggest that the infection controls imposed in Wuhan dropped the R_0 from ~3.5 to ~0.35, or about 10-fold. Note that they also include a period of partial controls, which in their case means:

  • Blocking outward transportation from Wuhan
  • Closing public transit and vehicular traffic inside Wuhan
  • Compulsory mask-wearing in public places
  • Cancellation of public events
  • Self-quarantine of confirmed or suspected cases

They estimate that during this period, R_0 dropped to ~1.25. For comparison, the full controls included as well:

  • Full quarantine of confirmed or suspected cases (i.e., extraction to a separate quarantine site), including contacts of confirmed cases
  • Temperature monitoring of all residents
  • Universal and strict stay-at-home orders for all residents

My expectation is that the current set of controls imposed by most European and US State governments is closer to the partial Wuhan controls than the full controls, and I fully expect growth rates to remain positive, though slower.

Conclusion

There are at least 5 serious, obvious problems with the IHME model, most seriously uncritical acceptance of Communist government data and wild overestimation of the effectiveness of US infection control measures. Given that I know from personal communication that multiple hospitals are using this to project out expectations for medical care required – and taking as valid that peak care will be required for only a matter of weeks – these reports are hugely irresponsible.

Further, the fact that all the problems I identified with their model cause it to be biased towards underestimating the scope of the problem, I suspect there are serious prejudices or biases in the people generating the model. Frankly, I think they’re projecting wishful thinking and looking for reasons to support optimism.

Join the Conversation

36 Comments

    1. Pretty much, that’s what I did, yeah. The trickiest part is figuring out what the effect of the infection controls are, but even that’s not too bad. They criticize SIR models for various reasons, and those reasons are valid, which is why they chose a crazy approach instead, I suppose.

      Like

  1. Despite your alarmist reactions, it turns our as today, April 7 in New York, IHME predicted a range of hospital beds required at 25,482, with a range between approximalely 14,000 to 40,000. On Cuomo’s slide today, the number of hospitalizations is 17,493 — off by about a third from the projection. So maybe the model isn’t overly optimistic, at least a far as NY is concerned — perhaps it’s in fact highly pessimistic.

    Or maybe the IHME error band is so large as to useless

    It’s frustrating that the IHME does not overlay actual numbers against their projections across all geographies. I don’t know of any such comparison that exists out there, and I am certainly not going to through the tedious process of doing so. for all their numbers. But somebody ought to.

    Like

      1. To the degree it’s validly discredited by this post, it would be in the opposite direction to skepticism, so I feel like something was lost in translation somewhere there

        Like

        1. What I tell you three times is true. The only criticism above that has been mooted is the over-reliance on Chinese numbers, since there’s now data from a few other areas in Europe that have pushed the growth rate negative. They’re still using an absurd curve-fitting approach that is overly optimistic about the tail of the epidemic, they’re still too optimistic about the effect of US/EU restrictions on the growth rate, they’re still undercounting deaths in high-impacted areas, and they’re still making rosy assumptions about control measures that still haven’t been imposed nation-wide.

          Like

          1. Their forecasts for early US states (e.g., Washington) actually look pretty solid so far. You didn’t put numbers on your criticism, but the exasperation makes it seem like you would not have expected hospitalizations to fall across the US over the past week or so like they have? If the peak is now, as it appears to be based on net new hospitalizations, then is your main concern at this point that there will be a much longer tail than IMHE predicts?

            Like

  2. @AR –

    Latest in COVID denial is this:

    https://www.nejm.org/doi/full/10.1056/NEJMc2009316

    You can read it better than I, so I won’t waste time explaining it to you.

    Would appreciate feedback.

    Question. Why can’t we take number of COVID deaths and, working with R-naught, estimate numbers of infected? We know the first, are reasonably sure of the 2nd.

    Like

    1. Yeah, saw that one. Here’s the breakdown: 215 women gave birth over the course of 2 weeks at a particular hospital in NY, and all but one of them were tested on admission for COVID-19 with qtPCR from nasopharyngeal swabs. The backstory is that this same hospital had two mothers who developed symptoms after delivery and then tested positive, so they decided to do full screening of all mothers.

      Super useful, since that means you’ve got a situation like the Diamond Princess, where you’ve got a fairly arbitrary sample of people that you’re testing fully. Probably better because on the cruise ship the individual patients are correlated, while the pregnant women are more like a random sample of NYC. Now, the downside is that pregnant women are very unusual immunologically, especially right up around childbirth, so you might be seeing results that aren’t representative of other individuals. But still super useful.

      Alright, so those 214 admission tests they report:
      – 181 tests were negative. One patient that initially tested negative developed symptoms post-delivery, and tested positive 3d after the initial test.
      – 33 tests were positive. Of those, 4 were symptomatic (febrile) on admission. 3 more were asymptomatic on admission but then developed a fever during their postpartum stay (of ~2d).

      Just on the data they present, 8/34 = 23.5% of positive-testing women had symptoms, about twice that of what’s implied by the pie chart (4/33=12.1%), and what I presume folks are focusing on to minimize the problem. it’s just that half of the symptomatic women developed symptoms in the ~2 days after delivery but weren’t showing symptoms on admission. Since with an exponentially-growing epidemic, most cases are recent, I’d put strong odds that more of the putatively asymptomatic cases developed symptoms post-discharge. All you’d need is someone who had just been infected long enough to be detectable at admission and lasted 2 days without developing a fever, then got sick after she was sent home.

      Since what we really care about is how many people are symptomatic/asymptomatic over the course of the disease before recovering and no longer being infectious, I’d very much like to see a followup in a couple of weeks where they check on the final status of all these women. As is, it’s reporting by and large the symptomatic rate in the ~ first few days after infection.

      Like

      1. “and what I presume folks are focusing on to minimize the problem. ”

        I think what they are focusing on is this. Assuming (and it’s a big fat assumption) that this random sample is representative of NYC’s infection rate at present: approx 14%.

        Deaths have now been revised a bit upwards to count people who died at home:

        It’s a bit more than 10K now.

        10K/13% of NYC’s population…. well you get the point.

        “It’s just flu, bro!”

        Like

    1. I get 0.81% with the most recent numbers. 34/214=15.9% of the population infected based on these numbers. NYC population is ~8.4 million. So that estimates us at ~1.33m infected, with 10,834 deaths as of the revision, for a death rate per estimated infection of 0.81%.

      It’s pretty back of the envelope, but it’s not terrible. Key points here are:
      1) When were the patients tested?
      2) How long does it take an infection to go from first exposure to death?

      For the first we’ll dig. There’s three papers from this group:
      – a case study of 7 infections in pregnant women (written March 25)
      – a study of 43 confirmed infections in pregnant women (written April 2)
      – the study you linked, of 214 tested women undergoing delivery (written April 13)

      Those cover different dates. For the first, the two critical patients, both of whom delivered, were admitted 3/18 and 3/19. The other five were undated, and it’s not clear whether they delivered or not.

      The second study covers March 13-27, with 43 confirmed infections, and 18 deliveries. 14 presented without symptoms, 4 were symptomatic. 2 of them are the two detailed case studies from the previous paper, and the other 12 were after March 22, when universal testing was instituted. For the 4 deliveries that presented with symptoms, it’s not clear whether they were before or after March 22. My guess is that it was after, since the first case study doesn’t mention any deliveries.

      The final study covers March 22 to April 4, all deliveries after universal testing. We know that of that group 181 were negative on admission, with 1 of those later testing positive. 33 were positive on admission, 4 of whom were symptomatic on admission, 3 of whom developed symptoms while on the unit.

      We can use the data from the first two to split the data up into March 22-27 and March 28-April 4. There were 12 positive cases in the first 6 days and 20 (one of whom was negative on admission) in the second 8 days. Of the symptomatic positives, the last report says there were 4 symptomatic on admission, all of whom must have been in the first 6 days. Unfortunately, there’s no data to split the total deliveries, so we’ll make the assumption that the daily delivery rate was constant over the period, at 92 during the first period, for 12/92=13.0% positive and 122 in the second, for 20/122=16.4% positive. That’s useful! It suggests that the infection rate is growing over time. It’s ~3wks after the first groups admission and ~2wks after the second group’s, so the current positive rate in NYC is probably >20% or so now.

      Now we need to figure out how long it takes from infection to death. If the infection rate is ~20%, then nearly half of those folks will have been infected in the last two weeks and won’t have had time to progress to critical or dead by now. Let’s say you have to have caught it at least three weeks ago to have died. In that case, we’d use a lower infection rate, since we’d only want to calculate the death rate of those who were infected 3 wks ago, not those who are infected now. So we’ll take a 13% rate, or ~0.99% of those infected >= 3 weeks ago have been identified as dying of COVID. Seems reasonable for having missed some deaths and not actually run out of oxygen for patients yet.

      Like

      1. Greg Cochran suggested that that hospital has a lot of Hasidic women; I added that it prob has a lot of black & Hispanic women — in NYC we don’t have a lot of what you might call white middle class mainstream people… we tend to be either some kind of ethnic group or very rich (who would live on the Upper East Side and not go to that hospital).

        Still, Columbia/Pres is a huge, topflight medical institution that serves what passes for the general population in NYC.

        Which includes a lot of A-A’s, Hispanics, and Hasidic Jews, all of which are overrepped in infections & deaths.

        If we knocked the overall infection rate down to 10% – we’d be right around 1% CFR, which is what pretty much every other country is getting (apart from the whopper outliers). That 1% is a nagging repetitive number.

        Not flu.

        Like

        1. Paper two has some details on the population:

          The majority of women were obese, with a BMI ≥30 (n=26, 60.5%). The mean BMI for the cohort was 30.9 ± 5.3 kg/m2 and two women (4.7%) had a BMI of 40 or greater. Eighteen women (41.8%) had an additional comorbid condition, with mild-intermittent asthma (n=8, 18.6%) representing the most common co-morbidity. Other comorbid conditions included type 2 diabetes mellitus (n=3, 7.0%) and chronic hypertension (n=3, 7.0%). Patients mostly resided in the Bronx (n=21, 48.8%) or upper Manhattan (n=19, 44.2%), and one was from out of state.

          Like

          1. Thanks. From those data we can almost certainly conclude they were mostly black/Hispanic. Hasidic women tend to be normal weight. (Not the men.)

            Like

            1. The letter is dated April 13 & refers to “Between March 22 and April 4, 2020, a total of 215 pregnant women delivered infants at the New York–Presbyterian Allen Hospital and Columbia University Irving Medical Center . ”

              The pre-proof study talks about “a series of 43 test-confirmed cases of COVID-19 presenting to a pair of affiliated New York City hospitals over two weeks from March 13 to 27, 2020.”

              Is this the same population?

              Like

                1. Also – no reason to believe that the larger population from the letter isn’t from Bronx and Upper Manhattan. That’s the catchment area, although hospitals aren’t schools and anyone can go there.

                  I can assure you that if you live on the Upper East Side, the moms aren’t obese and they go to Lenox Hill or Weill Cornell to give birth.

                  Like

          2. Being obese has nothing to do with susceptibility to the virus. Being from Upper Manhattan and the Bronx does.

            Like

  3. 5/4/20 – IMHE must be reading your blog – they’ve drastically upped their numbers. Today it’s 135K deaths. From something like 73K a day or two ago. WTF?

    Like

Leave a comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create your website at WordPress.com
Get started
%d bloggers like this: