Misapplying the Newsvendor Problem

The Stitch Fix blog recently had a good intro to the newsvendor problem. For more details and extensions of the basic problem, I’d recommend this paper by Qing et. al.  

I think the newsvendor problem is a very valuable mental model, but I have seen newsvendor-ish thinking misapplied very frequently, particularly in healthcare management. In this post, I’m going to try to explain two common misapplications of the newsvendor framework.

Newsvendor problem setup

But first, let’s recap the setup of the newsvendor problem.

1. Agents: There is one newsvendor, and several customers who show up during the course of the day, each of whom buys one newspaper.

2. Goals: The newsvendor wants to stock newspapers so as to match the daily demand. For example, if 100 people show up, then the newsvendor would like to have a stock of 100 newspapers, no more, no less. 

3. Randomness: However, demand is stochastic and varies from day to day. We’ll assume here that daily demand is drawn from a normal distribution.

4. Stockouts: If a customer arrives and finds that there are no newspapers left in stock, then that customer leaves without getting anything. This is important, we’ll come back to it later.

5. Required solution: How many newspapers should the newsvendor stock?

Solution using overage and underage costs

To solve this problem, the newsvendor first makes two calculations, to the find the “cost of overage” and the “cost of underage”. Here’s what those terms mean:

• Cost of overage (too many newspapers): this is the cost to the newsvendor of each paper that is left unsold. This might be the price that the newsvendor paid to the publisher/printer. Let’s denote this CO.

• Cost of underage (too few newspapers): this is the cost in terms of lost sales for each unit of demand that is unmet. We’ll denote this COU

From these two numbers we’ll calculate a ratio called the critical ratio. The critical ratio is CU/(CU+CO). Note that CU and CO are specific to the particular circumstances that prevail for this particular newsvendor.

The next and final step is to plug the critical ratio into the inverse CDF of the demand distribution. This gives us the optimum number of newspapers that the newsvendor should stock. There we go, problem solved.

As an example, let’s say that overage and underage costs are the same. Then the critical ratio is 0.5. Plug this into the inverse CDF of the demand distribution and we’ll get back the mean demand/50th percentile (assuming normality).

Misapplications of the newsvendor framework

Here are a couple of common misapplications that I see:

1. Defaulting to “magic numbers” such as the 95th percentile

2. Using the newsvendor formulation in cases where the assumptions don’t apply, and therefore focusing on the wrong output.

Let’s look at each in turn.

Misapplication 1: Defaulting to the 95th percentile

It might be a bit inaccurate to call this a misapplication of the newsvendor problem, as it’s more of a failure to apply the framework at all. However, I think it comes from a vague sort of newsvendor-ish thinking.

Let’s imagine a conversation between the Chief Marketing Officer and the Chief Financial Officer of a retail company.

The CMO says, “We want to serve as much demand as possible. We may not be able to hold enough stock to meet the maximum possible demand, but we should go for the 95th percentile of demand.”

This reasoning comes from a good place – the CMO is not just trying to meet average demand, but instead recognizing that there is variability around the average. However, they are only looking at the problem from one side – avoiding underage costs.

The CFO is likely to respond, “Do we really need to stock that much? By definition, the 95th percentile almost never happens. I don’t want to tie up so much working capital in inventory if we don’t need it.”

This is the other side of the problem – avoiding overage costs.

Within the newsvendor framework, the solution is based on the critical ratio that takes into account the underage and overage costs. And there’s no reason why this ratio would always be 0.95, or 0.75, or any other specific number. It depends on the specifics of the situation.

Misapplication 2: Failing to recognize a queueing problem

This is a more serious misapplication. I’ll explain using a specific example: imagine you’re working in a hospital, and you’re trying to find the right capacity for the emergency department (ED) to meet demand.

Here’s how this maps back to the newsvendor setup:

• The number of beds in the ED is analogous to the stock of newspapers that the newsvendor has.

• Just like in the newsvendor problem, you have to decide on the number of beds (the “stock”) before the stochastic demand materializes, and you can’t change capacity during the day to meet daily demand.

So, can you use the newsvendor framework here? I would say no, not in any simple way.

Remember point 4 in the newsvendor setup, the one about what happens if there’s a stockout? Well, it doesn’t apply here. If a patient arrives to the ED and there are no free beds, they aren’t turned away. Instead, they end up waiting until a bed gets freed up. This fundamental property of the system makes it a queueing problem, not a newsvendor problem.

How does this change our recommended solution?

It is almost always true that wait times matter as a key output in queuing problems. As a patient going to the ED I wouldn’t care at all that the number of beds is equal to the Xth percentile of the demand distribution. What I would care about is: “Can I access a bed in a reasonably short amount of time?”

The newsvendor formulation just doesn’t have any way of accounting for wait times. It also fails to highlight one of the most important properties of queueing systems: performance (in terms of wait times) decreases rapidly and nonlinearly as the average utilization rate approaches 100%. Furthermore, it’s important to realize that the rate of performance decline depends on the size of the system and the level of variability. These are all features that are important to account for.

Therefore, for this kind of problem, I would strongly recommend some kind of queueing model, rather than the newsvendor-ish thinking of “just match the 95th percentile of demand”, or even a more informed application using the critical ratio.

I see your AUC and I'm not impressed ... yet

Eric Topol’s paper on AI/ML in healthcare made a big splash when it came out earlier this year. It’s a broad review of published models that have the potential to impact three different stakeholder groups: clinicians, patients, and the healthcare delivery system.

One of the key takeaways of the paper is that although researchers report pretty good AUC scores for models that predict health outcomes (Table 3 of the paper), this doesn’t really mean that the models would have any clinical benefits if deployed. In other words, who cares that you can predict well, if you’re not improving patient well-being? This point also came up in an interview with Marzyeh Ghassemi, a researcher at the University of Toronto.

Topol stresses that prospective trials are necessary to show actual clinical benefit. This means it’s not enough to just get high AUC scores on historical data; instead, someone has to actually deploy the model and then track patient outcomes to see whether they improve significantly after the deployment. To date there have been very few of these prospective trials – Topol calls this the “AI chasm”.

What if we look beyond healthcare – has anyone been doing this kind of prospective trial to evaluate their ML/AI models? Yes, it’s standard practice at Booking.com, as this paper shows.

The authors say:

In Booking.com we are very much concerned with the value a model brings to our customers and our business. Such value is estimated through Randomized Controlled Trials (RCTs) and specific business metrics like conversion, customer service tickets, or cancellations.

They go on to say that in their work there are diminishing returns to model performance (better AUC =/= more $$$), and hypothesize about what might explain this. Except for the uncanny valley effect, I think all of the explanations could apply to ML/AI in healthcare as well.

Complex systems don't necessarily need more black boxes

In the book Prediction Machines, the authors break down the anatomy of a decision task, noting that the following components must come together for an AI system to make accurate predictions: input data, model training, and feedback data. Furthermore, for the AI system to be useful, there must be clear, agreed-upon actions that follow from a prediction, based on judgments about risk. 

Figure 7-1 from Prediction Machines

Perception tasks are a clear example where all these ingredients come together, and this is why we now have AI systems that can usefully diagnose certain eye diseases, for example. 

In contrast, in predicting "social outcomes" or the outcomes of poorly-understood processes, such as recidivism, hospital readmissions, and sepsis, the appropriate actions to take can be very unclear, and any two experts could make different conclusions about the costs and benefits of a given action. 

Generating useful predictions in this kind of context is still very much an open problem. The best we have are "good habits" and rules of thumb. Perhaps one rule of thumb we should add to the list is: "the most accurate models [in a complex social context] aren't necessarily the most un-interpretable/complicated ones … the standard should be changed from the assumption that interpretable models do not exist, to the assumption that they do, until proven otherwise".

This is the argument made in an article in the latest issue of the Harvard Data Science Review. As part of the Explainable AI Challenge at NeurIPS 2018, the authors built an interpretable model for credit scoring, and found that the difference in accuracy between that model and a black box model was negligible. As they say, "The false dichotomy between the accurate black box and the not-so accurate transparent model has gone too far."