A method for open-ended research cases

An open case has no right answer, which is what makes candidates flounder. It is scored on structure: whether you clarify the objective, define what you are predicting, enumerate the data, state assumptions aloud, start from a baseline, and know how you would be proved wrong. Follow a fixed order and the content takes care of itself.

Step	The question you answer	What it prevents
1. Objective	What decision does this support, and what does "good" mean?	Solving the wrong problem elegantly
2. Target	What exactly am I predicting, at what horizon, for whom?	A vague label that cannot be evaluated
3. Data	What would I want, what exists, and when was each knowable?	Look-ahead built into the design
4. Assumptions	What am I taking for granted, and which matter most?	Being caught out by a premise you never noticed
5. Baseline	What is the simplest thing that could work?	Being unable to say whether the clever model helped
6. Improvement	What does the baseline miss, and what fixes that?	Model choice as a reflex rather than a reason
7. Validation	How would I know this works out of sample?	A number nobody can trust
8. Failure modes	What would make me abandon this?	Sounding like an advocate rather than a researcher

Table 1.1 · The order to work in. The last row is the one candidates skip, and it is the one interviewers remember. Volunteering how your own idea could fail is the strongest single move in a case.

Proposition 1.2

Define the target variable out loud

Most cases are lost in step two. "Predict demand" is not a target; "predict the number of bike hires per station per hour, one hour ahead, for a dispatcher deciding where to move bikes" is. The horizon, the unit of observation and the consumer of the forecast all change the model, and naming them takes fifteen seconds.

Holds when

Ask what the prediction will be used for. A forecast that feeds a threshold decision needs calibration, not accuracy.
Classification or regression is a consequence of that answer, not a preference.
If the target is a return, say over what horizon and net of what costs.

Why the baseline is not a formality. Interviewers ask for a baseline to see whether you know what your model must beat. "Yesterday’s value" is a formidable competitor for most time series, the seasonal mean beats a great deal of machine learning on demand data, and a single well-chosen feature in a linear model often beats an ensemble on financial returns. Starting with the baseline also gives you something to deploy while the clever thing is still being argued about, which is the answer to "what would you ship first?".

Figure 1.3 · Where a forty-five minute case goes. A suggested budget, and the block worth defending is the baseline. Candidates skip it to get to the interesting model and then have nothing to compare against — which makes every later claim unfalsifiable and is the single commonest way a case goes wrong.

A method for open-ended research cases

After this lesson you should be able to

Define the target variable out loud

The rest of this lesson is in Premium