On the medical-school admissions algorithm reform.

The post-affirmative-action reform of US medical-school admissions algorithms since the 2023 ruling has produced a class of changes that read as substantial in the trade press and are likely to look thin once the data arrives. The schools running the most ambitious reforms are getting the most coverage, and they should expect the smallest measurable effect. The schools that leave their algorithms mostly intact and change two upstream variables should see the largest. That predicted shape is the inverse of what most observers expect, and it is the second variable that explains it.

The most-covered reforms are the algorithm rewrites. The post-ruling conversation has treated the moment as an opportunity to overhaul the weighting of admissions rubrics: lower weight on standardized test scores, higher weight on letters of recommendation, higher weight on a structured interview score, higher weight on socioeconomic-context factors that the legal reading of the ruling permits as proxies. The rewrites are defensible. They are also being thoroughly relitigated in public, with each school's choices treated as a referendum on what diversity in medical education should mean. The trade press loves them.

What the trade press is not focusing on is that the rewrites are likely to produce small effect sizes. There is no outcome data yet. The ruling landed a month into the application cycle for the class entering this fall, and that class has not matriculated. But a reweighted rubric run on the same applicant pool should produce entering classes that are, on the dimensions that matter most to those schools, only marginally different from what the old algorithm would have produced. A rewrite moves a few percentage points. It does not move the distribution.

The schools that move the distribution can keep the algorithm largely intact and change two upstream variables. The first upstream variable is the candidate pool: aggressive recruiting in geographic and institutional contexts where the school has been under-recruiting for a decade, paired with funded-application waivers that remove the cost barrier from applicants for whom the cost is actually a barrier. The second upstream variable is the rubric for what counts as relevant clinical experience, broadened from "clinical experience that maps to a traditional pre-med pathway" to "clinical experience that maps to any service-delivery context where the candidate spent enough hours to know what care delivery actually looks like." That second variable is the one that will do the work.

Broadening the clinical-experience rubric matters for a structural reason that is not specific to medical school. Selection algorithms, when run on populations with similar inputs, produce outputs distributed similarly to the inputs. The output diversity is upper-bounded by the input diversity. Rewriting the algorithm without changing the inputs produces modest effect sizes. Changing the inputs without rewriting the algorithm produces a different distribution of outputs because the algorithm now has different things to choose between. The candidates who come through community-health programs and rural-hospital aide pipelines and EMS routes and pharmacy-tech roles are already there. They get screened out at the rubric step and never reach the algorithm. Letting them reach the algorithm is the change that matters.

The part that compounds is that the algorithm has little to do with the result. The result is decided three steps upstream, in the office where someone defines what "relevant clinical experience" means. A school measuring pipeline candidates against a relevance-of-experience filter calibrated to a population that did not include them will keep missing them, however its weights are tuned. Recalibrate the filter and the algorithm will do the rest of the work without complaint. The upstream office has been holding the lever the whole time.

The early takeaway, before the first post-ruling class has even arrived, is that the algorithm-reform discourse is looking at the wrong layer. The layer that matters is the upstream filter that decides which candidates the algorithm gets to evaluate. The schools that focus on the algorithm will get the press cycle. The schools that focus on the upstream filter will get the result.

There is a cleaner version of this lesson outside admissions. Any selection process that runs on a homogeneous input distribution will produce a homogeneous output distribution, regardless of how the selection process is tuned. Tuning the process is the visible work. Diversifying the input is the work that compounds. The medical-school admissions reform is going to be a case study for this distinction for the next decade, and the case study will read better the longer the data accumulates.

—TJ