I should probably have said the change in life expectancy is a weighted average, weighted by how far you are from the average. If average life expectancy is 80, removing a data point where somebody died at 40 has 8x the effect of removing a data point where somebody died at 75.
(To reproduce exactly the scenario being discussed, you fit a constant-only model to the data using least squares: that gives the average as the best fit. Then, you measure the leverage of each point of interest.)