The least squares criterion is a mathematical rule for choosing the “best” curve (most commonly a straight line) to represent a set of observed data. It chooses the curve that makes the sum of the squared vertical distances (residuals) between the observed points and the curve as small as possible. In practice this produces the familiar “line of best fit” used in regression analysis.
Understanding the Least Squares Criterion
– Basic idea: For observed pairs (xi, yi), a candidate model y = f(x; θ) (θ = model parameters) yields residuals ri = yi − f(xi; θ). Least squares selects θ to minimize S(θ) = Σi ri^2.
– Why square residuals? Squaring penalizes larger deviations more heavily and ensures all contributions are nonnegative, producing a single scalar objective to minimize.
– Linear vs. nonlinear least squares:
• Linear (ordinary least squares, OLS): f is linear in parameters (e.g., y = a + b x). Closed-form solutions exist for parameter estimates.
• Nonlinear least squares: f is nonlinear in parameters (e.g., exponential, logistic). Iterative numerical methods (Gauss–Newton, Levenberg–Marquardt) are typically required.
– Relationship to statistics: Under the common assumption that errors are independent and normally distributed with constant variance, ordinary least squares estimates coincide with maximum-likelihood estimates.
Common Uses of Least Squares
– Trend estimation and forecasting (economics, finance)
– Estimating sensitivity or beta (finance: regression of asset returns on a market index)
– Curve fitting in science and engineering (calibration, signal processing)
– Time-series modeling and residual analysis
– Option and risk-model calibration in quantitative finance
– Basis for many machine-learning methods (linear regression, generalized linear models)
Key Takeaways
– Least squares finds parameters that minimize the sum of squared residuals; OLS is the simplest and most used version.
– OLS has a closed-form solution and convenient properties under standard statistical assumptions (unbiasedness, minimum variance in the class of linear unbiased estimators).
– Diagnostics (residual plots, R^2, standard errors) are essential to check model validity.
– If assumptions are violated (heteroskedasticity, outliers, nonlinearity), consider weighted least squares, robust regression, or nonlinear methods.
What Does Least Squares Tell You?
– Coefficient estimates: the fitted slope and intercept (or other parameters) that best explain variation in y given x under the least-squares criterion.
– Goodness-of-fit metrics: R^2 (fraction of variance explained), standard error of regression, and residual standard deviation quantify how well the model fits.
– Residual pattern: structure in residuals (autocorrelation, heteroskedasticity, nonlinearity) indicates model inadequacy and suggests model improvements.
Practical steps — Linear (Ordinary) Least Squares (manual / conceptual)
1. Plot data: Scatter xi vs yi to check linearity and outliers.
2. Compute sample means x̄ and ȳ.
3. Compute slope b and intercept a using the OLS formulas:
• b = Σ (xi − x̄)(yi − ȳ) / Σ (xi − x̄)^2
• a = ȳ − b x̄
4. Compute fitted values ŷi = a + b xi and residuals ri = yi − ŷi.
5. Assess fit:
• R^2 = 1 − (Σ ri^2 / Σ (yi − ȳ)^2)
• Standard error of estimate = sqrt(Σ ri^2 / (n − p)), where p = number of parameters estimated
6. Diagnose:
• Plot residuals vs fitted values to check for nonlinearity or changing variance.
• Plot residuals vs time (if time series) to check autocorrelation.
• Use statistical tests (Breusch–Pagan for heteroskedasticity, Durbin–Watson for autocorrelation) as needed.
7. If assumptions fail, consider transformations, weighted least squares, adding predictors, or robust regression.
Practical steps — Nonlinear Least Squares (conceptual)
1. Choose a parametric form f(x; θ) suggested by theory or exploratory plots.
2. Provide initial guesses for θ (important for convergence).
3. Use an iterative solver (Gauss–Newton, Levenberg–Marquardt, or trust-region methods) to minimize Σ (yi − f(xi; θ))^2.
4. Inspect residuals and parameter standard errors (from approximate Hessian or Jacobian).
5. If fit is poor or solver fails to converge, try different starting values, reparameterize, or change model.
Implementations — Quick practical notes
– Excel: Use the built-in trendline (chart), LINEST function, or Solver for nonlinear fits.
– Python:
• Linear: numpy.polyfit(x, y, 1) or statsmodels.api.OLS for detailed output (standard errors, t-stats).
• Nonlinear: scipy.optimize.curve_fit or least_squares.
– R: lm(y ~ x) for OLS; nls() for nonlinear least squares.
– Always examine diagnostic output (confidence intervals, p-values, residual plots).
Worked mini-example (linear)
Data: (x, y) = {(1,2), (2,3), (3,5)}
– x̄ = 2, ȳ = 10/3 ≈ 3.333
– Numerator Σ(xi − x̄)(yi − ȳ) = (−1)(−1.333) + 0*(−0.333) + 1*(1.667) = 1.333 + 0 + 1.667 = 3.0
– Denominator Σ(xi − x̄)^2 = 1 + 0 + 1 = 2
– b = 3.0 / 2 = 1.5 ; a = ȳ − b x̄ = 3.333 − 1.5*2 = 0.333
– Fitted line: ŷ = 0.333 + 1.5 x
Limitations and common extensions
– Outliers: Squared residuals give outliers high influence. Robust regression (e.g., Huber, least absolute deviations) reduces sensitivity.
– Heteroskedasticity (non-constant variance): Use weighted least squares or heteroskedasticity-consistent standard errors.
– Multicollinearity: In multiple regression, highly correlated predictors inflate variance of estimated coefficients; consider principal components or regularization (ridge, lasso).
– Overfitting: With many parameters relative to data, use cross-validation or penalized methods.
– Non-normal errors: Parameter estimates are still least-squares-optimal for minimizing squared error, but inferential results based on normality may not hold.
When to prefer nonlinear or more advanced methods
– When the relationship suggested by theory is nonlinear in parameters.
– When residual analysis reveals structured departures (e.g., curvature) from linearity.
– When measurement error is present in predictors (errors-in-variables models).
– When you need regularization to guard against overfitting.
Further reading and sources
– Investopedia — Least Squares:
– For algorithms and theory: texts on regression and numerical optimization (e.g., standard econometrics and numerical methods references)
– Produce code examples for Excel, Python, or R to fit and diagnose an OLS model.
– Walk through a larger worked example (with real dataset) and show diagnostic plots and interpretation.