Key takeaways
– VIF measures how much the variance (i.e., the standard error squared) of an estimated regression coefficient is inflated because of linear correlation with the other independent variables.
– VIF = 1/(1 − R_i^2), where R_i^2 is the R-squared from regressing the i-th predictor on the remaining predictors.
– Low VIF values mean little or no multicollinearity; high values show predictors are redundant and coefficient estimates are unstable.
– Common rules of thumb: VIF < 3 (or 10 indicates serious multicollinearity. Use domain knowledge, not just cutoffs.
– Remedies include removing or combining variables, principal components/PLS, ridge or Lasso regression, centering interaction terms, or collecting more data.
What is VIF?
A Variance Inflation Factor (VIF) quantifies how much the variance of a regression coefficient is increased because that predictor is linearly related to the other predictors in the model. VIF is used to detect multicollinearity — a situation where two or more independent variables convey overlapping information (are correlated). Multicollinearity inflates coefficient standard errors, making hypothesis tests and interpretation unreliable.
The formula and interpretation
– Formula: VIF_i = 1 / (1 − R_i^2)
• R_i^2 is the R-squared from regressing predictor X_i on all the other predictors.
– Equivalent metric: Tolerance_i = 1 − R_i^2 = 1 / VIF_i.
– Interpretation:
• VIF_i = 1: X_i is uncorrelated with the other predictors (no multicollinearity for that variable).
• VIF_i between 1 and ~3 (or 5): usually acceptable.
• VIF_i > 10: widely considered to indicate severe multicollinearity (rules of thumb vary; use context).
• VIF_i → ∞: occurs when R_i^2 → 1 (perfect linear dependence).
What VIF can and cannot tell you
– It tells you which predictors are highly collinear with others and roughly how much the coefficient variance is inflated.
– It does not tell you which variable causes the outcome to change (no causal inference).
– It measures linear multicollinearity only — non-linear relations or measurement error need other checks.
– Multicollinearity detected by VIF harms interpretability of coefficients but often does not reduce predictive accuracy for held-out data (although it can if coefficients vary widely).
Example (simple numeric)
– Suppose you regress X1 on X2 and X3 and obtain R_1^2 = 0.60. Then VIF_1 = 1/(1−0.60) = 2.5. Interpretation: the variance of the coefficient estimate for X1 is 2.5 times what it would be if X1 were uncorrelated with X2 and X3.
– If X1 were perfectly correlated with some combination of the other predictors (R_1^2 = 1), VIF_1 would be infinite — the model is not estimable without removing or re-specifying variables.
Practical workflow: Diagnose and address multicollinearity
Step 1 — Initial checks
1. Inspect the pairwise correlation matrix among predictors. High pairwise correlations (e.g., > 0.8 or 0.9) are an immediate red flag.
2. Fit your regression model and compute VIFs for all predictors.
3. Optionally compute the condition number/index and examine eigenvalues of the predictors’ correlation matrix for more general multicollinearity detection (condition index > 30 suggests problems).
Step 2 — Interpret VIFs
1. If all VIFs are low (e.g., 5 or > 10), identify which predictors are implicated and consider next steps.
Step 3 — Resolve problematic multicollinearity (options)
Use domain knowledge first — pick the option that preserves the hypothesis you wish to test.
1. Remove redundant variables
• If two variables measure nearly the same construct, drop one (or combine them).
2. Combine variables
• Create an index or composite (e.g., average of related measures) to capture the shared construct.
3. Re-specify the model
• Try alternative functional forms, interaction terms, or hierarchical modeling.
4. Orthogonalize or center variables
• Mean-centering reduces multicollinearity involving interaction or polynomial terms (it does not change multicollinearity among original predictors, but can greatly help with interactions).
5. Use dimensionality reduction
• Principal components regression (PCR) or partial least squares (PLS) create uncorrelated components used as predictors.
6. Use regularization
• Ridge regression (L2) shrinks coefficients and stabilizes estimates in presence of multicollinearity. Lasso (L1) can select variables.
7. Collect more data
• More observations that break existing correlation patterns can reduce sampling-driven multicollinearity.
8. Keep the model but change interpretation
• If the goal is prediction, multicollinearity may be tolerable; if the goal is interpreting individual coefficients, you must address it.
Step 4 — Re-evaluate
After applying a remedy, recalculate VIFs and check coefficient stability, standard errors, and model fit. Iterate until results align with your analytical goals.
Computing VIF: practical recipes
– Python (statsmodels):
• Build an N x k design matrix (include constant if desired).
• For each column j, compute variance_inflation_factor(exog, j) from statsmodels.stats.outliers_influence.
• Example (conceptual):
• from statsmodels.stats.outliers_influence import variance_inflation_factor
• import statsmodels.api as sm
• X = sm.add_constant(X) # if using intercept
• vif = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
– R:
• Fit lm_model <- lm(y ~ x1 + x2 + x3, data=…)
• Use car package: library(car); vif(lm_model)
– Excel / manual:
• For each predictor X_i: regress X_i on all other predictors, get R_i^2, then compute VIF_i = 1/(1 − R_i^2).
• (Many users prefer statistical software for convenience.)
Additional tests for multicollinearity
– Tolerance (1/VIF).
– Pairwise correlation matrix (simple but limited).
– Condition number / condition index (based on eigenvalues of X'X): higher values indicate near-linear dependencies.
– Eigenvalues of correlation matrix: near-zero eigenvalues indicate collinearity.
Special notes and tips
– Dummy variable trap: including all category dummies plus an intercept causes perfect multicollinearity — drop one category or omit intercept.
– Centering: subtracting the mean (centering) reduces multicollinearity between polynomial or interaction terms and their constituent variables, but does not change linear dependence among untransformed predictors.
– Interpretation: a variable with high VIF can still be important for prediction; the issue is with coefficient precision and interpretability.
– No hard universal cutoffs: the commonly used thresholds (3, 5, 10) are rules of thumb. Use them as guidance combined with domain knowledge.
Checklist to act on high VIFs
1. Confirm the VIFs are driven by meaningful collinearity (not coding error).
2. Check pairwise correlations among flagged predictors.
3. Decide whether your goal is interpretation (fix multicollinearity) or prediction (regularize or keep).
4. Try removing or combining variables tied to the same conceptual construct.
5. If you need to keep all variables but want stable estimates, consider ridge regression or PCR/PLS.
6. Recalculate VIFs and test stability and predictive performance (cross-validation, out-of-sample error).
Bottom line
VIF is a simple, practical metric to quantify linear multicollinearity in multiple regression. It helps identify predictors that inflate coefficient variances and complicate interpretation. Use it as part of a broader diagnostic process (correlations, condition indices, domain knowledge) and choose remedies that align with your inferential or predictive goals — e.g., remove or combine variables, use dimensionality reduction, or apply regularization.
Sources
– Investopedia: “Variance Inflation Factor (VIF)” — Yurle Villegas. (Source material used to construct this explanation.)
– iSixSigma: “Variance Inflation Factor (VIF)” — practical notes on VIF interpretation and remedies.
Editor’s note: The following topics are reserved for upcoming updates and will be expanded with detailed examples and datasets.