ST3189 - Linear Regression Quiz (Part 2)

51. Why is it important to standardize predictors before fitting Ridge or Lasso models?

To ensure the intercept is correctly estimated. To ensure the penalty is applied fairly to all coefficients, regardless of their original scale. It is not important; the result is the same either way. To make the design matrix invertible.

52. In the context of the bias-variance trade-off, a very complex model (e.g., a high-degree polynomial) is likely to have:

Low bias and high variance. Low bias and low variance. High bias and low variance. High bias and high variance.

53. What does a VIF (Variance Inflation Factor) value of 1 indicate for a predictor?

The predictor is perfectly correlated with another predictor. The predictor is not correlated with any other predictors. The predictor is highly influential. The variance of the predictor's coefficient is inflated by 100%.

54. When using polynomial regression, what is a major risk of choosing a very high degree for the polynomial?

The model will be too biased. The model will be difficult to interpret. The model will overfit the data, showing strange behavior, especially at the boundaries. The R-squared will decrease.

55. What is the primary advantage of using cross-validation to select the tuning parameter \(\lambda\) in Ridge or Lasso regression?

It is computationally faster than using a single validation set. It provides a more robust estimate of the test error, making the choice of \(\lambda\) less dependent on the specific split of data into training and test sets. It guarantees that the chosen model will have no bias. It always selects the simplest possible model.

56. If a residual plot (residuals vs. fitted values) shows a distinct U-shape, what does this suggest?

The presence of multicollinearity. The presence of non-linearity in the data that the model is not capturing. The error terms are correlated. The variance of the error terms is not constant (heteroscedasticity).

57. Comparing model selection criteria, which of the following tends to select the most parsimonious (simplest) model?

Adjusted R-squared AIC (Akaike Information Criterion) BIC (Bayesian Information Criterion) Mallows's Cp

58. In Bayesian linear regression, what is a 'conjugate prior'?

A prior that is difficult to compute. A prior that results in a posterior distribution that has the same functional form as the prior. A prior that is uninformative. A prior that is based on previous experimental results.

59. What is a potential issue with using p-values from hypothesis tests for variable selection?

P-values are not valid for linear regression. They do not account for model complexity. When performing many tests (one for each variable), the probability of making a Type I error (falsely rejecting a true null hypothesis) increases. P-values can only be used for simple linear regression.

60. How does the degrees of freedom of a regression spline relate to its flexibility?

Degrees of freedom are unrelated to flexibility. A higher number of degrees of freedom results in a more flexible, wigglier curve. A higher number of degrees of freedom results in a less flexible, smoother curve. Degrees of freedom only apply to polynomial regression.

61. What is Cook's distance used to measure?

The collinearity between two predictors. The overall fit of the regression model. The influence of a single data point on the regression coefficients. The non-linearity of the data.

62. What is the difference between an outlier and a high-leverage point?

There is no difference; they are the same. An outlier has a large residual, while a high-leverage point has an unusual predictor value. An outlier has an unusual predictor value, while a high-leverage point has a large residual. High-leverage points are always outliers.

63. What is the purpose of the 'hat matrix' H in linear regression?

It transforms the response values Y into the coefficient estimates \(\hat{\beta}\). It is used to calculate the R-squared value. It transforms the response values Y into the fitted values \(\hat{Y}\) via the equation \(\hat{Y} = HY\). It contains the p-values for the hypothesis tests.

64. In a simple linear regression, the leverage of an observation \(x_i\) increases as:

\(x_i\) moves closer to the mean of X. The sample size n increases. \(x_i\) moves farther from the mean of X. The residual for \(x_i\) increases.

65. What is a key advantage of Principal Components Regression (PCR) over standard least squares?

It produces a more interpretable model. It can handle situations with high multicollinearity by using a smaller number of uncorrelated principal components as predictors. It always results in a higher R-squared value. It can be used for categorical response variables.

66. What is a potential disadvantage of Principal Components Regression (PCR)?

The principal components are chosen in an unsupervised way, so the components with the highest variance may not be the ones most strongly related to the response. It is more computationally expensive than best subset selection. It cannot be used when p > n. It does not perform any variable selection.

67. How does Partial Least Squares (PLS) differ from Principal Components Regression (PCR)?

PLS is a shrinkage method, while PCR is a subset selection method. PLS is a supervised dimension reduction technique that uses the response Y to help find the new predictors. PLS can only be used for classification. There is no significant difference.

68. What is meant by a 'studentized residual'?

A residual that has been divided by the mean of the residuals. A residual that has been divided by its estimated standard error, which can vary from point to point. A residual that is calculated from a model fit to a subset of the data. The absolute value of the residual.

69. If you have a categorical predictor with 4 levels, how many dummy variables should you include in the regression model?

4 3 2 1

70. What is a confounding variable?

A variable that interacts with another predictor. A variable that is associated with both the predictor and the response, potentially creating a spurious relationship. A variable that has a non-linear relationship with the response. A variable that is not included in the model.

71. In the context of GAMs, what does the 'additive' assumption mean?

The model can only be used for predictors that are added together. The effect of each predictor on the response is independent of the other predictors. The response variable must be on an additive scale. The error terms must sum to zero.

72. What is the main benefit of the elastic net penalty over the lasso penalty?

It is more sparse, selecting fewer variables. It is better at handling groups of highly correlated predictors, often selecting them together. It is computationally simpler. It does not require a tuning parameter.

73. A 95% confidence interval for a coefficient \(\beta_1\) is found to be [-0.1, 0.8]. What is the correct conclusion at the 5% significance level?

The coefficient is statistically significant. The coefficient is not statistically significant. The model is a poor fit. The true effect of the predictor is 0.

74. What does it mean if the error terms in a time series regression are positively autocorrelated?

The model is biased. The variance of the errors is increasing over time. A positive error at one time point is likely to be followed by another positive error at the next time point. The model is a perfect fit.

75. In Bayesian inference, what is a 'non-informative' or 'vague' prior?

A prior that contains no information and cannot be used. A prior that is intended to have minimal influence on the posterior distribution, letting the data speak for itself. A prior that is uniform over all possible values. A prior that leads to a biased posterior.

76. What is the primary motivation for using a log transformation on the response variable Y?

To handle categorical predictors. When the relationship between X and Y is multiplicative, or when the variance of the errors increases with the mean of Y. To increase the R-squared value. To simplify the interpretation of the coefficients.

77. If the goal of modeling is pure prediction, which of the following is generally more important?

Having an interpretable model. Having a low test MSE (Mean Squared Error). Having statistically significant coefficients. Having a high training R-squared.

78. What is the 'irreducible error' in the context of a statistical learning model?

The error that can be reduced by using a more flexible model. The error caused by a biased model. The random error inherent in the data-generating process, which cannot be eliminated by any model. The error on the training data.

79. In backward stepwise selection, what is the starting point?

The null model with only an intercept. A model with a random subset of predictors. The full model containing all available predictors. A model with only the most significant predictor.

80. What is a major limitation of using traditional subset selection methods (best, forward, backward) when the number of predictors p is very large, especially when p > n?

They are guaranteed to overfit. The initial least squares fit for the full model may not even be possible. They become computationally too simple. They tend to have high bias.

81. The shape of the constraint region for Ridge regression is a ________, while for Lasso it is a ________.

Square; Circle Diamond (Rhombus); Circle Circle; Diamond (Rhombus) Triangle; Square

82. In a Bayesian context, the posterior predictive distribution represents:

The distribution of the parameters. The distribution of new, unseen data points, averaged over the posterior distribution of the parameters. The prior distribution updated with the data. The distribution of the training data.

83. What is the main purpose of a Q-Q (quantile-quantile) plot of the residuals?

To check for heteroscedasticity. To check if the residuals are normally distributed. To check for multicollinearity. To check for influential points.

84. If the true relationship between X and Y is \(Y = X^2\), and you fit a simple linear model \(Y = \beta_0 + \beta_1 X\), the model will have:

High variance and low bias. Low variance and low bias. High bias and low variance. High variance and high bias.

85. Which of the following is NOT a method for dealing with heteroscedasticity?

Transforming the response variable (e.g., using log(Y)). Using weighted least squares. Using heteroscedasticity-consistent standard errors (robust standard errors). Using Variance Inflation Factors (VIFs).

86. In a regression model with an interaction term between a quantitative predictor X and a binary dummy variable D, what does the coefficient of the interaction term (\(\beta_3\) in \(Y = \beta_0 + \beta_1 X + \beta_2 D + \beta_3 XD\)) represent?

The intercept for the group where D=1. The difference in the slope of X between the group where D=1 and the group where D=0. The main effect of the dummy variable. The overall slope of X.

87. What is the main drawback of k-fold cross-validation?

It produces a biased estimate of the test error. It is computationally expensive as it requires fitting the model k times. It can only be used for linear models. It does not provide an estimate of the variance of the test error.

88. Leave-one-out cross-validation (LOOCV) is a special case of k-fold cross-validation where:

k = 1 k = 2 k = n (the number of observations) k = p (the number of predictors)

89. Compared to 5-fold or 10-fold CV, LOOCV generally has:

Lower bias but higher variance. Higher bias but lower variance. Lower bias and lower variance. Higher bias and higher variance.

90. What is the primary risk of data dredging or p-hacking?

Finding spurious relationships and reporting them as statistically significant. Using models that are too complex. Failing to check model assumptions. Using insufficient data.

91. In a Bayesian setting, if the prior and posterior distributions are very different, what does this imply?

The model is incorrect. The prior was a poor choice. The data provided a lot of information to update our initial beliefs. The computation has failed.

92. What is the main difference between prediction and inference?

Prediction uses machine learning, while inference uses statistics. Prediction aims to estimate Y for new values of X, while inference aims to understand the relationship between X and Y. Prediction is easier than inference. There is no difference.

93. A key assumption of linear regression is that the predictors are...

Normally distributed. Independent of each other. Considered fixed and are measured without error. Categorical.

94. If the tuning parameter \(\lambda\) in Ridge regression is set to 0, the resulting model is equivalent to:

The null model (intercept only). The least squares regression model. The Lasso regression model. Principal Components Regression.

95. As the tuning parameter \(\lambda\) in Ridge regression approaches infinity, what happens to the coefficient estimates?

They approach the least squares estimates. They all approach zero. They all approach infinity. Some coefficients are set to exactly zero, while others are not.

96. Which statement about the relationship between training error and test error is true?

Training error is always a good estimate of test error. Training error is typically lower than test error. Training error is typically higher than test error. Training error and test error are unrelated.

97. A simple linear regression model is fit to a dataset where the true relationship is not linear. The training RSS will be ________ and the test RSS will be ________.

low; high high; high low; low high; low

98. A very flexible model is fit to a dataset. The training RSS will be ________ and the test RSS will be ________.

very low; high high; high very low; low high; low

99. What is the main advantage of using a smoothing spline over a polynomial or regression spline?

It is more interpretable. It avoids the need to choose the number and location of knots by using a knot at every data point and regularizing the fit. It is always linear. It is computationally less expensive.

100. In the context of Bayesian inference, what does MCMC stand for?

Mean-Centered Model Comparison Markov Chain Monte Carlo Multiple-Choice Model Criterion Minimum Chi-squared Monte Carlo