ST3189 - Linear Regression Quiz

1. In the context of linear regression, what does the term 'least squares' refer to?

Minimizing the number of predictors. Minimizing the sum of the squared differences between the observed and predicted values. Minimizing the absolute differences between the observed and predicted values. Maximizing the correlation between predictors.

2. What is the primary assumption about the relationship between predictors and the response in a simple linear regression model?

The relationship is exponential. The relationship is approximately linear. There is no relationship. The relationship is logarithmic.

3. In the linear model \(Y = \beta_0 + \beta_1 X + \epsilon\), what does \(\beta_1\) represent?

The intercept, or the value of Y when X is zero. The error term. The average increase in Y associated with a one-unit increase in X. The predicted value of Y.

4. The Residual Sum of Squares (RSS) is defined as:

\(\sum_{i=1}^n (y_i - \hat{y}_i)^2\) \(\sum_{i=1}^n |y_i - \hat{y}_i|\) \(\frac{1}{n}\sum_{i=1}^n (y_i - \hat{y}_i)^2\) \(\sqrt{\sum_{i=1}^n (y_i - \hat{y}_i)^2}\)

5. What does an R-squared (\(R^2\)) value of 0.75 signify?

75% of the data points are correct. The model is incorrect 75% of the time. 75% of the variability in the response variable is explained by the predictor variables. The correlation between the predictor and response is 0.75.

6. In multiple linear regression, what is the null hypothesis for the F-statistic?

All regression coefficients are equal to 1. At least one regression coefficient is non-zero. All regression coefficients are zero. The model fits the data perfectly.

7. What problem does multicollinearity refer to?

A non-linear relationship between predictors and response. The presence of outliers in the data. High correlation between two or more predictor variables. The error terms are correlated.

8. Which of the following is a potential solution for dealing with non-linearity in a linear regression model?

Removing outliers. Applying a transformation to the response or predictor variables (e.g., log, square root). Increasing the sample size. Using weighted least squares.

9. What is an interaction effect in a regression model?

When the effect of one predictor on the response depends on the value of another predictor. When two predictors are highly correlated. When the response variable is categorical. When the error terms have non-constant variance.

10. What is the purpose of a prediction interval?

To estimate the range where the true population regression line lies. To estimate the range where a single future observation will fall. To estimate the range of the regression coefficients. To estimate the variance of the error terms.

11. In the context of linear regression, what is a 'loss function'?

A function that measures the amount of information lost by using a subset of predictors. A function that describes how much accuracy is lost by using a linear model instead of a non-linear one. A function that quantifies the difference between the true and predicted values, which the model aims to minimize. A function that measures the financial cost of making an incorrect prediction.

12. What is the primary goal of the 'least squares' approach in linear modelling?

To find the model that passes through the most data points. To find the values of the model parameters (e.g., \(w_0, w_1\)) that minimize the average loss across the dataset. To ensure the model's predictions are always positive. To maximize the number of parameters in the model.

13. In vector/matrix notation for linear regression, \(y = Xw + \epsilon\), what does the matrix \(X\) represent?

A column vector of the target values. A column vector of the model weights. The design matrix, containing the input features for all observations. A diagonal matrix of error variances.

14. What is the primary difference between a parametric and a non-parametric statistical learning approach?

Parametric methods are only for regression, while non-parametric are for classification. Parametric methods make an assumption about the functional form of f, while non-parametric methods do not. Parametric methods are always more accurate. Non-parametric methods are easier to interpret.

15. What is the main disadvantage of a very flexible (complex) statistical learning model?

They are always less accurate on the training data. They are harder to fit. They can lead to overfitting, where the model follows the noise in the training data too closely. They are always difficult to interpret.

16. The mean squared error (MSE) is composed of which two quantities?

The model's accuracy and the sample size. The bias of the model squared and the variance of the model. The training error and the test error. The reducible error and the irreducible error.

17. What is the 'bias-variance trade-off'?

The trade-off between using a biased or an unbiased estimator. The fact that as model flexibility increases, its variance tends to increase and its bias tends to decrease. The trade-off between the error on the training data and the error on the test data. The choice between a parametric and a non-parametric model.

18. In the context of linear regression, what is the purpose of the 't-statistic'?

To measure the total variance in the response. To measure the goodness-of-fit of the model. To test the null hypothesis that a particular regression coefficient is zero. To calculate the R-squared value.

19. What is a 'dummy variable' used for in linear regression?

To represent a placeholder for missing data. To represent a qualitative (categorical) predictor with two or more levels. To represent an interaction term. To represent a non-linear transformation of a predictor.

20. What does the 'Residual Standard Error' (RSE) measure?

The average amount that the response will deviate from the true regression line. The standard deviation of the predictor variables. The total sum of squares (TSS). The proportion of variance explained by the model.

21. What is the primary purpose of using subset selection methods like best subset selection or forward stepwise selection?

To handle non-linear relationships. To identify and fit a model using only a subset of the available predictors, improving interpretability and potentially prediction accuracy. To deal with correlated error terms. To increase the R-squared of the model.

22. Which of the following is a drawback of best subset selection?

It cannot be used for linear regression. It is computationally intensive and becomes infeasible for a large number of predictors (p). It always results in a model with high bias. It can only select one variable at a time.

23. How does forward stepwise selection work?

It starts with all predictors and removes them one by one. It tries every possible combination of predictors. It starts with no predictors and adds them one by one, at each step adding the predictor that gives the greatest additional improvement to the fit. It randomly selects a subset of predictors.

24. What is the main idea behind shrinkage methods like Ridge and Lasso regression?

To remove outliers from the dataset. To fit a linear model with a penalty that shrinks the magnitude of the coefficient estimates towards zero. To transform the predictors to have a linear relationship with the response. To select the best subset of predictors.

25. What is the key difference between the penalty used in Ridge regression and Lasso regression?

Ridge uses an L1 penalty (\(\lambda\sum|\beta_j|\)), while Lasso uses an L2 penalty (\(\lambda\sum\beta_j^2\)). Ridge can shrink coefficients to exactly zero, while Lasso cannot. Ridge uses an L2 penalty (\(\lambda\sum\beta_j^2\)), while Lasso uses an L1 penalty (\(\lambda\sum|\beta_j|\)). There is no difference in the penalty term.

26. A key advantage of Lasso regression over Ridge regression is that:

It is computationally faster. It can produce 'sparse' models by forcing some coefficient estimates to be exactly zero. It always has a lower test error. It can handle non-linear relationships better.

27. In Bayesian linear regression, what is the role of the prior distribution?

It represents the distribution of the error terms. It is another name for the likelihood function. It represents our uncertainty about the model parameters *before* observing the data. It is the final distribution of the parameters after seeing the data.

28. The posterior distribution in Bayesian inference is proportional to:

The prior distribution divided by the likelihood. The likelihood minus the prior distribution. The likelihood times the prior distribution. The sum of the likelihood and the prior distribution.

29. Ridge regression can be viewed as a Bayesian linear regression model with what kind of prior on the coefficients?

A Uniform prior. A Laplace (double-exponential) prior. A Gaussian (Normal) prior centered at zero. No prior (equivalent to least squares).

30. Lasso regression is equivalent to finding the MAP estimate under which prior distribution for the coefficients?

A Gaussian (Normal) prior. A Laplace (double-exponential) prior. A Bernoulli prior. A Uniform prior.

31. What is a 'confidence interval' in frequentist inference?

An interval that has a 95% probability of containing the true parameter value. An interval that contains 95% of the data points. An interval which, if the experiment were repeated many times, would contain the true parameter value in 95% of the repetitions. An interval representing the range of the predicted values.

32. What is a 'credible interval' in Bayesian inference?

An interval that is more believable than a confidence interval. An interval that has a certain probability (e.g., 95%) of containing the true parameter value, given the data. An interval that contains 95% of the posterior predictions. The same as a confidence interval, just with a different name.

33. What is the primary purpose of using basis functions (e.g., polynomials, splines) in a linear model?

To reduce the number of predictors. To make the model easier to interpret. To allow the linear model to capture non-linear relationships. To ensure the error terms are normally distributed.

34. What is a 'spline' in the context of regression?

A method for smoothing noisy data. A type of non-parametric regression. A function defined by piecewise polynomials, joined together at points called knots. A tool for visualizing high-dimensional data.

35. What is a key advantage of using a natural cubic spline over a regular cubic spline?

It is more flexible. It has more knots. It is constrained to be linear at the boundaries, which can result in more stable and realistic estimates at the edges of the data range. It is easier to compute.

36. What is the main idea behind Generalized Additive Models (GAMs)?

They generalize linear models to handle any type of response variable. They allow for non-linear functions of each predictor to be fitted, while maintaining an additive structure. They are a type of neural network. They are used exclusively for unsupervised learning.

37. What is a potential problem with fitting a linear regression model when the error terms \(\epsilon_i\) are correlated?

The coefficient estimates will be biased. The model will overfit the data. The estimated standard errors for the coefficients will be incorrect, leading to unreliable confidence intervals and hypothesis tests. The R-squared value will be artificially low.

38. What is an 'outlier' in a regression context?

A data point with an unusual predictor (x) value. A data point whose response (y) value is far from the value predicted by the model. Any data point that is removed from the analysis. A data point that lies on the regression line.

39. What is a 'high leverage' point?

A point with a large residual. A point that has a large effect on the R-squared value. A data point with an unusual predictor (x) value. A point that is always an outlier.

40. The Gauss-Markov theorem states that under certain assumptions, the least squares estimator is:

The most computationally efficient estimator. The only unbiased estimator. The Best Linear Unbiased Estimator (BLUE), meaning it has the minimum variance among all linear unbiased estimators. Always normally distributed.

41. What does 'homoscedasticity' mean in the context of linear regression?

The error terms are normally distributed. The variance of the error terms is constant across all levels of the predictor variables. The predictors are uncorrelated with each other. The relationship between predictors and response is linear.

42. What is the primary purpose of examining residual plots in regression diagnostics?

To check for multicollinearity. To assess the significance of the overall model. To identify potential problems with the model assumptions, such as non-linearity and heteroscedasticity. To calculate the R-squared value.

43. If you fit a multiple linear regression model and the p-value for the F-statistic is very small, but the p-values for individual coefficients are large, what is a likely cause?

The model is a very poor fit for the data. There is significant multicollinearity among the predictors. The sample size is too small. The error terms are not normally distributed.

44. What is the 'curse of dimensionality'?

The fact that models become harder to interpret as the number of dimensions increases. The phenomenon where, in high-dimensional spaces, data becomes very sparse, and methods that work well in low dimensions, like k-NN, can perform poorly. The computational cost of fitting models with many dimensions. The difficulty of visualizing data in more than three dimensions.

45. In the formula for the least squares estimators \(\hat{\beta} = (X^T X)^{-1}X^T y\), what must be true about the matrix \(X^T X\)?

It must be a diagonal matrix. It must be invertible. It must be symmetric. It must have a determinant of 1.

46. Which of these is NOT a standard assumption of the classical linear regression model?

The relationship between X and Y is linear. The error terms are normally distributed. The error terms have constant variance (homoscedasticity). The predictor variables (X) are normally distributed.

47. What is the primary difference between a confidence interval and a prediction interval for a given value of X?

There is no difference. A confidence interval is for the mean response, while a prediction interval is for an individual response. A prediction interval is always narrower than a confidence interval. A confidence interval is used for classification, while a prediction interval is for regression.

48. What does the 'Adjusted R-squared' do that the regular R-squared does not?

It adjusts for the number of observations in the dataset. It adjusts for the number of predictors in the model, applying a penalty for adding predictors that do not improve the model. It is always higher than the regular R-squared. It can be used for non-linear models.

49. Which method is considered a 'greedy' approach to subset selection?

Best subset selection Ridge regression Forward and Backward stepwise selection Lasso regression

50. The elastic net penalty is a combination of which two other penalties?

Best subset and forward selection. Ridge (L2) and Lasso (L1). Polynomial and spline. AIC and BIC.