CS 507 – Statistical Learning Assignment 2 Fall 2019 Assignment 2: Gaussians and linear regression 1. (15 points) Mathematical preliminaries: Manipulating Gaussian densities Consider the func- tion f(x) = 1Z exp(ax 2 + bx+ c), with a < 0, and the Gaussian density g(x) = 1√ 2piσ2 exp ( − 12 ( x−µ σ )2) (a) Find the maximum of log(f(x)) as well its curvature (i.e. second derivative) at the maximum. (b) Show that, for appropriate choices of a, b, c, and Z, f(x) is equivalent to g(x). [Hint: Multiply out the square term in the exponent of g(x), and equate the coefficients of f(x) and g(x).] Express µ and σ in terms of a, b, and c. 2. (25 points) Linear regression [Python]. Suppose you are given training-inputs D = {x1, . . . ,xN}, where each xi is an M × 1-dimensional vector, and outputs {y1, . . . , yN} You want to fit a function of the form f(x, ω) = ω>x = ∑M i=1 wixi to this data by minimising the cost function L(ω,D) = N∑ n=1 (f(xn, ω)− yn)2 + λ ( ω>ω ) (a) By taking the derivatives of L with respect to each ωi and setting them to 0, show that this cost function is minimised for ωλ = (∑ n xnxn > + λIM )−1 ( ∑ n xnyn), where I is an identify matrix of size M . (b) Load the variables in the file Assignment2.mat and fit the parameters to the data xTrain and yTrain (where each row of xTrain corresponds to one data-point), using λ = 0, i.e. no regularisa- tion. Plot the vector ωo that you obtain. (c) Train multiple versions of your model on the training-set, using values λ = 1, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000. Make a plot that shows how the least-squares errors of your model on the training and the test set changes as a function of λ. [Note: Please make clearly labelled plots with meaningful axes.] (d) Using this plot, decide which value of λ you expect to give you the best generalization performance, and report the value. Use this model to predict the y-values for the data xValidation, save your predictions as a variable named yValidationPredicted in a file Assignment2YourName.mat and submit this file. (e) Your friendly instructors included a column of ’ones’ for the input data (i.e. the first entry of each x is 1). Explain why (for this model), including a constant term in the input-data can be useful and results in a more flexibel regression model. (f) By inspecting the vector ω obtained for the best model, make a guess as to which of the dimensions of x are important for predicting y, and which are irrelevant.
