Inf2B Coursework (Ver. 0.9) Submission due: 4pm, Friday 3rd April 2020 Hiroshi Shimodaira 1 Outline The coursework consists of two tasks, Task 1 – data analysis and classification with multivariate Gaussian classifiers, Task 2 – neural networks. You are required to submit (i) two reports, one for each Task, (ii) code, and (iii) results of exper- iments if specified, using the electronic submission command. Details are given in the corresponding task sections below. Some of the code and results of experiments submitted will be checked with an automated marking system in the DICE computing environment, so that it is essential that you follow the syntax of function and file format specified. No marks will be given if it does not meet the specifications. Some helper tools to check your files and function template files will be provided. Please check the following coursework web-page frequently to see any updates. https://www.inf.ed.ac.uk/teaching/courses/inf2b/coursework/cwk.html Efficiency of code and programming style (e.g. comments, indentation, and variable names) count. Those pieces of code that do not run or that do not finish in approximately five minutes on a standard DICE machine will not be marked. This coursework is out of 100 marks and forms 25% of your final Inf2b grade. This coursework is individual coursework – group work is forbidden. You should work alone to complete the coursework. You are not allowed to show any written materials, the data provided to you, results of your experiments, or code to anyone else. This includes posting your coursework to the internet and making it accessible to other people not only during the coursework period, but also after that. Never copy-and-paste material of other people (including those available on the internet) into your coursework and edit it. You can, however, use the code provided in the lecture notes, slides, and labs of this course, excluding some functions described later. High-level discussion that is not directly related to this coursework is fine. Please note that assessed work is subject to University regulations on academic misconduct: http://web.inf.ed.ac.uk/infweb/admin/policies/academic-misconduct For late coursework and extension requests, see the page: http://web.inf.ed.ac.uk/infweb/student-services/ ito/admin/coursework-projects/late-coursework-extension-requests Note that any extension request must be made to the ITO, and not to the lecturer. Programming: Write code in Matlab(R2018a)/Octave or Python(version 3.6)+Numpy+Scipy+Matplolib. Your code should run on standard DICE machines without the need of any additional software. There are some functions that you should write the code by yourself rather than using those of standard libraries available. See section 4 for details. This document assumes programming in Matlab. For Python, put all the specified functions into a single file for each Task, so that task1.py for Task 1, and task2.py for Task 2. Output data should be stored in Matlab’s MAT binary format. 2 Data 2.1 Data for Task 1 The coursework employs the Anuran Calls (MFCCs) Data Set introduced by J. Colonna etal..1 Your data set file, ‘dset.mat’, which is a subset of the original data set, should be found in your coursework data directory (denoted as YourDataDir hereafter) : 1 https://doi.org/10.1007/978-3-319-46307-0_13 1 3 Task specifications 2 /afs/inf.ed.ac.uk/group/teaching/inf2b/cwk/d/UUN/ where UUN denotes your UUN (DICE login name). You can use Matlab’s load() function to load the data set in the following manner: load(pathname); where pathname denotes the absolute pathname of your data set file. Once you load the data set, you will find the following variables. Matlab variable (Class) Description X[N, D] (double)) feature vectors Y family[N,1] (int32) family class labels Y genus[N,1] (int32) genus class labels Y species[N,1] (int32) species class labels list family[4,1] (cell) family class names list genus[8,1] (cell) genus class names list species[10,1] (cell) species class names where N and D denotes the total number of samples and the dimension of feature vector (D=24), respectively. Among the three different levels of taxonomic rank provided in the original data set, we use ’species’ in the coursework. There are ten different species, so that the number of classes for classification is ten, i.e., C = 10. The variable, Y species(i), contains the integer number that corresponds to the species of i-th sample, whose feature vector is X(i,:). Hereafter, Y denotes Y species. The variable, list species, holds the list of species names. The following table shows the number of samples for each species in the original data set, which may be different from the number samples in your data set. Family Genus Species # of samples Leptodactylidae Leptodyctylus Leptodactylus fuscus 222 Adenomera Adenomera andreae 496 Adenomera hylaedactyla 3049 Hylidae Dendropsophus Hyla minuta 229 Scinax Scinax ruber 96 Osteocephalus Osteocephalus oophagus 96 Hypsiboas Hypsiboas cinerascens 429 Hypsiboas cordobae 702 Bufonidae Rhinella Rhinella granulosa 135 Dendrobatidae Ameerega Ameerega trivittata 544 The data set has not been split in two sets for training and testing. You need to split the data set according to the instructions described later. 2.2 Data for Task 2 The data for Task 2 is stored in the plain-text file named ‘task2 data.txt’ in YourDataDir. For details, see the Task 2 specifications. 3 Task specifications Task1 – Anuran-Call analysis and classification [50 marks] Task 1.1 [5 marks] (a) Write a Matlab function task1 1() that • calculates the covariance matrix, S and correlation matrix, R, for the whole data set X, using the maximum likelihood estimation (MLE), • saves S as ‘t1 S.mat’, 3 Task specifications 3 • saves R as ‘t1 R.mat’. Save the code as ‘task1 1.m’. Note that, hereafter, function and file names are case sensi- tive, and your code should save output files in the current working directory. The syntax of the function should be as follows. function task1 1(X, Y) where X N-by-D matrix of feature vector (of floating-point numbers in double-precision format, which is the default in Matlab), where N is the number of samples, and D is the the number of elements in a sample. Note that each sample is represented as a row vector rather than a column vector. Y N-by-1 label vector (of int32) for X. Y(i) is the class number of X(i,:). (b) Run the following: function task1 1(X, Y) Make sure that the two output files are created properly. It will be a good idea that you write a script to run the above. Task 1.2 [5 marks] Look into the correlation matrix, R, you obtained, and describe your findings in your report, using graphs. Task 1.3 [10 marks] (a) Write a Matlab function task1 3() that • calculates the eigenvectors, EVecs and eigenvalues, EVals, of a covariance matrix, and calculates the cumulative variance, Cumvar, • finds the minimum number of PCA dimensions to cover each 70%, 80%, 90%, 95% of the total variance, and store the values to a vector MinDims, • saves the eigenvectors to a file named ‘t1 EVecs.mat’, • saves the eigenvalues to a file named ‘t1 EVals.mat’, • saved the cumulative variances to a file named ‘t1 Cumvar.mat’, • saves the the numbers of minimum dimensions, MinDims, to a fle named ‘t1 MinDims.mat’, Save the function as ‘task1 3.m’. The syntax of the function should be as follows. function task1 3(Cov) where Cov is a D-by-D covariance matrix (double). The specifications of the variables are as follows. EVecs D-by-D matrix (in double) EVals D-by-1 vector (in double) Cumvar D-by-1 vector (in double) MinDims 4-by-1 vector (in int32) The eigenvalues should be sorted in descending order, so that λ1 is the largest and λD is the smallest, and i’th column of EVecs should hold the eigenvector that corresponds to λi. Eigenvectors are not unique by definition in terms of scale (length) and sign, but we make them unique in this coursework by putting the following additional constraints, which your program should employ. • The first element of each eigenvector is non-negative. If it is not
the case, i.e. if the first element is negative, multiply -1 to the eigenvector (i.e. v ← −v) so that it gets the opposite direction. • Each eigenvector is a unit vector, i.e. ‖v‖ = 1, where v denotes an eigenvector. As far as you use Matlab’s eig() or Python’s numpy.linalg.eig(), you do not need to care about this, since either function ensures unit vectors. 3 Task specifications 4 (b) Run the following: task1 3(S); In your report, show a graph of cumulative variance. (c) Plot all data on a 2D-PCA plane, clarifying data of different classes, and show the graph in your report. Task 1.4 [25 marks] (a) Write a Matlab function task1 mgc cv() that carries out a classification experiment with multivariate Gaussian classifiers, using k-fold cross validation, and save the code as ‘task1 mgc cv.m’. The syntax of the function is as follows function task1 mgc cv(X, Y, CovKind, epsilon, Kfolds) where CovKind is the type of covariance matrix – 1 for full covariance matrix, 2 for diagonal covariance matrix, and 3 for shared covariance matrix, epsilon is a scalar (double) for the regularisation of covariance matrix described in Lecture 8, in which we add a small positive number () to the diagonal elements of covariance matrix, i.e. Σ ← Σ + I, where I is the identity matrix, Kfolds is the number of folds (partitions) in k-fold cross validation. Assume a uniform prior distribution over class, and use MLE for the estimation of model parameters. At first, the function should split the data set in Kfolds partitions for cross validation, whose information is stored in a N-by-1 vector, PMap, where PMap(i) holds the partition number that i-th sample is assigned to, and save it to a file named ‘t1 mgc cv PMap.mat’, where is the number of folds. For each fold, p, the function should • estimate the mean vector and covariance matrix for each class from the samples that do not belong to partition p. • save the mean vectors ((Ms)to ‘t1 mgc cv Ms.mat’, • save the covariance matrices (Covs) to ‘t1 mgc cv ck Covs.mat’, • carry out a classification experiment using the samples of partition p, and save the confusion matrix (CM) to ‘t1 mgc cv ck CM.mat’, • calculate the final confusion matrix (where each element is a relative frequency) and save it to ‘t1 mgc cv ck CM.mat’, where L = Kfolds + 1. In the above, replace , , , and with the actual values. Details of partitioning algorithm for k-fold cross validation and the variables to save will be specified in a separate sheet. (b) Run the function with epsilon=0.01 and Kfolds=5 for each CovKind=1,2,3, and report the accuracy (correct classification rate) in your report. Task 1.5 [5 marks] Using CovKind=1 (i.e. full covariance), investigate how the classification accuracy changes with respect to the regularisation parameter, epsilon. Plot a graph and describe your findings in your report. Task 2 – Neural networks [50 marks] In this task, you implement neural networks for binary classification problems, in which input feature is represented as a two-dimensional vector (x1, x2) T . We assume that decision regions are defined with polygon(s), whose specifications are given in the polygon specification file ‘task2 data.txt’ 2 in YourDataDir. The file is a plain-text file, in which each line specifies the name of the polygon and the coordinates of its vertices {(xp1, xp2)}Pp=1, where P is the number of vertices. The following is an example of the file. 2 You are not allowed to show this file of yours to anyone else. 3 Task specifications 5 Polygon A: -1 -0.5 6 1.25 6 6.25 1 6 Polygon B: 2.5 3 3.5 3 3.5 3.5 2.5 3.5 where two polygons, Polygon A and Polygon B, are defined. In each line, the first two numbers (e.g. -1 and -0.5 for Polygon A) from the left specify the coordinates (x11, x12) of the first vertex, followed by the coordinates (x21, x22) for the second vertex, and so one. You will see that each polygon has four vertices, meaning a quadrangle in this case. Task 2.1 [3 marks] Consider a single neuron with a unit function, whose output is defined as y(x) = h(wTx), where h(a) is a step function such that h(a) = 1 if a > 0, and h(a) = 0 otherwise 3. Implement this neuron as a Matlab function: function [Y] = task2 hNeuron(W, X) where X is a N-by-D data matrix (double), W is a (D+1)-by-1 weight matrix (double), Y is a N-by-1 output vector (double). Save the function as ‘task2 hNeuron.m’. Note that this function can take more than one input vector stored in a matrix X, where each input vector is represented as a row vector rather than a column one, and gives corresponding output as a vector Y. Task 2.2 [3 marks] Similar to task2 hNeuron() above, but consider another neuron which employs the logistic sigmoid function g(a) = 11+exp(−a) . Implement this neuron as a Matlab function: function [Y] = task2 sNeuron(W, X) and save it as ‘task2 sNeuron.m’. Task 2.3 [8 marks] Find the structure (i.e. connection of neurons) and weights of the neural network that classifies the inside and periphery of Polygon A as Class 1 (i.e. y(x) = 1), and the outside as Class 0 (i.e. y(x) = 0), where each neuron is modelled with task2 hNeuron(). This task is meant for you to work using pen and paper (and calculator), but it is also fine that you write a piece of code to find the weights. If it is the case, save the script or function as ‘task2 find hNN A weights.m’. Let w`ji denote the weight of neuron j in layer ` from neuron i in layer `−1 4. Normalise your weights in such a way that maxi |w`ji| = 1. Write the weights in a plain text file ‘task2 hNN A weights.txt’ in the following format. You write each w`ji in a separate line, for ` = 1, …, j = 1, …, and i = 0, 1, …, so that the first line contains w110 followed by w 1 11 and w 1 12 in the second line and the third line, respectively. The format of each line should be as follows: W(`,j,i) : where “” is the actual value of the weight. For example, if w 1 10 = 0.35, the first line should look like this: W(1,1,0) : 0.35 Spaces are only allowed just before and after “:”, and none in other places. In your report, show the structure of the network and explain how you found the weights. Task 2.4 [5 marks] Implement the neural network above as a function: 3 NB: The step function defined here is slightly different from the one in the lectures. 4 The input layer where input date are fed is regarded as layer 0 (zero). The output node of a single-layer neural network is in layer 1. 4 Functions that are not allowed to use 6 function [Y] = task2 hNN A(X) and save it as ‘task2 hNN A.m’, where X and Y follow the same format as was shown in Task 3.1. Task 2.5 [4 marks] Using task2 hNN A(), write a script that plots the decision regions in a 2D space, and save the code as ‘task2 plot regions hNN A.m’. Save the graph as a PDF file named ‘t2 regions hNN A.pdf’. Task 2.6 [6 marks] We now consider the decision regions formed with Polygon A and Polygon B, whose classification rule is shown below: Class 1 : A ∩ B¯ Class 0 : A¯ ∪B where A and B denote the inside and periphery of the corresponding polygon, B¯ denotes the complement of B. Implement the corresponding neural network as a function: function [Y] = task2 hNN AB(X) and save it as ‘task2 hNN AB.m’. Note that each neuron should be modelled with task2 hNeuron(). Task 2.7 [4 marks] Using task2 hNN AB(), write a script that plots the decision regions in a 2D space, and save the code as ‘t2 plot regions hNN AB.m’. Save the graph as a PDF file named ‘t2 regions hNN AB.pdf’. Task 2.8 [5 marks] We now consider another network task2 sNN AB() obtained by replacing all nodes of task2 hNeuron() with those of task2 sNeuron() in task2 hNN AB(), so that each neuron is now modelled with task2 sNeuron(). Implement the neural network as a function: function [Y] = task2 sNN AB(X) and save it as ‘task2 sNN AB.m’. Note that you will need to modify the weights to approximate the decision regions properly. Task 2.9 [4 marks] Using task2 sNN AB(), write a script that plots the decision regions in a 2D space, and save the code as ‘task2 plot regions sNN AB.m’. Save the graph as a PDF file named ‘t2 regions sNN AB.pdf’. Task 2.10 [8
marks] Investigate and discuss the decision regions for task2 sNN AB(), clarifying how and why they are different from those for task2 hNN AB(). 4 Functions that are not allowed to use Since one of the objectives of this coursework is to understand and implement basic algorithms for machine learning, you are not allowed to use those functions in standard libraries listed below. You should write the code by yourself using the basic operations of arithmetic for scalars, vectors, and matrices. If it is the case, use a different function name from the original one in standard libraries (e.g. MyCov() for cov() as shown in the table below). You may, however, use them for comparison purposes, i.e. to check your code. 5 Submission 7 Description of function Typical names Suggested name to implement Pairwise (squared) Euclidean distance pdist2() MySqDist() Compute the mean mean() MyMean() Compute the covariance matrix cov() MyCov() Compute Gaussian probability densities mvnpdf() K-NN classification fitcknn() run knn classifier() K-means clustering kmeans() my kMeansClustering() Compute confusion matrix confusion() comp confmat() Other utilities for classification You may use those functions or operations: Description Typical names Sum function sum() Cumulative sum cumsum() Square root function sqrt() Exponential function e, exp() Logarithmic function log(), ln() Matrix transpose transpose(), ‘ Matrix inverse inv() Determinant det() Log determinant logdet() · · · available in Inf2b cwk directory Eigen values/vectors eig() Sort sort() Sample mode mode() Vectorisation helpers bsxfun(), arrayfun() (NB: the list is not exhaustive) 5 Submission You should submit your work electronically via the DICE submit command by the deadline. No submission of printed document is required. Since marking for each task will be done separately, you should prepare separate reports for the two tasks, and save your report files in PDF format and name them ‘report task1.pdf’ and ‘report task2.pdf’. Remember to place your student number and the task name prominently at the top of each report. Do not indicate your name anywhere. Your report should be concise and brief for each task. Create a directory named LearnCW, copy all of the requested files to the directory, but do NOT put the data set files in it. A checklist will be available from the coursework web page. Submit your coursework from a DICE machine using: submit inf2b cw1 LearnCW
