Pa ge 1 ECE1512 2019 FALL ECE1512 – Homework 3: Monday, October 28, 2019 Background Machine learning (ML) has played a significant role in data analysis and decision making. Though promising, there is little insight in how and why a decision is made. The lack of interpretability hurdles the practical usage of ML (especially neural network) in applications that are highly regulated and may have serious consequences (for example, medical diagnosis, autonomous cars, financial decision, etc.). To understand a rationale of a ML solution and thus to build trust on the outcomes, explainable AI (XAI) that attempts to “opening the black box of AI algorithms” is highly desirable. Introduction In this assignment, you will be exploring a simple approach to detecting anomalies in a dataset of industrial inspection texture images, MVTec Anomaly Detection Dataset. Finding defective components is an important problem in manufacturing and black-box machine learning algorithms is unacceptable. You will be asked to apply a texture classification CNN pre-trained on normal images, infer pixel-level activation maps with Grad-CAM to detect defects, and assess the quality of the binarized defect activation maps to explain the defect decision. Dataset MVTec Anomaly Detection (AD) Dataset: https://www.mvtec.com/company/research/datasets/mvtec-ad/ 5000 images (700-1024 pixels) = 3629 normal training images + 467 normal test images + 1258 defective test images 15 classes in total: 5 texture and 10 object classes Instructions Setup 1. Prerequisites: a. python b. keras (TensorFlow backend) c. sklearn (suggested) d. numpy (suggested) e. matplotlib (suggested) 2. Download a3_release.zip from Quercus 3. data is the re-formatted MVTec AD dataset for texture classes only a. 1781 images = 1266 normal training images + 133 normal test images + 382 defective test images b. Only includes images from the 5 texture classes (carpet, grid, leather, tile, and wood), re-formatted into a folder structure compatible with Keras c. data/train includes normal training images Pa ge 2 ECE1512 2019 FALL d. data/test includes normal and defective test images 1. Normal images: _test_good_.png 2. Defective images: _test__.png e. data/ground_truth includes binary defect masks for defective test images 4. save/texturenet.h5 is a pre-trained classification CNN for Keras trained on texture classification (alternatively, you may train your own CNN for texture classification) a. Architecture customized to be lightweight and ideal for applying Grad-CAM (first three blocks of VGG16, GAP, and one FC layer) b. The CNN was trained on classifying the 1266 normal training images to the five texture classes 5. gradcam_utils.py is a Python file including utility functions needed for Grad-CAM in this assignment (alternatively, you may use other code for Grad-CAM, such as that provided on GitHub: https://github.com/jacobgil/keras-grad-cam) 6. train_cnn.ipynb is a Python notebook explaining how the CNN was trained, included as a reference (train_cnn.pdf is the PDF version) 7. demo_gradcam.ipynb is a Python notebook explaining how to use the Grad-CAM utility functions, included as a reference (demo_gradcam.pdf is the PDF version) Task #1: Texture classification [4 marks in total] 1. Load the pre-trained classification CNN save/texturenet.h5 and the test set images from “data/test”, then classify the texture classes of the test set images, assuming that the most-confident class is predicted for each image 2. Report the following [2 marks]: a. Overall classification accuracy on the test set b. Plot the classification ROC curves (and find the AUC) for each class c. Plot the normalized confusion matrix 3. Analyze the success cases and failure modes: for which classes does the texture classifier succeed? For which classes does it fail? Can you explain why? [2 marks] Task #2: Defect detection [6 marks in total] 1. Apply the pre-trained CNN to each test image (both normal and defective) and implement the Grad-CAM method using gradcam_utils.py to infer pixel-level activation maps for each predicted texture class 2. Without training on the defective test images, design your own approaches to the following – report and justify your methodology [1 mark]: a. Create a “defect activation map” to quantify the probability that a pixel is “defective” in each image b. Classify the test image as normal or defective using your “defect activation map” 3. Report the following [2 marks]: a. Overall defect accuracy on the test set b. Plot the defect ROC curves (and find the AUC) for each class 4. Analyze the success cases and failure modes: for which classes does the defect detector succeed? For which classes does it fail? Can you explain why? [3 marks] Pa ge 3 ECE1512 2019 FALL Task #3: Defect detection explanation [5 marks in total] 1. Load the ground-truth binary defect masks of the defective test images from data/ground_truth (assume a blank mask for normal test images) 2. Obtain your predicted binary defect masks for all test images (normal and defective) by applying a threshold to your “defect activation map” – this is your defect detection explanation 3. Report the following [1 mark]: a. Overall mean Intersection-over-Union (mIoU) between the ground-truth and predicted binary defect masks b. Intersection-over-Union (IoU) for each ground-truth class between the ground-truth and predicted binary defect masks (regardless of the predicted texture class) 4. Analyze the success cases and failure modes: for which classes does the defect detection explainer succeed? For which classes does it fail? Can you explain why? [2 marks] 5. Analyze the general psychology of the defect detection explanation – how well does it explain the defect decision to a human? Qualitatively, how is it successful or lacking? [2 marks] Notes 1. Coding is expected for this assignment, and your code must be included in your submitted report – use the Python programming language 2. No machine learning models need to be trained for this assignment 3. External code may be used only if properly cited and intended for other problems – you cannot use code intended for the MVTec dataset! 4. Include as many data visualizations as possible – a good visual is worth more than a thousand words 5. It is recommended that you write your Python code as a Jupyter notebook and export as a PDF report for submission – Jupyter notebooks enable easy insertion of figures with Markdown text Resources Certain concepts and methods used in this assignment may be unfamiliar to you. Refer to these online resources for more details (cite if code is used): Keras Installation: https://keras.io/#installation Training and test sets: https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets Evaluating a pre-trained CNN in Keras: https://medium.com/@vijayabhaskar96/tutorial- image-classification-with-keras-flow-from-directory-and-generators-95f75ebe5720 Summary of Grad-CAM: https://medium.com/@ninads79shukla/gradcam-73a752d368be ROC curves: https://machinelearningmastery.com/roc-curves-and-precision-recall- curves-for-classification-in-python/ Confusion matrix: https://www.geeksforgeeks.org/confusion-matrix-machine-learning/ Intersection-over-Union / mean Intersection-over-Union: https://www.jeremyjordan.me/evaluating-image-segmentation-models/ Pa ge 4 ECE1512 2019 FALL Format and Grading Your report should be approximately 10 pages and list any additional references you may have used for your answers. The page limit is a general guideline only, and includes any figures and code that you might include in the report. Links to external web pages and code repositories, listing your results and code implementations, such as “GitHub” are permissible. Kindly provide answers to the ab
ove questions in complete sentences and in paragraph form. You will be marked on the correctness, preciseness and comprehensiveness of your answers and quality of your presentation. You are not required to observe the “Required Format for Submitting Assignment Reports” guidelines included in your course syllabus for this assignment. Turnitin scores will be visible upon submission, so make sure that you are not penalized for plagiarism in uncited text and code portions. The report should be submitted as PDF email attachment, using the following subject line: ECE1512_HOMEWORK3-by-STUDENTNAME
