CSE142-Fall 2019 Project Predicting Star Ratings from User Reviews Handed out Date: Nov 6, 2019 Evaluation deadline: Dec 4, 2019 Report and code due date: Dec 6, 2019 • The project has to be done in groups of 3. • 10% of the points are for write-up on the group’s diversity. • We recommend implementing all the codes in Python3. • You are allowed to use any external library including Machine Learning and Natural Language Processing libraries. • One (and only one) member of the group has to submit the project report using his/her account on Canvas. All group members will get points for that submission. • How to submit your solutions: Your project report must be typed up separately (in at least an 11-point font) and submitted on the Canvas website as a PDF file. The code and related files should be submitted in a .zip file. • Your report should clearly mention your group’s number (from the shared project sign up sheet), and the names, email addresses and student ids of all group members. • You are very strongly encouraged to format your report in LATEX. You can use other software but hand- written reports are not acceptable. • The Computer Science and Engineering Department of UCSC has a zero-tolerance policy for any incident of academic dishonesty. If cheating occurs, consequences within the context of the course may range from getting zero on a particular assignment, to failing the course. In addition, every case of academic dishonesty will be referred to as the student’s college Provost, who sets in motion an official disciplinary process. Cheating in any part of the course may lead to failing the course and suspension or dismissal from the university. 1 1 Course Project [100 points] The rise in E-commerce has brought a significant rise in the importance of customer reviews. There are hundreds of review sites online and massive amounts of reviews for every product. The ability to successfully decide whether a review will be helpful to other customers and thus give the product more exposure is vital to companies that support these reviews. This project is about automatically identifying the appropriate ratings for a given review. Specifically, the Machine Learning classification task is as follows: given an input text (review), you have to predict the respective ratings. (from 1 to 5) 1.1 Dataset The training dataset provided to you is the modified Yelp Open dataset. The dataset consists of reviews and their respective ratings in JSON format. You will be provided: • data train.json: – This dataset has around 330000 entries. – Each entry consists of a review in multiple sentences, corresponding rating and also the usefulness of the review. – There are five fields – ‘stars’, ‘useful’,‘funny’,‘text’ – You need to predict ‘stars’ (ratings) from the ‘text’ (reviews) – If you think it helps, you may use the attributes of the reviews about them being useful or funny! – You can download this file from this link this link • data test.json: – This contains only the reviews and their attributes. The ratings will not be provided for this dataset. The test set will be provided shortly before the project evaluation deadline. 1.2 Evaluation Your trained model will be evaluated on a held-out and hidden test set. As mentioned above, the goal at test time is to predict a rating of each review in the test set. The test set will be provided to you on the day of the evaluation. Your code should take as input the test file with no labels and output predictions in a .csv file. We will evaluate your predictions against the ground truth (hidden from you at all times) using the following performance measures: Accuracy, Precision, Recall, and F1-score. Your system should be able to accept such a file as input. Note that the file is in JSON format. Also, ‘text’ entry of each data point represents a review and it can contain punctuation marks including commas and quotation marks. Your predictions file (output) should contain only one column (predicted rating). The file should be comma-separated (it should not contain any other punctuation marks like quotation marks). Outputs that do not conform to this format will not be evaluated. Please see the template file “data test wo label template.json”. This is just a template test file with 5 entries. The actual test file that we will release near the end will have around 50,000 entries. We also provide the template for output prediction file. See “predictions.csv”. There should be one column with header predictions. The prediction numbers should NOT be quoted. Please make sure your prediction file format is similar to the one provided as template. 1.3 Report Page Limit: 3 pages You are also expected to write a short report on your findings. The report will describe the details of your approach like the data cleaning/pre-processing, feature extraction, model details, and experiments done to build 2 your model. The first section of the report should be titled ‘Tools used’ and should list all the tools/libraries that you use for the project. In the report, indicate whether you wrote code for a particular step or used a library. For example, if you try Logistic Regression, when describing your approach indicate if you used a library or coded the algorithm. The first page should also contain a small paragraph on diversity. Diversity of the group can be based on a variety of factors and as mentioned in class you don’t have to limit yourself to race/gender. Talk to your teammates and find out how you might be different from them. You will be evaluated on your description of diversity. A typical report would have the following components/sections, but feel free to customize the suggested components according to your project. Required Components: 1. Title 2. Group details (full names, email addresses and student ids of all group members) 3. Tools Used (including a short 1-2 sentence description of what they were used for) 4. Diversity Suggested Components: 1. Abstract (1 paragraph summary of your approach and key findings) 2. Data Pre-processing 3. Feature Extraction 4. Approach(es) 5. Experimental Set-up 6. Results 7. Conclusion 8. Ideas for future work 1.4 What to Submit: 1. Report (.pdf file) to be submitted on Canvas. 2. 〈Names〉 code.zip: This file should contain any code that you write for the project. It should contain a ReadMe and the code should be properly documented. This file should be submitted on Canvas with your report. 3. 〈Names〉 predictions.csv containing the predictions of your model on the provided test set. Please note that we will not be able to evaluate your predictions if your predictions file is not in the correct format. In the above description, 〈Names〉 should be replaced by the last names of all group members in alphabetical order. For example, if there were two members in the group named: Joe Smith and Mary Johnson, then the zip file would be named JohnsonSmith code.zip. 3
