This repository contains the code for my MSc project. The project uses the Civil Comments dataset and compares five models: Logistic Regression, Linear SVM, DistilBERT, BERT-base and HateBERT.
The notebooks follow the order of the experiments:
01cleans the data and creates the training, validation and test splits.02runs the traditional machine learning models using the 200k training set.03–05run DistilBERT, BERT-base and HateBERT using the 200k training set.06runs the traditional machine learning models using the full training data.07–09run the three Transformer models using the full training data.10combines the model predictions and calculates the final summary results.11performs the error analysis and creates a sample of 300 comments for manual review.
The original and prepared data files are not included directly in this repository because they are too large. The data files can be accessed from this OneDrive data folder.
The paths in the notebooks are the paths I used for the experiments. If the notebooks are run on another computer, the paths need to be changed based on the local data location.
Notebooks 03–09 were run on Kaggle, so they use Kaggle input and output paths.
Most of the smaller result files generated by each model are included in the results folder. The individual prediction files are not included because they are too large.
The combined prediction files generated by notebook 10 are also not included because they are large. However, the final summary results and error-analysis results are included.
Notebook 11 first creates a sample of 300 comments from the combined predictions. After the sample was generated, I manually read the comments and added my own labels and analysis. The final table with the manual labels is included in the tables folder. These labels were added after the automated sampling and were not generated by the notebook.
The trained model weights and checkpoints are also not included because these files are large.