Transformer Transfer Learning & Commonsense Reasoning

RoBERTa sentiment fine-tuning, multiple-choice reasoning, and sentence embeddings — Neural Networks / Deep Learning laboratory, ECE NTUA, academic year 2022–2023, Team 24.

This project explores how pretrained transformers can be adapted to sentiment classification and evaluated on language-understanding tasks. It combines a small Yelp Polarity fine-tuning study with experiments on PIQA, TruthfulQA, and WinoGrande, using Hugging Face Transformers, PyTorch and Sentence Transformers.

Original submission · Analysis notebook · Experiment notes · Colab guide

What the project explores

Experiment	Data	Approach
Sentiment classification	Yelp Polarity	Fine-tune roberta-base; compare learning-rate, batch-size and weight-decay settings.
Physical commonsense	PIQA	Select one of two solutions with transformer multiple-choice heads.
Answer selection and similarity	TruthfulQA generation split	Select from four constructed candidates and compare sentence embeddings at a cosine threshold.
Pronoun resolution	WinoGrande XL	Compare classifier scores for the two possible sentence completions.

The original notebook contains the Greek assignment instructions, implementation, discussion and saved outputs. The Python modules provide portable commands with explicit seeds, configuration files, checkpoint saving and corrected evaluation logic.

Recorded coursework observations

These values come from the 2023 notebook, not new runs of the reorganized code.

Experiment	Recorded observation	Context
Yelp Polarity	95.67% best reported accuracy	Hyperparameter table; 300 training / 300 test reviews, seven epochs.
PIQA	56% for ALBERT	100 validation examples; the base checkpoints can initialize untrained multiple-choice heads.
TruthfulQA custom semantic score	56% for LUKE + all-MiniLM-L6-v2	100 examples; custom threshold metric with a duplicated candidate bug.
WinoGrande	78% for DeepPavlov RoBERTa	100 validation examples; original label handling is inconsistent across classifiers.

Interpretation matters. The sentiment notebook repeatedly uses test data for model selection, and its explicit optimizer overrides the learning rate shown in TrainingArguments. The multiple-choice heads are not established task-tuned baselines. The TruthfulQA score is not an official truthfulness benchmark metric. These historical scores should not be presented as validated benchmark rankings. Experiment notes explain the original protocol and each correction.

Repository guide

Location	Purpose
notebooks/original_submission.ipynb	Unmodified shared submission, including all saved outputs.
notebooks/analysis.ipynb	English walkthrough and historical result tables.
notebooks/colab_runner.ipynb	Guided setup, tests, GPU training, Drive output and optional reasoning experiments.
src/data.py	Dataset adapters, seeded sampling, balanced train/validation splits and candidate construction.
src/train.py	Sentiment fine-tuning, validation, checkpoint selection and training history.
src/evaluate.py	Final sentiment test evaluation from a saved checkpoint.
src/commonsense.py	PIQA, TruthfulQA and WinoGrande inference with explicit scoring assumptions.
src/visualize.py	Recreate the historical comparison figure from CSV.
results/	Transcribed historical tables, including all 30 TruthfulQA model/encoder combinations.
tests/	Offline sampling, scoring, tiny-model training and checkpoint checks.
docs/	Protocols, limitations, Colab instructions and attribution.

Setup

Use Python 3.10–3.12. CPU checks were run with Python 3.12 and PyTorch 2.5.1. From a terminal:

git clone https://github.com/johnskop7/Neural_Networks_Ex2.git
cd Neural_Networks_Ex2
python -m venv .venv
source .venv/bin/activate
python -m pip install torch==2.5.1 --index-url https://download.pytorch.org/whl/cpu
python -m pip install -r requirements.txt

On Windows PowerShell, activate with .venv\Scripts\Activate.ps1. For GPU training, install a compatible CUDA PyTorch build using PyTorch's installation instructions, then install the requirements. The scripts automatically use CUDA when available; --device cpu or --device cuda selects explicitly. JupyterLab is optional (python -m pip install jupyterlab).

Datasets and pretrained checkpoints download on first use. Yelp is subsampled after downloading the source split. The adapter reads published Hugging Face Parquet exports without enabling remote dataset scripts. PIQA currently uses the lighteval/piqa mirror. Dataset/model availability and licenses are controlled by their publishers. Trained weights and new runs are excluded from Git; original trained checkpoints are not included.

Run in Google Colab

Open the badge above, save a copy in Drive and select a GPU runtime. The runner clones this public fork, installs the Hugging Face dependencies, checks the environment and runs the offline tests. It keeps Colab's existing PyTorch build. Start with one epoch, then increase to seven. Optional PIQA/TruthfulQA runs are disabled until you enable them. See the Colab guide for Drive storage, resuming and troubleshooting.

Run the experiments

Run commands from the repository root. Each training run should use a fresh output directory.

Sentiment fine-tuning

python -m src.train --model roberta-base --train-size 300 --val-size 100 --epochs 7 --batch-size 8 --learning-rate 5.32e-6 --output-dir runs/roberta-yelp
python -m src.evaluate --checkpoint runs/roberta-yelp/best --samples 300 --output runs/roberta-yelp/test.json

The revised recipe reserves a balanced validation subset from the training split; the test set is used only by the evaluation command. Trainer derives the learning-rate schedule from the actual run length. Outputs include config.json, split_indices.json, history.json, validation.json, epoch checkpoints and best/. The final test command writes metrics and sampled indices. Results will differ from the historical notebook because the protocol has changed.

PIQA and TruthfulQA

The following commands explicitly reproduce the exploratory base-model approach, accepting newly initialized multiple-choice weights. They are useful for inspecting the pipeline, not for claiming task-tuned performance:

python -m src.commonsense --task piqa --model albert-base-v2 --allow-random-head --samples 100 --output runs/piqa-albert.json
python -m src.commonsense --task truthfulqa --model bert-base-uncased --allow-random-head --semantic-model sentence-transformers/all-MiniLM-L6-v2 --samples 100 --output runs/truthfulqa-bert.json

For meaningful model comparisons, supply a suitable fine-tuned multiple-choice checkpoint via --model and omit --allow-random-head. Missing weights then stop execution. Absence of missing weights alone does not prove task suitability. The TruthfulQA command fixes the second correct-answer slot and reports both correct-slot accuracy and the custom semantic acceptance rate; neither is official MC1/MC2 accuracy. All predictions and the selected seed are saved in JSON.

WinoGrande

Use a sequence-classification checkpoint whose input format and positive-class semantics you have verified. The command requires its positive-class index explicitly:

python -m src.commonsense --help

Set --task winogrande --model MODEL_ID --positive-label-id CLASS_INDEX --output runs/winogrande.json, replacing both placeholders based on the model card/configuration. The adapter scores each completed sentence with that same class probability. Class IDs are model-specific; an NLI checkpoint is not interchangeable with a WinoGrande-tuned classifier. See protocol details.

Offline checks and plots

python -m unittest discover -s tests -v
python -m src.visualize

The tests create a tiny BERT locally, train it on synthetic text, reload its checkpoint and exercise both classification heads. They also check balanced/disjoint splitting, the corrected TruthfulQA candidates and model-specific positive-class scoring. They require no model or dataset download. Full pretrained experiments have not been rerun during organization.

Team and attribution

Team 24, ECE NTUA, 2022–2023:

Contributor	GitHub
Ioannis Litsos	johnskop7
Achilleas Tsimichodimos	tsimiho

Forked from tsimiho/Neural_Networks_Ex2. The shared notebook and course template remain attributed to their original authors. The modular code and English documentation were added later. See references and attribution. No new project-wide license is asserted for the supplied coursework.