Reproduction on PersonaChat

[hzhwcmhf]

Weizhen Qi (@qiweizhen) Hello, thanks for your great work!

I am reproducing the result on personachat (in the NAT setting) but find it hard to reach the performance reported in your paper.

Specifically, I use the dataset from GLGE, and then finetune BANG use following command:

DATA_DIR=processed_data
ARCH=bang_ar_nar_mixed_base
CRITERION=ngram_language_loss_NAR_mixed
SAVE_DIR=models/model_nar
TENSORBOARD_LOGDIR=models/logs_nar
PRETRAINED_MODEL=checkpoint_base_9gram_ck35.pt
NAR_RATIO=1.0

MAX_TOKENS=12000
LR=5e-5

# on 8 GPUs
fairseq-train $DATA_DIR \
--user-dir ./bang/bang  \
--task translation_bang --arch $ARCH \
--optimizer adam --adam-betas '(0.9, 0.999)' --clip-norm 0.1 \
--lr $LR --min-lr 1e-09 --nar-ratio $NAR_RATIO --ngram 1 --disable-ngram-loss \
--lr-scheduler inverse_sqrt --warmup-init-lr 1e-07 --warmup-updates 1000 \
--dropout 0.1 --attention-dropout 0.1 --weight-decay 0.01 \
--criterion $CRITERION --label-smoothing 0.1 \
--update-freq 1  --max-tokens $MAX_TOKENS \
--num-workers 8  \
--load-from-pretrained-model $PRETRAINED_MODEL \
--ddp-backend=no_c10d --max-epoch 50 \
--max-source-positions 512 --max-target-positions 512 \
--truncate-source \
--save-dir $SAVE_DIR \
--keep-last-epochs 10  --save-interval 5 \
--tensorboard-logdir $TENSORBOARD_LOGDIR

Then I use the evaluation script here, but only get:

--	B1	B2	D1	D2
My reproduction	30.69	19.27	2.1	23.0
Your Result	39.82	30.72	1.9	14.2

Can you give some advice for the reproduction? For example, the training and evaluation script?

[hzhwcmhf]

Also I use post_processed_nar.py for deduplication. I upload the output here: chat.dedup.txt

[Paulmzr]

I also found it hard to reproduce the results. Could you please specify batch size and learning rate used in each downstream task? Thank you. Weizhen Qi (@qiweizhen)