Test Dataset for Dream-coder

[preordinary]

Thank you for your work in DLLM community! Have you tried to test Dream-coder in other types of datasets, such as gsm8k, math dataset?
Thanks again for your patience!

[zhxieml]

Hi @preordinary, we haven't tested our model on math benchmarks, but you can directly use the evaluation toolkit from Dream for Dream-coder since they share the same architecture. For details, please refer to: https://github.com/DreamLM/Dream/tree/main/eval and https://github.com/DreamLM/Dream/tree/main/eval_instruct.

[preordinary]

Since I have encountered a large number of EOS when using Dream-Instruct for math and code datasets. This problem can be solved with EOS penalty, which was added in Dream-coder. I am wondering if you will adapt EOS penalty into the original Dream Model?

[zhxieml]

I am wondering if you will adapt EOS penalty into the original Dream Model

Could you add the EOS penalty to the generation utilities? Please see to https://huggingface.co/Dream-org/Dream-Coder-v0-Instruct-7B/blob/main/generation_utils.py#L422 for reference.