4bit AWQ/GPTQ quantized model?

[kenvix]

Thank you for your work.

I would like to know if there are any plans to release a 4bit AWQ/GPTQ quantized version for the 70B size model, as I don't have enough resources locally to run the quantization procedures.

[yuxie11]

Thank you for your question. We plan to do this in the future.

[aoji0606]

There is a simple way, you can load the model using quantization of transformers, such as modifying inference code:

tokenizer, model, image_processor, context_len = load_pretrained_model(model_path, args.model_base, model_name, load_4bit=True)

If you set load_4bit=True, please set nproc_per_node=1 in scripts.
This will bring some loss of accuracy.