SLM-Python-Buddy

An offline Python coding assistant that runs on your phone. Fine-tune small language models (like Qwen2.5-Coder-1.5B) and deploy them as lightweight GGUF files for mobile devices.

What it does

• Fine-tune small language models on Python instruction datasets using LoRA
• Convert & Quantize models to GGUF format (Q8: ~1.5GB) for mobile deployment
• Deploy offline on Android devices for Python learning and coding help
• Answer questions like "What is a loop?" or "Explain list comprehension" with ~10-15 tokens/s on mid-range phones

Perfect for students and developers who want AI coding assistance on their phone without internet dependency or privacy concerns.

Demo

🎥 Watch android demo on YouTube

Performance on different devices:

• Redmi Note 13 (Snapdragon 685, 8GB): ~7-12 tokens/s
• Redmi Note 11 (Snapdragon 680, 4GB): ~7-10 tokens/s
• AMD Ryzen 5 5000 (16GB RAM): ~80-120 tokens/s

Try Yourself

Finetuned models available:

• HuggingFace: https://huggingface.co/taufiq-ai/qwen2.5-coder-1.5-instruct-ft (recommended)
• Ollama: ollama run taufiq-ai/qwen2.5-coder-1.5b-instruct-ft-taufiq-04092025

To try on phone, download the GGUF file and follow the demo video.

Quick test using Python:

# pip install llama-cpp-python huggingface-hub
from llama_cpp import Llama

model = Llama.from_pretrained(
   repo_id="taufiq-ai/qwen2.5-coder-1.5-instruct-ft",
   filename="qwen2.5-coder-1.5b-instruct-mt-04092025-v2.gguf" # use the latest gguf file
)

response = model.create_chat_completion(
   messages=[
       {"role": "system", "content": "You are a helpful python coding assistant."},
       {"role": "user", "content": "How to install Python on Ubuntu?"}
   ],
   max_tokens=512
)

print(response['choices'][0]['message']['content'])

---

Quick Start

pipx install uv
git clone https://github.com/taufiq-ai/slm-python-buddy.git
cd slm-python-buddy
cp .env.example .env
uv venv
uv sync
export PYTHONPATH=.

Usage

1. Download Pre-trained Model from HuggingFace

# syntax
uv run python -m scripts.download_pretrained_model --model-name <hf_model_name> --model-dir <dir> --device <auto|cpu|cuda>

# example:
uv run python -m scripts.download_pretrained_model --model-name Qwen/Qwen2.5-Coder-1.5B-Instruct --model-dir model --device auto

2. Infer Pre-trained Model

# syntax
uv run -m scripts.infer_pretrained_model "<prompt>" --max_tokens <tokens> --model_path <path_to_model> --device <auto|cpu|cuda>

# example
uv run -m scripts.infer_pretrained_model "What is list comprehension?" --max_tokens 500 --model_path model/Qwen/Qwen2.5-Coder-1.5B-Instruct --device auto

3. Fine-tune with 4bit-Quantization and LoRA Config

1. Finetuning Pretrained Model

# syntax
uv run -m pybuddy.train \
  --dataset_path <path/to/dataset.json> \
  --model_path <path/to/base/model> \
  --output_dir <path/to/output> \
  --batch_size <int> \
  --grad_accum <int> \
  --lr <float> \
  --epochs <int> \
  --max_length <int>

# example
uv run -m pybuddy.train \
  --dataset_path data/data.json \
  --model_path model/Qwen/Qwen2.5-Coder-1.5B-Instruct \
  --output_dir model/ft_model \
  --batch_size 2 \
  --grad_accum 4 \
  --lr 2e-4 \
  --epochs 3 \
  --max_length 4096

2. Finetuning Finetuned Model

# Syntax
uv run -m pybuddy.train \
 --dataset_path <path/to/dataset.json> \
 --model_path <path/to/merged/model> \
 --output_dir <path/to/output> \
 --batch_size <int> \
 --grad_accum <int> \
 --lr <float> \
 --epochs <int> \
 --max_length <int>

# example
uv run -m pybuddy.train \
 --dataset_path data/python-code-inst-dataset.json \
 --model_path model/Qwen/qwen2.5-1.5b-instruct-ft-merged \
 --output_dir model/ft_model_04092025_0400 \
 --batch_size 2 \
 --grad_accum 4 \
 --lr 2e-4 \
 --epochs 3 \
 --max_length 4096

4. Inference Fine-tuned Model

1. Single Prompt:

# syntax
uv run python -m pybuddy.inference "<prompt>" --max_tokens <max_tokens> --base-model <path/to/base_model> --ftmodel <path_to_finetuned_lora_adapter> --device <auto|cpu|cuda>

# example
uv run python -m pybuddy.inference \
 "What is a loop?" \
 --max_tokens 512 \
 --base-model model/Qwen/Qwen2.5-Coder-1.5B-Instruct \
 --ftmodel model/ft_model_04092025_0400/lora-adapter \
 --device auto

2. Chat Mode (Long Context):

# syntax
uv run python -m pybuddy.chat --base-model <path/to/base_model> --ftmodel <path/to/lora_adapter> --device <auto|cpu|cuda>

# example
uv run python -m pybuddy.chat \
  --base-model model/Qwen/Qwen2.5-Coder-1.5B-Instruct \
  --ftmodel model/ft_model_04092025_0400/lora-adapter \
  --device auto

6. Merge Finetuned LoRa Adapter with base model

# uv run python
from pybuddy import utils
utils.merge_ft_model(
    base_model_path="<path_to_basemodel>" # "model/Qwen/Qwen2.5-Coder-1.5B-Instruct",
    lora_adapter_path="<path_to_finetuned_lora_adapter>" #"model/ft_model/lora-adapter", 
    output_path="<path_to_save_merged_model>" # "model/Qwen/qwen2.5-1.5b-instruct-ft-merged"
)

7. Run on Android (Offline)

Convert your fine-tuned model to GGUF format for offline Android inference.

Step 1: Install llama.cpp

# NOTE: Make sure that you clone `llama.cpp` repo outside the `slm-python-buddy` project directory
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j 8
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Follow llama.cpp build guide for more information.

Step 2: Convert to GGUF

# syntax
~/llama.cpp$ python convert_hf_to_gguf.py \
  ~/<path_to_project>/slm-python-buddy/<path_to_merged_model> \  
  --outfile ~/<path_to_save>/<filename.gguf> \
  --outtype <f32|f16|bf16|q8_0|tq1_0|tq2_0|auto>

# example
~/llama.cpp$ python ~/llama.cpp/convert_hf_to_gguf.py \
  ~/slm-python-buddy/model/Qwen/qwen2.5-1.5b-instruct-ft-merged \
  --outfile ~/slm-python-buddy/model/gguf/qwen2.5-1.5b-q8.gguf \
  --outtype q8_0

Step 3: Test locally

# syntax:
~/llama.cpp$ llama-cli \
  -m <path_to_filename.gguf> \
  -p "<prompt>"

# example:
~/llama.cpp$ llama-cli \
  -m ~/slm-python-buddy/model/gguf/qwen2.5-1.5b-q8.gguf \
  -p "What is list comprehension?"

Step 4: Deploy to Android

• Copy GGUF file to your Android device
• Install Llama Chat or similar local LLM android/ios app
• Import the GGUF file
• Enjoy offline inference (~10-15 tokens/s on mid-range devices)

Acknowledgments

Special thanks to LalokaLabs for providing GPU compute resources that made this project possible. The fine-tuning was conducted on their Quadro RTX 8000 (46GB VRAM) infrastructure.

Without access to high-end GPU resources, training and experimenting with language models would be significantly more challenging for individual developers and researchers.