Empirical comparison between the frozen microsoft/phi-2 base model (2.78B) and its 4-bit QLoRA adapter (5.24M parameters / 0.19%) evaluated on a 50-task leak-free algorithmic benchmark.
# Loading base generation...# Loading adapted generation...### Exercise 5, ### Instruction:), and hitting the maximum token limit. The fine-tuned QLoRA adapter cleanly terminates generation with <|endoftext|> immediately upon completing the requested function.
| Model / Configuration | Training Scope | Target Modules | Resp Loss | Core Pass% (50 Tasks) | EOS Term% | Max-Token% | Mean Length |
|---|---|---|---|---|---|---|---|
| Raw Phi-2 Base | Pretrained (2.78B) | None | 0.5792 | 52.0% (26/50) | 18.0% | 84.0% | 284.6 tok |
| Phase 2 Corrected (Ref) | 2,000 samples | q_proj, v_proj | 0.4471 | 62.0% (31/50) | 94.0% | 6.0% | 112.8 tok |
| Config A (Attn-Only, 2k) | 2,000 samples | q_proj, v_proj | 0.4522 | 62.0% (31/50) | 98.0% | 2.0% | 112.8 tok |
| Config B (Full-Attn, 2k) | 2,000 samples | q, k, v, dense | 0.4477 | 54.0% (27/50) | 90.0% | 10.0% | 121.9 tok |
| Config C (All-Linear, 2k) | 2,000 samples | q, k, v, dense, fc1, fc2 | 0.4442 | 60.0% (30/50) | 98.0% | 2.0% | 115.6 tok |
| Config D (Masking-Ablation, 2k) | 2,000 samples | q_proj, v_proj (no mask) | 0.4489 | 62.0% (31/50) | 98.0% | 2.0% | 114.1 tok |
| Config E (5k, Step 100) [BEST] | 5,000 samples | q_proj, v_proj | 0.4803 | 68.0% (34/50) | 96.0% | 4.0% | 115.4 tok |
| Algorithmic Category | Task Count | Raw Phi-2 Base | Config E Step 100 | Improvement Delta |
|---|---|---|---|---|
| String Manipulation | 6 | 2/6 (33%) | 3/6 (50%) | +17% |
| Data Structures | 8 | 3/8 (38%) | 5/8 (63%) | +25% |
| Sorting & Searching | 6 | 3/6 (50%) | 5/6 (83%) | +33% |
| Recursion & Dynamic Programming | 8 | 5/8 (63%) | 7/8 (88%) | +25% |
| OOP & Class Definitions | 6 | 3/6 (50%) | 4/6 (67%) | +17% |
| File I/O & Exception Handling | 4 | 1/4 (25%) | 2/4 (50%) | +25% |
| Algorithm Complexity & Edge Cases | 6 | 5/6 (83%) | 3/6 (50%) | -33% |
| Bug Fixing | 6 | 4/6 (67%) | 5/6 (83%) | +16% |
| Overall 50-Task Total | 50 | 26/50 (52.0%) | 34/50 (68.0%) | +16.0% |
The project systematically addressed common low-resource fine-tuning failure modes:
input column in 69% of records. Restored full input preservation.<|endoftext|> token loss.Phase2CompletionCollator to restrict gradient updates solely to the generated code response, cutting training wall-clock time by 5.27× over full-sequence masking.q_proj, v_proj) vs Full-Attention vs All-Linear, confirming Attention-only is parameter-optimal at 5.24M parameters.Load the adapter directly from Hugging Face Hub on your local GPU or CPU:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
# Configure 4-bit NF4 quantization (or remove for CPU)
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
)
# Load base model & attach adapter
base_model = AutoModelForCausalLM.from_pretrained(
"microsoft/phi-2",
quantization_config=bnb_config,
device_map="auto",
trust_remote_code=True,
)
model = PeftModel.from_pretrained(base_model, "alfeenafsal/phi2-qlora-finetune")
model.eval()
tokenizer = AutoTokenizer.from_pretrained("alfeenafsal/phi2-qlora-finetune", trust_remote_code=True)
prompt = "### Instruction:\nWrite a Python function that checks if two strings are anagrams.\n\n### Response:\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
out = model.generate(**inputs, max_new_tokens=150, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))