Finetuning Gemma with ReFT

What is this project about?

The notebook titled "Finetuning Gemma with ReFT" explores applying ReFT (Representation Fine-Tuning) to the Gemma 2 language model on Kaggle. The goal: fine-tune Gemma 2 (specifically the 2B instruction-tuned model) using ReFT, a parameter-efficient, representation-based tuning method (Kaggle,Hugging Face).

Why was it made?

Efficiency: Traditional fine-tuning (SFT or PEFT like LoRA) updates model weights. But ReFT focuses on modifying internal representations. This allows tuning with fewer parameters—ReFT methods are 15x–65x more parameter-efficient than LoRA (arXiv,Wikipedia).

Interpretability: By learning interventions on hidden representations, you gain conceptual control over the model—understanding how concepts are mapped inside the model (Hugging Face,arXiv).

This project demonstrates how to fine-tune Gemma 2 using this method to adapt it to new tasks more efficiently, potentially unlocking custom capabilities or improving performance without full re-training.

How does it work?

Here’s a breakdown:

Select the Base Model

We start with the Gemma 2 (2B instruction-tuned) model available via Hugging Face (e.g. "google/gemma-2-2b-it"). (Hugging Face,Kaggle)

Configure ReFT (LoReFT method)

• ReFT freezes the base model’s parameters.
• Instead of modifying weights, it learns an intervention that projects hidden representations into a low-rank subspace.
• This intervention is injected into a specific layer’s hidden states (e.g. residual stream at layer 20) (Hugging Face,arXiv,Wikipedia)
• ReFT freezes the base model’s parameters. Instead of modifying weights, it learns an intervention that projects hidden representations into a low-rank subspace.This intervention is injected into a specific layer’s hidden states (e.g. residual stream at layer 20) (Hugging Face,arXiv,Wikipedia).

Training the ReFT Intervention

• Using PyReFT, the intervention is trained on task-specific data (e.g., prompting formats, question-answer pairs).
• Only the low-rank projection parameters are learned, keeping the base model frozen.
• Using PyReFT, the intervention is trained on task-specific data (e.g., prompting formats, question-answer pairs).Only the low-rank projection parameters are learned, keeping the base model frozen.

Applying and Testing

• Once trained, the intervention can be injected into the model at inference.
• The model now reacts differently—performing better on the target task thanks to the learned representation modification.
• Once trained, the intervention can be injected into the model at inference.The model now reacts differently—performing better on the target task thanks to the learned representation modification.

The Kaggle notebook shows code for setting this up, training, and evaluating Gemma with the ReFT adaptation.

Context from ReFT

For extra understanding, here’s what ReFT is about:

• ReFT methods intervene on hidden representations, not weights—enabling tuning with minimal parameters (Wikipedia,arXiv).
• LoReFT is the low-rank subspace variant, offering both efficiency and effectiveness across tasks like reasoning, GLUE, and instruction tuning (arXiv).
• The notebook builds on this idea by practically applying it to Gemma 2 to adapt the model using this cutting-edge approach.

Summary Table

What’s the project about?

Fine-tuning Gemma 2 using ReFT (representation fine-tuning) techniques.

Why build it?

To achieve efficient, interpretable model tuning with minimal parameter updates.

How does it work?

Apply a low-rank intervention (LoReFT) to hidden representations of a frozen Gemma model, train using PyReFT, and evaluate.