Hosting local AI models

apprenti.dev's AI features can run entirely offline, on a local model, with no cloud provider and no per-token cost. This page is for whoever runs a curriculum fork and wants to offer that to their apprentices: how model hosting works, how to build the catalog the app reads, and which models are a reasonable fit on a phone versus a desktop.

Local model weights are never stored in Git — not in this repository, not in a curriculum fork. They live on Google Drive, and the app downloads and verifies them directly onto the apprentice's device.

How hosting works

A small JSON catalog tells the app which models exist and where to fetch each one. The app ships with a single bundled fallback entry, but you can point it at your own catalog instead — Settings → AI lets an apprentice enter a catalog's Google Drive file id, and the app fetches that file directly by id. It never lists or browses your Drive folder, so nothing else in that folder is exposed. 

{
  "schemaVersion": 1,
  "source": "google_drive",
  "models": [
    {
      "id": "llama-3.2-1b-instruct-q4",
      "displayName": "Llama 3.2 1B Instruct Q4",
      "driveFileId": "REPLACE_WHEN_UPLOADED",
      "fileName": "Llama-3.2-1B-Instruct-Q4_K_M.gguf",
      "sha256": "",
      "sizeBytes": 0,
      "quantization": "Q4_K_M",
      "contextWindow": 4096,
      "recommended": true
    }
  ]
}

driveFileId is the id segment from a Drive share link — the part between /file/d/ and /view in the URL you get from Drive's Share button. A row with an empty id still shows up in the app's model list, just disabled until you fill it in.

Use apprenti.dev's own hosted catalog

You don't have to host anything yourself to try local models — apprenti.dev publishes its own catalog. In the app, go to Settings → AI → Local models and paste this catalog file id:

Catalog file id: 147raTcMO52l3Jzslewm_-mSYvWHwkiVm

That's the same id whether you copy it from the file's share link or type it in directly — the app only needs the id, not the full link. If you'd rather browse what's there first, the models live in this public Drive folder — browsing the folder is just for humans; the app itself never lists it, only the catalog file by id.

This is a convenience, not a requirement — everything below explains how to host your own catalog instead, for a curriculum fork that wants different models or full control over where the weights come from.

Setting up the Drive folder

  1. Put your GGUF files and the catalog file together in one Drive folder — a Google Drive desktop sync folder works well for this.
  2. Share each GGUF file (or the whole folder) as Anyone with the link → Viewer.
  3. Share the catalog file itself the same way, and copy its file id into Settings → AI on your own device to confirm it resolves.
  4. Hand that same catalog file id to your apprentices.

Building the catalog

Writing that catalog file by hand means hashing every file yourself and keeping sizes in sync whenever you add a model. A small PowerShell script does that for you:

Download New-GgufCatalog.ps1

Run it against the folder that holds your GGUF files: 

.\New-GgufCatalog.ps1 -Path D:\Drive\apprenti-gguf

It scans for GGUF files, hashes each one (SHA-256), fills in file size, and writes the catalog file next to the weights — in the schema the app expects. Re-running it after adding more models merges in the Drive file ids and context-window values you already filled in, so you never lose that work. The one thing it can't do for you is upload the files and paste each Drive id in — that's still a manual step, done once per file.

Requires PowerShell 5.1 or later — already on every Windows machine, and available on macOS and Linux too if you'd rather run it there.

What an apprentice's device actually does

Once a model is selected, the app downloads it directly from Drive, streams it to a temporary file, and only renames it into place once the SHA-256 you provided actually matches — so a corrupted or partial download never silently becomes "the model." Downloads can be cancelled, retried, or deleted at any point.

Which models to offer

apprenti.dev doesn't restrict which GGUF model you host — any llama.cpp-compatible model works. The practical limit is the apprentice's own device memory, not anything the app enforces. A reasonable spread, smallest to largest:

  • • Any phone, including older or lower-memory ones — Llama 3.2 1B Instruct (GGUF). The smallest reasonable default; a good first model to offer since almost every device can run it.
  • • Newer phones, or a light desktop — Qwen2.5 3B Instruct (GGUF) or Gemma 2 2B Instruct (GGUF). Noticeably better reasoning than the 1B tier for a modest size increase. Gemma's repository is gated — you'll need your own Hugging Face account and to accept Google's license there before you can download it and re-host it on Drive; apprentices themselves never touch Hugging Face.
  • • Desktop (Windows, Linux, or macOS with 8 GB+ RAM) — Phi-3.5-mini Instruct (GGUF) or Mistral 7B Instruct v0.3 (GGUF) for the most capable option in this set — plan for a few gigabytes of storage per model.

Each model page lists several quantizations (file size versus quality trade-off). Q4_K_M is a reasonable default for most devices; for phones with limited storage, check the file size on the model page before choosing and consider a smaller quantization.

Performance differs by platform

The app tries GPU acceleration first on macOS and Linux. On Windows, it currently loads local models on CPU only — a known issue with GPU context creation on that platform — so the same model will feel slower on Windows than on a similarly specced Mac or Linux machine. If a model fails to load at all, the app automatically retries with a smaller context window before giving up, so a device running low on memory degrades gracefully rather than simply failing.

Keep in mind

Never commit a GGUF file to this repository or to a curriculum fork — they belong on Drive, not in Git history. Local models are one option among several; apprentices can just as easily use a cloud provider instead, and nothing about hosting local models is required to use apprenti.dev at all.

← All Docs Back to apprenti.dev