RAG on iOS: Grounding an On-Device LLM in Your App's Own Content

Ritus, the coffee app I'm building, has a chat tab called the Lab for coffee questions. It runs 100% on the iPhone with Apple's Foundation Models framework, the model behind Apple Intelligence. This post is how I made that small on-device model answer from the app's own recipes, grind guide and glossary instead of making things up.

In short:

• Search does most of the work: keyword search (BM25) re-ranked by Apple's on-device embeddings, merged with Reciprocal Rank Fusion.
• Only the best ~1,400 characters go into the prompt, so they fit the model's 4k-token window, and the app only cites what the model actually read.
• If a question shares no words with the app's content, nothing is retrieved, and the app says it's answering from general knowledge instead of citing a wrong source.
• 44 real questions run as a unit test: 97% of the Portuguese ones find the right chunk in the top 4, and the test caught four bugs.

The Going deeper blocks are closed by default. Open them if you want the math and the code.

Here's where it ends up, on a real iPhone:

"What should water be like per SCA?" comes back with two sources from Ritus. The follow-up, "What is SCA?", pulls four, including two glossary terms.

The problem

A small model that has never seen the app

On-device means privacy (nothing leaves the phone), offline support and zero cost per token. It also means real constraints:

• Small: about 3B parameters, with a 4,096-token context window.
• Offline: it can't search the web.
• Not built for facts: Apple positions it for focused text tasks inside apps (summarizing, extracting, rewriting), not as a general-knowledge chatbot.
• It has never seen the app: not its recipes, grind guide or glossary.

And Ritus isn't short on content. Besides the app's own recipes, grind guide and glossary, it shares its educational content with my site, ritus.club: 52 pages on brewing, roasting, fermentation and more, in English and Portuguese. The Portuguese version alone is about 237k characters of text, and the English one is about the same size. The model just doesn't know any of it.

So ask it for the V60 recipe and it makes one up:

User: What's the right V60 recipe?

AI: Use 10 g of coffee for 700 ml of water at 100 °C, steep for 10 minutes… (totally wrong)

Fluent, confident, and not from the app. That's a hallucination: when a language model doesn't know something, it doesn't say "I don't know"; it generates the most plausible-sounding text.

Four ways to fix it, and only one fits

What RAG is

RAG = Retrieval-Augmented Generation: hand the model the right page before it answers.

1. Retrieve: search your own content for the chunks that match the question.
2. Augment: put those chunks into the prompt, next to the question.
3. Generate: the model answers from what it just read, and the app shows the sources.

To retrieve, you first need something to search. So the full pipeline has four steps: one done once, and three done for every question.

      ONCE                          EVERY QUESTION
┌─────────────┐    ┌─────────────┐   ┌─────────────┐   ┌─────────────┐
│  0  Index   │ →  │ 1 Retrieve  │ → │ 2  Augment  │ → │ 3 Generate  │
└─────────────┘    └─────────────┘   └─────────────┘   └─────────────┘
 Split the content  Find the chunks   Put those chunks  The on-device
 into chunks and    that best match   into the prompt   model answers
 prepare them for   the question                        from them
 search

Step 0: index the content

Split the content into chunks, one per section

The model only reads small pieces, so the content is cut before any search. Since ritus.club is my own site, I don't need to scrape anything: a build script reads the content straight from the site's code and writes one JSON file per language, which the app bundles.

• Cut at each heading, so every chunk holds one whole idea.
• Long sections are split by sentence, ~700 characters each.
• Ritus ends up with ~560 chunks from the site and the app.

Each chunk starts with its page and section title, so it still makes sense on its own.

Step 1: retrieve the best chunks

Two ways to search: by words (BM25) and by meaning (embeddings). Then combine both.

BM25: keyword search, ranked

Lexical search: like Ctrl+F, but it scores every chunk and sorts the results.

• Matches the words of the question.
• Rare words weigh more: "Hoffmann" is in 4 of 558 chunks, "grind" in 101.
• Repeating a word helps, but less each time.
• Precise for exact terms like V60, TDS, Gesha.

Query "ideal grind for french press": the Grind Guide: French Press chunk wins, because "french" and "press" are rare and outweigh "grind". A V60 chunk that only shares "grind" scores under 25% of the best match and is dropped.

Weakness: no notion of synonyms. "Japanese pour-over method" shares no words with "V60".

static func tokenize(_ text: String) -> [String] {
    text.lowercased()
        .folding(options: .diacriticInsensitive, locale: nil)
        .components(separatedBy: CharacterSet.alphanumerics.inverted)
        .filter { $0.count >= 3 }
}
 
// Poor man's stemmer: keep the first 7 letters
static func stem(_ token: String) -> String {
    guard token.count > 7, !token.contains(where: \.isNumber) else { return token }
    return String(token.prefix(7))
}

// IDF: the rarer the term in the corpus, the higher its weight
let idf = log(1 + (n - df + 0.5) / (df + 0.5))
 
// BM25: saturating term frequency (k1 = 1.2), length normalization (b = 0.75)
let lengthNorm = k1 * (1 - b + b * Double(length) / averageLength)
score += idf * Double(tf) * (k1 + 1) / (Double(tf) + lengthNorm)

Embeddings turn meaning into coordinates

Semantic search: compare what texts mean, not which words they share.

• A model maps text to a vector: 512 numbers for Apple's sentence embedder.
• Similar meaning, nearby vectors, even with different words.
• Cosine similarity: 1 = same direction, 0 = unrelated.
• Strong on paraphrases, weaker on exact technical terms.
• Every vector is scaled to length 1 when the index is built, so comparing two of them is just multiplying and adding their numbers.

Think of a map of ideas: "French press", "prensa francesa" and "plunger coffee" sit together; "V60" is nearby; "latte art" is far away. And Apple's embedder runs on the device:

// Built into iOS: no server, no API key
let embedder = NLEmbedding.sentenceEmbedding(for: .english)
let vector = embedder?.vector(for: "grind for french press")
// -> 512 numbers. Similar meaning, nearby vectors.
 
// ~560 chunk vectors: computed once, cached on disk
let vectors = cachedVectors(for: chunks, embedder: embedder)
 
// The question is embedded off the main thread
actor QueryEmbedder { func vector(for text: String) -> [Float]? }

Use both searches, but trust words more

Words and meaning fail on different questions, so combine them: hybrid search.

• Combine by rank, not raw score: the two scores use different scales.
• In Ritus, words count 65%, meaning 35%.
• No word in common? It retrieves nothing, so the app says "not in Ritus" instead of citing a wrong source.

A wins by doing well in both lists, even though it isn't first for words.

// No word in common? Retrieve nothing, not even by meaning.
guard snapshot.hasLexicalMatch(queryTokens) else { return [] }
 
// Drop weak matches: under 25% of the best BM25 score
lexical = lexical.filter { $0.score >= best * 0.25 }
 
// Weighted Reciprocal Rank Fusion, k = 60
var score = 0.65 / (60 + Double(lexicalRank))          // BM25
if let rank = semanticRank[candidate.index] {
    score += 0.35 / (60 + Double(rank))                // embeddings
}
// -> sort by score, keep the top 4

Step 2: augment the prompt

Fit the best chunks, and only those, into a small context window.

• Budget: add chunks by relevance until the limit.
• Delimit the context from the question with clear headers.
• Instruct the model for the no-context case.
• Cite only what the model actually read.

PREVIOUS CONVERSATION:        <- last 2 exchanges
User: Hoffmann's V60 recipe?
Ritus: ## V60 James Hoffmann…
---
CONTEXT:                      <- retrieved chunks
• V60 James Hoffmann 300mL by James Hoffmann.
  18g of coffee to 300ml of water…
---
Question: and for two cups?

The sources shown are only the chunks that made it into the prompt:

private static let contextBudget = 1400    // characters
 
static func contextChunks(for hits: [KnowledgeHit]) -> [KnowledgeChunk] {
    var used = 0
    var included: [KnowledgeChunk] = []
    for hit in hits {
        let cost = hit.chunk.text.count + 4
        if used + cost > contextBudget, !included.isEmpty { break }
        included.append(hit.chunk)
        used += cost
    }
    return included    // these, and only these, become sources
}

1,400 characters leaves room for instructions, history and a 1,200-token answer. The top hit always gets in, even if it alone is over budget.

Step 3: generate the answer

Apple's on-device model writes the reply and streams it to the screen:

for try await partial in session.streamResponse(to: prompt) {
    pending = partial.content            // full text so far
    if ContinuousClock.now - lastFlush >= .milliseconds(50) {
        messages[index].text = partial.content   // ~20 updates/s
        lastFlush = .now
    }
}

• Each partial is the full text so far, not a delta.
• Updating SwiftUI on every token would stall the UI, so updates are batched every 50 ms.
• Sources are attached only after the reply finishes, so citation cards don't flash under a half-written answer.

No Apple Intelligence? Search is the product

switch SystemLanguageModel.default.availability {
case .available:
    availability = .chat
case .unavailable(.deviceNotEligible):
    availability = .smartSearchOnly(reason: "This device isn't compatible with Apple Intelligence.")
case .unavailable(.appleIntelligenceNotEnabled):
    availability = .smartSearchOnly(reason: "Turn on Apple Intelligence in Settings.")
case .unavailable(.modelNotReady):
    availability = .smartSearchOnly(reason: "The model is still being prepared on this device.")
default:
    availability = .smartSearchOnly(reason: "Apple Intelligence is unavailable.")
}

Without Apple Intelligence, the Lab doesn't break or show an error: it becomes smart search over the same index, with results as cards that open the real screens. Search is the product; the LLM is the conversation layer.

Smart search on the Simulator, in Portuguese. "V60" and "bloom" each return four sources from the articles, the grind guide and the glossary, with no model involved.

All three steps in one function

ChatSession.send(_:), simplified:

// 1. Retrieve
var hits = await knowledgeBase.search(question)
if hits.isEmpty, let previous {        // follow-up question
    hits = await knowledgeBase.search(previous + " " + question)
}
let context = Self.contextChunks(for: hits)
 
// 2. Augment
let prompt = augmentedPrompt(question: question,
                             history: history,
                             context: context)
 
// 3. Generate
let stream = session.streamResponse(to: prompt)

Does it work? I test it

An eval is a fixed set of questions with known answers, run again after every change. Without one, every tweak is a guess.

1. Write 44 real questions, each with the chunk that answers it. For example: "ideal grind for french press" → Grind Guide: French Press.
2. Run the search for every question.
3. Count how often the right chunk is in the top 4 results.

recall@4 is how often the right chunk appears among the top 4 results. The eval also tracks recall@2, because the top 2 chunks almost always fit the 1,400-character budget. It runs as a regular unit test:

// coffee-toolsTests/RAGRetrievalEvalTests.swift
private static let casesPT: [(String, [String])] = [
    ("moagem ideal pra prensa francesa", ["grind-Prensa Francesa"]),
    ("o que é TDS?",                     ["glossary-tds"]),
    ("o que é maceração carbônica",      ["fermentations-"]),
    // … 29 Portuguese + 15 English questions
]
 
@Test func retrievalRecallPT() async {
    let recall = await evaluate(Self.casesPT, language: .pt)
    #expect(recall.top4 >= 0.9)
    #expect(recall.top2 >= 0.8)
}

Expected IDs are prefixes, so any chunk from the right page counts. It also checks that small talk retrieves nothing and that chunk IDs are unique. Since it runs on the Simulator, where embeddings are off, it measures the lexical engine, the part that gates everything else.

The four bugs the eval caught

All from the first version, all invisible in day-to-day use:

• Duplicate IDs. Site copies of the app's recipes shared their IDs, so one recipe showed up twice in the top 4.
• A filter that never filtered. On iPhone, the semantic score alone passed the old absolute relevance threshold. That's why the floor is now relative and lexical.
• Sources the model never read. Source cards listed chunks that didn't fit in the prompt.
• Wrong language. The grind troubleshooting article was in English inside the Portuguese base, so "my coffee is bitter" found nothing.

Known limits and next steps

What's still not great:

• 3 eval questions still miss. 7-letter stemming is weak in English ("store" vs. "storage"), and "my coffee tastes bitter" falls outside the top 4.
• Semantics can't rescue a question with no shared word, by design.
• NLEmbedding's sentence embedder is older. NLContextualEmbedding (iOS 17+) is transformer-based and multilingual.
• Chat needs Apple Intelligence (iPhone 15 Pro or newer); other devices get smart search only.

What I want to try next: swapping in NLContextualEmbedding, a real stemmer per language (or a coffee synonym list), tool calling so the model decides when to search ("agentic RAG"), full contextual retrieval with an LLM writing a context line per chunk at build time, and growing the eval with real user questions.

Questions people ask

Why on-device and not OpenAI or Claude? Privacy (nothing leaves the device), offline support, zero cost per user, no network latency. The trade-off is a smaller model and Apple Intelligence devices only.

Why not a vector database? There are ~560 chunks. Searching all of them in memory takes under a millisecond. Vector databases solve millions of documents; here they'd only add complexity. The real bottleneck is the LLM generating text.

Why combine by rank instead of adding the scores? The scores live on different scales. Adding them needs calibration; RRF only uses positions. Adding scores is exactly what broke the relevance filter in the first version.

Why not let the model search with tool calling? A searchKnowledge tool calling the same search() is still RAG, just triggered by the model. But for a 3B model with a 4k window, always retrieving is the safer choice: it's predictable and directly measured by the eval, it costs only the chunks in the window, and it avoids the main risk of tool calling, a small model not calling the tool and hallucinating anyway.

Is this "real" RAG? Yes: retrieve → augment → generate, with BM25, embeddings, rank fusion, structure-based chunking and an eval, at app scale and 100% on-device. What's missing is LLM re-ranking, on purpose: an extra on-device call would cost seconds.

Final thoughts

The whole pipeline, as built in the app:

Chunk by structure → Search by words + meaning → Combine by rank → Prompt within a budget → Generate on device → Measure with evals

Most of this is plain search engineering. The model is the easy bit. Getting the right 1,400 characters in front of it, and knowing that you did, is the actual work.

RAG isn't a bigger model. It's giving the model the right page at the right time.

References

• Apple, TN3193: Managing the on-device foundation model's context window
• Apple, Apple Intelligence Foundation Language Models Tech Report 2025
• Anthropic, Introducing Contextual Retrieval
• Robertson & Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond (2009)
• Cormack, Clarke & Buettcher, Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods (SIGIR 2009)