Shipping an LLM-Powered iOS App; Embeddings, performance, and other Gotchas

I tried to ship an on-device LLM & RAG functionality on my own app for iOS; the hardest parts weren't the model.

From Places to Prompts: Building a Local-First On-Device AI App on iOS

Building an AI app usually implies a big server, API costs, and privacy trade-offs. I took a different path: Local-First.

I built an iOS app that lets users save places from anywhere (Instagram, Maps, Web), organizes them, and then uses an on-device Large Language Model (LLM) to answer questions like "Where can I get a quiet coffee nearby?" based solely on their data.

Here is the engineering journey of shipping an on-device RAG (Retrieval-Augmented Generation) system, from the unglamorous data ingestion to the "gotchas" of Swift 6 concurrency.

Act 1: The Foundation — "Places on a Map"

Before we could have AI, we needed data. I chose a Local-First architecture. There is no cloud database; the "truth" lives on the user's device in JSON files. This prioritized privacy and eliminated sync complexity during the MVP phase and still does.

MVVM Architecture Diagram

┌─────────────┐
│   Views     │ ← SwiftUI (MapView, ChatView, etc.)
└──────┬──────┘
       │
┌──────▼──────┐
│ ViewModels  │ ← @Observable (PlacesViewModel, ChatViewModel)
└──────┬──────┘
       │
┌──────▼──────┐
│   Models    │ ← Codable structs (Place, ChatMessage)
└──────┬──────┘
       │
┌──────▼──────┐
│  Services   │ ← StorageManager, RAGService, LlamaLLMService
└─────────────┘

The Architecture: I utilized a standard MVVM pattern backed by Swift's new @Observable macro. This allows the UI to react instantly to data changes without the boilerplate of Combine.

Key Decision: I chose local Codable structs over CoreData or SwiftData for the initial prototype.

• Pros: rapid iteration, human-readable data (places.json), simple backups.
• Cons: I had to write my own relationships (IDs) rather than relying on an object graph.

Act 2: Ingestion — The Share Extension Pipeline

The Hard Truth: Getting data into the app was significantly harder than building the AI.

Users don't input data manually; they share it. They hit "Share" on a Google Maps link, an Instagram Reel, or a TikTok restaurant review. Every source formats data differently.

Share Extension Ingestion Flow Diagram

Share Sheet (URL) 
     │
     ▼
┌────────────────────-┐
│ ShareContentParser  │
│  - isGoogleMapsURL? │
│  - isSocialMediaURL?│
│  - isGenericWebsite?│
└────────┬────────────┘
         │
    ┌────┴────┐────────────────┐
    ▼         ▼                ▼
GoogleMaps  SocialMedia    Website
URLResolver URLResolver    PlaceParser
    │         │                │
    └────┬────┴────────────────┘
         ▼
┌─────────────────────┐
│   PlaceDraft        │ (name, coords, category, notes)
└────────┬────────────┘
         ▼
┌─────────────────────┐
│ SharedStorageBridge │ (App Groups)
└────────┬────────────┘
         ▼
Main App (places.json)

The Pipeline Challenge

1. Google Maps Short Links: https://goo.gl/maps/... often uses JavaScript redirects. URLSession cannot handle these; it just sees the redirect page. I had to use WKWebView to resolve the final URL.
2. Tracking Params: URLs are dirty (?g_st=...). I built cleaners to strip these before parsing.
3. Platform Variance: Instagram hides location data in captions; Google Maps puts it in coordinates; Websites use OpenGraph tags, all of these I needed to figure out and handle.

The "Share Sheet" Gotcha iOS Share Extensions are ruthless environments to build for.

• Memory Limit: If you spike memory (e.g., loading a heavy library), the OS kills the extension.
• Time Limit: You have roughly 30 seconds to parse, resolve, and save.
• Networking: You must explicitly add the com.apple.security.network.client entitlement, or network calls fail silently.
• Cannot open your app from the extension: If you need to open your app from the extension, you don't... you have to open the app from outside the extension.

Act 3: On-Device AI — The RAG Stack

The chat feature is a Retrieval-Augmented Generation (RAG) pipeline using an implementation of llama.cpp for the LLM on iOS. It runs entirely offline.

Apple please improve your foundation model, or better yet allow us developers to have better access to iOS's on-device AI capabilities. It would make it so much better and performant for us developers.

RAG Pipeline Diagram

User Query ("romantic dinner nearby")
        │
        ▼
┌───────────────────-──┐
│ PlaceEmbeddingManager│
│  - Apple NL (512d)   │
│  - Query expansion   │
│  - Cached embeddings │
└─────────┬────────────┘
          ▼
┌─────────────────────┐
│ InMemoryVectorStore │
│  - Accelerate cosine│
│  - Top-K + threshold│
└─────────┬───────────┘
          ▼
┌─────────────────────┐
│    PromptBuilder    │
│  - Query analysis   │
│  - Context building │
│  - Token estimation │
└─────────┬───────────┘
          ▼
┌─────────────────────┐
│   LlamaLLMService   │
│  - llama.cpp (GGUF) │
│  - Gemma 3 1B/4B    │
│  - Streaming tokens │
└─────────┬───────────┘
          ▼
Streaming Response ("Try **Café Luna** (0.5 km)...")

1. The Embeddings (Apple NL)

I needed to turn user queries (text) into vectors (numbers).

• Decision: I used Apple's Natural Language framework. It produces 512-dimensional vectors.
• Why? It is built-in (no download required), fast (~25ms per query), and "good enough" for semantic matching.
• Upgrade Path: I built an optional toggle to use a BERT-based CoreML model (MiniLM) for users who want higher accuracy at the cost of disk space (43MB).

2. The Vector Store (In-Memory)

Since a user rarely has more than 5,000 saved places, I didn't need a heavy vector database like Chroma or others. I keep vectors in memory and use Apple's Accelerate framework to calculate Cosine Similarity. It takes roughly 5-10ms to search 1,500 places, which to me seems and felt really fast.

3. The LLM (Llama.cpp)

I integrated llama.cpp to run GGUF models (like Gemma 3 1B||4B, Ministral 8B or Llama 3.2 1B||4B).

Performance on my iPhone 16 Pro:

Key Engineering Lesson: Context Window Management On-device models have small context windows, about 32,000 tokens. If we pass the entire chat history plus all the place details, we could run out of tokens quickly but the bigger issue is memory usage. We will run out of memory before we run out of context window and more so with 4B models.

• Strategy: I estimate tokens (approx. 4 chars = 1 token). If history > 80% of the window, we trim the oldest messages.
• Warmup: Loading a 1GB or 4GB model takes seconds. The app "warms up" the model in the background when the user opens the chat tab, masking the latency.

Act 4: iOS Production Constraints ("The Gotchas")

Shipping a "demo" is easy; shipping a production app requires fighting the platform.

Gotcha 1: Swift 6 Strict Concurrency

Swift 6 enforces thread safety rigorously.

• The Issue: llama.cpp is C++ code interacting with Swift. It manages its own state and callbacks.
• The Fix: I had to mark the LLM service as @unchecked Sendable and manually manage locks (SerialDispatchQueue) to ensure we didn't crash the app by accessing the model from two threads simultaneously, but it's a small price to pay for the performance when compared to Apple's LLM.

Gotcha 2: Navigation Race Conditions

If a user taps a notification ("You are near Café Luna") while the app is in the foreground, SwiftUI often crashes or fails to navigate to the new view, I am still debugging and fighting this issue.

• Hacky Fix: The "Transaction Timing" hack. I dismiss all sheets, wait 0.5 seconds, and then push the new view.

// NotificationManager.swift
DispatchQueue.main.asyncAfter(deadline: .now() + 0.5) {
    self.handleNotificationResponse(...)
}

Gotcha 3: The "Broken" Share Extension

I spent days debugging why the Share Extension would sometimes fail to save data.

• Root Cause: It wasn't the code; it was the App Group container. The extension and the main app run in different processes. If both tried to write to places.json at the exact same millisecond, the file would corrupt.
• Solution: I implemented a "Drafts" folder. The extension writes a unique file per share (draft_uuid.json), and the main app consumes/merges them on launch.

Act 5: Quality & Polish

A technical marvel is useless if it feels the UX is trash. I invested time in:

1. 4K Video Export: A generic "Recap" feature that generates a video travelogue of visited places using AVAssetWriter, it is not the best export but it works.
2. Localization: Full support for English, Spanish, and French using .stringsdict for correct pluralization (e.g., "1 place found" vs "2 places found").
3. Testing: I wrote extensive Unit Tests for the URL parsers because external URLs change formats constantly, and I also invested time in testing the app on different devices.

Lessons I Learned building this App

Something To Do Mobile App aka. Swift To Do

1. Local-First works: No server latency, no API costs, no privacy concerns.You don't need a server to build powerful AI experiences but it sure simplify the architecture and experience.
2. Ingestion is a product: Treating the "Share to App" experience as a first-class feature (rather than an afterthought) changed how I and hopefully users perceived the app's utility.
3. RAG is fragile: The AI is only as good as the data retrieval. If the embeddings are weak, the smartest LLM will hallucinate. Embeddings, retrieval, prompting, inference — each can fail independently
4. iOS has opinions: You must respect the platform's constraints... memory limits, main thread rules, and sandbox security... or the OS will terminate your process and crash your app lol.
5. Fallbacks everywhere: Semantic → Distance, LLM → Mock, WebView → HTTP → HTML

Next steps for the app:

• CloudKit sync for multi-device
• Improve performance on the areas of the app were there are bottlenecks/hangs
• Model evaluation harness (automated quality testing)
• Improve the LLM / RAG pipeline, maybe fine-tuning a model to better suit the app's needs.
• Apple Intelligence integration when available