public final class LocalLlmClient

An LlmClient that runs a GGUF model on the device with llama.cpp. Create one with LlmClient.local(String).

Nothing leaves the device and no network is needed. The model file is loaded on the first request, which can take several seconds for a large model, and stays in memory until close().

LocalLlmClient llm = LlmClient.local(FileSystemStorage.getInstance()
        .getAppHomePath() + "qwen2.5-0.5b-instruct-q4_k_m.gguf");
llm.chat(ChatRequest.builder()
        .addMessage(ChatMessage.user("Name three primes"))
        .build()).ready(r -> Log.p(r.getText()));

The prompt is rendered by the chat template the model file carries, so any instruction-tuned GGUF works without configuration. ResponseFormat.JSON_OBJECT is enforced with a grammar, so the output is always valid JSON. Tools and image parts are not supported and fail the request with LlmException.ErrorType.INVALID_REQUEST.

Availability

Android and iOS. The builders link llama.cpp only into applications that call LlmClient.local(String). Elsewhere, the simulator included, every request fails with LlmException.ErrorType.INVALID_REQUEST; use LlmClient.ollama() to develop against the same model on the desktop.

Requests run one at a time on a background thread, in the order they were made. Results and streaming callbacks arrive on the EDT.