models

      ███╗   ███╗ ██████╗ ██████╗ ███████╗██╗     ███████╗     ██╗
      ████╗ ████║██╔═══██╗██╔══██╗██╔════╝██║     ██╔════╝     ╚██╗
█████╗██╔████╔██║██║   ██║██║  ██║█████╗  ██║     ███████╗█████╗╚██╗
╚════╝██║╚██╔╝██║██║   ██║██║  ██║██╔══╝  ██║     ╚════██║╚════╝██╔╝
      ██║ ╚═╝ ██║╚██████╔╝██████╔╝███████╗███████╗███████║     ██╔╝
      ╚═╝     ╚═╝ ╚═════╝ ╚═════╝ ╚══════╝╚══════╝╚══════╝     ╚═╝

In-process small-language-model inference for Java 25.

Models is an in-process inference library for Java applications. It loads an open-weight language model and generates text without Python, a separate model server, or a network request.

Models currently reads GGUF, a local-model file format that packages model structure, tokenizer metadata, and trained numeric weights. Those weights are often quantized to reduce disk and memory requirements. Efficient inference must then map the weights, tokenize input, execute the transformer layers, maintain the attention KV cache, and sample and stream output tokens. Models implements that complete pipeline on Java 25 and uses the Vector API for CPU SIMD execution:

• backend-java executes every inference kernel in Java.
• backend-native runs the same Java 25 and Vector API pipeline, substituting only selected, measured bottleneck kernels with a small Models-owned Rust library through Java's Foreign Function and Memory (FFM) API.

On supported Apple Silicon Macs, backend-apple also exposes Apple's OS-managed, on-device SystemLanguageModel to Java through FFM and a small Models-owned Swift binary. It uses Apple Intelligence rather than GGUF weights, and its client plugs into the same Models, LangChain4j, and Spring AI text-generation contract. See the Apple Foundation Models guide.

The native backend is not a wrapper around llama.cpp or Ollama. Those runtimes are controlled performance comparators only. The project intends to replace each Rust kernel with pure Java when a released JDK can provide equivalent correctness and performance. See Runtime architecture for the exact boundary and migration policy.

Capabilities

Implemented functionality includes:

• GGUF v2/v3 parsing with memory-mapped tensor access
• Llama, Qwen2, and Qwen3 decoder architectures
• F32, F16, Q4_0, Q5_0, Q8_0, Q4_K, Q5_K, and Q6_K tensor paths
• byte-level BPE and Llama SentencePiece tokenizers
• grouped-query attention, RoPE, SwiGLU, KV caching, and autoregressive decode
• greedy, temperature, top-k, top-p, and repetition-penalty sampling
• plain Java, LangChain4j, Spring AI, and Spring Boot integrations
• Apple Foundation Models on supported Apple Silicon Macs
• framework-neutral guarded RAG
• compact WordTour semantic-order models

Supported Models

Committed same-host evidence covers 27 exact artifacts across the 25 model identities below. Support is bound to an artifact SHA, workload, runtime selector, backend plan, correctness result, and latency measurements; consult the qualification ledger for those exact details.

Model identity	Domain
SmolLM2 360M	General
SmolLM2 1.7B	General
SmolLM3 3B	General
Qwen3 1.7B	General
Qwen2.5 0.5B	General
Qwen2.5 1.5B	General
Llama 3.2 1B	General
Llama 3.2 3B	General
Gemma 3 1B	General
H2O Danube2 1.8B	General
DeepSeek-R1-Distill-Qwen 1.5B	General
TinyLlama 1.1B Chat	General
Qwen3 0.6B	Coding
Qwen2.5-Coder 0.5B	Coding
Qwen2.5-Coder 1.5B	Coding
DeepSeek-Coder 1.3B	Coding
MiniCPM5 1B	Coding
Yi-Coder 1.5B	Coding
Qwen2.5-Math 1.5B	Math
EuroLLM 1.7B	Multilingual
UmarTransit 1B	Transportation
Indian-Legal-Qwen2.5 3B	Legal
Nexus Legal	Legal
Nexus Finance	Finance
Nexus Medical	Healthcare

• Model support and qualification
• Production RAG results
• Controlled inference results

Install

Models requires Java 25 or newer. For a versioned, qualified artifact, add ModelJars and the marker JAR for the selected model:

dependencies {
    implementation("org.modeljars:modeljars:0.1.0")
    implementation(
        "org.modeljars.huggingface:" +
            "ggml-org.qwen3-0.6b-gguf.q4_0:" +
            "3.0.0-q4_0.1",
    )
}

The ModelJars facade brings Models and both execution backends, then selects the backend qualified for that exact artifact. Applications that manage their own GGUF files can depend on Models directly:

dependencies {
    implementation("com.integrallis:models:0.1.0")
    implementation("com.integrallis:backend-java:0.1.0") // or backend-native
}

Use Apple's on-device system model on a supported Apple Silicon Mac:

dependencies {
    implementation("com.integrallis:backend-apple:0.1.0")
}

Model marker artifacts, checksums, variants, and measured runtime profiles are provided by ModelJars.org. ModelJars is a separate project and depends on Models; the Models artifacts remain usable without a catalog dependency.

Quick Start

import static org.modeljars.catalog.Qwen3_0_6b_Q4_0.MODEL;

var prompt = ChatTemplate.CHATML_NO_THINK.render(List.of(
    ChatMessage.system("Classify the user's intent in one phrase."),
    ChatMessage.user("I want to cancel my order")));
var options = SamplingOptions.builder()
    .temperature(0.0f)
    .maxTokens(20)
    .build();

try (var model = ModelJars.open(MODEL)) {
    String result = model.generate(prompt, options);
    System.out.println(result);
}

ModelJars.open resolves the pinned artifact, downloads and verifies it when needed, chooses its qualified Models backend, and applies a matching performance profile. Applications that manage their own GGUF files can use the lower-level PureJavaBackend.load(Path) and RustFfmBackend.load(Path) APIs described in the getting-started guide.

Streaming uses the same loaded model:

try (var model = ModelJars.open(MODEL)) {
    model.generate(prompt, options, new TokenStream() {
        @Override
        public void onToken(String token) {
            System.out.print(token);
        }

        @Override
        public void onComplete() {
            System.out.println();
        }

        @Override
        public void onError(Throwable error) {
            error.printStackTrace();
        }
    });
}

Backend diagnostics expose the exact plan selected for the loaded model:

try (var model = ModelJars.open(MODEL)) {
    model.diagnostics().optimizations().forEach(System.out::println);
}

Profile matching, explicit overrides, and every execution-plan switch are documented in Execution planning.

Integrations

Integration	Module	Surface
Plain Java	models-runtime	GenerationLoop and TokenStream
LangChain4j	models-langchain4j	ModelsChatModel
Spring AI	models-spring-ai	blocking and streaming chat models
Guarded RAG	models-rag	retrieval abstention, citation validation, and fallback
Vector storage	models-embedding	optional bridge to vectors
Apple on-device model	backend-apple	Apple Foundation Models through Java FFM

These adapters are implemented and tested against the same backend contracts; they do not select hidden inference paths. See Framework integrations.

Documentation

The documentation site covers architecture, model qualification, execution planning, integrations, guarded RAG, Javadocs, and release testing.

• Executable Java notebooks
• Apple Foundation Models bridge
• Native kernel backend

Build

./gradlew build
./gradlew test
./gradlew integrationTest
./gradlew spotlessApply

Real-model tests resolve immutable ModelJars revisions, download missing GGUF files, verify size and SHA-256, and fail if a required model cannot run. The complete test matrix and individual large-model tasks are in Building and testing.

Scope

Models is intended for local and private inference with small, qualified model artifacts. It is not a training framework, a high-throughput GPU serving system, or a claim that every GGUF architecture works. Use the qualification ledger for the exact model, quantization, runtime, workload, and hardware evidence.

License

Licensed under the Apache License 2.0.