feat(search): Kotlin Multiplatform search for iOS and Android 10+

JVM_RUNTIME.md

@@ -0,0 +1,75 @@
+# Lucene on a JVM, on every platform
+
+Lucene needs a JVM, and keeps up with the newest Java (21 for Lucene 10, 25 for Lucene 11): ART can't, so the search
+runs Lucene on a real JVM everywhere, and everything else (query building, dictionary, ranking, snippets, highlights,
+fusion, the semantic model) in common Kotlin. The bridge between both is
+[Nucleus JVM Access](https://github.com/NucleusFramework/NucleusJvmAccess) (NJA).
+
+```
+search (Kotlin Multiplatform)                         search-facade (Kotlin/JVM): what runs on the JVM
+──────────────────────────────                        ─────────────────────────────────────────────────
+LexicalSearchEngine, MagicDictionaryIndex, snippets,
+highlights... use the façade's API as if it were      LuceneIndex(indexDir, cacheQueries): analyze, search,
+local: io.github...search.index.LuceneIndex,     ──▶  cursors, facets, collectLongs, count, KNN
+IndexQuery, IndexPage, KnnRequest... (generated by    IndexQuery (a sealed value), IndexPage, KnnRequest...
+NJA from the façade's sources)                        LuceneIndexBackend (internal): Lucene 10
+```
+
+| | Desktop (JVM) | Android 10+ | iOS |
+|---|---|---|---|
+| The JVM | the app's own | HotSpot (server VM, JIT), in the app's process next to ART | none: the façade compiled ahead of time into a GraalVM native image, linked into the app (iOS forbids generated code) |
+| A call | a plain call into NJA's dispatcher, on the calling thread | handed to a HotSpot thread paired with the calling one | a call into the image's entry point, on the calling thread |
+
+- `search-facade` holds the façade: its public API (`LuceneIndex`, `IndexCursor` and the value types: data classes,
+  enums, the sealed `IndexQuery`) is what NJA exports; `LuceneIndexBackend` is internal.
+- NJA generates, from the façade's sources, the proxies and value types `search`'s common code uses, in
+  `io.github.kdroidfilter.seforimlibrary.search.index` (`njaFacade.clientPackage`: apart from the façade's own package,
+  as the desktop has both on its class path). Query helpers built in the app (`bool { }`) stay in common code.
+- `LuceneIndex(path)` is the same constructor on every platform; `close()` closes the JVM's index.
+
+## The runtimes
+
+`nucleusJvm.iosRuntime` is `GRAALVM`: Zero, jdk-mobile's interpreter, searched about 8× slower than the native image
+(a first page 1.2 s against 140 ms in the simulator, on real data). Android keeps HotSpot: an Android native image
+builds on Linux only, and HotSpot is the fastest once warm.
+
+- **iOS**: NJA downloads [graalvm-mobile](https://github.com/NucleusFramework/graalvm-mobile)'s latest kit (GraalVM
+  CE's static libraries, for iOS) and the GraalVM CE release it names, for the build host, and compiles the façade
+  into `search/build/nja/nativeImage/<target>/nja.o` (`njaNativeImage<Target>`). The binaries link it with the kit's
+  libraries (`-export_dynamic`: NJA's client finds the image's entry points at run time): nothing to bundle. Lucene's
+  reflection and resources are in the façade's jar (`META-INF/native-image/.../reachability-metadata.json`), traced on
+  the JVM tests with the real index: `./gradlew :search:jvmTest -PindexDir=<seforim.db.lucene>
+  -PnativeImageAgent=<GraalVM home>` (then drop what the tests alone reach). The image reads the index through NIO.
+  For `search`'s own tests NJA does it all; an app links as Zayit's `searchkit` shows (`assembleIosJavaRuntimeDevice`:
+  `static/`, the object and the kit's libraries).
+
+Android's JVM is jdk-mobile's: a stable OpenJDK (27, `openjdk/jdk27u`'s GA releases) with openjdk/mobile's port,
+built by the CI of [NucleusFramework/openjdk-mobile](https://github.com/NucleusFramework/openjdk-mobile) and published
+as releases named like the JBR's, `jdk-mobile-<version>b<build>.<revision>` (`jdk-mobile-27b35.3`: `jdk-27-ga`, the
+JDK's build 35). NJA downloads the latest one (the runtimes `search` needs: arm64-v8a and x86_64 Android),
+checks them against their `.sha256`, keeps them in Gradle's cache and extracts them into `search/build/nja/jdkMobile`;
+`nucleusJvm.jdkMobileRelease` pins one.
+
+- **Android**: NJA links HotSpot and the JDK's natives into `libnja.so` (one per ABI: arm64-v8a, x86_64) and puts the
+  runtime (`java.home` per ABI, the façade's class path, the VM's options) in `search`'s assets, extracted once per
+  version on the device: an app depending on `search` gets both through the library, with nothing to configure.
+- **The desktop**: the façade is on the app's class path; nothing else.
+- The VM's options are `nucleusJvm.jvmOptions` (a 128 MB heap: about 140 MB of memory on Android; the native image
+  keeps the heap's maximum, about 60 MB of memory on iOS for Lucene alone).
+
+## Tests
+
+```bash
+./gradlew :search:jvmTest -PindexDir=<seforim.db.lucene> -PluceneQueryCache=off               # build/parity/jvm.txt
+./gradlew :search:iosSimulatorArm64Test -PindexDir=<seforim.db.lucene> -PluceneQueryCache=off # build/parity/ios.txt
+```
+
+They dump what the search answers on a real index (`SearchDump`). `-PluceneQueryCache=off` turns off Lucene's query
+cache: it changes the last digits of the scores from one search to the next. Two runs differ anyway by Lucene's
+process-dependent clause order (`BooleanQuery` rewrites through a `HashMap` keyed by class identity hashes), so the
+comparison allows score differences: facets, snippets and highlights must be identical. On Android, Zayit's
+`searchkit-check` app runs the whole search on a device or an emulator.
+
+Verified on the real index (7M documents): the desktop, the iOS simulator and Android 11 emulator give the same facets,
+snippets, highlights, titles and find-in-book candidates (Zayit's `searchkit`, hybrid with the semantic search: the
+same dump as the desktop, but for a hit total Lucene counts as a lower bound).

cache/build.gradle.kts

@@ -0,0 +1,68 @@
+
+plugins {
+    alias(libs.plugins.multiplatform)
+    alias(libs.plugins.android.library)
+    alias(libs.plugins.maven.publish)
+}
+
+group = "io.github.kdroidfilter.seforimlibrary"
+
+/** Caches read and written from several threads at once, on every platform (ConcurrentCache.kt). */
+kotlin {
+    jvmToolchain(libs.versions.jvmToolchain.get().toInt())
+
+    android {
+        namespace = "io.github.kdroidfilter.seforimlibrary.cache"
+        compileSdk = 35
+        minSdk = 21
+    }
+    jvm()
+    iosArm64()
+    iosSimulatorArm64()
+
+    sourceSets {
+        commonTest.dependencies {
+            implementation(kotlin("test"))
+            implementation(libs.kotlinx.coroutines.core)
+        }
+
+        // No java.util.concurrent on iOS: cachemap's map, lock-free reads
+        iosMain.dependencies {
+            implementation(libs.cachemap)
+        }
+    }
+
+}
+
+//Publishing your Kotlin Multiplatform library to Maven Central
+//https://www.jetbrains.com/help/kotlin-multiplatform-dev/multiplatform-publish-libraries.html
+mavenPublishing {
+    publishToMavenCentral()
+    coordinates("io.github.kdroidfilter.seforimlibrary", "cache", "1.0.0")
+
+    pom {
+        name = "SeforimLibrary"
+        description = "Kotlin Multiplatform library"
+        url = "github url" //todo
+
+        licenses {
+            license {
+                name = "MIT"
+                url = "https://opensource.org/licenses/MIT"
+            }
+        }
+
+        developers {
+            developer {
+                id = "" //todo
+                name = "" //todo
+                email = "" //todo
+            }
+        }
+
+        scm {
+            url = "github url" //todo
+        }
+    }
+    if (project.hasProperty("signing.keyId")) signAllPublications()
+}

cache/src/androidMain/kotlin/io/github/kdroidfilter/seforimlibrary/cache/ConcurrentCache.android.kt

@@ -0,0 +1,17 @@
+package io.github.kdroidfilter.seforimlibrary.cache
+
+import java.util.concurrent.ConcurrentHashMap
+
+actual fun <K : Any, V : Any> concurrentCache(): ConcurrentCache<K, V> =
+    object : ConcurrentCache<K, V> {
+        private val map = ConcurrentHashMap<K, V>()
+
+        override fun get(key: K): V? = map[key]
+
+        override fun set(
+            key: K,
+            value: V,
+        ) {
+            map[key] = value
+        }
+    }

cache/src/commonMain/kotlin/io/github/kdroidfilter/seforimlibrary/cache/ConcurrentCache.kt

@@ -0,0 +1,14 @@
+package io.github.kdroidfilter.seforimlibrary.cache
+
+/** A cache read and written from several threads at once: concurrent lookups of the same key may both load it. */
+interface ConcurrentCache<K : Any, V : Any> {
+    operator fun get(key: K): V?
+
+    operator fun set(
+        key: K,
+        value: V,
+    )
+}
+
+/** On the JVM and Android, ConcurrentHashMap; on iOS (no java.util.concurrent), cachemap's lock-free reads. */
+expect fun <K : Any, V : Any> concurrentCache(): ConcurrentCache<K, V>

cache/src/commonTest/kotlin/io/github/kdroidfilter/seforimlibrary/cache/ConcurrentCacheTest.kt

@@ -0,0 +1,31 @@
+package io.github.kdroidfilter.seforimlibrary.cache
+
+import kotlinx.coroutines.Dispatchers
+import kotlinx.coroutines.async
+import kotlinx.coroutines.awaitAll
+import kotlinx.coroutines.runBlocking
+import kotlin.test.Test
+import kotlin.test.assertEquals
+import kotlin.test.assertNull
+
+class ConcurrentCacheTest {
+    @Test
+    fun storesAndReplacesValues() {
+        val cache = concurrentCache<String, Int>()
+        assertNull(cache["a"])
+        cache["a"] = 1
+        cache["a"] = 2
+        assertEquals(2, cache["a"])
+    }
+
+    @Test
+    fun keepsEveryValueWrittenFromSeveralThreads() =
+        runBlocking {
+            val cache = concurrentCache<Int, Int>()
+            (0 until 8)
+                .map { worker ->
+                    async(Dispatchers.Default) { for (i in 0 until 1_000) cache[worker * 1_000 + i] = i }
+                }.awaitAll()
+            for (key in 0 until 8_000) assertEquals(key % 1_000, cache[key])
+        }
+}

cache/src/iosMain/kotlin/io/github/kdroidfilter/seforimlibrary/cache/ConcurrentCache.ios.kt

@@ -0,0 +1,17 @@
+package io.github.kdroidfilter.seforimlibrary.cache
+
+import io.github.charlietap.cachemap.cacheMapOf
+
+actual fun <K : Any, V : Any> concurrentCache(): ConcurrentCache<K, V> =
+    object : ConcurrentCache<K, V> {
+        private val map = cacheMapOf<K, V>()
+
+        override fun get(key: K): V? = map[key]
+
+        override fun set(
+            key: K,
+            value: V,
+        ) {
+            map[key] = value
+        }
+    }

cache/src/jvmMain/kotlin/io/github/kdroidfilter/seforimlibrary/cache/ConcurrentCache.jvm.kt

@@ -0,0 +1,17 @@
+package io.github.kdroidfilter.seforimlibrary.cache
+
+import java.util.concurrent.ConcurrentHashMap
+
+actual fun <K : Any, V : Any> concurrentCache(): ConcurrentCache<K, V> =
+    object : ConcurrentCache<K, V> {
+        private val map = ConcurrentHashMap<K, V>()
+
+        override fun get(key: K): V? = map[key]
+
+        override fun set(
+            key: K,
+            value: V,
+        ) {
+            map[key] = value
+        }
+    }

core/build.gradle.kts

@@ -12,11 +12,13 @@ kotlin {
     jvmToolchain(libs.versions.jvmToolchain.get().toInt())
 
     android {
-        namespace = "io.github.kdroidfilter.seforimlibrary"
+        namespace = "io.github.kdroidfilter.seforimlibrary.core"
         compileSdk = 35
         minSdk = 21
     }
     jvm()
+    iosArm64()
+    iosSimulatorArm64()
 
     sourceSets {
         commonMain.dependencies {

dao/build.gradle.kts

@@ -14,20 +14,24 @@ kotlin {
     jvmToolchain(libs.versions.jvmToolchain.get().toInt())
 
     android {
-        namespace = "io.github.kdroidfilter.seforimlibrary"
+        namespace = "io.github.kdroidfilter.seforimlibrary.dao"
         compileSdk = 35
         minSdk = 21
     }
     jvm()
+    iosArm64()
+    iosSimulatorArm64()
 
     sourceSets {
         commonMain.dependencies {
             api(project(":core"))
+            implementation(project(":cache"))
             implementation(libs.kotlinx.coroutines.core)
             implementation(libs.kotlinx.serialization.json)
             implementation(libs.kotlinx.serialization.protobuf)
             implementation(libs.kotlinx.datetime)
             implementation(libs.kermit)
+            implementation(libs.kotlinx.io.core)
         }
 
         commonTest.dependencies {
@@ -48,6 +52,10 @@ kotlin {
             implementation(libs.sqlDelight.driver.sqlite)
         }
 
+        iosMain.dependencies {
+            implementation(libs.sqlDelight.driver.native)
+        }
+
     }
 
 }