SeforimLibrary

A Kotlin Multiplatform library + JVM tooling to build and query a unified SQLite database of Jewish texts (Sefaria + Otzaria), with Lucene-based full-text search.

Overview

SeforimLibrary is a comprehensive solution for working with Jewish religious texts from the Otzaria database. The project converts the original Otzaria database into a modern SQLite database with full-text search capabilities using FTS5, making it efficient to search through large volumes of text.

The library is structured as a set of modules that can be imported via Maven:

• core: Contains data models representing entities like books, authors, categories, and lines of text
• dao: Provides database access objects and repositories for interacting with the SQLite database
• otzariasqlite: Otzaria import/enrichment tooling (append into an existing DB)
• catalog: Precomputed catalog builder (catalog.pb)
• searchindex: Lucene index builders (text + lookup)
• packaging: Release/bundling tooling (release info + .tar.zst bundle)
• sefariasqlite: One-step Sefaria export → SQLite importer

Generation tooling modules are grouped under generator/ on disk (Gradle module names stay :sefariasqlite, :otzariasqlite, etc.).

Features

• Convert Otzaria database to SQLite format
• Efficient full-text search using SQLite's FTS5
• Hierarchical category and book organization
• Table of contents navigation for books
• Support for links between related texts
• Comprehensive data model for Jewish religious texts

Demo / App

SeforimLibrary is typically consumed from the SeforimApp/ project in the parent repo.

Requirements

• JDK 11 or higher
• Kotlin 1.9.0 or higher
• SQLite 3.35.0 or higher (for FTS5 support)

Usage

Initializing the Database

// Initialize the database
val dbPath = "path/to/your/database.db"
val driver = JdbcSqliteDriver(url = "jdbc:sqlite:$dbPath")
val repository = SeforimRepository(dbPath, driver)

Searching for Text

// Search in all books
val searchResults = repository.search("your search query", limit = 20, offset = 0)

// Search in a specific book
val bookSearchResults = repository.searchInBook(bookId, "your search query")

// Search by author
val authorSearchResults = repository.searchByAuthor("author name", "your search query")

Browsing Categories and Books

// Get root categories
val rootCategories = repository.getRootCategories()

// Get subcategories
val subcategories = repository.getCategoryChildren(parentId)

// Get books in a category
val books = repository.getBooksByCategory(categoryId)

Reading Book Content

// Get book details
val book = repository.getBook(bookId)

// Get lines of text
val lines = repository.getLines(bookId, startIndex, endIndex)

// Get table of contents
val toc = repository.getBookToc(bookId)

Database Generation

Recommended pipeline (Sefaria → SQLite → Otzaria)

./gradlew generateSeforimDb

Outputs (by default): build/seforim.db + build/catalog.pb

Full pipeline (DB + indexes + bundle)

./gradlew packageSeforimBundle

Manual steps

# 1) Base DB from Sefaria
./gradlew :sefariasqlite:generateSefariaSqlite

# 2) Append Otzaria (lines + links)
./gradlew :otzariasqlite:appendOtzaria

# 3) Build precomputed catalog
./gradlew :catalog:buildCatalog

# 4) Build Lucene indexes (creates build/seforim.db.lucene + build/seforim.db.lookup.lucene)
./gradlew :searchindex:buildLuceneIndexDefault

# 5) Download lexical.db next to the DB (used by the app; auto-run by packaging)
./gradlew :packaging:downloadLexicalDb

# 6) Package everything into a single .tar.zst (plus split parts)
./gradlew :packaging:packageArtifacts

Project Structure

• core: Contains data models and extensions

  • models: Data classes representing entities in the database
  • extensions: Utility extensions for working with the models

• dao: Database access layer

  • repository: Repository classes for accessing the database
  • extensions: Extensions for converting between database and model objects
  • sqldelight: SQL queries and database schema

• generator/: Grouping folder for JVM generation tooling modules

• otzariasqlite: Otzaria enrichment tools

  • DatabaseGenerator: Converts Otzaria sources into SQLite rows
  • GenerateLines / GenerateLinks: phase tasks (see Gradle tasks)

• catalog: Precomputed catalog tools

  • BuildCatalog: builds catalog.pb from a SQLite DB

• searchindex: Lucene indexing tools

  • LuceneTextIndexWriter / LuceneLookupIndexWriter: JVM Lucene writers
  • BuildLuceneIndex: CLI entrypoint used by Gradle tasks

• packaging: Release/bundling tools

  • WriteReleaseInfo / PackageArtifacts: bundle .tar.zst for distribution

• sefariasqlite: Sefaria direct importer

  • SefariaDirectImporter / GenerateSefariaSqlite: Sefaria export → SQLite

Contributing

Please read CONTRIBUTING.md first. In short: contributions are licensed under the same terms as the project (the Zayit Public License), you keep your copyright, there is no CLA to sign, and every commit must be signed off (git commit -s, DCO).

License

SeforimLibrary is part of Zayit and is published under the Zayit Public License: the GNU AGPL v3, plus a grant back to the maintainer, an attribution term and a trademark clause. Its attribution term (Section 4): any use, reproduction, modification, or distribution of this software, or of any work derived from it, must conspicuously display:

Powered by the technologies that drive Zayit — https://zayitapp.com/

See the top of LICENSE for the full text.

Like Zayit, it is open core: Section 3 of the license (the grant back) also lets the maintainer distribute contributions, and the modifications of any fork, under other terms, including in Zayit's official builds with their proprietary modules (see "Open core" in Zayit's README).