feat(knowledge): filter knowledge bases by availability status by KevinAndrewDong · Pull Request #45322 · langgenius/dify

KevinAndrewDong · 2026-11-02T11:58:16Z

Summary

Fixes #43848. Adds a Status filter to the Knowledge list so available and unavailable knowledge bases can be reviewed separately.

GET /console/api/datasets accepts a new optional availability query parameter: all (default, identical to today), available, or unavailable.
The filter is applied server-side inside the same query that produces the page and total, before pagination, and composes with workspace permissions, keyword search, tag matching, the creator filter and the existing ids filter.
The predicate is the same rule the knowledge cards already use for embedding_available: economy indexing needs no embedding model, and high-quality indexing is available when its stored embedding model is an active text embedding model in the workspace. The rule now lives in one function (api/services/knowledge/datasets/availability.py) shared by the card flag and the filter, so the two cannot drift.
Web: a Status chip sits beside the existing tag, creator and search filters with All / Available / Unavailable (All is the default). Changing it starts the list over at page 1 because the filter is part of the infinite-query input; combining it with search and tags keeps sending them together.
i18n added for en-US and zh-Hans; the remaining locales are handled by the repository translation workflow.

Availability stays a configuration-state check, not a live model health probe.

Relation to #43853

#43853 (still open) adds reranking_available to the card payload. I deliberately did not copy its card/API changes. This PR filters on the availability state that cards expose on main today. When #43853 lands, the reranking check should be folded into the shared predicate in api/services/knowledge/datasets/availability.py (and the adapter's request-time evaluation), then the filter is retested against the updated card logic. I am happy to rebase and retest once that PR is merged instead of depending on an unmerged branch.

Screenshots

Before	After
Knowledge toolbar has tag, creator and search filters only.	A Status chip (All / Available / Unavailable) sits between the creator filter and search.

Screenshots were not captured because this environment has no browser session; the chip reuses the existing Knowledge/Dataset filter chip styling and the menu radio pattern already used by the snippets list.

Checklist

This change requires a documentation update, included: Dify Document
No separate docs update is needed for this in-repo UI/API change; the console OpenAPI Markdown and generated contracts are regenerated in this PR.
I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
Discussed on [Feature Request] Filter knowledge bases by availability status #43848 and maintainer approval was given to work on the filtering portion.
I've verified the change and added or updated tests where meaningful regression risk justifies coverage.
I've updated the documentation accordingly.
I ran make lint && make type-check (backend) and vp staged (frontend) to appease the lint gods

Tests (real local runs)

Backend:

uv run --project api --dev pytest -o addopts="" -q api/tests/unit_tests/services/knowledge/datasets api/tests/unit_tests/services/knowledge/test_dataset_service.py api/tests/unit_tests/controllers/console/datasets/test_datasets.py api/tests/unit_tests/controllers/openapi/test_knowledge_bases.py — 327 passed, 0 failed.
Coverage includes filtering before pagination and totals, keyword+tag combinations, RBAC/permission scope, active model configuration changes, requested-id filtering, legacy provider normalization, and filter/card agreement through the application service.
ruff format / ruff check ./api: passed.
pyrefly on the changed backend files: passed.
lint-imports (architecture contracts): 82 kept, 0 broken.
Response contract lint: 762 valid, 0 mismatch.

Frontend:

vp test --project unit app/components/datasets/list/__tests__/use-dataset-list.spec.ts app/components/datasets/list/__tests__/index.spec.tsx — 33 passed, 0 failed.
vp lint on the changed files: 0 warnings, 0 errors.
tsrs --runExternalCode --noEmit (web type check): passed.
vp fmt on the changed files: passed.
pnpm -C web i18n:check-types: passed.

Generated artifacts: pnpm --dir packages/contracts gen-api-contract and api/dev/generate_swagger_markdown_docs.py were rerun locally; the checked-in contract/Markdown diffs come from those scripts.

Not run: the full repository unit suites, browser tests, and the Docker-backed integration suites (CI-owned / not runnable in this environment).

From DeepSeek Harness

codecov · 2026-11-03T02:17:23Z

Codecov Report

❌ Patch coverage is 94.59459% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.77%. Comparing base (24da932) to head (b3e7c79).
⚠️ Report is 16 commits behind head on main.

Files with missing lines	Patch %	Lines
api/services/knowledge/dataset_service.py	77.77%	1 Missing and 1 partial ⚠️
api/services/knowledge/datasets/adapters.py	88.88%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #45322    +/-   ##
========================================
  Coverage   89.77%   89.77%            
========================================
  Files        6238     6242     +4     
  Lines      390586   390964   +378     
  Branches    79215    79298    +83     
========================================
+ Hits       350631   350998   +367     
- Misses      33863    33867     +4     
- Partials     6092     6099     +7

Flag	Coverage Δ
api	`89.63% <92.00%> (-0.01%)`	⬇️
cli	`94.26% <ø> (ø)`
dify-ui	`92.39% <ø> (ø)`
web	`89.75% <100.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-11-03T02:17:57Z

Pyrefly Diff

base → PR

--- /tmp/pyrefly_base.txt	2026-11-03 02:17:10.211862760 +0000
+++ /tmp/pyrefly_pr.txt	2026-11-03 02:16:57.491746769 +0000
@@ -732,6 +732,8 @@
   --> tests/e2e_tests/containers/services/test_dataset_service_permissions.py:42:58
 ERROR Argument `Literal['active']` is not assignable to parameter `status` with type `AccountStatus | SQLCoreOperations[AccountStatus]` in function `models.account.Account.__init__` [bad-argument-type]
   --> tests/e2e_tests/containers/services/test_dataset_service_permissions.py:69:20
+ERROR Argument `str` is not assignable to parameter `value` with type `IndexingStatus | SQLCoreOperations[IndexingStatus]` in function `sqlalchemy.orm.base.Mapped.__set__` [bad-argument-type]
+  --> tests/e2e_tests/containers/services/test_document_service_display_status.py:50:32
 ERROR Argument `Literal['completed']` is not assignable to parameter `value` with type `IndexingStatus | SQLCoreOperations[IndexingStatus]` in function `sqlalchemy.orm.base.Mapped.__set__` [bad-argument-type]
   --> tests/e2e_tests/containers/services/test_document_service_rename_document.py:78:27
 ERROR `None` is not subscriptable [unsupported-operation]
@@ -2114,6 +2116,10 @@
    --> tests/helpers/workflow/table_runner.py:390:37
 WARN `URLError` is already caught earlier in this `try` statement, so it never matches here [unreachable-except-clause]
    --> tests/pytest_dify.py:155:34
+ERROR Object of class `object` has no attribute `package` [missing-attribute]
+   --> tests/unit_tests/commands/test_data_migration.py:218:16
+ERROR Object of class `object` has no attribute `options_override` [missing-attribute]
+   --> tests/unit_tests/commands/test_data_migration.py:219:16
 ERROR Argument `list[FromClause]` is not assignable to parameter `tables` with type `Sequence[Table] | None` in function `sqlalchemy.sql.schema.MetaData.create_all` [bad-argument-type]
    --> tests/unit_tests/commands/test_retention.py:188:20
 ERROR Expected a callable, got `None` [not-callable]
@@ -2156,6 +2162,8 @@
    --> tests/unit_tests/context/test_flask_app_context.py:266:20
 ERROR Object of class `Flask` has no attribute `_get_current_object` [missing-attribute]
    --> tests/unit_tests/context/test_flask_app_context.py:295:20
+ERROR Object of class `Task` has no attribute `queue` [missing-attribute]
+   --> tests/unit_tests/controllers/console/app/test_annotation.py:100:16
 ERROR Object of class `FromClause` has no attribute `create` [missing-attribute]
    --> tests/unit_tests/controllers/console/app/test_app.py:173:13
 ERROR Argument value `Literal[0]` violates Pydantic `ge` constraint `Literal[1]` for field `page` [bad-argument-type]
@@ -2296,6 +2304,8 @@
    --> tests/unit_tests/controllers/console/datasets/rag_pipeline/test_rag_pipeline_workflow.py:876:9
 ERROR Argument `datetime | None` is not assignable to parameter `value` with type `SQLCoreOperations[datetime] | datetime` in function `sqlalchemy.orm.base.Mapped.__set__` [bad-argument-type]
    --> tests/unit_tests/controllers/console/datasets/rag_pipeline/test_rag_pipeline_workflow.py:924:27
+ERROR Argument `Literal['unavailable']` is not assignable to parameter `availability` with type `DatasetAvailabilityFilter` in function `services.knowledge.datasets.application.DatasetListFilter.__init__` [bad-argument-type]
+   --> tests/unit_tests/controllers/console/datasets/test_datasets.py:141:26
 ERROR Unpacked keyword argument `int | str | Unknown` is not assignable to parameter `tenant_id` with type `SQLCoreOperations[str] | str` in function `models.dataset.Document.__init__` [bad-argument-type]
    --> tests/unit_tests/controllers/console/datasets/test_datasets_document.py:615:9
 ERROR Unpacked keyword argument `int | str | Unknown` is not assignable to parameter `dataset_id` with type `SQLCoreOperations[str] | str` in function `models.dataset.Document.__init__` [bad-argument-type]
@@ -2884,14 +2894,8 @@
    --> tests/unit_tests/controllers/service_api/test_wraps.py:415:16
 ERROR Argument `Literal['abnormal']` is not assignable to parameter `value` with type `AppStatus | SQLCoreOperations[AppStatus]` in function `sqlalchemy.orm.base.Mapped.__set__` [bad-argument-type]
    --> tests/unit_tests/controllers/service_api/test_wraps.py:450:28
-ERROR Object of class `NoneType` has no attribute `tenant_id` [missing-attribute]
-   --> tests/unit_tests/controllers/service_api/test_wraps.py:893:16
-ERROR Object of class `NoneType` has no attribute `dataset_ids` [missing-attribute]
-   --> tests/unit_tests/controllers/service_api/test_wraps.py:894:16
-ERROR Object of class `NoneType` has no attribute `dataset_ids` [missing-attribute]
-   --> tests/unit_tests/controllers/service_api/test_wraps.py:961:16
 ERROR Object of class `NoneType` has no attribute `last_used_at` [missing-attribute]
-   --> tests/unit_tests/controllers/service_api/test_wraps.py:963:16
+   --> tests/unit_tests/controllers/service_api/test_wraps.py:951:16
 ERROR `object` is not assignable to attribute `request` with type `Request` [bad-assignment]
   --> tests/unit_tests/controllers/trigger/test_trigger.py:11:22
 ERROR Object of class `tuple` has no attribute `status_code` [missing-attribute]
@@ -5926,6 +5930,20 @@
   --> tests/unit_tests/services/enterprise/test_plugin_manager_service.py:59:25
 ERROR Argument `None` is not assignable to parameter `response` with type `Response` in function `httpx._exceptions.HTTPStatusError.__init__` [bad-argument-type]
   --> tests/unit_tests/services/enterprise/test_plugin_manager_service.py:60:26
+ERROR Unpacked keyword argument `object` is not assignable to parameter `page` with type `int` in function `services.knowledge.datasets.application.DatasetListFilter.__init__` [bad-argument-type]
+   --> tests/unit_tests/services/knowledge/datasets/test_adapters.py:495:54
+ERROR Unpacked keyword argument `object` is not assignable to parameter `limit` with type `int` in function `services.knowledge.datasets.application.DatasetListFilter.__init__` [bad-argument-type]
+   --> tests/unit_tests/services/knowledge/datasets/test_adapters.py:495:54
+ERROR Unpacked keyword argument `object` is not assignable to parameter `keyword` with type `str | None` in function `services.knowledge.datasets.application.DatasetListFilter.__init__` [bad-argument-type]
+   --> tests/unit_tests/services/knowledge/datasets/test_adapters.py:495:54
+ERROR Unpacked keyword argument `object` is not assignable to parameter `include_all` with type `bool` in function `services.knowledge.datasets.application.DatasetListFilter.__init__` [bad-argument-type]
+   --> tests/unit_tests/services/knowledge/datasets/test_adapters.py:495:54
+ERROR Unpacked keyword argument `object` is not assignable to parameter `ids` with type `Sequence[str]` in function `services.knowledge.datasets.application.DatasetListFilter.__init__` [bad-argument-type]
+   --> tests/unit_tests/services/knowledge/datasets/test_adapters.py:495:54
+ERROR Unpacked keyword argument `object` is not assignable to parameter `tag_ids` with type `Sequence[str]` in function `services.knowledge.datasets.application.DatasetListFilter.__init__` [bad-argument-type]
+   --> tests/unit_tests/services/knowledge/datasets/test_adapters.py:495:54
+ERROR Unpacked keyword argument `object` is not assignable to parameter `creator_ids` with type `Sequence[str]` in function `services.knowledge.datasets.application.DatasetListFilter.__init__` [bad-argument-type]
+   --> tests/unit_tests/services/knowledge/datasets/test_adapters.py:495:54
 ERROR Argument `str` is not assignable to parameter `type` with type `Literal['basic', 'bearer', 'custom'] | None` in function `services.entities.external_knowledge_entities.external_knowledge_entities.AuthorizationConfig.__init__` [bad-argument-type]
    --> tests/unit_tests/services/knowledge/external/test_service.py:116:60
 ERROR Argument `str` is not assignable to parameter `type` with type `Literal['api-key', 'no-auth']` in function `services.entities.external_knowledge_entities.external_knowledge_entities.Authorization.__init__` [bad-argument-type]
@@ -5945,41 +5963,41 @@
 ERROR `None` is not subscriptable [unsupported-operation]
    --> tests/unit_tests/services/knowledge/metadata/test_application.py:434:20
 ERROR Object of class `NoneType` has no attribute `keyword_number` [missing-attribute]
-    --> tests/unit_tests/services/knowledge/test_dataset_service.py:1519:20
+    --> tests/unit_tests/services/knowledge/test_dataset_service.py:1438:20
 ERROR Argument `str` is not assignable to parameter `data_source_type` with type `DataSourceType | SQLCoreOperations[DataSourceType]` in function `models.dataset.Document.__init__` [bad-argument-type]
-    --> tests/unit_tests/services/knowledge/test_dataset_service.py:1834:26
+    --> tests/unit_tests/services/knowledge/test_dataset_service.py:1753:26
 ERROR Argument `str` is not assignable to parameter `indexing_status` with type `IndexingStatus | SQLCoreOperations[IndexingStatus]` in function `models.dataset.Document.__init__` [bad-argument-type]
-    --> tests/unit_tests/services/knowledge/test_dataset_service.py:1840:25
+    --> tests/unit_tests/services/knowledge/test_dataset_service.py:1759:25
 ERROR Argument `Literal['opendal']` is not assignable to parameter `storage_type` with type `StorageType` in function `tests.helpers.unit.model_factories.make_upload_file` [bad-argument-type]
-    --> tests/unit_tests/services/knowledge/test_dataset_service.py:1857:22
+    --> tests/unit_tests/services/knowledge/test_dataset_service.py:1776:22
 ERROR Argument `str` is not assignable to parameter `mode` with type `ProcessRuleMode | SQLCoreOperations[ProcessRuleMode]` in function `models.dataset.DatasetProcessRule.__init__` [bad-argument-type]
-    --> tests/unit_tests/services/knowledge/test_dataset_service.py:1878:37
+    --> tests/unit_tests/services/knowledge/test_dataset_service.py:1797:37
 ERROR Argument `Literal['opendal']` is not assignable to parameter `storage_type` with type `StorageType` in function `models.model.UploadFile.__init__` [bad-argument-type]
-    --> tests/unit_tests/services/knowledge/test_dataset_service.py:2118:30
+    --> tests/unit_tests/services/knowledge/test_dataset_service.py:2037:30
 ERROR Argument `Literal['account']` is not assignable to parameter `created_by_role` with type `CreatorUserRole` in function `models.model.UploadFile.__init__` [bad-argument-type]
-    --> tests/unit_tests/services/knowledge/test_dataset_service.py:2124:33
+    --> tests/unit_tests/services/knowledge/test_dataset_service.py:2043:33
 ERROR `None` is not subscriptable [unsupported-operation]
-    --> tests/unit_tests/services/knowledge/test_dataset_service.py:2149:20
+    --> tests/unit_tests/services/knowledge/test_dataset_service.py:2068:20
 ERROR Object of class `NoneType` has no attribute `name` [missing-attribute]
-    --> tests/unit_tests/services/knowledge/test_dataset_service.py:2150:20
+    --> tests/unit_tests/services/knowledge/test_dataset_service.py:2069:20
 ERROR Argument `Literal['opendal']` is not assignable to parameter `storage_type` with type `StorageType` in function `models.model.UploadFile.__init__` [bad-argument-type]
-    --> tests/unit_tests/services/knowledge/test_dataset_service.py:2592:30
+    --> tests/unit_tests/services/knowledge/test_dataset_service.py:2511:30
 ERROR Argument `Literal['account']` is not assignable to parameter `created_by_role` with type `CreatorUserRole` in function `models.model.UploadFile.__init__` [bad-argument-type]
-    --> tests/unit_tests/services/knowledge/test_dataset_service.py:2598:33
+    --> tests/unit_tests/services/knowledge/test_dataset_service.py:2517:33
 ERROR Object of class `NoneType` has no attribute `status` [missing-attribute]
-    --> tests/unit_tests/services/knowledge/test_dataset_service.py:2677:20
+    --> tests/unit_tests/services/knowledge/test_dataset_service.py:2596:20
 ERROR Object of class `NoneType` has no attribute `status` [missing-attribute]
-    --> tests/unit_tests/services/knowledge/test_dataset_service.py:2776:20
+    --> tests/unit_tests/services/knowledge/test_dataset_service.py:2695:20
 ERROR Argument `list[PreProcessingRule] | None` is not assignable to parameter `obj` with type `Sized` in function `len` [bad-argument-type]
-    --> tests/unit_tests/services/knowledge/test_dataset_service.py:2880:24
+    --> tests/unit_tests/services/knowledge/test_dataset_service.py:2799:24
 ERROR `None` is not subscriptable [unsupported-operation]
-    --> tests/unit_tests/services/knowledge/test_dataset_service.py:2881:20
+    --> tests/unit_tests/services/knowledge/test_dataset_service.py:2800:20
 ERROR Object of class `NoneType` has no attribute `rules` [missing-attribute]
-    --> tests/unit_tests/services/knowledge/test_dataset_service.py:3881:42
+    --> tests/unit_tests/services/knowledge/test_dataset_service.py:3800:42
 ERROR Argument `Literal['completed']` is not assignable to parameter `indexing_status` with type `IndexingStatus | SQLCoreOperations[IndexingStatus]` in function `models.dataset.Document.__init__` [bad-argument-type]
-    --> tests/unit_tests/services/knowledge/test_dataset_service.py:4316:25
+    --> tests/unit_tests/services/knowledge/test_dataset_service.py:4235:25
 ERROR Argument `str` is not assignable to parameter `doc_form` with type `IndexStructureType | SQLCoreOperations[IndexStructureType]` in function `models.dataset.Document.__init__` [bad-argument-type]
-    --> tests/unit_tests/services/knowledge/test_dataset_service.py:4317:18
+    --> tests/unit_tests/services/knowledge/test_dataset_service.py:4236:18
 ERROR Object of class `FromClause` has no attribute `create` [missing-attribute]
   --> tests/unit_tests/services/plugin/test_plugin_parameter_service.py:47:9
 ERROR Object of class `NoneType` has no attribute `marked_name` [missing-attribute]

github-actions · 2026-11-03T02:18:00Z

Pyrefly Type Coverage

Metric	Base	PR	Delta
Type coverage	82.88%	82.72%	-0.16%
Strict coverage	82.54%	82.38%	-0.16%
Typed symbols	85,474	85,333	-141
Untyped symbols	17,725	17,896	+171
Modules	4373	4374	+1

Add an `availability` query parameter to the console dataset list so the Knowledge list can show all, available-only, or unavailable-only knowledge bases. The filter is applied inside the same query as the page and total, before pagination, and composes with workspace permissions, keyword search, tag matching, the creator filter, and requested dataset ids. The availability rule now lives in one shared function used by both the `embedding_available` card flag and the filter, so a filtered list cannot disagree with the badge on a card. Legacy provider ids are normalized the same way list responses already normalize them. The web list exposes a Status chip beside the tag, creator, and search filters, defaults to All, and restarts at the first page when the filter changes. English and Simplified Chinese copy added; other locales follow the repository translation workflow. Refs langgenius#43848.

KevinAndrewDong requested review from JohnJyong, QuantumGhost, WTW0313, crazywoola, iamjoel and laipz8200 as code owners November 2, 2026 11:58

github-actions Bot added size:XL This PR changes 500-999 lines, ignoring generated files. web This relates to changes on the web. labels Nov 2, 2026

KevinAndrewDong mentioned this pull request Nov 2, 2026

[Feature Request] Filter knowledge bases by availability status #43848

Open

4 of 5 tasks

KevinAndrewDong force-pushed the feat/knowledge-availability-filter branch from b3e7c79 to c9ccde2 Compare November 3, 2026 13:12

Navigation Menu

Repository navigation

feat(knowledge): filter knowledge bases by availability status - #45322

feat(knowledge): filter knowledge bases by availability status#45322
KevinAndrewDong wants to merge 1 commit into
langgenius:mainfrom
KevinAndrewDong:feat/knowledge-availability-filter

KevinAndrewDong commented Nov 2, 2026

Uh oh!

codecov Bot commented Nov 3, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 3, 2026

Uh oh!

github-actions Bot commented Nov 3, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

KevinAndrewDong commented Nov 2, 2026

Summary

Relation to #43853

Screenshots

Checklist

Tests (real local runs)

Uh oh!

codecov Bot commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Nov 3, 2026

Pyrefly Diff

Uh oh!

github-actions Bot commented Nov 3, 2026

Pyrefly Type Coverage

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

codecov Bot commented Nov 3, 2026 •

edited

Loading