-
Notifications
You must be signed in to change notification settings - Fork 55
feat: support bulk-export command #261
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
ohadedry
wants to merge
13
commits into
microsoft:main
Choose a base branch
from
ohadedry:support-bulk-export-command
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+19,699
−96
Open
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
1aec777
feat: support bulk-export command
ohadedry 1c2c809
Merge branch 'main' into support-bulk-export-command
ohadedry ebfd5f6
Fix lint issue, return dict
ohadedry d760057
Fix changie file
ohadedry 50cd1b3
Review fixes
ohadedry 0a15d1d
fix tests, remove test which is no longer required
ohadedry a107136
Add more tests and review fixes
ohadedry 87d103b
fix doc and remove redundant check
ohadedry 4ca5a7a
review fixes
ohadedry 2292d22
fix bulk-export output
ohadedry 76046f2
review fix and supported items update
ohadedry 193cb89
remove unreachable code and add tests
ohadedry 6c9c36c
fix test typo
ohadedry File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| kind: added | ||
| body: Add new 'bulk-export' command for exporting workspace or folder items | ||
| time: 2026-07-06T09:11:26.916814199Z | ||
| custom: | ||
| Author: ohadedry | ||
| AuthorLink: https://github.com/ohadedry |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| # `bulk-export` Command | ||
|
|
||
| Export a workspace or folder in bulk while preserving folder structure and item bindings. | ||
|
|
||
| The `bulk-export` command exports all supported items from a workspace or folder in a single bulk API request, preserving the folder hierarchy and item bindings (logical IDs) in the exported definitions. | ||
|
|
||
| !!! warning "When exporting items, the item definitions are exported without their sensitivity labels" | ||
|
|
||
| !!! note "Permissions" | ||
| - **Workspace export**: Only items that the caller has both read and write permissions for are exported. | ||
| - **Folder export**: The caller must have read and write permissions for all folder items. | ||
|
ohadedry marked this conversation as resolved.
|
||
|
|
||
| !!! note "Differences from `export`" | ||
| - **Folder structure**: `bulk-export` preserves the workspace folder hierarchy in the output. `export` exports items flat. | ||
| - **Item bindings**: `bulk-export` preserves item logical IDs, enabling round-trip import/update. `export` uses nil UUIDs. | ||
| - **Format**: `bulk-export` does not support the `--format` flag. Items are exported in their default format only. | ||
| - **Target scope**: `bulk-export` operates on workspaces and folders. `export` operates on individual items or workspaces. | ||
|
|
||
| **Supported Targets:** | ||
|
|
||
| - `.Workspace` — exports all supported items in the workspace | ||
| - `.Folder` — exports all supported items in the folder and its sub-folders | ||
|
|
||
| **Supported Item Types:** | ||
|
|
||
| `CopyJob`, `CosmosDBDatabase`, `Dataflow`, `DataPipeline`, `DigitalTwinBuilder`, `DigitalTwinBuilderFlow`, `Environment`, `Eventhouse`, `Eventstream`, `GraphQLApi`, `GraphQuerySet`, `KQLDashboard`, `KQLDatabase`, `KQLQueryset`, `Lakehouse`, `Map`, `MirroredDatabase`, `MLExperiment`, `MLModel`, `MountedDataFactory`, `Notebook`, `Reflex`, `Report`, `SemanticModel`, `SparkJobDefinition`, `SQLDatabase`, `UserDataFunction`, `VariableLibrary` | ||
|
|
||
| Unsupported item types are automatically skipped and reported in the output summary. | ||
|
|
||
| **Usage:** | ||
|
|
||
| ``` | ||
| fab bulk-export <path> -o <output_path> --recursive [-f] | ||
|
ohadedry marked this conversation as resolved.
|
||
| ``` | ||
|
|
||
| **Parameters:** | ||
|
|
||
| | Parameter | Description | | ||
| |-----------|-------------| | ||
| | `<path>` | Path to the workspace or folder to export. | | ||
| | `-o, --output <output_path>` | Output directory path. Required. | | ||
| | `--recursive` | Recursively export folder contents. Required for workspace and folder targets. | | ||
| | `-f, --force` | Skip confirmation prompts. Exports without sensitivity label confirmation and proceeds when the output folder is not empty. Optional. | | ||
|
|
||
| **Examples:** | ||
|
|
||
| ```bash | ||
| # Bulk-export an entire workspace to a local directory | ||
| fab bulk-export ws1.Workspace -o /tmp --recursive --force | ||
|
|
||
| # Bulk-export a specific folder | ||
| fab bulk-export ws1.Workspace/folder1.Folder -o /tmp --recursive --force | ||
| ``` | ||
|
|
||
| **Output:** | ||
|
|
||
| On success, the command prints a summary with the number of exported and skipped items: | ||
|
|
||
| ``` | ||
| Exported 15 items to '/tmp'. Skipped 1 items due to unsupported item types: Dashboard (1) | ||
| ``` | ||
|
|
||
| When using `--output_format json`, the output includes structured data: | ||
|
|
||
| ```json | ||
| { | ||
| "exported": 15, | ||
| "exported_types": { | ||
| "SemanticModel": 5, | ||
| "Report": 4, | ||
| "Notebook": 3, | ||
| "DataPipeline": 3 | ||
| }, | ||
| "skipped": 1, | ||
| "skipped_types": { | ||
| "Dashboard": 1 | ||
| }, | ||
| "output": "/tmp" | ||
| } | ||
| ``` | ||
|
|
||
| **Output Directory Structure:** | ||
|
|
||
| The exported output mirrors the workspace folder structure: | ||
|
|
||
| ``` | ||
| <output_path>/ | ||
| ├── notebook1.Notebook/ | ||
| │ ├── .platform | ||
| │ └── notebook-content.ipynb | ||
| ├── Folder1/ | ||
| │ ├── notebook2.Notebook/ | ||
| │ │ ├── .platform | ||
| │ │ └── notebook-content.ipynb | ||
| │ └── report1.Report/ | ||
| │ ├── .platform | ||
| │ ├── definition.pbir | ||
| │ └── StaticResources/ | ||
| │ └── ... | ||
| └── Folder2/ | ||
| └── model1.SemanticModel/ | ||
| ├── .platform | ||
| └── definition/ | ||
| ├── model.tmdl | ||
| └── ... | ||
| ``` | ||
|
ohadedry marked this conversation as resolved.
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| # Copyright (c) Microsoft Corporation. | ||
| # Licensed under the MIT License. |
70 changes: 70 additions & 0 deletions
70
src/fabric_cli/commands/fs/bulk_export/fab_fs_bulk_export_folder.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| # Copyright (c) Microsoft Corporation. | ||
| # Licensed under the MIT License. | ||
|
|
||
| from argparse import Namespace | ||
| from copy import deepcopy | ||
| import json | ||
|
|
||
| from fabric_cli.core.hiearchy.fab_folder import Folder | ||
| from fabric_cli.core.fab_exceptions import FabricCLIError | ||
| from fabric_cli.core import fab_constant | ||
| from fabric_cli.client import fab_api_item as item_api | ||
| from fabric_cli.errors.bulk_export import BulkExportErrors | ||
| from fabric_cli.core.hiearchy.fab_hiearchy import Item | ||
| from fabric_cli.utils import fab_cmd_fs_utils as utils_fs | ||
| from fabric_cli.utils.fab_cmd_bulk_export_utils import ContextItemsSupportMap | ||
| from fabric_cli.utils import fab_cmd_bulk_export_utils as bulk_export_utils | ||
|
|
||
|
|
||
| def bulk_export_folder(context: Folder, args: Namespace) -> None: | ||
| """Bulk-export a folder with its contents, including sub-folders recursively.""" | ||
| args = deepcopy(args) | ||
|
|
||
| workspace_id = context.workspace.id | ||
| args.ws_id = workspace_id | ||
| args.from_path = context.path.strip("/") | ||
|
|
||
| items_support = _collect_context_items(context) | ||
| if not items_support["supported_items"]: | ||
| error_message = ( | ||
| BulkExportErrors.empty_target(context.full_name) | ||
| if not items_support["unsupported_items"] | ||
| else BulkExportErrors.no_exportable_items() | ||
| ) | ||
| raise FabricCLIError( | ||
| error_message, | ||
| fab_constant.ERROR_INVALID_OPERATION, | ||
| ) | ||
|
|
||
| exported_item_ids = [item.id for item in items_support["supported_items"]] | ||
| payload = bulk_export_utils.create_bulk_export_payload(exported_item_ids) | ||
| response = item_api.bulk_export_definitions(args, payload) | ||
| exported_definitions = json.loads(response.text) | ||
| bulk_export_utils.export_definition_parts_to_storage( | ||
| args, context.full_name, exported_definitions | ||
| ) | ||
| bulk_export_utils.print_bulk_export_summary(args, items_support) | ||
|
|
||
|
|
||
| def _collect_context_items( | ||
| context: Folder, | ||
| ) -> ContextItemsSupportMap: | ||
| """Recursively collect all item IDs under a folder and its sub-folders.""" | ||
| supported_items: list[Item] = [] | ||
| unsupported_items: list[Item] = [] | ||
| elements = utils_fs.get_ws_elements(context) | ||
|
|
||
| for element in elements: | ||
| if isinstance(element, Item): | ||
| try: | ||
| if bulk_export_utils.is_command_supported(element): | ||
| supported_items.append(element) | ||
| except FabricCLIError: | ||
| unsupported_items.append(element) | ||
|
ohadedry marked this conversation as resolved.
|
||
| pass | ||
| elif isinstance(element, Folder): | ||
| folder_items = _collect_context_items(element) | ||
| supported_items.extend(folder_items["supported_items"]) | ||
| unsupported_items.extend(folder_items["unsupported_items"]) | ||
|
|
||
| return {"supported_items": supported_items, "unsupported_items": unsupported_items} | ||
123 changes: 123 additions & 0 deletions
123
src/fabric_cli/commands/fs/bulk_export/fab_fs_bulk_export_workspace.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| # Copyright (c) Microsoft Corporation. | ||
| # Licensed under the MIT License. | ||
|
|
||
| from argparse import Namespace | ||
| from copy import deepcopy | ||
| import json | ||
|
|
||
| from fabric_cli.utils import fab_mem_store as utils_mem_store | ||
| from fabric_cli.core.hiearchy.fab_hiearchy import Item, Workspace | ||
| from fabric_cli.core.fab_exceptions import FabricCLIError | ||
| from fabric_cli.errors.bulk_export import BulkExportErrors | ||
| from fabric_cli.core import fab_constant | ||
| from fabric_cli.client import fab_api_item as item_api | ||
| from fabric_cli.utils.fab_cmd_bulk_export_utils import ContextItemsSupportMap | ||
| from fabric_cli.utils import fab_cmd_bulk_export_utils as bulk_export_utils | ||
|
|
||
|
|
||
| def bulk_export_workspace(context: Workspace, args: Namespace) -> None: | ||
| """Bulk-export a workspace with its contents, including sub-folders recursively.""" | ||
| args = deepcopy(args) | ||
|
|
||
| args.ws_id = context.id | ||
| args.from_path = context.path.strip("/") | ||
|
|
||
| ws_items = utils_mem_store.get_workspace_items(context) | ||
| if not ws_items: | ||
| raise FabricCLIError( | ||
| BulkExportErrors.empty_target(context.full_name), | ||
| fab_constant.ERROR_INVALID_OPERATION, | ||
| ) | ||
|
|
||
| # bulk export can be called with no items collected and create_bulk_export_payload will handle it as "export all", so we don't need to pre-collect items here like we do for folders. Just call bulk export directly with the workspace ID and let the API determine what to export based on the from_path. | ||
| payload = bulk_export_utils.create_bulk_export_payload(item_ids=[]) | ||
| response = item_api.bulk_export_definitions(args, payload) | ||
| exported_definitions = json.loads(response.text) | ||
|
|
||
| items_support = _filter_response_by_supported_items(exported_definitions, context) | ||
| if not items_support["supported_items"]: | ||
| raise FabricCLIError( | ||
| BulkExportErrors.no_exportable_items(), | ||
| fab_constant.ERROR_INVALID_OPERATION, | ||
| ) | ||
|
|
||
| bulk_export_utils.export_definition_parts_to_storage( | ||
| args, context.full_name, exported_definitions | ||
| ) | ||
| bulk_export_utils.print_bulk_export_summary(args, items_support) | ||
|
|
||
|
|
||
| def _filter_response_by_supported_items( | ||
| exported_definitions: dict, workspace: Workspace | ||
| ) -> ContextItemsSupportMap: | ||
| """Filter response items by matching IDs from itemDefinitionsIndex against known workspace items. | ||
|
|
||
| Uses itemDefinitionsIndex to identify returned items by ID, matches them against cached | ||
| workspace items, and checks command support. Mutates exported_definitions['definitionParts'] in-place | ||
| to remove parts belonging to unsupported or unknown item types. | ||
| Returns a map of supported/unsupported Item objects. | ||
| """ | ||
| items_definitions_index = exported_definitions.get("itemDefinitionsIndex", []) | ||
| parts = exported_definitions.get("definitionParts", []) | ||
|
|
||
| # Build a lookup of workspace items by ID | ||
| ws_items = utils_mem_store.get_workspace_items(workspace) | ||
| items_by_id: dict[str, Item] = {item.id: item for item in ws_items} | ||
|
|
||
| supported_item_prefixes: set[str] = set() | ||
| supported_items_list: list[Item] = [] | ||
| unsupported_items_list: list[Item] = [] | ||
|
|
||
| for index_entry in items_definitions_index: | ||
| item_id = index_entry.get("id", "") | ||
| root_path = index_entry.get("rootPath", "") | ||
|
|
||
| ws_item = items_by_id.get(item_id) | ||
| if ws_item is None: | ||
| # Item not known in workspace - skip | ||
| continue | ||
|
|
||
| try: | ||
| if bulk_export_utils.is_command_supported(ws_item): | ||
| supported_item_prefixes.add(root_path) | ||
| supported_items_list.append(ws_item) | ||
| else: | ||
| unsupported_items_list.append(ws_item) | ||
| except FabricCLIError: | ||
| unsupported_items_list.append(ws_item) | ||
|
|
||
| # find unsupported items that were not in the response index and add them to the unsupported_items_list | ||
| supported_item_ids: set[str] = {item.id for item in supported_items_list} | ||
| unsupported_item_ids: set[str] = {item.id for item in unsupported_items_list} | ||
| for ws_item in ws_items: | ||
| if ( | ||
| ws_item.id not in supported_item_ids | ||
| and ws_item.id not in unsupported_item_ids | ||
| ): | ||
| unsupported_items_list.append(ws_item) | ||
|
ohadedry marked this conversation as resolved.
|
||
|
|
||
| # Filter response parts to keep only supported items | ||
| filtered_parts = [ | ||
| part | ||
| for part in parts | ||
| if _path_belongs_to_item(part.get("path", ""), supported_item_prefixes) | ||
| ] | ||
| exported_definitions["definitionParts"] = filtered_parts | ||
|
|
||
| return { | ||
| "supported_items": supported_items_list, | ||
| "unsupported_items": unsupported_items_list, | ||
| } | ||
|
|
||
|
|
||
| def _path_belongs_to_item(path: str, item_prefixes: set[str]) -> bool: | ||
| """Check if a definition part path belongs to any of the given item prefixes. | ||
|
|
||
| A part belongs to an item if its path starts with the item prefix followed by '/'. | ||
| e.g., path '/Folder1/MyReport.Report/definition/pages/file.json' | ||
| belongs to item prefix '/Folder1/MyReport.Report'. | ||
| """ | ||
| for prefix in item_prefixes: | ||
| if path.startswith(prefix + "/"): | ||
| return True | ||
| return False | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.