diff --git a/.github/scripts/bundle-install.sh b/.github/scripts/bundle-install.sh new file mode 100755 index 00000000..6e4cb1a0 --- /dev/null +++ b/.github/scripts/bundle-install.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +# Wrapper script for bundle install +# Works around git bare repository issues when invoked from certain environments +# (VS Code/Copilot sets GIT_CONFIG_* to enforce safe.bareRepository=explicit) + +set -euo pipefail + +cd "$(dirname "$0")/../.." + +# Clear VS Code's git config overrides that break bundler's bare repo clones +unset GIT_CONFIG_PARAMETERS + +if [[ -n "${GIT_CONFIG_COUNT:-}" ]]; then + for ((i=0; i< GIT_CONFIG_COUNT; i++)); do + unset "GIT_CONFIG_KEY_${i}" "GIT_CONFIG_VALUE_${i}" + done + unset GIT_CONFIG_COUNT +fi +echo "==> Running bundle install..." +bundle install "$@" +echo "==> bundle install completed successfully" diff --git a/.github/skills/search-context/SKILL.md b/.github/skills/search-context/SKILL.md new file mode 100644 index 00000000..9f9d2aee --- /dev/null +++ b/.github/skills/search-context/SKILL.md @@ -0,0 +1,69 @@ +--- +name: search-context +description: Search the shared context system for information. Use this whenever the AI or human needs contextual information about the products. +--- + +## Initialization Required + +If `context/shared/map.md` does not exist, run /start-developement. + +## Load Context Map + +Load the context map. Understand the context types. Understand the levels of detail. + +## Figure Out How to Walk the Levels Of Specificity + +Ideally, the first time you search, you will discover how to search by learning the answers to these questions. + +### Need to know repo + +Determine which repository you are working in. This can usually be determined from git remote -v. + +### Need to know if repo uses local context or shared context + +Check for `context/shared/by-repo/ORG/REPO`. If so, anticipate searching per-repo context there. +If not, anticipate searching per-repo context at `context/local`. + +### Need to know product + +Determine product info from the per-repo `background/product-info.md` file. This determines `context/shared/by-product/PRODUCT/` search entries. There may be zero or multiple product associations. + +### Need to know division + +Determine which division produces the product from the per-repo `background/product-info.md` file. This determines `context/shared/by-division/DIVISION/` search entries. There may be zero or multiple division associations. + +### Need to know business unit + +Determine which business unit produces the product from the per-repo `background/product-info.md` file. This determines `context/shared/by-business-unit/UNIT/` search entries. There may be zero or multiple unit associations. + +## Determine type of query + +Decide if you are looking for designs, background, specifications, or what, based on the context types listed in the Context Map. + +## Construct queries + +You don't have to use grep, but these are examples. + +```bash +grep -r context/shared/global/progress/background/**/*.md 'string' +grep -r context/shared/by-business-unit/infra/background/**/*.md 'string' +grep -r context/shared/by-division/chef/background/**/*.md 'string' +grep -r context/shared/by-product/chef-infra-client/background/**/*.md 'string' +grep -r context/shared/by-repo/chef/chef/background/**/*.md 'string' +``` + +```bash +grep -r context/shared/global/progress/standards/**/*.md 'string' +grep -r context/shared/by-business-unit/infra/standards/**/*.md 'string' +grep -r context/shared/by-division/next/standards/**/*.md 'string' +grep -r context/shared/by-product/alsi/standards/**/*.md 'string' +grep -r context/local/standards/**/*.md 'string' +``` + +## Reconcile Results + +You will likely have multiple results. Merge the results and reconcile contradictions as follows: + +1. Policy specifications higher in the tree are more influential. So a division-level standard should generally apply more than a product-level standard. +2. Technical specifications lower in the tree override specs context higher in the tree. So a technical specification to use a particular driver api might be needed for a good reason (which must be justified) and this override a higher-level mandate. +3. Any confusion or unresolved issues should be brough to the user's attention for a decision. diff --git a/.github/skills/start-development/SKILL.md b/.github/skills/start-development/SKILL.md new file mode 100644 index 00000000..6c0cfeda --- /dev/null +++ b/.github/skills/start-development/SKILL.md @@ -0,0 +1,74 @@ +--- +name: start-development +description: Configure the repo for AI development. Do this before doing any work in the repo. +--- + +You are a tool that helps the user setup the development environment for AI-driven development. + +You will do several tasks to set the user up. + +First, determine if the user is running Windows, MacOS, or Linux. Use that information to decide what scripts to run. + +## Load env file if present + +Read eny env vars from etc/env.sh if present or etc/env.default.sh if not. You should source this in any shell you run. + +## Setup gh + +### Install gh + +Install the gh GitHub CLI tool if it is not already installed. + +### Ensure gh is authenticated + +Make sure `gh auth status` works, and run `gh auth login` if not. + +## Clone the shared-context repo + +### Determine the location of the shared-context repo + +The shared context repo location is at $PROGRESS_SHARED_CONTEXT_REPO which looks like org/repo@branch (branch defaults to main). + +If no value is present, this defaults to `chef/shared-context@main` + +### Clone, Re-Remote, Or Pull + +If there are local changes, warn and do nothing. + +If context/shared does not exist, clone the repo into it. + +If it does exist confirm it is on the right remote and switch. + +If it does exist confirm it is on the git branch and switch. + +Pull. + +## Ensure the list of reference repos is checked out + +Look for the file `etc/reference-repo-list.txt`. Re-read the repo list each time you run — it may have changed. It is a list of GitHub repos to clone. Some of them may be private or internal; you may not have access. The list may include branch specifications like @branch. + +Try to clone each one into `context/reference-repos`. If it has already been cloned, pull it. If a branch has been specified, make sure you are on that branch. If it has local changes, inform the user and do nothing. + +Each time you run, check the repo status again. Do not remove repos, only add them. + +## Ensure the Atlassian MCP server is running + +Check for `.vscode/mcp.json` and look for the atlassian entry. If it is not running or has errored, ask the user to restart it. + +## Ensure the user has rbenv installed and configured + +Check if `rbenv` is installed by running `rbenv --version`. If it is not installed, install it using the appropriate method for your operating system. On MacOS, you can use `brew install rbenv`. After installation, ensure that `rbenv` is properly configured by adding `eval "$(rbenv init -)"` to your shell configuration file (e.g., `.bashrc`, `.zshrc`). + +## Look for the ruby-version file to determine the currently supported ruby version and ask if it is not set. + +Look for the file `.ruby-version` in the root of the repo. It should have a number like 3.4.8 or similar. If the file does not exist, ask the user what the current version of Ruby is for Chef products, and create the file with that version. Default to 3.4.8 if the user does not know. + +## Ensure the user has the current ruby installed + +Use `rbenv version` to check the currently installed Ruby version. If it does not match the version specified in `.ruby-version`, install the correct version using `rbenv install `. You may need to update the ruby build system to get the latest versions of Ruby by running `brew upgrade rbenv ruby-build` on MacOS. + +## Run bundle install using script + +Run `bash .github/scripts/bundle-install.sh` + +**Note:** Use the wrapper script instead of `bundle install` directly — it clears VS Code/Copilot git environment variables that break bundler's bare repository clones. diff --git a/.gitignore b/.gitignore index 4d56b365..ffb6e672 100644 --- a/.gitignore +++ b/.gitignore @@ -9,4 +9,9 @@ terraform.tfstate.backup .bundle .gems coverage/ -Berksfile.lock \ No newline at end of file +Berksfile.lock +etc/env.sh +context/reference-repos/** +context/shared +tmp/ +.ruby-version diff --git a/.vscode/mcp.json b/.vscode/mcp.json new file mode 100644 index 00000000..733e0180 --- /dev/null +++ b/.vscode/mcp.json @@ -0,0 +1,8 @@ +{ + "servers": { + "atlassian-mcp-server": { + "url": "https://mcp.atlassian.com/v1/sse", + "type": "http" + } + } +} \ No newline at end of file diff --git a/context/local/design/img/archetypes.png b/context/local/design/img/archetypes.png new file mode 100644 index 00000000..53bee5ba Binary files /dev/null and b/context/local/design/img/archetypes.png differ diff --git a/context/local/design/img/architecture.png b/context/local/design/img/architecture.png new file mode 100644 index 00000000..7cb5aa1e Binary files /dev/null and b/context/local/design/img/architecture.png differ diff --git a/context/local/design/img/class-diagram.png b/context/local/design/img/class-diagram.png new file mode 100644 index 00000000..a6e94b7a Binary files /dev/null and b/context/local/design/img/class-diagram.png differ diff --git a/context/local/design/img/sequence-load-connect.png b/context/local/design/img/sequence-load-connect.png new file mode 100644 index 00000000..29f20fb5 Binary files /dev/null and b/context/local/design/img/sequence-load-connect.png differ diff --git a/context/local/design/img/sequence-run-command.png b/context/local/design/img/sequence-run-command.png new file mode 100644 index 00000000..796d2ae5 Binary files /dev/null and b/context/local/design/img/sequence-run-command.png differ diff --git a/context/local/design/img/src/archetypes.puml b/context/local/design/img/src/archetypes.puml new file mode 100644 index 00000000..71af05ed --- /dev/null +++ b/context/local/design/img/src/archetypes.puml @@ -0,0 +1,71 @@ +@startuml archetypes +title Train Plugin V1 - Real-World Connection Archetypes + +skinparam shadowing false +skinparam defaultFontName Helvetica +skinparam classAttributeIconSize 0 + +legend top + Green = implemented Red = deliberately omitted ("cheat") + All four are Train transports; three ship as train- gems, gcp ships in train core. +endlegend + +class "train-winrm\n(remote-protocol / shell)" as WinRM { + +run_command_via_connection YES + +file_via_connection -> Remote::Windows YES + +upload/download (WinRM::FS) YES + +wait_until_ready / login_command YES + +connection(state,&block) reuse YES + +validate_options override YES + +lazy gem activation YES + +socks monkeypatch YES + -- + family: (OS via Remote::Windows) +} + +class "train-aws\n(API-only)" as Aws { + +run_command_via_connection NO + +file_via_connection NO + +aws_client / aws_resource custom + +enable_cache :api_call (forced) YES + +URI+ENV option mapping YES + +unique_identifier (STS) YES + -- + platform: force_platform!("aws") + family: cloud +} + +class "train-kubernetes\n(hybrid API + shell)" as K8s { + +run_command_via_connection -> kubectl exec YES + +file_via_connection -> custom File::Linux YES + +K8s::Client (k8s-ruby) API YES + +per-call opts (pod/ns/container) YES + -- + platform: force_platform!("k8s") + family: cloud +} + +class "gcp\n(API-only, in train core)" as Gcp { + +run_command_via_connection NO + +file_via_connection NO + +gcp_client(klass) / gcp_*_client custom + +enable_cache :api_call (forced) YES + +ENV-lazy option defaults YES + +unique_identifier (auth id) YES + -- + platform: force_platform!("gcp") + family: cloud +} + +WinRM -[hidden]right- Aws +Aws -[hidden]right- K8s +K8s -[hidden]right- Gcp + +note bottom of Gcp + Not a separate gem: lib/train/transports/gcp.rb, + class Gcp < Train.plugin(1), name "gcp". + Sibling of the in-core azure transport; the closest + structural twin of train-aws. There is no train-gcp gem. +end note + +@enduml diff --git a/context/local/design/img/src/architecture.puml b/context/local/design/img/src/architecture.puml new file mode 100644 index 00000000..9a71c6da --- /dev/null +++ b/context/local/design/img/src/architecture.puml @@ -0,0 +1,47 @@ +@startuml architecture +title Train Plugin V1 - Component Architecture + +skinparam componentStyle rectangle +skinparam shadowing false +skinparam defaultFontName Helvetica + +actor "Consumer\n(InSpec / Chef Workstation / your app)" as App + +package "Train core (the 'train' gem)" { + [Train.create / Train.options] as Facade + [load_transport\n(registry -> core -> gem)] as Loader + [Train::Plugins.registry\n(name -> Transport class)] as Registry + [Train::Options\n(option DSL, merge/validate)] as Options + [Train::Platforms::Detect\n(Scanner, families)] as Detect + [Train::File::*\n(Local/Remote file impls)] as Files + [Train::Plugins::Transport\n(= Train.plugin(1))] as BaseTransport + [BaseConnection\n(run_command/file/upload/cache)] as BaseConn +} + +package "Plugin gem 'train-'" { + [Entry point\nlib/train-.rb] as Entry + [Transport\n< Train.plugin(1)] as PTransport + [Connection\n< BaseConnection] as PConn + [Platform mixin\n(force_platform!)] as PPlat +} + +cloud "Target\n(host / API / cluster)" as Target + +App --> Facade : create(name, opts)\noptions(name) +Facade --> Loader : resolve name +Loader ..> Registry : lookup +Loader ..> Entry : require "train-"\n(on miss) +Entry --> PTransport : loads +Entry --> PConn : loads +Entry --> PPlat : loads +PTransport --|> BaseTransport +PConn --|> BaseConn +PTransport ..> Registry : name "" registers self +PTransport --> PConn : connection() builds +PConn ..> PPlat : include (platform detection) +PConn ..> Files : wraps file objects +PConn ..> Detect : platform() / force_platform! +BaseTransport ..> Options : option / default_options +PConn --> Target : commands / files / API calls + +@enduml diff --git a/context/local/design/img/src/class-diagram.puml b/context/local/design/img/src/class-diagram.puml new file mode 100644 index 00000000..3c6f3747 --- /dev/null +++ b/context/local/design/img/src/class-diagram.puml @@ -0,0 +1,85 @@ +@startuml class-diagram +title Train Plugin V1 - Class Model + +skinparam shadowing false +skinparam defaultFontName Helvetica +skinparam classAttributeIconSize 0 + +class "Train::Plugins" as Plugins { + +registry : Hash +} + +class "Train::Plugins::Transport" as Transport { + {static} +name(name) + +initialize(options={}) + +connection(_opts=nil) <> + -logger +} +note right of Transport + Returned by Train.plugin(1). + Includes Train::Options and + Train::Extras. +end note + +class "Train::Options" as Options <> { + +option(name, conf=nil, &blk) + +default_options() + +include_options(other) + +merge_options(base, opts) + +validate_options(opts) +} + +class "BaseConnection" as BaseConn { + +initialize(options=nil) + +run_command(cmd, opts={}, &h) + +file(path, *args) + +upload(locals, remote) + +download(remotes, local) + +platform() <> + +login_command() + +wait_until_ready() + +close() + +enable_cache(type) / disable_cache(type) + +cached_client(type, key) + +force_platform!(name, details=nil) + +to_json / load_json(j) + -- + #run_command_via_connection(cmd, opts, &h) <> + #file_via_connection(path, *args) <> +} + +class "CommandResult" as CmdResult <> { + +stdout + +stderr + +exit_status +} + +class "Train::File" as TFile { + +DATA_FIELDS + +exist? mode owner group content mtime size ... +} + +' --- Plugin-provided classes --- +class "TrainPlugins::X::Transport" as XTransport { + +name "x" + +connection(_=nil) +} +class "TrainPlugins::X::Connection" as XConn { + #run_command_via_connection(...) + #file_via_connection(...) +} +class "TrainPlugins::X::Platform" as XPlat <> { + +platform() +} + +Plugins ..> Transport : registers +Transport ..|> Options : includes +Transport +-- BaseConn : nested +BaseConn ..> CmdResult : returns +BaseConn ..> TFile : returns +XTransport --|> Transport +XConn --|> BaseConn +XConn ..|> XPlat : include +XTransport --> XConn : builds + +@enduml diff --git a/context/local/design/img/src/sequence-load-connect.puml b/context/local/design/img/src/sequence-load-connect.puml new file mode 100644 index 00000000..e61dcfd5 --- /dev/null +++ b/context/local/design/img/src/sequence-load-connect.puml @@ -0,0 +1,58 @@ +@startuml sequence-load-connect +title Train Plugin V1 - Load & Connect Lifecycle + +skinparam shadowing false +skinparam defaultFontName Helvetica + +actor App +participant "Train" as Train +participant "Train::Plugins.registry" as Reg +participant "Plugin entry\ntrain-.rb" as Entry +participant "TrainPlugins::X::Transport" as Xport +participant "TrainPlugins::X::Connection" as Xconn +participant "Train::Platforms::Detect" as Detect + +App -> Train : create("x", opts) +activate Train +Train -> Train : load_transport("x") +Train -> Reg : registry["x"] ? +alt not registered yet + Train -> Train : require "train/transports/x" (core)\nelse require "train-x" (gem) + Train -> Entry : load + activate Entry + Entry -> Xport : require transport + Xport -> Reg : name "x" => registry["x"]=self + Entry -> Xconn : require connection + deactivate Entry +end +Train -> Xport : new(opts) +activate Xport +Xport -> Xport : merge_options + (validate_options) +Xport --> Train : transport instance +deactivate Xport +Train --> App : transport +deactivate Train + +App -> Xport : connection() +activate Xport +Xport -> Xconn : new(@options) +activate Xconn +Xconn -> Xconn : super(options)\n(setup cache, audit log) +Xconn -> Xconn : authenticate / configure SDK\n(optional) +Xconn --> Xport : connection +deactivate Xconn +Xport --> App : connection +deactivate Xport + +App -> Xconn : platform (a.k.a. os) +activate Xconn +alt plugin overrides platform (typical) + Xconn -> Xconn : force_platform!("x", release: ...) +else default + Xconn -> Detect : scan(self) + Detect --> Xconn : detected Platform +end +Xconn --> App : Platform +deactivate Xconn + +@enduml diff --git a/context/local/design/img/src/sequence-run-command.puml b/context/local/design/img/src/sequence-run-command.puml new file mode 100644 index 00000000..f4636a3f --- /dev/null +++ b/context/local/design/img/src/sequence-run-command.puml @@ -0,0 +1,52 @@ +@startuml sequence-run-command +title Train Plugin V1 - run_command / file Runtime Flow (rot13 example) + +skinparam shadowing false +skinparam defaultFontName Helvetica + +actor App +participant "Connection\n(BaseConnection)" as Base +participant "audit_log" as Audit +participant "@cache" as Cache +participant "Plugin impl\nrun_command_via_connection" as Impl +participant "Target" as Target + +== run_command == +App -> Base : run_command("echo hello", opts) +activate Base +Base -> Audit : info(type: cmd) [if enabled] +Base -> Base : method(:run_command_via_connection).arity +note right of Base + Arity dispatch keeps compatibility: + arity 1 => (cmd, &h) + arity 2 => (cmd, opts, &h) +end note +alt cache_enabled?(:command) + Base -> Cache : @cache[:command][cmd] ||= ... +end +Base -> Impl : run_command_via_connection(cmd[, opts], &h) +activate Impl +Impl -> Target : execute (e.g. Mixlib::ShellOut) +Target --> Impl : stdout, stderr, exitstatus +Impl -> Impl : transform stdout (Rot13.rotate) +Impl --> Base : CommandResult(stdout, stderr, exit_status) +deactivate Impl +Base --> App : CommandResult +deactivate Base + +== file == +App -> Base : file("/path") +activate Base +Base -> Audit : info(type: file) [if enabled] +alt cache_enabled?(:file) (default true) + Base -> Cache : @cache[:file][path] ||= ... +end +Base -> Impl : file_via_connection(path, *args) +activate Impl +Impl -> Impl : wrap Train::File::* (e.g. Local::Unix)\nand/or override content +Impl --> Base : file object (Train::File-shaped) +deactivate Impl +Base --> App : file object +deactivate Base + +@enduml diff --git a/context/local/design/img/src/v2-actor-boundaries.puml b/context/local/design/img/src/v2-actor-boundaries.puml new file mode 100644 index 00000000..c0c035dc --- /dev/null +++ b/context/local/design/img/src/v2-actor-boundaries.puml @@ -0,0 +1,48 @@ +@startuml v2-actor-boundaries +title Train Plugin Ecosystem — Actors & Contract Seams + +skinparam rectangle { + BorderColor #333333 + FontSize 12 +} +skinparam ArrowColor #444444 +skinparam defaultTextAlignment center + +rectangle "CLIENT APPLICATION\n(InSpec, Chef Target Mode, Test Kitchen)" as client #E8F0FE { + rectangle "credential/target resolution\noption marshalling\nresult consumption\nexception rescue" as clientwork + rectangle "PLUGIN HOST (InSpec today)\nprivate ~/.inspec/gems\ninstall + activate train-*/inspec-*\n(separate from system gem)" as clienthost #D6E4FF +} + +rectangle "TRAIN CORE (facade + framework)" as core #E6FFEA { + rectangle "Loader & registry\nTrain.create / load_transport\nglobal Plugins.registry\nTrain.plugin(version) gate" as coreload + rectangle "Option DSL\nURI parsing\nvalidate_backend\nerror taxonomy" as coreopt + rectangle "BaseConnection\nrun_command/file dispatch\nresult cache (file/command/api)\nupload/download\naudit-log hook" as coreconn + rectangle "Platform-detection engine\nDetect.scan + specifications\nOS/family taxonomy" as coredetect + rectangle "**In core: only local + mock**\n(bootstrap + test transports)" as corebuiltin #CFF0D4 +} + +rectangle "TRANSPORT PLUGINS (all separate gems)\ntrain-ssh, train-winrm, train-docker,\ntrain-aws, train-gcp, train-rest, ..." as transport #FFF4E5 { + rectangle "declare options\nimplement connection()\nrun_command_via_connection\nfile_via_connection\nbuild CommandResult\nconnection reuse (each reinvents)\nauth + sudo/PTY" as tport +} + +rectangle "ECOSYSTEM / PACKAGING\n(RubyGems, bundler)" as eco #F3E8FF { + rectangle "gem publish/install (fallback)\ntrain- naming convention" as ecowork +} + +client -down-> coreopt : **Seam 1**\nTrain.options(name),\ntarget hash, creds +coreload -down-> transport : **Seam 3**\nsubclass Train.plugin(1),\nself.name(reg) +client -down-> coreconn : **Seam 2**\nrun_command / file /\nplatform (primitives) +coreconn -down-> tport : template-method\n*_via_connection +coredetect -right-> tport : drives via\nprimitives / force_platform! +clienthost -right-> transport : installs + activates\n(today: per-client) +eco -down-> clienthost : gems + +legend right + Seam 1 Config -> transport options (Train.options is a de-facto public contract) + Seam 2 Runtime primitives (tiny: run_command, file, platform) + Seam 3 Plugin authoring contract (inherit + register) + Today's problems (red in companion diagram): + - core over-bundles ssh/docker/gcp/azure/... (should be local+mock only) + - the plugin host is reinvented per client (InSpec has one; Chef will too) +endlegend +@enduml diff --git a/context/local/design/img/src/v2-locus-shift.puml b/context/local/design/img/src/v2-locus-shift.puml new file mode 100644 index 00000000..2e6390e5 --- /dev/null +++ b/context/local/design/img/src/v2-locus-shift.puml @@ -0,0 +1,54 @@ +@startuml v2-locus-shift +title Recommended Locus Shifts for Plugin API v2 (what moves, and where) + +skinparam defaultTextAlignment left +skinparam rectangle { + FontSize 11 + BorderColor #555555 +} +skinparam ArrowColor #B03030 +left to right direction + +package "TODAY (v1)" #FDECEC { + rectangle "Core bundles many transports\nssh/docker/gcp/azure/vmware/...\nin train-core, privileged load path" as c0 + rectangle "Plugin host reinvented per client\nInSpec has a full one\n(~/.inspec/gems); Chef will too" as c00 + rectangle "Capability discovery\n= client sniffs by class name\n(is_a? Mock, local_transport?)" as c1 + rectangle "Error handling\n= clients rescue Train::* classes" as c2 + rectangle "Secret redaction\n= nobody (no sensitive flag)" as c3 + rectangle "URI parsing quirks\n= hardcoded in core\n(# move logic into ssh/winrm)" as c4 + rectangle "Connection reuse/pool\n= each transport reinvents\n+ client caches backend" as c5 + rectangle "Audit logging\n= bolt-on beside option DSL" as c6 + rectangle "HTTP/API facility\n= absent; clients improvise\n(curl, Net::HTTP)" as c7 + rectangle "API versioning\n= Train.plugin(1) only" as c8 + rectangle "Registration\n= global mutable singleton" as c9 + rectangle "Platform detection\n= engine assumes shell/file\nprimitives exist" as c10 +} + +package "RECOMMENDED (v2)" #EAF6EC { + rectangle "Core ships only local + mock\nall others are plugin gems\n(train-ssh, train-gcp, ...)" as n0 + rectangle "Shared train-level plugin host\nclients delegate install/activate" as n00 + rectangle "Core-declared **capability manifest**\ntransports advertise; clients query" as n1 + rectangle "Core **error contract**\ndeclared codes/categories, not classes" as n2 + rectangle "Core option **metadata (sensitive)**\nenforced in log/audit redaction" as n3 + rectangle "Transport-provided **URI parser hook**\ncore stays generic" as n4 + rectangle "Core **connection lifecycle/pool**\ntransports implement open/close only" as n5 + rectangle "Core cross-cutting **middleware**\n(audit + diag logging + metrics)" as n6 + rectangle "Core **request primitive**\nuniform HTTP/API surface" as n7 + rectangle "Core **negotiated version**\n+ capability gating" as n8 + rectangle "Core **scoped registry**\n(instance/isolated, versioned)" as n9 + rectangle "Core detection w/ transport-declared\n**strategy** (shell | api | force)" as n10 +} + +c0 --> n0 +c00 --> n00 +c1 --> n1 +c2 --> n2 +c3 --> n3 +c4 --> n4 +c5 --> n5 +c6 --> n6 +c7 --> n7 +c8 --> n8 +c9 --> n9 +c10 --> n10 +@enduml diff --git a/context/local/design/img/v2-actor-boundaries.png b/context/local/design/img/v2-actor-boundaries.png new file mode 100644 index 00000000..fb2f5761 Binary files /dev/null and b/context/local/design/img/v2-actor-boundaries.png differ diff --git a/context/local/design/img/v2-locus-shift.png b/context/local/design/img/v2-locus-shift.png new file mode 100644 index 00000000..f25fd814 Binary files /dev/null and b/context/local/design/img/v2-locus-shift.png differ diff --git a/context/local/design/plugins-v1.md b/context/local/design/plugins-v1.md new file mode 100644 index 00000000..175ab1f0 --- /dev/null +++ b/context/local/design/plugins-v1.md @@ -0,0 +1,555 @@ +# Train Plugin API — Version 1 (V1) Reference & Design Notes + +> Scope: this document describes the **current, shipping Train Plugin API (V1)** — the only +> version Train actually supports today (`Train.plugin(1)`). It is written as the factual +> baseline for a future redesign of the plugin API. Everything here is grounded in the Train +> source tree and in five real transports: the teaching example +> `examples/plugins/train-local-rot13`, three production reference plugins under +> `context/reference-repos/` (`train-winrm`, `train-aws`, `train-kubernetes`), and the +> in-core `gcp` transport (`lib/train/transports/gcp.rb`). + +## Contents + +1. [Overview & scope](#1-overview--scope) +2. [Anatomy of a V1 plugin](#2-anatomy-of-a-v1-plugin) +3. [Registration & discovery/loading](#3-registration--discoveryloading) +4. [The Transport class](#4-the-transport-class) +5. [The Connection class (`BaseConnection`)](#5-the-connection-class-baseconnection) +6. [Platform & family declaration](#6-platform--family-declaration) +7. [Data contracts](#7-data-contracts) +8. [Options, target URIs & credentials](#8-options-target-uris--credentials) +9. [Runtime flow of `run_command` and `file`](#9-runtime-flow-of-run_command-and-file) +10. [Testing a plugin](#10-testing-a-plugin) +11. [Walkthrough: `train-local-rot13`](#11-walkthrough-train-local-rot13) +12. [Real-world patterns & variances](#12-real-world-patterns--variances) +13. [Limitations & observations (springboard for V2)](#13-limitations--observations-springboard-for-v2) + +--- + +## 1. Overview & scope + +Train is the **transport interface** underpinning Chef InSpec. It provides a *uniform* way +to talk to a target — a local machine, a remote host over SSH/WinRM, a container, or a cloud +API — exposing three primitive capabilities to callers: + +- **Command execution** — `connection.run_command("...")` → a result with + `stdout`/`stderr`/`exit_status`. +- **File access** — `connection.file("/path")` → a file object with a uniform metadata API. +- **Platform detection** — `connection.platform` (aliased `os`) → what the target *is*. + +A **Train plugin** adds a new transport (a new "backend") to this system. Train itself has +no CLI and only a light test harness; plugins are normally exercised through InSpec, but they +can be developed and tested standalone (see [§10](#10-testing-a-plugin)). + +> **Train plugin vs. InSpec plugin.** A *Train* plugin (this document) provides +> *connectivity*. An *InSpec* plugin (a different, "v2" plugin system) provides *resources*, +> CLI commands, reporters, etc. They use different factory calls, namespaces, and gem +> prefixes. InSpec resource packs (for example the GCP resource pack) are InSpec plugins that +> sit *on top of* a Train transport — they are **not** transports themselves. + +![Component architecture](img/architecture.png) + +## 2. Anatomy of a V1 plugin + +A Train plugin is a Ruby **gem** named `train-` with three logical components plus an +entry point: + +| Component | Role | Base / form | +|---|---|---| +| **Entry point** | `lib/train-.rb`; sets load path, requires the parts | plain file | +| **Transport** | glue: registers the plugin name, declares options, builds the Connection | `class < Train.plugin(1)` | +| **Connection** | does the real work (commands, files, API, caching) | `class < Train::Plugins::Transport::BaseConnection` | +| **Platform** | declares/detects the target platform & family | usually a `module` mixed into Connection | + +Conventional gem layout (from `train-local-rot13`): + +``` +train-/ +├── train-.gemspec # gem name MUST start with "train-" +├── Gemfile +├── Rakefile # test + rubocop tasks +├── lib/ +│ ├── train-.rb # entry point +│ └── train-/ +│ ├── version.rb # VERSION constant, loaded cheaply by gemspec +│ ├── transport.rb +│ ├── connection.rb +│ └── platform.rb +└── test/ + ├── helper.rb # require "train/plugin_test_helper" + ├── unit/… # class-shape tests + └── functional/… # behavior via Train.create +``` + +The entry point should be minimal — set the load path, then require the pieces +(`lib/train-local-rot13.rb`): + +```ruby +libdir = __dir__ +$LOAD_PATH.unshift(libdir) unless $LOAD_PATH.include?(libdir) +require "train-local-rot13/version" +require "train-local-rot13/transport" +require "train-local-rot13/platform" +require "train-local-rot13/connection" +``` + +Plugins are conventionally namespaced under `TrainPlugins::`. + +## 3. Registration & discovery/loading + +### The versioned base class + +Everything hangs off `Train.plugin(version = 1)` (`lib/train/plugins.rb`). It returns the +class you inherit your Transport from, and it is the sole version gate: + +```ruby +def self.plugin(version = 1) + if version != 1 + raise ClientError, "Only understand train plugin version 1. ..." + end + ::Train::Plugins::Transport +end +``` + +> **There is only V1.** Any argument other than `1` raises. This is the concrete artifact a +> future "V2" would extend. + +### The registry + +`Train::Plugins.registry` is a plain `Hash` mapping **String** plugin names → Transport +classes. A plugin adds itself by calling the class-level `name` DSL inside its Transport +(`lib/train/plugins/transport.rb`): + +```ruby +def self.name(name) + Train::Plugins.registry[name] = self +end +``` + +Note the name is the transport scheme (`"winrm"`, `"aws"`, `"k8s"`, `"local-rot13"`), **not** +the gem name (`train-winrm`, …). Keys are Strings, not Symbols. + +### Discovery / loading + +`Train.create(name, *args)` (`lib/train.rb`) resolves a name to a class via +`load_transport`, then instantiates it. Resolution order: + +1. **Registry hit** — return `registry[name]` if already loaded. +2. **Core transport** — `require "train/transports/"` (built-ins like `ssh`, `local`, + `docker`, `winrm`…). +3. **Gem convention** — `require "train-"` (this is what loads external plugins; the + entry point registers the name as a side effect of being required). +4. Otherwise raise `Train::PluginLoadError` ("Please install it first"). + +![Load & connect lifecycle](img/sequence-load-connect.png) + +## 4. The Transport class + +Minimal Transport (from `train-local-rot13`): + +```ruby +module TrainPlugins + module LocalRot13 + class Transport < Train.plugin(1) + name "local-rot13" + + def connection(_instance_opts = nil) + @connection ||= TrainPlugins::LocalRot13::Connection.new(@options) + end + end + end +end +``` + +Responsibilities and available machinery: + +- **`name "x"`** — required; registers the plugin. +- **`connection(_opts = nil)`** — the *only* method you must implement. Return a + `BaseConnection` subclass instance. The base implementation raises `ClientError`. The + argument is "undocumented and rarely used" (per the base class comments) — some plugins + ignore it, WinRM uses it as mutable `state` (see [§12](#12-real-world-patterns--variances)). +- **`initialize(options = {})`** — the base stores `@options = merge_options({}, options)` + and sets up `@logger`. If you override it, call `super`. +- **Options DSL** — `Train::Options` is attached to every Transport + (`lib/train/options.rb`). At class level you get `option`, `default_options`, + `include_options`; at instance level `merge_options`, `validate_options`. Declare options + with defaults or a lazy block: + + ```ruby + option :host, required: true + option :port, default: 5985 + option(:region, required: true) { ENV["AWS_REGION"] } # block = lazy default + ``` + + `merge_options` fills defaults (evaluating `Proc` defaults and `:coerce`), and also folds + in the **audit-log options** (`enable_audit_log`, `audit_log_location`, …). + `validate_options` enforces `required: true`. + +## 5. The Connection class (`BaseConnection`) + +`Train::Plugins::Transport::BaseConnection` (`lib/train/plugins/base_connection.rb`) is the +heart of the API. Its own comments admit it is overloaded: *"Later generations of the plugin +API will likely separate out these responsibilities."* Today one object handles auth, +platform detection, caching, file access, command execution, API execution, and JSON +marshalling. + +![Class model](img/class-diagram.png) + +### Public API (what callers use) + +| Method | Purpose | +|---|---| +| `run_command(cmd, opts = {}, &data_handler)` | Execute a command (with caching + audit + arity dispatch). | +| `file(path, *args)` | Get a file object (with caching + audit). | +| `platform` / `os` | Detected platform (memoized `Train::Platforms::Detect.scan(self)`). | +| `upload(locals, remote)` / `download(remotes, local)` | File transfer (default impls use `file(...).content`). | +| `login_command` | Return a `LoginCommand` for an interactive session (default raises). | +| `wait_until_ready` | Block until the target is usable (default no-op). | +| `close` | Tear down (default no-op). | +| `enable_cache(type)` / `disable_cache(type)` | Toggle caching for `:file` / `:command` / `:api_call`. | +| `cached_client(type, key) { … }` | Memoize an expensive client under the cache. | +| `force_platform!(name, details = nil)` | Bypass detection and assert a platform (see [§6](#6-platform--family-declaration)). | +| `to_json` / `load_json` | Marshal cached file state (used by InSpec's mock/`--json-config`). | + +### Methods you implement (the real contract) + +These are **private** and raise `NotImplementedError` in the base — you override the ones +your transport supports: + +```ruby +def run_command_via_connection(command, opts = {}, &data_handler); end +def file_via_connection(path, *args); end +``` + +- Implement **both** for a shell-style transport (winrm, k8s, rot13, local). +- Implement **neither** for a pure API transport (aws) — callers use plugin-specific + accessors instead. +- The optional **`data_handler`** block lets a transport stream inbound data + (`data_handler.call(chunk)`); implementations must call it explicitly or it's ignored. + +### Caching + +The base sets defaults `{ file: true, command: false, api_call: false }` and maintains +`@cache[type]`. `run_command`/`file` consult it (`@cache[type][key] ||= …`). +`enable_cache`/`disable_cache` validate the type against the known set (else +`Train::UnknownCacheType`). `cached_client` is a convenience for memoizing SDK clients. + +### Arity-based dispatch (compatibility wart) + +Because plugins are separate gems that can't all be updated in lockstep, `run_command` +inspects `method(:run_command_via_connection).arity` and calls the 1-arg or 2-arg form +accordingly. New plugins should use the 2-arg `(cmd, opts, &h)` form. + +### Audit logging + +If `enable_audit_log` is set, `run_command`/`file`/`upload` emit structured entries to a +`Train::AuditLog` created in the base initializer. + +## 6. Platform & family declaration + +Train has a full platform/family detection engine +(`lib/train/platforms/…`): a tree of platforms and families walked by `Scanner` +(`Detect.scan(backend)`), matching on files/uname/etc. Real OS transports (SSH, local, +WinRM's remote file class) rely on it. + +Most *plugin* authors, however, know exactly what they connect to, so they **bypass +detection** with `force_platform!`. The idiom (a `Platform` module mixed into the +Connection) declares the platform's family membership, then forces it: + +```ruby +module TrainPlugins::Aws + module Platform + def platform + Train::Platforms.name("aws").in_family("cloud") + force_platform!("aws", release: "train-aws: v#{VERSION}, aws-sdk-core: v#{sdk_ver}") + end + end +end +``` + +- `Train::Platforms.name(x).in_family(y)` registers the platform and its family. +- `force_platform!(name, details)` (in `BaseConnection`) sets the backend, builds the family + hierarchy, and attaches platform helper methods — no scanning. +- **Family choice matters.** Cloud/API plugins declare the **`cloud`** family (aws, k8s). A + filesystem-capable plugin declares an OS family (`unix`/`windows`) so InSpec's file logic + works — rot13 declares *both* `unix` and `windows`. +- `release:` is a free-form version string — commonly the plugin version and/or the + underlying SDK/API version. + +## 7. Data contracts + +### CommandResult + +`run_command_via_connection` must return an object exposing `stdout`, `stderr`, +`exit_status`. Train ships `Train::Extras::CommandResult` +(`Struct.new(:stdout, :stderr, :exit_status)`), but any duck-typed object works — rot13 +returns an `OpenStruct`, WinRM/k8s/local return `CommandResult`. + +```ruby +CommandResult = Struct.new(:stdout, :stderr, :exit_status) # lib/train/extras.rb +``` + +### File object + +`file_via_connection` must return a `Train::File`-shaped object. The interface is defined by +`Train::File::DATA_FIELDS` (`lib/train/file.rb`): + +``` +exist? mode owner group uid gid content mtime size selinux_label path +``` + +Plugins usually **reuse** a built-in file class (`Train::File::Local::Unix`, +`Train::File::Remote::Windows`, `Train::File::Remote::Linux`) and optionally wrap/subclass it +to customize behavior (rot13 wraps and overrides `content`; k8s subclasses +`Remote::Linux`). + +### LoginCommand + +`LoginCommand = Struct.new(:command, :arguments)` — returned by `login_command` to describe +how to open an interactive session (e.g. WinRM builds `mstsc`/`rdesktop`/`open` invocations). + +## 8. Options, target URIs & credentials + +Callers usually specify a target as a URI. `Train.unpack_target_from_uri` / +`Train.target_config` (`lib/train.rb`) parse `scheme://user:password@host:port/path?query` +into the symbol-keyed options hash the Transport receives: + +- `scheme` → `:backend` (selects the plugin name) +- `host`/`port`/`user`/`password`/`path` → matching keys +- `?query=values` → merged as additional options + +Plugins then re-interpret these fields as they see fit inside `Connection#initialize`. For +example, `train-aws` treats the URI as `aws:///`: + +```ruby +options[:region] = options[:host] || options[:region] +options[:profile] = options[:path].sub(%r{^/}, "") if options[:path] +``` + +New transports are encouraged to use the `transport://credset` convention rather than +inventing bespoke field mappings. + +## 9. Runtime flow of `run_command` and `file` + +![run_command / file runtime flow](img/sequence-run-command.png) + +For a command: audit-log (if enabled) → arity check → cache check → your +`run_command_via_connection` (which talks to the target and may transform output) → +`CommandResult`. For a file: audit-log → cache check (files cached by default) → your +`file_via_connection` → file object. Both memoize into `@cache[...]` when the relevant cache +type is enabled. + +## 10. Testing a plugin + +`require "train/plugin_test_helper"` (`lib/train/plugin_test_helper.rb`) loads Train, +`minitest/spec` + `minitest/autorun`, and mixes in `TrainPluginBaseHelper` / +`TrainPluginFunctionalHelper` which provide `let` helpers like `plugin_fixtures_path`, +`registry`, etc. A plugin's `test/helper.rb` is typically just: + +```ruby +require "train/plugin_test_helper" +``` + +Two common layers (all references follow this): + +- **Unit tests** assert *shape*: the transport is registered without the `train-` prefix, the + Transport `< Train.plugin(1)`, it defines `connection`, and the Connection defines + `file_via_connection` / `run_command_via_connection`. +- **Functional tests** assert *behavior* through the public entrypoint, e.g. + `Train.create("local-rot13").connection.run_command("echo hello")`, and check the output. + +The `Rakefile` wires a `test` task (unit + functional) and a `lint` (RuboCop) task that +reuses Train's own `.rubocop.yml` via `Train.src_root` (`lib/train/globals.rb`). + +## 11. Walkthrough: `train-local-rot13` + +The canonical teaching plugin (`examples/plugins/train-local-rot13`) is a *local* transport +that applies the ROT13 cipher to every file it reads and every command's stdout. It shows the +full skeleton with nothing extraneous: + +- **Transport** (`transport.rb`) — `name "local-rot13"`; `connection` returns a memoized + `Connection`. +- **Connection** (`connection.rb`) — `include`s the Platform module; implements + `file_via_connection` by wrapping `Train::File::Local::Unix` in a `FileContentRotator` + (which overrides `content` and `method_missing`-delegates the rest); implements + `run_command_via_connection` with `Mixlib::ShellOut`, returning an `OpenStruct` with + rot13'd stdout. +- **Platform** (`platform.rb`) — declares membership in both `unix` and `windows`, then + `force_platform!("local-rot13", release: VERSION)`. +- **Support** — `file_content_rotator.rb` (has-a `Train::File`, meddles with `content`), + `version.rb`. + +It's a deliberately "honest" plugin: it implements both capabilities and reuses Train's +file/command machinery. The production references each diverge from this baseline in +instructive ways. + +## 12. Real-world patterns & variances + +Four production transports show how plugins are *actually* built: three references under +`context/reference-repos/` (`train-winrm`, `train-aws`, `train-kubernetes`) plus the in-core +`gcp` transport (`lib/train/transports/gcp.rb`). `train-winrm` is the traditional case; the +other three "cheat" — two are API-only (no command/file access) and one shells out for a +"remote." + +![Real-world archetypes](img/archetypes.png) + +### 12.1 Comparison matrix + +| Aspect | train-winrm | train-aws | train-kubernetes | gcp (in-core) | +|---|---|---|---|---| +| Kind | **Train plugin** (remote protocol) | **Train plugin** (API-only) | **Train plugin** (hybrid) | **Train transport** (API-only, **in train core**) | +| Base class | `Train.plugin(1)` | `Train.plugin(1)` | `Train.plugin(1)` | `Train.plugin(1)` | +| Namespace | `TrainPlugins::WinRM` | `TrainPlugins::Aws` | `TrainPlugins::TrainKubernetes` | `Train::Transports::Gcp` | +| Gem prefix | `train-` | `train-` | `train-` | — (ships in `train-core`) | +| `run_command_via_connection` | ✅ threaded, timeout | ❌ | ✅ via `kubectl exec` | ❌ | +| `file_via_connection` | ✅ `Remote::Windows` | ❌ | ✅ custom `File::Linux` | ❌ | +| `upload`/`download` | ✅ `WinRM::FS` | ❌ | ❌ | ❌ | +| Options DSL | very rich (~25) | few, ENV-lazy | few, ENV default | few, ENV-lazy (3) | +| Caching | default | forces `:api_call` | default | forces `:api_call` | +| Platform / family | OS (remote windows) | `force_platform!("aws")`, `cloud` | `force_platform!("k8s")`, `cloud` | `force_platform!("gcp")`, `cloud` | +| Connection reuse | ✅ `connection(state, &block)` | memoized | on init (`connect`) | memoized (`@connection ||=`) | +| Lazy dependency load | ✅ `load_needed_dependencies!` | eager `require` | eager `require` | eager `require` (`google-apis-*`) | +| Custom accessors | — | `aws_client` / `aws_resource` | `client`, `KubectlClient` | `gcp_client(klass)` + `gcp_*_client` | +| "Cheat" | (none — canonical) | no files/commands | shells out for a "remote" | no files/commands; bundled in core (no gem) | + +### 12.2 train-winrm — the traditional/canonical case + +Everything the API offers, used properly: + +- **Rich options** (`option :host, required: true`, ssl/kerberos/socks/…) plus a + **`validate_options` override** that normalizes symbols, computes the `endpoint`, and + auto-detects the Kerberos realm from `/etc/krb5.conf`. +- **Connection reuse via the `connection` argument**: `connection(state = nil, &block)` + merges `state` over options, and reuses the cached connection when the computed + `connection_options` are unchanged, otherwise `create_new_connection` (closing the old + one). This is the one real user of the "undocumented" `connection` argument. +- **Lazy dependency activation** — `load_needed_dependencies!` → `load_dependency` uses + `gem`/`require` with a friendly `LoadError`→`Train::UserError` message telling the user + what to `gem install`. +- **Full capability set** — `run_command_via_connection` runs in a `Thread` to support + timeouts (raising `Train::CommandTimeoutReached`); `file_via_connection` → + `Train::File::Remote::Windows`; `upload`/`download` via `WinRM::FS::FileManager`; + `wait_until_ready` pings; `login_command` builds per-OS RDP commands; `close` tears down + the session; `to_s` hides the password. +- **Runtime monkeypatch** — `socks_proxy_patch.rb` patches `HTTPClient` when a SOCKS proxy is + configured — an escape hatch used because the API has no first-class proxy concept. + +### 12.3 train-aws — API-only + +- Implements **neither** `run_command_via_connection` nor `file_via_connection`. Instead it + exposes **`aws_client(klass)`** (memoized under `:api_call` cache) and **`aws_resource`**; + resource packs built on top call these. +- **Forces `enable_cache :api_call`** in the initializer so repeated client construction is + cheap. +- **ENV-lazy option defaults** via the block form (`option(:region) { ENV["AWS_REGION"] }`) + so tests can override the environment; it also maps the URI (`aws://region/profile`) into + options and *writes back* to `ENV` for the SDK. +- **Platform**: `Train::Platforms.name("aws").in_family("cloud")` + + `force_platform!`, with `release:` combining the plugin and `aws-sdk-core` versions. +- `unique_identifier` returns the AWS account id (via STS) — a convention some cloud plugins + use to identify the target. +- Pulls in a large fan of `aws-sdk-*` gems as dependencies. + +### 12.4 train-kubernetes — hybrid API + shell + +- **Both worlds**: it holds a real API client (`K8s::Client` from `k8s-ruby`, built in + `parse_kubeconfig`/`connect`) *and* executes commands by **shelling out to `kubectl exec`** + (`KubectlClient` + `Mixlib::ShellOut`) — the "cheat": a "remote" command transport that is + really a local `kubectl` subprocess. +- **Custom File subclass**: `File::Linux < Train::File::Remote::Linux` implements + `content`/`exist?`/`content=`/… by running remote commands (`cat`, `test -e`, base64 + round-trips) through the connection. +- **Per-call options threading**: both `run_command_via_connection(cmd, opts = {})` and + `file_via_connection(path, **args)` pass `pod`/`namespace`/`container` through on each call, + because a single connection can target many pods. This exercises the `*args`/`opts` + flexibility of the base API. +- **Platform**: `force_platform!("k8s")` in the `cloud` family. + +### 12.5 gcp — in-core, API-only transport + +Unlike the other three, the GCP transport is **not a separate gem** — it ships **inside train +core** at [`lib/train/transports/gcp.rb`](../../../lib/train/transports/gcp.rb) +(`class Gcp < Train.plugin(1)`, `name "gcp"`), alongside its sibling `azure`. Structurally it +is the closest twin of `train-aws`: + +- **API-only** — implements neither `run_command_via_connection` nor `file_via_connection`. + Instead it exposes client accessors: `gcp_client(klass)` (memoized under the `:api_call` + cache) plus conveniences `gcp_compute_client`, `gcp_iam_client`, `gcp_project_client`, + `gcp_storage_client`, `gcp_admin_client`. +- **Forces `@cache_enabled[:api_call] = true`** in the initializer, exactly like aws/azure. +- **ENV-lazy options** — `google_application_credentials`, `google_cloud_project`, + `google_super_admin_email`, each defaulting from the matching env var via the `option { … }` + block form; `connect` writes them back to `ENV` for the Google SDK and sets up + application-default credentials. +- **Platform** — `force_platform!("gcp")`; the `gcp` platform is registered + `in_family("cloud")` (`lib/train/platforms/detect/specifications/api.rb`). `release:` is + derived from the installed `google-apis-core` gem version. +- **Eager dependencies** — the file `require`s the `google-apis-*` client gems at load time + (no lazy activation like winrm). +- `unique_identifier` derives an id from the auth `client_id`/`issuer`. + +**There is no `train-gcp` gem.** The `gcp://` backend has always lived in train core (added in +commit `d991228`, #283; `gcp_admin_client` in `c502811`, #349) — there is no separate +`train-gcp` gem on RubyGems nor an `inspec/train-gcp` repo. The InSpec **GCP resource pack** +consumes this transport via `inspec.backend` (calling `gcp_client` / `gcp_compute_client` +etc.). A code comment in that resource pack ("…will be moved into the `train-gcp` plugin +itself") refers to an extraction that never happened. + +### 12.6 Recurring idioms across the references + +- URI → options remapping inside `Connection#initialize` (aws, k8s). +- ENV-lazy option defaults via the `option { … }` block form (aws, gcp). +- `force_platform!` + explicit `in_family(...)` instead of detection (aws, k8s, gcp, rot13). +- Custom `*_client` / client accessors for API transports (aws `aws_client`, k8s `client`, + gcp `gcp_client`). +- Per-call option threading through `run_command_via_connection` / `file_via_connection` + (k8s). +- Reusing/subclassing built-in `Train::File::*` classes rather than reimplementing files + (all file-capable plugins). +- Lazy gem activation with friendly errors (winrm); runtime monkeypatching as an escape hatch + (winrm socks). +- Connection reuse driven by the (otherwise undocumented) `connection` argument (winrm). + +## 13. Limitations & observations (springboard for V2) + +Distilled from the code and the real-world variances above — the pain points a new API +version should address: + +1. **`BaseConnection` is a god-object.** One class owns auth, platform detection, caching, + files, commands, API execution, and JSON marshalling. The base class comments themselves + anticipate splitting these responsibilities. Capability should be composable, not + all-or-nothing on one base class. +2. **Capabilities are implicit and unenforced.** Whether a plugin supports commands and/or + files is expressed only by *which private methods you happen to override*. There is no + declaration of capabilities, so callers can't ask "does this transport support files?" + without probing (aws implements neither; k8s fakes commands). +3. **Arity-based dispatch** in `run_command` is a compatibility hack for un-versioned plugin + method signatures — a symptom of having no per-method versioning. +4. **The `connection` argument is undocumented** yet load-bearing (winrm uses it for reuse). + Its semantics vary per plugin. +5. **File/command coupling to OS classes.** Reusing `Train::File::*` is powerful but leaks OS + assumptions into API/cloud plugins; the `cloud` family + OS-family fudging (rot13 claims + both `unix` and `windows`) is a workaround. +6. **No load-on-demand / capability activation** on the Train side. Plugins `require` heavy + SDKs eagerly (aws pulls dozens of gems); the InSpec v2 side has an activator model Train + lacks. WinRM hand-rolls `load_needed_dependencies!` to compensate. +7. **Single version only.** `Train.plugin(1)` hard-rejects any other version; there is no + forward-compatible negotiation, so a V2 must be introduced carefully. +8. **Transport vs. resource-pack confusion.** The ecosystem mixes `train-*` transports and + `inspec-*` resource packs (gcp); clearer separation / documentation of the boundary would + help. + +--- + +### Appendix: source references + +- Core: `lib/train.rb`, `lib/train/plugins.rb`, `lib/train/plugins/transport.rb`, + `lib/train/plugins/base_connection.rb`, `lib/train/options.rb`, `lib/train/extras.rb`, + `lib/train/file.rb`, `lib/train/platforms/**`, `lib/train/plugin_test_helper.rb`, + `lib/train/globals.rb`. +- Teaching example: `examples/plugins/train-local-rot13/**`. +- References: `context/reference-repos/{train-winrm,train-aws,train-kubernetes}/**` and + `lib/train/transports/gcp.rb`. +- InSpec v2 plugin API (contrast): `context/reference-repos/inspec/dev-docs/plugins.md`. +- Diagram sources: `img/src/*.puml` (regenerate with + `java -jar plantuml.jar -tpng -o "$PWD/img" img/src/*.puml`). diff --git a/context/local/design/plugins-v2-separation-of-concerns.md b/context/local/design/plugins-v2-separation-of-concerns.md new file mode 100644 index 00000000..9a2f9017 --- /dev/null +++ b/context/local/design/plugins-v2-separation-of-concerns.md @@ -0,0 +1,573 @@ +# Train Plugin API v2 — Separation of Concerns + +> Scope: this is the **groundwork** for a Train Plugin API v2 — *not* the v2 API itself. +> It inventories every area of concern in the transport/plugin ecosystem and, for each, +> records **who owns it today**, **the problem or tension**, and a **recommended locus of +> responsibility for v2**. It is grounded in the Train source tree and in the two consumer +> studies already written ([InSpec](../integrations/inspec.md), +> [Chef Target Mode](../integrations/chef-infra-client-agentless.md)) and builds directly on +> the [V1 reference](./plugins-v1.md). + +## Contents + +1. [Purpose & how to read this](#1-purpose--how-to-read-this) +2. [The four actors](#2-the-four-actors) +3. [Master responsibility index](#3-master-responsibility-index) +4. [Group A — Discovery, packaging & loading](#4-group-a--discovery-packaging--loading) +5. [Group B — Configuration & connection setup](#5-group-b--configuration--connection-setup) +6. [Group C — Runtime facilities (the primitives)](#6-group-c--runtime-facilities-the-primitives) +7. [Group D — Cross-cutting operational concerns](#7-group-d--cross-cutting-operational-concerns) +8. [Cross-cutting meta-problems](#8-cross-cutting-meta-problems) +9. [Open questions](#9-open-questions) +10. [Cross-references](#10-cross-references) + +--- + +## 1. Purpose & how to read this + +A plugin API is, fundamentally, a **contract about who is responsible for what**. Train V1 +grew organically: responsibilities landed wherever was convenient at the time, and several +of them landed in the *wrong* place — transport-specific parsing baked into core, most +transports (ssh, docker, gcp, azure…) bundled into a gem that should carry only `local` and +`mock`, the plugin install/load *lifecycle* pushed entirely onto each client to reinvent, +secret handling owned by nobody, and capability discovery done by clients reaching past the +interface to sniff concrete classes. Before proposing a v2 API we must first name every +concern and decide, deliberately, where it belongs. + +Each concern below is written as a short record: + +- **What** — the concern in one line. +- **Current locus** — who owns it today, with file/method evidence. +- **Tension** — why the current placement is a problem (or fine). +- **v2** — the recommended locus of responsibility for a redesign. + +Concerns marked **[+]** are ones not on the original brainstorm list — the "what have I +missed" additions. + +Two diagrams frame the analysis: + +![Actors & contract seams](./img/v2-actor-boundaries.png) + +*The four actors and the three contract seams between them. The runtime seam (Seam 2) is +deliberately tiny; most of the surface area — and most of the misplaced responsibility — +lives around configuration (Seam 1) and plugin authoring (Seam 3).* + +![Recommended locus shifts](./img/v2-locus-shift.png) + +*The concerns whose recommended v2 owner differs from today: what moves, and to where. These +ten shifts are the core of the v2 opportunity.* + +--- + +## 2. The four actors + +Every responsibility row is assigned to one (occasionally two) of these: + +| Actor | Who | Role | +|---|---|---| +| **Core** | Train / train-core gem | Façade (`Train.create`, `load_transport`), global `Plugins.registry`, option DSL, URI parsing, `validate_backend`, `BaseConnection` template methods, result cache, platform-detection engine, error taxonomy, audit-log subsystem. **Should ship only the two bootstrap transports it cannot live without — `local` and `mock` — but today over-bundles ssh/docker/podman/gcp/azure/vmware/cisco_ios.** | +| **Transport** | A transport plugin (ssh, winrm, docker, aws, gcp, `train-rest`, `train-local-rot13`) | Declares options, implements `connection()` + `*_via_connection`, builds `CommandResult`, does auth/sudo, reuses its own socket. In v2 **every** transport except `local`/`mock` is a separate gem. | +| **Client** | InSpec, Chef Target Mode, Test Kitchen | Resolves target/credentials, marshals options into Train, consumes results, rescues `Train::*` exceptions — **and, in InSpec's case, embeds a full plugin host** (install/activate; see below). | +| **Ecosystem** | RubyGems, bundler, **client-embedded plugin host** (InSpec's plugin v2 manager) | Installs and loads gems. The `train-` naming is enforced only by convention. Installation is *not* only the system `gem` command — see the plugin-host note below. | + +**The client-embedded plugin host is a first-class actor, not just "the ecosystem."** InSpec +ships a **complete plugin manager independent of the system `gem` command**: it installs +plugin gems into a private prefix `~/.inspec/gems` via `Gem::Installer` / `Gem::RequestSet` / +`Gem::Resolver` (`lib/inspec/plugin/v2/installer.rb`), records them in a `plugins.json` +(`config_file.rb`), and at runtime adds that prefix to `Gem.path` and *activates* the gems and +their dependencies with version constraints (`loader.rb#activate_managed_gems_for_plugin`). It +manages **both** `inspec-*` and `train-*` gems (`list_installed_plugin_gems`), so **installing +and loading a transport is, today, an InSpec responsibility** — not merely RubyGems, and not +Train. Chef Target Mode currently leans on bundler, but is expected to grow its own equivalent. +This duplication (each client re-implementing a transport plugin host) is itself a v2 concern +(§4.2–4.3). + +The recurring failure mode is **responsibility bleeding across actors**: Core hardcoding +transport specifics *and shipping transports it shouldn't own*, clients sniffing transport +classes, clients each re-implementing a plugin host, transports each re-implementing what Core +should provide once. + +--- + +## 3. Master responsibility index + +Legend: **C** = Core, **T** = Transport, **A** = Client application, **E** = Ecosystem. +An arrow (→) marks a recommended shift. + +| # | Concern | Today | v2 (recommended) | +|---|---|---|---| +| **A. Discovery, packaging & loading** |||| +| 1 | Packaging, naming & **core-vs-plugin boundary** | E conv.; C over-bundles | E conv., **C-enforced; core = local+mock only** | +| 2 | Gem installation | E / **A (InSpec plugin host)** | **C-provided host**, used by A | +| 3 | Gem loading / activation | C (in-core path) + **A (InSpec loader)** | **C-provided host**; in-core path = local+mock only | +| 4 | Plugin registration | C (global singleton) | C (**scoped/isolated**) | +| 5 **[+]** | API versioning & capability negotiation | C (`plugin(1)` only) | **C (negotiated)** | +| 6 **[+]** | Discovery & **capability advertisement** | A (class sniffing) | **C + T (manifest)** | +| **B. Configuration & connection setup** |||| +| 7 | Option definition (DSL) | C + T | C + T (unify audit opts) | +| 8 **[+]** | Option namespacing / merge precedence | A + C | A + C (formalize) | +| 9 | URL / target parsing | C (with T-hacks) | **C generic + T hook** | +| 10 | Credential access / credential sets | A | A + **C schema** | +| 11 | Auth & privilege escalation | T | T (unchanged) + C sudo helper | +| 12 **[+]** | Secret / sensitive handling & redaction | *nobody* | **C (option metadata)** | +| 13 **[+]** | Backend validation | C | C (unchanged) | +| 14 **[+]** | Connection lifecycle (open/close/ready) | T | T + **C contract** | +| 15 | Connection reuse / pooling | T + A | **C lifecycle** | +| **C. Runtime facilities** |||| +| 16 | Command execution + result + timeout + stream | C dispatch / T impl | C / T (unchanged) | +| 17 | File read / metadata | C dispatch / T impl | C / T (unchanged) | +| 18 **[+]** | File write / transfer (upload/download) | C + T | C + T (make optional-capability) | +| 19 | HTTP / API request facility | *absent* (A improvises) | **C (uniform primitive)** | +| 20 **[+]** | Interactive session / login / PTY | T | T (unchanged) | +| 21 **[+]** | Platform detection & taxonomy | C engine / T primitives | C + **T strategy** | +| 22 | Result caching (file/command/api) | C | C (unchanged) | +| **D. Cross-cutting operational** |||| +| 23 **[+]** | Error taxonomy & exception contract | C classes / A rescue | **C error codes** | +| 24 **[+]** | Diagnostic logging | C + T | C middleware | +| 25 | Audit logging | C (bolt-on) | **C middleware** | +| 26 **[+]** | Serialization / marshalling | C (partial) | C (generalize) | +| 27 | Test mocking / fixtures | C (Mock) + A | C (first-class) | +| 28 **[+]** | Concurrency / thread-safety | *undefined* | **C contract** | +| 29 **[+]** | Documentation / help generation | T (ad hoc) | C from metadata | + +--- + +## 4. Group A — Discovery, packaging & loading + +### 4.1 Packaging, naming & the core-vs-plugin boundary +- **What:** A transport ships as a gem; the gem name encodes the transport name — and, + separately, **which transports Core itself bundles.** +- **Current locus:** **Ecosystem (naming, by convention) + Core (bundling, badly).** + `Train.load_transport` maps transport `foo` to `require "train/transports/foo"` **first** + (in-core) and only then `require "train-foo"` (`lib/train.rb:39-64`). Core therefore ships a + pile of transports directly in the gem: `local`, `mock`, **plus** `ssh`, `docker`, `podman`, + `gcp`, `azure`, `vmware`, `cisco_ios`. The `train-` prefix is also load-bearing inside + InSpec's plugin bridge. +- **Tension:** **No transport belongs in Core except the two it structurally cannot live + without.** `local` is required because the default target is local and platform detection + must run *somewhere*; `mock` is required because testing and fixtures are core machinery. Every + other bundled transport — ssh, docker, podman, and especially the heavyweight cloud API ones + (`gcp`, `azure`, `vmware`) — drags optional, dependency-heavy code into `train-core` and blurs + the plugin boundary: the "in-core first" load path means these transports never had to prove + they work as plugins. `gcp` living in core is the sharpest example — it is an API transport + with cloud-SDK dependencies sitting in the base gem. +- **v2:** **Core ships only `local` and `mock`.** Everything else is extracted to its own gem + (`train-ssh`, `train-docker`, `train-gcp`, …) loaded through the *same* plugin path as + third-party transports — no privileged in-core shortcut. Naming stays an ecosystem convention + but is **Core-verified** via declared gem metadata (§4.2). This makes the plugin API + dogfood itself: the built-in transports become plugins like everyone else's. + +### 4.2 Gem installation +- **What:** Getting a transport gem onto the system and recorded as available. +- **Current locus:** **Client-embedded plugin host (InSpec), not just the system `gem` + command.** InSpec's plugin v2 installer (`lib/inspec/plugin/v2/installer.rb`) resolves and + installs plugin gems into a private prefix (`~/.inspec/gems`) with `Gem::Installer` / + `Gem::RequestSet` / `Gem::Resolver`, tracks them in `plugins.json` (`config_file.rb`), and + supports install/update/uninstall/search — a full package manager parallel to (and + independent of) `gem install`. It manages `train-*` and `inspec-*` gems alike. Chef Target + Mode relies on bundler today but is expected to grow its own host. +- **Tension:** Installation of *transports* is currently a **client** concern, and **each + client reinvents the host**. That means transport install/activation semantics (private + prefix, version pinning, dependency resolution) differ per consumer, and a transport author + has no single "how do I get installed" story. Train — the thing that actually *defines* what a + transport is — owns none of this. +- **v2:** Provide a **Core (train-level) plugin host** — a reusable transport install/activate + facility — that clients delegate to instead of hand-rolling. InSpec's mature manager is the + template; the goal is that InSpec, Chef, and Kitchen share one transport-plugin lifecycle + rather than three. The system `gem` command remains a valid fallback path. + +### 4.3 Gem loading / activation +- **What:** Loading and activating a transport's code (and its dependencies) at runtime. +- **Current locus:** **Core in-core path + Client loader.** Core's `load_transport` handles the + `require`; InSpec's `loader.rb` additionally puts `~/.inspec/gems` on `Gem.path` and + *activates* managed gems with version constraints + (`activate_managed_gems_for_plugin`). So activation of a plugin transport is, again, a + client responsibility. +- **Tension:** The in-core `require "train/transports/"` shortcut (§4.1) means bundled + transports load by a different mechanism than plugin transports — two code paths for one + concept. And activation-with-constraints lives in the client, not Train. +- **v2:** Collapse to **one** loading path via the Core plugin host: `local`/`mock` load + in-core, everything else activates as a managed gem through the same facility, with the + version/capability handshake (§4.5) applied uniformly. + +### 4.4 Plugin registration +- **What:** Making a loaded transport findable by name. +- **Current locus:** **Core**, via a **global mutable singleton**: `self.name(name)` writes + `Train::Plugins.registry[name] = self` (`plugins/transport.rb`), and `registry` is a + process-wide `@registry ||= {}` (`plugins.rb`). +- **Tension:** Global mutable state: no isolation between clients embedding Train, no + namespacing, last-registration-wins, and no way to run two API versions side by side. +- **v2:** **Core, scoped/isolated** registry (an injectable registry instance), versioned so + a v1 and v2 transport can coexist during migration. + +### 4.5 API versioning & capability negotiation **[+]** +- **What:** How a plugin declares which API it targets, and how Core decides what to offer it. +- **Current locus:** **Core**, but frozen: `Train.plugin(version=1)` raises for anything but + `1` (`plugins.rb`). There is exactly one version and no negotiation. +- **Tension:** This is *the* v2 seam. Today a plugin author gets an all-or-nothing base class; + there is no way to opt into new facilities, advertise what it supports, or migrate + incrementally. +- **v2:** **Core-owned negotiated version.** `Train.plugin(2)` returns a base whose available + mixins are gated by a declared **capability set** (§4.6). Version + capabilities together + replace the current monolith. + +### 4.6 Discovery & capability advertisement **[+]** +- **What:** Enumerating available transports and knowing what each can *do* (run commands? + files? http? platform detect? upload?). +- **Current locus:** **Client, by class-name sniffing** — even though the client-embedded + plugin host *already enumerates installed plugins* (`list_installed_plugin_gems` filters + gems by `train-`/`inspec-` prefix; `plugins.json` records them). Enumeration exists; a + *capability* API does not, so consumers still reach past the interface: InSpec's `http` + resource branches on `is_a?(Train::Transports::Local::Connection)`; the resource base + bypasses platform support when `backend.class == Train::Transports::Mock::Connection`; Chef + Target Mode makes similar class checks. (See [InSpec study §9](../integrations/inspec.md) and + [Chef study](../integrations/chef-infra-client-agentless.md).) +- **Tension:** Class-name sniffing is the single most pervasive smell in the ecosystem. It + couples clients to concrete transport classes, breaks encapsulation, and makes new + transports invisible unless clients are patched. The plugin host can tell you a transport is + *installed*, but nothing tells you what it can *do*. +- **v2:** **Core + Transport capability manifest.** Transports *declare* capabilities + (`:command`, `:file_read`, `:file_write`, `:http`, `:platform_detect`, `:interactive`); + Core exposes `connection.supports?(:http)` and clients query capabilities instead of + sniffing classes. + +--- + +## 5. Group B — Configuration & connection setup + +### 5.1 Option definition (the option DSL) +- **What:** Declaring a transport's configuration options with defaults/validation. +- **Current locus:** **Core provides the DSL, Transport declares.** `Train::Options` gives + `option name, default:/required:/coerce:` and `include_options` (`lib/train/options.rb`); + transports call it at class level. +- **Tension:** Adequate, but the **audit-log options are bolted on beside the DSL** + (`default_audit_log_options`, `validate_audit_log_options`) with comments admitting they + are kept separate "so it will not break existing functionality." Cross-cutting options are + not composable through the same mechanism. +- **v2:** Keep the DSL; make cross-cutting option groups (audit, logging, timeouts) first-class + **composable modules** rather than special-cased methods. + +### 5.2 Option namespacing / prefixing / merge precedence **[+]** +- **What:** Turning CLI flags / env / config files into a validated per-transport option hash, + with a defined precedence. +- **Current locus:** **Client + Core.** Clients filter/unprefix using `Train.options(name)` + as the authoritative key list (InSpec's `_utc_merge_transport_options`; Chef's Target Mode + does the same). `Train.options(name)` is therefore a **de-facto public contract**. +- **Tension:** The precedence (CLI > env > file > default) lives entirely in each client and + is re-implemented per consumer; Core only supplies the key list. +- **v2:** Keep marshalling in the client, but have Core **formally publish** the option schema + (types, precedence hints, sensitivity) so clients stop re-deriving it. + +### 5.3 URL / target parsing +- **What:** Turning `scheme://user:pass@host:port/path?query` into a credentials hash and a + transport name. +- **Current locus:** **Core**, `Train.unpack_target_from_uri` / `parse_uri` (`lib/train.rb`). +- **Tension:** Core contains **transport-specific hacks**: `www_form_encoded_password` for one + case, and literal `# TODO: move logic into winrm plugin` / `# move logic into SSH plugin` + comments where core special-cases empty paths and key handling. The generic parser knows too + much about specific transports. +- **v2:** **Core stays generic**; transports provide an optional **URI-parsing hook** to + interpret their own scheme (as the code comments already wish). New transports are + encouraged toward `transport://credset` (RFC-099) to avoid bespoke field mapping. + +### 5.4 Credential access & credential sets +- **What:** Sourcing secrets, including layered credential files. +- **Current locus:** **Client.** InSpec implements RFC-099 + `credentials//` resolution in `config.rb`; Chef resolves its own. +- **Tension:** Every client re-implements credential-set precedence; Core has no notion of it. +- **v2:** Keep sourcing in the client, but define a **Core credential schema** (which options + are credentials, which are sensitive) so redaction (§5.6) and validation are consistent. + +### 5.5 Auth & privilege escalation +- **What:** Authenticating (keys, passwords, tokens) and elevating (sudo, runas). +- **Current locus:** **Transport.** SSH owns keys + `sudo`/`sudo_password`/`sudo_options` and + `with_sudo_pty`; WinRM owns its auth; cloud transports own token/credential chains. +- **Tension:** Reasonable that auth is transport-specific, but **sudo/PTY logic is duplicated** + across shell transports with no shared helper, and `with_sudo_pty` is a no-op in the base. +- **v2:** Auth stays with the Transport; extract a **Core shell-escalation helper** shared by + shell-family transports. + +### 5.6 Secret / sensitive option handling & redaction **[+]** +- **What:** Marking options as secret so they are never logged/serialized in cleartext. +- **Current locus:** **Nobody.** There is no `sensitive` flag anywhere in Train (`grep + sensitive|redact lib/train` → nothing). Passwords/tokens flow through option hashes and can + surface in logs, audit records, and `inspect` output. +- **Tension:** A real security gap. The audit-log subsystem writes option-derived fields; the + diagnostic logger debug-prints; neither redacts. +- **v2:** **Core option metadata** (`option :password, sensitive: true`) enforced centrally in + logging, audit, and `inspect`. + +### 5.7 Backend validation **[+]** +- **What:** Rejecting a misconfigured target before connecting. +- **Current locus:** **Core.** `Train.validate_backend` picks the transport; `validate_options` + enforces `required:` options (`options.rb`). +- **Tension:** Minimal — validation is shallow (required-key presence only; no type or + cross-field checks) and audit-log options are validated by a *separate* method. +- **v2:** Keep in Core; deepen to schema-based validation (types, mutually-exclusive fields) + and fold audit-option validation into the same path. + +### 5.8 Connection lifecycle **[+]** +- **What:** Open → ready → use → close (and reconnect) semantics. +- **Current locus:** **Transport.** `connection()` builds it; `close`, `wait_until_ready`, + `login_command` are base no-ops/`NotImplementedError` each transport fills in + (`base_connection.rb`). +- **Tension:** No contract for *when* Core or clients should call `close`; lifecycle is + implicit and leaks (InSpec keeps connections alive by caching the whole backend). +- **v2:** Transports still implement open/close, but **Core owns the lifecycle contract** — + defined states, guaranteed teardown, reconnect policy. + +### 5.9 Connection reuse / pooling +- **What:** Avoiding re-dialing the same target. +- **Current locus:** **Transport + Client.** Each transport reinvents reuse (SSH keeps + `@connection` + `reusable_connection?` comparing `@connection_options`, + `ssh.rb:83,125-128,285-319`); clients additionally cache the whole backend object. +- **Tension:** Reuse logic is duplicated per transport and again per client; there is no shared + pool, no keyed cache in Core (the Core cache is for *results*, not connections — §6.7). +- **v2:** **Core connection lifecycle/pool** keyed by connection options; transports implement + only open/close. + +--- + +## 6. Group C — Runtime facilities (the primitives) + +### 6.1 Command execution +- **What:** Run a command, get stdout/stderr/exit status, optionally streamed, with a timeout. +- **Current locus:** **Core dispatch, Transport implementation.** `BaseConnection#run_command` + handles caching, the audit hook, and an **arity shim** (1- vs 2-arg + `run_command_via_connection`) before delegating; transports implement the private method and + build `CommandResult` (`extras.rb:8` = `Struct.new(:stdout,:stderr,:exit_status)`). Timeouts + surface as `Train::CommandTimeoutReached`. A `data_handler` block enables streaming. +- **Tension:** The arity shim is a compatibility hack betraying that the signature is not a firm + contract. `CommandResult` has no exit-signal / duration / truncation fields. +- **v2:** Keep the dispatch/implement split; freeze a **richer result contract** and a single + command signature (options hash always present). + +### 6.2 File read / metadata +- **What:** `connection.file(path)` → uniform metadata (content, mode, owner, mtime, type…). +- **Current locus:** **Core dispatch, Transport implementation.** `BaseConnection#file` caches + and delegates to `file_via_connection`; per-OS file classes live under `lib/train/file/`. +- **Tension:** The `Train::File` API is large and implicitly contractual — arguably the second + hidden contract after platform detection (§6.6). Not every transport can honor all of it. +- **v2:** Keep the split; make the file API a **capability-gated** interface (a transport + declares which metadata it supports) rather than an all-or-nothing base class. + +### 6.3 File write / transfer **[+]** +- **What:** Writing file content and moving files (`content=`, `upload`, `download`). +- **Current locus:** **Core + Transport.** `BaseConnection` implements `upload`/`download` in + terms of `file(...).content` / `content=`; transports back the actual write. +- **Tension:** Train is commonly assumed read-only, but it is not — yet write support is + uneven and undiscoverable (no capability flag), so clients cannot know if `upload` works. +- **v2:** Keep Core orchestration; gate behind a declared `:file_write` capability (§4.6). + +### 6.4 HTTP / API request facility +- **What:** A uniform way to make an HTTP/API request through the transport. +- **Current locus:** **Absent — clients improvise.** Train has no HTTP primitive. InSpec's + `http` resource branches on local-vs-remote and uses Ruby `Net::HTTP` locally or a remote + worker; Chef Target Mode faked HTTP via `curl`/`wget`. API transports (aws/azure/gcp) expose + *client objects* (`gcp_compute_client`, etc.) instead of a request primitive. +- **Tension:** The most impactful missing primitive: every consumer reinvents remote HTTP, and + API transports each expose bespoke client accessors with no common shape. +- **v2:** **Core-defined request primitive** (`connection.http(...)` or similar), capability + gated, so consumers stop improvising and API transports share a surface. + +### 6.5 Interactive session / login / PTY **[+]** +- **What:** Opening an interactive shell or feeding stdin/PTY. +- **Current locus:** **Transport.** `login_command` (base raises `NotImplementedError`), + `with_sudo_pty`. +- **Tension:** Minor; genuinely transport-specific. +- **v2:** Stays with the Transport, capability gated (`:interactive`). + +### 6.6 Platform detection & OS/family taxonomy **[+]** +- **What:** Deciding *what the target is* — OS, release, family hierarchy, uuid. +- **Current locus:** **Core engine driven through Transport primitives.** `Detect.scan(backend)` + (`platforms/detect.rb:6`) runs a decision tree using the transport's `run_command`/`file`; + the taxonomy lives in `specifications/os.rb` and `specifications/api.rb`; API transports skip + detection via `force_platform!`. `Platforms.export` publishes the taxonomy (InSpec embeds it + in its JSON schema). +- **Tension:** This is arguably the **largest and least-acknowledged contract** in Train. The + detection engine *assumes* shell/file primitives exist, which is why API transports must + bypass it entirely. Adding a platform means editing core specifications — plugins cannot + contribute taxonomy. +- **v2:** Core keeps the engine + taxonomy, but transports declare a **detection strategy** + (`:shell` | `:api` | `:force`) and can **contribute platform definitions**, so detection is + not hardwired to shell transports. + +### 6.7 Result caching +- **What:** Memoizing file/command/api results within a connection. +- **Current locus:** **Core.** `BaseConnection` has a three-way cache: `:file` **on by + default**, `:command` and `:api_call` **off** (`base_connection.rb`), with + `enable_cache`/`disable_cache` and `UnknownCacheType`. +- **Tension:** Cache defaults are surprising (files cached, commands not) and are per-connection + policy that clients must know to override. It is separate from *connection* reuse (§5.9), + which is easy to confuse. +- **v2:** Keep in Core; make defaults explicit and per-capability; clearly separate *result* + cache from *connection* pool in naming. + +--- + +## 7. Group D — Cross-cutting operational concerns + +### 7.1 Error taxonomy & exception contract **[+]** +- **What:** The set of errors Core/transports raise and how clients handle them. +- **Current locus:** **Core defines classes, Client rescues them.** `lib/train/errors.rb`: + `Error` → `UserError`/`ClientError`/`TransportError` + `PlatformDetectionFailed`, + `CommandTimeoutReached`, `PluginLoadError`, etc. Clients `rescue Train::ClientError`, + `Train::TransportError` **by class name** (InSpec `backend.rb`, Chef Target Mode). +- **Tension:** Rescue-by-class couples clients to Core's class hierarchy; transports sometimes + raise generic `RuntimeError`; the `reason` symbol on `Error` is under-used. There is no + stable, documented error *code* contract. +- **v2:** **Core error contract** with declared, documented **error codes/categories** + (machine-readable `reason`), so clients switch on a code, not a class. + +### 7.2 Diagnostic logging **[+]** +- **What:** Developer/debug logging of connection activity. +- **Current locus:** **Core + Transport.** A `:logger` option threads a `Logger` into + `Transport`/`BaseConnection`; transports `logger.debug(...)`. +- **Tension:** Distinct from audit logging (§7.3) but not clearly separated; no redaction + (§5.6); default logger config is ad hoc (`Logger.new($stdout, level: :fatal)`). +- **v2:** Fold into a **Core cross-cutting middleware** layer with consistent redaction. + +### 7.3 Audit logging +- **What:** Structured record of every command/file/upload for compliance. +- **Current locus:** **Core (bolt-on).** `Train::AuditLog` (JSON logger) is instantiated in + `BaseConnection#initialize` from `enable_audit_log`/`audit_log_*` options, and hooks fire + inside `run_command`, `file`, and `upload`. +- **Tension:** Grafted on beside the option DSL (its own option set + own validation method, + per the code's own comments), duplicates option data into `@audit_log_data`, and logs + unredacted fields. +- **v2:** Promote to a proper **Core middleware/interceptor** around the primitive dispatch, + sharing redaction and option handling with diagnostic logging (§7.2). + +### 7.4 Serialization / marshalling **[+]** +- **What:** Persisting/transferring connection state (e.g. file cache to a remote worker or a + mock fixture). +- **Current locus:** **Core (partial).** `BaseConnection#to_json` / `load_json` serialize the + *file* cache into/out of the Mock transport's file objects. +- **Tension:** Only the file cache is covered; command results and platform aren't; it is + entangled with the Mock transport specifically. +- **v2:** Generalize into a **Core-owned serialization contract** for results/fixtures, + independent of Mock. + +### 7.5 Test mocking / fixtures +- **What:** Emulating targets for tests without real connectivity. +- **Current locus:** **Core (Mock transport) + Client.** `Train::Transports::Mock` and + `plugin_test_helper` live in core; InSpec builds the `mock_loader` (~40 emulated OSes) and, + critically, the resource base **special-cases the Mock class** to bypass platform-support + gating. +- **Tension:** Mock is a test double wired into *production* control flow via class-name checks + — the capability-discovery smell (§4.6) in its purest form. +- **v2:** Keep Mock in Core as a **first-class transport** advertising capabilities, so clients + test against capabilities rather than `== Mock::Connection`. + +### 7.6 Concurrency / thread-safety **[+]** +- **What:** Whether a connection may be shared across threads / used concurrently. +- **Current locus:** **Undefined.** No documented thread-safety guarantee; SSH keeps a single + mutable `@connection`; the global registry is shared. +- **Tension:** Consumers that parallelize (runners, servers) have no contract to rely on. +- **v2:** **Core-defined thread-safety contract** per connection (e.g. "not thread-safe; one + connection per thread") and a thread-safe registry. + +### 7.7 Documentation / help generation **[+]** +- **What:** Human-facing docs for a transport's options and capabilities. +- **Current locus:** **Transport (ad hoc).** Option help is prose in READMEs; nothing is + generated from the option DSL. +- **Tension:** Options carry no `desc:`/help metadata, so CLIs cannot auto-document transports. +- **v2:** Add help/description metadata to the option DSL and **generate** transport docs from + it in Core. + +--- + +## 8. Cross-cutting meta-problems + +Eight themes recur across the table above; a v2 design should treat these as first-order +goals, not incidental fixes: + +1. **Convention over contract.** The `train-` gem prefix and the `require` path are + load-bearing but unenforced and undiscoverable until failure (§4.1, §4.3). v2 should make + plugin identity *declared*, not filename-derived. + +2. **Class-name sniffing instead of capability negotiation.** The single most pervasive smell: + `is_a?(Local::Connection)`, `== Mock::Connection`, `local_transport?` across InSpec and + Chef (§4.6, §6.4, §7.5). v2 must provide a **capability manifest** so no consumer reaches + past the interface. + +3. **Global mutable singleton state.** One process-wide registry, last-writer-wins, no + isolation or versioning (§4.4). v2 needs scoped, versioned registration to allow migration. + +4. **Bolt-on cross-cutting concerns.** Audit logging (and to a lesser extent diagnostic + logging, redaction, timeouts) are grafted beside the core mechanisms rather than composed + through a middleware layer (§5.1, §7.2, §7.3). v2 should have one interceptor pipeline. + +5. **Leaky core.** Transport-specific knowledge is hardcoded into generic core code — + `www_form_encoded_password`, the `# move logic into ssh/winrm plugin` TODOs, the + detection engine assuming shell primitives (§5.3, §6.6). v2 should push these to + transport-provided hooks. + +6. **Rescue/read coupling to concrete types.** Clients depend on Core exception *classes* + (§7.1) and concrete transport *classes* (theme 2) rather than declared codes/capabilities. + v2 should expose stable machine-readable contracts (error codes, capability flags) so + consumers never bind to Ruby class identity. + +7. **Core over-bundles transports.** `train-core` ships ssh, docker, podman, gcp, azure, + vmware, and cisco_ios directly, with a privileged "in-core first" load path that ordinary + plugins don't get (§4.1). Only `local` and `mock` are structurally required in Core. v2 + should extract every other transport to its own gem loaded through the *same* path as + third-party plugins — so the built-in transports dogfood the plugin API. + +8. **Every client reinvents the plugin host.** InSpec already has a complete, mature transport + install/activate manager (private `~/.inspec/gems`, `Gem::Resolver`, `plugins.json`, + version-constrained activation) that is entirely separate from the system `gem` command + (§4.2, §4.3); Chef is expected to grow its own. Train — which *defines* what a transport is + — owns none of this, so the lifecycle is duplicated and divergent per consumer. v2 should + offer a shared, train-level plugin host that clients delegate to. + +The through-line: **V1 leaks implementation identity across every seam** — filenames, class +names, exception classes, hardcoded transport specifics, and privileged in-core transports — +while pushing the plugin *lifecycle* out to each client to reinvent. A v2 API is fundamentally +about replacing leaked identity with **declared contracts** (capabilities, error codes, option +schemas, lifecycle states) and pulling the **plugin host** into a shared, reusable facility so +that Core owns only `local`+`mock` and everyone — built-in or third-party — is a plugin on +equal terms. + +--- + +## 9. Open questions + +These need decisions before (or during) the v2 API design proper: + +1. **Migration model.** Can v1 and v2 transports coexist in one process (dual registry), or is + v2 a hard break? This gates the registry/versioning design (§4.4, §4.5). +2. **Capability granularity.** How fine-grained should capabilities be — coarse + (`:command`/`:file`/`:http`) or fine (`:file_write`, `:file_chmod`, `:command_stream`)? +3. **Is the file API too big?** Should the rich `Train::File` metadata surface be trimmed or + split into capability tiers (§6.2)? +4. **HTTP primitive shape.** A generic request/response object, or a thin adapter over + `Faraday`/`Net::HTTP`, and how do API-client transports (aws/gcp) map onto it (§6.4)? +5. **Platform taxonomy ownership.** Can plugins contribute platform definitions, or does the + taxonomy stay a curated core artifact that clients (InSpec schema) depend on (§6.6)? +6. **Where does credential-set resolution belong** long term — permanently in each client, or + a shared Core/`train`-level resolver (§5.4)? +7. **Thread-safety guarantee.** What contract do we commit to, given parallel runners + (§7.6)? +8. **Backward compatibility of `Train.options`.** It is a de-facto public contract for InSpec + *and* Chef; any change to option publishing must preserve it (§5.2). +9. **Plugin host ownership & reuse.** Should Train ship a reusable transport plugin host + (install/activate) that InSpec and Chef delegate to, or does each client keep its own? If + shared, does it subsume InSpec's mature `~/.inspec/gems` manager or wrap it (§4.2, §4.3)? +10. **Extracting the built-in transports.** What is the migration path to move ssh/docker/ + gcp/azure/vmware/cisco_ios out of `train-core` into gems without breaking consumers who + assume they are always present (§4.1)? Do `local`/`mock` stay the only in-core transports? + +--- + +## 10. Cross-references + +- [Train Plugin API — Version 1 (V1) Reference](./plugins-v1.md) — the current contract this + redesign starts from; especially [§7 Data contracts](./plugins-v1.md#7-data-contracts), + [§8 Options, target URIs & credentials](./plugins-v1.md#8-options-target-uris--credentials), + and [§13 Limitations & observations](./plugins-v1.md#13-limitations--observations-springboard-for-v2). +- [InSpec's usage of Train](../integrations/inspec.md) — the class-name-sniffing, + `Train.options` contract, and Mock-as-core evidence behind §4.6, §5.2, §7.5. +- [Chef Infra Client "Target Mode" (Agentless)](../integrations/chef-infra-client-agentless.md) + — the parallel consumer showing the missing HTTP primitive (§6.4) and the same option- + filtering contract (§5.2). diff --git a/context/local/integrations/chef-infra-client-agentless.md b/context/local/integrations/chef-infra-client-agentless.md new file mode 100644 index 00000000..2e1ac224 --- /dev/null +++ b/context/local/integrations/chef-infra-client-agentless.md @@ -0,0 +1,468 @@ +# Chef Infra Client "Target Mode" (Agentless) and its reliance on Train + +> Scope & sourcing: this document analyzes how Chef Infra Client's **Target Mode** (informally +> "Agentless") uses **Train**. It is based on a full clone of `chef/chef` (branch `main`) at +> `context/reference-repos/chef`, cross-referenced with the Train source and the Train Plugin +> V1 analysis in `../design/plugins-v1.md`. Every claim below cites a concrete file/path in +> the Chef tree. The final section is a deliberately critical assessment. + +## Contents + +1. [What "Target Mode" is](#1-what-target-mode-is) +2. [How it is turned on (config & CLI)](#2-how-it-is-turned-on-config--cli) +3. [The connection: `Chef::TrainTransport` → `Train.create`](#3-the-connection-cheftraintransport--traincreate) +4. [Credentials (RFC-099 profiles)](#4-credentials-rfc-099-profiles) +5. [The TargetIO shim layer](#5-the-targetio-shim-layer) +6. [Exactly which Train features Chef uses](#6-exactly-which-train-features-chef-uses) +7. [Resource opt-in & platform detection](#7-resource-opt-in--platform-detection) +8. [Where plugins fit in the ecosystem](#8-where-plugins-fit-in-the-ecosystem) +9. [End-to-end flow](#9-end-to-end-flow) +10. [Critical assessment](#10-critical-assessment) +11. [Appendix: file map & source references](#appendix-file-map--source-references) + +--- + +## 1. What "Target Mode" is + +Normally Chef Infra Client runs **on** the node it configures (an installed agent). **Target +Mode** inverts this: the client process runs on a workstation/bastion and manages a **remote** +target — a server you can't or don't want to install Ruby/Chef on (a network device, an +appliance, a locked-down host, or an API). This is the "agentless" story. + +The mechanism is: instead of touching the local filesystem and running local commands, Chef +routes its I/O and command execution **through a Train connection** to the remote target. The +`--target` option was designed as the entry point (`lib/chef/application/base.rb:219`, an +`option :target`), and the feature is labeled internally as "RFCxxx Target Mode support" +(`chef-config/lib/chef-config/config.rb`, just above `config_context :target_mode`). + +Train is the *only* remoting layer used — Chef does not implement its own SSH/WinRM. Chef's +gemspec makes the dependency explicit (`chef.gemspec:46-48`): + +```ruby +s.add_dependency "train-core", "~> 3.13", ">= 3.13.4" +s.add_dependency "train-winrm", ">= 0.2.17" +s.add_dependency "train-rest", ">= 0.4.1" # target mode with rest APIs +``` + +So Chef ships `train-core` (which contains the `ssh`, `local`, `docker`, … transports) plus +the `train-winrm` and `train-rest` plugin gems out of the box. + +![Target Mode architecture over Train](img/agentless-architecture.png) + +## 2. How it is turned on (config & CLI) + +Target Mode is a **config context** in `chef-config` +(`chef-config/lib/chef-config/config.rb`): + +```ruby +configurable(:target) # the --target shortcut value + +config_context :target_mode do + config_strict_mode false # do NOT validate keys — accept arbitrary Train options + default :enabled, false + default :protocol, "ssh" # default transport + # typical additional keys: host, user, password +end + +def self.target_mode? + target_mode.enabled +end +``` + +Two things are load-bearing here: + +- **`config_strict_mode false`** — the `target_mode` context intentionally accepts *any* key, + because those keys are really **Train transport options** (host, user, password, port, + `winrm_transport`, etc.). Chef does not want to enumerate every Train option, so it lets + them pass through and filters them later against `Train.options(protocol)` (see §3). +- **`protocol` defaults to `"ssh"`** — i.e. the default Train transport is SSH. + +The CLI wires `--target` in `Chef::Application::Client#reconfigure` +(`lib/chef/application/client.rb:122-138`): + +```ruby +if config[:target] || Chef::Config.target + require "ed25519" # net-ssh ed25519 key support + Chef::Config.target_mode.host = config[:target] || Chef::Config.target + if URI.parse(Chef::Config.target_mode.host).scheme + train_config = Train.unpack_target_from_uri(Chef::Config.target_mode.host) + Chef::Config.target_mode = train_config + end + # Capture the OPERATOR's Chef Server identity before target mode rewrites paths + Chef::Config[:api_client_name] ||= Chef::Config[:node_name] + Chef::Config[:api_client_key] ||= Chef::Config[:client_key] + Chef::Config.target_mode.enabled = true + Chef::Config.node_name = Chef::Config.target_mode.host +end +``` + +Notable details: + +- If `--target` is a **URI** (e.g. `winrm://user@host`), Chef calls + **`Train.unpack_target_from_uri`** — a *Train* helper — to split scheme/host/user/password + into the `target_mode` hash. This is a direct reuse of Train's URI credential parser. +- The **node identity is remapped**: `node_name` becomes the target host, and the operator's + original Chef Server API credentials are preserved in `api_client_name`/`api_client_key` + (`config.rb:490-493`). This is a subtle but important consequence — the Chef Server still + authenticates as the operator, while the *node object* is named after the target. Cache + paths are also namespaced by target host (`config.rb:382-386`, `428`). + +## 3. The connection: `Chef::TrainTransport` → `Train.create` + +The connection object is created lazily and memoized on the run context +(`lib/chef/run_context.rb:660-674`): + +```ruby +def transport # Train transport plugin instance + @transport ||= Chef::TrainTransport.new(logger).build_transport +end + +def transport_connection # Train BaseConnection + @transport_connection ||= transport&.connection +end +``` + +`Chef::TrainTransport` (`lib/chef/train_transport.rb`) is a thin wrapper that mixes in the +real logic from `ChefConfig::Mixin::TrainTransport` +(`chef-config/lib/chef-config/mixin/train_transport.rb`). The heart is `build_transport`: + +```ruby +def build_transport + return nil unless config.target_mode? + + tm_config = config.target_mode + credentials = load_credentials(tm_config.host) # RFC-099 profile (see §4) + protocol = credentials&.dig(:transport_protocol) || tm_config.protocol + + # Keep ONLY keys that the chosen Train transport actually understands: + train_config = tm_config.to_hash.select { |k| Train.options(protocol).key?(k) } + + if credentials + valid_settings = credentials.select { |k| Train.options(protocol).key?(k) } + valid_settings[:enable_password] = credentials[:enable_password] if credentials.key?(:enable_password) + train_config.merge!(valid_settings) + end + + train_config[:logger] = logger + Train.create(protocol, train_config) # <-- the Train handshake +rescue SocketError => e + e.message.replace "Error connecting to #{train_config[:target]} via #{protocol} - #{e.message}" + raise e +rescue Train::PluginLoadError + logger.error("Invalid target mode protocol: #{protocol}") + exit(1) +end +``` + +This single method is where almost all of Chef's *setup-time* dependence on Train lives. The +Train API surface used here: + +| Train API | Used for | +|---|---| +| `Train.options(protocol)` | Discover the option schema of the transport, to filter Chef config down to keys the transport accepts. Called for **both** the config context and the credentials file. | +| `Train.create(protocol, cfg)` | Load the transport plugin (by name = `protocol`) and instantiate it with the merged options. Returns a `Train::Plugins::Transport`. | +| `Train::PluginLoadError` | Distinguish "unknown/uninstalled transport" from other failures → friendly error + `exit(1)`. | +| `Train.unpack_target_from_uri` | (in the CLI, §2) parse a `scheme://…` target into credential fields. | +| `transport.connection` | (in run_context) obtain the `BaseConnection`. | + +The comment *"Train handles connection retries for us"* is telling: Chef deliberately offloads +retry/backoff to the transport plugin rather than owning it. + +![Connection setup & resource execution](img/agentless-connection-sequence.png) + +## 4. Credentials (RFC-099 profiles) + +Target credentials live in a **TOML** file of named profiles (one per target), following the +Chef RFC-099 "credentials" convention. `ChefConfig::Mixin::TrainTransport` (via +`ChefConfig::Mixin::Credentials`) resolves the file with this precedence +(`credentials_file_path`): + +1. `ENV["CHEF_CREDENTIALS_FILE"]` +2. `target_mode.credentials_file` +3. `/etc/chef//credentials` +4. `$HOME/.chef/target_credentials` + +`load_credentials(profile)` parses the TOML, symbolizes keys "to match `Train.options()`", and +warns about a very common mistake — an unquoted FQDN like `[host.example.org]` becomes a +nested TOML table instead of a profile name (`contains_split_fqdn?`). A profile can override +the transport with a `transport_protocol` key, which is why `build_transport` prefers +`credentials[:transport_protocol]` over `target_mode.protocol`. + +The important architectural point: **credentials are expressed in Train's vocabulary.** Chef +does not define its own credential schema; a profile is essentially "a bag of Train options" +that is validated against `Train.options(protocol)`. + +## 5. The TargetIO shim layer + +Setting up a connection is only half the story. For a *converge* to work agentlessly, every +place in Chef that would normally touch `File`, `Dir`, `Etc`, `FileUtils`, `IO`, HTTP, or the +shadow password DB must transparently redirect to the remote target. Chef does this with a +**`TargetIO::*` shim layer** (`lib/chef/target_io/`). + +Each shim is a constant that resolves to either the native class or a Train-backed compat +class, keyed on `Config.target_mode?`. For example `lib/chef/target_io/file.rb`: + +```ruby +backend = ChefConfig::Config.target_mode? ? TrainCompat::File : ::File +``` + +The same pattern appears for `Dir`, `Etc`, `IO`, `FileUtils`, `Shadow` +(`target_io/{dir,etc,io,fileutils,shadow}.rb`). Providers are written against +`::TargetIO::File`, `::TargetIO::Dir`, etc., so the *same provider code* runs locally or +remotely with no branching (e.g. `lib/chef/provider/file.rb:103,402` uses `::TargetIO::File`). + +The Train-backed implementations live under `lib/chef/target_io/train/` and all funnel through +`TargetIO::Support` (`lib/chef/target_io/support.rb`), which is the actual adapter to the Train +connection: + +```ruby +def read_file(filename) + # sudo path stages the file into a readable temp dir first + transport_connection.file(accessible_file).content +end +def write_file(remote, content) # tempfile -> upload + upload(tempfile.path, remote) +end +def upload(local, remote) # transport_connection.upload(...) +def run_command(cmd) # transport_connection.run_command(cmd) +def sudo? transport_connection.transport_options[:sudo] end +def remote_user transport_connection.transport_options[:user] end +def transport_connection Chef.run_context&.transport_connection end +``` + +Highlights of the compat classes: + +- **`TrainCompat::File`** (`target_io/train/file.rb`) is the richest. `read`/`readlines` go + through `Support#read_file` → `connection.file(path).content`. A large `method_missing` + routes: `stat`/`lstat` → `connection.file(path, follow_symlink).stat` (wrapped in an + `OpenStruct`); `mode`/`owner`/`group`/`uid`/`gid`/`size`/`selinux_label` → + `connection.file(path).stat[...]`; `mtime` → `connection.file(path).mtime` (converted to a + `Time`); `exist?`/`file?`/`symlink?`/etc. → `connection.file(path).send(...)`; write-ish + ops (`chmod`, `chown`, `delete`, `symlink`) are **redirected to `TargetIO::FileUtils`** + (i.e. remote shell commands). Pure path math (`join`, `dirname`, `extname`) is done + **locally**. +- **`TrainCompat::FileUtils`** (`target_io/train/fileutils.rb`) implements ~16 operations by + **shelling out** (`run_command("cp …")`, `mv`, `rm`, `mkdir`, `chmod`, `chown`, …). +- **`TrainCompat::Etc`** (`target_io/train/etc.rb`) has **no Train primitive of its own** — it + reads `/etc/passwd` and `/etc/group` via `TargetIO::File.read` and parses them in Ruby + (the parser is explicitly *"Courtesy of InSpec"*), reconstructing `Etc::Passwd`/`Etc::Group` + structs. This is a strong signal of the InSpec/Train lineage. +- **`TrainCompat::HTTP`** (`target_io/train/http.rb`) is the most surprising: there is **no + HTTP transport primitive in Train**, so Chef synthesizes HTTP by running `curl`/`wget` **on + the target** via `run_command` (it even `which`es for the binary first). `remote_file` over + HTTP in target mode therefore requires curl/wget on the target. +- **`TrainCompat::Shadow`** and `Etc` guard with `!transport_connection.os.unix?` (they raise + or bail on non-Unix), using Train's platform detection. + +![TargetIO dispatch and Train primitives](img/targetio-dispatch.png) + +## 6. Exactly which Train features Chef uses + +Distilling §3–§5, the **complete** set of Train capabilities Chef depends on: + +**Setup time (once, in `build_transport`/CLI):** +- `Train.options(name)` — option schema introspection (used to filter config + credentials). +- `Train.create(name, opts)` — plugin load + transport instantiation. +- `Train.unpack_target_from_uri(uri)` — URI → credential fields. +- `Train::PluginLoadError` — error taxonomy. +- `transport.connection` — obtain the `BaseConnection`. + +**Runtime (per operation, via `TargetIO::Support` + compat classes):** +- `connection.run_command(cmd)` → `CommandResult` (`.stdout`, `.exit_status`). The single most + used primitive (all of `FileUtils`, `HTTP`, parts of `File`/`Dir`). +- `connection.file(path[, follow_symlink])` → a `Train::File`, from which Chef reads + `.content`, `.stat`, `.mode`, `.owner`, `.group`, `.uid`, `.gid`, `.size`, + `.selinux_label`, `.mtime`, `.exist?`, `.file?`, `.symlink?`, `.block_device?`, + `.character_device?`. +- `connection.upload(local, remote)` — file writes (via a local tempfile then upload; `sudo` + writes stage through a temp dir). +- `connection.os` / `connection.platform` — platform gating (`.unix?`), also handed to Ohai. +- `connection.transport_options[:sudo | :user]` — privilege/user awareness. + +Notably Chef does **not** use Train's caching API, `login_command`, `wait_until_ready`, or the +JSON marshalling — it treats the connection as a live command/file/upload pipe plus a platform +oracle. + +## 7. Resource opt-in & platform detection + +**Not every resource works in target mode.** Availability is gated in `Chef::NodeMap` +(`lib/chef/node_map.rb`). The `set` signature carries two mode flags +(`node_map.rb:69`): `target_mode: nil` (default) and `agent_mode: true` (default). Matching +logic (`node_map.rb:255-283`): + +```ruby +def matches_target_mode?(filters) + return true unless Chef::Config.target_mode? + !!filters[:target_mode] # must explicitly opt in +end +def matches_agent_mode?(filters) + return true if Chef::Config.target_mode? + !!filters[:agent_mode] # default true +end +``` + +So in target mode a resource is only selected if it was registered with +`target_mode: true`. Resources opt in through their `provides` call, e.g. +`lib/chef/resource/apt_package.rb:25-27`: + +```ruby +provides :apt_package, target_mode: true +provides :package, platform_family: "debian", target_mode: true +target_mode support: :full +``` + +At the time of writing, **94** resource files declare `target_mode: true` +(`grep -rlE 'target_mode:\s*true' lib/chef/resource/ | wc -l`). `Resource.target_mode` +(`lib/chef/resource.rb:1495`) is a *documentation-only* keyword recording the support level +(`:full`, etc.) surfaced by `resource_inspector`. + +**Platform detection** in target mode is delegated to Ohai *running over the Train +connection*: `lib/chef/client.rb:614` sets `ohai.transport_connection = transport_connection if +Chef::Config.target_mode?`. Individual shims also consult `transport_connection.os` directly +(e.g. `target_io/etc.rb`, `target_io/shadow.rb` require `os.unix?`). Thus Train's platform +family/`force_platform!` machinery (see `../design/plugins-v1.md` §6) becomes Chef's notion of +the target's OS. + +## 8. Where plugins fit in the ecosystem + +Target Mode consumes Train **transport plugins** exactly as any Train host would (see the V1 +plugin model in `../design/plugins-v1.md`). The `protocol` config value *is* the plugin's +registered `name`: + +- `protocol "ssh"` / `"winrm"` / `"local"` / `"docker"` → transports bundled in `train-core` + or the `train-winrm` gem Chef depends on. +- `protocol "rest"` → the `train-rest` gem (Chef ships it specifically for "target mode with + rest APIs", per the gemspec comment). +- **Any third-party `train-` gem** works automatically: `Train.create(protocol, …)` + will `require "train-"` on a registry miss (Train's loader, `lib/train.rb`), so + installing a plugin gem is enough to add a new Chef target type. No Chef-side change is + required. + +Conversely, a plugin must satisfy the runtime contract Chef relies on (§6). A **pure-API +transport** like `train-aws` (which implements *neither* `run_command_via_connection` nor +`file_via_connection`; see `../design/plugins-v1.md` §12.3) is essentially unusable for a +general Chef converge — the `TargetIO` shims would call `run_command`/`file` and hit +`NotImplementedError`. This is why the practically useful Chef target transports are the +shell/file-capable ones (ssh, winrm, and command-shims like k8s), while REST targets are +handled by purpose-built resources (e.g. the `*_api`/inspec resources) rather than the generic +file/command providers. + +There is also a **naming/adjacency subtlety** worth calling out: the InSpec resource-pack +gems (e.g. `inspec-gcp-resources`, which sit on top of a Train transport such as the in-core +`gcp` transport described in `../design/plugins-v1.md` §12.5) are **not** Train plugins and +play no role in Chef Target Mode's transport; they belong to InSpec's own plugin system. Chef Target Mode ↔ Train transport plugins is a strictly separate axis from +InSpec ↔ resource packs, even though both ecosystems share Train and overlap in code lineage +(the `/etc/passwd` parser "Courtesy of InSpec" is a visible seam). + +## 9. End-to-end flow + +1. Operator runs `chef-client --target host` (or sets `target_mode` in config). +2. `Application::Client#reconfigure` sets `target_mode.host`, optionally unpacks a URI via + `Train.unpack_target_from_uri`, preserves the operator's Chef Server identity, flips + `target_mode.enabled = true`, and renames the node to the target. +3. On first access to `run_context.transport_connection`, `Chef::TrainTransport#build_transport` + loads the RFC-099 credentials profile, chooses the `protocol`, filters config+credentials + through `Train.options(protocol)`, and calls `Train.create(protocol, cfg)` → `.connection`. +4. Ohai is given the connection and detects the target's platform over Train. +5. During converge, only resources registered `target_mode: true` are available. Their + providers use `::TargetIO::{File,Dir,Etc,FileUtils,IO,HTTP,Shadow}`. +6. Each `TargetIO` call resolves to a `TrainCompat` class → `TargetIO::Support` → + `transport_connection.{file,run_command,upload}` → the transport plugin → the target. +7. Reads return `Train::File#content`/`#stat`; commands return `CommandResult`; writes go + through a local tempfile + `upload` (staged through a temp dir when `sudo`). + +## 10. Critical assessment + +Strengths: + +- **Clean separation of concerns.** Chef owns *what* to do (resources/providers) and defers + *how to reach the target* entirely to Train. The `TargetIO` indirection means the vast + majority of provider code is transport-agnostic — the same `provider/file.rb` runs locally + or remotely. That is a genuinely elegant reuse of the Train abstraction. +- **Zero-integration plugin extensibility.** Because `protocol` maps to a Train plugin name + and Train auto-`require`s `train-`, new target types are added by installing a gem. + Chef inherits Train's whole plugin ecosystem for free. +- **Options/credentials expressed in Train's vocabulary.** Filtering through + `Train.options(protocol)` avoids Chef re-declaring every transport's option and keeps the + two projects loosely coupled. + +Weaknesses & risks (the critical part): + +1. **The abstraction is leaky and Unix-shell-centric.** Large parts of the compat layer are + not really "remote I/O" — they are **`run_command` with hardcoded coreutils** (`cp`, `mv`, + `rm`, `chmod`, `realpath`, `readlink`, `test -r`). `TrainCompat::File#realpath` even notes + *"coreutils, not MacOSX"*. This silently assumes a POSIX shell with GNU utilities on the + target, undermining the promise of a uniform transport. WinRM/PowerShell or minimal + busybox targets will hit gaps. +2. **HTTP is faked via `curl`/`wget` on the target.** `TrainCompat::HTTP` shells out because + Train has no HTTP primitive. This is fragile (depends on a binary being present and on the + argument formats), and it explicitly *"does not yet support sending data"* for + PUT/POST bodies. `remote_file` from an HTTP source in target mode is therefore only + partially functional. +3. **Capability mismatch with the plugin model.** Train V1 lets a plugin implement *neither* + files nor commands (e.g. `train-aws`), but Chef's generic providers assume *both*. There is + no capability negotiation — a resource that isn't target-aware, or a transport that lacks + `run_command`/`file`, fails at runtime with `NotImplementedError` rather than being + detected up front. The `target_mode: true` opt-in gate mitigates this only for the 94 + curated resources; everything else is simply absent in target mode. +4. **Performance.** Each `File`/`FileUtils`/`Etc` operation is a separate round trip + (`run_command` or `file`), and Chef does **not** use Train's caching. A converge that + touches many files or shells out repeatedly can be very chatty over SSH/WinRM. `Etc` + re-reads and re-parses `/etc/passwd`/`/etc/group` on every lookup. +5. **`sudo` file writes are non-atomic and racy.** `Support#upload`/`write_file` upload to a + staging dir then `mv` into place via a *second* command; the staging cleanup swallows + `Errno::ENOENT`. This is a reasonable workaround for privilege boundaries but weaker than + Chef's local atomic-write guarantees. +6. **Partial/inconsistent implementation.** The compat classes are peppered with + `raise NotImplementedError` (`File.new`, block-less `File.foreach`, non-read `File.open` + modes) and commented-out methods (`ftype`). The surface is "enough to run the curated + resources," not a complete `File`/`IO` replacement, so third-party cookbooks that reach for + unsupported stdlib calls will break in subtle ways. +7. **Identity remapping is subtle.** Renaming `node_name` to the target host while keeping the + operator's API keys (`api_client_name`/`api_client_key`) is clever but easy to + misconfigure; credentials-file path resolution changes once target mode is enabled, which + is exactly why the code has to capture the operator identity *before* flipping the flag. + This ordering dependency is a latent foot-gun. +8. **Reliance on Train internals/lineage.** Parsing `/etc/passwd` "Courtesy of InSpec" and + depending on `Train::File#stat` shape (wrapped in `OpenStruct`) couples Chef to Train's + file object semantics beyond the documented `DATA_FIELDS`. Changes in a transport's `stat` + contents would ripple into Chef. + +Bottom line: Target Mode is a **thin, pragmatic remoting adapter** — it leans on Train for the +transport handshake and for `run_command`/`file`/`upload`, then rebuilds a *Unix-flavored* +stdlib on top of `run_command`. It is powerful for SSH/WinRM Unix-like targets and for the +curated resource set, but the "agentless everywhere" ambition is bounded by (a) the +shell-command assumptions baked into the compat layer, (b) the lack of capability negotiation +with the plugin API, and (c) the absence of caching/atomicity guarantees. A future Train +plugin API redesign (the eventual goal of the `../design/` docs) could address several of +these by making capabilities explicit (files? commands? http? atomic writes?) and by offering +first-class primitives (an HTTP verb, a stat contract, batched/streamed I/O) that Chef today +has to synthesize. + +## Appendix: file map & source references + +Chef side (`context/reference-repos/chef`): + +- CLI/enable: `lib/chef/application/base.rb` (`option :target`), + `lib/chef/application/client.rb:122-138` (reconfigure). +- Config: `chef-config/lib/chef-config/config.rb` (`config_context :target_mode`, + `target_mode?`, cache-path namespacing, `api_client_name/key`). +- Connection: `lib/chef/run_context.rb:660-674` (`transport`/`transport_connection`), + `lib/chef/train_transport.rb`, + `chef-config/lib/chef-config/mixin/train_transport.rb` (`build_transport`, + `load_credentials`, `credentials_file_path`). +- Shim layer: `lib/chef/target_io/{file,dir,etc,fileutils,io,http,shadow}.rb`, + `lib/chef/target_io/train_compat.rb`, `lib/chef/target_io/support.rb`, + `lib/chef/target_io/train/{file,dir,etc,fileutils,http,shadow,io}.rb`. +- Gating & platform: `lib/chef/node_map.rb:69,255-283`, + `lib/chef/resource.rb:1495` (`target_mode` keyword), + `lib/chef/resource/apt_package.rb:25-27` (opt-in example), + `lib/chef/client.rb:614` (Ohai over transport). +- Dependencies: `chef.gemspec:46-48` (`train-core`, `train-winrm`, `train-rest`). + +Train side (see also `../design/plugins-v1.md`): `Train.create`, `Train.options`, +`Train.unpack_target_from_uri`, `Train::PluginLoadError`, `BaseConnection#{run_command,file, +upload,os,transport_options}`, `Train::File#{content,stat,mtime,…}`. + +Diagram sources: `img/src/*.puml` (regenerate with +`java -jar -tpng -o "$PWD/img" img/src/*.puml`). diff --git a/context/local/integrations/img/agentless-architecture.png b/context/local/integrations/img/agentless-architecture.png new file mode 100644 index 00000000..485b4cd0 Binary files /dev/null and b/context/local/integrations/img/agentless-architecture.png differ diff --git a/context/local/integrations/img/agentless-connection-sequence.png b/context/local/integrations/img/agentless-connection-sequence.png new file mode 100644 index 00000000..0d8e0b5d Binary files /dev/null and b/context/local/integrations/img/agentless-connection-sequence.png differ diff --git a/context/local/integrations/img/inspec-train-architecture.png b/context/local/integrations/img/inspec-train-architecture.png new file mode 100644 index 00000000..06982779 Binary files /dev/null and b/context/local/integrations/img/inspec-train-architecture.png differ diff --git a/context/local/integrations/img/inspec-train-connect-sequence.png b/context/local/integrations/img/inspec-train-connect-sequence.png new file mode 100644 index 00000000..3a1d161a Binary files /dev/null and b/context/local/integrations/img/inspec-train-connect-sequence.png differ diff --git a/context/local/integrations/img/inspec-train-resource-dispatch.png b/context/local/integrations/img/inspec-train-resource-dispatch.png new file mode 100644 index 00000000..1c46c2fd Binary files /dev/null and b/context/local/integrations/img/inspec-train-resource-dispatch.png differ diff --git a/context/local/integrations/img/src/agentless-architecture.puml b/context/local/integrations/img/src/agentless-architecture.puml new file mode 100644 index 00000000..808e6976 --- /dev/null +++ b/context/local/integrations/img/src/agentless-architecture.puml @@ -0,0 +1,54 @@ +@startuml agentless-architecture +title Chef Infra Client - Target Mode (Agentless) over Train + +skinparam shadowing false +skinparam defaultFontName Helvetica +skinparam componentStyle rectangle + +actor Operator + +package "Chef Infra Client (workstation / bastion)" { + [CLI: chef-client --target HOST\n(-t / target_mode)] as CLI + [ChefConfig::Config.target_mode\n(enabled, protocol, host, ...)] as Cfg + [Chef::TrainTransport\n(ChefConfig::Mixin::TrainTransport)] as TT + [run_context.transport /\ntransport_connection] as RC + + package "TargetIO shim layer" { + [TargetIO::File / Dir / Etc /\nFileUtils / IO / HTTP / Shadow] as TIO + [TargetIO::TrainCompat::*] as TC + [TargetIO::Support\n(read_file/write_file/upload/run_command)] as Sup + } + + [Ohai (transport_connection=)] as Ohai + [Resources / Providers\n(opt-in: provides target_mode: true)] as Res +} + +package "Train (train-core + plugins)" { + [Train.create(protocol, cfg)\nTrain.options(protocol)] as TrainF + [Transport plugin\n(ssh / winrm / rest / ...)] as Plugin + [BaseConnection\n(run_command / file / upload / os)] as Conn +} + +cloud "Remote target\n(host / device / API)" as Target + +Operator --> CLI +CLI --> Cfg : sets host, enabled=true +Cfg --> TT +TT --> TrainF : build_transport +TrainF --> Plugin : loads & instantiates +Plugin --> Conn : connection() +TT --> RC : transport +RC --> Conn : transport_connection + +Res --> TIO : ::TargetIO::File.read(...) +TIO --> TC : if target_mode? +TC --> Sup +Sup --> Conn : file(...).content / run_command / upload +Ohai --> Conn : run plugins over transport +Conn --> Target : SSH/WinRM/REST/... + +note bottom of TIO + When NOT in target mode, TargetIO::X + resolves to the native ::X (File, Dir, ...). +end note +@enduml diff --git a/context/local/integrations/img/src/agentless-connection-sequence.puml b/context/local/integrations/img/src/agentless-connection-sequence.puml new file mode 100644 index 00000000..72733d3e --- /dev/null +++ b/context/local/integrations/img/src/agentless-connection-sequence.puml @@ -0,0 +1,53 @@ +@startuml agentless-connection-sequence +title Target Mode - Connection Setup & Resource Execution + +skinparam shadowing false +skinparam defaultFontName Helvetica + +actor Operator +participant "Application::Client\n#reconfigure" as App +participant "ChefConfig::Config\n.target_mode" as Cfg +participant "Chef::TrainTransport\n(build_transport)" as TT +participant "Credentials file\n(target_credentials TOML)" as Creds +participant "Train" as Train +participant "BaseConnection" as Conn +participant "TargetIO::File\n+ Support" as TIO + +== Startup / reconfigure == +Operator -> App : chef-client --target host.example.org +App -> Cfg : target_mode.host = target\n(unpack URI if scheme present) +App -> Cfg : api_client_name/key := operator identity +App -> Cfg : target_mode.enabled = true\nnode_name = host + +== Lazy transport build (first use of transport_connection) == +App -> TT : run_context.transport_connection +activate TT +TT -> Creds : load_credentials(host) (RFC099 profile) +Creds --> TT : {user, password, protocol, ...} +TT -> Train : Train.options(protocol) +Train --> TT : valid option keys +TT -> TT : select tm_config + creds keys\nthat Train knows about +TT -> Train : Train.create(protocol, train_config) +activate Train +Train -> Conn : transport.connection() +Train --> TT : transport +deactivate Train +TT --> App : transport_connection (BaseConnection) +deactivate TT + +== Ohai / platform == +App -> Conn : ohai.transport_connection = conn\n(detect platform via Train) + +== Resource execution (converge) == +Operator -> TIO : ::TargetIO::File.read("/etc/passwd") +activate TIO +alt Config.target_mode? + TIO -> Conn : file(path).content\n(or run_command / upload) + Conn -> Conn : execute over SSH/WinRM/REST + Conn --> TIO : content / CommandResult +else not target mode + TIO -> TIO : delegate to native ::File +end +TIO --> Operator : result +deactivate TIO +@enduml diff --git a/context/local/integrations/img/src/inspec-train-architecture.puml b/context/local/integrations/img/src/inspec-train-architecture.puml new file mode 100644 index 00000000..89a40201 --- /dev/null +++ b/context/local/integrations/img/src/inspec-train-architecture.puml @@ -0,0 +1,62 @@ +@startuml inspec-train-architecture +title InSpec over Train — Layered Architecture & the Two Plugin Systems + +skinparam shadowing false +skinparam defaultFontName Helvetica +skinparam rectangle { + BorderColor #444444 + BackgroundColor #F7F7F7 +} + +package "InSpec (inspec-core gem)" #EAF2FF { + rectangle "CLI / base_cli\n-t/--target, -b/--backend,\n--host --sudo* --winrm_* --key_files\n(transport-specific flags)" as CLI + rectangle "Inspec::Config\nunpack_train_credentials:\n- determine backend (scheme://)\n- RFC-099 credset lookup\n- Train.options(name) remap\n- symbolize keys" as CFG + rectangle "Inspec::Runner\nBackend.create(conf);\nprofile.supports_platform?" as RUN + rectangle "Inspec::Backend (wrapper)\nbackend => Train connection\nlocal_transport?, add_resource_methods,\nmethod_missing -> resource DSL" as IB + rectangle "Resources (Inspec.resource(1))\ncommand / file / os / platform / http ...\ncall inspec.backend.{run_command,file,platform}" as RES + rectangle "Plugin v2 (Inspec.plugin(2))\ntypes: cli, dsl(input/resource_dsl),\nmock, reporter, streaming_reporter,\nresource_pack — NO transport type" as PV2 +} + +package "Train (train / train-core gem)" #EAFBEA { + rectangle "Facade\nTrain.create / validate_backend /\noptions / unpack_target_from_uri /\nplugin(1) / Plugins.registry" as TF + rectangle "BaseConnection (god object)\nrun_command · file · upload/download ·\nplatform detect · caching\n(file/command/api_call)" as BC + rectangle "Bundled transports (core):\nlocal, ssh, docker, podman,\nmock, azure, gcp, vmware, cisco_ios" as CORE + rectangle "External transport gems\n(Train v1 plugins):\ntrain-winrm, train-aws,\ntrain-kubernetes, train-habitat,\ntrain-rest" as EXT +} + +package "Targets" #FFF6E6 { + rectangle "OS/shell\n(local, ssh, winrm,\ndocker, podman, k8s exec)" as TOS + rectangle "Cloud/REST APIs\n(aws, azure, gcp, rest)" as TAPI +} + +CLI --> CFG +CFG --> RUN +RUN --> IB +IB --> RES +RES --> IB : inspec.backend.* +IB --> TF : Backend.create +TF --> BC +BC --> CORE +BC --> EXT +CORE --> TOS +EXT --> TOS +CORE --> TAPI +EXT --> TAPI + +' The bridge between the two plugin systems +PV2 ..> TF : registry bridge\nloaded_plugin?("train-x") ->\nTrain::Plugins.registry;\nplugin-mgr installs train-* gems + +note bottom of PV2 + Two SEPARATE plugin systems: + InSpec Plugin v2 owns resources/CLI/reporters (NOT transports). + Train v1 owns transports. InSpec only *bridges* to Train's + registry to recognize/install train-* gems. +end note + +note right of BC + InSpec uses a SMALL slice of the god object: + run_command, file, platform, hostname (+ .class checks). + No HTTP primitive exists -> http resource has its own path. +end note + +@enduml diff --git a/context/local/integrations/img/src/inspec-train-connect-sequence.puml b/context/local/integrations/img/src/inspec-train-connect-sequence.puml new file mode 100644 index 00000000..8f59a9ad --- /dev/null +++ b/context/local/integrations/img/src/inspec-train-connect-sequence.puml @@ -0,0 +1,47 @@ +@startuml inspec-train-connect-sequence +title Target Resolution & Backend Construction (InSpec -> Train) + +skinparam shadowing false +skinparam defaultFontName Helvetica +autonumber + +actor User +participant "base_cli\n(-t scheme://credset)" as CLI +participant "Inspec::Config" as CFG +participant "Inspec::Backend\n.create" as IB +participant "Train (facade)" as TRAIN +participant "Transport\n(train-)" as TR +participant "BaseConnection" as CONN + +User -> CLI : inspec exec ... -t winrm://prod +CLI -> CFG : new(cli_opts) +CFG -> CFG : unpack_train_credentials +activate CFG +CFG -> CFG : _utc_determine_backend\n(parse scheme:// -> :backend) +CFG -> CFG : _utc_merge_credset\n(RFC-099 credentials//)\nelse Train.unpack_target_from_uri +CFG -> TRAIN : Train.options(transport_name) +TRAIN --> CFG : known option keys +CFG -> CFG : remap/unprefix CLI opts\n(winrm_transport, sudo, ...)\n+ symbolize keys +CFG --> IB : train_credentials (Hash) +deactivate CFG + +IB -> TRAIN : validate_backend(creds) +TRAIN --> IB : transport_name +IB -> TRAIN : create(name, creds) +activate TRAIN +TRAIN -> TRAIN : load_transport\n(registry / require train-) +TRAIN -> TR : new(options) +TRAIN --> IB : transport +deactivate TRAIN +IB -> TR : connection +TR -> CONN : new(options) / connect +CONN --> IB : connection + +IB -> CONN : enable/disable_cache(:file,:command)\n(mock => always cached) +IB --> User : Inspec::Backend.new(connection) + +note over IB + Errors mapped: Train::ClientError / TransportError + / Errno::ENOENT -> friendly "can't connect" strings. +end note +@enduml diff --git a/context/local/integrations/img/src/inspec-train-resource-dispatch.puml b/context/local/integrations/img/src/inspec-train-resource-dispatch.puml new file mode 100644 index 00000000..c38f6b62 --- /dev/null +++ b/context/local/integrations/img/src/inspec-train-resource-dispatch.puml @@ -0,0 +1,45 @@ +@startuml inspec-train-resource-dispatch +title Resource -> Train Primitive Dispatch (runtime & test paths) + +skinparam shadowing false +skinparam defaultFontName Helvetica + +rectangle "Resource instance\n(Inspec.resource(1))" as R { +} +rectangle "command\ninspec.backend.run_command(cmd, timeout:)" as CMD +rectangle "file / directory\ninspec.backend.file(path)" as FILE +rectangle "os / platform\ninspec.backend.platform" as OS + +rectangle "Inspec::Backend\nbackend (Train connection)" as IB +rectangle "Train::Plugins::Transport::BaseConnection" as CONN + +rectangle "CommandResult\n(stdout, stderr, exit_status)" as CR +rectangle "Train::File::* object\n(content, mode, owner, ...)" as FO +rectangle "Train::Platforms::Platform\n(name, release, family_hierarchy)" as PL + +R --> CMD +R --> FILE +R --> OS +CMD --> IB +FILE --> IB +OS --> IB +IB --> CONN : delegates (small slice) +CONN --> CR +CONN --> FO +CONN --> PL + +note bottom of CONN + Runtime target: real transport (ssh/winrm/local/...). + Test target: Train::Transports::Mock::Connection via MockLoader + (mock_os + fixture files + mocked commands). +end note + +note right of R + Coupling seams: + * command resource rescues Train::CommandTimeoutReached. + * check_supported! is BYPASSED when + backend.backend is a Mock::Connection. + * http resource forks its own path when + inspec.local_transport? (Train has no HTTP primitive). +end note +@enduml diff --git a/context/local/integrations/img/src/targetio-dispatch.puml b/context/local/integrations/img/src/targetio-dispatch.puml new file mode 100644 index 00000000..69d109e5 --- /dev/null +++ b/context/local/integrations/img/src/targetio-dispatch.puml @@ -0,0 +1,68 @@ +@startuml targetio-dispatch +title TargetIO Shim Dispatch and Train Primitives Used + +skinparam shadowing false +skinparam defaultFontName Helvetica +skinparam classAttributeIconSize 0 + +class "TargetIO::File" as File { + read / readlines / open + exist? / stat / mode / owner ... + --dispatch-- + target_mode? ? TrainCompat::File : ::File +} +class "TargetIO::Dir" as Dir { + entries / glob / mktmpdir + target_mode? ? TrainCompat::Dir : ::Dir +} +class "TargetIO::Etc" as Etc { + getpwnam / getgrnam + (parses /etc/passwd via File.read) +} +class "TargetIO::FileUtils" as FU { + cp / mv / rm / mkdir / chmod / chown + (all via run_command) +} +class "TargetIO::IO" as IO +class "TargetIO::HTTP" as HTTP { + get/put/post via curl|wget + (run_command on target) +} +class "TargetIO::Shadow" as Shadow { + (guards: unix only) +} + +class "TargetIO::Support" as Support { + read_file(f) -> connection.file(f).content + write_file/upload -> connection.upload(...) + run_command(c) -> connection.run_command(c) + sudo? / remote_user -> connection.transport_options[...] + transport_connection = run_context.transport_connection +} + +class "Train BaseConnection" as Conn { + +file(path) -> Train::File (content, stat, mode, ...) + +run_command(cmd) -> CommandResult(stdout, exit_status) + +upload(local, remote) + +os / platform (unix? etc.) + +transport_options (sudo, user) +} + +File ..> Support +Dir ..> Support +Etc ..> File +FU ..> Support +IO ..> Support +HTTP ..> Support +Shadow ..> Support +Support ..> Conn : uses + +note bottom of Conn + Train features actually used by Chef: + file().content / .stat / .mode / .owner / .mtime / .exist? + run_command().stdout / .exit_status + upload() ; os.unix? ; transport_options[:sudo|:user] + (Train.create, Train.options, Train.unpack_target_from_uri, + Train::PluginLoadError at setup time) +end note +@enduml diff --git a/context/local/integrations/img/targetio-dispatch.png b/context/local/integrations/img/targetio-dispatch.png new file mode 100644 index 00000000..78679153 Binary files /dev/null and b/context/local/integrations/img/targetio-dispatch.png differ diff --git a/context/local/integrations/inspec.md b/context/local/integrations/inspec.md new file mode 100644 index 00000000..48656a59 --- /dev/null +++ b/context/local/integrations/inspec.md @@ -0,0 +1,357 @@ +# InSpec's Use of the Train Framework & Plugin System + +> Scope: a thorough, critical assessment of how **Chef InSpec** consumes **Train** — the +> transport interface — and how the two projects' plugin systems relate. Grounded in the +> `inspec/inspec` source (`context/reference-repos/inspec`, `train ~> 3.16`) and the Train +> source in this repo. Companion to [`../design/plugins-v1.md`](../design/plugins-v1.md) +> (the Train V1 plugin API) and +> [`chef-infra-client-agentless.md`](./chef-infra-client-agentless.md) (Chef's parallel use of +> Train). Written as baseline material for a future Train plugin-API redesign. + +## Contents + +1. [Executive summary](#1-executive-summary) +2. [Dependency & version surface](#2-dependency--version-surface) +3. [The two plugin systems (and why InSpec has no "transport" plugin type)](#3-the-two-plugin-systems) +4. [Target resolution & credentials](#4-target-resolution--credentials) +5. [Backend construction & caching](#5-backend-construction--caching) +6. [The `Inspec::Backend` wrapper](#6-the-inspecbackend-wrapper) +7. [Resources → Train primitives](#7-resources--train-primitives) +8. [Platform detection & schema export](#8-platform-detection--schema-export) +9. [Testing via the Mock transport](#9-testing-via-the-mock-transport) +10. [Critical assessment](#10-critical-assessment) +11. [Cross-references & sources](#11-cross-references--sources) + +--- + +## 1. Executive summary + +InSpec is, at its core, a **DSL and runner layered directly on top of Train**. Every audit +resource ultimately reaches its target through exactly three Train primitives — +`run_command`, `file`, and `platform` (plus an occasional `hostname` and `.class` check). +Train supplies *connectivity and platform identity*; InSpec supplies *resources, controls, +reporting, and a CLI*. + +![InSpec over Train — architecture](img/inspec-train-architecture.png) + +Three things stand out and drive the rest of this document: + +1. **The coupling is narrow but deep.** InSpec touches only a tiny slice of Train's + `BaseConnection` god-object API, yet it is *totally* dependent on it — there is no + abstraction seam or adapter; resources call `inspec.backend.run_command(...)` directly and + rescue `Train::` exception classes by name. +2. **There are two separate plugin systems.** InSpec's own Plugin v2 (`Inspec.plugin(2)`) has + plugin *types* for CLI/DSL/input/reporter/resource-pack/mock — but **not for transports**. + Transports live entirely in Train's v1 plugin system (`Train.plugin(1)`). InSpec only + *bridges* to Train's registry to recognize and install `train-*` gems. +3. **The Mock transport is a first-class citizen of InSpec's core.** Not just tests — the + resource base class special-cases `Train::Transports::Mock::Connection` to bypass its + platform-support gate. Train's test double is wired into InSpec's production code path. + +## 2. Dependency & version surface + +From the gemspecs: + +| Gem | Declares | Notes | +|---|---|---| +| `inspec-core` | `train-core ~> 3.16, >= 3.16.5` | the slim transport core | +| `inspec` | `train ~> 3.16, >= 3.16.5` | full Train (adds ssh/winrm helpers etc.) | +| `inspec` | `train-habitat ~> 0.1`, `train-aws ~> 0.2`, `train-winrm ~> 0.4.0`, `train-kubernetes >= 0.3.1` | "Train plugins we ship with InSpec" | + +Observations: + +- **Tight major-version pin to Train 3.x.** InSpec rides Train's 3.16 line closely; a Train + major bump (e.g. a V2 plugin API) is an InSpec-breaking event by construction. +- **Transport gems are split across two homes.** Train *core* bundles `local, ssh, docker, + podman, mock, azure, gcp, vmware, cisco_ios`; the *external* transports InSpec ships as + direct deps are `train-winrm, train-aws, train-kubernetes, train-habitat` (and, in the + Chef ecosystem, `train-rest` — now at `prospectra/train-rest`). See + [`../design/plugins-v1.md` §12](../design/plugins-v1.md#12-real-world-patterns--variances) + for the archetypes. +- **InSpec never `require`s the transport gems itself.** There is no `require "train-aws"` + anywhere in `lib/`. InSpec relies entirely on Train's `load_transport` fallback + (`require "train-"`) when `Train.create(name)` is first called. The gemspec deps only + guarantee the gems are *installed*; Train decides *when* to load them. + +## 3. The two plugin systems + +This is the single most important structural fact for a redesign. + +**InSpec Plugin v2** (`lib/inspec/plugin/v2/`) registers plugin *types* via +`register_plugin_type`, and the shipped types are: + +``` +cli · dsl (input / resource_dsl) · mock · reporter · streaming_reporter · resource_pack +``` + +There is **no `transport` plugin type**. InSpec deliberately does not own transports — it +delegates 100% of connectivity to Train's own, older, entirely separate v1 plugin system +(`Train.plugin(1)`, `Train::Plugins.registry`, keyed on Strings). + +The only coupling between the systems is a **registry bridge** used for *management*, not +*runtime dispatch*: + +```ruby +# lib/inspec/plugin/v2/registry.rb +def loaded_plugin?(name) + # HACK: ... unless it is a train plugin; then the Train::Registry is the source of truth. + return registry.dig(name.to_sym, :loaded) unless name.to_s.start_with?("train-") + Train::Plugins.registry.key?(name.to_s.sub(/^train-/, "")) +end +``` + +And the `inspec plugin install` command probe-loads a candidate gem and, if its name starts +with `train`, verifies it registered itself into **Train's** registry (not InSpec's): + +```ruby +# inspec-plugin-manager-cli/cli_command.rb +if plugin_name.to_s.start_with?("train") + registry_key = plugin_name.to_s.sub(/^train-/, "") + unless Train::Plugins.registry.key?(registry_key) + ui.red("... Ensure something inherits from 'Train.plugin(1)' - installation failed.") + end +end +``` + +So a `train-*` gem is a **second-class, specially-cased citizen** of InSpec's plugin manager: +InSpec can *install* it and *report* it as loaded, but it is owned, loaded, and dispatched by +Train. The `train-`/`inspec-` gem-name prefix is load-bearing — it is how InSpec decides which +registry to consult. (This is exactly the `inspec-`-vs-`train-` distinction analyzed in +[`../design/plugins-v1.md` §12.5](../design/plugins-v1.md#125-gcp--in-core-api-only-transport).) + +## 4. Target resolution & credentials + +All target/credential handling lives in `Inspec::Config#unpack_train_credentials` +(`lib/inspec/config.rb`) and produces the symbol-keyed hash Train expects. + +![Connect sequence](img/inspec-train-connect-sequence.png) + +The pipeline: + +1. **Determine the backend** (`_utc_determine_backend`). No `--target` ⇒ `backend = "local"`. + Otherwise the transport name is parsed from the URI scheme with a strict regex + `^(?[a-z_\-0-9]+)://` — a bare hostname with no `scheme://` is a hard error. +2. **Resolve a credential set** (`_utc_merge_credset`). InSpec implements **RFC-099** + credentials files: `-t winrm://prod` looks up `credentials → winrm → prod` in the config + file. If there is no credset match, it falls back to + `Train.unpack_target_from_uri(target)` — i.e. Train parses `scheme://user:pass@host:port`. +3. **Filter & un-prefix transport options** (`_utc_merge_transport_options`). InSpec asks + Train which options a transport accepts and copies the matching CLI flags in: + + ```ruby + transport_options = Train.options(transport_name).keys.map(&:to_s) + ``` + +4. **Symbolize keys** so they match Train's `option :name` declarations. + +**This is the identical mechanism Chef Target Mode uses** (`Train.options(protocol)` filtering); +see [`chef-infra-client-agentless.md` §4](./chef-infra-client-agentless.md). `Train.options` +is effectively a *public contract* two major consumers depend on for config marshalling. + +**Leaky abstraction — InSpec's CLI hardcodes transport-specific flags.** `base_cli.rb` defines +`--host`, `--sudo`/`--sudo_password`/`--sudo_command`, `--key_files`, `--ssl`, +`--winrm_transport`, `--winrm_shell_type`, `--winrm_disable_sspi`, `--ssh_config_file`, etc. +The "uniform" transport layer is not uniform at the UI: InSpec must know SSH- and WinRM-specific +option names, and `Train.options()` is the only thing keeping them loosely coupled. A new +transport with novel options cannot be fully driven from the InSpec CLI without InSpec changes +(unless the user drops to `-t scheme://…` URI form or a credentials file). + +## 5. Backend construction & caching + +`Inspec::Backend.create(config)` (`lib/inspec/backend.rb`) is the sole runtime entry point: + +```ruby +train_credentials = config.unpack_train_credentials +transport_name = Train.validate_backend(train_credentials) +transport = Train.create(transport_name, train_credentials) +connection = transport.connection +# caching policy... +Inspec::Backend.new(connection) +``` + +Notable details: + +- **Caching is InSpec's decision, applied to Train.** With `--backend-cache` (or *always* for + the Mock transport), InSpec calls `connection.enable_cache(:file)` and + `enable_cache(:command)`; otherwise it disables both. InSpec reaches directly into Train's + caching switches — another point where the god-object's internals are part of the contract. +- **Error mapping.** `Train::ClientError`, `Train::TransportError`, and `Errno::ENOENT` are + rescued and rewritten into human "can't connect to '' backend" strings. InSpec depends + on Train's exception taxonomy by class name. +- **`Train.validate_backend`** is trusted to normalize/resolve the backend name (including the + `local`-vs-explicit logic) before `create`. + +## 6. The `Inspec::Backend` wrapper + +`Inspec::Backend` is a thin façade around a single Train connection: + +- `#backend` → the raw Train connection. In a resource, `inspec.backend` **is** this Train + connection object, so resources call Train's connection API directly. +- `add_resource_methods` dynamically defines one method per registered resource, and + `method_missing` routes unknown names through the resource DSL — so `inspec.file(...)`, + `inspec.command(...)` etc. resolve to resources, while `inspec.backend.file(...)` reaches + Train. +- `local_transport?` = `backend.is_a?(Train::Transports::Local::Connection)` — a **hard + `is_a?` check on a specific Train class**, used by resources that want a Ruby-stdlib fast + path (only the `http` resource actually uses it today). +- `version` exposes the InSpec version; `inspect` prints `@transport=`. + +The wrapper adds almost no abstraction: it does not hide Train's API, normalize its data, or +insulate resources from Train class names. It is a convenience holder, not an adapter. + +## 7. Resources → Train primitives + +![Resource dispatch](img/inspec-train-resource-dispatch.png) + +Across all bundled resources, the entire Train runtime surface InSpec uses is: + +| Call | Count* | Consumers | +|---|---|---| +| `inspec.backend.run_command` | ~6 direct (+ every shell-based resource via `command`) | `command`, `bash`, `powershell`, `script`, … | +| `inspec.backend.file` | 2 direct (+ every file-based resource via `file`) | `file`, `directory`, config-file resources | +| `inspec.backend.platform` | 1 | `platform`/`os` resources, `supports` gate | +| `inspec.backend.hostname` | 1 | `hostname` resource | +| `inspec.backend.class` / `respond_to?` | few | local/mock detection, diagnostics | + +*direct call sites in `lib/inspec/resources`; most resources compose the `command`/`file` +resources rather than calling Train themselves. + +Representative seams: + +- **`command` resource** calls `inspec.backend.run_command(@command, timeout: @timeout)` and + **rescues `Train::CommandTimeoutReached`** to raise an InSpec `ResourceFailed`. The timeout + contract (and its quirky "sleep 0.1 so the connection isn't broken") is Train-specific + knowledge living in an InSpec resource. +- **`file` resource** wraps `inspec.backend.file(path)` and surfaces Train's file-object + metadata (content, mode, owner, …) as InSpec matchers. The data contract is Train's + `Train::File::*` object (see [`plugins-v1.md` §7](../design/plugins-v1.md#7-data-contracts)). +- **`os`/`platform` resources** expose `inspec.backend.platform` fields (`family`, `release`, + `arch`, `name`, `family_hierarchy`) and delegate matchers like `redhat?`, `linux?`, + `windows?` straight to Train's platform object. +- **`http` resource** is the exception that proves the rule: **Train has no HTTP primitive**, + so the resource branches on `inspec.local_transport?` — running Ruby `Net::HTTP` locally, or + shelling out to a remote worker otherwise. (Chef Target Mode hit the same gap and faked HTTP + via curl/wget; see [`chef-infra-client-agentless.md` §7](./chef-infra-client-agentless.md).) + +The `CommandResult = Struct.new(:stdout, :stderr, :exit_status)` contract is consumed verbatim +by InSpec matchers (`its('stdout')`, `its('exit_status')`). + +## 8. Platform detection & schema export + +InSpec does **not** implement OS detection — it reads whatever `connection.platform` returns +and builds its `supports`/`os` machinery on top: + +- The resource base's `check_supported!` calls `inspec.platform.supported?(@supports)` and + raises "Unsupported resource/backend combination" using `backend.platform.name`. +- **Schema/enum generation** literally boots a Mock connection and dumps Train's platform + table into InSpec's JSON schema: + + ```ruby + # lib/inspec/schema.rb & schema/output_schema.rb + Train.create("mock").connection + Train::Platforms.export + ``` + + So InSpec's *published output schema* embeds Train's platform taxonomy — a direct data + dependency on Train's `Platforms` registry. + +## 9. Testing via the Mock transport + +Train's Mock transport is the backbone of InSpec's ~hundreds of resource unit tests, via +`test/helpers/mock_loader.rb`: + +```ruby +@backend = Inspec::Backend.create(Inspec::Config.mock) # backend == Mock::Connection +mock = @backend.backend +mock.mock_os(@platform) # pick an emulated OS +# fixture files loaded through a REAL local connection, then handed to the mock: +local = Train.create("local", command_runner: :generic).connection +mock_files["/etc/passwd"] = local.file(fixture_path("passwd")) +``` + +`MockLoader::OPERATING_SYSTEMS` enumerates ~40 emulated platforms (centos5→8, ubuntu, windows +2016/2019/2025, aix, solaris, hpux, …) as `{name, family, release, arch}` hashes fed to +`mock_os`. Commands and files are pre-registered; the mock returns canned `CommandResult`s and +fixture-backed file objects. + +**The coupling that matters: InSpec's *production* code knows about the Mock transport.** + +```ruby +# lib/inspec/resource.rb#check_supported! +test_backend = defined?(Train::Transports::Mock::Connection) && + backend.backend.class == Train::Transports::Mock::Connection +unless supported || test_backend + raise ArgumentError, "Unsupported resource/backend combination..." +end +``` + +The platform-support gate is **deliberately bypassed when the backend is Train's Mock +connection**. A test double from the transport library is referenced by class name inside a +core control-flow decision. It works, but it is a textbook leaky abstraction: the abstraction +(Train connection) leaks a specific concrete subclass into InSpec's business logic. + +## 10. Critical assessment + +Strengths: + +- **Clean separation of concerns at the macro level.** InSpec = resources/DSL/runner/reporting; + Train = connectivity/platform. The division is real and has let both evolve (Target Mode in + Chef, resource packs in InSpec) on a shared substrate. +- **Small runtime surface.** Only `run_command`/`file`/`platform` are load-bearing, so in + principle Train's contract to InSpec is tiny and stable. +- **Config marshalling is data-driven** via `Train.options(name)`, avoiding (most) hardcoded + per-transport wiring in the resolver. + +Weaknesses / risks a redesign should target: + +1. **No adapter seam.** Resources call `inspec.backend.` and rescue `Train::*` + exceptions directly. There is no interface InSpec owns; a Train API change ripples into + dozens of resources. The "tiny surface" is not *encapsulated*, just *small*. +2. **The god-object leaks.** InSpec pokes Train's caching switches (`enable_cache(:file)`), + `is_a?(Train::Transports::Local::Connection)`, and `== Train::Transports::Mock::Connection`. + These are internals-as-contract. A V2 API should expose *capabilities* (can-run-commands, + can-read-files, is-local, is-mock) rather than concrete classes. +3. **Two plugin systems, one blurred boundary.** Newcomers cannot tell a `train-` transport + plugin from an `inspec-` resource pack without knowing the prefix convention, and InSpec's + plugin manager special-cases `train-*` by *string prefix*. A unified or explicitly-federated + plugin model (with a real "transport provider" contract on the InSpec side) would remove the + `HACK`-commented bridges. +4. **The abstraction isn't uniform at the edges.** No HTTP primitive (so `http` and Chef both + improvise); CLI hardcodes SSH/WinRM/sudo flags; API-only transports (aws/gcp/azure) don't + implement `run_command`/`file` at all, so resource packs built on them bypass the very + primitives InSpec's core is organized around. The "one uniform target" promise frays for + cloud/API targets. +5. **Mock-as-core.** Wiring `Train::Transports::Mock::Connection` into `check_supported!` + couples product behavior to a test fixture. A redesign should let resources declare + capability/support requirements that a mock can *satisfy* declaratively, instead of the core + branching on the mock's class. +6. **Version lock-step.** `train ~> 3.16` plus by-class-name coupling makes Train and InSpec a + de-facto single release train; a Train plugin-API V2 cannot land without a coordinated + InSpec major. + +Net: InSpec is a *well-behaved but intimate* consumer of Train. The relationship is healthy in +spirit (thin runner over a transport lib) but under-specified in contract — it leans on concrete +classes, exception taxonomy, cache internals, a test double, and a gem-name prefix convention +rather than on an explicit, versioned interface. Those are precisely the seams a V2 plugin API +should formalize. + +## 11. Cross-references & sources + +- Train V1 plugin API: [`../design/plugins-v1.md`](../design/plugins-v1.md) (esp. §5 + BaseConnection god-object, §7 data contracts, §12 real-world archetypes). +- Chef's parallel use of Train: [`chef-infra-client-agentless.md`](./chef-infra-client-agentless.md) + (`Train.options` filtering, missing HTTP primitive, `/etc/passwd` "Courtesy of InSpec"). +- InSpec sources (under `context/reference-repos/inspec/`): + `lib/inspec/backend.rb`, `lib/inspec/config.rb` (`unpack_train_credentials`), + `lib/inspec/runner.rb`, `lib/inspec/resource.rb` (`check_supported!`), + `lib/inspec/resources/{command,file,os,platform,http}.rb`, + `lib/inspec/plugin/v2/{registry,plugin_base}.rb`, + `lib/plugins/inspec-plugin-manager-cli/.../cli_command.rb`, + `lib/inspec/schema.rb`, `test/helpers/mock_loader.rb`, `inspec.gemspec`, + `inspec-core.gemspec`. +- Shared context consulted: + `context/shared/by-repo/inspec/inspec/interfaces/train-backends.md` (marked *INCOMPLETE*), + `context/shared/by-product/inspec/specifications/resource-pack-backends.md`. +- Diagrams: `img/inspec-train-architecture.png`, + `img/inspec-train-connect-sequence.png`, `img/inspec-train-resource-dispatch.png` + (regen: `java -jar plantuml.jar -tpng -o .. src/*.puml` from `img/src/`). diff --git a/etc/env.default.sh b/etc/env.default.sh new file mode 100644 index 00000000..55c6df45 --- /dev/null +++ b/etc/env.default.sh @@ -0,0 +1,7 @@ + +# Set this to an alternate location or branch to experiment +# with different shared context locations, then re-run /start-development +export PROGRESS_SHARED_CONTEXT_REPO=chef/shared-context@main + +# Chef/InSpec licensing +export CHEF_LICENSE="accept" diff --git a/etc/reference-repo-list.txt b/etc/reference-repo-list.txt new file mode 100644 index 00000000..72189985 --- /dev/null +++ b/etc/reference-repo-list.txt @@ -0,0 +1,19 @@ +# This file lists GitHub repos to clone as reference material for AI-assisted development. +# Edit this list to add repos relevant to your specific resource pack domain. +# Format: org/repo (one per line, optionally @branch) +# + +# Major client applications +inspec/inspec +chef/chef + +# Docs of client applications +inspec/chef-inspec-docs +chef/chef-web-docs + +# A variety of plugin (transport) implementations +inspec/train-winrm +prospectra/train-rest +inspec/train-aws +inspec/train-kubernetes +