From e2efafd41d6c12d87b2a50d0a9c442c6d7fb7a98 Mon Sep 17 00:00:00 2001 From: David Date: Tue, 8 Sep 2026 17:15:35 +0200 Subject: [PATCH 1/2] Add Vimeo importer and CLI authentication improvements --- README.md | 5 +- docs/development.md | 23 +- docs/usage.md | 151 ++++++++- lib/mave_cli/cli.ex | 165 +++++++++- lib/mave_cli/client.ex | 5 + lib/mave_cli/config.ex | 5 +- lib/mave_cli/output.ex | 4 + lib/mave_cli/vimeo.ex | 155 ++++++++++ lib/mave_cli/vimeo/auth.ex | 165 ++++++++++ lib/mave_cli/vimeo/catalog.ex | 158 ++++++++++ lib/mave_cli/vimeo/import.ex | 237 ++++++++++++++ lib/mave_cli/vimeo/rate_limit.ex | 113 +++++++ lib/mave_cli/vimeo/state.ex | 96 ++++++ test/mave_cli/cli_test.exs | 320 ++++++++++++++++++- test/mave_cli/vimeo_auth_test.exs | 304 ++++++++++++++++++ test/mave_cli/vimeo_import_test.exs | 393 ++++++++++++++++++++++++ test/mave_cli/vimeo_rate_limit_test.exs | 134 ++++++++ test/mave_cli/vimeo_test.exs | 168 ++++++++++ 18 files changed, 2572 insertions(+), 29 deletions(-) create mode 100644 lib/mave_cli/vimeo.ex create mode 100644 lib/mave_cli/vimeo/auth.ex create mode 100644 lib/mave_cli/vimeo/catalog.ex create mode 100644 lib/mave_cli/vimeo/import.ex create mode 100644 lib/mave_cli/vimeo/rate_limit.ex create mode 100644 lib/mave_cli/vimeo/state.ex create mode 100644 test/mave_cli/vimeo_auth_test.exs create mode 100644 test/mave_cli/vimeo_import_test.exs create mode 100644 test/mave_cli/vimeo_rate_limit_test.exs create mode 100644 test/mave_cli/vimeo_test.exs diff --git a/README.md b/README.md index 7e071e5..b7fe9cb 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,9 @@ A command-line client for [Mave](https://www.mave.io/). Manage videos and collections, upload files, and use JSON output in your scripts. +Import videos from Vimeo with their titles and folder structure using +[`mave import vimeo`](docs/usage.md#import-from-vimeo). + ## Installation Install with Homebrew: @@ -26,7 +29,7 @@ For running or building from source, see the [development guide](docs/developmen ## Get started ```sh -mave auth login +mave login mave videos list --format table mave videos upload ./video.mp4 --wait mave collections create --name "Product videos" diff --git a/docs/development.md b/docs/development.md index c9ca9d1..8715f01 100644 --- a/docs/development.md +++ b/docs/development.md @@ -42,16 +42,31 @@ mix deps.get MIX_ENV=test mix run -e 'System.halt(MaveCli.CLI.run(System.argv()))' -- --help ``` -For the `mave` examples in the README, define this helper in your current shell -from the repository directory: +To run the checkout from the repository directory, define `mave-local` in your +current shell. The installed `mave` executable continues to run its released +version; local source changes appear only through this helper until rebuilt. + +Bash or Zsh: ```sh -mave() { +mave-local() { MIX_ENV=test mix run -e 'System.halt(MaveCli.CLI.run(System.argv()))' -- "$@" } -mave auth login +mave-local --help ``` +Fish: + +```fish +function mave-local + env MIX_ENV=test mix run -e 'System.halt(MaveCli.CLI.run(System.argv()))' -- $argv +end +mave-local --help +``` + +Use `mave-local` instead of `mave` in the usage examples, for example +`mave-local import vimeo --help`. + Replace `--help` with a CLI command and its options, including [custom endpoints](usage.md#custom-endpoints) where needed. The test configuration starts dependencies without automatically invoking the CLI. This command diff --git a/docs/usage.md b/docs/usage.md index 5f7f756..8c960b2 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -5,7 +5,7 @@ ## Everyday commands ```sh -mave auth login +mave login mave videos list --format table mave videos get VIDEO_ID mave videos upload https://example.com/video.mp4 --wait @@ -14,12 +14,30 @@ mave collections create --name "Product videos" mave spaces create --domain video.example.com ``` -`mave auth login` opens a short-lived authorization page in your browser. After you approve the CLI for a space, the resulting API token is stored locally. If the connected Mave server does not support browser authorization yet, or the authorization endpoint cannot be reached, the CLI gracefully falls back to a hidden interactive API-token prompt. You can choose that prompt directly with `mave auth login --no-browser`; positional tokens and `--token` remain supported for automation and staged rollouts. +`mave login` (also available as `mave auth login`) opens a short-lived +authorization page in your browser. After you approve the CLI for a space, the +resulting API token is stored locally. If the connected Mave server does not +support browser authorization yet, or the authorization endpoint cannot be +reached, the CLI falls back to a hidden interactive API-token prompt. You can +choose that prompt directly with `mave login --no-browser`; positional tokens +and `--token` remain supported for automation and staged rollouts. + +If a login is already saved for the selected server, either login command reports +that you are already logged in and exits without opening a browser or creating +another API key. Run `mave logout` first to replace the login or recover +from a revoked key. This also applies when supplying a token manually. When +`MAVE_TOKEN` is set, login uses that existing authentication instead; unset the +variable before changing to a saved or browser login. A login saved for a +different server does not prevent logging in to the selected server. + +`mave logout` removes the locally saved login. The longer `mave auth login` and +`mave auth logout` commands remain available as aliases; use `mave auth status` +to inspect the current authentication source. Browser login also sends the computer name (the macOS computer name, or the hostname) so the connection can be recognized under API Keys in settings. If no usable name is available, it is omitted. Existing keys keep their original -metadata; log in again to create a connection with the computer name. +metadata; log out and log in again to create a connection with the computer name. `videos upload` accepts either a URL or a local file. For URL input, the CLI first downloads the source locally and then uploads it directly to Mave using the resumable TUS protocol: @@ -59,10 +77,133 @@ Bearer authentication is the default. Mave also documents Basic authentication; mave videos list --basic ``` -Token precedence is `--token`, followed by `MAVE_TOKEN`, followed by the locally stored token. On Unix, `auth login` writes the token with `0600` permissions to `${MAVE_CONFIG_HOME:-${XDG_CONFIG_HOME:-~/.config}}/mave/config.json`. For stricter secret storage, prefer your CI platform's environment-variable mechanism. +Token precedence is `--token`, followed by `MAVE_TOKEN`, followed by the locally stored token. On Unix, `mave login` writes the token with `0600` permissions to `${MAVE_CONFIG_HOME:-${XDG_CONFIG_HOME:-~/.config}}/mave/config.json`. For stricter secret storage, prefer your CI platform's environment-variable mechanism. Run `mave --help` to see all commands. JSON is the default output format; add `--format table` for human-readable tables. +## Import from Vimeo + +Import videos into the selected Mave space with their **titles and folder +structure**, including subfolders and empty folders. The CLI sends video file +links to Mave's existing import API; Mave fetches and processes the videos. + +Run `mave import` to list available importers, or `mave import vimeo --help` +for Vimeo setup and options. Importing starts with `mave import vimeo`. + +```sh +mave login +mave import vimeo --dry-run --format table +mave import vimeo +``` + +When no Vimeo token is saved and `VIMEO_ACCESS_TOKEN` is not set, the importer shows a link to +[your Vimeo apps](https://developer.vimeo.com/apps) and asks for a token using +hidden input. Select an existing app or create one, then choose +**Authentication → Generate Access Token → Authenticated (you)**. Enable +**Public**, **Private**, and **Video Files**, select **Generate**, and paste the +token into the CLI. The CLI validates it with Vimeo and saves it for subsequent +imports. Signing in to Vimeo in your browser does not authenticate +the CLI. See [Vimeo's token setup guide](https://help.vimeo.com/hc/en-us/articles/12427789081745-How-to-generate-a-personal-access-token). + +For automation, set `VIMEO_ACCESS_TOKEN` instead. The token needs the `public`, `private`, and +`video_files` scopes, plus a Vimeo plan that allows access to video file links. +See [Vimeo's download-link documentation](https://help.vimeo.com/hc/en-us/articles/12427806914577-About-video-file-download-links-from-the-API). +The Vimeo token is used only for Vimeo API requests. Tokens entered at the prompt +are saved atomically with `0600` permissions on Unix, alongside the Mave login in +`${MAVE_CONFIG_HOME:-${XDG_CONFIG_HOME:-~/.config}}/mave/vimeo.json`. +`VIMEO_ACCESS_TOKEN` takes precedence over the saved token and is never persisted. + +Manage this login through the importer: + +```sh +mave import vimeo login # Optional: save a token before your first import +mave import vimeo status # Show whether the token comes from the environment or a file +mave import vimeo logout # Remove the saved Vimeo token +``` + +These commands do not require a Mave login. Repeated Vimeo login keeps the existing +token; log out first to replace it. Logout removes only the local Vimeo token, +preserving your Mave login and import progress. It does not revoke the token at +Vimeo. If `VIMEO_ACCESS_TOKEN` is set, logout explains that you also need to unset +that variable. Plain `mave logout` continues to remove only the Mave login. +A first import with `--dry-run` can save the login, but never writes import progress +or creates videos or collections. + +```sh +# Import a folder and its descendants, including the selected folder itself +mave import vimeo --folder 12345 + +# Put the imported hierarchy inside an existing Mave collection +mave import vimeo --collection MAVE_COLLECTION_ID + +# Preview without creating collections, submitting videos, or saving state +mave import vimeo --dry-run --format table + +# Resume with the same source selection and destination +mave import vimeo --resume + +# Wait until each video is playable, with a timeout per video +mave import vimeo --wait --timeout 1800 +``` + +Folders become Mave collections. Videos without a folder stay at the destination +root (or in `--collection`). Folder identity comes from Vimeo IDs, so folders +with the same name remain separate. Selecting `--folder` imports its subtree +without importing its ancestors. Existing Mave collections are not merged by +name. Only video titles and folder names are copied as metadata. + +The importer fetches a fresh file link immediately before submitting each video. +It prefers an available source file, otherwise the highest-resolution available +video file. Originals are not guaranteed to be available. Vimeo read requests +retry temporary failures. On HTTP 429, the CLI pauses until Vimeo's `Retry-After` +or `X-RateLimit-Reset` allows another request, using the later value if both are +present. Without a usable reset header it waits 60 seconds. The wait is reported +on stderr, including during login, and does not prompt for the token again while +retrying. See [Vimeo's rate-limit guidance](https://help.vimeo.com/hc/en-us/articles/12427783954065-Rate-limits). + +There are at most three retries per request. If a cooldown exceeds five minutes +or the retries are exhausted, the command stops and reports when to try again. +An import stops requesting further videos and preserves progress; repeat the +same command with `--resume` after the cooldown. During first login, the token is +saved only after Vimeo validates it; if validation still fails, the error explains +that it has not been saved yet. Mave create requests are not retried +automatically when their outcome is uncertain. + +By default the command finishes after Mave has accepted the imports; `submitted` +does not mean playback is ready. Use `--wait` to verify playback, or +`mave import vimeo --resume --wait` to check previously submitted videos. +Unavailable videos are reported as `failed` while the remaining videos continue. +The command exits with status `1` if any video failed. Progress goes to stderr, +with results on stdout as JSON or `--format table`. + +### Resume state + +The CLI saves Vimeo-to-Mave IDs under the same configuration directory as your +Mave login, in `mave/imports/vimeo-.json`. The result includes `state_file`. +Use `--state-file FILE` to choose a path. The state is bound to the Vimeo account, +Mave server and space, selected Vimeo folder, and destination collection. Repeat +the same options with `--resume`; saved IDs are reused. A different state file +starts a separate import. Run only one importer against a given state file at a +time, and keep the state file to avoid importing the same library twice. + +State is written atomically with private file permissions and contains no access +tokens or signed file links. An existing state file is never reset automatically. +Videos that could not supply a file link are retried on resume. Videos already +accepted by Mave retain their IDs even if waiting for playback failed. + +If a process stops during a create request, the state can contain a `pending` +item. The server might have created it without the CLI receiving the ID, so +resuming stops instead of risking a duplicate. Check the destination in Mave: + +1. If the item exists, add its Mave ID to the state's `folders` or `videos` + mapping under the `source_id` shown in `pending`. +2. If the item definitely was not created, leave that mapping unchanged. +3. Set `pending` to `null`, save the file, and run the same command with `--resume`. + +For example, a pending video with `source_id` `12345` that exists in Mave is +reconciled by adding `"12345": "MAVE_VIDEO_ID"` to `videos` and clearing `pending`. +Do not clear it until you have checked whether the request succeeded. + ## Custom endpoints Use the API, upload socket, TUS, and CDN addresses configured by your server @@ -78,7 +219,7 @@ available for local development. Stored tokens are bound to the API base URL used at login. Switching servers does not reuse the saved token. Older unbound logins must be saved again with -`mave auth login` against the intended server. Use a separate `MAVE_CONFIG_HOME` +`mave login` against the intended server. Use a separate `MAVE_CONFIG_HOME` for each environment to keep multiple logins. An explicit `--token` or `MAVE_TOKEN` still applies to the server selected for that command. diff --git a/lib/mave_cli/cli.ex b/lib/mave_cli/cli.ex index 1c3695a..8b22851 100644 --- a/lib/mave_cli/cli.ex +++ b/lib/mave_cli/cli.ex @@ -11,6 +11,7 @@ defmodule MaveCli.CLI do Sources, Upload, UploadToken, + Vimeo, Webhook } @@ -27,6 +28,10 @@ defmodule MaveCli.CLI do per_page: :integer, uploaded: :boolean, collection: :string, + folder: :string, + dry_run: :boolean, + resume: :boolean, + state_file: :string, show_collections: :boolean, input_url: :string, name: :string, @@ -63,7 +68,7 @@ defmodule MaveCli.CLI do 0 opts[:help] == true or positional == [] -> - IO.puts(help()) + IO.puts(help(positional)) 0 true -> @@ -71,16 +76,58 @@ defmodule MaveCli.CLI do end end - defp execute(["auth", "login" | rest], opts) do - with {:ok, token} <- login_token(rest, opts), - :ok <- Config.save_token(token, base_url: opts[:base_url]) do - IO.puts("Mave token saved to #{Config.path()}") - 0 - else - {:error, reason} -> fail(reason) + defp execute([action | rest], opts) when action in ["login", "logout"], + do: execute(["auth", action | rest], opts) + + defp execute(["import"], _opts) do + IO.puts(help(["import"])) + 0 + end + + defp execute(["import", provider | _rest], _opts) when provider != "vimeo", + do: + fail("unknown importer #{inspect(provider)}; use `mave import` to see available importers") + + defp execute(["import", "vimeo", action], _opts) when action in ["login", "logout", "status"] do + result = + case action do + "login" -> Vimeo.Auth.login() + "logout" -> Vimeo.Auth.logout() + "status" -> Vimeo.Auth.status() + end + + case result do + {:ok, message} -> IO.puts(message) && 0 + {:error, message} -> fail(message) end end + defp execute(["import", "vimeo", _action | _rest], _opts), + do: fail("usage: mave import vimeo [login | logout | status]; use --help for import options") + + defp execute(["auth", "login" | rest], opts) when length(rest) <= 1 do + case Config.source(nil, base_url: opts[:base_url]) do + :config -> + IO.puts( + "Already logged in to this Mave server. Run `mave logout` before logging in again." + ) + + 0 + + :environment -> + IO.puts( + "Already authenticated via MAVE_TOKEN. Unset MAVE_TOKEN before logging in with another account." + ) + + 0 + + _ -> + login(rest, opts) + end + end + + defp execute(["auth", "login" | _rest], _opts), do: fail("usage: mave login [TOKEN]") + defp execute(["auth", "logout"], _opts) do case Config.delete_token() do :ok -> IO.puts("Saved Mave token removed.") && 0 @@ -139,9 +186,18 @@ defmodule MaveCli.CLI do :ok <- Output.print(result, opts[:format] || "json") do 0 else - {:error, {:api, status, message}} -> fail("Mave API (HTTP #{status}): #{message}") - {:error, {:transport, message}} -> fail("network error: #{message}") - {:error, message} when is_binary(message) -> fail(message) + {:error, {:import, result}} -> + Output.print(result, opts[:format] || "json") + 1 + + {:error, {:api, status, message}} -> + fail("Mave API (HTTP #{status}): #{message}") + + {:error, {:transport, message}} -> + fail("network error: #{message}") + + {:error, message} when is_binary(message) -> + fail(message) end end @@ -157,6 +213,9 @@ defmodule MaveCli.CLI do Client.list_videos(client, query) end + defp dispatch(["import", "vimeo"], opts, client), + do: Vimeo.Import.run(client, opts) + defp dispatch(["videos", "get", id], _opts, client), do: Client.get_video(client, id) defp dispatch(["videos", "create"], opts, client), @@ -328,7 +387,17 @@ defmodule MaveCli.CLI do defp auth_status(:different_server), do: fail("stored token belongs to a different server") defp auth_status(:missing), - do: fail("no token found; use `mave auth login` or MAVE_TOKEN") + do: fail("no token found; use `mave login` or MAVE_TOKEN") + + defp login(rest, opts) do + with {:ok, token} <- login_token(rest, opts), + :ok <- Config.save_token(token, base_url: opts[:base_url]) do + IO.puts("Mave token saved to #{Config.path()}") + 0 + else + {:error, reason} -> fail(reason) + end + end defp login_token([token], _opts), do: validate_token(token) @@ -355,7 +424,7 @@ defmodule MaveCli.CLI do end end - defp login_token(_, _opts), do: {:error, "usage: mave auth login [TOKEN]"} + defp login_token(_, _opts), do: {:error, "usage: mave login [TOKEN]"} defp validate_token(token) do case String.trim(token) do @@ -461,13 +530,79 @@ defmodule MaveCli.CLI do 1 end - defp help do + defp help(["import"]) do + """ + Import videos into Mave from another platform. + + Usage: + mave import PROVIDER [OPTIONS] + + Available importers: + vimeo Import videos with their titles and folder structure + + Examples: + mave import vimeo --dry-run --format table + mave import vimeo --folder 12345 + + Run `mave import vimeo --help` for setup and all import options. + """ + end + + defp help(["import", "vimeo" | _rest]) do + """ + Import Vimeo videos with their titles and folder structure, including subfolders. + + Usage: + mave import vimeo [OPTIONS] + + Setup: + Run `mave login` to select your Mave space. + Paste a token on your first import; it is saved locally for future imports. + VIMEO_ACCESS_TOKEN overrides the saved Vimeo token without being stored. + Create a token at https://developer.vimeo.com/apps with + Public, Private and Video Files access (Authenticated (you)). + + Vimeo login: + mave import vimeo login save a token before importing (optional) + mave import vimeo status show the current Vimeo authentication source + mave import vimeo logout remove the saved Vimeo token + + Import options: + --folder ID import this Vimeo folder and its descendants (default: all) + --collection ID place imports inside an existing Mave collection + --dry-run preview titles and folders without making changes + --resume continue the same import using saved Mave IDs + --state-file FILE choose where to save import progress + --wait wait until each video is playable + --timeout SECONDS maximum playback wait per video (default: 600) + --no-progress hide progress messages + --format FORMAT json (default) or table + --base-url URL alternative Mave API base URL + --token TOKEN override MAVE_TOKEN/stored Mave token + + Examples: + mave import vimeo --dry-run --format table + mave import vimeo --folder 12345 --wait + mave import vimeo --folder 12345 --resume --wait + + Without --wait, success means Mave accepted the import for processing. + Vimeo rate limits pause requests automatically; long cooldowns stop with retry guidance. + """ + end + + defp help(_command) do """ mave #{MaveCli.version()} — command-line client for mave.io Usage: + mave login [TOKEN] [--no-browser] + mave logout mave auth login [TOKEN] [--no-browser] mave auth status | logout + mave import list available importers + mave import vimeo [--folder ID] [--collection ID] [--dry-run] [--resume] + [--state-file FILE] [--wait] [--timeout SECONDS] + mave import vimeo login | status | logout mave videos list [--page N] [--per-page N] [--collection ID] mave videos get ID mave videos upload FILE_OR_URL [--collection ID] [--wait] @@ -501,6 +636,8 @@ defmodule MaveCli.CLI do --timeout SECONDS maximum wait time (default: 600) -h, --help show this help -v, --version show the version + + Run `mave import vimeo --help` for Vimeo setup and import options. """ end end diff --git a/lib/mave_cli/client.ex b/lib/mave_cli/client.ex index dc4d889..b82ebce 100644 --- a/lib/mave_cli/client.ex +++ b/lib/mave_cli/client.ex @@ -28,6 +28,11 @@ defmodule MaveCli.Client do def upload_key(%__MODULE__{upload_key: token}), do: token + def base_url(client), do: Req.Request.get_option(client.request, :base_url) + + def without_retries(client), + do: %{client | request: Req.merge(client.request, retry: false)} + def list_videos(client, query), do: request(client, :get, "videos", query: query) def get_video(client, id), do: request(client, :get, "videos/#{segment(id)}") def create_video(client, body), do: request(client, :post, "videos", body: body) diff --git a/lib/mave_cli/config.ex b/lib/mave_cli/config.ex index 7994179..dfa3590 100644 --- a/lib/mave_cli/config.ex +++ b/lib/mave_cli/config.ex @@ -77,11 +77,10 @@ defmodule MaveCli.Config do {:ok, data["token"]} :missing -> - {:error, "no token found; use `mave auth login` or MAVE_TOKEN"} + {:error, "no token found; use `mave login` or MAVE_TOKEN"} :unbound -> - {:error, - "stored token is not bound to a server; run `mave auth login` again for this server"} + {:error, "stored token is not bound to a server; run `mave login` again for this server"} :different_server -> {:error, diff --git a/lib/mave_cli/output.ex b/lib/mave_cli/output.ex index 10ca099..7ba4e5f 100644 --- a/lib/mave_cli/output.ex +++ b/lib/mave_cli/output.ex @@ -26,6 +26,10 @@ defmodule MaveCli.Output do do: {:error, "unknown output format: #{format} (use json or table)"} defp columns_for([%{"object" => "collection"} | _]), do: @collection_columns + + defp columns_for([%{"object" => "vimeo_import_item"} | _]), + do: ["type", "vimeo_id", "name", "folder", "mave_id", "status", "error"] + defp columns_for([%{"object" => "space"} | _]), do: @space_columns defp columns_for([row]) when is_map(row), do: row |> Map.keys() |> Enum.sort() defp columns_for(_), do: @video_columns diff --git a/lib/mave_cli/vimeo.ex b/lib/mave_cli/vimeo.ex new file mode 100644 index 0000000..32411ac --- /dev/null +++ b/lib/mave_cli/vimeo.ex @@ -0,0 +1,155 @@ +defmodule MaveCli.Vimeo do + @moduledoc false + + alias MaveCli.Vimeo.RateLimit + + @base_url "https://api.vimeo.com" + + def new(token) do + Req.new( + base_url: @base_url, + headers: [ + {"authorization", "Bearer #{token}"}, + {"accept", "application/vnd.vimeo.*+json;version=3.4"}, + {"user-agent", "mave-cli/#{MaveCli.version()}"} + ], + redirect: false, + compressed: true, + decode_body: false, + retry: :safe_transient, + retry_log_level: false, + max_retries: 3, + receive_timeout: 30_000 + ) + |> Req.Request.prepend_response_steps(vimeo_rate_limit: &RateLimit.handle/1) + end + + defdelegate token(), to: MaveCli.Vimeo.Auth + + def get(client, path, query \\ []) do + with {:ok, url} <- api_url(path) do + case Req.get(client, url: url, params: query) do + {:ok, %{status: status, body: body}} when status in 200..299 -> + decode_response(body, status) + + {:ok, %Req.Response{status: 429} = response} -> + {:error, RateLimit.from_response(response)} + + {:ok, %{status: status}} -> + {:error, api_error(status)} + + {:error, _reason} -> + {:error, "could not reach the Vimeo API"} + end + end + end + + # Vimeo endpoints return different media types; decode their JSON explicitly. + defp decode_response(body, status) when is_binary(body) do + case Jason.decode(body) do + {:ok, decoded} when is_map(decoded) -> {:ok, decoded} + _ -> invalid_response(status) + end + end + + defp decode_response(body, _status) when is_map(body), do: {:ok, body} + defp decode_response(_body, status), do: invalid_response(status) + + defp invalid_response(status), + do: {:error, "Vimeo returned an invalid JSON object (HTTP #{status})"} + + def all(client, path, query \\ []) do + list_pages(client, path, Keyword.put(query, :per_page, 100), MapSet.new(), []) + end + + def id(uri) when is_binary(uri) do + case Regex.run(~r{^/(?:videos|(?:users/\d+|me)/projects)/(\d+)(?:/[^/]+)?$}, uri) do + [_, id] -> {:ok, id} + _ -> {:error, "Vimeo returned an invalid video or folder identifier"} + end + end + + def id(_), do: {:error, "Vimeo returned a missing video or folder identifier"} + + def source(client, video_id) do + # Keep a required field in the response even when file access is unavailable. + with {:ok, video} <- get(client, "/videos/#{video_id}", fields: "uri,files,download") do + files = List.wrap(video["download"]) ++ List.wrap(video["files"]) + + select_file(files) + end + end + + defp select_file(files) do + case files |> Enum.filter(&video_file?/1) |> Enum.max_by(&quality/1, fn -> nil end) do + nil -> {:error, "no accessible video file; check Vimeo's video_files scope and plan"} + file -> {:ok, file["link"]} + end + end + + defp list_pages(client, path, query, visited, pages) do + with {:ok, url} <- api_url(path), + false <- MapSet.member?(visited, url), + {:ok, %{"data" => data, "paging" => paging}} when is_list(data) and is_map(paging) <- + get(client, url, query) do + pages = [data | pages] + + case paging["next"] do + nil -> {:ok, pages |> Enum.reverse() |> List.flatten()} + next -> list_pages(client, next, [], MapSet.put(visited, url), pages) + end + else + {:error, _} = error -> error + _ -> {:error, "Vimeo returned invalid or repeating pagination"} + end + end + + # Never send the Vimeo bearer token to a pagination/connection URL on another origin. + defp api_url(path) when is_binary(path) do + url = URI.merge(@base_url, path) + + if url.scheme == "https" and url.host == "api.vimeo.com" and url.port == 443 and + is_nil(url.userinfo) and is_nil(url.fragment) do + {:ok, URI.to_string(url)} + else + {:error, "Vimeo returned a URL outside its API"} + end + end + + defp api_url(_), do: {:error, "Vimeo returned an invalid API URL"} + + defp video_file?(%{"link" => link} = file) when is_binary(link) do + uri = URI.parse(link) + type = file["type"] || "" + + uri.scheme in ["http", "https"] and is_binary(uri.host) and is_nil(uri.userinfo) and + (String.starts_with?(type, "video/") or file["quality"] == "source") and + file["quality"] != "hls" and not expired?(file["expires"]) + end + + defp video_file?(_), do: false + + defp quality(file) do + {file["quality"] == "source", number(file["width"]) * number(file["height"]), + number(file["size"])} + end + + defp number(value) when is_number(value), do: value + defp number(_), do: 0 + + defp expired?(value) when is_binary(value) do + case DateTime.from_iso8601(value) do + {:ok, expires, _} -> DateTime.compare(expires, DateTime.utc_now()) != :gt + _ -> false + end + end + + defp expired?(_), do: false + + defp api_error(401), do: "Vimeo rejected the access token" + + defp api_error(403), + do: "Vimeo denied access; check the public, private and video_files token scopes" + + defp api_error(status), do: "Vimeo API returned HTTP #{status}" +end diff --git a/lib/mave_cli/vimeo/auth.ex b/lib/mave_cli/vimeo/auth.ex new file mode 100644 index 0000000..4971c39 --- /dev/null +++ b/lib/mave_cli/vimeo/auth.ex @@ -0,0 +1,165 @@ +defmodule MaveCli.Vimeo.Auth do + @moduledoc false + + alias MaveCli.{Config, PrivateFile, Terminal, Vimeo} + alias MaveCli.Vimeo.RateLimit + + def token do + case credentials() do + {:ok, token, _source} -> {:ok, token} + :missing -> authorize() + error -> error + end + end + + def login do + case credentials() do + {:ok, _token, :config} -> + {:ok, + "Already logged in to Vimeo. Run `mave import vimeo logout` before logging in again."} + + {:ok, _token, :environment} -> + {:ok, + "Already authenticated via VIMEO_ACCESS_TOKEN. Unset VIMEO_ACCESS_TOKEN before saving another login."} + + :missing -> + case authorize() do + {:ok, _token} -> {:ok, "Logged in to Vimeo."} + error -> error + end + + error -> + error + end + end + + # The path is fixed under the operator-selected configuration directory. + # sobelow_skip ["Traversal.FileModule"] + def logout do + case File.rm(path()) do + result when result in [:ok, {:error, :enoent}] -> + suffix = + if environment_token(), + do: " VIMEO_ACCESS_TOKEN is still set; unset it to stop using that token.", + else: "" + + {:ok, "Saved Vimeo token removed." <> suffix} + + {:error, _reason} -> + {:error, "could not remove the saved Vimeo token"} + end + end + + def status do + case credentials() do + {:ok, _token, :environment} -> + {:ok, "Logged in to Vimeo via VIMEO_ACCESS_TOKEN."} + + {:ok, _token, :config} -> + {:ok, "Logged in to Vimeo via #{path()}."} + + :missing -> + {:error, "no Vimeo token found; use `mave import vimeo login` or VIMEO_ACCESS_TOKEN"} + + error -> + error + end + end + + def path, do: Path.join(Path.dirname(Config.path()), "vimeo.json") + + defp credentials do + case environment_token() do + nil -> stored_credentials() + token -> {:ok, token, :environment} + end + end + + defp environment_token, do: normalize_token(System.get_env("VIMEO_ACCESS_TOKEN")) + + # Read only the fixed Vimeo credential file; remote data never selects a path. + # sobelow_skip ["Traversal.FileModule"] + defp stored_credentials do + case File.read(path()) do + {:ok, content} -> decode_credentials(content) + {:error, :enoent} -> :missing + {:error, _reason} -> {:error, "could not read the saved Vimeo token"} + end + end + + defp decode_credentials(content) do + with {:ok, %{"token" => value}} <- Jason.decode(content), + token when is_binary(token) <- normalize_token(value) do + {:ok, token, :config} + else + _ -> {:error, "invalid saved Vimeo login; run `mave import vimeo logout` and log in again"} + end + end + + defp normalize_token(value) when is_binary(value) do + case String.trim(value) do + "" -> nil + token -> token + end + end + + defp normalize_token(_value), do: nil + + defp authorize do + with {:ok, token} <- prompt_for_token(), + :ok <- validate(token), + :ok <- save(token) do + IO.puts(:stderr, "Vimeo token saved to #{path()}") + {:ok, token} + end + end + + defp validate(token) do + case Vimeo.get(Vimeo.new(token), "/me", fields: "uri") do + {:ok, %{"uri" => "/users/" <> id}} when id != "" -> + :ok + + {:ok, _body} -> + {:error, "could not identify the Vimeo account; token was not saved"} + + {:error, %RateLimit{} = limit} -> + {:error, + RateLimit.message(limit) <> + ". The token has not been saved; retry login after the cooldown"} + + error -> + error + end + end + + defp save(token) do + case PrivateFile.write_atomic(path(), Jason.encode!(%{"token" => token}, pretty: true)) do + :ok -> :ok + {:error, _reason} -> {:error, "could not save the Vimeo token to #{path()}"} + end + end + + defp prompt_for_token do + IO.puts(:stderr, """ + To import your videos, create a Vimeo access token: + https://developer.vimeo.com/apps + + 1. Select an app, or create one (for example, "Mave import"). + 2. Under Authentication, select Generate Access Token, then Authenticated (you). + 3. Enable Public, Private and Video Files, then select Generate. + + Paste the generated token below (input is hidden). It will be saved for future imports. + """) + + case Terminal.read_secret("Vimeo access token: ") do + value when is_binary(value) or is_list(value) -> + case value |> to_string() |> normalize_token() do + nil -> {:error, "Vimeo token must not be empty; set VIMEO_ACCESS_TOKEN"} + token -> {:ok, token} + end + + _ -> + {:error, "could not read Vimeo token; set VIMEO_ACCESS_TOKEN"} + end + end +end diff --git a/lib/mave_cli/vimeo/catalog.ex b/lib/mave_cli/vimeo/catalog.ex new file mode 100644 index 0000000..59fde6f --- /dev/null +++ b/lib/mave_cli/vimeo/catalog.ex @@ -0,0 +1,158 @@ +defmodule MaveCli.Vimeo.Catalog do + @moduledoc false + + alias MaveCli.Vimeo + + def load(client, folder_id \\ nil) do + with {:ok, roots} <- roots(client, folder_id), + {:ok, folders} <- walk(client, roots, nil, %{}, []), + {:ok, folders} <- order(folders), + {:ok, videos} <- folder_videos(client, folders), + {:ok, videos} <- unfiled_videos(client, videos, folder_id) do + {:ok, %{folders: folders, videos: videos |> Map.values() |> Enum.sort_by(& &1.id)}} + end + end + + defp roots(client, nil), do: Vimeo.all(client, "/me/projects") + + defp roots(client, id) do + if Regex.match?(~r/^\d+$/, id) do + with {:ok, folder} <- Vimeo.get(client, "/me/projects/#{id}"), do: {:ok, [folder]} + else + {:error, "--folder must be a numeric Vimeo folder ID"} + end + end + + defp walk(client, folders, parent, found, ancestors) do + Enum.reduce_while(folders, {:ok, found}, fn folder, {:ok, found} -> + case visit(client, folder, parent, found, ancestors) do + {:ok, found} -> {:cont, {:ok, found}} + error -> {:halt, error} + end + end) + end + + defp visit(client, folder, parent, found, ancestors) do + with {:ok, item} <- item(folder, parent), + :ok <- valid_ancestry(item.id, ancestors), + {:ok, item} <- merge_parent(item, found[item.id]) do + visit_children(client, folder, item, found, ancestors) + end + end + + defp visit_children(client, folder, item, found, ancestors) do + if Map.has_key?(found, item.id) do + {:ok, Map.put(found, item.id, item)} + else + with {:ok, children} <- children(client, folder) do + walk(client, children, item.id, Map.put(found, item.id, item), [item.id | ancestors]) + end + end + end + + defp children(client, folder) do + connections = get_in(folder, ["metadata", "connections"]) || %{} + connection = connections["folders"] || connections["items"] + + case connection do + %{"total" => 0} -> + {:ok, []} + + %{"uri" => uri} -> + with {:ok, items} <- Vimeo.all(client, uri, filter: "folder") do + {:ok, Enum.flat_map(items, &child_folder/1)} + end + + _ -> + {:ok, []} + end + end + + defp child_folder(%{"type" => "folder", "folder" => folder}), do: [folder] + defp child_folder(_), do: [] + + defp valid_ancestry(id, ancestors) do + if id in ancestors or length(ancestors) >= 100, + do: {:error, "Vimeo returned a cyclic or excessively deep folder structure"}, + else: :ok + end + + defp merge_parent(item, nil), do: {:ok, item} + defp merge_parent(%{parent_id: nil}, existing), do: {:ok, existing} + defp merge_parent(item, %{parent_id: nil}), do: {:ok, item} + defp merge_parent(%{parent_id: parent} = item, %{parent_id: parent}), do: {:ok, item} + defp merge_parent(_, _), do: {:error, "Vimeo returned a folder with conflicting parents"} + + defp order(folders) do + Enum.reduce_while(Map.values(folders), {:ok, []}, fn folder, {:ok, result} -> + case folder_path(folder, folders, []) do + {:ok, path} -> {:cont, {:ok, [Map.put(folder, :path, path) | result]}} + error -> {:halt, error} + end + end) + |> case do + {:ok, result} -> {:ok, Enum.sort_by(result, &{length(&1.path), &1.path, &1.id})} + error -> error + end + end + + defp folder_path(folder, folders, ancestors) do + with :ok <- valid_ancestry(folder.id, ancestors) do + parent_path(folder, folders, ancestors) + end + end + + defp parent_path(%{parent_id: nil} = folder, _folders, _ancestors), do: {:ok, [folder.name]} + + defp parent_path(folder, folders, ancestors) do + with {:ok, path} <- folder_path(folders[folder.parent_id], folders, [folder.id | ancestors]), + do: {:ok, path ++ [folder.name]} + end + + defp folder_videos(client, folders) do + Enum.reduce_while(folders, {:ok, %{}}, fn folder, {:ok, found} -> + with {:ok, videos} <- + Vimeo.all(client, folder.uri <> "/videos", + fields: "uri,name", + include_subfolders: false + ), + {:ok, found} <- merge_videos(videos, found, folder.id) do + {:cont, {:ok, found}} + else + error -> {:halt, error} + end + end) + end + + defp unfiled_videos(client, found, nil) do + with {:ok, videos} <- Vimeo.all(client, "/me/videos", fields: "uri,name"), + do: merge_videos(videos, found, nil) + end + + defp unfiled_videos(_client, found, _folder), do: {:ok, found} + + defp merge_videos(videos, found, parent) do + Enum.reduce_while(videos, {:ok, found}, fn video, {:ok, found} -> + case item(video, parent) do + {:ok, item} -> + item = preserve_parent(item, found[item.id]) + {:cont, {:ok, Map.put(found, item.id, item)}} + + error -> + {:halt, error} + end + end) + end + + defp preserve_parent(%{parent_id: nil} = item, %{parent_id: parent}), + do: %{item | parent_id: parent} + + defp preserve_parent(item, _existing), do: item + + defp item(%{"uri" => uri, "name" => name}, parent) when is_binary(name) and name != "" do + with {:ok, id} <- Vimeo.id(uri), + do: {:ok, %{id: id, uri: uri, name: name, parent_id: parent}} + end + + defp item(_, _), do: {:error, "Vimeo returned an item without an identifier or title"} +end diff --git a/lib/mave_cli/vimeo/import.ex b/lib/mave_cli/vimeo/import.ex new file mode 100644 index 0000000..9401e95 --- /dev/null +++ b/lib/mave_cli/vimeo/import.ex @@ -0,0 +1,237 @@ +defmodule MaveCli.Vimeo.Import do + @moduledoc false + + alias MaveCli.{Client, Progress, Vimeo} + alias MaveCli.Vimeo.{Catalog, RateLimit, State} + + def run(client, opts \\ []) do + with :ok <- validate_options(opts), + {:ok, token} <- Vimeo.token() do + run_with_client(client, Vimeo.new(token), opts) + end + end + + def run_with_client(client, vimeo, opts \\ []) do + report(opts, "Reading Vimeo videos and folders…") + + with :ok <- validate_options(opts), + {:ok, scope} <- scope(client, vimeo, opts), + {:ok, state} <- State.open(scope, opts), + {:ok, catalog} <- Catalog.load(vimeo, opts[:folder]) do + report(opts, "Import state: #{state.path}") + + if opts[:dry_run] do + {:ok, preview(catalog, state)} + else + execute(Client.without_retries(client), vimeo, catalog, state, opts) + end + else + {:error, %RateLimit{} = limit} -> {:error, RateLimit.message(limit)} + error -> error + end + end + + defp validate_options(opts) do + cond do + opts[:format] not in [nil, "json", "table"] -> + {:error, "--format must be json or table"} + + opts[:timeout] && opts[:timeout] <= 0 -> + {:error, "--timeout must be greater than zero"} + + opts[:folder] && not Regex.match?(~r/^\d+$/, opts[:folder]) -> + {:error, "--folder must be a numeric Vimeo folder ID"} + + true -> + :ok + end + end + + defp scope(client, vimeo, opts) do + with {:ok, %{"uri" => account}} when is_binary(account) <- + Vimeo.get(vimeo, "/me", fields: "uri"), + {:ok, %{"space_id" => space}} when is_binary(space) <- + Client.list_videos(client, per_page: 1) do + {:ok, + %{ + "vimeo_account" => account, + "mave_server" => String.trim_trailing(Client.base_url(client), "/"), + "mave_space" => space, + "collection" => opts[:collection], + "folder" => opts[:folder] + }} + else + {:error, _} = error -> error + _ -> {:error, "could not identify the Vimeo account or Mave space"} + end + end + + defp preview(catalog, state) do + rows = + Enum.map( + catalog.folders, + &row(&1, "folder", catalog, State.lookup(state, "folders", &1.id), "planned") + ) ++ + Enum.map( + catalog.videos, + &row(&1, "video", catalog, State.lookup(state, "videos", &1.id), "planned") + ) + + result(rows, state, true) + end + + defp execute(client, vimeo, catalog, state, opts) do + with {:ok, state} <- State.save(state), + {:ok, state, folders} <- create_folders(client, catalog, state, opts), + {:ok, state, videos} <- create_videos(client, vimeo, catalog, state, opts) do + result = result(folders ++ videos, state, false) + if result["failed"] == 0, do: {:ok, result}, else: {:error, {:import, result}} + end + end + + defp create_folders(client, catalog, state, opts) do + Enum.reduce_while(catalog.folders, {:ok, state, []}, fn folder, {:ok, state, rows} -> + case create_folder(client, folder, state, opts) do + {:ok, state, id, status} -> + {:cont, {:ok, state, rows ++ [row(folder, "folder", catalog, id, status)]}} + + error -> + {:halt, error} + end + end) + end + + defp create_folder(client, folder, state, opts) do + case State.lookup(state, "folders", folder.id) do + nil -> + report(opts, "Creating folder #{inspect(folder.name)}") + body = body(folder, state, opts) + + with {:ok, state} <- State.begin_item(state, "folders", folder), + {:ok, id} <- created_id(Client.create_collection(client, body), state.path), + {:ok, state} <- State.finish_item(state, "folders", folder.id, id) do + {:ok, state, id, "created"} + end + + id -> + {:ok, state, id, "existing"} + end + end + + defp create_videos(client, vimeo, catalog, state, opts) do + Enum.reduce_while(catalog.videos, {:ok, state, []}, fn video, {:ok, state, rows} -> + case create_video(client, vimeo, video, state, opts) do + {:ok, state, id, status} -> + entry = row(video, "video", catalog, id, status) + {:cont, {:ok, state, rows ++ [entry]}} + + {:unavailable, message} -> + entry = row(video, "video", catalog, nil, "failed") |> Map.put("error", message) + report(opts, "Could not import #{inspect(video.name)}: #{message}") + {:cont, {:ok, state, rows ++ [entry]}} + + error -> + {:halt, error} + end + end) + end + + defp create_video(client, vimeo, video, state, opts) do + case State.lookup(state, "videos", video.id) do + nil -> submit_video(client, vimeo, video, state, opts) + id -> wait(client, state, id, "existing", opts) + end + end + + defp submit_video(client, vimeo, video, state, opts) do + report(opts, "Importing #{inspect(video.name)}") + + case Vimeo.source(vimeo, video.id) do + {:ok, url} -> + body = body(video, state, opts) |> Map.put("input_url", url) + + with {:ok, state} <- State.begin_item(state, "videos", video), + {:ok, id} <- created_id(Client.create_video(client, body), state.path), + {:ok, state} <- State.finish_item(state, "videos", video.id, id) do + wait(client, state, id, "submitted", opts) + end + + {:error, %RateLimit{} = limit} -> + {:error, + RateLimit.message(limit) <> + ". Import stopped; progress is saved. Rerun with --resume after the cooldown"} + + {:error, message} -> + {:unavailable, message} + end + end + + defp wait(client, state, id, status, opts) do + if opts[:wait] do + case Progress.wait_for_video(client, id, + timeout: opts[:timeout] || 600, + progress: opts[:progress] != false + ) do + {:ok, _} -> {:ok, state, id, "playable"} + {:error, _} -> {:ok, state, id, "failed"} + end + else + {:ok, state, id, status} + end + end + + defp body(item, state, opts) do + parent = + if item.parent_id, + do: State.lookup(state, "folders", item.parent_id), + else: opts[:collection] + + if parent, do: %{"name" => item.name, "collection" => parent}, else: %{"name" => item.name} + end + + defp created_id({:ok, %{"id" => id}}, _path) when is_binary(id) and id != "", do: {:ok, id} + + defp created_id(_, path) do + {:error, + "Mave did not confirm the create request; import stopped to avoid duplicates. " <> + "Check the destination and reconcile the pending item in #{path} before resuming"} + end + + defp row(item, kind, catalog, id, status) do + folder = Enum.find(catalog.folders, &(&1.id == item.parent_id)) + + %{ + "object" => "vimeo_import_item", + "type" => kind, + "vimeo_id" => item.id, + "name" => item.name, + "folder" => if(folder, do: Enum.join(folder.path, "/"), else: "/"), + "mave_id" => id, + "status" => if(id && status == "planned", do: "existing", else: status) + } + |> processing_error(status, id) + end + + defp processing_error(row, "failed", id) when is_binary(id), + do: + Map.put(row, "error", "Mave has not confirmed playback; resume with --wait to check again") + + defp processing_error(row, _status, _id), do: row + + defp result(rows, state, dry_run) do + %{ + "data" => rows, + "dry_run" => dry_run, + "state_file" => state.path, + "space_id" => state.data["scope"]["mave_space"], + "collection" => state.data["scope"]["collection"], + "folders" => Enum.count(rows, &(&1["type"] == "folder")), + "videos" => Enum.count(rows, &(&1["type"] == "video")), + "failed" => Enum.count(rows, &(&1["status"] == "failed")) + } + end + + defp report(opts, message) do + if opts[:progress] != false, do: IO.puts(:stderr, message) + end +end diff --git a/lib/mave_cli/vimeo/rate_limit.ex b/lib/mave_cli/vimeo/rate_limit.ex new file mode 100644 index 0000000..e6973a3 --- /dev/null +++ b/lib/mave_cli/vimeo/rate_limit.ex @@ -0,0 +1,113 @@ +defmodule MaveCli.Vimeo.RateLimit do + @moduledoc false + + defexception [:seconds, :retry_at, :reset_known?] + + @fallback_seconds 60 + @max_wait_seconds 300 + + def handle({request, %Req.Response{status: 429} = response}) do + limit = from_response(response) + retries = Req.Request.get_private(request, :req_retry_count, 0) + max_retries = Req.Request.get_option(request, :max_retries, 3) + + if request.options[:retry] != false and retries < max_retries and + limit.seconds <= @max_wait_seconds do + IO.puts(:stderr, waiting_message(limit, retries + 1, max_retries)) + {request, Req.Response.put_header(response, "retry-after", to_string(limit.seconds))} + else + Req.Request.halt(request, response) + end + end + + def handle(request_response), do: request_response + + def from_response(response, now \\ DateTime.utc_now()) do + server_now = datetime(header(response, "date")) || now + + seconds = + [ + retry_after(header(response, "retry-after"), server_now), + reset_after(header(response, "x-ratelimit-reset"), server_now) + ] + |> Enum.filter(&is_integer/1) + |> Enum.max(fn -> nil end) + + wait = seconds || @fallback_seconds + %__MODULE__{seconds: wait, retry_at: DateTime.add(now, wait), reset_known?: seconds != nil} + end + + @impl true + def message(%{reset_known?: true} = limit) do + "Vimeo rate limit reached (HTTP 429); try again after #{timestamp(limit.retry_at)} " <> + "(in about #{limit.seconds} seconds)" + end + + def message(_limit) do + "Vimeo rate limit reached (HTTP 429); no usable reset time was supplied. " <> + "Wait at least #{@fallback_seconds} seconds before trying again" + end + + defp waiting_message(limit, attempt, max_retries) do + until = if limit.reset_known?, do: " until #{timestamp(limit.retry_at)}", else: "" + + "Vimeo rate limit reached; waiting #{limit.seconds}s#{until}, " <> + "then retrying (#{attempt}/#{max_retries})." + end + + defp timestamp(time), do: Calendar.strftime(time, "%Y-%m-%d %H:%M:%S UTC") + + defp header(response, name), do: List.first(Req.Response.get_header(response, name)) + + defp retry_after(value, now) when is_binary(value) do + case Integer.parse(String.trim(value)) do + {seconds, ""} when seconds >= 0 -> max(seconds, 1) + _ -> seconds_until(datetime(value), now) + end + end + + defp retry_after(_value, _now), do: nil + + defp reset_after(value, now) when is_binary(value) do + time = + case Integer.parse(String.trim(value)) do + {seconds, ""} -> + case DateTime.from_unix(seconds) do + {:ok, time} -> time + _ -> nil + end + + _ -> + datetime(value) + end + + seconds_until(time, now) + end + + defp reset_after(_value, _now), do: nil + + defp seconds_until(nil, _now), do: nil + + defp seconds_until(time, now) do + case DateTime.diff(time, now, :millisecond) do + milliseconds when milliseconds >= 0 -> div(milliseconds, 1000) + 1 + _ -> nil + end + end + + defp datetime(value) when is_binary(value) do + case DateTime.from_iso8601(value) do + {:ok, time, _offset} -> time + _ -> http_datetime(value) + end + end + + defp datetime(_value), do: nil + + defp http_datetime(value) do + case Req.Utils.parse_http_date(value) do + {:ok, time} -> time + _ -> nil + end + end +end diff --git a/lib/mave_cli/vimeo/state.ex b/lib/mave_cli/vimeo/state.ex new file mode 100644 index 0000000..b3f190e --- /dev/null +++ b/lib/mave_cli/vimeo/state.ex @@ -0,0 +1,96 @@ +defmodule MaveCli.Vimeo.State do + @moduledoc false + + alias MaveCli.{Config, PrivateFile} + + def open(scope, opts) do + path = Path.expand(opts[:state_file] || default_path(scope)) + + with {:ok, data} <- load(path, scope, opts), + :ok <- reconciled(data, path) do + {:ok, %{path: path, data: data}} + end + end + + def save(state) do + case PrivateFile.write_atomic(state.path, Jason.encode!(state.data, pretty: true)) do + :ok -> + {:ok, state} + + {:error, _} -> + {:error, + "could not save import state to #{state.path}; stop and reconcile before resuming"} + end + end + + def lookup(state, kind, id), do: get_in(state.data, [kind, id]) + + def begin_item(state, kind, item) do + pending = %{"kind" => kind, "source_id" => item.id, "name" => item.name} + save(%{state | data: Map.put(state.data, "pending", pending)}) + end + + def finish_item(state, kind, source_id, mave_id) do + data = state.data |> put_in([kind, source_id], mave_id) |> Map.put("pending", nil) + save(%{state | data: data}) + end + + defp default_path(scope) do + suffix = + :crypto.hash(:sha256, :erlang.term_to_binary(Enum.sort(scope))) + |> Base.encode16(case: :lower) + + Path.join([Path.dirname(Config.path()), "imports", "vimeo-#{suffix}.json"]) + end + + # The path is the operator's state file or a hash under their config directory. + # Remote data never selects a filesystem path. + # sobelow_skip ["Traversal.FileModule"] + defp load(path, scope, opts) do + resume = opts[:resume] == true + dry_run = opts[:dry_run] == true + + case File.read(path) do + {:ok, contents} when resume -> decode(contents, scope) + {:ok, _} when dry_run -> {:ok, empty(scope)} + {:ok, _} -> {:error, "an import state already exists at #{path}; use --resume"} + {:error, :enoent} when not resume -> {:ok, empty(scope)} + {:error, :enoent} -> {:error, "no import state found at #{path}; start without --resume"} + {:error, _} -> {:error, "could not read import state at #{path}"} + end + end + + defp empty(scope), + do: %{"version" => 1, "scope" => scope, "folders" => %{}, "videos" => %{}, "pending" => nil} + + defp decode(contents, scope) do + with {:ok, + %{"version" => 1, "scope" => ^scope, "folders" => folders, "videos" => videos} = data} <- + Jason.decode(contents), + true <- valid_mapping?(folders) and valid_mapping?(videos) do + {:ok, data} + else + _ -> + {:error, + "import state is invalid or belongs to a different Vimeo account, Mave destination or folder selection"} + end + end + + defp valid_mapping?(mapping) when is_map(mapping) do + Enum.all?(mapping, fn {source, target} -> + is_binary(source) and is_binary(target) and target != "" + end) + end + + defp valid_mapping?(_), do: false + + defp reconciled(%{"pending" => nil}, _path), do: :ok + + defp reconciled(%{"pending" => %{"kind" => kind, "source_id" => id}}, path) do + {:error, + "the previous create request for #{kind} Vimeo ID #{id} has an unknown outcome; " <> + "check Mave and reconcile the pending item in #{path} before using --resume (see the import documentation)"} + end + + defp reconciled(_, _path), do: {:error, "invalid pending item in import state"} +end diff --git a/test/mave_cli/cli_test.exs b/test/mave_cli/cli_test.exs index 1cbfe7b..c52ba1a 100644 --- a/test/mave_cli/cli_test.exs +++ b/test/mave_cli/cli_test.exs @@ -2,6 +2,8 @@ defmodule MaveCli.CLITest do use ExUnit.Case, async: false import ExUnit.CaptureIO + alias MaveCli.Vimeo.Auth, as: VimeoAuth + setup {Req.Test, :verify_on_exit!} setup do @@ -60,6 +62,210 @@ defmodule MaveCli.CLITest do assert output =~ "mave spaces create" assert output =~ "mave webhooks verify" assert output =~ "mave sources manifest" + assert output =~ "mave import vimeo" + assert output =~ "mave login" + assert output =~ "mave logout" + end + + test "import discovery and help do not require credentials or contact either API" do + Req.Test.stub(__MODULE__, fn _conn -> flunk("help must not make API requests") end) + + for args <- [["import"], ["import", "--help"], ["import", "vimeo", "--help"]] do + prompt = + capture_io(:stderr, fn -> + output = capture_io(fn -> assert MaveCli.CLI.run(args) == 0 end) + assert output =~ "mave import vimeo" + assert output =~ "folder structure" + refute output =~ "mave videos delete" + end) + + assert prompt == "" + end + + output = capture_io(fn -> assert MaveCli.CLI.run(["import", "vimeo", "--help"]) == 0 end) + assert output =~ "VIMEO_ACCESS_TOKEN" + assert output =~ "--folder ID" + assert output =~ "--resume" + end + + test "unknown importers explain how to find supported providers before authentication" do + error = capture_io(:stderr, fn -> assert MaveCli.CLI.run(["import", "unknown"]) == 1 end) + assert error =~ "unknown importer" + assert error =~ "use `mave import`" + end + + describe "Vimeo imports" do + @describetag :tmp_dir + + setup %{tmp_dir: directory} do + previous = System.get_env("VIMEO_ACCESS_TOKEN") + previous_home = System.get_env("MAVE_CONFIG_HOME") + System.put_env("VIMEO_ACCESS_TOKEN", "synthetic-vimeo-token") + System.put_env("MAVE_CONFIG_HOME", directory) + + on_exit(fn -> + restore_env("VIMEO_ACCESS_TOKEN", previous) + restore_env("MAVE_CONFIG_HOME", previous_home) + end) + + Req.Test.stub(__MODULE__, fn conn -> + body = + case {conn.host, conn.request_path} do + {"api.vimeo.com", "/me"} -> + %{"uri" => "/users/42"} + + {"api.vimeo.com", "/me/projects"} -> + %{"data" => [], "paging" => %{"next" => nil}} + + {"api.vimeo.com", "/me/videos"} -> + %{ + "data" => [%{"uri" => "/videos/101", "name" => "Example title"}], + "paging" => %{"next" => nil} + } + + {"api.vimeo.com", "/videos/101"} -> + %{"files" => []} + + {"local.test", "/v1/videos"} -> + %{"space_id" => "space-1", "data" => []} + end + + assert conn.method == "GET" + Req.Test.json(conn, body) + end) + end + + test "guides users without a Vimeo token and keeps token input out of output" do + for token <- [nil, " "] do + restore_env("VIMEO_ACCESS_TOKEN", token) + assert {:ok, _message} = VimeoAuth.logout() + + prompt = + capture_io(:stderr, fn -> + output = + capture_io([input: " synthetic-pasted-token\n"], fn -> + assert MaveCli.Vimeo.token() == {:ok, "synthetic-pasted-token"} + end) + + assert output == "" + end) + + assert prompt =~ "https://developer.vimeo.com/apps" + assert prompt =~ "Authenticated (you)" + assert prompt =~ "Public, Private and Video Files" + assert prompt =~ "Vimeo access token: " + refute prompt =~ "synthetic-pasted-token" + end + end + + test "an environment token bypasses the setup guidance and prompt" do + prompt = + capture_io(:stderr, fn -> + output = + capture_io(fn -> + assert MaveCli.Vimeo.token() == {:ok, "synthetic-vimeo-token"} + end) + + assert output == "" + end) + + assert prompt == "" + refute File.exists?(VimeoAuth.path()) + end + + test "imports reuse the token saved by the first prompt", %{tmp_dir: directory} do + System.delete_env("VIMEO_ACCESS_TOKEN") + + args = [ + "import", + "vimeo", + "--dry-run", + "--no-progress", + "--base-url", + "http://local.test/v1/", + "--token", + "mave-secret", + "--state-file", + Path.join(directory, "import.json") + ] + + for input <- ["pasted-vimeo-token\n", ""] do + prompt = + capture_io(:stderr, fn -> + output = + capture_io([input: input], fn -> + assert MaveCli.CLI.run(args) == 0 + end) + + assert Jason.decode!(output)["videos"] == 1 + refute output =~ "pasted-vimeo-token" + end) + + if input == "", do: assert(prompt == ""), else: assert(prompt =~ "Vimeo token saved") + end + + refute File.exists?(Path.join(directory, "import.json")) + end + + test "dry-run accepts importer flags and prints a readable preview", %{tmp_dir: directory} do + path = Path.join(directory, "state.json") + + output = + capture_io(fn -> + assert MaveCli.CLI.run([ + "import", + "vimeo", + "--dry-run", + "--collection", + "target", + "--format", + "table", + "--state-file", + path, + "--no-progress", + "--base-url", + "http://local.test/v1/", + "--token", + "mave-token" + ]) == 0 + end) + + assert output =~ "Example title" + assert output =~ "planned" + refute File.exists?(path) + end + + test "unavailable videos produce JSON and a nonzero exit status", %{tmp_dir: directory} do + output = + capture_io(fn -> + assert MaveCli.CLI.run([ + "import", + "vimeo", + "--state-file", + Path.join(directory, "state.json"), + "--no-progress", + "--base-url", + "http://local.test/v1/", + "--token", + "mave-token" + ]) == 1 + end) + + assert %{"failed" => 1, "data" => [%{"name" => "Example title", "status" => "failed"}]} = + Jason.decode!(output) + + refute output =~ "synthetic-vimeo-token" + end + + test "rejects invalid folder IDs before accessing an account" do + error = + capture_io(:stderr, fn -> + assert MaveCli.CLI.run(["import", "vimeo", "--folder", "../other", "--token", "test"]) == + 1 + end) + + assert error =~ "--folder must be a numeric Vimeo folder ID" + end end test "generates an upload token for any documented upload subject" do @@ -111,7 +317,7 @@ defmodule MaveCli.CLITest do assert String.trim(output) == MaveCli.version() end - describe "manual login" do + describe "login" do @describetag :tmp_dir setup %{tmp_dir: directory} do @@ -134,7 +340,6 @@ defmodule MaveCli.CLITest do output = capture_io([input: input], fn -> assert MaveCli.CLI.run([ - "auth", "login", "--no-browser", "--base-url", @@ -152,6 +357,117 @@ defmodule MaveCli.CLITest do MaveCli.Config.token(nil, base_url: "http://local.test/api/v1/") end + test "repeated login keeps existing credentials and never starts browser authorization" do + assert :ok = MaveCli.Config.save_token("existing-secret", base_url: "http://local.test/v1/") + saved = File.read!(MaveCli.Config.path()) + + Req.Test.stub(__MODULE__, fn _conn -> + flunk("an existing login must not make HTTP requests") + end) + + for command <- [["auth", "login"], ["login"]], + arguments <- [ + [], + ["--no-browser"], + ["replacement-secret"], + ["--token", "replacement-secret"] + ] do + prompt = + capture_io(:stderr, fn -> + output = + capture_io(fn -> + assert MaveCli.CLI.run( + command ++ ["--base-url", "http://local.test/v1"] ++ arguments + ) == 0 + end) + + assert output =~ "Already logged in" + assert output =~ "mave logout" + refute output =~ "existing-secret" + refute output =~ "replacement-secret" + end) + + assert prompt == "" + assert File.read!(MaveCli.Config.path()) == saved + end + end + + test "an environment token prevents creating a new login without persisting the token" do + System.put_env("MAVE_TOKEN", "environment-secret") + + Req.Test.stub(__MODULE__, fn _conn -> + flunk("environment authentication must not start login") + end) + + output = capture_io(fn -> assert MaveCli.CLI.run(["auth", "login"]) == 0 end) + assert output =~ "Already authenticated via MAVE_TOKEN" + assert output =~ "Unset MAVE_TOKEN" + refute output =~ "environment-secret" + refute File.exists?(MaveCli.Config.path()) + end + + test "logout permits a new browser login" do + for command <- [["auth", "logout"], ["logout"]] do + assert :ok = + MaveCli.Config.save_token("existing-secret", base_url: "http://local.test/v1/") + + capture_io(fn -> assert MaveCli.CLI.run(command) == 0 end) + refute File.exists?(MaveCli.Config.path()) + end + + Req.Test.expect(__MODULE__, fn conn -> + assert conn.method == "POST" + assert conn.request_path == "/v1/cli/authorizations" + Plug.Conn.send_resp(conn, 501, "") + end) + + capture_io(:stderr, fn -> + capture_io([input: "replacement-secret\n"], fn -> + assert MaveCli.CLI.run(["auth", "login", "--base-url", "http://local.test/v1/"]) == 0 + end) + end) + + assert {:ok, "replacement-secret"} = + MaveCli.Config.token(nil, base_url: "http://local.test/v1/") + end + + test "credentials saved for a different server do not block login" do + assert :ok = MaveCli.Config.save_token("first-secret", base_url: "https://first.test/v1/") + + output = + capture_io(fn -> + assert MaveCli.CLI.run([ + "auth", + "login", + "second-secret", + "--base-url", + "https://second.test/v1/" + ]) == 0 + end) + + assert output =~ "Mave token saved" + + assert {:ok, "second-secret"} = + MaveCli.Config.token(nil, base_url: "https://second.test/v1/") + end + + test "legacy unbound credentials can be replaced with a server-bound login" do + File.mkdir_p!(Path.dirname(MaveCli.Config.path())) + File.write!(MaveCli.Config.path(), Jason.encode!(%{"token" => "unbound-secret"})) + + capture_io(fn -> + assert MaveCli.CLI.run([ + "auth", + "login", + "bound-secret", + "--base-url", + "https://local.test/v1/" + ]) == 0 + end) + + assert {:ok, "bound-secret"} = MaveCli.Config.token(nil, base_url: "https://local.test/v1/") + end + test "rejects blank input and EOF without saving credentials" do for {input, message} <- [ {" \n", "token must not be empty"}, diff --git a/test/mave_cli/vimeo_auth_test.exs b/test/mave_cli/vimeo_auth_test.exs new file mode 100644 index 0000000..ac4d5d6 --- /dev/null +++ b/test/mave_cli/vimeo_auth_test.exs @@ -0,0 +1,304 @@ +defmodule MaveCli.VimeoAuthTest do + use ExUnit.Case, async: false + import ExUnit.CaptureIO + + alias MaveCli.{CLI, Config, PrivateFile, Vimeo} + alias MaveCli.Vimeo.Auth + + @moduletag :tmp_dir + + setup {Req.Test, :verify_on_exit!} + + setup %{tmp_dir: directory} do + previous_env = + Map.new(~w(MAVE_CONFIG_HOME MAVE_TOKEN VIMEO_ACCESS_TOKEN), &{&1, System.get_env(&1)}) + + previous_req = Req.default_options() + System.put_env("MAVE_CONFIG_HOME", directory) + System.delete_env("MAVE_TOKEN") + System.delete_env("VIMEO_ACCESS_TOKEN") + Req.default_options(plug: {Req.Test, __MODULE__}) + + on_exit(fn -> + Enum.each(previous_env, fn + {name, nil} -> System.delete_env(name) + {name, value} -> System.put_env(name, value) + end) + + Req.default_options(previous_req) + end) + end + + test "first use validates and saves a private token, then reuses it without prompting" do + expect_validation("vimeo-secret") + + prompt = + capture_io(:stderr, fn -> + output = + capture_io([input: " vimeo-secret \n"], fn -> + assert Vimeo.token() == {:ok, "vimeo-secret"} + end) + + assert output == "" + end) + + assert prompt =~ "Vimeo token saved" + refute prompt =~ "vimeo-secret" + assert Jason.decode!(File.read!(Auth.path())) == %{"token" => "vimeo-secret"} + assert Bitwise.band(File.stat!(Auth.path()).mode, 0o777) == 0o600 + refute File.exists?(Config.path()) + + assert capture_io(:stderr, fn -> + assert Vimeo.token() == {:ok, "vimeo-secret"} + end) == "" + end + + test "importer login, status and logout work without Mave authentication" do + error = + capture_io(:stderr, fn -> + assert CLI.run(["import", "vimeo", "status"]) == 1 + end) + + assert error =~ "mave import vimeo login" + + expect_validation("first-secret") + + capture_io(:stderr, fn -> + output = + capture_io([input: "first-secret\n"], fn -> + assert CLI.run(["import", "vimeo", "login"]) == 0 + end) + + assert output =~ "Logged in to Vimeo" + refute output =~ "first-secret" + end) + + saved = File.read!(Auth.path()) + output = capture_io(fn -> assert CLI.run(["import", "vimeo", "login"]) == 0 end) + assert output =~ "Already logged in" + assert output =~ "mave import vimeo logout" + assert File.read!(Auth.path()) == saved + + output = capture_io(fn -> assert CLI.run(["import", "vimeo", "status"]) == 0 end) + assert output =~ Auth.path() + refute output =~ "first-secret" + + for _ <- 1..2 do + output = capture_io(fn -> assert CLI.run(["import", "vimeo", "logout"]) == 0 end) + assert output =~ "Saved Vimeo token removed" + refute File.exists?(Auth.path()) + end + + expect_validation("second-secret") + + capture_io(:stderr, fn -> + capture_io([input: "second-secret\n"], fn -> + assert CLI.run(["import", "vimeo", "login"]) == 0 + end) + end) + + assert Vimeo.token() == {:ok, "second-secret"} + refute File.exists?(Config.path()) + end + + test "Mave and Vimeo logout preserve each other's login and import progress" do + assert :ok = Config.save_token("mave-secret") + save_token("vimeo-secret") + state_path = Path.join([Path.dirname(Config.path()), "imports", "state.json"]) + assert :ok = PrivateFile.write_atomic(state_path, "saved-import-progress") + + capture_io(fn -> assert CLI.run(["logout"]) == 0 end) + assert Vimeo.token() == {:ok, "vimeo-secret"} + assert File.read!(state_path) == "saved-import-progress" + + assert :ok = Config.save_token("mave-secret") + saved_mave = File.read!(Config.path()) + capture_io(fn -> assert CLI.run(["import", "vimeo", "logout"]) == 0 end) + assert Config.token() == {:ok, "mave-secret"} + assert File.read!(Config.path()) == saved_mave + assert File.read!(state_path) == "saved-import-progress" + refute File.exists?(Auth.path()) + end + + test "environment tokens override saved credentials without being persisted" do + System.put_env("VIMEO_ACCESS_TOKEN", " environment-secret ") + assert Vimeo.token() == {:ok, "environment-secret"} + assert {:ok, message} = Auth.login() + assert message =~ "VIMEO_ACCESS_TOKEN" + refute message =~ "environment-secret" + refute File.exists?(Auth.path()) + + save_token("saved-secret") + saved = File.read!(Auth.path()) + assert Vimeo.token() == {:ok, "environment-secret"} + assert {:ok, "Logged in to Vimeo via VIMEO_ACCESS_TOKEN."} = Auth.status() + assert File.read!(Auth.path()) == saved + + System.put_env("VIMEO_ACCESS_TOKEN", " ") + assert Vimeo.token() == {:ok, "saved-secret"} + System.put_env("VIMEO_ACCESS_TOKEN", "environment-secret") + + output = capture_io(fn -> assert CLI.run(["import", "vimeo", "logout"]) == 0 end) + assert output =~ "VIMEO_ACCESS_TOKEN is still set" + refute output =~ "environment-secret" + refute File.exists?(Auth.path()) + assert Vimeo.token() == {:ok, "environment-secret"} + end + + test "rejected tokens are not saved or exposed" do + for status <- [401, 403] do + Req.Test.expect(__MODULE__, fn conn -> + Req.Test.json(Plug.Conn.put_status(conn, status), %{"error" => "vimeo-secret"}) + end) + + error = + capture_io(:stderr, fn -> + output = + capture_io([input: "vimeo-secret\n"], fn -> + assert CLI.run(["import", "vimeo", "login"]) == 1 + end) + + assert output == "" + end) + + refute error =~ "vimeo-secret" + refute error =~ "Vimeo token saved" + refute File.exists?(Auth.path()) + end + end + + test "a rate-limited login retries the same entered token and saves it after validation" do + Req.default_options(plug: {Req.Test, __MODULE__}, retry_delay: fn _ -> 0 end) + + Req.Test.expect(__MODULE__, fn conn -> + assert Plug.Conn.get_req_header(conn, "authorization") == ["Bearer vimeo-secret"] + Plug.Conn.send_resp(conn, 429, "private-body") + end) + + expect_validation("vimeo-secret") + + prompt = + capture_io(:stderr, fn -> + output = + capture_io([input: "vimeo-secret\n"], fn -> + assert CLI.run(["import", "vimeo", "login"]) == 0 + end) + + assert output =~ "Logged in to Vimeo" + refute output =~ "vimeo-secret" + end) + + assert prompt =~ "waiting 60s" + assert prompt =~ "Vimeo token saved" + assert length(Regex.scan(~r/^Vimeo access token: /m, prompt)) == 1 + assert Vimeo.token() == {:ok, "vimeo-secret"} + refute prompt =~ "vimeo-secret" + refute prompt =~ "private-body" + end + + test "a long cooldown reports when to retry login without saving an unvalidated token" do + Req.Test.expect(__MODULE__, fn conn -> + conn + |> Plug.Conn.put_resp_header("retry-after", "900") + |> Plug.Conn.send_resp(429, "private-body") + end) + + error = + capture_io(:stderr, fn -> + output = + capture_io([input: "vimeo-secret\n"], fn -> + assert CLI.run(["import", "vimeo", "login"]) == 1 + end) + + assert output == "" + end) + + assert error =~ "in about 900 seconds" + assert error =~ "token has not been saved" + refute error =~ "vimeo-secret" + refute error =~ "private-body" + refute File.exists?(Auth.path()) + end + + test "blank input and EOF do not save a token or contact Vimeo" do + for {input, message} <- [ + {" \n", "Vimeo token must not be empty"}, + {"", "could not read Vimeo token"} + ] do + error = + capture_io(:stderr, fn -> + capture_io([input: input], fn -> + assert CLI.run(["import", "vimeo", "login"]) == 1 + end) + end) + + assert error =~ message + refute File.exists?(Auth.path()) + end + end + + test "invalid saved credentials can be removed without exposing their contents" do + assert :ok = PrivateFile.write_atomic(Auth.path(), "invalid-private-content") + assert {:error, message} = Auth.token() + assert message =~ "mave import vimeo logout" + refute message =~ "invalid-private-content" + assert {:ok, _} = Auth.logout() + assert {:error, message} = Auth.status() + assert message =~ "no Vimeo token found" + end + + test "a failed save does not report a successful login" do + Req.Test.expect(__MODULE__, fn conn -> + File.mkdir_p!(Auth.path()) + Req.Test.json(conn, %{"uri" => "/users/42"}) + end) + + error = + capture_io(:stderr, fn -> + output = + capture_io([input: "vimeo-secret\n"], fn -> + assert CLI.run(["import", "vimeo", "login"]) == 1 + end) + + assert output == "" + end) + + assert error =~ "could not save the Vimeo token" + refute error =~ "vimeo-secret" + refute error =~ "Vimeo token saved" + assert File.ls!(Auth.path()) == [] + end + + test "importer help lists credential commands and unknown subcommands fail before login" do + output = capture_io(fn -> assert CLI.run(["import", "vimeo", "--help"]) == 0 end) + + for action <- ["login", "status", "logout"] do + assert output =~ "mave import vimeo #{action}" + end + + for args <- [["unknown"], ["login", "unexpected-secret"]] do + error = + capture_io(:stderr, fn -> + assert CLI.run(["import", "vimeo"] ++ args) == 1 + end) + + assert error =~ "usage: mave import vimeo" + refute error =~ "unexpected-secret" + end + + refute File.exists?(Auth.path()) + end + + defp expect_validation(token) do + Req.Test.expect(__MODULE__, fn conn -> + assert conn.host == "api.vimeo.com" + assert conn.method == "GET" + assert conn.request_path == "/me" + assert Plug.Conn.get_req_header(conn, "authorization") == ["Bearer " <> token] + Req.Test.json(conn, %{"uri" => "/users/42"}) + end) + end + + defp save_token(token), + do: assert(:ok == PrivateFile.write_atomic(Auth.path(), Jason.encode!(%{"token" => token}))) +end diff --git a/test/mave_cli/vimeo_import_test.exs b/test/mave_cli/vimeo_import_test.exs new file mode 100644 index 0000000..a47b1a1 --- /dev/null +++ b/test/mave_cli/vimeo_import_test.exs @@ -0,0 +1,393 @@ +defmodule MaveCli.VimeoImportTest do + use ExUnit.Case, async: true + + alias MaveCli.{Client, Vimeo} + alias MaveCli.Vimeo.Import + + @moduletag :tmp_dir + + setup {Req.Test, :verify_on_exit!} + + setup %{tmp_dir: directory} do + owner = self() + + Req.Test.stub(__MODULE__, fn conn -> + body = if conn.method == "POST", do: Jason.decode!(Req.Test.raw_body(conn)), else: nil + send(owner, {:request, conn.host, conn.method, conn.request_path, body}) + respond(conn, body) + end) + + %{ + mave: + Client.new("mave-secret", base_url: "https://mave.test/v1/", plug: {Req.Test, __MODULE__}), + vimeo: Vimeo.new("vimeo-secret") |> Req.merge(plug: {Req.Test, __MODULE__}), + opts: [state_file: Path.join(directory, "import.json"), progress: false] + } + end + + test "imports titles and nested folders using only the existing Mave API", ctx do + assert {:ok, result} = run(ctx, collection: "destination") + assert result["folders"] == 3 + assert result["videos"] == 3 + assert result["failed"] == 0 + + assert_receive {:request, "mave.test", "POST", "/v1/collections", + %{"name" => "Campaign", "collection" => "destination"}} + + assert_receive {:request, "mave.test", "POST", "/v1/collections", + %{"name" => "Empty", "collection" => "destination"}} + + assert_receive {:request, "mave.test", "POST", "/v1/collections", + %{"name" => "Edits", "collection" => "collection-Campaign"}} + + assert_receive {:request, "mave.test", "POST", "/v1/videos", + %{ + "name" => "Één & twee 🎬", + "collection" => "collection-Edits", + "input_url" => "https://cdn.test/101.mp4?signature=private" + }} + + assert_receive {:request, "mave.test", "POST", "/v1/videos", + %{ + "name" => "Overview", + "collection" => "collection-Campaign", + "input_url" => "https://cdn.test/102.mp4?signature=private" + }} + + assert_receive {:request, "mave.test", "POST", "/v1/videos", + %{ + "name" => "Unfiled", + "collection" => "destination", + "input_url" => "https://cdn.test/103.mp4?signature=private" + }} + + state = File.read!(ctx.opts[:state_file]) + refute state =~ "secret" + refute state =~ "signature" + refute Jason.encode!(result) =~ "signature" + assert Bitwise.band(File.stat!(ctx.opts[:state_file]).mode, 0o777) == 0o600 + assert Enum.find(result["data"], &(&1["vimeo_id"] == "101"))["folder"] == "Campaign/Edits" + end + + test "dry-run previews nesting without changing Mave or writing state", ctx do + assert {:ok, result} = run(ctx, dry_run: true) + assert result["dry_run"] + assert result["folders"] == 3 + assert result["videos"] == 3 + refute File.exists?(ctx.opts[:state_file]) + refute_receive {:request, _, "POST", _, _} + refute_receive {:request, "api.vimeo.com", _, "/videos/101", _} + end + + test "a selected folder imports its subtree including the selected folder", ctx do + assert {:ok, result} = run(ctx, folder: "10", dry_run: true) + assert result["folders"] == 2 + assert result["videos"] == 2 + refute_receive {:request, "api.vimeo.com", _, "/me/videos", _} + refute Enum.any?(result["data"], &(&1["name"] == "Empty")) + end + + test "resume reuses saved IDs without creating videos or collections twice", ctx do + assert {:ok, _} = run(ctx) + flush_requests() + assert {:ok, result} = run(ctx, resume: true) + assert Enum.all?(result["data"], &(&1["status"] == "existing")) + refute_receive {:request, _, "POST", _, _} + refute_receive {:request, "api.vimeo.com", _, "/videos/101", _} + assert {:error, message} = run(ctx) + assert message =~ "use --resume" + end + + test "resume is bound to the Mave destination", ctx do + assert {:ok, _} = run(ctx) + assert {:error, message} = run(ctx, resume: true, collection: "different") + assert message =~ "different Vimeo account, Mave destination or folder selection" + end + + test "an interrupted create remains pending and is never retried blindly", ctx do + owner = self() + + Req.Test.stub(:failing_mave, fn conn -> + if conn.method == "POST" do + send(owner, :create_attempt) + state = Jason.decode!(File.read!(ctx.opts[:state_file])) + assert state["pending"]["kind"] == "folders" + Plug.Conn.send_resp(conn, 503, "private response") + else + respond(conn, nil) + end + end) + + mave = + Client.new("mave-secret", + base_url: "https://mave.test/v1/", + plug: {Req.Test, :failing_mave} + ) + + assert {:error, message} = Import.run_with_client(mave, ctx.vimeo, ctx.opts) + assert message =~ "did not confirm" + refute message =~ "private response" + assert_receive :create_attempt + refute_receive :create_attempt + assert {:error, message} = run(ctx, resume: true) + assert message =~ "unknown outcome" + refute_receive {:request, _, "POST", _, _} + end + + test "videos without files are reported while other videos import and can be retried", ctx do + Req.Test.stub(:missing_file, fn conn -> + if conn.request_path == "/videos/101", + do: Req.Test.json(conn, %{"files" => []}), + else: respond(conn, nil) + end) + + vimeo = Vimeo.new("vimeo-secret") |> Req.merge(plug: {Req.Test, :missing_file}) + assert {:error, {:import, result}} = Import.run_with_client(ctx.mave, vimeo, ctx.opts) + assert result["failed"] == 1 + state = Jason.decode!(File.read!(ctx.opts[:state_file])) + refute Map.has_key?(state["videos"], "101") + assert Map.has_key?(state["videos"], "102") + assert {:ok, result} = run(ctx, resume: true) + assert result["failed"] == 0 + end + + test "rate limiting stops further video requests and preserves progress for resume", ctx do + owner = self() + + Req.Test.stub(:limited_vimeo, fn conn -> + send(owner, {:vimeo_request, conn.request_path}) + + if conn.request_path == "/videos/102" do + conn + |> Plug.Conn.put_resp_header("retry-after", "900") + |> Plug.Conn.send_resp(429, "private-body") + else + respond(conn, nil) + end + end) + + vimeo = Vimeo.new("vimeo-secret") |> Req.merge(plug: {Req.Test, :limited_vimeo}) + + assert {:error, message} = Import.run_with_client(ctx.mave, vimeo, ctx.opts) + assert message =~ "Vimeo rate limit reached" + assert message =~ "--resume" + refute message =~ "private-body" + refute_receive {:vimeo_request, "/videos/103"} + state = Jason.decode!(File.read!(ctx.opts[:state_file])) + assert state["videos"] == %{"101" => "video-101"} + assert state["pending"] == nil + + flush_requests() + assert {:ok, result} = run(ctx, resume: true) + assert result["failed"] == 0 + refute_receive {:request, "api.vimeo.com", _, "/videos/101", _} + refute_receive {:request, _, "POST", "/v1/collections", _} + end + + test "wait checks processing after saving the returned video ID", ctx do + assert {:ok, result} = run(ctx, wait: true) + + assert Enum.all?( + Enum.filter(result["data"], &(&1["type"] == "video")), + &(&1["status"] == "playable") + ) + + assert_receive {:request, "mave.test", "GET", "/v1/videos/video-101", nil} + end + + test "folder lists that also include children do not flatten or duplicate them", ctx do + Req.Test.stub(:all_folders, fn conn -> + if conn.request_path == "/me/projects" do + Req.Test.json( + conn, + page([folder("11", "Edits", 0), folder("10", "Campaign", 1), folder("12", "Empty", 0)]) + ) + else + respond(conn, nil) + end + end) + + vimeo = Vimeo.new("vimeo-secret") |> Req.merge(plug: {Req.Test, :all_folders}) + + assert {:ok, result} = + Import.run_with_client(ctx.mave, vimeo, Keyword.put(ctx.opts, :dry_run, true)) + + assert result["folders"] == 3 + edits = Enum.find(result["data"], &(&1["vimeo_id"] == "11")) + assert edits["folder"] == "Campaign" + assert Enum.find(result["data"], &(&1["vimeo_id"] == "101"))["folder"] == "Campaign/Edits" + end + + test "cycles fail during discovery before any Mave writes", ctx do + Req.Test.stub(:cyclic_folders, fn conn -> + if conn.request_path == "/users/42/projects/10/items" do + Req.Test.json( + conn, + page([%{"type" => "folder", "folder" => folder("10", "Campaign", 1)}]) + ) + else + respond(conn, nil) + end + end) + + vimeo = Vimeo.new("vimeo-secret") |> Req.merge(plug: {Req.Test, :cyclic_folders}) + assert {:error, message} = Import.run_with_client(ctx.mave, vimeo, ctx.opts) + assert message =~ "cyclic" + refute_receive {:request, _, "POST", _, _} + refute File.exists?(ctx.opts[:state_file]) + end + + test "processing failure retains the ID and resumes polling without another import", ctx do + Req.Test.stub(:processing_error, fn conn -> + if conn.method == "GET" and String.starts_with?(conn.request_path, "/v1/videos/") do + Plug.Conn.send_resp(conn, 500, "processing failed") + else + body = if conn.method == "POST", do: Jason.decode!(Req.Test.raw_body(conn)), else: nil + respond(conn, body) + end + end) + + mave = + Client.new("mave-secret", + base_url: "https://mave.test/v1/", + plug: {Req.Test, :processing_error} + ) + + assert {:error, {:import, result}} = + Import.run_with_client(mave, ctx.vimeo, Keyword.put(ctx.opts, :wait, true)) + + assert result["failed"] == 3 + assert map_size(Jason.decode!(File.read!(ctx.opts[:state_file]))["videos"]) == 3 + flush_requests() + assert {:ok, result} = run(ctx, resume: true, wait: true) + assert result["failed"] == 0 + refute_receive {:request, _, "POST", _, _} + end + + test "invalid or missing resume state cannot start an import", ctx do + assert {:error, message} = run(ctx, resume: true) + assert message =~ "no import state" + File.write!(ctx.opts[:state_file], "not json") + assert {:error, message} = run(ctx, resume: true) + assert message =~ "import state is invalid" + refute_receive {:request, _, "POST", _, _} + end + + defp run(ctx, extra \\ []), + do: Import.run_with_client(ctx.mave, ctx.vimeo, Keyword.merge(ctx.opts, extra)) + + defp respond(%{host: "mave.test"} = conn, body) do + assert Plug.Conn.get_req_header(conn, "authorization") == ["Bearer mave-secret"] + + case {conn.method, conn.request_path} do + {"GET", "/v1/videos"} -> + Req.Test.json(conn, %{"space_id" => "space-1", "data" => []}) + + {"POST", "/v1/collections"} -> + Req.Test.json(conn, %{"id" => "collection-#{body["name"]}"}) + + {"POST", "/v1/videos"} -> + id = body["input_url"] |> URI.parse() |> Map.fetch!(:path) |> Path.basename(".mp4") + Req.Test.json(conn, %{"id" => "video-#{id}"}) + + {"GET", "/v1/videos/" <> id} -> + Req.Test.json(conn, %{"id" => id, "renditions" => ["hls"]}) + end + end + + defp respond(%{host: "api.vimeo.com"} = conn, _body) do + assert Plug.Conn.get_req_header(conn, "authorization") == ["Bearer vimeo-secret"] + assert_folder_query(conn) + vimeo_json(conn, vimeo_response(conn.request_path)) + end + + defp vimeo_json(%{request_path: "/videos/" <> _id} = conn, body) do + conn + |> Plug.Conn.put_resp_header("content-type", "application/vnd.vimeo.video+json;version=3.4") + |> Plug.Conn.put_resp_header("content-encoding", "gzip") + |> Plug.Conn.send_resp(200, :zlib.gzip(Jason.encode!(body))) + end + + defp vimeo_json(conn, body), do: Req.Test.json(conn, body) + + # Synthetic responses follow Vimeo's project and project-item response schemas. + # https://developer.vimeo.com/api/reference/response/project-item + defp assert_folder_query(%{request_path: "/users/42/projects/" <> path} = conn) do + query = URI.decode_query(conn.query_string) + + if String.ends_with?(path, "/items") do + assert query["filter"] == "folder" + else + assert query["fields"] == "uri,name" + assert query["include_subfolders"] == "false" + end + end + + defp assert_folder_query(_conn), do: :ok + + defp vimeo_response("/me"), do: %{"uri" => "/users/42"} + + defp vimeo_response("/me/projects"), + do: page([folder("10", "Campaign", 1), folder("12", "Empty", 0)]) + + defp vimeo_response("/me/projects/10"), do: folder("10", "Campaign", 1) + + defp vimeo_response("/users/42/projects/10/items"), + do: page([%{"type" => "folder", "folder" => folder("11", "Edits", 0)}]) + + defp vimeo_response("/users/42/projects/10/videos"), do: page([video("102", "Overview")]) + + defp vimeo_response("/users/42/projects/11/videos"), + do: page([video("101", "Één & twee 🎬")]) + + defp vimeo_response("/users/42/projects/12/videos"), do: page([]) + + defp vimeo_response("/me/videos"), + do: + page([ + video("101", "Één & twee 🎬"), + video("102", "Overview"), + video("103", "Unfiled") + ]) + + defp vimeo_response("/videos/" <> id), + do: %{ + "files" => [ + %{ + "type" => "video/mp4", + "width" => 1920, + "height" => 1080, + "link" => "https://cdn.test/#{id}.mp4?signature=private" + } + ] + } + + defp folder(id, name, children) do + %{ + "uri" => "/users/42/projects/#{id}", + "name" => name, + "metadata" => %{ + "connections" => %{ + "folders" => %{"total" => children, "uri" => "/users/42/projects/#{id}/items"} + } + } + } + end + + defp video(id, name) do + %{ + "uri" => "/videos/#{id}", + "name" => name + } + end + + defp page(items), do: %{"data" => items, "paging" => %{"next" => nil}} + + defp flush_requests do + receive do + {:request, _, _, _, _} -> flush_requests() + after + 0 -> :ok + end + end +end diff --git a/test/mave_cli/vimeo_rate_limit_test.exs b/test/mave_cli/vimeo_rate_limit_test.exs new file mode 100644 index 0000000..94e43e4 --- /dev/null +++ b/test/mave_cli/vimeo_rate_limit_test.exs @@ -0,0 +1,134 @@ +defmodule MaveCli.VimeoRateLimitTest do + use ExUnit.Case, async: true + import ExUnit.CaptureIO + + alias MaveCli.Vimeo + alias MaveCli.Vimeo.RateLimit + + @now ~U[2026-09-08 15:00:00Z] + setup {Req.Test, :verify_on_exit!} + + test "reads Vimeo reset timestamps and accounts for server clock differences" do + for reset <- [ + "2026-09-08T15:01:30Z", + "Tue, 08 Sep 2026 15:01:30 GMT", + to_string(DateTime.to_unix(DateTime.add(@now, 90))) + ] do + response = + response([{"x-ratelimit-reset", reset}, {"date", "Tue, 08 Sep 2026 15:00:00 GMT"}]) + + limit = RateLimit.from_response(response, DateTime.add(@now, -120)) + assert limit.seconds == 91 + assert limit.reset_known? + assert limit.retry_at == ~U[2026-09-08 14:59:31Z] + assert RateLimit.message(limit) =~ "2026-09-08 14:59:31 UTC" + end + end + + test "honors Retry-After seconds and dates, using the later of both reset headers" do + assert RateLimit.from_response(response([{"retry-after", "120"}]), @now).seconds == 120 + + assert RateLimit.from_response( + response([{"retry-after", "Tue, 08 Sep 2026 15:02:00 GMT"}]), + @now + ).seconds == 121 + + limit = + RateLimit.from_response( + response([{"retry-after", "5"}, {"x-ratelimit-reset", "2026-09-08T15:03:00Z"}]), + @now + ) + + assert limit.seconds == 181 + end + + test "missing, malformed and stale reset headers use a minute without exposing header contents" do + for headers <- [ + [], + [{"retry-after", "invalid-private-header"}], + [{"retry-after", "-1"}], + [{"retry-after", "1.5"}], + [{"x-ratelimit-reset", "invalid-private-header"}], + [{"x-ratelimit-reset", "2020-01-01T00:00:00Z"}] + ] do + limit = RateLimit.from_response(response(headers), @now) + assert limit.seconds == 60 + refute limit.reset_known? + assert RateLimit.message(limit) =~ "Wait at least 60 seconds" + refute RateLimit.message(limit) =~ "private" + end + end + + test "the response step supplies a conservative Retry-After before Req retries" do + message = + capture_io(:stderr, fn -> + {request, response} = RateLimit.handle({Vimeo.new("test-secret"), response([])}) + refute request.halted + assert Req.Response.get_header(response, "retry-after") == ["60"] + end) + + assert message =~ "waiting 60s" + assert message =~ "(1/3)" + end + + test "long waits and exhausted or disabled retries halt without a misleading wait message" do + request = Vimeo.new("test-secret") + exhausted = Req.Request.put_private(request, :req_retry_count, 3) + + for {request, response} <- [ + {request, response([{"retry-after", "900"}])}, + {exhausted, response([])}, + {Req.merge(request, retry: false), response([])} + ] do + assert capture_io(:stderr, fn -> + {request, _response} = RateLimit.handle({request, response}) + assert request.halted + end) == "" + end + end + + test "a real retry waits for the declared cooldown and keeps stdout clean" do + started = System.monotonic_time(:millisecond) + + Req.Test.expect(__MODULE__, fn conn -> + conn + |> Plug.Conn.put_resp_header("retry-after", "1") + |> Plug.Conn.send_resp(429, "private-body") + end) + + Req.Test.expect(__MODULE__, fn conn -> + assert System.monotonic_time(:millisecond) - started >= 1000 + Req.Test.json(conn, %{"uri" => "/users/42"}) + end) + + message = + capture_io(:stderr, fn -> + assert capture_io(fn -> + assert {:ok, %{"uri" => "/users/42"}} = Vimeo.get(client(), "/me") + end) == "" + end) + + assert message =~ "waiting 1s" + refute message =~ "private-body" + end + + test "persistent rate limiting stops after the retry budget" do + for _ <- 1..4 do + Req.Test.expect(__MODULE__, fn conn -> Plug.Conn.send_resp(conn, 429, "private-body") end) + end + + message = + capture_io(:stderr, fn -> + assert {:error, %RateLimit{seconds: 60}} = + Vimeo.get(Req.merge(client(), retry_delay: fn _ -> 0 end), "/me") + end) + + assert length(Regex.scan(~r/waiting 60s/, message)) == 3 + refute message =~ "private-body" + end + + defp response(headers), + do: Req.Response.new(status: 429, headers: headers, body: "private-body") + + defp client, do: Vimeo.new("test-secret") |> Req.merge(plug: {Req.Test, __MODULE__}) +end diff --git a/test/mave_cli/vimeo_test.exs b/test/mave_cli/vimeo_test.exs new file mode 100644 index 0000000..cda5f84 --- /dev/null +++ b/test/mave_cli/vimeo_test.exs @@ -0,0 +1,168 @@ +defmodule MaveCli.VimeoTest do + use ExUnit.Case, async: true + + alias MaveCli.Vimeo + + setup {Req.Test, :verify_on_exit!} + + test "paginates Vimeo responses with isolated credentials" do + Req.Test.expect(__MODULE__, fn conn -> + assert conn.host == "api.vimeo.com" + assert Plug.Conn.get_req_header(conn, "authorization") == ["Bearer vimeo-secret"] + assert URI.decode_query(conn.query_string)["per_page"] == "100" + + Req.Test.json(conn, %{ + "data" => [%{"uri" => "/videos/1"}], + "paging" => %{"next" => "/me/videos?page=2&per_page=100"} + }) + end) + + Req.Test.expect(__MODULE__, fn conn -> + assert URI.decode_query(conn.query_string)["page"] == "2" + Req.Test.json(conn, %{"data" => [%{"uri" => "/videos/2"}], "paging" => %{"next" => nil}}) + end) + + assert {:ok, [%{"uri" => "/videos/1"}, %{"uri" => "/videos/2"}]} = + Vimeo.all(client(), "/me/videos") + end + + test "rejects cross-origin pagination and does not follow HTTP redirects" do + Req.Test.expect(__MODULE__, fn conn -> + Req.Test.json(conn, %{"data" => [], "paging" => %{"next" => "https://other.test/token"}}) + end) + + assert {:error, "Vimeo returned a URL outside its API"} = Vimeo.all(client(), "/me/videos") + + Req.Test.expect(__MODULE__, fn conn -> + conn + |> Plug.Conn.put_resp_header("location", "https://other.test/token") + |> Plug.Conn.send_resp(302, "") + end) + + assert {:error, "Vimeo API returned HTTP 302"} = Vimeo.get(client(), "/me") + end + + test "detects repeating pages rather than looping forever" do + Req.Test.expect(__MODULE__, fn conn -> + Req.Test.json(conn, %{"data" => [], "paging" => %{"next" => "/me/videos"}}) + end) + + assert {:error, "Vimeo returned invalid or repeating pagination"} = + Vimeo.all(client(), "/me/videos") + end + + test "selects a source video ahead of renditions and ignores expired and HLS links" do + Req.Test.expect(__MODULE__, fn conn -> + Req.Test.json(conn, %{ + "download" => [ + file("old-original", %{"quality" => "source", "expires" => "2000-01-01T00:00:00Z"}), + file("original", %{"quality" => "source"}), + file("4k", %{"width" => 3840, "height" => 2160}) + ], + "files" => [ + %{ + "link" => "https://cdn.test/video.m3u8", + "type" => "application/x-mpegURL", + "quality" => "hls" + } + ] + }) + end) + + assert {:ok, "https://cdn.test/original.mp4"} = Vimeo.source(client(), "1") + end + + test "falls back to the best MP4 in files when downloads are unavailable" do + Req.Test.expect(__MODULE__, fn conn -> + Req.Test.json(conn, %{ + "files" => [file("sd", %{"height" => 360}), file("hd", %{"height" => 1080})] + }) + end) + + assert {:ok, "https://cdn.test/hd.mp4"} = Vimeo.source(client(), "1") + end + + test "reads compressed video responses with Vimeo's vendor JSON content type" do + Req.Test.expect(__MODULE__, fn conn -> + assert URI.decode_query(conn.query_string)["fields"] == "uri,files,download" + body = Jason.encode!(%{"files" => [file("original", %{"quality" => "source"})]}) + + conn + |> Plug.Conn.put_resp_header("content-type", "application/vnd.vimeo.video+json;version=3.4") + |> Plug.Conn.put_resp_header("content-encoding", "gzip") + |> Plug.Conn.send_resp(200, :zlib.gzip(body)) + end) + + assert {:ok, "https://cdn.test/original.mp4"} = Vimeo.source(client(), "1") + end + + test "reads JSON objects even when the response is not labelled as JSON" do + for content_type <- [nil, "text/plain", "application/octet-stream"] do + Req.Test.expect(__MODULE__, fn conn -> + conn = + if content_type, + do: Plug.Conn.put_resp_header(conn, "content-type", content_type), + else: conn + + Plug.Conn.send_resp(conn, 200, Jason.encode!(%{"files" => [file("hd", %{})]})) + end) + + assert {:ok, "https://cdn.test/hd.mp4"} = Vimeo.source(client(), "1") + end + end + + test "reports invalid successful responses without exposing their contents" do + for body <- ["", "private-response", "{invalid", "[]", "null", "42"] do + Req.Test.expect(__MODULE__, fn conn -> + conn + |> Plug.Conn.put_resp_header("content-type", "application/json") + |> Plug.Conn.send_resp(200, body) + end) + + assert {:error, "Vimeo returned an invalid JSON object (HTTP 200)"} = + Vimeo.get(client(), "/videos/1") + end + end + + test "reports inaccessible videos without exposing responses or tokens" do + for fields <- [%{}, %{"files" => [], "download" => []}, %{"files" => nil, "download" => nil}] do + Req.Test.expect(__MODULE__, fn conn -> + assert "uri" in String.split(URI.decode_query(conn.query_string)["fields"], ",") + Req.Test.json(conn, Map.put(fields, "uri", "/videos/1")) + end) + + assert {:error, message} = Vimeo.source(client(), "1") + assert message =~ "no accessible video file" + end + + Req.Test.expect(__MODULE__, fn conn -> + Req.Test.json(Plug.Conn.put_status(conn, 403), %{"error" => "secret: vimeo-secret"}) + end) + + assert {:error, message} = Vimeo.get(client(), "/me") + refute message =~ "vimeo-secret" + end + + test "retries rate-limited read requests" do + Req.Test.expect(__MODULE__, fn conn -> Plug.Conn.send_resp(conn, 429, "") end) + Req.Test.expect(__MODULE__, fn conn -> Req.Test.json(conn, %{"uri" => "/users/1"}) end) + + request = Req.merge(client(), retry_delay: fn _ -> 0 end, retry_log_level: false) + + message = + ExUnit.CaptureIO.capture_io(:stderr, fn -> + assert {:ok, %{"uri" => "/users/1"}} = Vimeo.get(request, "/me") + end) + + assert message =~ "waiting 60s" + end + + defp file(name, attrs) do + Map.merge( + %{"link" => "https://cdn.test/#{name}.mp4", "type" => "video/mp4", "width" => 1920}, + attrs + ) + end + + defp client, do: Vimeo.new("vimeo-secret") |> Req.merge(plug: {Req.Test, __MODULE__}) +end From 9cf0f9e26f8f56e6c747dd3e628c864acca4eea8 Mon Sep 17 00:00:00 2001 From: David Date: Thu, 10 Sep 2026 10:03:20 +0200 Subject: [PATCH 2/2] Mark Vimeo importer as experimental and document testing limits --- README.md | 5 +++-- docs/usage.md | 21 ++++++++++++++++++++- lib/mave_cli/cli.ex | 9 +++++++-- lib/mave_cli/vimeo/import.ex | 1 + 4 files changed, 31 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index b7fe9cb..caf64b3 100644 --- a/README.md +++ b/README.md @@ -10,8 +10,9 @@ A command-line client for [Mave](https://www.mave.io/). Manage videos and collections, upload files, and use JSON output in your scripts. -Import videos from Vimeo with their titles and folder structure using -[`mave import vimeo`](docs/usage.md#import-from-vimeo). +The **experimental** [`mave import vimeo`](docs/usage.md#import-from-vimeo) +imports videos with their titles and folder structure. Full live import +verification is still pending. ## Installation diff --git a/docs/usage.md b/docs/usage.md index 8c960b2..65c31b7 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -83,6 +83,10 @@ Run `mave --help` to see all commands. JSON is the default output format; add `- ## Import from Vimeo +**Experimental:** automated tests cover the importer using simulated Vimeo and +Mave responses. A complete live import, including playable videos in Mave, has +not yet been verified. Start with `--dry-run` and a small test folder. + Import videos into the selected Mave space with their **titles and folder structure**, including subfolders and empty folders. The CLI sends video file links to Mave's existing import API; Mave fetches and processes the videos. @@ -106,7 +110,9 @@ imports. Signing in to Vimeo in your browser does not authenticate the CLI. See [Vimeo's token setup guide](https://help.vimeo.com/hc/en-us/articles/12427789081745-How-to-generate-a-personal-access-token). For automation, set `VIMEO_ACCESS_TOKEN` instead. The token needs the `public`, `private`, and -`video_files` scopes, plus a Vimeo plan that allows access to video file links. +`video_files` scopes. Transferring video files requires a Vimeo **Standard, +Advanced, Pro, Business, Premium, or Enterprise** plan; Free, Basic, Starter, +and Plus do not provide the file-link access used by this importer. See [Vimeo's download-link documentation](https://help.vimeo.com/hc/en-us/articles/12427806914577-About-video-file-download-links-from-the-API). The Vimeo token is used only for Vimeo API requests. Tokens entered at the prompt are saved atomically with `0600` permissions on Unix, alongside the Mave login in @@ -176,6 +182,19 @@ Unavailable videos are reported as `failed` while the remaining videos continue. The command exits with status `1` if any video failed. Progress goes to stderr, with results on stdout as JSON or `--format table`. +### Testing without a paid Vimeo plan + +Vimeo's [general API is available on free accounts](https://help.vimeo.com/hc/en-us/articles/12427702473105-API-technical-and-developer-prerequisites). +You can test login and `mave import vimeo --dry-run --format table` to check titles +and the folders your account can access. The preview does not request video file +links or submit videos, so a successful preview does not verify a full import. + +For complete verification, use a small folder in an eligible account, such as a +customer's test folder, and a Mave test space. Include two short videos and a +subfolder, import with `--folder ID --wait`, then check titles, nesting and playback. +Repeat with the same folder and `--resume --wait` to confirm videos are not duplicated. +The account owner can run these commands themselves without sharing their token. + ### Resume state The CLI saves Vimeo-to-Mave IDs under the same configuration directory as your diff --git a/lib/mave_cli/cli.ex b/lib/mave_cli/cli.ex index 8b22851..db04b2d 100644 --- a/lib/mave_cli/cli.ex +++ b/lib/mave_cli/cli.ex @@ -538,7 +538,7 @@ defmodule MaveCli.CLI do mave import PROVIDER [OPTIONS] Available importers: - vimeo Import videos with their titles and folder structure + vimeo Import videos with their titles and folder structure (experimental) Examples: mave import vimeo --dry-run --format table @@ -552,6 +552,9 @@ defmodule MaveCli.CLI do """ Import Vimeo videos with their titles and folder structure, including subfolders. + Experimental: a complete live import has not yet been verified. + Start with --dry-run and a small test folder. + Usage: mave import vimeo [OPTIONS] @@ -561,6 +564,8 @@ defmodule MaveCli.CLI do VIMEO_ACCESS_TOKEN overrides the saved Vimeo token without being stored. Create a token at https://developer.vimeo.com/apps with Public, Private and Video Files access (Authenticated (you)). + Video file imports require Vimeo Standard, Advanced, Pro, Business, Premium or Enterprise. + Free accounts can preview titles and accessible folders with --dry-run. Vimeo login: mave import vimeo login save a token before importing (optional) @@ -601,7 +606,7 @@ defmodule MaveCli.CLI do mave auth status | logout mave import list available importers mave import vimeo [--folder ID] [--collection ID] [--dry-run] [--resume] - [--state-file FILE] [--wait] [--timeout SECONDS] + [--state-file FILE] [--wait] [--timeout SECONDS] (experimental) mave import vimeo login | status | logout mave videos list [--page N] [--per-page N] [--collection ID] mave videos get ID diff --git a/lib/mave_cli/vimeo/import.ex b/lib/mave_cli/vimeo/import.ex index 9401e95..2528d67 100644 --- a/lib/mave_cli/vimeo/import.ex +++ b/lib/mave_cli/vimeo/import.ex @@ -12,6 +12,7 @@ defmodule MaveCli.Vimeo.Import do end def run_with_client(client, vimeo, opts \\ []) do + report(opts, "Experimental Vimeo importer: start with --dry-run and a small test folder.") report(opts, "Reading Vimeo videos and folders…") with :ok <- validate_options(opts),