TestQuality CLI

The TestQuality CLI is a robust, command-line tool designed to streamline your testing workflows with TestQuality projects. It helps you to programmatically interact with TestQuality organize your testing efforts and upload your automated test results from CI/CD pipelines or your local development environment

Prerequisites

Before you begin, ensure you have the following:

1. Node.js and npm: Required for the primary installation method.
2. A TestQuality Account: You'll need credentials to log in.
3. A Target Cycle in TestQuality: Create a Cycle in your TestQuality Project where you intend to upload results.
4. Automated Test Results: Your test automation framework should produce results in JUnit XML format.

Installation

Install the TestQuality CLI globally using npm (or your preferred Node.js package manager):

npm install -g @testquality/cli

After installation, you can run the CLI using the testquality command.

Alternative: Standalone Binary

If you prefer not to use npm or need a standalone executable, you can download compiled binaries. These are larger as they include the Node.js runtime.

Download

Download the binary from cli.testquality.com or directly from GitHub releases:

wget https://github.com/BitModern/testQualityCli/releases/download/{{ version }}/testquality-linux -O testquality

Replace {{ version }} with the actual release tag (e.g., v1.0.0) and adjust the filename based on your operating system:

• Windows: testquality-win.exe
• MacOS: testquality-macos
• Linux: testquality-linux
• Alpine Linux: testquality-alpine

Set Permissions

Once downloaded, grant execute permissions:

chmod 744 testquality

For easier access, consider moving the binary to a directory in your PATH (e.g., /usr/local/bin for Linux/MacOS).

Note for Alpine Linux: If using the standalone binary on Alpine Linux, you must install libstdc++:

apk add --no-cache libstdc++

Authentication

You need to authenticate with TestQuality to use the CLI.

1. Login with Email and Password

Use the login command with your TestQuality email and password.

testquality login your_email@example.com YourPassword

Save Credentials: To avoid logging in repeatedly, include the --save flag. This will store your authentication token locally for subsequent commands.

testquality login your_email@example.com YourPassword --save

2. Personal Access Token (PAT)

You can also authenticate using a Personal Access Token (PAT).

Option A: Environment Variable Set the TQ_ACCESS_TOKEN environment variable:

export TQ_ACCESS_TOKEN=your_personal_access_token

The CLI will automatically pick up this variable.

Option B: Command Line Argument Add the --access_token=<YOUR_PAT> argument to any command:

testquality upload_test_run 'path/to/*.xml' --project_name=MyProject --plan_name=MyPlan --access_token=your_personal_access_token

Core Usage: Uploading Test Results

This is the primary function of the CLI. Ensure your test results are in JUnit XML format.

Basic Upload

testquality upload_test_run 'sampleXml/*.xml' --project_name="Your Project Name" --plan_name="Your Cycle Name"

• Replace 'sampleXml/*.xml' with the glob pattern matching your JUnit XML files.
• Replace "Your Project Name" and "Your Cycle Name" with the actual names from your TestQuality instance.
• --project_name matches the exact name first, then falls back to a case-insensitive match. If that fallback matches more than one project (for example "Alpha" and "ALPHA"), the command stops and lists them; use --project_id instead.
• A single upload creates one run, so it is limited to 200 files and 32 MB in total, counting XML files and attachments together. Larger sets fail before anything is sent; narrow the glob or split them into separate runs.

Including Attachments

Attach files like screenshots or logs to your test results.

• From Test Name Tag: [[ATTACHMENT|ScreenshotFileName.png]]
  • Example: Test with UI screenshot [[ATTACHMENT|error_screenshot.png]]
• From Console Output: [[ATTACHMENT|path/to/your/file.log]]
  • Example: If your test prints [[ATTACHMENT|logs/test_run_details.log]] to console output.

Important: When using attachments, you must specify the root directory where these attachment files are located using the run_result_output_dir option:

testquality upload_test_run 'output/results/*.xml' --project_name="MyProject" --plan_name="MainCycle" --run_result_output_dir="output/attachments_and_logs/"

In this example, if a test tag mentions [[ATTACHMENT|error_screenshot.png]], the CLI will look for output/attachments_and_logs/error_screenshot.png. If console output mentions [[ATTACHMENT|logs/test_run_details.log]], it will look for output/attachments_and_logs/logs/test_run_details.log.

Linking Defects

You can link test results to defects in GitHub or Jira directly from your test reports. Modify your test names or include console output within your tests using the following formats:

• GitHub Defects: [[DEFECT|GH_ISSUE_NUMBER]]
  • Example: My Test Case [[DEFECT|22]]
• Jira Defects: [[DEFECT|JIRA_ISSUE_KEY]]
  • Example: Another Test Scenario [[DEFECT|TQ-123]]

The CLI will parse these tags and create the necessary links in TestQuality.

Gherkin / BDD Feature Files

Import Feature Files as Test Cases

Import Gherkin .feature files into a project. Each Feature becomes a folder and each Scenario becomes a test case.

testquality upload_feature 'features/**/*.feature' --project_id=1234

• Quote the glob so your shell doesn't expand it.
• Use --folder_id to import into a specific folder.
• Large sets of feature files are uploaded automatically in batches of up to 200 files and 32 MB per request (the server's per-request limits); a new batch starts when either limit would be exceeded. A single file larger than 32 MB cannot be uploaded and fails before anything is sent. Batches are sent one after another; if one fails, the command stops, reports which batch and files were not uploaded, and exits non-zero. Files in earlier batches have already been imported. Use --batch_size=<n> (1-200) to send smaller batches.

Keeping TestQuality in step with your repository (--write_tags, --sync)

If your .feature files in Git are the source of truth, a sync keeps one TestQuality folder in step with them: edited steps update the existing test, a renamed scenario keeps its test and history, and a deleted scenario is archived rather than destroyed. It needs the server that shipped with CLI 1.4.0.

1. Give every scenario its TestQuality key, once, locally. A tag such as @TC6865 above a scenario is its identity: rename the scenario, move it to another file or rename its Feature: and it is still the same test.

testquality upload_feature 'features/**/*.feature' --project_id=1234 --folder_id=5678 --write_tags --dry-run   # preview
testquality upload_feature 'features/**/*.feature' --project_id=1234 --folder_id=5678 --write_tags
git diff        # one @TC<key> line above each scenario that had none
git commit -am "Add TestQuality keys to scenarios"

• --write_tags is a real import: scenarios without a key become tests, and their new keys are written back. Run it again at any time; tagged scenarios are left alone, so it only adds what is missing.
• It writes nothing unless every batch was imported, only where the reported line is still that scenario, and never into a file that changed since it was read. Indentation and line endings are kept.
• It refuses to run under CI (CI set) and cannot be combined with --sync: tag locally, review, commit.
• --dry-run imports and lists file:line → @TC<key> without writing any file.

2. Sync from CI after every merge. --sync requires --folder_id: a folder that holds only these tests (not the project root).

testquality upload_feature 'features/**/*.feature' --project_id=1234 --folder_id=5678 --sync

• A sync spans every batch of the run. Scenarios no longer in the files are archived only once the server has every batch, the file count and the content digest: a failed or partial run archives nothing.
• Archiving moves a test into an Archived folder under the sync folder, with the removed-from-source label; its run history stays with it. Put the scenario back with its @TC tag and it is restored.
• Only folders the sync created or that hold only imported tests are managed. A test made by hand in TestQuality is never archived.
• If a sync would archive more than 50 tests, or more than 20% of the folder's tests (and more than 5), it is refused and lists what it would archive. That usually means the glob was narrowed by mistake; if the scenarios really were removed, run again with --force.
• --sync --dry-run reports what would be created, updated and archived, and changes nothing.
• A folder left empty (for example after a Feature: is renamed) is kept and reported, not deleted.

What happens to your tests:

Change in Git	Tagged @TC	Untagged
Step edited, inserted, reordered or removed	Updated in place; results stay attached	Same, while its name matches
Scenario renamed	Same test, renamed	New test; the old one is archived
File moved	No change	No change
Feature: renamed	Tests move to the new folder; the old folder is kept	New tests; the old ones are archived
Scenario deleted	Archived	Archived
Scenario restored	Moved back from Archived	New test

Notes:

• A step's recorded results stay attached when you edit its text, and show against the current text. A removed step loses its per-step results; the test's result for the run is kept.
• An outline may have only one Examples: table; a file with several is refused, naming the scenario.
• Rule: blocks are not supported. A file that cannot be parsed is named in the error.

Upload Feature Results

Upload Cucumber JSON results for your feature files:

testquality upload_feature_results 'reports/**/*.json' --project_name="MyProject" --plan_name="MainCycle"

Like upload_test_run, this creates one run per upload, so it is limited to 200 files and 32 MB in total, and is not batched.

Running From CI (GitHub Actions)

Sync feature files whenever changes are merged into main. --sync needs CLI 1.4.0 or later. Pin the version in npx rather than relying on whatever latest resolves to.

Run it only from the default branch, and give it a concurrency group so two merges never sync the same folder at once. The server refuses an overlapping sync with a 409; the group makes the second one wait instead.

on:
  push:
    branches: [main]
    paths: ['**/*.feature']

concurrency:
  group: tq-sync-${{ vars.TQ_FOLDER_ID }}
  cancel-in-progress: false

jobs:
  sync-features:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: >
          npx @testquality/cli@1.4.0 upload_feature 'features/**/*.feature'
          --project_id=${{ vars.TQ_PROJECT_ID }} --folder_id=${{ vars.TQ_FOLDER_ID }} --sync
        env:
          TQ_ACCESS_TOKEN: ${{ secrets.TQ_ACCESS_TOKEN }}

Other Common Commands

Upload CSV Files

For specific scenarios, you might need to upload results via CSV:

testquality upload_csv ./test_run_results.csv --cf ./test_run_results.config

(Ensure you have the corresponding config file for your CSV structure.)

Getting Help

To see a list of all available commands:

testquality --help

For help with a specific command (e.g., login):

testquality login --help

Advanced Usage

Restoring a Plan or Suite

This section is for recovering deleted items.

Restore a Plan:

1. Login (if you haven't already).
2. List deleted plans:

   testquality plans --revision_log -p _sort=-updated_at -p operation=delete

3. Restore the plan using its ID:

   testquality restore --plan_id <PLAN_ID>

Restore a Suite:

1. Login.
2. List deleted suites:

   testquality suites --revision_log -p _sort=-updated_at -p operation=delete

3. Find associated plans for the deleted suite:

   testquality plan_suite --revision_log -p _sort=-updated_at -p operation=delete -p suite_id=<SUITE_ID>

4. Restore the suite with its ID and an associated plan ID:

   testquality restore --suite_id <SUITE_ID> --plan_id <ASSOCIATED_PLAN_ID>

Using Custom Parameters

You can pass additional parameters to the API using the --params prefix:

testquality some_command --params.per_page -1 --params._with test

This translates to API parameters: {per_page: -1, _with: 'test'}

Contributing (For Developers)

If you want to contribute to the development of the TestQuality CLI itself:

Development Environment

The project uses Yarn for dependency management. You'll need Node.js and Yarn installed.

1. Clone the repository.
2. Install dependencies:

   yarn install

Running in Development Mode (with watch): This command will watch for file changes and automatically rebuild.

yarn dev

Running a Specific Command Directly (without watch): Use yarn start followed by the command and its arguments.

yarn start login <username> <password>

(Replace <username> and <password> or other command arguments as needed.)

Running the tests:

yarn test

Building for Production: This command builds the project for production (creates the binaries mentioned in the alternative installation).

yarn build