diff --git a/.github/agents/analyze-results.agent.md b/.github/agents/analyze-results.agent.md index c993fa0..088af8f 100644 --- a/.github/agents/analyze-results.agent.md +++ b/.github/agents/analyze-results.agent.md @@ -1,6 +1,6 @@ --- name: analyze-results -description: Analyzes crowdsourced subjective test results — runs result_parser.py for data cleaning, quality checks, and per-clip/per-worker MOS aggregation. +description: Analyzes crowdsourced subjective test results — runs result_parser.py for data cleaning, quality checks, and per-clip/per-worker MOS aggregation, and writes a re-runnable rerun_result_parser.bat that re-runs result_parser.py. --- # Analyze subjective test results @@ -45,6 +45,12 @@ Do not guess these values if they are missing: (AMT) or HIT App server. Contains worker responses. 4. **Prolific demographic CSV** (optional): `prolific_demographic_export_*.csv` — only needed if the study was run on Prolific via HIT App server. +5. **Payment per session** (Prolific only): the reward paid to a participant per + session/HIT (e.g. `2.10`). Prolific does not include the reward in its export, + so the parser cannot compute payment-per-hour statistics without it. Ask the + user for this value whenever the study was run on Prolific; pass it to the + parser as `--payment_per_session`. Not needed for AMT (the reward is already a + column in the AMT batch). ## Execution workflow @@ -54,6 +60,8 @@ Do not guess these values if they are missing: - The path to the project directory (where the `*_result_parser.cfg` is). - The test method used. - Whether they used Prolific or AMT. +- **If Prolific**: the payment per session (reward per HIT, e.g. `2.10`) — this is + not in the Prolific export and is required for payment-per-hour statistics. Then instruct: "Please download the answers file (`Batch_XXX.csv`) and, if using Prolific, the demographic export (`prolific_demographic_export_*.csv`) and place @@ -94,13 +102,17 @@ python REPO_ROOT\src\result_parser.py ` **With Prolific demographic data:** +Also pass `--payment_per_session` (the reward per HIT, e.g. `2.10`) so payment-per-hour +statistics are computed — Prolific does not include the reward in its export. + ```powershell Set-Location PROJECT_DIR python REPO_ROOT\src\result_parser.py ` --cfg RESULT_PARSER_CFG ` --method METHOD ` --answers Batch_XXX.csv ` - --prolific_answers prolific_demographic_export_XXX.csv + --prolific_answers prolific_demographic_export_XXX.csv ` + --payment_per_session 2.10 ``` **Notes:** @@ -109,6 +121,39 @@ python REPO_ROOT\src\result_parser.py ` - The working directory should be the project directory so output files are written there. +### 3b. Write a re-run batch file (rerun_result_parser.bat) + +Always write a `rerun_result_parser.bat` in the results directory so the requester can +re-run the analysis later (e.g. after a config change or with an updated answers export) +without re-deriving the command. It must reproduce the exact `result_parser.py` invocation +from step 3, using `%BASE%` (the folder the `.bat` lives in) for the input files and the +absolute repo path for the script. Include the `--prolific_answers` and +`--payment_per_session` lines only if Prolific was used. + +```bat +@echo off +REM Re-run the result parser for PROJECT (METHOD). +REM Rebuilds the data-cleaning report and per-clip / per-worker MOS outputs from the +REM batch answers CSV using the existing result-parser config. + +setlocal +set "BASE=%~dp0" +set "REPO=REPO_ROOT" + +cd /d "%BASE%" + +python "%REPO%\src\result_parser.py" ^ + --cfg "%BASE%RESULT_PARSER_CFG" ^ + --method METHOD ^ + --answers "%BASE%Batch_XXX.csv" ^ + --prolific_answers "%BASE%prolific_demographic_export_XXX.csv" ^ + --payment_per_session 2.10 + +endlocal +``` + +Save as `rerun_result_parser.bat` in the results directory. + ### 4. Analyze the output and summarize After the parser completes, provide a summary covering: @@ -124,22 +169,51 @@ Calculate: `rejection_percentage = XXXX / YYYY * 100` **⚠️ If rejection rate > 35%**: flag as alarming. Ask the user to investigate the rejection reasons in the data cleaning report. +**Rejection reason breakdown**: because a submission can fail several checks at +once, a flat per-reason count over-counts. Use the parser's "Rejection breakdown" +log and the two files it writes to report the marginal count per reason, how many +were rejected by a single reason ("only-this-reason", e.g. removed *only* because +of performance), and the most common reason combinations: +- `Batch_XXX_rejection_reason_matrix.csv` — reason co-occurrence matrix (diagonal / + `total` = submissions failing that reason; `only_this_reason` = failed that reason + alone; off-diagonal = failed both). +- `Batch_XXX_rejection_reason_combinations.csv` — each distinct reason combination + with its count and percentage. + #### 4b. Gold question performance -Read `detailed_gold_question_performance.csv` from the working directory. +Read `Batch_XXX_detailed_gold_question_performance.csv` (next to the answers file). + +The report has **one row per gold question** (so a P.804 submission with two gold +clips contributes two rows). Key columns: +- `gold_url` — the gold clip evaluated on that row (exactly one per row). +- `wrong` / `correct` — number of dimensions answered wrong / correct for that gold + on that submission; per-dimension detail is in `_wrong` (e.g. `sig_wrong`). +- `worker_id`, `HITID` — the submission the row belongs to. -- Look for columns matching `wrong*` — these indicate how many times each gold - clip received a wrong answer. -- Look for columns matching `url*` — these identify the gold clip URLs. -- **Any row where the sum of `wrong*` columns > 0** means that gold clip received - at least one wrong answer. -- Calculate the rejection rate per gold clip: - `gold_rejection_rate = wrong_count / total_times_shown * 100` +To assess gold clips, **group by `gold_url`**: +- A row counts as a failed presentation when `wrong > 0`. +- `gold_rejection_rate = (rows with wrong > 0 for that gold_url) / (total rows for that gold_url) * 100`. **⚠️ If any gold clip is rejected > 20% of the time**: flag as alarming. Ask the user to check that clip and verify the expected answer is correct. It may indicate a bad gold clip rather than bad workers. +For P.804, `Batch_XXX_gold_summary.csv` gives this directly: one row per gold clip with +`url`, `n_submission`, `max_wrong_pct` (worst scale wrong rate for that clip), and +per scale `_expected` (correct answer, blank if not targeted), +`_mean` (mean rating participants gave), and `_wrong_pct`. Sort by +`max_wrong_pct`; when the mean rating is far from the expected answer with a high +wrong rate, the gold clip is likely mis-keyed or too hard on that scale. + +**Correcting gold answers**: to override mis-keyed gold clips, build a CSV with a +`url` column plus, per scale, `` (correct answer) and optional `_var` +(variance), and pass it as `--gold_overrides`. A listed clip is authoritative: the +scales you fill are checked with those values and blank scales are skipped (not +checked); clips not listed keep the encoded answers. A good starting point per +scale is the rounded `_mean` for scales with a high `_wrong_pct`; +review before re-running. + #### 4c. Summary to present Provide the user with a structured summary: @@ -166,8 +240,11 @@ After analysis, direct the user to the key output files: | `Batch_XXX_votes_per_worker_[SCALE].csv` | Per-worker rating statistics | | `Batch_XXX_all_votes_per_clip.csv` | All individual votes per clip (key: `all_votes` in name) | | `Batch_XXX_data_cleaning_report.csv` | Detailed per-submission data cleaning report | -| `detailed_gold_question_performance.csv` | Per-gold-clip acceptance/rejection statistics | -| `Batch_XXX_quantity_bonus_report.csv` | Quantity bonus calculations | +| `Batch_XXX_rejection_reason_matrix.csv` | Reason co-occurrence matrix (total, only-this-reason, and pairwise counts) | +| `Batch_XXX_rejection_reason_combinations.csv` | Count/percentage of each distinct reason combination | +| `Batch_XXX_detailed_gold_question_performance.csv` | One row per gold question per submission (single `gold_url` per row) | +| `Batch_XXX_gold_summary.csv` | Per-gold-clip summary (P.804): `url`, `n_submission`, `max_wrong_pct`, and per scale `_expected` + `_mean` + `_wrong_pct` | +| `Batch_XXX_quantity_bonus_report.csv` | Quantity bonus calculations (MTurk only; not generated when the platform is Prolific) | **Scale suffixes by method:** diff --git a/.github/agents/create-study.agent.md b/.github/agents/create-study.agent.md index e1810af..4816681 100644 --- a/.github/agents/create-study.agent.md +++ b/.github/agents/create-study.agent.md @@ -1,6 +1,6 @@ --- name: create-study -description: Creates subjective speech quality tests using the P.808 toolkit — handles study setup, gold/trapping clip generation, storage upload, and project building for crowdsourcing platforms. +description: Creates subjective speech quality tests using the P.808 toolkit — handles study setup, gold/trapping clip generation, storage upload, project building for crowdsourcing platforms, prompts for study-specific screen-out completion codes (never reused across studies) and for the internal-vs-general master-script assets, and writes a re-runnable regenerate_study.bat that re-runs master_script.py. --- # Create subjective test instructions @@ -78,6 +78,11 @@ gold/trapping clips, training clips), **do not silently reuse them**. Instead: - If the user says they want to modify → ask about each item one by one, then proceed with the updated values. +**Never reuse the platform screen-out completion code** (`screenout_code`) from a prior +study or session. The crowd platform issues it per study, so a copied code is invalid. +Always **[ASK]** for the new study's code, or leave it unset to emit the HIT App Server +`${...}` placeholder — never silently carry it over. + ## Supported test methods | Method | `--method` flag | Gold clip generation | Trapping config | Template | @@ -132,6 +137,9 @@ Do not guess these values if they are missing: - Can they identify clean clips by a URL pattern (e.g. `*/clean/*`, `*/reference/*`)? - Would they like the agent to download a small subset of rating clips for the user to **listen to and manually remove any clips with distortion** before gold generation? + **Mandatory**: however the candidate clean clips are obtained, they **must** go through + the user review in section 5a (keep only clips the user would rate 5 on all scales) + before gold generation. This step is required and must not be skipped. **Important**: never use the sample clips bundled in this repository (`src\test_inputs\`). Source clips must come from the same dataset as the rating clips. - **Trapping clips**: if `trapping_clips.csv` is not provided, the quality of source @@ -142,8 +150,20 @@ Do not guess these values if they are missing: See "Storage and public accessibility" below for the full check procedure. 7. **Contact email**: the email address to show in the HIT app for worker inquiries. Do **not** use a hardcoded default — always ask. -8. **Max assignments per worker** (for Prolific) or worker requirements and payment (for AMT). -9. **Target valid votes per clip**: suggest publishing `target + BEST_PRACTICE_VALID_VOTE_BUFFER`. +8. **Screen-out completion code** (`screenout_code`): a study-specific code issued by the + crowd platform for paying screened-out participants. A single code covers both the + qualification and setup screen-out (platforms such as Prolific support only one code + per study). **Never copy it from a prior study** — always ask for the new study's code, + or leave it unset to emit the HIT App Server `${...}` placeholder. See the config template below. +9. **Max assignments per worker** (for Prolific) or worker requirements and payment (for AMT). +10. **Target valid votes per clip**: suggest publishing `target + BEST_PRACTICE_VALID_VOTE_BUFFER`. +11. **Master-script general assets** (`--general_assets`): the shared reference material for the + setup section — the math questions, comparison (CMP) pairs, hearing test and bandwidth-check + clips with their answers/hashes. **[ASK]** whether the requester will provide/point to the + **internal** assets (`src/assets_master_script/general_assets_internal.csv`, the larger + internal set) or use the default **general** set (`src/assets_master_script/general.csv`, + applied automatically when `--general_assets` is omitted). Pass `--general_assets` only when + using the internal set. ## Storage and public accessibility @@ -439,6 +459,31 @@ If downloading clips from Azure private storage, either: **How many source clips?** Use `BEST_PRACTICE_GOLD_SOURCE_COUNT` capped at `BEST_PRACTICE_MAX_GOLD_SOURCE_CLIPS`. +#### 5a. MANDATORY — user review of clean reference clips before generation + +The clean reference clips are the foundation of every gold question: the generator assumes +each source clip is **perfect** and would be rated **5 on all scales**. If a reference clip +has any audible flaw, every gold clip derived from it will have a wrong expected answer and +will wrongly fail workers. Therefore this review step is **required** and must not be skipped, +regardless of how the candidate clips were obtained. + +**You must:** + +1. Copy the candidate clean/reference clips into a temporary review directory, e.g. + `RATING_CLIPS_PATH\gold_source_review`. +2. **[ASK]** Instruct the user, in these exact terms, to review them: + - "Please listen to every clip in `gold_source_review`." + - "**Keep only the clips you would personally rate 5 on ALL scales** + (coloration, discontinuity, loudness, noise, reverb, signal, and overall)." + - "If a clip is not a perfect 5 on every scale, **delete that file** from the folder." + - "Do not edit the clips — only keep or delete them." +3. **Wait for the user to explicitly confirm** they have finished reviewing and deleting. + Do not proceed on assumption. +4. After confirmation, use **only the clips that remain** in the review directory as the + gold source (copy the survivors into `RATING_CLIPS_PATH\gold_source`). If the user + deleted everything, or too few remain, ask for more candidate clips and repeat — do not + fall back to unreviewed clips. + Generate gold clips (filenames are **anonymized** by default — do **not** use `--no_anonymize`): @@ -581,6 +626,7 @@ number_of_clips_per_session:10 number_of_trapping_per_session:1 number_of_gold_clips_per_session:GOLD_PER_SESSION clip_packing_strategy: random +platform: PLATFORM [hit_app_html] allowed_max_hit_in_project:COMPUTED_VALUE @@ -592,14 +638,51 @@ quantity_bonus: 0.1 quality_top_percentage: 20 quality_bonus: 0.15 contact_email:USER_PROVIDED_EMAIL +# Screen-out completion code, paid on the platform for the screening effort. A single +# code covers both the qualification and setup screen-out (platforms such as Prolific +# support only one code per study). It is STUDY-SPECIFIC: the crowd platform issues it +# per study, so never copy it from another study's cfg. Ask the requester for this study's +# code, or leave unset to keep the ${screenout_code} placeholder for the HIT App Server to +# fill; with no code the participant returns the task. +#screenout_code: XXXXXXXX +# Online (in-HIT) grading of the qualification / setup sections (both default true). +# Set to false to grade that section only offline in result_parser (answers not embedded, +# check button hidden). +#run_online_eval_qualification: true +#run_online_eval_setup: true +# Objective listening-device check (acoustic echo test right after qualification). +# required_playback_device: headset | loudspeaker | any (default headset; "any" skips the check). +# device_check_threshold_db: net-coupling dB above which a loudspeaker is inferred (default 20). +# device_check_probe_gain: probe-tone level 0..1, kept low for hearing safety (default 0.1). +#required_playback_device: headset +#device_check_threshold_db: 20 +#device_check_probe_gain: 0.1 ``` **Key rules:** - `number_of_gold_clips_per_session` = **2 for P.804**, 1 for others. - `bw_min` defaults to `FB`. Valid: `NB-WB`, `SWB`, `FB`. +- `screenout_code` (optional) = completion code a screened-out participant enters on the + platform to be paid for the screening effort (one code for both qualification and setup + screen-out). It is **study-specific — never copy it between studies**; ask for each new + study's code. If unset, the master script prints a warning and leaves a `${...}` placeholder + for the HIT App Server to fill; when no code is provided at all, the participant is asked to + return the task. +- `run_online_eval_qualification` / `run_online_eval_setup` (optional, default `true`) = grade the + qualification / setup section in the browser. Set to `false` to hide that section's answers and + "Check answers" button and grade it only offline in `result_parser`. +- `required_playback_device` (optional, default `headset`) = the device the task requires: + `headset`, `loudspeaker`, or `any`. An objective acoustic echo test verifies it right after + qualification (and confirms audibility via a beep count to catch a muted device). With `any` the + check is skipped. Tune `device_check_threshold_db` (default `20`) and `device_check_probe_gain` + (default `0.1`) only if browser testing shows detection issues or the tone is too loud. - `contact_email` = user-provided. Never hardcode. - `allowed_max_hit_in_project` = `BEST_PRACTICE_ALLOWED_MAX_HITS`. - `quantity_hits_more_than` ≈ `floor(total_sessions / 2)`, at least 2. +- `platform` = the crowd platform (`prolific` or `mturk`), from the requester's answer + (defaults to `prolific`). It is written into the generated `*_result_parser.cfg` so the + result parser knows the platform; for `prolific` no bonus report is generated (bonuses are + handled on Prolific). ### 8. Run the master script @@ -643,6 +726,10 @@ python REPO_ROOT\src\master_script.py ` Note: when `--training_gold_clips` is used, the `--training_clips` flag is **not** needed — training clips are embedded in the training gold CSV. +Add `--general_assets REPO_ROOT\src\assets_master_script\general_assets_internal.csv` when +the requester chose the internal assets (input 11). Omit the flag to use the default general +set (`assets_master_script/general.csv`). + **Notes:** - Use **full absolute paths** for all arguments to avoid path resolution issues. @@ -653,6 +740,45 @@ needed — training clips are embedded in the training gold CSV. - If `quantity_hits_more_than` triggers a warning, update the config file with the suggested value and re-run. +### 8b. Write a re-run batch file (regenerate_study.bat) + +Always write a `regenerate_study.bat` next to the input CSVs so the requester can rebuild +the study later (e.g. after a config tweak) without re-deriving the command. It must +reproduce the exact `master_script.py` invocation from step 8, using `%BASE%` (the folder +the `.bat` lives in) for all input paths and the absolute repo path for the script. + +Include only the flags that were actually used: `--training_gold_clips` (P.804/pp835) or +`--training_clips` (other methods), and `--general_assets` if an internal assets CSV was +passed. Omit `--check_urls` (URLs were already validated on first run); keep +`--create_local_test` so a preview is regenerated. + +```bat +@echo off +REM Regenerate the METHOD study PROJECT_NAME from existing input CSVs. +REM Clips (rating, gold, trapping) are already uploaded to public storage; this +REM only rebuilds the HIT app, publish batch, result-parser cfg, and preview. + +setlocal +set "BASE=%~dp0" +set "REPO=REPO_ROOT" +set "PROJECT=PROJECT_NAME" + +cd /d "%BASE%" + +python "%REPO%\src\master_script.py" ^ + --project %PROJECT% ^ + --method METHOD ^ + --cfg "%BASE%PROJECT_CONFIG.cfg" ^ + --clips "%BASE%rating_clips.csv" ^ + --gold_clips "%BASE%gold_clips.csv" ^ + --trapping_clips "%BASE%trapping_clips.csv" ^ + --create_local_test + +endlocal +``` + +Save as `regenerate_study.bat` next to the input CSVs and confirm it runs. + ### 9. Verify the generated project artifacts The output project directory (`PROJECT_NAME\`) should contain: @@ -703,10 +829,11 @@ remind the requester to run the azcopy commands before publishing the study. 1. The project directory with all three artifacts. 2. The config file used (saved next to input CSVs for future re-runs). -3. The azcopy commands for uploading generated clips (if applicable). -4. The method and scale used. -5. Any warnings or deviations from the documented flow. -6. Instructions for the requester to publish on their chosen platform: +3. The `regenerate_study.bat` (next to the input CSVs) for one-command re-runs. +4. The azcopy commands for uploading generated clips (if applicable). +5. The method and scale used. +6. Any warnings or deviations from the documented flow. +7. Instructions for the requester to publish on their chosen platform: - **Prolific**: follow the team's Prolific workflow or `docs\running_test_prolific.md`. - **AMT**: follow `docs\running_test_mturk.md`. diff --git a/.gitignore b/.gitignore index 0b3f9ae..ad6c95e 100644 --- a/.gitignore +++ b/.gitignore @@ -5,3 +5,6 @@ desktop.ini .idea/ .env .vscode + +# internal-only math assets (keep local, do not commit) +src/assets_master_script/general_assets_internal.csv \ No newline at end of file diff --git a/README.md b/README.md index ed3b6d8..d357e40 100644 --- a/README.md +++ b/README.md @@ -96,7 +96,7 @@ You can use an AI coding agent (e.g. GitHub Copilot, Claude) to create and run s automatically. The agent will generate gold clips, trapping clips, training clips, upload them to Azure storage, and build the complete project — all from a single prompt. -**Setup**: This experince is trilored to use Azure Storage, if you use any other cloud provider for serving your clips, adapt the code accordingly.Otherwise, make sure you have `az login` configured with write access to your Azure Blob +**Setup**: This experience is tailored to use Azure Storage; if you use any other cloud provider for serving your clips, adapt the code accordingly. Otherwise, make sure you have `az login` configured with write access to your Azure Blob Storage account. **Usage**: Open the repository in your IDE with an AI agent and ask it to create a study: diff --git a/docs/bandwidth_check_clips.md b/docs/bandwidth_check_clips.md new file mode 100644 index 0000000..5598dfd --- /dev/null +++ b/docs/bandwidth_check_clips.md @@ -0,0 +1,130 @@ +> **⚠️ Note: This document is AI-generated and has not been reviewed yet. After review, remove this comment and below LLM targeted message.** + + + +[Home](../README.md) > [Preparation](preparation.md) > Bandwidth Check Clips + +# Bandwidth Check Clips + +Bandwidth-check clips are the "same vs. different quality" discrimination items used in the +P.80x qualification test. Each clip plays two speech segments joined by a short beep +(`segment A | beep | segment B`) and the participant is asked whether the audio *after* the +beep sounds different from the audio *before* it. + +They screen the participant's playback chain and hearing: high-frequency band-limited noise is +added to the second half of some clips. A participant only perceives the difference if their +equipment and hearing reproduce that band, which lets the test discriminate wideband (WB), +super-wideband (SWB), and full-band (FB) capable setups. + +`create_bandwidth_check_clips.py` generates these clips from a set of clean, full-band +reference recordings. + +## Overview + +Both halves of every clip always come from the **same** source reference. The five cases per +reference are: + +| Case | Second half | Correct answer | +|------|-------------|----------------| +| q1 | reference + noise (3.5–22 kHz) | `dq` (different) | +| q2 | reference + noise (8.5–22 kHz) | `dq` (different) | +| q3 | reference + noise (15–22 kHz) | `dq` (different) | +| q4 | reference (no audible change) | `sq` (same) | +| q5 | reference (no audible change) | `sq` (same) | + +The band-limited noise is set about **+13 dB** above the reference active speech level (ITU-T +P.56), so it is clearly audible *within its band*. q1 uses the widest band and is the obvious +attention case; q3 (15–22 kHz) only reveals itself on full-band capable equipment. q4 and q5 +carry no audible change and are the obvious "same" trapping cases. These design answers match +the hosted production clips `d_g1_cmb.wav` … `d_g5_cmb.wav` referenced by `master_script.py` +and `P808Template/Qualification.html` (`dq, dq, dq, sq, sq`). + +### Inaudible dither + +Every segment also receives an independent, **inaudible** dither (around −75 dBov, far below +the speech but above the 16-bit LSB). This keeps the two halves — and the two "same" clips +(q4/q5) — from ever being bit-identical, so exact-match and deduplication detection cannot flag +them, while humans still perceive the two halves as identical. + +### Anonymized names + +Output clip file names are random UUIDs so the hosted names do not reveal the source reference. +The manifest CSV keeps the source-to-clip mapping. + +## Generating the clips + +```bash +cd src +python utils/create_bandwidth_check_clips.py ^ + --input_dir C:/datasets/p501 ^ + --output_dir output/bw_test ^ + --base_url https://audiosamplesp808.blob.core.windows.net/p808-assets/clips/bw-test +``` + +### Arguments + +| Argument | Required | Default | Description | +|----------|----------|---------|-------------| +| `--input_dir`, `-i` | Yes | — | Directory containing clean, full-band reference WAV files. | +| `--output_dir`, `-o` | Yes | — | Directory for the generated clips and the manifest CSV. | +| `--base_url` | No | — | Base URL where clips will be hosted. When set, the `q1`…`q5` CSV columns hold full URLs instead of bare file names. | +| `--noise_gain_db` | No | 13.0 | Band-noise level relative to the reference active speech level, in dB. | +| `--beep_freq` | No | 440.0 | Beep tone frequency in Hz. | +| `--beep_sec` | No | 1.0 | Beep tone duration in seconds. | +| `--gap_sec` | No | 0.2 | Silence gap on each side of the beep, in seconds. | +| `--seed` | No | — | Integer seed for reproducible noise. | +| `--limit` | No | — | Cap on the number of references processed. | +| `--no_anonymize` | No | False | Use descriptive `_q{n}.wav` names instead of random UUIDs (useful for listening review). | + +### Output + +The script writes, to `--output_dir`: + +- Five WAV clips per reference, named with random UUIDs (source sample rate and subtype + preserved). Pass `--no_anonymize` for descriptive `_q{n}.wav` names when reviewing. +- `bandwidth_check_clips.csv`, one row per reference, with columns `ref_clip`, `q1`…`q5` + (clip file name, or full URL when `--base_url` is given) and `ans_q1`…`ans_q5` + (the correct answer `dq`/`sq` for each case). + +References sampled below twice the lowest band edge (3.5 kHz) cannot carry the high-frequency +noise and are skipped with a warning; use full-band (48 kHz) sources. + +## Source clip recommendations + +- Use clean, full-band speech at 48 kHz (for example the ITU-T P.501 references in + `C:/datasets/p501`). +- Include a variety of speakers (male and female). +- Four to six references are enough to build a diverse qualification pool. + +## Using the clips in the qualification test + +The P.80x qualification page (`P808Template/Qualification.html`) references five bandwidth +clips as `comb_bw1` … `comb_bw5`, and `master_script.py` (`create_qualification_only`) lists +their correct answers. To use freshly generated clips: + +1. Upload the generated WAV files to the bandwidth-test container + (`p808-assets/clips/bw-test`, or your own public location). +2. Map `q1`…`q5` from `bandwidth_check_clips.csv` to `comb_bw1`…`comb_bw5`, keeping the + `ans_q1`…`ans_q5` answers (`dq, dq, dq, sq, sq`). +3. Validation is performed server-side against these answers; no correct answers are embedded + in the HTML. + +## Reproducibility + +- **Model:** Claude Opus 4.8 (model ID `claude-opus-4.8`) +- **Generated:** 2026-07-01, updated 2026-07-02 (UTC+02:00) +- **Generation parameters:** managed by the GitHub Copilot CLI and not exposed to the + assistant (no explicit temperature or max-token values were set by the author). +- **Context:** Authored alongside `src/utils/create_bandwidth_check_clips.py`, based on the + script's implementation, the legacy `speech_impairment_utility.py` design, and the + `comb_bw*` usage in `master_script.py` and `P808Template/Qualification.html`. +- **Regeneration prompt:** "Write `docs/bandwidth_check_clips.md` documenting + `src/utils/create_bandwidth_check_clips.py`: explain the bandwidth-check qualification clips + (`A | beep | B`), the five cases and their `dq`/`sq` answers, the band-limited noise + (3.5/8.5/15–22 kHz at +13 dB over active speech level), the same-source halves with inaudible + dither, the UUID-anonymized names, the CLI arguments, the manifest CSV columns, and how to + wire `q1`…`q5` to `comb_bw1`…`comb_bw5`. Follow the repo doc style (breadcrumb header, + argument table) and include the AI-generated disclaimer banner, hidden LLM watermark, and this + reproducibility section." diff --git a/docs/conf_master.md b/docs/conf_master.md index 404119d..00c6e84 100644 --- a/docs/conf_master.md +++ b/docs/conf_master.md @@ -3,6 +3,23 @@ # Configure for `master_script.py` This describes the configuration for the `master_script.py`. A sample configuration file can be found in [`configurations\master.cfg`](.\src\configurations\master.cfg). + +## Command-line arguments + +* `--project`: Name of the project (required). +* `--cfg`: Configuration file path (required). See sections below. +* `--method`: Test method — `acr`, `dcr`, `ccr`, `p835`, `pp835`, `p804`, or `echo_impairment_test` (required). +* `--clips`: CSV with rating clip URLs in column `rating_clips`. +* `--gold_clips`: CSV with gold clip URLs and answers. +* `--training_clips`: CSV with training clip URLs. +* `--trapping_clips`: CSV with trapping clip URLs and answers. +* `--training_gold_clips`: CSV with gold training question details (P.804). +* `--general_assets`: Path to the general assets CSV. Defaults to `assets_master_script/general.csv`. + Use `assets_master_script/general_assets_internal.csv` for projects with internally generated + math clips (see `utils/generate_math_questions.py`). +* `--check_urls`: Validate that all links in the CSV files are accessible. +* `--create_local_test`: Generate a local preview HTML file after the project is created. +* `--p831_fest`: Use the question set of P.831 for FEST. ## `[create_input]` @@ -41,6 +58,64 @@ eligible for quantity bonus). * `quality_bonus: 0.15`: The amount of the quality bonus per accepted assignment. * `bw_min: FB `: minimum bandwidth that participants playback should support, can be "NB-WB", "SWB", "FB" * `bw_max: FB `: maximum bandwidth that participants playback should support, can be "NB-WB", "SWB", "FB" +* (optional) `show_qualification: true`: Whether the in-HIT qualification section is shown (`true`/`false`). +Set to `false` when the qualification is run as a separate study (e.g. a Prolific screener). Default: `true`. +* (optional) `screenout_code:`: Study-level completion code shown to a participant who is screened out of the +in-HIT **qualification** (bandwidth/hearing) check or the **setup** (listening environment/attention) check on +their final attempt. They enter it on the crowdsourcing platform (e.g. Prolific) to be paid for their screening +effort. A single code is used for both screen-outs, because platforms such as Prolific support only one +screen-out completion code per study. + +If a screen-out code is not set, `master_script.py` prints a warning at generation time and leaves the +`${screenout_code}` placeholder in the generated HIT so the HIT App Server can fill it at run time. When neither +the configuration nor the server provides a code, the participant is asked to **return the task** instead of +being paid for the screening. +* (optional) `run_online_eval_qualification: true`: Whether the **qualification** section (bandwidth/hearing) +is graded in the browser (`true`/`false`). Default: `true`. +* (optional) `run_online_eval_setup: true`: Whether the **setup** section (listening environment/attention) +is graded in the browser (`true`/`false`). Default: `true`. + +When a `run_online_eval_*` flag is `false`, that section's correct answers are **not embedded** in the generated +HTML, its client-side check and its **"Check answers"** button are hidden, and the participant is not gated by +it — all grading for that section is then performed **offline** by `result_parser.py` from the submitted answers +(the bandwidth answers and the setup pair-comparison / math answers are always recorded, so the parser can grade +them regardless of the online setting). + +### Objective listening-device check + +A short, objective acoustic check verifies which playback device the participant is actually using. Right after +qualification, a **longer probe tone** is played through the active playback device while the microphone is +recorded (with browser echo cancellation / automatic gain control / noise suppression disabled). Strong +microphone-to-playback coupling means the sound is leaking into the room (a **loudspeaker**); weak coupling means +a **headset**. The decision uses `net_db`, the probe-band level rise minus a control-band rise, which isolates the +tonal echo from broadband microphone-level changes. When a headset is detected the probe is immediately followed +by a few **shorter beeps** and the participant is asked how many of the shorter beeps they heard — this confirms +audibility and catches a **muted or too-quiet** device (a muted device is otherwise indistinguishable from a +well-sealed headset). The check is **mandatory**: the participant cannot continue until the detected device +matches the requirement. No audio is recorded, stored, or transmitted — the measurement happens locally in the +browser and is discarded immediately. The measured values (`net_db`, `coupling_db`, `ref_db`, detected device and +the audio device names) are logged to the `webrtc_raw` field for offline analysis. + +* (optional) `required_playback_device:`: the device the task requires — `headset`, `loudspeaker`, or `any` +(accept either). Default: `headset`. The qualification self-report and the instruction rules are adapted to this +value. With `any` the device check adds no value, so it is **skipped** (the section is hidden and the following +sections are unlocked directly); a muted device is then caught by the setup section instead. +* (optional) `device_check_threshold_db:`: the `net_db` value at/above which a **loudspeaker** is inferred. +Default: `20`. Tune per your audio setup by browser testing (headset vs loudspeaker readings are printed to the +browser console). +* (optional) `device_check_probe_gain:`: probe-tone level (linear gain `0..1`). Default: `0.1`. Kept low for +hearing safety because the check runs before the volume-adjust step; detection is ratio-based, so a quiet tone +still works. Lower it if the tone is too loud, raise it if detection is unreliable. +* (optional) `device_check_dispute_max_net_db:`: gray-zone dispute policy. When the detected device does not +match the requirement, the participant may **dispute** the detection ("wrong — I am using the required device") +only when `net_db` is within this value of the threshold (a plausible false positive). A **confident** detection +(`net_db` at/beyond this) **cannot be disputed** — the participant must switch to the required device and retry, +or declare they do not have it (screen-out). Default: `30` (threshold `20` + a 10 dB band). Raise it to allow +disputes further from the threshold, lower it to enforce the requirement more strictly. +* (optional) `allow_cannot_rate:`: **P.804 only.** When `true`, a per-scale **"Cannot tell"** option (value +`0`) is shown on the Coloration, Discontinuity, Reverb and Signal-quality scales so raters can mark a dimension +they cannot assess. Default: `false`. These votes are **excluded from the per-scale MOS** by `result_parser.py` +and reported per clip as `cannot_rate_percentage`. ## `[acr_html]` or `[p835_html]` _deprecated_ diff --git a/docs/general_res.md b/docs/general_res.md index 6817bff..005ccd5 100644 --- a/docs/general_res.md +++ b/docs/general_res.md @@ -11,8 +11,13 @@ files: - `src/P808Template/P835_template.html` - `src/P808Template/Qualification.html` -1. Upload the links in the `src/assets_master_script/general.csv`: +1. Upload the links in the `src/assets_master_script/general.csv` (or `general_assets_internal.csv` + for internal assets): - Column `math` should contain URLs of files you find here `src/P808Template/assets/clips/math/*`. + You can generate additional math clips with `src/utils/generate_math_questions.py`. + - Column `math_ans` should contain the correct answer (sum) for each math clip. + - Column `math_hash` should contain a SHA-256 hash of the clip URL and answer for client-side + verification. The `generate_math_questions.py` script computes these automatically. - Columns `pair_a`, `pair_b` should contain URLs of files you find here `src/P808Template/assets/clips/environment_test/*`. Use files starting by `40` in `pair_a`, and corresponding file starting by `50` in `pair_b`. \ No newline at end of file diff --git a/docs/gold_clips.md b/docs/gold_clips.md index 9b55984..5d91dc9 100644 --- a/docs/gold_clips.md +++ b/docs/gold_clips.md @@ -79,8 +79,8 @@ degradation, so `bak` is set to 4 rather than 5 for that type. ### P.804 -For P.804, gold clips target multiple quality dimensions. Only dimensions with an expected answer -of 1 are listed in the CSV; empty cells mean the dimension is not targeted (implicitly 5). +For P.804, gold clips target multiple quality dimensions. Only targeted dimensions are listed in the CSV +(with their expected answer); empty cells mean the dimension is not targeted (implicitly 5). | Type | Description | col | disc | loud | noise | sig | ovrl | |------|-------------|-----|------|------|-------|-----|------| @@ -88,16 +88,39 @@ of 1 are listed in the CSV; empty cells mean the dimension is not targeted (impl | Background noise | Pink noise | | | | 1 | | 1 | | Signal distortion | Hard clipping | | | | | 1 | 1 | | Discontinuity | Random segment dropouts (choppy) | | 1 | | | 1 | 1 | -| Discontinuity + noise | Choppy + noise | | | | 1 | 1 | 1 | -| Coloration | Resonant/muffled/telephone filter | 1 | | | | 1 | 1 | -| Coloration + noise | Coloration + noise | | | | 1 | 1 | 1 | -| Distortion + noise | Clipping + noise | | | | 1 | 1 | 1 | -| Loudness | Too loud (+25 dB) or too quiet (-25 dB) | | | 1 | | | 1 | -| Loudness + distortion | Loudness + clipping | | | 1 | | 1 | 1 | -| Loudness + noise | Loudness + noise | | | 1 | 1 | | 1 | - -**Note:** When distortion is combined with noise, only `sig` and `noise` are flagged because the -specific type of underlying distortion is not clearly distinguishable to raters. +| Discontinuity + noise | Choppy + noise | | | | 1 | | 1 | +| Coloration | Resonant/muffled/telephone filter | 1 | | | | 2 | 2 | +| Coloration + noise | Coloration + noise | | | | 1 | | 1 | +| Distortion + noise | Clipping + noise | | | | 1 | | 1 | +| Loudness (too loud) | Gain raised so speech is too loud | | | 1 | | | | +| Loudness (too quiet) | Gain lowered so speech is too quiet | | | 1 | | | | +| Loudness (too loud) + distortion | Too loud + clipping | | | 1 | | 1 | 1 | +| Loudness (too quiet) + distortion | Too quiet + clipping | | | 1 | | | 1 | +| Loudness (too loud) + noise | Too loud + noise | | | 1 | 1 | | 1 | +| Loudness (too quiet) + noise | Too quiet + noise | | | 1 | 1 | | 1 | + +**Note:** The `sig` dimension cannot be judged reliably when the signal is masked or hidden, so it is not +flagged for: the noise-combined types (discontinuity/coloration/distortion + noise, where noise masks the +signal) and the **too-quiet** loudness + distortion case (a very low-gain clip hides signal detail). The +**too-loud** loudness + distortion case still flags `sig`, since the distortion remains audible. In all +these cases the `ovrl` flag is retained, driven by the noise or loudness/distortion degradation. + +**Note:** Loudness alone does not flag `ovrl`. A level offset by itself is not treated as an overall +quality degradation, so only the `loud` dimension is targeted. When loudness is combined with another +artifact (distortion or noise), that other artifact drives the `ovrl` flag. + +**Note:** For the standalone **Coloration** type the `sig` and `ovrl` answers default to **2** (not 1): +listeners tend to perceive coloration as milder on signal quality and overall than a "1". These are +still suggested defaults — **listen to each colored clip and confirm/adjust** the answers, since some +clips' coloration is subtle enough that even `col=1` may not hold. You can correct per-clip gold answers +after collection with `result_parser.py --gold_overrides` (a CSV of `url` plus per-scale answer/variance; +a listed clip is authoritative, and a blank scale for it is skipped). + +**Note:** The loudness and coloration degradations keep the first few seconds of audio as a clean +reference (`GOLD_CLEAN_PREFIX_SEC`, default 2 seconds) and only apply the degradation afterwards, so a +rater can perceive the change relative to the clean start. Other artifacts (noise, distortion, +discontinuity) are applied from the beginning of the clip; when combined with loudness or coloration, +the other artifact is present throughout while the loudness/coloration change appears after the prefix. **Output CSV columns:** `gold_clips`, `col_ans`, `disc_ans`, `loud_ans`, `noise_ans`, `reverb_ans`, `sig_ans`, `ovrl_ans` diff --git a/docs/prep_acr.md b/docs/prep_acr.md index 7cb5b30..741c9c7 100644 --- a/docs/prep_acr.md +++ b/docs/prep_acr.md @@ -81,6 +81,7 @@ a column named `trapping_clips` and expected answer to each clip in a column nam Optionally: - Add `--check_urls` to validate that all links in the CSV files are accessible before creating the project. - Add `--create_local_test` to generate a local preview HTML file for testing. See [preview_html](preview_html.md) for details. + - Add `--general_assets path/to/general.csv` to use a custom general assets CSV instead of the default `assets_master_script/general.csv`. Note: file paths are expected to be relative to the current working directory. diff --git a/docs/prep_dcr_ccr.md b/docs/prep_dcr_ccr.md index 29ca2a1..8974d31 100644 --- a/docs/prep_dcr_ccr.md +++ b/docs/prep_dcr_ccr.md @@ -47,6 +47,7 @@ column named `training_clips` and URLs to corresponding reference clips in colum Optionally: - Add `--check_urls` to validate that all links in the CSV files are accessible before creating the project. - Add `--create_local_test` to generate a local preview HTML file for testing. See [preview_html](preview_html.md) for details. + - Add `--general_assets path/to/general.csv` to use a custom general assets CSV instead of the default `assets_master_script/general.csv`. Note: file paths are expected to be relative to the current working directory. diff --git a/docs/prep_p804.md b/docs/prep_p804.md index ce7f2f9..45b8e84 100644 --- a/docs/prep_p804.md +++ b/docs/prep_p804.md @@ -26,7 +26,7 @@ column named `training_clips` (see [training_gold_clips.csv](../src/test_inputs/ They should approximately cover the range from worst to best quality to be expected in the test. In P.804, it is possible to add the correct answer, variance, and a message to be shown if the given answer is out of expected range per dimension. -1. Upload your **gold standard clips** in a cloud server and create `gold_clips_p804.csv` file which contains all URLs in a +1. Upload your **gold standard clips** in a cloud server and create `gold_clips.csv` file which contains all URLs in a column named `gold_url` and expected answer to each clip/dimension in corresponding column e.g. `col_ans`, `disc_ans`, etc. There should be a column name `ver` with value either 1 or 2. Each session will include two gold clips, one from each version. (see [gold_clips_p804.csv](../src/test_inputs/gold_clips.csv) as an example). When a correct answer for a dimension is not provided, @@ -79,13 +79,14 @@ a column named `trapping_clips` and expected answer to each clip in a column nam --method p804 ^ --cfg your_configuration_file.cfg ^ --clips rating_clips.csv ^ - --training_gold_clips training_clips.csv ^ + --training_gold_clips training_gold_clips.csv ^ --gold_clips gold_clips.csv ^ --trapping_clips trapping_clips.csv ``` Optionally: - Add `--check_urls` to validate that all links in the CSV files are accessible before creating the project. - Add `--create_local_test` to generate a local preview HTML file for testing. See [preview_html](preview_html.md) for details. + - Add `--general_assets path/to/general.csv` to use a custom general assets CSV instead of the default `assets_master_script/general.csv`. Note: file paths are expected to be relative to the current working directory. @@ -93,6 +94,6 @@ a column named `trapping_clips` and expected answer to each clip in a column nam directory which contains: * `YOUR_PROJECT_NAME_p804.html`: Customized HIT app to be used in Amazon Mechanical Turk (AMT). * `YOUR_PROJECT_NAME_publish_batch.csv`: List of dynamic content to be used during publishing batch in AMT. - * `YOUR_PROJECT_NAME_acr_result_parser.cfg`: Customized configuration file to be used by `result_parser.py` script + * `YOUR_PROJECT_NAME_p804_result_parser.cfg`: Customized configuration file to be used by `result_parser.py` script Now, you are ready for running the test on [Prolific](running_test_prolific.md) or [Amazon Mechanical Turk](running_test_mturk.md). \ No newline at end of file diff --git a/docs/prep_p835.md b/docs/prep_p835.md index 267f9f6..5f17763 100644 --- a/docs/prep_p835.md +++ b/docs/prep_p835.md @@ -86,6 +86,7 @@ a column named `trapping_clips` and expected answer to each clip in a column nam Optionally: - Add `--check_urls` to validate that all links in the CSV files are accessible before creating the project. - Add `--create_local_test` to generate a local preview HTML file for testing. See [preview_html](preview_html.md) for details. + - Add `--general_assets path/to/general.csv` to use a custom general assets CSV instead of the default `assets_master_script/general.csv`. Note: file paths are expected to be relative to the current working directory. @@ -93,6 +94,6 @@ a column named `trapping_clips` and expected answer to each clip in a column nam directory which contains: * `YOUR_PROJECT_NAME_p835.html`: Customized HIT app to be used in Amazon Mechanical Turk (AMT). * `YOUR_PROJECT_NAME_publish_batch.csv`: List of dynamic content to be used during publishing batch in AMT. - * `YOUR_PROJECT_NAME_acr_result_parser.cfg`: Customized configuration file to be used by `result_parser.py` script + * `YOUR_PROJECT_NAME_p835_result_parser.cfg`: Customized configuration file to be used by `result_parser.py` script Now, you are ready for running the test on [Prolific](running_test_prolific.md) or [Amazon Mechanical Turk](running_test_mturk.md). \ No newline at end of file diff --git a/docs/preparation.md b/docs/preparation.md index 604c513..0d3b059 100644 --- a/docs/preparation.md +++ b/docs/preparation.md @@ -35,4 +35,6 @@ URLs associated to them as described in [General Resources](general_res.md) ## Utility Scripts - [Gold Standard Clips](gold_clips.md) — Generate gold clips for quality control. +- [Bandwidth Check Clips](bandwidth_check_clips.md) — Generate the bandwidth-discrimination clips for the P.80x qualification test. +- [Setup (JND) Clips](setup_jnd_clips.md) — Generate the just-noticeable-difference A/B pairs for the P.80x setup/environment check. - [Upload Clips to Storage](upload_clips.md) — Upload local clips or copy from private to public Azure storage. diff --git a/docs/results.md b/docs/results.md index 9290a82..1d78ec4 100644 --- a/docs/results.md +++ b/docs/results.md @@ -35,22 +35,34 @@ created in the first step ([preparation](preparation.md)). --quality_bonus ``` * `--cfg` use the configuration file generated for your project in the [preparation](preparation.md) step here (i.e.`YOUR_PROJECT_NAME_ccr_result_parser.cfg`). - * `--method` could be either `acr`, `dcr`, `ccr`, `p835`, `pp835` or `p804`. + * `--method` could be either `acr`, `dcr`, `ccr`, `p835`, `pp835`, `p804`, or `echo_impairment_test`. * if using Prolific, provide the csv file you downloaded from that platform as `--prolific_answers`. The answer you downloaded from HITAppServer should be provided as `--answers` * `--quantity_bonus` could be `all`, or `submitted`. It specify which assignments should be considered when calculating the amount of quantity bonus (everything i.e. `all` or just the assignments with status submitted i.e. `submitted`). Besides the console outputs, following files will be generated in the same directory as the `--answers` file is located in. All file names will start with the `--answers` file name. - * `[downloaded_batch_result]_data_cleaning_report`: Data cleansing report. Each line refers to one line in answer file. + * `[downloaded_batch_result]_data_cleaning_report`: Data cleansing report. Each line refers to one line in answer file. + It also includes the objective listening-device check log parsed from the `webrtc_raw` field: `device_detected`, + `device_required`, `device_net_db`, `device_coupling_db`, `device_ref_db`, `device_audible`, `device_user_choice`, + `device_names`, and `device_check_flag` (set to 1 for submissions worth a manual look — a disputed detection, an + audibility failure, or a detected device that does not match the requirement). + * `[downloaded_batch_result]_device_check_summary.csv`: Aggregate summary of the objective listening-device check + (counts of detected/required devices, disputes, no-device and audibility failures, how many were flagged, and the + `net_db` distribution). Useful for reviewing edge cases and calibrating `device_check_threshold_db` at scale. It is + only written when the answers contain the objective-check log (older studies are skipped). * `[downloaded_batch_result]_accept_reject_gui.csv`: A report to be used for approving and rejecting assignments. One line for each assignment which has a status of "submitted". * `[downloaded_batch_result]_votes_per_clip.csv`: Aggregated result per clip, including MOS, standard deviations, and 95% Confidence Intervals. * `[downloaded_batch_result]_votes_per_cond.csv`: Aggregated result per condition. * `[downloaded_batch_result]_votes_per_worker.csv`: Long format of rating per clip, includes: HITId, workerid, file, vote and condition. - * `[downloaded_batch_result]_quantity_bonus_report.csv`: List of workers who are eligible for quantity bonus with the amount of bonus (to be used with the mturk_utils.py). - * `[downloaded_batch_result]_quality_bonus_report.csv`: List of workers who are eligible for quality bonus with the amount of bonus (to be used with the mturk_utils.py). + * `[downloaded_batch_result]_quantity_bonus_report.csv`: List of workers who are eligible for quantity bonus with the amount of bonus (to be used with the utils/mturk_utils.py). + * `[downloaded_batch_result]_quality_bonus_report.csv`: List of workers who are eligible for quality bonus with the amount of bonus (to be used with the utils/mturk_utils.py). * `[downloaded_batch_result]_extending.csv`: List of HITIds with number of assignment per each which are needed to reach a specific number of votes per clip. + * `[downloaded_batch_result]_gold_summary.csv`: Per gold clip, the expected answer, mean rating and wrong-vote percentage for each scale, plus `max_wrong_pct`. Useful for spotting mis-set gold answers. + * `[downloaded_batch_result]_detailed_gold_question_performance.csv`: One row per (submission, gold clip) with the per-scale given answer and wrong flags. + * `[downloaded_batch_result]_rejection_reason_matrix.csv` and `[downloaded_batch_result]_rejection_reason_combinations.csv`: Breakdown of why submissions were rejected (per reason, and per reason-combination). + * `[downloaded_batch_result]_screened_out.csv` (Prolific only): Participants Prolific marked as SCREENED OUT (failed the in-HIT screening). They are reported separately and excluded from rejection and blocking. @@ -69,6 +81,10 @@ created in the first step ([preparation](preparation.md)). * In addition a summary in the condition level will be provided for all three scales in `[downloaded_batch_result]_votes_per_cond_all`. + Note for **P804** method: + * For each scale (`noise`, `col`, `loud`, `disc`, `reverb`, `sig`, `ovrl`) an aggregated `[downloaded_batch_result]_votes_per_clip_[scale].csv` is written (MOS, std, 95% CI), plus a combined `[downloaded_batch_result]_votes_per_clip_all-scales.csv`. + * Each per-clip file also reports `is_silent_percentage` (share of ratings where the clip was marked silent, next to `n`) and, for the scales that offer the **"Cannot tell"** option, `cannot_rate_percentage` (share of ratings where the rater could not assess that scale, next to the scale's MOS). Silent and "Cannot tell" votes are excluded from the MOS. + ## Approve/Reject submissions - Prolific - Get your API token from the Prolific website and add it to your [Prolific config file](../src/configurations/prolific.cfg). @@ -76,7 +92,7 @@ created in the first step ([preparation](preparation.md)). ``` bash cd src - python prolific_utils.py ^ + python utils/prolific_utils.py ^ --cfg your_prolific_configuration_file.cfg ^ --review [path to you project's root directory] ^ ``` @@ -98,7 +114,7 @@ submission. ```bash cd src - python mturk_utils.py ^ + python utils/mturk_utils.py ^ --cfg mturk.cfg ^ --approve_reject [downloaded_batch_result]_accept_reject_gui.csv ``` @@ -111,7 +127,7 @@ submission. ```bash cd src - python mturk_utils.py ^ + python utils/mturk_utils.py ^ --cfg mturk.cfg ^ --send_bonus [downloaded_batch_result]_*_bonus_report.csv ``` @@ -124,7 +140,7 @@ submission. ```bash cd src - python mturk_utils.py ^ + python utils/mturk_utils.py ^ --cfg mturk.cfg ^ --extend_hits [downloaded_batch_result]_extending.csv ``` @@ -138,7 +154,7 @@ submission. ```bash cd src - python mturk_utils.py ^ + python utils/mturk_utils.py ^ --cfg mturk.cfg ^ --extended_hits_status [downloaded_batch_result]_extending.csv ``` diff --git a/docs/running_test_mturk.md b/docs/running_test_mturk.md index 7cdb2c3..7139b08 100644 --- a/docs/running_test_mturk.md +++ b/docs/running_test_mturk.md @@ -68,7 +68,7 @@ all actions (like download results, approve/reject assignments, etc.) should be ```bash cd src - python mturk_utils.py ^ + python utils/mturk_utils.py ^ --cfg mturk.cfg ^ --create_hit create_hit.cfg ^ --create_hit_input YOUR_PROJECT_NAME_publish_batch.csv @@ -80,7 +80,7 @@ all actions (like download results, approve/reject assignments, etc.) should be 1. Later, download the results ```bash cd src - python mturk_utils.py ^ + python utils/mturk_utils.py ^ --cfg mturk.cfg ^ --answers Batch_123_456.csv ``` diff --git a/docs/setup_jnd_clips.md b/docs/setup_jnd_clips.md new file mode 100644 index 0000000..86bb6c5 --- /dev/null +++ b/docs/setup_jnd_clips.md @@ -0,0 +1,161 @@ +> **⚠️ Note: This document is AI-generated and has not been reviewed yet. After review, remove this comment and below LLM targeted message.** + + + +[Home](../README.md) > [Preparation](preparation.md) > Setup (JND) Clips + +# Setup (JND) Clips + +Setup clips are the "just-noticeable-difference" (JND) A/B comparison items used in the +**setup / environment** section of the P.80x HIT. Each item plays two versions of the *same* +clean speech clip — one cleaner (higher SNR) and one noisier (lower SNR) — and asks the +participant which sample has the better quality. The cleaner (higher-SNR) clip is the correct +answer. + +Because the two clips differ *only* in the level of added white noise, a participant can hear +the difference only in a quiet environment with a proper headset. This is what the setup +section screens, alongside the math attention question. + +`create_jnd_check_clips.py` generates these pairs from a set of clean reference recordings. It +mirrors the existing hosted pairs (`sample_jnd/40S_*.wav` vs `50S_*.wav`) that `master_script.py` +reads from the `pair_a` / `pair_b` columns of the general assets CSV. + +## Overview + +Each reference produces one A/B pair: + +| Slot | Content | Correct answer | +|------|---------|----------------| +| cleaner clip | reference + white noise at `--snr_high` (default 50 dB) | this one (better quality) | +| noisier clip | reference + white noise at `--snr_low` (default 40 dB) | — | + +The default 50 dB vs 40 dB gap matches the current production pairs. A **smaller** gap (e.g. +`--snr_high 50 --snr_low 45`) makes the pair harder to tell apart, i.e. a more demanding setup +check. + +### White-noise mixing + +The speech is normalized to **−25 dBFS** and white noise is scaled to reach the requested SNR +(the MS-SNSD `snr_mixer` convention, matching the legacy `create_jnd_dataset.py`). Both clips in +a pair use the *same* clean speech and the *same* noise realization, scaled to the two levels, +so the pair is a pure SNR difference. + +### Anonymized names and explicit answers + +Unlike the legacy `{SNR}S_.wav` naming (where the 2-digit SNR prefix leaks the answer in +the URL), output clip file names are random UUIDs. The correct answer is recorded **explicitly** +in the manifest (`ans_pair` / `ans_url`) and the A/B order is randomized, so the hosted file +names reveal nothing about which clip is correct. + +## Building source clips with silence gaps (optional) + +A silent gap between the two sentences of a clip exposes the noise floor: once white noise is +added, the listener hears the faint hiss in the quiet gap, which makes the small SNR difference +much easier to notice on a good setup. `merge_jnd_sources.py` builds such source clips from +pre-split segments (`_p1.wav`, `_p2.wav`, …), joining two segments with a silent +lead-in, a silent gap, and a silent tail: + +``` +[pre silence] segment_i [mid silence] segment_j [post silence] +``` + +```bash +cd src +python utils/merge_jnd_sources.py ^ + --input_dir C:/datasets/clean_speech/split ^ + --output_dir C:/datasets/clean_speech/jnd_sources ^ + --pre_sec 1.5 --mid_sec 1.5 --post_sec 0.5 +``` + +By default every ordered pair of distinct segments is produced (`1+2`, `2+1`, `1+3`, …), named +`_p.wav`, so a source with three segments yields six merged clips. Use `--sources`, +`--combos`, and `--limit` to build a subset while tuning the silence lengths. Feed the resulting +directory to `create_jnd_check_clips.py` as its `--input_dir`. A pre/mid/post of +**1.5 / 1.5 / 0.5 s** was found to work well. + +## Generating the clips + +```bash +cd src +python utils/create_jnd_check_clips.py ^ + --input_dir C:/datasets/clean_speech ^ + --output_dir output/jnd_test ^ + --base_url https://audiosamplesp808.blob.core.windows.net/p808-assets/clips/jnd-test ^ + --snr_high 50 --snr_low 40 +``` + +### Arguments + +| Argument | Required | Default | Description | +|----------|----------|---------|-------------| +| `--input_dir`, `-i` | Yes | — | Directory containing clean reference WAV files. | +| `--output_dir`, `-o` | Yes | — | Directory for the generated clips and the manifest CSV. | +| `--base_url` | No | — | Base URL where clips will be hosted. When set, the pair/answer CSV columns hold full URLs instead of bare file names. | +| `--snr_high` | No | 50.0 | SNR in dB of the cleaner (correct) clip. | +| `--snr_low` | No | 40.0 | SNR in dB of the noisier clip. A smaller gap to `--snr_high` makes the pair harder. | +| `--seed` | No | — | Integer seed for reproducible noise and A/B order. | +| `--limit` | No | — | Cap on the number of references processed. | +| `--no_anonymize` | No | False | Use descriptive `_snr{snr}.wav` names instead of random UUIDs (useful for listening review). | + +### Output + +The script writes, to `--output_dir`: + +- Two WAV clips per reference, named with random UUIDs (source sample rate and subtype + preserved). Pass `--no_anonymize` for descriptive `_snr{snr}.wav` names when reviewing. +- `jnd_check_clips.csv`, one row per reference, with columns: + - `ref_clip` — the source file name. + - `pair_a`, `pair_b` — the two clip file names (or full URLs when `--base_url` is given). + - `ans_pair` — which slot is correct, `a` or `b` (the cleaner clip). + - `ans_url` — the correct clip file name / URL (convenient for hashing the answer). + - `snr_a`, `snr_b` — the SNR in dB of each slot, for listening review. + +## Source clip recommendations + +- Use clean speech clips (48 kHz recommended, matching the current sources). +- Include a variety of speakers (male and female). +- A few well-chosen references are enough; the setup section shows four pairs per session, drawn + from the pool. + +## Using the clips in the setup test + +The setup section shows four A/B pairs as `CMP1` … `CMP4`, filled per session from the +`pair_a` / `pair_b` columns of the general assets CSV. `master_script.py` +(`prepare_basic_cfg`) SHA-256-hashes the correct clip's URL into `cmp_correct_answers`, and the +client verifies a selected clip by hashing its URL — no plain answer is embedded in the page. + +To use freshly generated clips: + +1. Upload the generated WAV files to a public location (for example + `p808-assets/clips/jnd-test`). +2. Add the `pair_a` / `pair_b` URLs and the correct answer (`ans_url`, as a `pair_ans` column) to + the internal general resource (`src/assets_master_script/general_assets_internal.csv`). +3. `master_script.py` reads `pair_ans` to hash the correct clip into `cmp_correct_answers` (online + check), and `create_input.py` records the correct slot per shown pair as `ans_cmp1`…`ans_cmp4` + in the publish batch so `result_parser.py` can grade the setup **offline** too (needed when + `run_online_eval_setup` is `false`). + +> **Note:** the legacy pairs encode the correct answer in the file name (higher SNR = cleaner), +> which `master_script.py` and `result_parser.py` read via the 2-digit SNR prefix. Anonymized clips +> carry the answer explicitly (`pair_ans` / `ans_cmp`) instead, so wiring them into the general +> resource uses the explicit answer rather than the file name. + +## Reproducibility + +- **Model:** Claude Opus 4.8 (model ID `claude-opus-4.8`) +- **Generated:** 2026-07-10 (UTC+02:00) +- **Generation parameters:** managed by the GitHub Copilot CLI and not exposed to the + assistant (no explicit temperature or max-token values were set by the author). +- **Context:** Authored alongside `src/utils/create_jnd_check_clips.py`, based on the script's + implementation, the legacy `src/environment test/script/create_jnd_dataset.py` design, and the + `pair_a` / `pair_b` → `cmp_correct_answers` usage in `master_script.py`. +- **Regeneration prompt:** "Write `docs/setup_jnd_clips.md` documenting + `src/utils/create_jnd_check_clips.py`: explain the setup-section JND A/B pairs (same clip mixed + with white noise at two SNR levels, cleaner = correct), the −25 dBFS / SNR mixing convention, + the default 50/40 dB pair and how the gap sets difficulty, the UUID-anonymized names with the + correct answer recorded in the manifest, the CLI arguments, the manifest CSV columns, and how + to wire `pair_a` / `pair_b` into the general resource for the hashed `cmp_correct_answers` + check. Follow the repo doc style (breadcrumb header, argument table) and include the + AI-generated disclaimer banner, hidden LLM watermark, and this reproducibility section." diff --git a/src/P808Template/ACR_template.html b/src/P808Template/ACR_template.html index c3a7734..c09ccac 100644 --- a/src/P808Template/ACR_template.html +++ b/src/P808Template/ACR_template.html @@ -47,6 +47,7 @@ font-size:0.9em; } fieldset { padding: 10px; background:#fbfbfb; border-radius:5px; margin-bottom:5px; } + .disabled2_section{pointer-events: none; opacity: 0.4;} @@ -1413,7 +2042,7 @@

You should follow the below mentioned rules, otherwise your answers will be invalid.

Rules:

    -
  • Use a headset, not the loudspeaker: otherwise your response will be rejected
    • +
  • Use a headset, not the loudspeaker: otherwise your response will be rejected
  • Perform the task in a quite environment
  • Do not change the volume after modifying it in the Setup section.
@@ -1433,7 +2062,7 @@

Attention:

-

This hit includes one or more Control clips (gold clips). Control clips are ones that we know that answer for and should be very easy to rate (they are clearly very good or very poor). We include control clips in the HIT to ensure raters are paying attention and their environment hasn't changed. +

This hit includes one or more Control clips (gold clips). Control clips are ones that we know the answer for and should be very easy to rate (they are clearly very good or very poor). We include control clips in the HIT to ensure raters are paying attention and their environment hasn't changed. Wrong answer to control clip(s) will result in rejection of the HIT.

@@ -1474,11 +2103,20 @@

-
-
- -
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
@@ -1533,16 +2171,8 @@

-
-
- -
-
- -
-
-
+
@@ -1560,7 +2190,7 @@

attention Please wear your headphones now.
-
+

 

@@ -1570,7 +2200,7 @@

-
+
@@ -1609,14 +2239,14 @@

-
+ {% if cfg.run_online_eval_setup == "true" %} +
+ +
+ {% endif %} + @@ -1837,33 +2549,9 @@

- -
-
-
-

- Listening device check -

-
-
-
- Introduction -

We need to check your listening device. It is an automated test using your Web-browser functionality. Please click on Start. - After that, you might see a popup message from your browser asking for allowance to use your microphone. - Please click on Allow (on some of smartphones there will be no message). We do not process or record any audio with your microphone. -

- -
-
-
-
- -
-
- -
+

@@ -1920,7 +2608,7 @@

-
+

Ratings

@@ -1968,13 +2656,51 @@

Ratings

+
+ + diff --git a/src/P808Template/DCR_template.html b/src/P808Template/DCR_template.html index ce02a58..071a9cc 100644 --- a/src/P808Template/DCR_template.html +++ b/src/P808Template/DCR_template.html @@ -47,6 +47,7 @@ font-size:0.9em; } fieldset { padding: 10px; background:#fbfbfb; border-radius:5px; margin-bottom:5px; } + .disabled2_section{pointer-events: none; opacity: 0.4;} @@ -1120,7 +1749,7 @@

You should follow the below mentioned rules, otherwise your answers will be invalid.

Rules:

    -
  • Use a headset, not the loudspeaker: otherwise your response will be rejected
    • +
  • Use a headset, not the loudspeaker: otherwise your response will be rejected
  • Perform the task in a quite environment
  • Do not change the volume after modifying it in the Setup section.
@@ -1139,7 +1768,7 @@

Please perform up to 0 HITs from this group. If you do more than that, the rest will be rejected.

Attention:

-

This hit includes one or more Control clips (gold clips). Control clips are ones that we know that answer for and should be very easy to rate (they are clearly very good or very poor). We include control clips in the HIT to ensure raters are paying attention and their environment hasn't changed. +

This hit includes one or more Control clips (gold clips). Control clips are ones that we know the answer for and should be very easy to rate (they are clearly very good or very poor). We include control clips in the HIT to ensure raters are paying attention and their environment hasn't changed. Wrong answer to control clip(s) will result in rejection of the HIT.

@@ -1180,11 +1809,20 @@

-
-
- -
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
@@ -1239,16 +1877,8 @@

-
-
- -
-
- -
-
-
+
@@ -1266,7 +1896,7 @@

attention Please wear your headphones now.
-
+

 

@@ -1276,7 +1906,7 @@

-
+
@@ -1314,14 +1944,14 @@

-
+ {% if cfg.run_online_eval_setup == "true" %} +
+ +
+ {% endif %} + @@ -1539,33 +2251,9 @@

- -
-
-
-

- Listening device check -

-
-
-
- Introduction -

We need to check your listening device. It is an automated test using your Web-browser functionality. Please click on Start. - After that, you might see a popup message from your browser asking for allowance to use your microphone. - Please click on Allow (on some of smartphones there will be no message). We do not process or record any audio with your microphone. -

- -
-
-
-
- -
-
- -
+

@@ -1618,7 +2306,7 @@

-
+

Ratings

@@ -1660,13 +2348,51 @@

Ratings

+
+ + diff --git a/src/P808Template/P808_multi.html b/src/P808Template/P808_multi.html index 1bd86c6..33116e8 100644 --- a/src/P808Template/P808_multi.html +++ b/src/P808Template/P808_multi.html @@ -263,29 +263,13 @@ - randomizeRatingQuestions: boolean whether the questions in the rating section should be randomized on loading time. - allowedMaxHITsInProject: How many HITs each worker is allowed to answer in this project - allowedMaxContinuesSessionDurationInMinutes: how log is the maximum continue session. A break of 10min will be forced - - questionUrls: {{cfg.rating_urls}}, - trainingUrls: {{cfg.training_urls}}, - goldClip: { - url: "${gold_url_2}", - sig_ans:"${gold_sig_ans_2}", - noise_ans:"${gold_noise_ans_2}", - disc_ans:"${gold_disc_ans_2}", - col_ans:"${gold_col_ans_2}", - loud_ans:"${gold_loud_ans_2}", - reverb_ans:"${gold_reverb_ans_2}", - ovrl_ans: "${gold_ovrl_ans_2}", - n_fail_show_warning: 3, - n_fail_block: 6 - } - - */ var config = { cookieName: "{{cfg.cookie_name}}", qualificationCookieName: "{{cfg.qual_cookie_name}}", qualificationValidFor: 43200, + showQualification: {{cfg.show_qualification}}, forceRetrainingInHours: 3, showSetupEveryMinutes: 120, debug: "true", @@ -294,29 +278,36 @@ training_gold_clips: {{ cfg.training_gold_clips }}, knownQuestionInTrainingUrl: "{{cfg.training_trap_urls}}", knownQuestionInTrainingAns: "{{cfg.training_trap_ans}}", - knownQuestionUrl: "${TP}", - knownQuestionAns: "${TP_ANS}", + // Front-end gold/trapping quality control is DISABLED: the correct answers + // are intentionally NOT embedded in the page. (They could be trivially + // recovered client-side - gold answers were base64, trapping answers plain + // text - so a front-end check cannot be made tamper-proof for a 1-5 answer + // space.) Gold/trapping validation is performed server-side in + // result_parser.py. Training feedback (config.training_gold_clips and + // knownQuestionInTraining*) is unaffected and still works. + knownQuestionUrl: "", + knownQuestionAns: "", goldClip: { - url: "${gold_url}", - sig_ans:"${gold_sig_ans}", - noise_ans:"${gold_noise_ans}", - disc_ans:"${gold_disc_ans}", - col_ans:"${gold_col_ans}", - loud_ans:"${gold_loud_ans}", - reverb_ans:"${gold_reverb_ans}", - ovrl_ans: "${gold_ovrl_ans}", + url: "", + sig_ans:"", + noise_ans:"", + disc_ans:"", + col_ans:"", + loud_ans:"", + reverb_ans:"", + ovrl_ans: "", n_fail_show_warning: 4, n_fail_block: 6 }, goldClip2: { - url: "${gold_url_2}", - sig_ans:"${gold_sig_ans_2}", - noise_ans:"${gold_noise_ans_2}", - disc_ans:"${gold_disc_ans_2}", - col_ans:"${gold_col_ans_2}", - loud_ans:"${gold_loud_ans_2}", - reverb_ans:"${gold_reverb_ans_2}", - ovrl_ans: "${gold_ovrl_ans_2}", + url: "", + sig_ans:"", + noise_ans:"", + disc_ans:"", + col_ans:"", + loud_ans:"", + reverb_ans:"", + ovrl_ans: "", dummy:"dummy" }, randomizeTrainingQuestions: "true", @@ -334,6 +325,20 @@ num4: "{{cfg.num4_ans}}", num5: "{{cfg.num5_ans}}" }, + runOnlineEvalQualification: {{cfg.run_online_eval_qualification}}, + runOnlineEvalSetup: {{cfg.run_online_eval_setup}}, + // listening-device check: the required active playback device + requiredPlaybackDevice: "{{cfg.required_playback_device}}", + // net coupling (dB) at/above which a loudspeaker is inferred (probe rise minus + // control-band rise); classifies headset vs loudspeaker + deviceCheckThresholdDb: {{cfg.device_check_threshold_db}}, + // probe-tone level (linear gain 0..1); kept low for hearing safety + deviceCheckProbeGain: {{cfg.device_check_probe_gain}}, + // gray-zone dispute: a loudspeaker detection may only be disputed when net_db is + // below this value (near the threshold); a clearly-loudspeaker reading cannot be disputed + deviceCheckDisputeMaxNetDb: {{cfg.device_check_dispute_max_net_db}}, + // P.804: show a per-scale "Cannot tell" option on col/disc/reverb/sig scales + allowCannotRate: {{cfg.allow_cannot_rate}}, cmp_correct_answers: {{ cfg.cmp_correct_answers }}, cmp_max_n_feedback: {{ cfg.cmp_max_n_feedback }}, cmp_pass_threshold: {{ cfg.cmp_pass_threshold }}, @@ -353,7 +358,15 @@ var generalErrorLog = ""; // how many times a feedback was given to the workers, var n_cmp_feedbacks = 0; - const Hide_HIT_REASON = Object.freeze({ "MAX_ACHIVED": 1, "QUALIFICATON": 2, "GOLD_FAILED":3 }) + var n_setup_feedback = 0; + // listening-device (acoustic echo) check state + var device_check_passed = false; + // audibility (beep-count) check state, used to catch a muted / silent device. + // dc_audibility_attempts persists across full-test re-runs (each retry is a whole + // new check via the Start button), so it is not reset per run. + var dc_beep_expected = 0; + var dc_audibility_attempts = 0; + const Hide_HIT_REASON = Object.freeze({ "MAX_ACHIVED": 1, "QUALIFICATON": 2, "GOLD_FAILED":3, "SETUP":4 }) /* Initializing the page: generate and hide sections depending to the cookies value */ @@ -396,46 +409,36 @@ cookie_debug_status = ""; // given cookie availability disable the qualification qual_passed = readCookie(config['qualificationCookieName']); - //------------- - is_test = false; - //------------ + // when the qualification section is disabled (e.g. it is run as a + // separate Prolific study), treat it as already passed and skip it. + if (config['showQualification'] === false) { + qual_passed = "true"; + } if (qual_passed !== null && qual_passed == "true") { $('.qualificationFieldset').prop("disabled", true); $("#qualification").hide(); $("#qualification :input").removeAttr("required"); - $("#setup").removeClass("disabled2_section"); - $("#device_check_section").removeClass("disabled2_section"); - $("#details_instruction").removeClass("disabled2_section"); - $("#training").removeClass("disabled2_section"); - $("#rating_section").removeClass("disabled2_section"); + // unlock the device check (which re-runs every session and gates the rest), + // or, when any device is accepted, skip it and open the following sections + unlockAfterQualification(); cookie_debug_status = cookie_debug_status + "ACR_LISTENER Exist*"; } - let gold_stat = getGoldQuestionStat(); - console.log("gold_stat:"+ gold_stat['failed'] +"warning:" + gold_stat['warning_showed'] ); + // Front-end gold QC disabled: never block or warn based on gold + // "failures" (validation is server-side). Keep force_re_training so the + // later training/cookie logic still works. let force_re_training = false; - if (gold_stat['failed']>= config['goldClip']['n_fail_block']) { - disableTheHIT(Hide_HIT_REASON.GOLD_FAILED); - }else if (gold_stat['failed']>= config['goldClip']['n_fail_show_warning'] && gold_stat['warning_showed'] == 0) { - console.log("show warning"); - // show warning - $("#gold_warning").show(); - // force training - force_re_training = true; - //gold_stat['warning_showed'] = 1; - //saveGoldQuestionStat(gold_stat); - } // detailed instruction - if (is_test || readCookie(config['cookieName'] + "_instra")) { + if (readCookie(config['cookieName'] + "_instra")) { $("#details_instruction").hide(); $("#training").removeClass("disabled_section"); } // given cookie availability disable the training - if (is_test || readCookie(config['cookieName'] + "_training") && !force_re_training) { + if (readCookie(config['cookieName'] + "_training") && !force_re_training) { $('.trainingFieldset').prop("disabled", true); $("#training").hide(); $("#training :input").removeAttr("required"); @@ -445,7 +448,7 @@ } // given cookie availability disable the setup - if (is_test || readCookie(config['cookieName'] + "_setup")) { + if (readCookie(config['cookieName'] + "_setup")) { $('.setupFieldset').prop("disabled", true); $("#setup").hide(); $("#setup :input").removeAttr("required"); @@ -538,6 +541,20 @@ updateResult(); // just to check checkScripts(); + $("#run_qualification_check").on("click", validateQualificationAnswer); + $("#run_setup_check").on("click", runSetupCheck); + // show which device the listening-device check requires, and adapt the + // instruction rule; hide the whole check when any device is accepted. + applyDeviceCheckTexts(); + if (config.requiredPlaybackDevice === "any") { + $("#device_check_section").hide(); + } + if (!config.runOnlineEvalQualification) { + // online qualification check is disabled -> do not gate on the answers, but + // the objective device check stays mandatory, so unlock the device check + // (which unlocks the rest on pass), or skip it when any device is accepted. + unlockAfterQualification(); + } }); async function generateHash(content) { @@ -639,6 +656,7 @@ Initialize the qualification section */ var qualification_ans = new Set(); + var qualification_attempts = 0; function initializeQualification() { randomizeHearingtestItems(); makeCheckboxBeRequired(); @@ -652,35 +670,41 @@ check if the qualification all answered */ function isQualificationDone() { - qualification_ans.add(this.name); - console.log("check if the qualification is done " + this.name + "," + qualification_ans.size); - - //validateQualificationAnswer(); - if (qualification_ans.size == 16) - validateQualificationAnswer(); + // validation runs only when the participant clicks "Check answers" } /* check the qualification passed */ function validateQualificationAnswer() { - m_tongue = $("input[name='3_mother_tongue']").val().toLowerCase().trim(); - m_tongue_check = config.qualification.mother_tongue.includes(m_tongue) - - // check listening devices - var listening_device = []; - $.each($("input[name='4_ld']:checked"), function () { - listening_device.push($(this).val()); - }); - device_check = false; - for (var i = 0; i < listening_device.length; i++) { - device_check = device_check || config.qualification.ld_in.includes(listening_device[i]); + // Require every qualification question to be answered before evaluating (and + // possibly screening out) the participant. If anything is missing, ask them to + // complete the section first instead of counting blanks as wrong. + var qualComplete = $("input[name='3_mother_tongue']:checked").length > 0 && + $("input[name='4_ld']:checked").length > 0 && + !!$("input[name='8_hearing']:checked").val(); + for (var qi = 1; qi <= 5 && qualComplete; qi++) { + if (!$("input[name='num" + qi + "']").val().trim()) qualComplete = false; + if (!$("input[name='comb_bw" + qi + "']:checked").val()) qualComplete = false; } - + if (!qualComplete) { + alert("Please answer all questions in the qualification section before checking your answers."); + return; + } + var mother_tongues = $("input[name='3_mother_tongue']:checked").map(function () { + return $(this).val().toLowerCase().trim(); + }).get(); + m_tongue_check = mother_tongues.some(function (v) { return config.qualification.mother_tongue.includes(v); }); + + // check listening-device self-report against the required playback device + // (headset -> in-ear/over-ear, loudspeaker -> ex-ls/ib-ls). When the study accepts + // "any" device, any single valid selection is fine. The objective device check + // later verifies the actual active device. + device_check = (config.requiredPlaybackDevice === "any") + ? (selfReportedDeviceClass() !== null) + : (selfReportedDeviceClass() === config.requiredPlaybackDevice); - working_area_checked = $("input[name='7_working_area']:checked").val() == "N" - hearing_self_report_check = $("input[name='8_hearing']:checked").val() == "normal" count_correct_ht = 0; @@ -696,67 +720,74 @@ if (checkQualNumCorrect($("input[name='num5']").val().trim(), config.qualification.num5)) count_correct_ht++; - bw_check = validate_bw_test(); - - //console.log("count_correct_ht:"+count_correct_ht); - passed = m_tongue_check && device_check && working_area_checked && hearing_self_report_check && bw_check && - count_correct_ht >= 3; - - - // add a qualification for 'qualificationValidFor' minutes - createCookie(config.qualificationCookieName, passed, config.qualificationValidFor); - if (passed){ - // activate the rest - $("#setup").removeClass("disabled2_section"); - $("#device_check_section").removeClass("disabled2_section"); - $("#training").removeClass("disabled2_section"); - $("#details_instruction").removeClass("disabled2_section"); - $("#rating_section").removeClass("disabled2_section"); - }else{ - // show the error message - disableTheHIT(Hide_HIT_REASON.QUALIFICATON); - } + // bandwidth check is async (hash-based); finish the decision once it resolves + validate_bw_test().then(function (bw_check) { + passed = m_tongue_check && device_check && + hearing_self_report_check && bw_check && count_correct_ht >= 3; + + // add a qualification for 'qualificationValidFor' minutes + createCookie(config.qualificationCookieName, passed, config.qualificationValidFor); + if (passed) { + // unlock what follows qualification. With a required device this only + // unlocks the device check (which then gates the rest); with "any" the + // device check is skipped and the remaining sections open directly. + unlockAfterQualification(); + var afterQual = document.getElementById( + config.requiredPlaybackDevice === "any" ? "setup" : "device_check_section"); + if (afterQual) afterQual.scrollIntoView({ behavior: "smooth", block: "start" }); + } else { + qualification_attempts++; + if (qualification_attempts < 2) { + // first failure: let the participant review and try once more + alert("Some of your answers are not correct. Please review your answers and click \"Check answers\" again. You have one more try."); + } else { + // second failure: not qualified -> screen-out + disableTheHIT(Hide_HIT_REASON.QUALIFICATON); + document.getElementById("qualification_rejected").scrollIntoView({ behavior: "smooth", block: "start" }); + } + } + }); } - bw_v2_test_data ={ - "comb_bw1":'dq', - "comb_bw2":'dq', - "comb_bw3":'dq', - "comb_bw4":'sq', - "comb_bw5":'sq', - } + // Per-position correct-answer hashes injected from the publish batch + // (sha256(clip_url:dq|sq)). The plain answers are never rendered into the page. + var bwHashes = {% if cfg.run_online_eval_qualification == "true" %}["${comb_bw_hash1}", "${comb_bw_hash2}", "${comb_bw_hash3}", "${comb_bw_hash4}", "${comb_bw_hash5}"]{% else %}[]{% endif %}; /** - * Validate the band-width check + * Verify one bandwidth-check answer against its hash (no plain answer in the page). **/ - function validate_bw_test() { - data = bw_v2_test_data; - count_correct = 0; - ans_array = Array(5).fill(0); - for (var i = 1; i <6; i++) { - q_name = 'comb_bw'+i; - ans = $("input[name='"+q_name+"']:checked").val(); - if (ans == data[q_name]) - ans_array[i-1] = 1; - else - ans_array[i-1] = 0; - } - console.log(ans_array); - bw_min = config['bw_min'].toUpperCase(); - bw_max = config['bw_max'].toUpperCase(); - - if (ans_array[0] + ans_array[3] + ans_array[4] !=3) - // failed in trapping of obvious questions - return false; - if ((bw_min == 'SWB' && ans_array[1] != 1) || (bw_min == 'FB' && ans_array[1]+ans_array[2] != 2)) - return false; + async function verifyBwHash(i, userAnswer) { + var h = bwHashes[i - 1]; + if (!h || h === "$" + "{comb_bw_hash" + i + "}") + return true; + var url = document.getElementById("comb_bw" + i).getAttribute("data-src"); + var data = new TextEncoder().encode(url + ":" + String(userAnswer).trim()); + var buf = await crypto.subtle.digest("SHA-256", data); + var hex = Array.from(new Uint8Array(buf)).map(function (b) { + return b.toString(16).padStart(2, "0"); + }).join(""); + return hex === h; + } - if ((bw_max == 'NB-WB' && ans_array[1]+ans_array[2] != 0) || (bw_max == 'SWB' && ans_array[2] != 0)) - return false; + /** + * Validate the band-width check locally (hash-based). Resolves to true only when + * every clip is identified correctly. The authoritative per-band bw_min/bw_max + * decision is done server-side by the result parser. + **/ + async function validate_bw_test() { + for (var i = 1; i < 6; i++) { + q_name = 'comb_bw' + i; + ans = $("input[name='" + q_name + "']:checked").val(); + if (!ans) + return false; + var ok = await verifyBwHash(i, ans); + if (!ok) + return false; + } return true; -} + } @@ -764,27 +795,38 @@ randomize clips in the hearing test */ function randomizeHearingtestItems() { - sections = [".rnd", ".rnd2", ".rnd4"]; + var sections = [".rnd", ".rnd2", ".rnd4"]; for (var j = 0; j < sections.length; j++) { - var cards = $(sections[j]); - for (var i = 0; i < cards.length; i++) { - var target = Math.floor(Math.random() * cards.length - 1) + 1; - var target2 = Math.floor(Math.random() * cards.length - 1) + 1; - cards.eq(target).before(cards.eq(target2)); - } + var $cards = $(sections[j]); + if ($cards.length < 2) + continue; + var $parent = $cards.eq(0).parent(); + var arr = $cards.toArray(); + // Fisher-Yates shuffle, then re-append in the shuffled order + for (var i = arr.length - 1; i > 0; i--) { + var k = Math.floor(Math.random() * (i + 1)); + var tmp = arr[i]; + arr[i] = arr[k]; + arr[k] = tmp; + } + for (var m = 0; m < arr.length; m++) { + $parent.append(arr[m]); + } } } /* Make a checkbox group to be required */ function makeCheckboxBeRequired() { - var requiredCheckboxes = $('.4_lds :checkbox[required]'); - requiredCheckboxes.change(function () { - if (requiredCheckboxes.is(':checked')) { - requiredCheckboxes.removeAttr('required'); - } else { - requiredCheckboxes.attr('required', 'required'); - } + $('.4_lds, .mts_group').each(function () { + var group = $(this).find(':checkbox'); + group.on('change', function () { + if (group.is(':checked')) { + group.removeAttr('required'); + } else { + group.attr('required', 'required'); + } + }); }); } /* @@ -880,10 +922,40 @@ } // add this to onload of page + function runSetupCheck() { + // Require every setup question to be answered before evaluating, then run the + // same validation used for the setup (CMP + math) section. + for (var i = 1; i < 5; i++) { + if (!$("input[name='cmp" + i + "']:checked").val()) { + alert("Please answer all questions in the setup section before checking your answers."); + return; + } + } + if (!$('#Math').val()) { + alert("Please answer all questions in the setup section before checking your answers."); + return; + } + validateCMPSection(); + } + function cmp_check_on_load() { - $('#setup').find(':input').filter(':visible').each(function () { - $(this).on('change', validateCMPSection); - }); + // The setup section is evaluated when the participant clicks "Check answers" + // (see runSetupCheck / #run_setup_check), not automatically on each input change. + } + + async function verifyMathHash(userAnswer) { + var url = document.getElementById("math1").getAttribute("data-src"); + var mathHash = {% if cfg.run_online_eval_setup == "true" %}"${math_hash}"{% else %}""{% endif %}; + if (!mathHash || mathHash === "$" + "{math_hash}") { + return true; + } + var payload = url + ":" + userAnswer.trim(); + var data = new TextEncoder().encode(payload); + var buf = await crypto.subtle.digest("SHA-256", data); + var hex = Array.from(new Uint8Array(buf)).map(function(b) { + return b.toString(16).padStart(2, "0"); + }).join(""); + return hex === mathHash; } /* @@ -902,41 +974,55 @@ if (!math_ans) return; - // 2. check math question - // do not check math question now + // 2. check math question via SHA-256 hash (client-side pre-check) + verifyMathHash(math_ans).then(function(mathCorrect) { + if (!mathCorrect) { + setupCheckFailed("Please listen carefully and answer the math question correctly. Make sure both earbuds are working and you are in a quiet environment."); + return; + } + validateCMPSectionAfterMath(cmp_answers); + }); + return; + } + + function setupCheckFailed(msg) { + // The setup allows up to two attempts. On the first failure the participant is + // asked to fix their setup and try once more; a second failure screens them out. + removeCookie(config['cookieName'] + "_setup"); + n_cmp_feedbacks++; + if (n_cmp_feedbacks < 2) { + makeAllCMPsUnChecked(); + alert(msg + " You have one more try."); + } else { + $('#setup').find(':input').filter(':visible').each(function () { + $(this).off('change'); + }); + disableTheHIT(Hide_HIT_REASON.SETUP); + document.getElementById("setup_rejected").scrollIntoView({ behavior: "smooth", block: "start" }); + } + } + + async function validateCMPSectionAfterMath(cmp_answers) { // 3. check cmps correct_ans = 0; for (i = 0; i < 4; i++) { if (cmp_answers[i] == "o") continue; url = $('span[id="CMP{0}_{1}"]'.f(i + 1, cmp_answers[i].toUpperCase())).attr("data-src"); - if (config["cmp_correct_answers"].includes(btoa(url))) + if (config["cmp_correct_answers"].includes(await generateHash(url))) correct_ans++; } // 4. shows the message if needed if (correct_ans >= config["cmp_pass_threshold"]) { // valid response in cmp createCookie(config['cookieName'] + "_setup", "un_important", config['showSetupEveryMinutes']); + unlockAfterSetup(); + alert("Your setup looks good. You can continue with the task."); + var setupNext = document.getElementById("details_instruction"); + if (setupNext) setupNext.scrollIntoView({ behavior: "smooth", block: "start" }); return; } - removeCookie(config['cookieName'] + "_setup"); - if (n_cmp_feedbacks < config["cmp_max_n_feedback"]) { - n_cmp_feedbacks++; - msg = "Based on your answers, your environment is NOT quite enough or your setup is NOT good enough. Please make sure to do your task in a quite environment with headset and use both earpods. Try the Setup section again."; - last_time_msg = " It is the last time you see this feedback."; - makeAllCMPsUnChecked(); - if (n_cmp_feedbacks == config["cmp_max_n_feedback"]) - alert(msg + last_time_msg); - else - alert(msg); - - } - // 4.2 remove the listener - if (n_cmp_feedbacks >= config["cmp_max_n_feedback"]) { - $('#setup').find(':input').filter(':visible').each(function () { - $(this).off('change'); - }); - } + setupCheckFailed("Based on your answers, your environment is not quiet enough or your setup is not good enough. Please make sure you are in a quiet environment, wearing a headset with both earbuds."); } function makeAllCMPsUnChecked() { @@ -994,12 +1080,37 @@ if (c == Hide_HIT_REASON.MAX_ACHIVED) $("#max_allowed_HITs_reached").show(); - if (c == Hide_HIT_REASON.QUALIFICATON) + if (c == Hide_HIT_REASON.QUALIFICATON) { + applyScreenoutMessage("qualification_rejected", "screenout_code"); $("#qualification_rejected").show(); + } + if (c == Hide_HIT_REASON.SETUP) { + applyScreenoutMessage("setup_rejected", "screenout_code"); + $("#setup_rejected").show(); + } if (c == Hide_HIT_REASON.GOLD_FAILED) $("#gold_rejected").show(); } + /* + Decide which screen-out message to show. If a completion code was provided (either + baked in from the config or filled by the HIT app server), ask the participant to + enter the code (and close the window) to be paid for the screening effort. Otherwise, + no code is available, so ask them to return the task. + */ + function applyScreenoutMessage(sectionId, codeVar) { + // build the placeholder token at runtime so the HIT app server's dollar-brace placeholder replacement + // does not rewrite this comparison string. + var placeholder = "$" + "{" + codeVar + "}"; + var section = document.getElementById(sectionId); + if (!section) return; + var codeEl = section.querySelector(".screenout_code_value"); + var code = codeEl ? codeEl.textContent.trim() : ""; + var hasCode = code !== "" && code !== placeholder; + section.querySelector(".screenout_paid_block").hidden = !hasCode; + section.querySelector(".screenout_return_block").hidden = hasCode; + } + /* Add a cookies with corresponding values */ @@ -1700,7 +1811,6 @@ (i.e. q2. math question). Here the cookie will be added/updated */ function userAnsweringToSetupQuestions() { - createCookie(config['cookieName'] + "_setup", "un_important", config['showSetupEveryMinutes']); $('input[name="math"]').off('change'); } @@ -1709,42 +1819,539 @@