Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 102 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ We used [MIT License](LICENSE) because it is simple, permissive and encourages d
- Team Members/Developers: Chikith Rishi Maddi, Vishakha Mathur, Samuel Keun

## Contact
- Ming-Dao Chia (Ming-Dao.Chia@anu.edu.au)

## Installation and Setup

Expand Down Expand Up @@ -63,6 +64,15 @@ The system requires two primary inputs:
- Metadata CSV file
- Raw hyperspectral .sig measurement files

### Input formatting

- Example raw .sig file data can be found in [sig_files](data/example_data/example1/sig_files)

- Example metadata csv file can be found in [example1](data/example_data/example1/metadata.csv)

- Example Location file can also be found in [example1](data/example_data/example1/positions.csv)


Recommended folder structure:

```text
Expand Down Expand Up @@ -157,17 +167,104 @@ Using absolute / full file paths (supported irrespective of file location):

```

#### Metadata File Path
## Sample and Test Data

Example data is available in [`data/example_data`](data/example_data). It contains:

* metadata CSV files;
* SVC `.sig` hyperspectral measurements; and
* location files used to arrange measurements in the heatmap.

Run HyperKey with the example dataset:

```bash
python hyperkey.py data/example_data/example1/metadata.csv \
-r data/example_data/example1 \
--outlier-analysis
```

Run the automated test suite after system changes:

```bash
pytest --cov=scripts --cov-report=term-missing
```

These tests verify file matching, path handling, report generation and other expected pipeline behaviour.

## Validation and Quality Assurance

HyperKey currently uses automated tests and manual output inspection for functional validation.

The following checks are performed:

* metadata rows are matched with the expected `.sig` files;
* missing, blank and invalid file numbers are reported;
* output files are created in both default and custom locations;
* NDVI uses the spectral bands nearest to the required red and near-infrared wavelengths;
* Markdown, HTML and PDF reports contain consistent processing results;
* outlier listings are checked against the generated outlier CSV; and
* CI runs the test suite whenever changes are pushed.


## Technical Documentation

HyperKey uses a modular Python workflow:

| Component | Responsibility |
| -------------------------- | ----------------------------------------------------------------------- |
| `hyperkey.py` | User-facing entry point |
| `workflow.py` | Coordinates all processing stages |
| `pipeline.py` | Validates metadata, finds `.sig` files and creates merged spectral data |
| `visualise_heatmap.py` | Calculates the selected vegetation index and generates a heatmap |
| `visualise_measurement.py` | Generates spectral-reflectance graphs |
| `outlier_analysis.py` | Detects and exports unusual spectral measurements |
| `report.py` | Generates Markdown, HTML and PDF reports |

The workflow follows these stages:

1. Validate the supplied paths and metadata.
2. Match metadata rows with `.sig` measurements.
3. Create the merged spectral CSV and processing summary.
4. Generate the heatmap and spectral graph.
5. Run optional outlier analysis.
6. Generate Markdown, HTML and PDF reports.

Output paths are resolved centrally and passed to each module. This allows the same workflow to support both the default output directory and a custom path supplied with `-o`.

## Version History

### Unreleased

* Added custom output paths.
* Added optional outlier analysis.
* Added outlier listings to Markdown, HTML and PDF reports.
* Added print-friendly HTML report styling.
* Replaced Playwright and Chromium with fpdf2 for PDF generation.
* Improved unique output filenames to prevent overwriting.


### Initial Prototype

* Added metadata and `.sig` file matching.
* Added merged spectral CSV generation.
* Added NDVI heatmaps and spectral-reflectance graphs.
* Added basic Markdown, HTML and PDF reports.

For detailed development history, see the repository’s [commit history](https://github.com/appn-anu/hyperkey/commits/main) and [releases](https://github.com/appn-anu/hyperkey/releases).

## References

## Input formatting
1. Rouse, J. W., Haas, R. H., Schell, J. A., and Deering, D. W. (1974). *Monitoring Vegetation Systems in the Great Plains with ERTS*. Third Earth Resources Technology Satellite Symposium, NASA SP-351.

Example raw .sig file data can be found in [sig_files](data/example_data/example1/sig_files)
2. Montero, D., Aybar, C., Mahecha, M. D., Martinuzzi, F., Söchting, M., and Wieneke, S. (2023). spyndex: A Python package for computing spectral indices. *SoftwareX, 23*, 101341. https://doi.org/10.1016/j.softx.2023.101341

Example metadata csv file can be found in [example1](data/example_data/example1/metadata.csv)
3. spyndex documentation: https://spyndex.readthedocs.io/

Example Location file can also be found in [example1](data/example_data/example1/positions.csv)
4. SVC HR-series spectroradiometer documentation: [add the exact manual or manufacturer URL used by the project].

5. fpdf2 documentation: https://py-pdf.github.io/fpdf2/

6. Outlier-detection method: [add the paper or statistical source corresponding to the method implemented in `outlier_analysis.py`].



Expand Down
Loading