Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 18 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -324,7 +324,8 @@ stack with the realistic sample, start to finish:

`gdi-dataset-tool publish <id>` then makes the dataset visible, and the node serves it at
`/fairdp/dataset/<id>` and in the Beacon. There is a scripted equivalent for automation:
`gdi-dataset-tool config init -o tool.toml` writes a commented template, and the minimum is
`gdi-dataset-tool config init` writes a commented template to `~/.config/gdi/tool.toml`,
and the minimum is

```toml
country_code = "EE"
Expand All @@ -341,14 +342,18 @@ gdi-aggregated = "Genome of Europe Aggregated Data"

```bash
export GDI_TOOL__PROFILES__DEFAULT__S3__ACCESS_KEY_ID=… GDI_TOOL__PROFILES__DEFAULT__S3__SECRET_ACCESS_KEY=…
gdi-dataset-tool --config tool.toml keys pin-recipient # fetches {service_url}/.well-known/c4gh-recipient and pins it; --file node.pub when handed over offline
gdi-dataset-tool init # scaffolds package.yaml: point it at your VCFs, fill the metadata
gdi-dataset-tool --config tool.toml package package.yaml # build + pack: <id>.tar.c4gh, encrypted to the node; mints your provider key on first use
gdi-dataset-tool --config tool.toml upload <id>.tar.c4gh # into the bucket, hidden; the node picks it up on its next poll
gdi-dataset-tool --config tool.toml publish <id> # flips the sidecar to visible
gdi-dataset-tool --config tool.toml status <id> # the node's verdict, once the operator sets the bucket's write_status = true or gives you a management_url
gdi-dataset-tool keys pin-recipient # fetches {service_url}/.well-known/c4gh-recipient and pins it; --file node.pub when handed over offline
gdi-dataset-tool init # scaffolds package.yaml: point it at your VCFs, fill the metadata
gdi-dataset-tool package package.yaml # build + pack: <id>.tar.c4gh, encrypted to the node; mints your provider key on first use
gdi-dataset-tool upload <id>.tar.c4gh # into the bucket, hidden; the node picks it up on its next poll
gdi-dataset-tool publish <id> # flips the sidecar to visible
gdi-dataset-tool status <id> # the node's verdict, once the operator sets the bucket's write_status = true or gives you a management_url
```

Your provider key and the pinned node key live in `~/.config/gdi/keys/` and
`~/.config/gdi/recipients/`, or next to the file `--config` names, so keep such a config
out of any directory you share or package from.

Every tool field, including the co-located inbox profile and `keyless = true`:
[`tool.example.toml`](tool.example.toml).

Expand Down Expand Up @@ -465,9 +470,12 @@ keys and no profile. Worth knowing before your first real build:

Health is `GET :9090/health/live` and `/health/ready` (`ready: true, degraded: true` means
a provider bucket is dark), metrics are Prometheus text at `:9090/metrics`, and a dataset's
state is `GET :9090/datasets/{id}/state`. Below is the shipped dashboard, captured on the
dev stack with the realistic sample under a few minutes of query load. The alert names in
the panel titles are the shipped Prometheus rules:
state is `GET :9090/datasets/{id}/state`. Route traffic only to a ready node: one that
started while Vault was unreachable can answer publicly as if it had no datasets until it is
restarted ([operating.md §1](docs/operating.md#1-health-and-readiness-endpoints)).
Kubernetes readiness probes do that; `docker run` and Compose do not. Below is the shipped
dashboard, captured on the dev stack with the realistic sample under a few minutes of query
load. The alert names in the panel titles are the shipped Prometheus rules:

![The shipped Grafana dashboard: node readiness, datasets by state, request rate, error ratio and latency, on the dev stack under query load](docs/images/grafana-dashboard.png)

Expand Down
91 changes: 81 additions & 10 deletions crates/core/src/validate_pkg.rs
Original file line number Diff line number Diff line change
Expand Up @@ -979,17 +979,26 @@ pub fn canonical_bcp47(tag: &str) -> Option<String> {
Some(tag.to_ascii_lowercase())
}

/// Reject a value not in the allowed set.
/// Reject a value not in the allowed set, naming what is allowed: the IRI a bare last
/// segment such as `PUBLIC` stands for, or else the whole set (at most a handful).
///
/// # Errors
///
/// Returns [`CoreError::InvalidManifest`] if `value` is not in `allowed`.
pub fn validate_enum(field: &str, value: &str, allowed: &[&str]) -> CoreResult<()> {
if allowed.contains(&value) {
Ok(())
} else {
Err(invalid(&format!("{field} value {value:?} is not allowed")))
return Ok(());
}
let hint = allowed
.iter()
.find(|iri| iri.rsplit(['/', '#']).next() == Some(value))
.map_or_else(
|| format!("allowed: {}", allowed.join(", ")),
|iri| format!("write the full IRI {iri}"),
);
Err(invalid(&format!(
"{field} value {value:?} is not allowed; {hint}"
)))
}

/// Validate `type`: at least one IRI, each from the closed [`DATASET_TYPES`] set.
Expand Down Expand Up @@ -1092,12 +1101,28 @@ fn reject_disallowed_scheme(field: &str, value: &str, parsed: &url::Url) -> Core
let scheme = parsed.scheme();
if !ALLOWED_IRI_SCHEMES.contains(&scheme) {
return Err(invalid(&format!(
"{field} value {value:?} uses a disallowed URI scheme {scheme:?} (allowed: {ALLOWED_IRI_SCHEMES:?})"
"{field} value {value:?} uses a disallowed URI scheme {scheme:?} (allowed: {ALLOWED_IRI_SCHEMES:?}){}",
full_address_hint(value)
)));
}
Ok(())
}

/// The hint for a web address without its scheme, such as `node.example.org` (unparseable)
/// or `localhost:8080` (parsed with the scheme `localhost`). A full URI with another scheme,
/// such as `mailto:x@example.org`, gets none.
fn full_address_hint(value: &str) -> &'static str {
let schemeless = match value.split_once(':') {
None => true,
Some((_, rest)) => rest.starts_with(|c: char| c.is_ascii_digit()),
};
if schemeless && !value.contains("://") {
"; write a web address in full, with its scheme (https://…)"
} else {
""
}
}

/// Validate an IRI metadata field: parses as an absolute URL with a scheme and a
/// non-empty host (or, for `urn:`/`w3id`-style identifiers, a non-empty path),
/// carries no `IRIREF`-forbidden character (see RFC 3987 §2.2), uses an allowed
Expand All @@ -1116,8 +1141,12 @@ pub fn validate_iri(field: &str, value: &str) -> CoreResult<()> {
)));
}
reject_iri_unsafe_chars(field, value)?;
let parsed = url::Url::parse(value)
.map_err(|_| invalid(&format!("{field} value {value:?} is not a valid IRI/URL")))?;
let parsed = url::Url::parse(value).map_err(|_| {
invalid(&format!(
"{field} value {value:?} is not a valid IRI/URL{}",
full_address_hint(value)
))
})?;
reject_disallowed_scheme(field, value, &parsed)?;
// Require a scheme and either a host (http/https/...) or a non-empty
// opaque path (urn:, ...). A relative or scheme-less string fails
Expand Down Expand Up @@ -1209,8 +1238,12 @@ pub fn validate_url(field: &str, value: &str) -> CoreResult<()> {
)));
}
reject_iri_unsafe_chars(field, value)?;
let parsed = url::Url::parse(value)
.map_err(|_| invalid(&format!("{field} value {value:?} is not a valid URL")))?;
let parsed = url::Url::parse(value).map_err(|_| {
invalid(&format!(
"{field} value {value:?} is not a valid URL{}",
full_address_hint(value)
))
})?;
// Same scheme allow-list as `validate_iri`: a `hasURL`/reference value is emitted as
// an IRI into the public RDF too.
reject_disallowed_scheme(field, value, &parsed)
Expand Down Expand Up @@ -1955,6 +1988,31 @@ mod tests {
);
}

/// A web address written without its scheme is told so, whether it fails to parse or
/// parses with its host as the scheme; a full URI with a disallowed scheme is not.
#[test]
fn a_schemeless_address_is_told_to_start_with_https() {
for (field, value) in [
("service_url", "localhost:8080"),
("service_url", "node.example.org"),
("hasURL", "www.example.org/data"),
] {
let iri = validate_iri(field, value).unwrap_err().to_string();
let url = validate_url(field, value).unwrap_err().to_string();
for err in [iri, url] {
assert!(err.contains("with its scheme"), "{value:?}: {err}");
}
}
for value in [
"file:///etc/passwd",
"javascript:alert(1)",
"mailto:x@example.org",
] {
let err = validate_iri("license", value).unwrap_err().to_string();
assert!(!err.contains("with its scheme"), "{value:?}: {err}");
}
}

#[test]
fn disallowed_iri_scheme_rejected() {
// Schemes that `url::Url::parse` accepts but that must never reach the public FDP
Expand Down Expand Up @@ -2405,7 +2463,20 @@ mod tests {
)
.is_ok()
);
assert!(super::validate_enum("accessRights", "PUBLIC", super::ACCESS_RIGHTS).is_err());
let bare = super::validate_enum("accessRights", "PUBLIC", super::ACCESS_RIGHTS)
.unwrap_err()
.to_string();
assert!(
bare.contains(&format!("write the full IRI {}", super::ACCESS_RIGHTS[0])),
"a bare last segment names its IRI: {bare}"
);
let unknown = super::validate_enum("accessRights", "OPEN", super::ACCESS_RIGHTS)
.unwrap_err()
.to_string();
assert!(
super::ACCESS_RIGHTS.iter().all(|iri| unknown.contains(iri)),
"an unknown value lists the allowed set: {unknown}"
);
assert!(
super::validate_health_category(
"http://data.gdi.eu/core/p2/HealthCategoryHumanGenomic"
Expand Down
Loading