Skip to content

Update reports with new jupycost functions #76

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
ateucher wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Update reports with new jupycost functions #76

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions _hub-usage-reporting/DESCRIPTION
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,8 @@ Imports:
khroma,
sixtyfour,
ggtext,
jupycost (>= 0.1.0)
jupycost (>= 0.2.0)
Remotes:
openscapes/kyber,
openscapes/jupycost,
gadenbuie/ggpomological,
getwilds/sixtyfour
gadenbuie/ggpomological
84 changes: 73 additions & 11 deletions _hub-usage-reporting/aws-usage-report-nasa.qmd
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "NASA Openscapes 2i2c JupyterHub\nUsage and Costs"
params:
year_month: "2025-02"
year_month: !expr format(lubridate::floor_date(Sys.Date(), "month") - 1, "%Y-%m")
subtitle: "Monthly report for `r format(lubridate::ym(params$year_month), '%B %Y')`"
format: typst
---
Expand All @@ -20,17 +20,17 @@ A key objective of NASA Openscapes is to minimize “the time to science” for
The main costs of running the JupyterHub come from two sources:

1. Compute, using AWS EC2
2. Storage using AWS EFS, via storage in users' home directories
2. Storage using AWS EBS volumes and snapshots, via storage in users' home directories

Compute costs scale up and down as the Hub is used, however storage costs are
fixed - we pay for "data at rest", with
[ongoing daily costs/GB](https://aws.amazon.com/efs/pricing/) even while the
[ongoing daily costs/GB](https://aws.amazon.com/ebs/pricing/) even while the
Hub is not running.

Storing large amounts of data in the cloud can incur significant ongoing costs if not done optimally. We are developing [technical strategies and policies](https://openscapes.cloud/data-storage.html) in the Earthdata Cloud Cookbook to reduce storage costs that will keep the Openscapes 2i2c Hub a shared resource for us all to use, while also providing reusable strategies for other admins.

This report is intended to give a monthly summary of usage of the Hub and its
resources, by tracking metrics on costs and usage of key components of storage (EFS)
resources, by tracking metrics on costs and usage of key components of storage (EBS)
and compute (EC2).

```{r setup}
Expand Down Expand Up @@ -102,7 +102,6 @@ The following plot shows the total monthly costs of all AWS services related to
the Hub, as well as a breakdown of costs by service each month.

```{r total-costs}

# https://www.paws-r-sdk.com/docs/costexplorer_get_cost_and_usage/
# https://docs.aws.amazon.com/aws-cost-management/latest/APIReference/API_GetDimensionValues.html
all_daily_costs_six_months <- get_daily_usage_costs(
Expand All @@ -125,9 +124,10 @@ total_monthly_cost <- total_monthly_usage_costs$cost[

total_monthly_cost_plot <- ggplot(total_monthly_usage_costs) +
geom_line(aes(x = start_date, y = cost)) +
scale_x_date(date_breaks = "1 month", date_labels = "%b") +
labs(
title = glue::glue(
"The total cost of all AWS Services for running the NASA Openscapes 2i2c",
"The total cost of all AWS Services for running the NASA Openscapes 2i2c ",
"Hub in {reporting_my} was ${round(total_monthly_cost)}"
),
x = "Month",
Expand Down Expand Up @@ -159,6 +159,7 @@ monthly_cost_by_service_plot <- ggplot(
aes(x = start_date, y = cost, fill = service)
) +
geom_col() +
scale_x_date(date_breaks = "1 month", date_labels = "%b") +
scale_fill_discrete(
type = aws_ce_palette(n_distinct(monthly_cost_service_summary$service))
) +
Expand All @@ -170,7 +171,7 @@ monthly_cost_by_service_plot <- ggplot(
) +
labs(
title = "Monthly cost of AWS Services",
subtitle = "Largest costs are EC2 compute (blue) and EFS (home directory) storage (red)",
subtitle = "Largest costs are EC2 compute (blue) and EBS (home directory) storage (red)",
caption = "*The top nine services are shown individually, with any remaining grouped into 'Other'",
x = "Month",
y = "Monthly cost ($)",
Expand All @@ -181,17 +182,46 @@ monthly_cost_by_service_plot <- ggplot(
total_monthly_cost_plot / monthly_cost_by_service_plot
```

The same costs can be grouped by category, following the cost allocation categories
used by 2i2c: "core" (always-on infrastructure that is needed whether or not the
Hub is used), "compute" (user sessions), and "home storage" (volumes and
snapshots for user home directories), as well as networking and object storage.

```{r monthly-costs-by-component}
#| fig-height: 4

monthly_cost_by_component_plot <- all_daily_costs_six_months |>
mutate(start_date = floor_date(date, unit = "month")) |>
summarise(cost = sum(cost), .by = c(start_date, service_component)) |>
mutate(
service_component = fct_reorder(service_component, cost, .fun = sum)
) |>
ggplot(aes(x = start_date, y = cost, fill = service_component)) +
geom_col() +
scale_x_date(date_breaks = "1 month", date_labels = "%b") +
scale_fill_brewer(palette = "Set2") +
theme(legend.position = "bottom") +
labs(
title = "Monthly cost by category",
x = "Month",
y = "Monthly cost ($)",
fill = "Category"
)

monthly_cost_by_component_plot
```

### Storage

Managing storage is an effective way to manage long-term costs in the Hub, as
data-at-rest is an ongoing cost, much of which can be avoided by monitoring and
reducing storage of data that is not required.

User home directories are in an AWS ["Elastic File System" (EFS)](https://aws.amazon.com/efs/) mount, which is
a relatively expensive option for long-term storage of large files. The
User home directories are stored on AWS [Elastic Block Store (EBS)](https://aws.amazon.com/ebs/) volumes, which
are backed up with snapshots. This is a relatively expensive option for long-term storage of large files. The
following figure plots the daily total size of data storage in the user home
directories in the Hub over the past six months. The size of the home drives
is directly correlated with the costs for "Amazon Elastic File System" in the
is related to the EBS volume and snapshot costs, which appear under "EC2 - Other" in the
previous chart.

```{r monthly-storage}
Expand All @@ -208,7 +238,7 @@ monthly_size |>
geom_line(aes(x = date, y = dirsize_gb)) +
scale_x_datetime(date_breaks = "1 month", date_labels = "%B") +
labs(
title = "Total size of user home directories in AWS EFS in the main Hub",
title = "Total size of user home directories in AWS EBS in the main Hub",
x = "Month",
y = "Total Size (GB)"
)
Expand All @@ -219,6 +249,38 @@ monthly_size |>
To understand more about usage and costs during the current month,
we can look at daily usage metrics and costs.

### Daily costs by category

The following figure shows the daily costs for `r reporting_month`, grouped by
the same categories as above.

```{r daily-costs-by-component}
#| fig-height: 3

daily_cost_by_component <- all_daily_costs_six_months |>
filter(date >= start_date, date <= end_date) |>
summarise(cost = sum(cost), .by = c(date, service_component)) |>
mutate(service_component = fct_reorder(service_component, cost, .fun = sum))

ggplot(
daily_cost_by_component,
aes(x = date, y = cost, fill = service_component)
) +
geom_col() +
scale_x_date(
breaks = seq(start_date, end_date, by = "week"),
date_labels = "%b %d"
) +
scale_fill_brewer(palette = "Set2") +
theme(legend.position = "bottom") +
labs(
title = glue::glue("Daily cost by category in {reporting_my}"),
x = "Date",
y = "Daily cost ($)",
fill = "Category"
)
```

### Home directory sizes

The Hub can currently be accessed via two different "namespaces": "production"
Expand Down
80 changes: 69 additions & 11 deletions _hub-usage-reporting/aws-usage-report-noaa.qmd
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "NMFS Openscapes 2i2c JupyterHub\nUsage and Costs"
params:
year_month: "2025-09"
year_month: !expr format(lubridate::floor_date(Sys.Date(), "month") - 1, "%Y-%m")
subtitle: "Monthly report for `r format(lubridate::ym(params$year_month), '%B %Y')`"
format: typst
---
Expand All @@ -20,17 +20,17 @@ A key objective of NMFS Openscapes is to minimize “the time to science” for
The main costs of running the JupyterHub come from two sources:

1. Compute, using AWS EC2
2. Storage using AWS EFS, via storage in users' home directories
2. Storage using AWS EBS volumes and snapshots, via storage in users' home directories

Compute costs scale up and down as the Hub is used, however storage costs are
fixed - we pay for "data at rest", with
[ongoing daily costs/GB](https://aws.amazon.com/efs/pricing/) even while the
[ongoing daily costs/GB](https://aws.amazon.com/ebs/pricing/) even while the
Hub is not running.

Storing large amounts of data in the cloud can incur significant ongoing costs if not done optimally. We are developing [technical strategies and policies](https://openscapes.cloud/data-storage.html) to reduce storage costs that will keep the Openscapes 2i2c Hubs a shared resource for us all to use, while also providing reusable strategies for other admins.

This report is intended to give a monthly summary of usage of the Hub and its
resources, by tracking metrics on costs and usage of key components of storage (EFS)
resources, by tracking metrics on costs and usage of key components of storage (EBS)
and compute (EC2).

```{r setup}
Expand Down Expand Up @@ -126,9 +126,10 @@ total_monthly_cost <- total_monthly_usage_costs$cost[

total_monthly_cost_plot <- ggplot(total_monthly_usage_costs) +
geom_line(aes(x = start_date, y = cost)) +
scale_x_date(date_breaks = "1 month", date_labels = "%b") +
labs(
title = glue::glue(
"The total cost of all AWS Services for running the NMFS Openscapes 2i2c",
"The total cost of all AWS Services for running the NMFS Openscapes 2i2c ",
"Hub in {reporting_my} was ${round(total_monthly_cost)}"
),
x = "Month",
Expand Down Expand Up @@ -157,9 +158,10 @@ monthly_cost_service_summary <- monthly_costs_by_service |>

monthly_cost_by_service_plot <- ggplot(
monthly_cost_service_summary,
aes(x = end_date, y = cost, fill = service)
aes(x = start_date, y = cost, fill = service)
) +
geom_col() +
scale_x_date(date_breaks = "1 month", date_labels = "%b") +
scale_fill_discrete(
type = aws_ce_palette(n_distinct(monthly_cost_service_summary$service))
) +
Expand All @@ -171,7 +173,7 @@ monthly_cost_by_service_plot <- ggplot(
) +
labs(
title = "Monthly cost of AWS Services",
# subtitle = "Largest costs are EC2 compute (blue) and EFS (home directory) storage (red)",
# subtitle = "Largest costs are EC2 compute (blue) and EBS (home directory) storage (red)",
caption = "*The top nine services are shown individually, with any remaining grouped into 'Other'",
x = "Month",
y = "Monthly cost ($)",
Expand All @@ -182,17 +184,44 @@ monthly_cost_by_service_plot <- ggplot(
total_monthly_cost_plot / monthly_cost_by_service_plot
```

The same costs can be grouped by category, following the cost allocation tags
used by 2i2c: "core" (always-on infrastructure that is needed whether or not the
Hub is used), "compute" (user sessions), and "home storage" (volumes and
snapshots for user home directories), as well as networking and object storage.

```{r monthly-costs-by-component}
#| fig-height: 4

monthly_cost_by_component_plot <- all_daily_costs_six_months |>
mutate(start_date = floor_date(date, unit = "month")) |>
summarise(cost = sum(cost), .by = c(start_date, service_component)) |>
mutate(service_component = fct_reorder(service_component, cost, .fun = sum)) |>
ggplot(aes(x = start_date, y = cost, fill = service_component)) +
geom_col() +
scale_x_date(date_breaks = "1 month", date_labels = "%b") +
scale_fill_brewer(palette = "Set2") +
theme(legend.position = "bottom") +
labs(
title = "Monthly cost by category",
x = "Month",
y = "Monthly cost ($)",
fill = "Category"
)

monthly_cost_by_component_plot
```

### Storage

Managing storage is an effective way to manage long-term costs in the Hub, as
data-at-rest is an ongoing cost, much of which can be avoided by monitoring and
reducing storage of data that is not required.

User home directories are in an AWS ["Elastic File System" (EFS)](https://aws.amazon.com/efs/) mount, which is
a relatively expensive option for long-term storage of large files. The
User home directories are stored on AWS [Elastic Block Store (EBS)](https://aws.amazon.com/ebs/) volumes, which
are backed up with snapshots. This is a relatively expensive option for long-term storage of large files. The
following figure plots the daily total size of data storage in the user home
directories in the Hub over the past six months. The size of the home drives
is directly correlated with the costs for "Amazon Elastic File System" in the
is related to the EBS volume and snapshot costs, which appear under "EC2 - Other" in the
previous chart.

```{r monthly-storage}
Expand All @@ -210,7 +239,7 @@ monthly_size |>
geom_line(aes(x = date, y = dirsize_gb)) +
scale_x_datetime(date_breaks = "1 month", date_labels = "%B") +
labs(
title = "Total size of user home directories in AWS EFS in the main Hub",
title = "Total size of user home directories in AWS EBS in the main Hub",
x = "Month",
y = "Total Size (GB)"
)
Expand All @@ -221,6 +250,35 @@ monthly_size |>
To understand more about usage and costs during the current month,
we can look at daily usage metrics and costs.

### Daily costs by category

The following figure shows the daily costs for `r reporting_month`, grouped by
the same categories as above.

```{r daily-costs-by-component}
#| fig-height: 3

daily_cost_by_component <- all_daily_costs_six_months |>
filter(date >= start_date, date <= end_date) |>
summarise(cost = sum(cost), .by = c(date, service_component)) |>
mutate(service_component = fct_reorder(service_component, cost, .fun = sum))

ggplot(daily_cost_by_component, aes(x = date, y = cost, fill = service_component)) +
geom_col() +
scale_x_date(
breaks = seq(start_date, end_date, by = "week"),
date_labels = "%b %d"
) +
scale_fill_brewer(palette = "Set2") +
theme(legend.position = "bottom") +
labs(
title = glue::glue("Daily cost by category in {reporting_my}"),
x = "Date",
y = "Daily cost ($)",
fill = "Category"
)
```

### Home directory sizes

The following figure shows the total size of home directories in the NMFS Hub. Usage is broken
Expand Down
Loading