diff --git a/_hub-usage-reporting/DESCRIPTION b/_hub-usage-reporting/DESCRIPTION index 6ef3221..9960b20 100644 --- a/_hub-usage-reporting/DESCRIPTION +++ b/_hub-usage-reporting/DESCRIPTION @@ -22,9 +22,8 @@ Imports: khroma, sixtyfour, ggtext, - jupycost (>= 0.1.0) + jupycost (>= 0.2.0) Remotes: openscapes/kyber, openscapes/jupycost, - gadenbuie/ggpomological, - getwilds/sixtyfour + gadenbuie/ggpomological diff --git a/_hub-usage-reporting/aws-usage-report-nasa.qmd b/_hub-usage-reporting/aws-usage-report-nasa.qmd index 5e7b370..fecb839 100644 --- a/_hub-usage-reporting/aws-usage-report-nasa.qmd +++ b/_hub-usage-reporting/aws-usage-report-nasa.qmd @@ -1,7 +1,7 @@ --- title: "NASA Openscapes 2i2c JupyterHub\nUsage and Costs" params: - year_month: "2025-02" + year_month: !expr format(lubridate::floor_date(Sys.Date(), "month") - 1, "%Y-%m") subtitle: "Monthly report for `r format(lubridate::ym(params$year_month), '%B %Y')`" format: typst --- @@ -20,17 +20,17 @@ A key objective of NASA Openscapes is to minimize “the time to science” for The main costs of running the JupyterHub come from two sources: 1. Compute, using AWS EC2 -2. Storage using AWS EFS, via storage in users' home directories +2. Storage using AWS EBS volumes and snapshots, via storage in users' home directories Compute costs scale up and down as the Hub is used, however storage costs are fixed - we pay for "data at rest", with -[ongoing daily costs/GB](https://aws.amazon.com/efs/pricing/) even while the +[ongoing daily costs/GB](https://aws.amazon.com/ebs/pricing/) even while the Hub is not running. Storing large amounts of data in the cloud can incur significant ongoing costs if not done optimally. We are developing [technical strategies and policies](https://openscapes.cloud/data-storage.html) in the Earthdata Cloud Cookbook to reduce storage costs that will keep the Openscapes 2i2c Hub a shared resource for us all to use, while also providing reusable strategies for other admins. This report is intended to give a monthly summary of usage of the Hub and its -resources, by tracking metrics on costs and usage of key components of storage (EFS) +resources, by tracking metrics on costs and usage of key components of storage (EBS) and compute (EC2). ```{r setup} @@ -102,7 +102,6 @@ The following plot shows the total monthly costs of all AWS services related to the Hub, as well as a breakdown of costs by service each month. ```{r total-costs} - # https://www.paws-r-sdk.com/docs/costexplorer_get_cost_and_usage/ # https://docs.aws.amazon.com/aws-cost-management/latest/APIReference/API_GetDimensionValues.html all_daily_costs_six_months <- get_daily_usage_costs( @@ -125,9 +124,10 @@ total_monthly_cost <- total_monthly_usage_costs$cost[ total_monthly_cost_plot <- ggplot(total_monthly_usage_costs) + geom_line(aes(x = start_date, y = cost)) + + scale_x_date(date_breaks = "1 month", date_labels = "%b") + labs( title = glue::glue( - "The total cost of all AWS Services for running the NASA Openscapes 2i2c", + "The total cost of all AWS Services for running the NASA Openscapes 2i2c ", "Hub in {reporting_my} was ${round(total_monthly_cost)}" ), x = "Month", @@ -159,6 +159,7 @@ monthly_cost_by_service_plot <- ggplot( aes(x = start_date, y = cost, fill = service) ) + geom_col() + + scale_x_date(date_breaks = "1 month", date_labels = "%b") + scale_fill_discrete( type = aws_ce_palette(n_distinct(monthly_cost_service_summary$service)) ) + @@ -170,7 +171,7 @@ monthly_cost_by_service_plot <- ggplot( ) + labs( title = "Monthly cost of AWS Services", - subtitle = "Largest costs are EC2 compute (blue) and EFS (home directory) storage (red)", + subtitle = "Largest costs are EC2 compute (blue) and EBS (home directory) storage (red)", caption = "*The top nine services are shown individually, with any remaining grouped into 'Other'", x = "Month", y = "Monthly cost ($)", @@ -181,17 +182,46 @@ monthly_cost_by_service_plot <- ggplot( total_monthly_cost_plot / monthly_cost_by_service_plot ``` +The same costs can be grouped by category, following the cost allocation categories +used by 2i2c: "core" (always-on infrastructure that is needed whether or not the +Hub is used), "compute" (user sessions), and "home storage" (volumes and +snapshots for user home directories), as well as networking and object storage. + +```{r monthly-costs-by-component} +#| fig-height: 4 + +monthly_cost_by_component_plot <- all_daily_costs_six_months |> + mutate(start_date = floor_date(date, unit = "month")) |> + summarise(cost = sum(cost), .by = c(start_date, service_component)) |> + mutate( + service_component = fct_reorder(service_component, cost, .fun = sum) + ) |> + ggplot(aes(x = start_date, y = cost, fill = service_component)) + + geom_col() + + scale_x_date(date_breaks = "1 month", date_labels = "%b") + + scale_fill_brewer(palette = "Set2") + + theme(legend.position = "bottom") + + labs( + title = "Monthly cost by category", + x = "Month", + y = "Monthly cost ($)", + fill = "Category" + ) + +monthly_cost_by_component_plot +``` + ### Storage Managing storage is an effective way to manage long-term costs in the Hub, as data-at-rest is an ongoing cost, much of which can be avoided by monitoring and reducing storage of data that is not required. -User home directories are in an AWS ["Elastic File System" (EFS)](https://aws.amazon.com/efs/) mount, which is -a relatively expensive option for long-term storage of large files. The +User home directories are stored on AWS [Elastic Block Store (EBS)](https://aws.amazon.com/ebs/) volumes, which +are backed up with snapshots. This is a relatively expensive option for long-term storage of large files. The following figure plots the daily total size of data storage in the user home directories in the Hub over the past six months. The size of the home drives -is directly correlated with the costs for "Amazon Elastic File System" in the +is related to the EBS volume and snapshot costs, which appear under "EC2 - Other" in the previous chart. ```{r monthly-storage} @@ -208,7 +238,7 @@ monthly_size |> geom_line(aes(x = date, y = dirsize_gb)) + scale_x_datetime(date_breaks = "1 month", date_labels = "%B") + labs( - title = "Total size of user home directories in AWS EFS in the main Hub", + title = "Total size of user home directories in AWS EBS in the main Hub", x = "Month", y = "Total Size (GB)" ) @@ -219,6 +249,38 @@ monthly_size |> To understand more about usage and costs during the current month, we can look at daily usage metrics and costs. +### Daily costs by category + +The following figure shows the daily costs for `r reporting_month`, grouped by +the same categories as above. + +```{r daily-costs-by-component} +#| fig-height: 3 + +daily_cost_by_component <- all_daily_costs_six_months |> + filter(date >= start_date, date <= end_date) |> + summarise(cost = sum(cost), .by = c(date, service_component)) |> + mutate(service_component = fct_reorder(service_component, cost, .fun = sum)) + +ggplot( + daily_cost_by_component, + aes(x = date, y = cost, fill = service_component) +) + + geom_col() + + scale_x_date( + breaks = seq(start_date, end_date, by = "week"), + date_labels = "%b %d" + ) + + scale_fill_brewer(palette = "Set2") + + theme(legend.position = "bottom") + + labs( + title = glue::glue("Daily cost by category in {reporting_my}"), + x = "Date", + y = "Daily cost ($)", + fill = "Category" + ) +``` + ### Home directory sizes The Hub can currently be accessed via two different "namespaces": "production" diff --git a/_hub-usage-reporting/aws-usage-report-noaa.qmd b/_hub-usage-reporting/aws-usage-report-noaa.qmd index 9271b0e..d2c5acf 100644 --- a/_hub-usage-reporting/aws-usage-report-noaa.qmd +++ b/_hub-usage-reporting/aws-usage-report-noaa.qmd @@ -1,7 +1,7 @@ --- title: "NMFS Openscapes 2i2c JupyterHub\nUsage and Costs" params: - year_month: "2025-09" + year_month: !expr format(lubridate::floor_date(Sys.Date(), "month") - 1, "%Y-%m") subtitle: "Monthly report for `r format(lubridate::ym(params$year_month), '%B %Y')`" format: typst --- @@ -20,17 +20,17 @@ A key objective of NMFS Openscapes is to minimize “the time to science” for The main costs of running the JupyterHub come from two sources: 1. Compute, using AWS EC2 -2. Storage using AWS EFS, via storage in users' home directories +2. Storage using AWS EBS volumes and snapshots, via storage in users' home directories Compute costs scale up and down as the Hub is used, however storage costs are fixed - we pay for "data at rest", with -[ongoing daily costs/GB](https://aws.amazon.com/efs/pricing/) even while the +[ongoing daily costs/GB](https://aws.amazon.com/ebs/pricing/) even while the Hub is not running. Storing large amounts of data in the cloud can incur significant ongoing costs if not done optimally. We are developing [technical strategies and policies](https://openscapes.cloud/data-storage.html) to reduce storage costs that will keep the Openscapes 2i2c Hubs a shared resource for us all to use, while also providing reusable strategies for other admins. This report is intended to give a monthly summary of usage of the Hub and its -resources, by tracking metrics on costs and usage of key components of storage (EFS) +resources, by tracking metrics on costs and usage of key components of storage (EBS) and compute (EC2). ```{r setup} @@ -126,9 +126,10 @@ total_monthly_cost <- total_monthly_usage_costs$cost[ total_monthly_cost_plot <- ggplot(total_monthly_usage_costs) + geom_line(aes(x = start_date, y = cost)) + + scale_x_date(date_breaks = "1 month", date_labels = "%b") + labs( title = glue::glue( - "The total cost of all AWS Services for running the NMFS Openscapes 2i2c", + "The total cost of all AWS Services for running the NMFS Openscapes 2i2c ", "Hub in {reporting_my} was ${round(total_monthly_cost)}" ), x = "Month", @@ -157,9 +158,10 @@ monthly_cost_service_summary <- monthly_costs_by_service |> monthly_cost_by_service_plot <- ggplot( monthly_cost_service_summary, - aes(x = end_date, y = cost, fill = service) + aes(x = start_date, y = cost, fill = service) ) + geom_col() + + scale_x_date(date_breaks = "1 month", date_labels = "%b") + scale_fill_discrete( type = aws_ce_palette(n_distinct(monthly_cost_service_summary$service)) ) + @@ -171,7 +173,7 @@ monthly_cost_by_service_plot <- ggplot( ) + labs( title = "Monthly cost of AWS Services", - # subtitle = "Largest costs are EC2 compute (blue) and EFS (home directory) storage (red)", + # subtitle = "Largest costs are EC2 compute (blue) and EBS (home directory) storage (red)", caption = "*The top nine services are shown individually, with any remaining grouped into 'Other'", x = "Month", y = "Monthly cost ($)", @@ -182,17 +184,44 @@ monthly_cost_by_service_plot <- ggplot( total_monthly_cost_plot / monthly_cost_by_service_plot ``` +The same costs can be grouped by category, following the cost allocation tags +used by 2i2c: "core" (always-on infrastructure that is needed whether or not the +Hub is used), "compute" (user sessions), and "home storage" (volumes and +snapshots for user home directories), as well as networking and object storage. + +```{r monthly-costs-by-component} +#| fig-height: 4 + +monthly_cost_by_component_plot <- all_daily_costs_six_months |> + mutate(start_date = floor_date(date, unit = "month")) |> + summarise(cost = sum(cost), .by = c(start_date, service_component)) |> + mutate(service_component = fct_reorder(service_component, cost, .fun = sum)) |> + ggplot(aes(x = start_date, y = cost, fill = service_component)) + + geom_col() + + scale_x_date(date_breaks = "1 month", date_labels = "%b") + + scale_fill_brewer(palette = "Set2") + + theme(legend.position = "bottom") + + labs( + title = "Monthly cost by category", + x = "Month", + y = "Monthly cost ($)", + fill = "Category" + ) + +monthly_cost_by_component_plot +``` + ### Storage Managing storage is an effective way to manage long-term costs in the Hub, as data-at-rest is an ongoing cost, much of which can be avoided by monitoring and reducing storage of data that is not required. -User home directories are in an AWS ["Elastic File System" (EFS)](https://aws.amazon.com/efs/) mount, which is -a relatively expensive option for long-term storage of large files. The +User home directories are stored on AWS [Elastic Block Store (EBS)](https://aws.amazon.com/ebs/) volumes, which +are backed up with snapshots. This is a relatively expensive option for long-term storage of large files. The following figure plots the daily total size of data storage in the user home directories in the Hub over the past six months. The size of the home drives -is directly correlated with the costs for "Amazon Elastic File System" in the +is related to the EBS volume and snapshot costs, which appear under "EC2 - Other" in the previous chart. ```{r monthly-storage} @@ -210,7 +239,7 @@ monthly_size |> geom_line(aes(x = date, y = dirsize_gb)) + scale_x_datetime(date_breaks = "1 month", date_labels = "%B") + labs( - title = "Total size of user home directories in AWS EFS in the main Hub", + title = "Total size of user home directories in AWS EBS in the main Hub", x = "Month", y = "Total Size (GB)" ) @@ -221,6 +250,35 @@ monthly_size |> To understand more about usage and costs during the current month, we can look at daily usage metrics and costs. +### Daily costs by category + +The following figure shows the daily costs for `r reporting_month`, grouped by +the same categories as above. + +```{r daily-costs-by-component} +#| fig-height: 3 + +daily_cost_by_component <- all_daily_costs_six_months |> + filter(date >= start_date, date <= end_date) |> + summarise(cost = sum(cost), .by = c(date, service_component)) |> + mutate(service_component = fct_reorder(service_component, cost, .fun = sum)) + +ggplot(daily_cost_by_component, aes(x = date, y = cost, fill = service_component)) + + geom_col() + + scale_x_date( + breaks = seq(start_date, end_date, by = "week"), + date_labels = "%b %d" + ) + + scale_fill_brewer(palette = "Set2") + + theme(legend.position = "bottom") + + labs( + title = glue::glue("Daily cost by category in {reporting_my}"), + x = "Date", + y = "Daily cost ($)", + fill = "Category" + ) +``` + ### Home directory sizes The following figure shows the total size of home directories in the NMFS Hub. Usage is broken