[feat] Extract and format data for asar to SIS function

[sbreitbart-NOAA]

Extract and format data for asar to SIS function
Note: this branch is a companion to https://github.com/nmfs-ost/asar/tree/sis-data, which intakes the templates (blank or filled in, with this function) and sends them to Google Drive

@Schiano-NOAA This works by extracting data from rdas, not creating them within the function. I'm not sure that it'd be worth adding in the capability to create rdas within the function, as it'd need to be highly interactive (asking authors for arguments that impact key quantities), and this works even if rdas are missing.

This still needs some work (see below) but wanted to share it/ask for feedback/spark discussion before I got too far:

• unit tests
• documentation: what the output should look like, which rdas are expected (and how to export them via the functions), etc. (in a vignette?)

[Schiano-NOAA]

What do you mean SIS will generate? I think clarity here is needed saying a specific function.

[sbreitbart-NOAA]

How about "automatically assign"?

[Schiano-NOAA]

Maybe we should make a wrapper like add_accessibility that performs this fxn and export_to_sis together

[sbreitbart-NOAA]

I like the idea of saving the user time, but that might come at the cost of them not inspecting their files before sending to SIS. I think we should encourage them to make sure it looks right before doing so

[Schiano-NOAA]

abundance and biomass are not the same

[sbreitbart-NOAA]

I was confused too; Jeff confirmed that "Abundance = Biomass - or - Spawners". I'm going to inquire further

[Schiano-NOAA]

This is really coming together! I do feel quite strongly that the users should be able to use this function without creating the RDAs; however, I do see why the approach here is more reasonable. Playing devil's advocate, I do think that if a user wants to send their info to SIS and doesn't use stockplotr for their figures and tables, they will need to make all of the necessary ones before running this function, which is just as highly interactive as having this function operate without rdas.

I think we need to really think about the purpose of this function. Is it only for users who use stockplotr in its entirety or is it for people to pick and choose what they want to do even if it's simply converting their output and easily sending their results to SIS?

I can see both sides to this. Since stockplotr isn't in v1.0 quite yet and not planned to get there until end of June 2027, adoption by a large potion of the community wouldn't be expected for sometime after that. I think this approach to separating functions out though will be a big help to the transition for SIS and the user base.

With that said and thinking on it, proceed as planned and continue with this approach. I think we can assess the utility of it with some test cases (the steering committee or other willing users) once we get this out there and talk with the SIS devs.

[Schiano-NOAA]

When testing this out I got the error:

Error in path_tidy(.Call(fs_path_, lapply(args, function(x) enc2utf8(as.character(x))), : Total path length must be less than PATH_MAX: 260

What I did:

plot_spawning_biomass(example_data, make_rda = TRUE)
extract_sis_data(example_data)

Was there a different step to follow? I think the directions in the documentation aren't clear if it is. I almost feel like there are a lot of directions included for documentation for a standard function and it needs to be made simpler, but just my opinion here.

I think I am mainly confused if I am supposed to run all of the functions mentioned for plotting and creating tables to run this or if I only have to run one of them.

[sbreitbart-NOAA]

When testing this out I got the error:

Error in path_tidy(.Call(fs_path_, lapply(args, function(x) enc2utf8(as.character(x))), : Total path length must be less than PATH_MAX: 260

What I did:

plot_spawning_biomass(example_data, make_rda = TRUE)
extract_sis_data(example_data)

Was there a different step to follow? I think the directions in the documentation aren't clear if it is. I almost feel like there are a lot of directions included for documentation for a standard function and it needs to be made simpler, but just my opinion here.

I think I am mainly confused if I am supposed to run all of the functions mentioned for plotting and creating tables to run this or if I only have to run one of them.

Ah, the error was that "example_data" is not an appropriate value for any of the arguments, which are all directories.

[Schiano-NOAA]

Ahh I knew I was doing something wong!

[Schiano-NOAA]

@sbreitbart-NOAA just ran it again but without the example_data in the function.

ℹ No existing sis_assmt_template.csv found in C:/Users/samantha.schiano.NMFS/Documents/GitHub/stockplotr. Using blank template.
✔ Found existing key_quantities.csv in C:/Users/samantha.schiano.NMFS/Documents/GitHub/stockplotr.
! Biomass data was not extracted from the 'biomass_figure.rda' file.
! Recruitment (age 1) data was not extracted from the 'abundance_at_age_figure.rda' file.
! Fishing mortality data was not extracted from the 'fishing_mortality_figure.rda' file.
! Index data was not extracted from the 'index_figure.rda' file.
ℹ 'tables' folder not found in C:/Users/samantha.schiano.NMFS/Documents/GitHub/stockplotr.
✖ Some time series data will not be extracted.
ℹ The following time series summaries were extracted:
• spawning_biomass
Extracted time series summaries:
• spawning_biomass
Eligible for Primary designation:
• fishing_mortality
• recruitment
• catch
Must select ONE of the following as a Primary time series:
• spawning_biomass
• biomass
ℹ Ineligible for Primary designation: "index"
Error in `dplyr::mutate()`:
ℹ In argument: `Unit = dplyr::case_when(...)`.
Caused by error in `dplyr::case_when()`:
! Can't recycle `..1 (left)` (size 159) to match `..5 (right)` (size 0).
Run `rlang::last_trace()` to see where the error occurred.

Is this expected? I do not see any new file in my folder. Also does this only export whatever time series is made in a plot or do you only need to make one?

[sbreitbart-NOAA]

@sbreitbart-NOAA just ran it again but without the example_data in the function.

ℹ No existing sis_assmt_template.csv found in C:/Users/samantha.schiano.NMFS/Documents/GitHub/stockplotr. Using blank template.
✔ Found existing key_quantities.csv in C:/Users/samantha.schiano.NMFS/Documents/GitHub/stockplotr.
! Biomass data was not extracted from the 'biomass_figure.rda' file.
! Recruitment (age 1) data was not extracted from the 'abundance_at_age_figure.rda' file.
! Fishing mortality data was not extracted from the 'fishing_mortality_figure.rda' file.
! Index data was not extracted from the 'index_figure.rda' file.
ℹ 'tables' folder not found in C:/Users/samantha.schiano.NMFS/Documents/GitHub/stockplotr.
✖ Some time series data will not be extracted.
ℹ The following time series summaries were extracted:
• spawning_biomass
Extracted time series summaries:
• spawning_biomass
Eligible for Primary designation:
• fishing_mortality
• recruitment
• catch
Must select ONE of the following as a Primary time series:
• spawning_biomass
• biomass
ℹ Ineligible for Primary designation: "index"
Error in `dplyr::mutate()`:
ℹ In argument: `Unit = dplyr::case_when(...)`.
Caused by error in `dplyr::case_when()`:
! Can't recycle `..1 (left)` (size 159) to match `..5 (right)` (size 0).
Run `rlang::last_trace()` to see where the error occurred.

Is this expected? I do not see any new file in my folder. Also does this only export whatever time series is made in a plot or do you only need to make one?

No, that error is not expected (but the console messages above it are). I can't replicate this by running your example above. This extracts TS data from the relevant rdas, then inserts them into the time series csv. You only need one relevant rda to run this (e.g., SB, B, index, etc.) Is this the same code you ran?

plot_spawning_biomass(example_data, make_rda = TRUE)
extract_sis_data()

[Schiano-NOAA]

Yes, that's what I ran. Let me start fresh and make sure everything is up to date then try again

[Schiano-NOAA]

Woop! Got it to work, but I have a few questions.

(1) I am a bit confused still if you need to create more figures/tables to export everything due to this message

! Biomass data was not extracted from the 'biomass_figure.rda' file.
! Recruitment (age 1) data was not extracted from the 'abundance_at_age_figure.rda' file.
! Fishing mortality data was not extracted from the 'fishing_mortality_figure.rda' file.
! Index data was not extracted from the 'index_figure.rda' file.
ℹ 'tables' folder not found in C:/Users/samantha.schiano.NMFS/Documents/GitHub/stockplotr.
✖ Some time series data will not be extracted.

(2) It is unclear what you mean in this message:

Eligible for Primary designation:
  • fishing_mortality
  • recruitment
  • catch
Must select ONE of the following as a Primary time series:
  • spawning_biomass
  • biomass
ℹ Ineligible for Primary designation: "index"

I think it might be helpful to put a vignette explaining it or just changing the terminology. It might also just be too much information.

Otherwise, this works! I am not sure if there are ways to check the validity of it, so I will just trust that it's extracting the correct values. I need to still do checks on this for other models

[sbreitbart-NOAA]

Woop! Got it to work, but I have a few questions.

(1) I am a bit confused still if you need to create more figures/tables to export everything due to this message

! Biomass data was not extracted from the 'biomass_figure.rda' file.
! Recruitment (age 1) data was not extracted from the 'abundance_at_age_figure.rda' file.
! Fishing mortality data was not extracted from the 'fishing_mortality_figure.rda' file.
! Index data was not extracted from the 'index_figure.rda' file.
ℹ 'tables' folder not found in C:/Users/samantha.schiano.NMFS/Documents/GitHub/stockplotr.
✖ Some time series data will not be extracted.

(2) It is unclear what you mean in this message:

Eligible for Primary designation:
  • fishing_mortality
  • recruitment
  • catch
Must select ONE of the following as a Primary time series:
  • spawning_biomass
  • biomass
ℹ Ineligible for Primary designation: "index"

I think it might be helpful to put a vignette explaining it or just changing the terminology. It might also just be too much information.

Otherwise, this works! I am not sure if there are ways to check the validity of it, so I will just trust that it's extracting the correct values. I need to still do checks on this for other models

Glad it works for you! There's a lot of info that's useful to know when using this function, and since it's not being conveyed clearly enough, I'll make a vignette.

Re: your questions:

1. Yes, the more figures/tables a user has already created, the more data they can import and then export into the templates. The function automates the insertion of the available data, so the more present, the better.
2. SIS has "Primary" category designations for some types of data. I'll try to explain this more thoroughly in the vignette

[Schiano-NOAA]

Sounds good! I am not sure if you need to include the "primary" category designations. I think it would be better to use a different term or not include the message unless you intend for the user to use that to indicate it to Jeff or the SIS team. I don't know if they would be up for trying to learn it and retain it for one step at the end of the process. If you feel like it isn't much more work to add it in the vignette then go for it, but I am hesitant to continue using that language.

I am unsure what being eligible for a primary designation means to provide an alternative.

[sbreitbart-NOAA]

@Schiano-NOAA I wrote a vignette in an effort to give more information about the new function, including the purpose of the Primary category. Let's ask Jeff for a review once he's back in the office too. But if you're on board with this, I say let's merge it and update as necessary

[Schiano-NOAA]

@sbreitbart-NOAA sounds like a plan! Merge away