diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 981a5b0..cddf466 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -29,8 +29,7 @@ jobs: - name: Set up the data directory run: | mkdir gdmData - python -c "import GeospaceDataManagement as gdm; gdm.params['data_dirs'] = './gdmData'; print(gdm.params); print(gdm.citation); import os; os.path.isfile(gdm.citation)" - ls ~/.gdm + python -c "import GeospaceDataManagement as gdm; gdm.params['data_dirs'] = './gdmData'" - name: Check documentation build run: sphinx-build -E -b html docs dist/docs diff --git a/CHANGELOG.md b/CHANGELOG.md index dafc5d2..2456651 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ This project adheres to [Semantic Versioning](https://semver.org/). a variable did not exist * DOC: - Added missing sections to docstrings + - Added information about space weather instruments and methods [0.0.1] - 2026-09-25 -------------------- diff --git a/README.md b/README.md index be6c514..51f46d8 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,5 @@ +[![Pytest with Flake8](https://github.com/USNavalResearchLaboratory/GeospaceDataManagement/actions/workflows/main.yml/badge.svg)](https://github.com/USNavalResearchLaboratory/GeospaceDataManagement/actions/workflows/main.yml) [![Coverage Status](https://coveralls.io/repos/github/USNavalResearchLaboratory/GeospaceDataManagement/badge.svg)](https://coveralls.io/github/USNavalResearchLaboratory/GeospaceDataManagement) + GeospaceDataManagement provides a simple and flexible interface for robust data analysis from beginning to end - with a focus on downloading, loading, cleaning and managing scientific data. The project's plug-in design allows analysis diff --git a/docs/examples.rst b/docs/examples.rst new file mode 100644 index 0000000..ff6645e --- /dev/null +++ b/docs/examples.rst @@ -0,0 +1,12 @@ +Examples +======== + +Here are some examples that demonstrate how to use various tools and features. + +.. toctree:: + examples/ex_sw_init.rst + examples/ex_f107.rst + examples/ex_kp.rst + examples/ex_ace.rst + examples/ex_mult_downloads.rst + examples/ex_mock_downloads.rst diff --git a/docs/examples/ex_ace.rst b/docs/examples/ex_ace.rst new file mode 100644 index 0000000..9d55779 --- /dev/null +++ b/docs/examples/ex_ace.rst @@ -0,0 +1,86 @@ +.. _exace: + +Make ACE SWEPAM behave like OMNI Data +===================================== + +NASA OMNI data is *the* place to go for solar wind (SW) parameters. However, +this data set is not available in real time. Fortunately, we can use the +`OMNI processing `_ to +normalize the ACE SWEPAM SW proton density and ion temperature. This is +performed using the :py:func:`GeospaceDataManagement.instruments.methods.ace.ace_swepam_hourly_omni_norm` +function. The example below uses historic ACE SWEPAM data so that we can +compare the results to OMNI. + +:: + + + import datetime as dt + import matplotlib as mpl + import matplotlib.pyplot as plt + + import GeospaceDataManagement as gdm + + ace = gdm.Instrument(inst_module=gdm.instruments.ace_swepam, + tag='historic', update_files=True) + + # Add the OMNI conversion routine to the ACE instrument + ace.custom_attach(gdm.instruments.methods.ace.ace_swepam_hourly_omni_norm) + + # Pick a day with a geomagnetic storm + stime = dt.datetime(2014, 3, 26) + + # If needed, download the data + if stime not in ace.files.files.index: + ace.download(start=stime) + + # Load the data + ace.load(date=stime) + + print(ace.variables) + + +This should yield ``Index(['jd', 'sec', 'status', 'sw_proton_dens', +'sw_bulk_speed', 'sw_ion_temp', 'sw_proton_dens_norm', 'sw_ion_temp_norm'], +dtype='object')``. The variables with the ``'norm'`` suffix were added by the +conversion function. Also get the OMNI Data. Now, let's plot the two data sets +together. + +:: + + + fig = plt.figure() + fig.suptitle("L1 ACE and L2 OMNI HRO data for {:}".format(stime.date())) + ax_dens = fig.add_subplot(211) + ax_temp = fig.add_subplot(212) + + ax_dens.plot(ace.index, ace['sw_proton_dens_norm'], 'k-', label='ACE') + ax_dens.plot(omni.index, omni['proton_density'], 'b-', label='OMNI') + ax_temp.plot(ace.index, ace['sw_ion_temp_norm'], 'k-', label='ACE') + ax_temp.plot(omni.index, omni['T'], 'b-', label='OMNI') + ax_temp.set_yscale('log') + ax_dens.xaxis.set_major_formatter(mpl.dates.DateFormatter('')) + ax_temp.xaxis.set_major_formatter(mpl.dates.DateFormatter('%H:%M')) + ax_dens.set_xlim(stime, stime + dt.timedelta(days=1)) + ax_temp.set_xlim(stime, stime + dt.timedelta(days=1)) + ax_temp.set_xlabel('Universal Time (HH:MM)') + ax_dens.set_ylabel(r'SW N$_p$ ({:s})'.format(omni.meta['proton_density', + 'units'])) + ax_temp.set_ylabel(r'SW T$_i$ ({:s})'.format(omni.meta['T', 'units'])) + ax_dens.legend(loc=1, fontsize='medium', ncol=2) + + # If not running in interactive mode + plt.show() + + +This figure shows that the temperature adjustment agrees relatively well, but +that the proton density has a large offset. This descrepency can be attributed +to the data source. The :py:mod:`GeospaceDataManagement.instruments.ace_swepam` +module uses L1 data, while OMNI uses L2 data that has undergone additional +processing. For this time period most of the temperature data is of high +quality in the L2 data, but all the proton density data is flagged for removal. +The ACE L2 data is accessible through CDAWeb or the `ACE database +`_. + +.. image:: ../images/ex_ace_omni.png + :align: center + diff --git a/docs/examples/ex_f107.rst b/docs/examples/ex_f107.rst new file mode 100644 index 0000000..c37b473 --- /dev/null +++ b/docs/examples/ex_f107.rst @@ -0,0 +1,88 @@ +.. _exf107: + +Load F\ :sub:`10.7`\ from Multiple Sources +========================================== + +As shown in :ref:`exswinit`, there are several sources that provide +F\ :sub:`10.7`\ over different time periods. It can be useful to combine these +data sets into a single :py:class:`GeospaceDataManagement.Instrument` object. +This may be done using the +:py:func:`~GeospaceDataManagement.instruments.methods.f107.combine_f107` +function. + +:: + + + import datetime as dt + import matplotlib as mpl + import matplotlib.pyplot as plt + + import GeospaceDataManagement as gdm + + f107_hist = gdm.Instrument(inst_module=gdm.instruments.sw_f107, + tag='historic', update_files=True) + f107_prel = gdm.Instrument(inst_module=gdm.instruments.sw_f107, + tag='prelim', update_files=True) + f107_fore = gdm.Instrument(inst_module=gdm.instruments.sw_f107, + tag='45day', update_files=True) + + # If needed, download the data + f107_hist.download(start=f107_hist.lasp_stime, stop=dt.datetime(2018, 4, 1)) + f107_prel.download(start=f107_hist.files.files.index[-1], + stop=f107_prel.today()) + f107_fore.download(start=f107_fore.today()) + + # Check the downloaded file range for the historic source + print(f107_hist.files.files.index.min(), f107_hist.files.files.index.max()) + + +This should yield ``1947-02-01 00:00:00 2018-04-30 00:00:00``. Now, combine the +data in these :py:class:`GeospaceDataManagement.Instruments` using the +:py:func:`~GeospaceDataManagement.instruments.methods.f107.combine_f107` +function twice, after loading the available data + +:: + + # Load all the data + f107_hist.load(date=f107_hist.lasp_stime, + end_date=f107_hist.files.files.index.max() + + dt.timedelta(days=1)) + f107_prel.load(fname=f107_prel.files.files[0], + stop_fname=f107_prel.files.files[-1]) + f107_fore.load(date=f107_fore.today()) + + # Combine the historic sources for all available times + f107 = gdm.instruments.methods.f107.combine_f107(f107_hist, f107_prel) + + # Combine the 45 day measurements and forecast + f107 = gdm.instruments.methods.f107.combine_f107(f107, f107_fore) + + # Check the combined Instrument index + print(f107.index[0], f107.index[-1]) + + +This print statement yields ``1947-02-14 00:00:00 2022-10-07 00:00:00``, where +the end date is roughly 45 days in the future to account for the forecasted +data. The combined :py:class:`GeospaceDataManagement.Instrument` varialbe, +``f107`` can be used to plot the F\ :sub:`10.7`\ over time. + +:: + + + fig = plt.figure() + ax = fig.add_subplot(111) + + ax.plot(f107.index, f107['f107'], 'k.', ms=1) + ax.xaxis.set_major_formatter(mpl.dates.DateFormatter('%Y')) + ax.xaxis.set_major_locator(mpl.dates.YearLocator(11)) + ax.set_xlim(f107.index[0], f107.index[-1]) + ax.set_xlabel('Universal Time (Year C.E.)') + ax.set_ylabel(r'F$_{10.7}$ (sfu)') + + # If not running in interactive mode + plt.show() + + +.. image:: ../images/ex_f107_all.png + :align: center + diff --git a/docs/examples/ex_kp.rst b/docs/examples/ex_kp.rst new file mode 100644 index 0000000..12b81e8 --- /dev/null +++ b/docs/examples/ex_kp.rst @@ -0,0 +1,97 @@ +.. _exkp: + +Load Kp from Multiple Sources +============================= + +Following from the previous F\ :sub:`10.7`\ example, :ref:`exf107`, the space +weather methods also have a routine to combine several sources of Kp over +different time periods. This may be done using +:py:func:`~GeospaceDataManagement.instruments.methods.kp_ap.combine_kp`. + +:: + + + import datetime as dt + import matplotlib as mpl + import matplotlib.pyplot as plt + + import GeospaceDataManagement as gdm + + kp_his = gdm.Instrument(inst_module=gdm.instruments.sw_kp, + tag='def', update_files=True) + kp_rec = gdm.Instrument(inst_module=gdm.instruments.sw_kp, + tag='recent', update_files=True) + kp_for = gdm.Instrument(inst_module=gdm.instruments.sw_kp, + tag='forecast', update_files=True) + + # Set the time range + stime = dt.datetime(1999, 1, 1) + etime = kp_for.today() + + # If needed, download the data + kp_his.download(start=stime, stop=etime) + kp_rec.download(start=stime, stop=etime) + kp_for.download(start=etime) + + # Combine the Kp sources for all available times + kp = gdm.instruments.methods.kp_ap.combine_kp(kp_his, kp_rec, kp_for, + stime, etime) + + # Check the combined Instrument index + print(kp.index[0], kp.index[-1]) + + +This yields ``1999-01-01 00:00:00 2022-08-23 21:00:00``, where the last date is +your current date (not the date that I created this example). Now, we can take +the combined :py:class:`GeospaceDataManagement.Instrument` can be used to plot +the Kp over time. + +:: + + + fig = plt.figure() + ax = fig.add_subplot(111) + + # Color code data by activity level + ax.bar(kp.data.loc[kp['Kp'] < 4, 'Kp'].index, + kp.data.loc[kp['Kp'] < 4, 'Kp'], color='green', label='Quiet') + ax.bar(kp.data.loc[(kp['Kp'] >= 4) & (kp['Kp'] < 7), 'Kp'].index, + kp.data.loc[(kp['Kp'] >= 4) & (kp['Kp'] < 7), 'Kp'], color='orange', + label='Active') + ax.bar(kp.data.loc[kp['Kp'] >= 7, 'Kp'].index, + kp.data.loc[kp['Kp'] >= 7, 'Kp'], color='firebrick', label='Storm') + + # Format the figure + ax.set_xlim(stime, etime) + ax.set_ylim(0, 9) + ax.xaxis.set_major_formatter(mpl.dates.DateFormatter('%Y')) + ax.set_xlabel('Universal Time (Year C.E.)') + ax.set_ylabel(r'Kp') + ax.legend(loc=1, fontsize='medium') + + # If not running in interactive mode + plt.show() + + +.. image:: ../images/ex_kp_all.png + :align: center + + + +Convert Kp to ap +================ + +The Kp and ap have a well established relationship, which takes the logarithmic +Kp index and converts it to a linear scale that is easier to handle numerically. +The :py:func:`~GeospaceDataManagement.instruments.methods.kp_ap.convert_3hr_kp_to_ap` +converts Kp to ap, as shown below. + +:: + + + gdm.instruments.methods.kp_ap.convert_3hr_kp_to_ap(kp) + + print("Max: {:.1f} -> {:.1f}, Min: {:.1f} -> {:.1f}".format( + kp['Kp'].max(), kp['3hr_ap'].max(), kp['Kp'].min(), kp['3hr_ap'].min())) + +This yields ``Max: 9.0 -> 400.0, Min: 0.0 -> 0.0``. diff --git a/docs/examples/ex_mock_downloads.rst b/docs/examples/ex_mock_downloads.rst new file mode 100644 index 0000000..eb81db3 --- /dev/null +++ b/docs/examples/ex_mock_downloads.rst @@ -0,0 +1,104 @@ +.. _exmockdown: + +Mock downloads +============== + +The space weather instruments all allow +:py:class:`~GeospaceDataManagement._instrument.Instrument` downloads to +"download" data from a local directory. This allows the user to have full +control over the remote access, while still allowing +:py:mod:`GeospaceDataManagement` to handle the data extraction and file +handling. + +When using the mock-download option, :py:mod:`GeospaceDataManagement` requires +that you specify the local directory where you have stored the desired data +files. If you don't employ this option, downloads will be attempted from the +remote source. If you do employ this option and the directory is wrong, and +IOError will be raised. If instead the expected file is missing, a logging +Information message will be returned. + +:: + + import GeospaceDataManagement as gdm + + f107 = gdm.Instrument(inst_module=gdm.instruments.sw_f107, tag='45day') + + # 'Download' the data from a local directory + f107.download(start=f107.tomorrow(), mock_download_dir='my/data/directory') + + +These data files must have the same name as the remote file. In instances where +there is no remote filename (e.g., when "downloading" data from LISIRD), the +filename should reflect the internal name used by +:py:mod:`GeospaceDataManagement`. The locations and expected filenames for each +of the Instruments are provided in the table below. In this table, YYYY stands +for the 4-digit year, MM stands for the 2-digit month, and DD stands for the +2-digit day-of-month. QUERY is a placeholder for a index and date dependent +query needed to return the desired data. See the GFZ and LISIRD methods to see +how the queries are constructed. + ++------------------------------------------------------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| Remote Location | Filename | Instrument(s) | ++========================================================================+====================================+============================================================================================================================+ +| https://services.swpc.noaa.gov/text/ | 3-day-geomag-forecast.txt | sw-ap-forecast, sw-kp-forecast, sw-stormprob-forecast | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | 3-day-solar-geomag-predictions.txt | sw-ap-preciction, sw-f107-forecast, sw-flare-prediction, sw-kp-preciction, sw-polarcap-prediction, sw-stormprob-prediction | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | 45-day-ap-forecast.txt | sw-f107-45day, sw-ap-45day | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | ace-epam.txt | ace-epam-realtime | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | ace-magnetometer.txt | ace-mag-realtime | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | ace-sis.txt | ace-sis-realtime | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | ace-swepam.txt | ace-swepam-realtime | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | daily-geomagnetic-indices.txt | sw-ap-recent, sw-kp-recent | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | daily-solar-indices.txt | sw-ssn-daily, sw-f107-daily, sw-flare-daily, sw-sbfield-daily | ++------------------------------------------------------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| https://sohoftp.nascom.nasa.gov/sdb/ace/daily/ | YYYYMMDD_ace_epam_5m.txt | ace-epam-historic | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | YYYYMMDD_ace_mag_1m.txt | ace-mag-historic | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | YYYYMMDD_ace_sis_5m.txt | ace-sis-historic | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | YYYYMMDD_ace_swepam_1m.txt | ace-swepam-historic | ++------------------------------------------------------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| ftp://ftp.swpc.noaa.gov/pub/indices/old_indices | YYYY_DSD.txt | sw-ssn-prelim, sw-f107-prelim, sw-flare-prelim, sw-sbfield-prelim | ++------------------------------------------------------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| \https://kp.gfz-potsdam.de/app/json/?QUERY | Fadj_YYYY-MM.txt | sw-f107-now-adj | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | Fobs_YYYY-MM.txt | sw-f107-now-obs | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | Hp30_YYYY-MM.txt | sw-hpo-now-30min | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | Hp60_YYYY-MM.txt | sw-hpo-now-60min | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | SN_YYYY-MM.txt | sw-ssn-now | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | ap30_YYYY-MM.txt | sw-apo-now-30min | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | ap60_YYYY-MM.txt | sw-apo-now-60min | ++------------------------------------------------------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| https://datapub.gfz-potsdam.de/download/10.5880.Kp.0001/Kp_nowcast/ | Kp_nowYYYY.wdc | sw-ap-now, sw-cp-now, sw-kp-now | ++------------------------------------------------------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| https://datapub.gfz-potsdam.de/download/10.5880.Kp.0001/Kp_definitive/ | kp_defYYYY.wdc | sw-ap-def, sw-cp-def, sw-kp-def | ++------------------------------------------------------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| ftp://ftp.ngdc.noaa.gov/STP/GEOMAGNETIC_DATA/INDICES/DST | dstYYYY.txt | sw-dst-noaa | ++------------------------------------------------------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| \https://lasp.colorado.edu/lisird/latis/dap/QUERY | f107_monthly_YYYY-MM.txt | sw-f107-historic | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | mgii_composite_YYYY-MM.txt | sw-mgii-composite | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | mgii_sorce_YYYY-MM-DD.txt | sw-mgii-sorce | ++------------------------------------------------------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| \https://lasp.colorado.edu/space_weather/dsttemerin/ | ae_last_96_hrs.txt | sw-ae-lasp | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | al_last_96_hrs.txt | sw-al-lasp | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | au_last_96_hrs.txt | sw-au-lasp | ++ +------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ +| | dst_last_96_hrs.txt | sw-dst-lasp | ++------------------------------------------------------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------------+ diff --git a/docs/examples/ex_mult_downloads.rst b/docs/examples/ex_mult_downloads.rst new file mode 100644 index 0000000..be4daf3 --- /dev/null +++ b/docs/examples/ex_mult_downloads.rst @@ -0,0 +1,91 @@ +.. _exmultdown: + +Downloading data from multiple Instruments +========================================== + +Unlike most data sets, space weather index data sets often contain a combination +of related indices. This means that data from multiple +:py:class:`GeospaceDataManagement.Instrument` sub-modules may be available in a +single file. This data is saved for all +:py:class:`GeospaceDataManagement.Instrument` sub-modules when downloading any +one of them. In this way, downloads are minimized and no data sets are hidden +or thrown away. The following example returns both F\ :sub:`10.7`\ and Ap +forecasts from SWPC. + +:: + + import GeospaceDataManagement as gdm + + f107 = gdm.Instrument(inst_module=gdm.instruments.sw_f107, tag='45day') + ap = gdm.Instrument(inst_module=gdm.instruments.sw_ap, tag='45day') + + # Test to see this data hasn't already been downloaded + assert (ap.today() not in f107.files.files) and (ap.today() + not in ap.files.files) + + # Download both data sets using only one Instrument + f107.download(start=f107.tomorrow()) + ap.download(start=ap.tomorrow()) + + # Show that both files have been downloaded + print(f107.files.files[f107.today()], ap.files.files[ap.today()]) + + +This will yield ``f107_45day_YYYY-MM-DD.txt ap_45day_YYYY-MM-DD.txt``, where +YYYY-MM-DD is the day you run this example. The table below shows a list of +the :py:class:`GeospaceDataManagement.Instrument` sub-modules and tags that +download multiple data sets. When possible, the same tag has been used across +:py:class:`GeospaceDataManagement.Instrument` sub-modules. + + ++----------------------+----------+-----------+------------+------------+ +| Remote Data Set | Platform | Name | Tag(s) | Inst ID(s) | ++======================+==========+===========+============+============+ +| GFZ Kp | SW | Kp | def, now | | ++----------------------+----------+-----------+------------+------------+ +| | | Ap | | | ++----------------------+----------+-----------+------------+------------+ +| | | Cp | | | ++----------------------+----------+-----------+------------+------------+ +| SWPC Daily DSD | SW | F107 | daily | | ++----------------------+----------+-----------+------------+------------+ +| | | Flare | | | ++----------------------+----------+-----------+------------+------------+ +| | | SSN | | | ++----------------------+----------+-----------+------------+------------+ +| | | SBField | | | ++----------------------+----------+-----------+------------+------------+ +| SWPC Prelim DSD | SW | F107 | prelim | | ++----------------------+----------+-----------+------------+------------+ +| | | Flare | | | ++----------------------+----------+-----------+------------+------------+ +| | | SSN | | | ++----------------------+----------+-----------+------------+------------+ +| | | SBField | | | ++----------------------+----------+-----------+------------+------------+ +| SWPC 3-day Solar-Geo | SW | F107 | forecast | | ++----------------------+----------+-----------+------------+------------+ +| Predictions | | Kp | prediction | | ++----------------------+----------+-----------+------------+------------+ +| | | Ap | | | ++----------------------+----------+-----------+------------+------------+ +| | | StormProb | | | ++----------------------+----------+-----------+------------+------------+ +| | | Flare | | | ++----------------------+----------+-----------+------------+------------+ +| | | PolarCap | | | ++----------------------+----------+-----------+------------+------------+ +| SWPC 3-day Geomag | SW | Kp | forecast | | ++----------------------+----------+-----------+------------+------------+ +| Forecast | | Ap | | | ++----------------------+----------+-----------+------------+------------+ +| | | StormProb | | | ++----------------------+----------+-----------+------------+------------+ +| SWPC Daily Geomag | SW | Kp | recent | | ++----------------------+----------+-----------+------------+------------+ +| | | Ap | | | ++----------------------+----------+-----------+------------+------------+ +| SWPC 45-day Forecast | SW | F107 | 45day | | ++----------------------+----------+-----------+------------+------------+ +| | | Ap | | | ++----------------------+----------+-----------+------------+------------+ diff --git a/docs/examples/ex_sw_init.rst b/docs/examples/ex_sw_init.rst new file mode 100644 index 0000000..efa165b --- /dev/null +++ b/docs/examples/ex_sw_init.rst @@ -0,0 +1,47 @@ +.. _exswinit: + +Loading F\ :sub:`10.7`\ +======================== + +The Space Weather Instruments included as an optional installation option +provide different historical, real-time, and forecasted solar, geomagnetic, and +other space weather indices. As specified in :ref:`tutorial_basics`, data may +be loaded using the following commands. Historic F\ :sub:`10.7`\ is used as +an example. + +:: + + + import GeospaceDataManagement as gdm + + f107 = gdm.Instrument(inst_module=gdm.instruments.sw_f107, + tag='historic', update_files=True) + f107.download(start=f107.lasp_stime, stop=f107.today()) + f107.load() + print(f107.data) + + +The output includes all available historic data (as implied by the tag name), +including the specified date. This data set starts on 14 February 1947, as +indicated by the special instrument attribute `f107.lasp_stime`, and will +not reach up to the present day. At the time of publication this produces the +output shown below (the index header has been added here for clarity). + +:: + + + f107 f107_adjusted f107_observed + 1947-02-14 253.9 253.9 NaN + 1947-02-17 228.5 228.5 NaN + 1947-02-19 178.8 178.8 NaN + 1947-02-20 163.7 163.7 NaN + 1947-02-24 164.1 164.1 NaN + ... ... ... ... + 2018-04-27 69.6 69.6 68.7 + 2018-04-28 71.1 71.1 70.2 + 2018-04-29 72.2 72.2 71.1 + 2018-04-30 71.3 71.3 70.2 + + [25367 rows x 3 columns] + + diff --git a/docs/images/ex_ace_omni.png b/docs/images/ex_ace_omni.png new file mode 100644 index 0000000..7772115 Binary files /dev/null and b/docs/images/ex_ace_omni.png differ diff --git a/docs/images/ex_f107_all.png b/docs/images/ex_f107_all.png new file mode 100644 index 0000000..7772111 Binary files /dev/null and b/docs/images/ex_f107_all.png differ diff --git a/docs/images/ex_kp_all.png b/docs/images/ex_kp_all.png new file mode 100644 index 0000000..4f06810 Binary files /dev/null and b/docs/images/ex_kp_all.png differ diff --git a/docs/index.rst b/docs/index.rst index 20d0295..5cc19e8 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -12,6 +12,7 @@ Welcome to the GeospaceDataManagement documentation installation.rst quickstart.rst tutorial.rst + examples.rst instruments.rst new_instrument.rst dependency.rst diff --git a/docs/instruments.rst b/docs/instruments.rst index c876c2a..a15e655 100644 --- a/docs/instruments.rst +++ b/docs/instruments.rst @@ -8,3 +8,4 @@ Instruments instruments/general_instruments.rst instruments/testing_instruments.rst + instruments/sw_instruments.rst diff --git a/docs/instruments/sw_instruments.rst b/docs/instruments/sw_instruments.rst new file mode 100644 index 0000000..35306e0 --- /dev/null +++ b/docs/instruments/sw_instruments.rst @@ -0,0 +1,446 @@ +.. _instruments-sw: + +Space Weather Instruments +========================= + +These :py:class:`~GeospaceDataManagement._instruments.Instrument` modules are +made available with the 'space_weather' installation option. + +.. _ace-inst: + +ACE +--- + +The Space Weather Prediction Center (SWPC) provides several Advanced +Composition Explorer (ACE) instrument data sets for use as real-time and +historic measurements of the solar wind. This differs from the ACE scientific +data, which is available at a greater latency from +`CDAWeb `_. Information about these +data sets can be found at the `SWPC ACE Solar-Wind page +`_. + + +.. _ace-epam-inst: + +ACE EPAM +^^^^^^^^ + +EPAM is the Electron, Proton, and Alpha Monitor onboard ACE. + + +.. automodule:: GeospaceDataManagement.instruments.ace_epam + :members: + + +.. _ace-mag-inst: + +ACE MAG +^^^^^^^ + +Supports ACE Magnetometer data. + + +.. automodule:: GeospaceDataManagement.instruments.ace_mag + :members: + + +.. _ace-sis-inst: + +ACE SIS +^^^^^^^ + +Supports ACE Solar Isotope Spectrometer data. + + +.. automodule:: GeospaceDataManagement.instruments.ace_sis + :members: + + +.. _ace-swepam-inst: + +ACE SWEPAM +^^^^^^^^^^ + +Supports ACE Solar Wind Electron Proton Alpha Monitor data. + + +.. automodule:: GeospaceDataManagement.instruments.ace_swepam + :members: + + +.. _norp-inst: + +NoRP +---- + +The Nobeyama Radio Polarameters (NoRP) platform encompasses solar flux +measurements provided by the Japanese Solar Science Observatory. +`NoRP `_ provides +additional information and processing tools on their website. + +.. _norp-rf-inst: + +RF +^^^ + +RF is the radio flux measured from the sun at different wavelengths. This +provides a different measure of solar activity and has been corrected to be +in solar flux units at 1 AU. The daily data set currently starts in Nov 1951 +and is updated to extend to the current period, but not in real-time. + + +.. automodule:: GeospaceDataManagement.instruments.norp_rf + :members: + + +.. _sw-inst: + +SW +--- + +The Space Weather (SW) platform encompasses space weather indices that may be +found across a variety of platforms. Many of the remote centers that provide +these data sets include multiple types of data in each file. From +:py:mod:`GeospaceDataManagement` version 0.1.0, the remote information is separated +by :py:class:`pysat.Instrument` and saved into appropriate files. For example, +the definitive Kp data from the German Research Centre for Geosciences at +Potsdam (GFZ) will also download Ap and Cp data files. + +.. _sw-ae-inst: + +AE +^^^ + +AE is an auroral electrojet index that reflects the level of magnetic deflection +in the auroral zone due to the difference between the eastward and westward +electroject currents at Earth. Real-time predictions (last 96 hours) are +provided by +`LASP `_. + + +.. automodule:: GeospaceDataManagement.instruments.sw_ae + :members: + + +.. _sw-al-inst: + + +AL +^^^ + +AL is an auroral electrojet index that reflects the lower envelope, the negative +peak of the electroject currents at Earth. Real-time predictions (last 96 +hours) are provided by the +`LASP repository `_. + + +.. automodule:: GeospaceDataManagement.instruments.sw_al + :members: + + +.. _sw-au-inst: + + +AU +^^^ + +AU is an auroral electrojet index that reflects the upper envelope, the positive +peak of the electroject currents at Earth. Real-time predictions (last 96 +hours) are provided at the +`LASP page `_. + + +.. automodule:: GeospaceDataManagement.instruments.sw_au + :members: + + +.. _sw-ap-inst: + +Ap +^^^ + +Ap is a geomagnetic index that reflects the magnitude of geomagnetic +disturbances at Earth but unlike Kp uses a linear scale. Historic, recent +(last 30 days), and forecasted values are available from the +`GFZ Kp `_ and the +`SWPC Forecasts page `_. + + +.. automodule:: GeospaceDataManagement.instruments.sw_ap + :members: + + +.. _sw-apo-inst: + +apo +^^^ + +apo is a linear (half)-hourly, planetary, open-ended, geomagnetic index that +reflects the magnitude of geomagnetic disturbances at Earth. It is like Ap but +does not have an upper limit. Values from 1995 onwards are available from the +`GFZ Hpo page `_. + + +.. automodule:: GeospaceDataManagement.instruments.sw_apo + :members: + + +.. _sw-cp-inst: + +Cp +^^^ + +Cp is a derivative geomagnetic index that provides a qualitative estimate of the +overall level of magnetic activity for the day. The C9 index provides the same +information on a scale from 0-9 instead of 0.0-2.5. Historic values are +available from the `GFZ Kp page `_. + + +.. automodule:: GeospaceDataManagement.instruments.sw_cp + :members: + + +.. _sw-dst-inst: + +Dst +^^^ + +The Disturbance Storm Time (Dst) Index is a measure of magnetic activity +associated with the ring current. The National Centers for Environmental +Information (NCEI), formerly the National Geophysical Data Center (NGDC), +maintains the current database from which the historic Dst is downloaded. +`LASP `_ +performs the calculations and provides the predicted Dst for the last 96 hours. +You can learn more about the Dst Index at the +`WDC Kyoto Observatory page `_. + + +.. automodule:: GeospaceDataManagement.instruments.sw_dst + :members: + + +.. _sw-f107-inst: + +F \ :sub:`10.7`\ +^^^^^^^^^^^^^^^^ + +F \ :sub:`10.7`\ is the 10.7 cm radio solar flux (measured in solar flux units, +sfu) `[Cortie 1912] `_. +Historic indices, real-time indices, and forecasted indices are available from +the +`LASP F107 `_ and the +`SWPC F107 page `_. + + +.. automodule:: GeospaceDataManagement.instruments.sw_f107 + :members: + + +.. _sw-flare-inst: + +Solar Flares +^^^^^^^^^^^^ + +Solar flares have been monitored for decades and the data has been compiled +into standard measurements from different data sets. Historic indices, +real-time indices, and forecasted indices are available from +`SWPC `_. + + +.. automodule:: GeospaceDataManagement.instruments.sw_flare + :members: + + +.. _sw-hpo-inst: + +Hpo +^^^ + +Hpo is a (half)-Hourly, planetary, open-ended, geomagnetic index that +reflects the magnitude of geomagnetic disturbances at Earth. It is like Kp but +does not have an upper limit. Values from 1995 onwards are available from the +`GFZ Hp30 and Hp60 page `_. + + +.. automodule:: GeospaceDataManagement.instruments.sw_hpo + :members: + + + +.. _sw-kp-inst: + +Kp +^^^ + +Kp is a geomagnetic index that reflects the magnitude of geomagnetic +disturbances at Earth. Historic, recent (last 30 days), and forecasted values +are available from the +`GFZ Kp repository `_, and the +`SWPC Kp page `_. + + +.. automodule:: GeospaceDataManagement.instruments.sw_kp + :members: + + +.. _mgii-inst: + +MgII Core-to-Wing Ratio +^^^^^^^^^^^^^^^^^^^^^^^ + +The core-to-wing ratio of the solar MgII line is a proxy for solar chromospheric +variability. It has been used to extract a precise measurement of solar +activity at Earth. The two data sets provided by LASP together provide index +values from 1978 through 2020. + + +.. automodule:: GeospaceDataManagement.instruments.sw_mgii + :members: + +.. _sw-pc-inst: + +Polar Cap +^^^^^^^^^ + +Polar cap indices have been developed to provide information about high-latitude +conditions and inform ionospheric space weather models. Currently, this +:py:class:`~pysat._instrument.Instrument` provides absorption predictions from SWPC. + +.. automodule:: GeospaceDataManagement.instruments.sw_polarcap + :members: + + +.. _sw-sbfield-inst: + +Solar Magnetic Field +^^^^^^^^^^^^^^^^^^^^ + +The solar mean field provides a measure of of mean solar magnetic field using +full-disk optical observations of the iron line. The first observations were +made at Stanford by +`Scherrer et al. `_. + +.. automodule:: GeospaceDataManagement.instruments.sw_sbfield + :members: + + +.. _sw-ssn-inst: + +Sunspot Number +^^^^^^^^^^^^^^ + +The Sunspot Number (SSN) is one of the oldest continuously measured solar +indices. Currently, this Instrument provides preliminary and daily values from +SWPC (for example, here is the +`forecast page `_). + +.. automodule:: GeospaceDataManagement.instruments.sw_ssn + :members: + + +.. _sw-stormprob-inst: + +Storm Probability +^^^^^^^^^^^^^^^^^ + +Geomagnetic storm predictions are provided by SWPC for global, high-latitude, +and mid-latitude regions. SWPC uses the +`NOAA SW scales `_, which +are explained here. + +.. automodule:: GeospaceDataManagement.instruments.sw_stormprob + :members: + + +.. _sw-methods: + +Methods +------- + +Several methods exist to help combine multiple data sets and convert between +equivalent indices. Most of these directly support the Instrument sub-modules, +but others contain useful tools for manipulating specific data sets. + +ACE Methods +^^^^^^^^^^^ + +Supports the ACE instrument by providing reference and acknowledgement +information. + + +.. automodule:: GeospaceDataManagement.instruments.sw_methods.ace + :members: + +Dst Methods +^^^^^^^^^^^ + +Supports the Dst ring current index by providing reference and acknowledgement +information. + + +.. automodule:: GeospaceDataManagement.instruments.sw_methods.dst + :members: + +F \ :sub:`10.7`\ Methods +^^^^^^^^^^^^^^^^^^^^^^^^ + +Supports the F \ :sub:`10.7`\ radio flux by providing reference and +acknowledgement information as well as a routine to combine +F \ :sub:`10.7`\ data obtained from multiple sources. + + +.. automodule:: GeospaceDataManagement.instruments.sw_methods.f107 + :members: + + +Space Weather General Methods +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +General functions that are useful across the Space Weather Instruments. + + +.. automodule:: GeospaceDataManagement.instruments.sw_methods.general + :members: + + +GFZ Methods +^^^^^^^^^^^ + +Supports the German Research Centre for Geosciences at Potsdam (GFZ) data +products. + + +.. automodule:: GeospaceDataManagement.instruments.sw_methods.gfz + :members: + + +Kp and Ap Methods +^^^^^^^^^^^^^^^^^ + +Supports the Kp instrument by providing reference and acknowledgement +information, a routine to combine Kp from multiple sources, routines to convert +between Kp and Ap, and a routine that uses Kp data as a geomagnetic activity +filter for other data sets. + + +.. automodule:: GeospaceDataManagement.instruments.sw_methods.kp_ap + :members: + +LISIRD Methods +^^^^^^^^^^^^^^ + +Contains functions to support LASP Interactive Solar IRradiance Datacenter +(LISIRD) data sets and downloads. + + +.. automodule:: GeospaceDataManagement.instruments.sw_methods.lisird + :members: + +SWPC Methods +^^^^^^^^^^^^ + +Supports the Space Weather Prediction Center (SWPC) data products. + + +.. automodule:: GeospaceDataManagement.instruments.sw_methods.swpc + :members: diff --git a/docs/new_instrument.rst b/docs/new_instrument.rst index 2ef408b..8c116f4 100644 --- a/docs/new_instrument.rst +++ b/docs/new_instrument.rst @@ -36,9 +36,9 @@ syntax: import my_package registry.register_from_module(my_package.instruments) -After registry the instrument module name is stored in the user's home -directory in a hidden directory named :code:`~.GeospaceDataManagement`. The -instrument may then be instantiated with the instrument's platform and name: +After registry the instrument module name is stored in the user's home directory +in a hidden directory named :code:`~/.gdm`. The instrument may then be +instantiated with the instrument's platform and name: .. code-block:: python