Using Application Usability Levels to support tracking the health of Heliophysics
Jeff Klenzing
Alexa J. Halford
Adam Kellerman
Barbara Thompson- 1Goddard Space Flight Center, National Aeronautics and Space Administration, Greenbelt, MD, United States
- 2Department of Earth, Planetary, and Space Sciences, University of California Los Angeles, Los Angeles, CA, United States
The Application Usability Level (AUL) framework describes how a project advances from basic research to operation-ready applications. Here we expand upon the current project-level usage of Application Usability Levels into a programmatic usage which can be used to help funding agencies track the health of the Heliophysics program. Examples are discussed throughout Heliophysics to show the utility of the Application Usability Level framework for describing the usability level of projects.
1 Introduction
Application Usability Levels (AULs) are used and often described within a research or research-to-operations context (e.g., Halford et al., 2019). While still a new framework, AULs have proven helpful to the community and have steadily grown in acceptance. For example, the United Kingdom Met office has now incorporated their use, and the United Kingdom and United States proposal calls for space weather applications also directly reference the AUL framework (e.g., Hapgood et al., 2020). However, this framework does not just describe progress at the project level, but can be used to support assessment of the health of the field overall.
This paper will outline how the AUL framework can be generalized to a funding organizational level. This extension was built in at the foundations of the framework, as part of its heritage from the Application Readiness Levels (ARLs). ARLs were developed by the Applied Sciences Program under the NASA HQ Earth Science Division (NASA Earth Science Applied Science, 2021). They have been successfully used in Earth Science for an extended period of time to track the health of the division and the roadblocks faced (e.g., Friedl and Sylak-Glassman, 2020). A framework such as the AULs or the ARLs can help guide strategic planning for new missions and new proposal calls. While the discussion will primarily focus on the AUL framework, these recommendations can apply to other readiness level frameworks, such as Technology Readiness Levels (e.g., Olechowski et al., 2015) and the Readiness Levels used by the National Oceanic and Atmospheric Administration (National Oceanic and Atmospheric Administration, 2021).
The AUL framework—when used across projects—provides a unique insight into the health of a program and where there may be common needs across the community. An overview of the phases is provided in Table 1. A healthy program should include projects across the AUL framework, ranging from exploratory research (AUL 1 through AUL 3) to research projects that are in their finalization and on-demand usage phases (AUL 6—AUL 9). The AUL framework provides an inventory of the balance across development levels, which can help determine whether the existing distribution of projects matches the needs of the community and funding agencies. In general, there are relatively many projects funded for early phase research, while a fraction of these reach the later phases. While it is expected that not every project will advance to an operational level, in general advancement of projects from mid to high level AULs is underfunded, resulting in the “Valley of Death” for research-to-operations (Merceret et al., 2013).
TABLE 1. A brief description of the AUL phases and levels.
2 Current examples of AULs in Heliophysics
One advantage of the AUL concept is that is a generalized framework that can be used for a variety of project types. Below are several examples of how this framework is being used in Heliophysics across the different phases outlines in Table 1.
2.1 Proof of concept studies
Phase 1 of the AUL framework covers the discovery and viability of possible applications, including proof of concept studies. While this phase is most similar to typical research projects, there is the added step of documenting the usage beyond immediate research. An example of a current Phase 1 project discussing potential applications is a new model of Solar Energetic Particle (SEP) injection and transport. van den Berg et al. (2020) outlines a list of requirements for predictive capability of SEPs needed for future crewed missions and uses this to inform the direction of future research (AUL 2). The potential use of solar soft X-ray activity as a proxy for SEPs is proposed as a proof-of-concept study (Steyn et al., 2020), which is consistent with AUL 3 (assess viability and current state of the art).
2.2 Prediction and validation of space weather events using models
Extending into Phase 2, testing and validation is now required. This describes projects that have moved beyond basic research and into applications. For example, the routine prediction of the growth of night-time Equatorial Plasma Bubbles is a space weather problem that has been outstanding for over 80 years due to the complex interplay of physical forces that can enhance or suppress the likelihood of growth on a given night (e.g., Klenzing et al., 2023). Recent work to improve these predictions has been outlined in Section 4.4 of Halford et al. (2019) using the AUL framework. An extensive validation campaign to compare 12 months of ground-based data with long-term runs of models was recently conducted as part of the validation track of the AUL framework (Carter et al., 2020).
2.3 Development of new space weather indices
Phase 3 projects have completed validation and are being implemented into operational solutions. The AUL framework has been employed in the development and validation of the LDi and LCi geomagnetic indices (Cid et al., 2020; Nahayo et al., 2022). These new indices were developed jointly by researchers and industry users for nowcasting local geomagnetic disturbances (LDi) and for geomagnetically induced current risk (LCi). They are currently being deployed real-time on ground-based stations (Guerrero et al., 2021).
2.4 Software development
The AUL framework has motivated the discussion of new metrics for model assessment (Liemohn et al., 2020, 2021; 2022). Additionally, it is useful for software projects that may have multiple applications, and therefore different AUL progress depending on the usage. For example, the sami2py software project consists of an open source python package that runs the SAMI2 (Sami2 is Another Model of the Ionosphere) ionospheric model (Huba et al., 2000) and archives the results. For a given version of the software, the different applications can be rated at different levels. For example, the usage of the software in early phase research projects was rated at a level of AUL 7 (application prototype) as this is actively being used by researchers, while the usage of the software as a educational tool is rated as Phase 1 since the needs of a broader user base (i.e., students from a wider range of computational backgrounds) is still actively being identified (Klenzing et al., 2022). These new user requirements will inform the future versions and releases of the software.
3 How the AULs can help funding agencies
We recommend that the AULs be used in a similar manner to how the ARLs have been successfully used within the Earth Science program (see Section 4. Program Performance and Budget; Friedl and Sylak-Glassman, 2020). Several examples and recommendations are given below.
3.1 Healthy program—Projects across levels
The AULs can help show that the funding programs are healthy in the respect of funding projects and research across all levels of development. A balanced portfolio may then encapsulate research proposals which will achieve 1 or 2 increases in AUL phases (e.g., a project that starts at AUL 3 and will reach an AUL 4 within the 3 years), as well as research projects which span the AUL pipeline (AULs 1–9). Looking at the funding elements in this fashion can help identify gaps in the development process funded by the current portfolio.
For example, the NASA Heliophysics program currently has Guest Investigator (NASA, 2023b) and Supporting Research (NASA, 2023c) programs which focus on basic research through analysis products which would be used by other scientists. The Living With a Star program (NASA, 2023e) supports early phase AUL research through more targeted research moving through the first few levels. The Heliophysics Data Environment Enhancement (NASA, 2023a) and Heliophysics Tools and Methods (NASA, 2023d) programs target new data products and code usability (Phases 2 and 3). Additional elements could be added as new needs are identified.
3.2 Identification of need for new models and techniques
While models and new analysis techniques have always been critical to the operation of Space Weather forecasting, they are now being incorporated in a more fundamental way to Heliophysics missions. For example, the ICON (Ionospheric CONnection) mission (Immel et al., 2018) incorporates high level geophysical models driven by ICON’s thermospheric measurements both as part of the mission science goals and as a publicly available community data product (Huba et al., 2017; Maute, 2017). During the development process of future missions with many distributed measurements—such as the Geospace Dynamics Constellation (Jaynes et al., 2019)—the incorporation of new models, coupling between models, and assimilation techniques will become even more critical. AULs offer a common language to discuss the usability of current models and techniques in a similar fashion to how the readiness of instrumentation is currently discussed with respect to mission development.
3.3 Identification of need for validation efforts and new data/missions
The AUL framework includes clear points where validation efforts are needed to be completed prior to advancement. In the framework outlined in Halford et al. (2019), a project can only claim an AUL level if it has completed all the milestones associated with it, regardless of whether work has been done on advanced milestones. For example, a project with an operational prototype (part of AUL 7) that has not undergone validation efforts (one of the milestones in AUL 6) can only be assessed at a maximum AUL level of 5, since the milestones for AUL 6 are not complete. This provides motivation for increased validation efforts within the community as well as a feedback mechanism for funding agencies. For example, if many projects are stuck at AUL 5 due to the lack of sufficient data, tools, or computational needs, this signals the need for new missions or new proposal calls to be initiated.
Another factor to consider is whether the AUL of a project may decrease over time. For example, projects that rely on real-time data inputs may lose access to these if an instrument ceases operation. A replacement data input would then be required, and the AUL would drop until validation and verification are completed again. It is important that the AULs of projects be periodically evaluated as part of the diagnostic efforts.
4 Discussion
The authors note that the AUL framework should be treated as a living framework, adjusting to user and researcher needs. The flexibility of this framework is key to its deployability across multiple projects. However, this flexibility also requires community engagement and continued discussion. Workshop sessions around roadblocks and new requirements and applications can also be guided by bringing researchers, developers, and end users together.
Here we have described how the AUL framework is generalizable to an organizational funding level. The use of ARLs by the Earth Science division shows an example of how a tracking framework can help ensure a healthy applied science funding line. We propose that AULs can act in a similar fashion for Heliophysics, tracking projects across maturity and helping find gaps in data observations and models that can benefit from new calls or new funding. By bringing the community together regularly, the AULs can be updated, and insights into common roadblocks can be found.
Data availability statement
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.
Author contributions
AH wrote a version of this manuscript as a Decadal Survey white paper, and JK expanded and revised the focus. All authors contributed to manuscript concept and revision, including reading and approving the submitted version.
Funding
This work has been completed as part of the Heliophysics Internal Scientist Funding Model (HISFM) Space Precipitation Impacts (SPI) group.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
Carter, B. A., Currie, J. L., Dao, T., Yizengaw, E., Retterer, J., Terkildsen, M., et al. (2020). On the assessment of daily equatorial plasma bubble occurrence modeling and forecasting. Space weather. 18, e2020SW002555. doi:10.1029/2020sw002555
Cid, C., Guerrero, A., Saiz, E., Halford, A. J., and Kellerman, A. C. (2020). Developing the LDi and LCi geomagnetic indices, an example of application of the AULs framework. Space weather. 18, e2019SW002171. doi:10.1029/2019sw002171
Friedl, L., and Sylak-Glassman, E. (2020). 2020 annual summary: NASA Earth science applied sciences program. Available at: https://appliedsciences.nasa.gov/sites/default/files/2021-05/ESD.Applied%20Sciences%20-%202020%20Annual%20Summary%20%281%29.pdf.
Guerrero, A., Cid, C., García, A., Domínguez, E., Montoya, F., and Saiz, E. (2021). The space weather station at the University of Alcala. J. Space Weather Space Clim. 11, 24. doi:10.1051/swsc/2021007
Halford, A. J., Kellerman, A. C., Garcia-Sage, K., Klenzing, J., Carter, B. A., McGranaghan, R. M., et al. (2019). Application usability levels: A framework for tracking project product progress. J. Space Weather Space Clim. 9, A34. doi:10.1051/swsc/2019030
Hapgood, M., Angling, M., Attrill, G., Bisi, M., Burnett, C., Cannon, P., et al. (2020). Summary of space weather worst-case environments. 2nd revised edition. Swindon, UK: Science and Technology Facilities Council. Technical Report RAL-TR-2020-005.
Huba, J. D., Joyce, G., and Fedder, J. A. (2000). Sami2 is another model of the ionosphere (Sami2): A new low-latitude ionosphere model. J. Geophys. Res. Space Phys. 105, 23035–23053. doi:10.1029/2000ja000035
Huba, J. D., Maute, A., and Crowley, G. (2017). Sami3_ICON: Model of the ionosphere/plasmasphere system. Space Sci. Rev. 212, 731–742. doi:10.1007/s11214-017-0415-z
Immel, T. J., England, S. L., Mende, S. B., Heelis, R. A., Englert, C. R., Edelstein, J., et al. (2018). The ionospheric connection explorer mission: Mission goals and design. Space Sci. Rev. 214, 13. doi:10.1007/s11214-017-0449-2
Jaynes, A., Ridley, A., Bishop, R., Heelis, R., Zesta, E., Anderson, B., et al. (2019). NASA science and Technology definition team for the Geospace Dynamics constellation final report. Washington, DC: NASA.
Klenzing, J., Halford, A., Liu, G., Smith, J. M., Zhang, Y., Zawdie, K., et al. (2023). A system science perspective of the drivers of equatorial plasma bubbles. Front. Astronomy Space Sci. 2022, 1064150. doi:10.3389/fspas.2022.1064150
Klenzing, J., Smith, J. M., Halford, A. J., Huba, J. D., and Burrell, A. G. (2022). Sami2py — overview and applications. Front. Astronomy Space Sci. 2022, 1066480. doi:10.3389/fspas.2022.1066480
Liemohn, M. W., Adam, J. G., and Ganushkina, N. Y. (2022). Analysis of features in a sliding threshold of observation for numeric evaluation (STONE) curve. Space weather. 20, e2022SW003102. doi:10.1029/2022sw003102
Liemohn, M. W., Azari, A. R., Ganushkina, N. Y., and Rastätter, L. (2020). The STONE curve: A ROC-derived model performance assessment tool. Earth Space Sci. 7, e2020EA001106. doi:10.1029/2020ea001106
Liemohn, M. W., Shane, A. D., Azari, A. R., Petersen, A. K., Swiger, B. M., and Mukhopadhyay, A. (2021). RMSE is not enough: Guidelines to robust data-model comparisons for magnetospheric physics. J. Atmos. Solar-Terrestrial Phys. 218, 105624. doi:10.1016/j.jastp.2021.105624
Maute, A. (2017). Thermosphere-Ionosphere-electrodynamics general circulation model for the ionospheric connection explorer: TIEGCM-ICON. Space Sci. Rev. 212, 523–551. doi:10.1007/s11214-017-0330-3
Merceret, F. J., O’Brien, T., Roeder, W. P., Huddleston, L. L., Bauman, W. H., and Jedlovec, G. J. (2013). Transitioning research to operations: Transforming the “valley of death” into a “valley of opportunity”. Space weather. 11, 637–640. doi:10.1002/swe.20099
Nahayo, E., Guerrero, A., Lotz, S., Cid, C., Tshisaphungo, M., and Saiz, E. (2022). Validating the LDi and LCi indices in the southern hemisphere. Space weather. 20, e2022SW003092. doi:10.1029/2022sw003092
NASA Earth Science Applied Science (2021). Expanded ARL definitions. Washington, DC: NASA. Available at: https://appliedsciences.nasa.gov/sites/default/files/2021-02/ExpandedARLDefinitions4813.pdf.
NASA (2023a). Heliophysics data environment enhancements, NNH23ZDA001N-hdee. Washington, DC: NASA. Available at: https://nspires.nasaprs.com/external/solicitations/summary!init.do?solId={FFAEE381-B6C0-F9A4-0CE1-66169DACD9E8}.
NASA (2023b). Heliophysics guest investigators-open, NNH23ZDA001N-HGIO. Washington, DC: NASA. Available at: https://nspires.nasaprs.com/external/solicitations/summary.do?solId={F67E67A4-E3F6-D0DD-CCAD-2E57D747BAC2}.
NASA (2023c). Heliophysics supporting research, NNH23ZDA001N-HSR. Washington, DC: NASA. Available at: https://nspires.nasaprs.com/external/solicitations/summary!init.do?solId={1FE03DBA-61BD-B248-316B-60ECB612CF2F}.
NASA (2023d). Heliophysics tools and methods, NNH23ZDA001N-HTM. Washington, DC: NASA. Available at: https://nspires.nasaprs.com/external/solicitations/summary!init.do?solId={17553B8F-C32A-1AD2-2A61-78670D6D21F4}.
NASA (2023e). Living with a star science, NNH23ZDA001N-LWS. Washington, DC: NASA. Available at: https://nspires.nasaprs.com/external/solicitations/summary!init.do?solId={EEA99B13-6209-5D69-4284-524A4B80ADBE}.
National Oceanic and Atmospheric Administration (2021). NAO 216-105B: Policy on research and development transitions. Washington, DC: NOAA. Available at: https://www.noaa.gov/organization/administration/nao-216-105b-policy-on-research-and-development-transitions.
Olechowski, A., Eppinger, S. D., and Joglekar, N. (2015). “Technology readiness levels at 40: A study of state-of-the-art use, challenges, and opportunities,” in 2015 Portland International Conference on Management of Engineering and Technology (PICMET) (Portland, OR, USA: IEEE), 2084–2094. doi:10.1109/PICMET.2015.7273196
Steyn, R., Strauss, D. T., Effenberger, F., and Pacheco, D. (2020). The soft X-ray Neupert effect as a proxy for solar energetic particle injection - a proof-of-concept physics-based forecasting model. J. Space Weather Space Clim. 10, 64. doi:10.1051/swsc/2020067
Keywords: space weather, Heliophysics, application usability level, operations, funding agencies
Citation: Klenzing J, Halford AJ, Kellerman A and Thompson B (2023) Using Application Usability Levels to support tracking the health of Heliophysics. Front. Astron. Space Sci. 10:1144053. doi: 10.3389/fspas.2023.1144053
Received: 13 January 2023; Accepted: 08 March 2023;
Published: 22 March 2023.
Edited by:
Nicholas Pedatella, National Center for Atmospheric Research, United StatesReviewed by:
Michael G. Henderson, Los Alamos National Laboratory (DOE), United StatesSteven Hill, NOAA Space Weather Prediction Center, United States
Copyright © 2023 Klenzing, Halford, Kellerman and Thompson. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Jeff Klenzing, jeff.klenzing@nasa.gov
†These authors have contributed equally to this work and share first authorship