Using Application Usability Levels to support tracking the health of Heliophysics

The Application Usability Level (AUL) framework describes how a project advances from basic research to operation-ready applications. Here we expand upon the current project-level usage of Application Usability Levels into a programmatic usage which can be used to help funding agencies track the health of the Heliophysics program. Examples are discussed throughout Heliophysics to show the utility of the Application Usability Level framework for describing the usability level of projects.


Introduction
Application Usability Levels (AULs) are used and often described within a research or research-to-operations context (e.g., Halford et al., 2019). While still a new framework, AULs have proven helpful to the community and have steadily grown in acceptance. For example, the United Kingdom Met office has now incorporated their use, and the United Kingdom and United States proposal calls for space weather applications also directly reference the AUL framework (e.g., Hapgood et al., 2020). However, this framework does not just describe progress at the project level, but can be used to support assessment of the health of the field overall.
This paper will outline how the AUL framework can be generalized to a funding organizational level. This extension was built in at the foundations of the framework, as part of its heritage from the Application Readiness Levels (ARLs). ARLs were developed by the Applied Sciences Program under the NASA HQ Earth Science Division (NASA Earth Science Applied Science, 2021). They have been successfully used in Earth Science for an extended period of time to track the health of the division and the roadblocks faced (e.g., Friedl and Sylak-Glassman, 2020). A framework such as the AULs or the ARLs can help guide strategic planning for new missions and new proposal calls. While the discussion will primarily focus on the AUL framework, these recommendations can apply to other readiness level frameworks, such as Technology Readiness Levels (e.g., Olechowski et al., 2015) and the Readiness Levels used by the National Oceanic and Atmospheric Administration (National Oceanic and Atmospheric Administration, 2021).
The AUL framework-when used across projects-provides a unique insight into the health of a program and where there may be common needs across the community. An overview of the phases is provided in Table 1. A healthy program should include projects across the AUL framework, ranging from exploratory research (AUL 1 through AUL 3) to research projects that are in their finalization and on-demand usage phases (AUL 6-AUL 9). The AUL framework provides an inventory of the balance across development levels, which can help determine whether the existing distribution of projects matches the needs of the community and funding agencies. In general, there are relatively many projects funded for early phase research, while a fraction of these reach the later phases. While it is expected that not every project will advance to an operational level, in general advancement of projects from mid to high level AULs is underfunded, resulting in the "Valley of Death" for research-to-operations (Merceret et al., 2013).

Current examples of AULs in Heliophysics
One advantage of the AUL concept is that is a generalized framework that can be used for a variety of project types. Below are several examples of how this framework is being used in Heliophysics across the different phases outlines in Table 1.

Proof of concept studies
Phase 1 of the AUL framework covers the discovery and viability of possible applications, including proof of concept studies. While this phase is most similar to typical research projects, there is the added step of documenting the usage beyond immediate research. An example of a current Phase 1 project discussing potential applications is a new model of Solar Energetic Particle (SEP) injection and transport. van den Berg et al. (2020) outlines a list of requirements for predictive capability of SEPs needed for future crewed missions and uses this to inform the direction of future research (AUL 2). The potential use of solar soft X-ray activity as a proxy for SEPs is proposed as a proof-of-concept study (Steyn et al., 2020), which is consistent with AUL 3 (assess viability and current state of the art).

Prediction and validation of space weather events using models
Extending into Phase 2, testing and validation is now required. This describes projects that have moved beyond basic research and into applications. For example, the routine prediction of the growth of night-time Equatorial Plasma Bubbles is a space weather problem that has been outstanding for over 80 years due to the complex interplay of physical forces that can enhance or suppress the likelihood of growth on a given night (e.g., Klenzing et al., 2023). Recent work to improve these predictions has been outlined in Section 4.4 of Halford et al. (2019) using the AUL framework. An extensive validation campaign to compare 12 months of groundbased data with long-term runs of models was recently conducted as part of the validation track of the AUL framework (Carter et al., 2020).

Development of new space weather indices
Phase 3 projects have completed validation and are being implemented into operational solutions. The AUL framework has been employed in the development and validation of the LDi and LCi geomagnetic indices (Cid et al., 2020;Nahayo et al., 2022). These new indices were developed jointly by researchers and industry users for nowcasting local geomagnetic disturbances (LDi) and for geomagnetically induced current risk (LCi). They are currently being deployed real-time on ground-based stations .

Software development
The AUL framework has motivated the discussion of new metrics for model assessment (Liemohn et al., 2020(Liemohn et al., , 2021. Additionally, it is useful for software projects that may have multiple applications, and therefore different AUL progress depending on the usage. For example, the sami2py software project consists of an open source python package that runs the SAMI2 (Sami2 is Another Model of the Ionosphere) ionospheric model (Huba et al., 2000) and archives the results. For a given version of the software, the different applications can be rated at different levels. For example, the usage of the software in early phase research projects was rated at a level of AUL 7 (application prototype) as this is actively being used by researchers, while the usage of the software as a educational tool is rated as Phase 1 since the needs of a broader user base (i.e., students from a wider range of computational backgrounds) is still actively being identified (Klenzing et al., 2022). These new user requirements will inform the future versions and releases of the software.

How the AULs can help funding agencies
We recommend that the AULs be used in a similar manner to how the ARLs have been successfully used within the Earth Science program (see Section 4. Program Performance and Budget; Friedl and Sylak-Glassman, 2020). Several examples and recommendations are given below.

Healthy program-Projects across levels
The AULs can help show that the funding programs are healthy in the respect of funding projects and research across all levels of development. A balanced portfolio may then encapsulate research proposals which will achieve 1 or 2 increases in AUL phases (e.g., a project that starts at AUL 3 and will reach an AUL 4 within the 3 years), as well as research projects which span the AUL pipeline (AULs 1-9). Looking at the funding elements in this fashion can help identify gaps in the development process funded by the current portfolio.  (NASA, 2023a) and Heliophysics Tools and Methods (NASA, 2023d) programs target new data products and code usability (Phases 2 and 3). Additional elements could be added as new needs are identified.

Identification of need for new models and techniques
While models and new analysis techniques have always been critical to the operation of Space Weather forecasting, they are now being incorporated in a more fundamental way to Heliophysics missions. For example, the ICON (Ionospheric CONnection) mission (Immel et al., 2018) incorporates high level geophysical models driven by ICON's thermospheric measurements both as part of the mission science goals and as a publicly available community data product (Huba et al., 2017;Maute, 2017). During the development process of future missions with many distributed measurements-such as the Geospace Dynamics Constellation (Jaynes et al., 2019)-the incorporation of new models, coupling between models, and assimilation techniques will become even more critical. AULs offer a common language to discuss the usability of current models and techniques in a similar fashion to how the readiness of instrumentation is currently discussed with respect to mission development.

Identification of need for validation efforts and new data/missions
The AUL framework includes clear points where validation efforts are needed to be completed prior to advancement. In the framework outlined in Halford et al. (2019), a project can only claim an AUL level if it has completed all the milestones associated with it, regardless of whether work has been done on advanced milestones. For example, a project with an operational prototype (part of AUL 7) that has not undergone validation efforts (one of the milestones in AUL 6) can only be assessed at a maximum AUL level of 5, since the milestones for AUL 6 are not complete. This provides motivation for increased validation efforts within the community as well as a feedback mechanism for funding agencies. For example, if many projects are stuck at AUL 5 due to the lack of sufficient data, tools, or computational needs, this signals the need for new missions or new proposal calls to be initiated.
Another factor to consider is whether the AUL of a project may decrease over time. For example, projects that rely on real-time data inputs may lose access to these if an instrument ceases operation. A replacement data input would then be required, and the AUL would drop until validation and verification are completed again. It is important that the AULs of projects be periodically evaluated as part of the diagnostic efforts.

Discussion
The authors note that the AUL framework should be treated as a living framework, adjusting to user and researcher needs. The flexibility of this framework is key to its deployability across multiple projects. However, this flexibility also requires community engagement and continued discussion. Workshop sessions around roadblocks and new requirements and applications can also be guided by bringing researchers, developers, and end users together.
Here we have described how the AUL framework is generalizable to an organizational funding level. The use of ARLs by the Earth Science division shows an example of how a tracking framework can help ensure a healthy applied science funding line. We propose that AULs can act in a similar fashion for Heliophysics, tracking projects across maturity and helping find gaps in data observations and models that can benefit from new calls or new funding. By bringing the community together regularly, the AULs can be updated, and insights into common roadblocks can be found.

Data availability statement
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

Author contributions
AH wrote a version of this manuscript as a Decadal Survey white paper, and JK expanded and revised the focus. All authors contributed to manuscript concept and revision, including reading and approving the submitted version.

Funding
This work has been completed as part of the Heliophysics Internal Scientist Funding Model (HISFM) Space Precipitation Impacts (SPI) group.