User centric metrics for a realistic assessment of broadband services

1 Introduction

Broadband Internet access, originally a nice to have yet expensive service for basic communications, has become a utility for households around the world. The value of broadband to families and individuals has been constantly increased by the emergence of new, productivity, education, and entertainment internet-based applications. These applications provide measurable and sustained improvements to the wellbeing and wealth of households, yet accessibility and utility depend on the availability, quality, and affordability of internet access services.

In economic terms, homes with access to broadband are valued higher than those lacking it (Molnar et al., 2015). Rural broadband access is associated with significant increases in agricultural yields (LoPiccalo, 2020). Broadband enables content rich collaboration and access to enterprise tools which facilitate remote work arrangements. Remote work is a major socio-economic opportunity (De et al., 2020) that became critical during the COVID-19 pandemic. In rural communities where remote work is less common, broadband access is a key economic enabler of precisions agriculture and data-based operation of farm and rural industries (Smith, 2020; Hennessy et al., 2016). The systemic lack of broadband access or at least internet access in rural areas (Perrin, 2019) significantly limits the economic opportunities for rural communities (Whitacre et al., 2014; Kandilov et al., 2017).

Modern K-12 and Higher-Ed education is more and more dependent on internet resources, from collaboration tools to educational resources, and administrative processes. Access to the Internet is critical to the success of students at all levels of their education. Middle and high school students with high-speed Internet access at home have more digital skills, higher grades, and perform better on standardized tests, such as the SAT and are more likely to attend college or university (Hampton et al., 2020). Higher education requires access to high-speed internet for class and project participation and for homework submissions. All the opportunities provided and needs met by broadband access were underlined by the COVID-19 pandemic (Lai and Widmar, 2020). Households without quality or even basic broadband access faced significant challenges in negotiating quarantine constraints. During the COVID-19 pandemic, the digital divide between those who have and those who do not have broadband access at home was clearly underlined.

Finally, Internet connectivity and particularly access to broadband, is playing an increasingly important role in both healthcare and public health (Bauerly et al., 2019). Population based medicine and the increasing application of telemedicine and outpatient care depend on good Internet connectivity to facilitate video conferencing and remote instrumentation. These services are especially important in rural areas seeing a continued decrease in the number of regional hospitals and clinics. Yet again, the pandemic underlined the importance of these services to scaling up medical resources as needed and where needed to address crisis scenarios.

2 Challenges of broadband internet access availability

The value of broadband to all households, regardless of geographical location has been repeatedly demonstrated by data. The improvements to the wellbeing of families naturally draw the attention of local, state, and federal governments to the need to push for and fund broadband services everywhere. The visibility and the impetus of these initiatives increased in recent years due to socio-economic drivers, yet the results are lagging, particularly when looking closer at service footprints and quality. While broadband access deserts can be found in even the largest urban areas, the problem is particularly acute in rural areas where only 51.6% of residents had broadband access as defined by the Federal Communications Commission (FCC) (Federal Communications Commission, 2020).

According to the FCC broadband (high-speed internet access) is currently defined as internet service with speeds of at least 25 Mbps download and 3 Mbps upload speeds (25/3 Mbps) (Federal Communications Commission, 2020). Communities with access to service between 25/3 Mbps and 10/1 Mbps are categorized as underserved while communities with service below 10/1 Mbps are identified as unserved. Internet access services with broadband speeds are not available to everyone, with broadband access deserts present in both rural and urban areas. While these challenges exist at the service level definitions mentioned above, it is worth noting that experts agree, a more appropriate definition of the broadband service should require a minimum of 100 Mbps to accommodate multiple devices and some of the key applications used today.

While service availability is an important factor, adoption of broadband also depends on the cost of these services. According to data collected by the Organization for Economic Co-operation and Development (OECD), the average cost of broadband services in US is $61.07 and $0.43/Mbps (OECD). These costs are significantly higher in the US than in many other countries in the World. By comparison, an Eastern European Country such as Romania has an average cost for broadband service of $8.15 or $0.05/Mbps. For this comparison, broadband services cost in US per average income is twice as high as in Romania.

The current state of broadband availability and adoption is challenging and while the causes of the current state are not the subject of this paper, it is worth noting that lack of data on the true state of broadband access combined with regulatory issues thwarted past attempts to comprehensively address the problem. With the renewed, and funded focus on broadband, it is imperative to approach this issue in a pragmatic and data driven way from day one.

3 Measuring the availability and performance of broadband services

Improving broadband access, quality and affordability is an effort that requires a quantitative understanding of the current state and a way to track progress of funded or regulatory driven initiatives. Past efforts to drive broadband adoption highlighted the truism that we cannot manage something we cannot accurately, reliably, and consistently measure. The self-reported data collected from service providers via FCC Form 477 (Federal Communications Commission, 2020) is inherently coarse and does not always agree with data collected directly from users. Figure 1 shows the discrepancy between self-reported data and data collected directly from the citizens in a study run by Merit Network in Berrien County, Michigan (Berrien).

FIGURE 1

FIGURE 1. FCC broadband availability data vs. data collected from end-users. Adapted from Berrien County Report, with permission from Berrien County.

The same study also states: “Out of 1,062 census blocks, we have found that in 76% of these, based on survey responses, there were customers who could not obtain service at their address though they are located in a census block that is reported as served based on FCC maps.” This underlines the limitations of census block level self-reporting.

The study went beyond surveying service availability and touched on another critical point: Quality of service. The study observed that 55% of users with >100/100 Mbps, 66% of those with 25/3 Mbps and 77% of those with 10/1 Mbps felt there are not enough broadband service options. Users who have some levels of service are generally unhappy with them and would be very interested in switching providers if available as shown in Figure 2.

FIGURE 2

FIGURE 2. User satisfaction with internet access services in Berrien County, MI. Adapted from Berrien County Report, with permission from Berrien County.

The need exists, 33% of households without broadband service are willing to pay over 23% above the average cost of service. The pain exists as well, with over 78% of households with broadband access being interested in switching providers.

The key takeaway is that the data we have been using to make funding and policy decisions with respect to broadband access development is neither accurate nor complete in measuring the experience users have with the service. We must change both the types of data we collect and the way we collect them, we must focus on service availability, service performance, and on service quality. Under the secretary Rosenworcel, the FCC recognizes some of these shortcomings of the past methodologies: “Collecting data from consumers who are directly affected by the lack of access to broadband will help inform the FCC’s mapping efforts and future decisions about where service is needed” (FCC, 2021).

To ensure effective funding and management of broadband initiatives, the focus must be on a more comprehensive approach to service assessment, one that accounts for service performance and sustained service quality over time. To meet this goal, we require a better assessment of the current state by adopting user centric metrics and deploying the instrumentation and processes that will enable accurate tracking of these metrics over time.

4 User centric metrics to assess and track broadband services

The common metrics used in defining and assessing broadband services are technology centric: bandwidth (speed) up and down and in some cases, latency (the average response time), drops (the average number of dropped packets over a period), and service availability (uptime). These metrics have been measured in various ways over time and have been commonly used in a variety of past reports (Bauer et al., 2010). Measuring these basic metrics, particularly to commonly used speed tests, has its own challenges as summarized by Feamster and Livingood (Feamster and Livingood, 2020). The traditional approach to measuring and documenting broadband service availability was recognized to be limited for the past decade (Grubesic, 2012) yet it continued to be used. The major shortcoming of this traditional view on measuring broadband access is that it does not reflect the real user experience with the service (Feamster and Livingood, 2020).

Broadband services should be measured in terms of their capability to support the applications users need, to measure the consistency over time in delivering those applications, and finally, the Return on Investment (ROI) for the user, or the economic value of the service. We will analyze each of these three components individually and then discuss options for describing a monitored broadband service by using the three metrics.

4.1 User experience

To measure user experience (UX) of a broadband service, we first need to identify a set of applications or general workflows that are expected to be supported by Internet access, individually and within a usage profile. For each one of these applications, we can identify an application specific metric to measure user experience. Table 1 lists the main applications to be considered and the corresponding metric.

TABLE 1

TABLE 1. Typical applications used by broadband subscribers and related UX metrics.

All the metrics listed in Table 1 can be inferred by individually collecting specific network and application response information such as: throughput, latency, jitter, drops, and DNS response time. However, applications typically use multiple infrastructure and cloud services delivered by broadband access providers which themselves impact UX. For example, a speed test’s results will depend on the DNS resolution times, on routing done by the provider and its upstream peers and finally on the end-to-end quality of the path, assuming the target is consistent in its performance.

The data can be collected by monitoring live traffic or by synthetic transactions. The former raises privacy concerns but with proper analysis can capture the true user experience of end users. The latter is a simulation of traffic generated by an application or an end-point device making it a representation of the actual UX. Synthetic transactions can easily simulate the usage profile and are highly deterministic. It is important to note that the renewed efforts of FCC to assess broadband service availability and quality are focused on user experience rather than singular network centric metrics (FCC Announces). The Measuring Broadband America initiative employs a user experience focused approach and uses synthetic transactions for assessment (Measuring Broadband, 2022).

Measuring a single application, as commonly done in the past, for example, measuring file transfer times (speed tests), is not sufficient to describe the overall user experience. Using a traffic monitoring approach, a common usage profile can be statistically extracted from the monitored traffic. However, this approach makes it difficult to consistently compare performance amongst users across wide service areas. Traffic monitoring-based evaluations are better suited for assessing the performance of individual applications. By contrast, synthetic transactions are better suited for assessing the performance of broadband services through deterministic measurements where a likely traffic profile consisting of a set of simulated applications can be used for consistent evaluation. An example profile could be to take UX measurements for simulated traffic specific to the applications in Table 1. The synthetic traffic-based approach and the use of set traffic profiles, in this case reflecting usage profiles, is in line with benchmarking methodologies used in Internet technologies performance evaluations (RFC 2544, 1999). It facilitates consistent testing, monitoring, analysis, and reporting.

User experience should be measured for each service considered in the usage profile. While application specific UX metrics can be compared across evaluated broadband services, the overall assessment of a service which consists of metrics for multiple applications becomes challenging. Moreover, the more metrics are involved in the service assessment, the harder it is to convey the results to end-users.

To simplify reporting, the individual UX measurements can be normalized to service target values and weight averaged according to the usage profile to provide a single number characterizing the overall quality (QS) of the broadband service observed. For example, in the case of the profile proposed in Table 1, the normalized metrics are calculated as follows:

${\text{NUX}}_{\text{VoIP}}=\frac{{\text{UX}}_{\text{VoIP}}}{{\mathbf{S}}_{\text{VoIP}}},{\text{NUX}}_{\text{Video}}=\frac{{\text{UX}}_{\text{Video}}}{{\mathbf{S}}_{\text{Video}}},{\text{NUX}}_{\text{FT}}=\frac{{\text{UX}}_{\text{FT}}}{{\mathbf{S}}_{\text{FT}}},{\text{NUX}}_{\text{Web}}=\frac{{\text{UX}}_{\text{Web}}}{{\mathbf{S}}_{\text{Web}}},{\text{NUX}}_{\text{DNS}}=\frac{{\mathit{U}\mathit{X}}_{\mathit{D}\mathit{N}\mathit{S}}}{{\mathit{S}}_{\mathit{D}\mathit{N}\mathit{S}}}$

where NUX_i represents the normalized value of measured (average) UX_i and S_i is the satisfactory value for UX_i. All normalized UX values greater than 1 are set to 1. The weights reflect the likelihood of use or the value to user: W_VoIP (weight VoIP), W_Video (weight Video), W_FT (weight File Transfer), W_Web (weight Web).

$\mathit{Q}\mathit{S}=\frac{{\text{NUX}}_{\text{VoIP}}*{\mathbf{W}}_{\text{VoIP}}+{\text{NUX}}_{\text{Video}}*{\mathbf{W}}_{\text{Video}}+{\text{NUX}}_{\text{FT}}*{\mathbf{W}}_{\text{FT}}+{\mathit{N}\mathit{U}\mathit{X}}_{\mathit{W}\mathit{e}\mathit{b}}*{\mathit{W}}_{\mathit{W}\mathit{e}\mathit{b}}+{\mathit{N}\mathit{U}\mathit{X}}_{\mathit{D}\mathit{N}\mathit{S}}*{\mathit{W}}_{\mathit{D}\mathit{N}\mathit{S}}}{{\mathit{W}}_{\text{VoIP}}+{\mathit{W}}_{\text{Video}}+{\mathit{W}}_{\text{FT}}+{\mathit{W}}_{\mathit{W}\mathit{e}\mathit{b}}+{\mathit{W}}_{\mathit{D}\mathit{N}\mathit{S}}}$

The overall service quality number is compared against service targets. As far as users are concerned, bandwidths and speeds are irrelevant if the applications they need are not working well. As far as engineers and operations staff are concerned, the more specific metrics related to individual applications and the infrastructure related metrics such as latency, jitter, drops, and DNS response time are more relevant to understanding the causes for the observed performance.

4.2 Consistency of user experience

The second dimension of assessing a broadband service relates to measuring the consistency of the quality of service. Many infrastructures providing broadband access today are shared amongst multiple users (George, 2019; Hoffman, 2017). This results in dramatic changes in the quality of service when many users are online at the same time. A 25/3 broadband service can quickly degrade to a nonbroadband service when experiencing load. This degradation can also happen at the most inconvenient times thus rendering the service sub-optimal. Moreover, many broadband infrastructures, particularly in sub-urban and rural areas have suboptimal distribution infrastructures that are not buried thus exposed to factors leading to interruption of service Heidemann et al. These service interruptions must be taken into consideration when assessing the overall quality of the service provided. Making 25/3 service available only during some days and some hour during the day can technically qualify as broadband by the FCC definition but it is not consistently delivering the expected user experience. It is thus important to monitor the performance of a service, through the UX metrics, over time.

Quantifying the consistency of user experience with a service can be done using APDEX (The APDEX users group, 2023). APDEX is a simple measure of the proportion of measured data points that are within the range of acceptable service quality compared to the proportion of measured data points that are not. For a given application “q”, the UX measurements over a period are placed into three buckets: Satisfactory UX, Tolerated UX, Frustrated UX. These buckets are separated by two predefined thresholds. The APDEX of application “q” over the period when the data was collected is calculated as:

${\mathit{A}\mathit{P}\mathit{D}\mathit{E}\mathit{X}}_{\mathit{q}}=\frac{\mathbf{S}\mathbf{a}\mathbf{t}\mathbf{i}\mathbf{s}\mathbf{f}\mathbf{a}\mathbf{c}\mathbf{t}\mathbf{o}\mathbf{r}\mathbf{y}+\mathbf{T}\mathbf{o}\mathbf{l}\mathbf{e}\mathbf{r}\mathbf{a}\mathbf{t}\mathbf{e}\mathbf{d}*\mathbf{0.5}}{\mathit{S}\mathit{a}\mathit{t}\mathit{i}\mathit{s}\mathit{f}\mathit{i}\mathit{e}\mathit{d}+\mathit{T}\mathit{o}\mathit{l}\mathit{e}\mathit{r}\mathit{a}\mathit{t}\mathit{e}\mathit{d}+\mathit{F}\mathit{r}\mathit{u}\mathit{s}\mathit{t}\mathit{r}\mathit{a}\mathit{t}\mathit{e}\mathit{d}}$

APDEX should be evaluated on a per application basis and the acceptable service threshold should also be defined for each application. Whenever the overall Internet Access is down, the failing scheduled UX measurements should be placed in the Frustrated bucket for each application assessed.

Similar to the challenge of determining the quality of a service when measuring the UX of individual applications, the APDEX of the broadband service could be a single number calculated as the weighted average of the APDEX scores for each monitored application. The weights used are the same as the ones used to calculate the quality of service described in the section above.

$\mathit{A}\mathit{P}\mathit{D}\mathit{E}\mathit{X}=\frac{{\text{APDEX}}_{\text{VoIP}}*{\mathbf{W}}_{\text{VoIP}}+{\text{APDEX}}_{\text{Video}}*{\mathbf{W}}_{\text{Video}}+{\text{APDEX}}_{\text{FT}}*{\mathbf{W}}_{\text{FT}}+{\mathbf{A}\mathbf{P}\mathbf{D}\mathbf{E}\mathit{X}}_{\mathit{W}\mathit{e}\mathit{b}}*{\mathit{W}}_{\mathit{W}\mathit{e}\mathit{b}}+{\mathbf{A}\mathbf{P}\mathbf{D}\mathbf{E}\mathit{X}}_{\mathit{D}\mathit{N}\mathit{S}}*{\mathit{W}}_{\mathit{D}\mathit{N}\mathit{S}}}{{\mathit{W}}_{\text{VoIP}}+{\mathit{W}}_{\text{Video}}+{\mathit{W}}_{\text{FT}}+{\mathit{W}}_{\mathit{W}\mathit{e}\mathit{b}}+{\mathit{W}}_{\mathit{D}\mathit{N}\mathit{S}}}$

It should be apparent that, like UX measurements, the APDEX measurements depend on the performance of the applications delivery infrastructure and not just the broadband access service quality.

4.3 ROI of broadband services

Finally, the third dimension in assessing broadband services is the return on investment (ROI) for individual households. Broadband must become a utility like water and electricity, however, before we achieve that goal with all its regulatory implications, internet access service offerings varied significantly in terms of economic value delivered. Many communities, especially rural and low-income communities are not covered by competitive broadband services, and they will pay disproportionately for the level of service. This opens the door for providers who fill in the coverage gaps with economically suboptimal solutions funded by broadband initiatives. These solutions will deliver some level of Internet access at prices driven by lack of supply rather than market competitiveness and delivery costs. In rural areas in particular, new companies are employing old technologies such as mesh WiFi (Moreno et al., 2013) to build infrastructure to support Internet access. New companies are exploring new technologies such as Citizens Broadband Radio Service (CBRS) wireless communications (Mun, 2017). While some of these solutions hold positive potential for success, in many cases they are not yet delivering true broadband access while charging significant subscription fees.

State and federal investments in broadband initiatives have proved to be effective according to an extended study performed on investments made across OECD member states (Briglauer and Grajek, 2021). The increases in per capita GDP, in the short and medium term, justify the investments in supporting broadband service coverage expansion. The benefits of these initiatives will depend on the quality of the services and on their affordability. Affordability and not just availability of service correlates with direct positive economic outcomes such as increased employment (Reddick et al., 2020; Zuo, 2021). According to a Pew Research Center report, the average income in areas most affected by a lack of broadband access is less than $30 k per year (Vogels, 2021). Moreover, over 34% of the households in the same income bracket stated they had trouble paying for broadband access service during the COVID-19 pandemic (McClain, 2023). For these reasons, broadband investments funded or subsidized by federal, state, and local governments must be evaluated in terms of both performance and ROI to users.

Specifically, targets should be set for the expected monthly subscription for a service with a measured QS and APDEX. Further analysis can be done on the economic value of the service by calculating the broadband ROI which can be performed based on usage profiles being tracked, usage profiles mapped to specific economic benefits. Each area of economic opportunity, education, healthcare, work mobility, etc., is enabled by the application types captured in the profile. The approach proposed in this paper allows for state and federal agencies to set affordability targets at community level based on specific socio-economic criteria yet still be able to monitor and assess broadband services at a larger scale. The three dimensions of assessing broadband: QS, APDEX, and ROI can be tracked independently or combined into a single metric. Both approaches are acceptable if the methodology of calculating composite metrics is clear and understood. Collecting the data for these metrics requires a systematic approach to engaging the users and the tools to monitor the services on an ongoing basis.

4.4 IPv6 considerations

One consideration in assessing the quality of broadband services is the availability of IPv6 and the quality of the infrastructure supporting it. Data transport and the protocols used in the process should be transparent to the user so whether the next-generation of the Internet Protocol is used or not, should in principle be of no consequence to the user experience. Nevertheless, over the past 5 years, broadband providers accelerated their plans to deploy IPv6 at the access layer to mitigate the exhaustion of IPv4 address space. These deployments vary in quality (architecture, use of translation) which in practice means users will have different experiences with applications running over IPv4 vs. IPv6 (Pickard et al., 2019). It is thus recommended that the three metrics defined in the earlier sections should be measured across each protocol independently and reported accordingly. The methodology for measuring IPv6 vs. IPv4 performance was described by the author in (Pickard et al., 2019) and it uses the guidelines shown in RFC 5180 (Popoviciu et al., 2022). If deployed properly, IPv6 has been shown to deliver improved user experience due to simpler transport infrastructures.

When services are evaluated for regulatory reasons or for federal and state local funding support, effective support of IPv6 should be considered an important criterion. This is a measure of future proofing of services. IPv6 is the current plan of record for the networking industry, and it provides the foundation for more diverse, more cost-effective services to be provided in the future.

5 Monitoring broadband initiatives

One of the key objections to the adoption of the proposed approach to assessing broadband services is the perception that it would require a very complex set of tools and analysis capabilities. This is a tractable problem that can be solved at scale with investments that would represent a fraction of the investments made to address the broadband adoption issues. In fact, all local, state, and federal investments should include single digit percentage allocations to managing these investments. These allocations should be used for instrumentation, monitoring, and reporting purposes, leading to accountable and delivery optimized initiatives.

The process of developing and executing an effective broadband strategy at local, state, and federal levels is complex and requires a systematic approach. The first step is to identify the meaningful metrics that will be used to understand the problem and to measure the outcomes of the solution. The metrics proposed by the author in the previous sections would enable the development of a strategy centered around the users, not the technology. Next, it is important to prepare the tools that will be used to collect the relevant data. These tools cover various data collection methods: surveys, polls, instrumentation. These initial steps will establish a clear, consistent framework for managing the initiative.

The third step is to develop a clear baseline for the current state, a baseline based on the metrics identified in step one. This baseline should be determined by engaging the community and applying monitoring tools to collect the relevant data from the users themselves. These steps will help build a more realistic picture of the current state, facilitate subsequent monitoring, and provide the data needed to evaluate the impacts of the initiative beyond simple adoption.

Lastly, instrumentation should be deployed to enable continued monitoring of the service quality. This instrumentation can be achieved through a crowdsourcing model or through an incentive program. The involvement of the community requires continuous, public reporting to actively show the value of the initiative to the end users and the positive outcomes of the community’s involvement. This reporting will also be a mechanism to offer quantitative guidance to service providers, both incumbents and new entrants on how to align their strategies with the goals of the initiative.

There are multiple options for assigning responsibility for the instrumentation and data collection, with options ranging from crowd sourcing with a state run backend, to entrusting service providers, to funding a third party to manage the mapping and monitoring of the quality of broadband access services. The model chosen should align with the primary sponsor of the broadband initiative. Research shows that state-level funding leads to an average increase in broadband availability by one to two percent (Whitacre and Gallardo, 2020). For this type of initiatives which often have dedicated staff, monitoring the availability and quality of broadband services can be led by the dedicated state office with technical support provided by the local REN (Research and Education Network) providers. Private-public partnerships and cooperatives are another driver for broadband adoption in smaller communities (Falch and Henten, 2010; Girth, 2014; Sadowski, 2017) with active development of service enablement and operation. For these initiatives, the communities can assume joint ownership of monitoring the progress and quality of deployment with the local provider or cooperative since transparency is expected. The technical aspects of data collection can be mitigated using paid monitoring services funded through the initial budget.

It is worth taking a moment to discuss the options available to underserved communities which might not have the financial, economic, political resources to drive broadband initiatives even though the state and federal governments might provide commercial organizations with the incentives to deliver broadband services in those communities. A good example in the US are tribal areas where lack of broadband access is a significant constraint to the wellbeing of the community (Mack et al., 2022; Blackwater, 2023; Hutto and Wheeler, 2023). These communities must assert their right to assess and track the quality of the services claimed in their name. Underserved communities should be provided direct funding to support the effort of independent monitoring of broadband services, to enable them to qualify the quality of the services provided. This function can be implemented through partnerships with academic institutions or with hired third party experts.

The author recommends the involvement of third-party experts who can be funded at state or federal level to provide the data needed to measure the return on the investment in broadband initiatives. A limited number of accredited organizations could step into this role, with the express requirement to adhere to common standards as described in this paper. This model would provide the level of data consistency needed for effective management across a state or across the country.

In an upcoming paper, the author will present recommendations for instrumentation, data management and data reporting to enable and manage effective broadband initiatives at local, state, and federal levels. The technology involved in the proposed instrumentation, the costs of deployment and operation, and the ways in which deployment can be facilitated by existing social and civic structures and services will be described.

6 Conclusion

Broadband Internet Access has become an essential service to the wellbeing of individuals, households, communities, and the country. The proven economic, social, health and educational benefits of broadband services are driving a renewed interest and effort to make the service ubiquitously available and delivering the performance necessary to support modern applications. The COVID pandemic underlined the significant challenges faced by communities lacking broadband access services and was a catalyst for the increased local, state, and federal level funding made available to support broadband initiatives.

A successful broadband initiative design and execution at local, state, or federal level depends on the metrics used to baseline the current state and track progress towards the target state. Past efforts were plagued using less relevant metrics and by data collection methods that are unreliable or not granular enough.

In this paper we propose a change in perspective with respect to broadband service assessment. We identify three dimensions of assessment: Quality of service (QS)—a user experience centric metric, Consistency of Service–an APDEX based evaluation of the quality of the user experience over time, and ROI–an economic value assessment of the service. The metrics are calculated based on elemental infrastructure data collected from the user premise over time. They are assessing the quality of the broadband service based on its ability to support a good user experience for a profile of relevant applications over time.

The case is made for the use of more complex metrics that will account for the true value delivered by the services to the communities it serves. Methods for collecting the necessary data are discussed and the case for the use of synthetic transactions is made, an approach that addresses consistency and control of measurements as well as privacy concerns. The paper is also addressing the concerns related to the feasibility and scalability of employing such methods to monitor broadband initiatives. Details of a working set of tools and the recommendations for applying them in practice are the subject of a follow up paper.

Data availability statement

The original contributions presented in the study are included in the article/supplementary materials, further inquiries can be directed to the corresponding author.

Author contributions

The author confirms being the sole contributor of this work and has approved it for publication.

Acknowledgments

The author would like to thank Phil Lunsford for his encouragement and insightful feedback.

Conflict of interest

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Bauer, S., Clark, D. D., and Lehr, W. (2010). “Understanding broadband speed measurements,” in Technology policy research conference (USA: TPRC).

Google Scholar

Bauerly, B. C., Hulkower, R., McCord, R. F., and Pepin, D. (2019). Broadband access as a public health issue: The role of law in expanding broadband access and connecting underserved communities for better health outcomes. doi:10.1177/1073110519857314

CrossRef Full Text | Google Scholar

Berrien county broadband internet task force. Available at: https://www.berriencounty.org/DocumentCenter/View/13977/BCBIT-Executive-Findings-Report-PDF.

Blackwater, D. (2023). Broadband internet access: A solution to tribal economic development challenges. Available at: https://heinonline.org/HOL/LandingPage?handle=hein.journals/indipeor6&div=8&id=&page=.

Google Scholar

Briglauer, W., and Grajek, M. (2021). Effectiveness and efficiency of state aid for new broadband networks: Evidence from OECD member states”, ESMT Working Paper, No. 21-01. Berlin: European School of Management and Technology ESMT. Available at: https://nbn-resolving.de/urn:nbn:de:101:1-2021082508030470783534.

Google Scholar

De, R., Pandey, N., and Pal, A. (2020). Impact of digital surge during covid-19 pandemic: A viewpoint on research and practice. Available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7280123/.

Google Scholar

Falch, M., and Henten, A., Public private partnerships as a tool for stimulating investments in broadband, Telecommun. Policy, Volume 34, 496, 504. 2010. doi:10.1016/j.telpol.2010.07.010

CrossRef Full Text | Google Scholar

FCC Announces FCC announces new outreach to collect consumer broadband availability experiences. Available at: https://docs.fcc.gov/public/attachments/DOC-370978A1.pdf.

Google Scholar

FCC (2021). FCC announces new outreach to collect consumer broadband availability. Available at: https://docs.fcc.gov/public/attachments/DOC-370978A1.pdf.

Google Scholar

Feamster, N., and Livingood, J. (2020). Measuring internet speed: current challenges and future recommendations. Commun. ACM 63 (12), 72–80. doi:10.1145/3372135

CrossRef Full Text | Google Scholar

Federal Communications Commission (2020). Broadband deployment report. Washington DC: Broadband Progress Report No. Available at: https://docs.fcc.gov/public/attachments/FCC-20-50A1.pdf.

Google Scholar

Federal Communications Commission Broadband speed guide. Available at: https://www.fcc.gov/consumers/guides/broadband-speed-guide.

Google Scholar

Federal Communications Commission Broadband speed guide. Available at: https://www.fcc.gov/form-477-census-tract-data-internet-access-services.

Google Scholar

George, N. (2019). Which providers’ internet speeds live up to the hype—actual vs. Advertised speeds. AllConnect.com. Available at: https://www.allconnect.com/blog/advertised-vs-actual-internet-speeds.

Google Scholar

Girth, A. M. (2014). What drives the partnership decision? Examining structural factors influencing public-private partnerships for municipal wireless broadband. Int. Public Manag. J. 17 (3), 344–364. doi:10.1080/10967494.2014.935240

CrossRef Full Text | Google Scholar

Grubesic, T. (2012). The U.S. National broadband map: data limitations and implications. Telecommun. Policy 36 (2), 113–126. doi:10.1016/j.telpol.2011.12.006

CrossRef Full Text | Google Scholar

Hampton, K. N., Fernandez, L., Robertson, C. T., Bauer, J. M., James, H., and Quello Center, M. B. (2020). Broadband and student performance gaps. Michigan State University. doi:10.25335/BZGY-3V91

CrossRef Full Text | Google Scholar

Heidemann, J., Pradkin, Y., and Baltra, G. The policy potential of measurements of internet outages (march 16, 2018). TPRC 46: the 46th research conference on communication. Available at: https://ssrn.com/abstract=3141698.

Google Scholar

Hennessy, T., Läpple, D., and Moran, B. (2016). The digital divide in farming: A problem of access or engagement? Appl. Econ. Perspect. Policy 38 (3), 474–491. doi:10.1093/aepp/ppw015

CrossRef Full Text | Google Scholar

Hoffman, C. 2017. Why you probably aren't getting the internet speeds you're paying for (and how to tell). Available at: https://www.howtogeek.com/165321/why-you-probably-arent-getting-the-internet-speeds-youre-paying-for-and-how-to-tell/.

Google Scholar

Hutto, H. D., and Wheeler, M. B. (2023). Tribal and rural digital inclusivity: an examination of broadband access in two neighboring Great Plains states. First Monday 28 (4). doi:10.5210/fm.v28i4.12519

CrossRef Full Text | Google Scholar

Kandilov, H., Amy, M. G., Kandilov, I. T., Liu, X., and Renkow, M. (2017). The impact of broadband on US agriculture: an evaluation of the USDA broadband loan program. Appl. Econ. Perspect. Policy 39 (4), 635–661. doi:10.1093/aepp/ppx022

CrossRef Full Text | Google Scholar

Lai, J., and Widmar, N. (2020). Revisiting the digital divide in the COVID-19 era. Appl. Econ. Perspect. Policy 43, 458–464. doi:10.1002/aepp.13104

PubMed Abstract | CrossRef Full Text | Google Scholar

LoPiccalo, K. (2020). Impact of broadband penetration on U.S. Farm productivity. Available at: https://docs.fcc.gov/public/attachments/DOC-368773A1.pdf.

Google Scholar

Mack, E. A., Helderop, E., Keene, T., Loveridge, S., Mann, J., Grubesic, T. H., et al. (2022). A longitudinal analysis of broadband provision in tribal areas. Telecommun. Policy 46, 102333. doi:10.1016/j.telpol.2022.102333

CrossRef Full Text | Google Scholar

McClain, C. (2023). 34% of lower-income home broadband users have had trouble paying for their service amid COVID-19 June 3, 2021. Available at: https://www.pewresearch.org/fact-tank/2021/06/03/34-of-lower-income-home-broadband-users-have-had-trouble-paying-for-their-service-amid-covid-19/.

Google Scholar

Measuring Broadband (2022). Measuring broadband America. Available at: https://www.measuringbroadbandamerica.com/test-methodology.

Google Scholar

Molnar, G., Savage, S., and Sicker, D. (2015).The impact of high-speed broadband availability on real estate values: evidence from United States property markets, TPRC 41: The 41st Research Conference on Communication, Information and Internet Policy, 30 Aug 2022, China. IEEE. doi:10.2139/ssrn.2241926

CrossRef Full Text | Google Scholar

Moreno, C., Roro, Z., Tucker, W., Siya, M., Bidwell, N., and Reigadas, J. (2013). Experiences, challenges and lessons from rolling out a rural WiFi mesh network. Proc. 3rd ACM symposium Comput. Dev. 11, 1. doi:10.1145/2442882.2442897

CrossRef Full Text | Google Scholar

Mun, K.C. B. R. S. (2017). New shared spectrum enables flexible indoor and outdoor mobile solutions and new business models; CBRS white paper; mobile experts: Silicon valley. Available at: https://www.federatedwireless.com/wp-content/uploads/2017/09/Mobile-Experts-CBRS-Overview.pdf.

Google Scholar

OECD 2022, Access network speed tests", OECD digital economy papers. Paris: OECD Publishing. No. 237. doi:10.1787/5jz2m5mr66f5-en

CrossRef Full Text | Google Scholar

Perrin, A. (2019). Digital gap between rural and nonrural America persists. Available at: https://www.pewresearch.org/fact-tank/2019/05/31/digital-gap-between-rural-and-nonrural-america-persists/.

Google Scholar

Pickard, J., Drummond, D., Lunsford, P., and Popoviciu, C. (2019). IPv6 user experience: An assessment of top US university domains”. USA: ASEE Annual Conference.

Google Scholar

Popoviciu, C., Hamza, A., Van de Velde, G., and Dugatkin, D. 2022, IPv6 benchmarking methodology for network interconnect devices”, RFC 5180. China: IETF. Available at: https://www.rfc-editor.org/rfc/rfc5180.html.

Google Scholar

Reddick, C. .G., Enriquez, R., Harris, R. .J., and Sharma, B. (2020). Determinants of broadband access and affordability: an analysis of a community survey on the digital divide. Cities 106, 102904. doi:10.1016/j.cities.2020.102904

PubMed Abstract | CrossRef Full Text | Google Scholar

RFC 2544 (1999). Benchmarking methodology for network interconnect devices. Available at: https://datatracker.ietf.org/doc/html/rfc2544.

Google Scholar

Sadowski, B. M. (2017). Consumer cooperatives as an alternative form of governance: the case of the broadband industry. Econ. Syst. 41, 86–97. doi:10.1016/j.ecosys.2016.04.004

CrossRef Full Text | Google Scholar

Smith, B. 2020. COVID-19 has only intensified the broadband gap - microsoft on the issues. Accessed May 22, 2022 Available at: https://blogs.microsoft.com/on-the-issues/2020/05/21/broadband-gap-covid-19-airband/.

Google Scholar

The APDEX users group (2023). Available at: https://www.apdex.org/.

Vogels, E. (2021). Digital divide persists even as Americans with lower incomes make gains in tech adoption. Available at: https://www.pewresearch.org/fact-tank/2021/06/22/digital-divide-persists-even-as-americans-with-lower-incomes-make-gains-in-tech-adoption/.

Google Scholar

Whitacre, B., and Gallardo, R. (2020). State broadband policy: impacts on availability. Telecommun. Policy 44, 102025. ISSN 0308-5961. doi:10.1016/j.telpol.2020.102025

PubMed Abstract | CrossRef Full Text | Google Scholar

Whitacre, B., Gallardo, R., and Strover, S. (2014). Broadband's contribution to economic growth in rural areas: moving towards a causal relationship. Telecommun. Policy 38 (11), 1011–1023. doi:10.1016/j.telpol.2014.05.005

CrossRef Full Text | Google Scholar

Zuo, G. W. (2021). Wired and hired: employment effects of subsidized broadband internet for low-income Americans. Am. Econ. J. Econ. Policy 13 (3), 447–482. doi:10.1257/pol.20190648

CrossRef Full Text | Google Scholar