Development of Key Performance Measures for E-Learning Knowledge Systems: The Case of Higher Education Institutions in Taiwan

To cope with the trend of e-learning and competition in global higher education, the Ministry of Education of Taiwan has promoted the “Project on deep plowing of e-Learning” to study the establishment of University e-learning knowledge systems, which will serve as a reference for universities to develop e-learning in response to the trends of the digital age. The research team developed a preliminary draft of the e-learning knowledge system through a literature review, modified it through the Delphi technique, and finally developed an e-learning knowledge system. The system consists of two dimensions: “organization” and “teaching team.” The organization dimension is divided into the “level of preparation” and “execution and implementation” sub-dimensions. The dimension of the teaching team is divided into the “basic ability” and “course design and application” sub-dimensions for a total of 4 sub-dimensions. The dimensions and sub-dimensions are divided into levels, sub-levels, indicators, and the content of indicators. The results of the research include 2 dimensions, 4 sub-dimensions, 14 levels, 40 sub-levels, and 84 indicators.


BACKGROUND AND MOTIVATION
In response to the trends of the digital age, learning methods have evolved over the past 10 years from distance learning and e-learning to today's Massive Open Online Courses (MOOCs). The development of today's education, with advances in information and communication technology, has gradually changed from predetermined and fixed to adaptive and dynamic development (Bojinov, 2016;Howarth et al., 2016).
Universities, as the highest institutions in the education framework, must keep up with this digital trend. Universities and professors must be able to respond to evolving social needs and self-learning needs in teaching development. How should University education change to cope with these development trends and challenges? Can universities make good use of digital technology and intelligence technology to improve the adaptability of education and students' learning performance? How can students assist or promote their own learning habits? Higher education institutions worldwide must actively pay attention to these issues and explore and implement solutions (Kaplan and Haenlein, 2016;Langset et al., 2018).
Relevant studies suggest that some foreign universities have considered possible changes in the school system and courses and have actively invested in higher education through various developments and actions to respond to changes in learning needs (e.g., Lamprou and Lvovskaya, 2015;Adekola et al., 2017;Ansong et al., 2017;Langset et al., 2018). For example, universities cooperate with online learning platform providers to offer bachelor's degree, master's degree and certificate courses. Students can obtain a degree according to the relevant regulations after completing the courses. Furthermore, the platform providers work with enterprises and universities to provide in-demand courses for employees, professional minidegrees, and required career development skills for learners and help leaners be more competitive in the job market.
In recent years, the efforts of Taiwanese universities have been applied to distance learning courses and credits, e-learning courses for in-service master's programs, Open CourseWare (OCW), and MOOCs. More than 60% of universities have distance learning courses, and 10% of college students have attended distance courses (Huang, 2014). Some universities have collaborated with online learning platform providers to develop on-campus and off-campus modes.
Although the international and Taiwanese efforts in elearning have shown good results, most of these efforts have involved project-based or scattered developments from universities that are not sufficient to effectively link the plans of digital education developments. Most universities still have room for growth in overall e-learning implementation, including standardized or normalized actions, supported resources, and overall performance. Therefore, universities must adopt a dynamic mindset to cope with adaptive and dynamic learning needs because worldwide competition in higher education will become increasingly fierce. Therefore, some countries or research institutions have begun to analyze relevant classifications or recommendations. For example, the Ministry of Education of New Zealand proposed enabling e-learning architecture, which includes beyond the classroom, teaching, professional learning, leadership, technology and ministry initiatives (Te Kete, 2019). Ali et al. (2018) collected 259 papers from 1990 to 2016 and classified the obstacles to promoting e-learning. Many authors have discussed this topic only from the course perspective, which is too narrow. These authors have found that implementation from organizations, teaching and research, technical system support, and development teams also affect the success of elearning implementation.
With regard to the development of e-learning, it must respond to paradigm shifts and trends in University competition. In a knowledge-based society, knowledge is an intangible asset, and the resources of the knowledge system provide an important method to enhance the competitiveness of e-learning. Therefore, the Ministry of Education of Taiwan promotes studies of elearning and research on the establishment of University elearning knowledge systems in combination with practice. This research can be adopted by other universities and colleges to set a direction and effectively implement technology for continuous learning and development. This approach can provide an educational environment and learning content that can meet the needs of future learning and maintain the international competitiveness of higher education. To achieve this goal, this study analyzes and identifies indicators of an e-learning knowledge system that can be used as a reference for colleges to implement e-learning.

Study Scope
The knowledge system in this study consists of three characteristics: it can be standardized and provide principles; it can provide guidance for learning; and it can be practiced concretely (IGI Global, 2018). The e-learning knowledge system in this study refers to e-learning as teaching in colleges and universities by obtaining digital content and materials through self-made content, outsourcing, and purchasing or adopting existing digital materials that are integrated into e-learning courses. Instructors and students can communicate interactively through assisted teaching tools and digital technology media, communication networks, computer networks, video channels, and other transmission media. The contents and activities include learning material, lectures, interactive teacher-student discussions, collaborative learning, quizzes, peer-to-peer assessments, individual and group assignments, and other learning activities.

Research Process
This project aimed to complete a set of guidelines for integrating e-learning knowledge and experience through research teams, experts from industry and government departments, and user feedback. There were three main stages. In the first stage, the project research team collected e-learning knowledge system data according to the literature review, invited experts with e-learning teaching experience, discussed the structure classification, defined the structure, and examined its related content. In the second stage, the research team created the content under each indicator according to the discussion results in the meetings at the previous stage, including dimensions, sub-dimensions, levels, sub-levels, levels, and indicators. In the third stage, the research team converted the content of the knowledge system into an online questionnaire, invited the experts to express their extent of agreements with the questionnaire and give feedback. The questionnaire was distributed in two rounds to achieve consensus.

Participating Experts
This study adopted the Delphi technique and invited 15 Taiwanese practitioners with practical e-learning teaching experience, experts from industry and government experts who had experience with e-learning implementation to discuss the classification of the structure in a meeting format. We defined two dimensions of the structure, organization and the teaching team, and examined their related contents.

Appropriateness and Project Consistency
The research team continually revised the content of the appropriateness of the Delphi technique in each round until the experts reached consensus in the third round. The average value of appropriateness was ≧ 4.5 and the mode was 5, indicating that most experts evaluated the appropriateness of the project's content as "very appropriate" and classified it as "very appropriate." The average was between 4 and 4.5 and the mode was 4, indicating that most experts evaluated the appropriateness of the project's content as "appropriate" and classified it as "appropriate." The average was between 3.5 and 4 and the mode was 3, indicating that most experts evaluated the appropriateness of the project's content as "no opinion" and classified it as "not necessary." In addition, this study adopted the quartile deviation and standard deviation to understand the distribution of the expert opinions. A quartile deviation <0.6 or standard deviation ≦0.5 indicates high consensus, a quartile deviation between 0.6 and standard deviation ≧1 indicates moderate consensus, and a quadruple deviation ≦1 or standard deviation >1 indicates low consensus.

KNOWLEDGE SYSTEM ARCHITECTURE AND INDICATOR DEVELOPMENT The Research Team Establishes the Draft
Relevant studies have noted (e.g., Nyoni, 2014;Okinda, 2014;Ireri and Omwenga, 2015;Tarus et al., 2015;Basak et al., 2016;Ansong et al., 2017) that e-learning implementation relies on promotion from the organization and input from teachers. It emphasizes that the concepts of organizational support and a level of preparation are required when implementing e-learning (Demir and Yurdugül, 2015;Mosa et al., 2016;Adekola et al., 2017). At the same time, scholars noted that in an organization, if a lecturer wants to use digital teaching or serve as an elearning lecturer, he or she should have the corresponding basic and core abilities to use digital teaching skills smoothly (Allison, 2015;Lamprou and Lvovskaya, 2015;Terosky and Heasley, 2015). To provide guidance for integrating e-learning knowledge and experience, this study analyzed the relevant literature (Chao et al., 2007;Demir and Yurdugül, 2015;Mosa et al., 2016;Adekola et al., 2017), identified the architecture of the e-learning knowledge system based on two aspects, the "organizational aspect" and the "teacher aspect, " and then developed the content of an elearning knowledge system for Taiwan colleges and universities as a reference to implement e-learning.

Organization Aspect
The research team further divided the implementation stage into two sub-dimensions: "the level of preparation" and "execution and implementation." In the sub-dimension of "the level of preparation, " the organization should evaluate internal readiness to conduct e-learning implementation activities smoothly (Chao et al., 2007;Adekola et al., 2017). According to the research results of the Economist Intelligence Unit (2003), "4C" indicators -connectivity, capability, content, and culture -are used to assess the level of preparation for e-learning with regard to education, industry, government, and social organizations. There are a total of 150 qualitative and quantitative indicators related to the study of e-learning to measure the e-learning situation in each country. The research team evaluated the suggestions of the Economist Intelligence Unit (2003) and reviewed other relevant literature (e.g., Chao et al., 2007). The level of preparation was divided into four levels, namely, "Infrastructure Connectivity, " "Team Capability, " "Courses and Contents, " and "Organizational Culture." The main contents are as follows.
• Infrastructure Connectivity: In terms of e-learning, the quality and popularity of infrastructure such as the Internet, e-learning platforms, tools, software and hardware are the basic conditions to consider. Furthermore, this area considers whether the software and hardware environment are suitable

Number of Level (L), Sub-level (SL), and Indicator (I).
Frontiers in Education | www.frontiersin.org Level: OR1 infrastructure connectivity Sub-level: OR1-1 organization basic infrastructure Indicator: OR1-1-1 for e-learning implementation; the quality and quantity of infrastructure are sufficient to fully support the teaching team and students.
Suggested learning course: introduction to e-learning software and hardware Indicator OR1-1-2: there is a security management mechanism to ensure smooth communication and data security for communication on the e-learning platform.
Suggested learning course: network security protocols Level: OR1 infrastructure connectivity Sub-level: OR1-2 organization supporting environment Indicator: OR1-2-1 there is infrastructure management staff for the e-learning platform.
Indicator: OR1-2-2 there is an appropriate document management mechanism for the e-learning platform according the demands of the organization and teaching team, such as learning material media, teaching plans and purchased materials.  Indicator: OR4-1-2 build an e-learning teaching unit in the organization.

Suggested learning course: organizational change and implementation
Level: OR4 organizational culture Sub-level: OR4-2 implementation mechanism Indicator: OR4-2-1 encourage mechanism for excellent e-learning staff .
Suggested learning course: organizational change and implementation for the development and implementation of e-learning from the perspective of management, teachers, and students.
• Team Capability: The organization evaluates whether team members have e-learning knowledge and skills and provides corresponding training, consulting services and assistants to enhance their abilities.
• Course and Content: The organization supports team members in developing e-learning courses and content and provides relevant resources for their use.
• Organizational Culture: The organization promotes elearning, adapts to changes, recruits e-learning professionals, and develops relevant incentives and promotion mechanisms.
Second, in the "execution and implementation" sub-dimension, the relevant literature (e.g., Penicina, 2011;Walasek et al., 2011;López and Builes, 2012) mentions the importance of adopting the "PDCA" quality management model in e-learning. This model is a four-step continuous management process: Plan, to establish a clear goal and develop an execution plan and its necessary procedures; Do, to implement according to the plan; Check, to check the execution status and determine the problem with the result; and Action, to review the results of execution and propose improvement suggestions, promote successful experience and standardize the processes. The research team divided the "execution and implementation" sub-dimension into Plan, Do and Support, Check, and Improve according to the requirements of organizational implementation. In addition to executing the plan, the organization should provide related resources to support the processes. Regular review, progress tracking, and improvement plans through quality management mechanisms will accumulate successful experience and produce benefits.

Teacher Aspects
In the initial planning of "teacher aspects, " most studies have considered teachers to be an important factor in e-learning implementation (Allison, 2015;Lamprou and Lvovskaya, 2015;Terosky and Heasley, 2015;Hamilton, 2016;Riehemann and Jucks, 2017;Smith et al., 2017;Thomas and Graham, 2017;Lin and Cantoni, 2018;Pettersson, 2018). The abilities of elearning teachers should involve two sub-dimensions: "Basic Competence" and "Course Design and Application." The research results of the "Basic Competence" sub-dimension of the knowledge system (Baran, 2014;Li et al., 2015;Tour, 2015;Al Khateeb, 2017) can be divided into four levels: "character, "    Suggested learning course: e-learning course review and improvement "teaching contents, " "information operation capability, " and "elearning teaching concepts." "Character" refers to whether teachers are willing to enter the digital teaching field and continue to maintain motivation related to their personality traits, such as resistance, innovative thinking, self-learning, positivity, respect, and community awareness. Institutions can provide resources and training courses to help teachers improve their positive motivations according to their needs. "Teaching content" refers to teachers' professional skills in the courses they teach and their capability in planning appropriate teaching strategies and content according to different teaching and learning objectives. "Information operation capability" refers to e-learning teachers' basic

Sub-level: TB1-1 basic information technology operation
Indicator: TB1-1-1 able to operate computers and peripheral hardware to aid teaching.
Suggested learning course: general teaching software and hardware operation.
Indicator: TB1-1-2 able to operate technology-based media and related hardware.
Suggested learning course: general teaching software and hardware operation Indicator: TB1-1-3 able to install by oneself or seek support to install and use the basic e-learning software and applications.
Suggested learning course: general teaching software and hardware operation Level: TB1 information operation capability Sub-level: TB1-2 internet resource application Indicator: TB1-2-1 able to search, download and manage internet resources.

Suggested learning course: internet resources application
Level: TB1 information operation capability Sub-level: TB1-3 online teaching and sharing Indicator: TB1-3-1 able to use online teaching activities to assist professional development.
Suggested learning course: online teaching and sharing Indicator: TB1-3-2 able to use online communication software or platform for discussion.

Suggested learning course: online teaching and sharing
Level: TB1 information operation capability

Suggested learning course: multimedia presentation and operation
Indicator: TB1-4-2 able to adopt multimedia appropriately according to learning objectives.

Suggested learning course: application of multimedia-aided teaching
Level: TB2 e-learning concept Sub-level: TB2-1 concept of e-learning implementation Indicator: TB2-1-1 able to understand the formats of implementing e-learning.

Suggested learning course: e-learning implementation formats
Indicator: TB2-1-2 able to decide the timing of using e-learning.

Suggested learning course: e-learning application timing
Level: TB2 e-learning concept Sub-level: TB2-2 concept of e-learning implementation planning Indictor: TB2-2-1 able to decide the feasibility of implementing e-learning course.
Suggested learning course: e-learning course planning Indicator: TB2-2-2 teaching team is able to define teaching plan.

Suggested learning course: teaching plan theory and practice
Level: TB2 e-learning concept Sub-level: TB2-3 design concept of e-learning Indicator: TB2-3-1 have basic concept of e-learning teaching models and processes.
(Continued) literacy and information skills. "E-learning concepts" are the basic concepts for the implementation, planning and design of e-learning. All members of the teaching team should have the basic required abilities for their responsibilities and should gradually improve the influencing factors when implementing e-learning.
In the sub-dimension of "Course Design and Application, " the "ADDIE Teaching Model" (Lynch and Roecker, 2007;Driscoll, 2010) is mainly used to provide systematic planning indicators for teaching designers to ensure teaching quality. The ADDIE teaching model is divided into five categories indicating the essential factors in each stage: "Analysis, " including demand analysis, learner analysis, teaching content analysis, tool analysis, and learning environment analysis; "Design, " including the course syllabus, course structure, teaching objectives, lesson design, and activity design; "Development, " including content presentation, learning material design, and learning material development; "Implementation, " including course testing, class opening and operation; and "Evaluation, " including performance evaluation and improvement.

Delphi Technique Process and Adjustment
The research team proposed the first draft of an e-learning knowledge system based on a literature review and internal discussion and invited experts to participate in the kick-off meeting and provide recommendations. The questionnaire with the Delphi technique was conducted in three rounds. The team adjusted indictors and contents to complete the final draft according to the experts' recommendations (as shown in Table 1).
In the preliminary structure stage (Step 0), the initial structure was proposed: two dimensions, four sub-dimensions, a total of 16 levels and 59 sub-levels. After the first round of the questionnaire, "teacher aspects" was changed to "teaching team aspects, " and the relevant terms and contents were adjusted. This means that the teaching team should work together for development rather than one teacher taking all responsibilities and tasks. The study also added the role of the teaching team and its responsibilities to help the management level of the organization understand the importance of this role. In addition, the study adjusted the "rewards and punishments mechanism" to the "encouragement mechanism, " which was mentioned at the level of "character" and "teaching content" in the original knowledge system, to avoid teachers' unwillingness to enter the e-learning field after their self-evaluation. Based on previous studies (e.g., Lamprou and Lvovskaya, 2015;Terosky and Heasley, 2015;Lin and Cantoni, 2018), this study moved the related content of the level of "learning abilities" in the dimension of "organization, " which provides training and consulting. Trainers should have training   Suggested learning course: e-learning materials evaluation theory and practice.
Indicator: TC3-3-2 development of evaluation activities. attitudes, domain knowledge and skills to make the training framework complete rather than a formality. Therefore, this study reduced the number of sub-levels from 59 to 42 and further developed 115 indicators. The breakdown is as follows.
The second-round questionnaire added 115 indicators under the sub-levels. After the experts responded to the questionnaire, the extreme values were removed. The average numbers of most questions were between 4.4 and 4.7, and standard deviations were between 0.4 and 0.7. The study removed the indicators with an average < 4 or a standard deviation > 1 indicator. Although the average number of most indicators reached consensus, the committees suggested adjusting the wording. For example, the study adjusted the wording and content of "course design" and "learning material design" given the operational difference between digital-aided in-class courses, mixed courses, and online courses. In addition, it emphasized the organization's building of indicators of hardware and software planning, featured courses, the development of resources and quality, and intellectual property management. In addition, the committees suggested consolidating the contents, removing unclear definitions of indicators and merging similar indicators. In summary, the third edition of the knowledge system was adjusted to 14 levels, 42 sub-levels and 104 indicators. The details of the indicators are listed. Finally, the study merged and adjusted indicators with an average number < 4 or standard deviation >1 and then completed the third round of the questionnaire. After adding the sub-levels, there was a total of 84 indicators. The average numbers of each indicator were between 4.6 and 4.9, and the standard deviations were between 0.2 and 0.5. Consensus was achieved after collecting feedback from the committees. The final draft was 14 levels, 40 sub-levels and 84 indicators and their detailed contents.

E-LEARNING KNOWLEDGE SYSTEM ARCHITECTURE AND INDICATORS E-Learning Knowledge System Architecture
According to the research results, the e-learning knowledge system included two levels, "organization" and "teaching team." The organization was divided into "the level of preparation" and "execution and implementation." The teaching team was divided into "basic ability" and "course design and application" with a total of four sub-dimensions. Under the dimensions and sub-dimension, levels, sub-levels, indicators, and contents were developed. The number of contents and outputs are summarized in Table 3. There was a total of 2 dimensions, 4 sub-dimensions, 14 levels, 40 sub-levels, and 84 indicators ( Table 2).

The Contents of E-Learning Knowledge System Architecture
This section describes the sub-dimension, levels, sub-levels, indicators, and indicators under the two major dimensions. Tables 3-6 list the organization -level of preparation (Table 3), organization -execution and implementation (Table 4), teaching team -basic ability (Table 5), and teaching team -course design and application ( Table 6).

CONCLUSIONS
To manage the trends in e-learning and the competition in global higher education, the Ministry of Education of Taiwan has promoted the "Project on deep plowing of e-Learning" to study the establishment of University e-learning knowledge systems. This project will serve as a reference for universities to develop digital teaching in response to digital age trends. The research team developed a preliminary draft of the e-teaching knowledge system through a literature analysis, modified the draft using the Delphi technique, and developed an e-learning knowledge system. The system consists of two dimensions: "organization" and "teaching team." The organization dimension is divided into the "level of preparation" and "execution and deployment" sub-dimensions. The teaching team dimension is divided into the "basic ability" and "course design and application" subdimensions for a total of four sub-dimensions. The dimensions and sub-dimensions are divided into levels, sub-levels, indicators, and indicator content depending on the content. The research results include 2 dimensions, 4 sub-dimensions, 14 levels, 40 sub-levels, and 84 indicators.
An e-learning knowledge system can provide a reference for universities to develop digital teaching in response to digital age trends. In this research, the e-learning knowledge system was established under the government's projects. In addition to providing a reference for other universities in Taiwan, this system can be used as a reference by other universities. However, the system should be moderately adapted when applied in other nations or universities based on intuitional situations because it was established by scholars and experts in Taiwan.
The purpose of this research is to establish e-learning knowledge systems. However, each University or college has a different starting point for introducing e-learning. For example, national private universities and national private technical universities in Taiwan have different organizational e-learning goals. Depending on the different goals, the organizational level and the index reference should be based on the university's own resources and human resource considerations. If resources are limited, achieving the complete system level and the recommended index practices at one time is difficult. Planning short-, medium-, and long-term promotional strategies is necessary. Therefore, in future research, promoting this system in different types of universities and colleges, establishing the reliability and validity of this indicator, and revising the content of the indicators are recommended to establish a more complete knowledge system.

DATA AVAILABILITY STATEMENT
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author/s.

ETHICS STATEMENT
Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The patients/participants provided their written informed consent to participate in this study.

AUTHOR CONTRIBUTIONS
S-PC, S-YW, and A-CL contributed to the design and implementation of the research, analysis of the results, and writing of the manuscript. All authors contributed to the article and approved the submitted version.

FUNDING
This research was supported by the Ministry of Education of Taiwan, which has promoted the project on the deep plowing of e-learning.