Assessing the quality of electronic medical records in academic hospitals: A multi-center study in Iran

Objective The present study aimed to assess the quality of electronic medical records (EMR) retrieved from hospital information systems (HIS) of three educational hospitals in Mashhad, Iran. Methods In this multi-center, cross-sectional study, inpatient electronic records collected from three academic hospitals were categorized into five data groups, namely demographics (D); care handler (CH), indicating the doers of the medical actions; diagnosis and treatment (DT); administrative and financial (AF); and laboratory and Para clinic (LP). Next, we asked 25 physicians from the three academic hospitals to determine data elements of medical research and education value (called research and educational data) in every group. Flowingly, the quality of the five data groups (completeness * accuracy) was reported for entire sampled data and those specified as research and educational data, based on the exact concordance between electronic medical records and corresponding paper records. HISRA, standing for HIS recording ability, was also assessed compared to data elements of standard paper forms. Results For entire data, HISRA was 58.5%. In all hospitals, the highest data quality (more than 90%) belongs to D and AF data groups, and the lowest quality goes to CH and DT groups (less than 50%, and 60%, respectively). For research and educational data, HISRA was 47%, and the quality of D and AF data groups were the highest (nearly 100%), while CH and DT stood around 50% and 60% in order. The quality of the LP data group was almost 85% in all hospitals but hospital C (well over 30%). Total data quality for the hospitals was almost less than 70%. Conclusions The low quality of electronic medical records was mostly a result of incompleteness, while the accuracy was relatively good. Results showed that the HIS application development mainly focused on administrative and financial aspects rather than academic and clinical goals.

Assessing the quality of electronic medical records in academic hospitals: A multi-center study in Iran Introduction A hospital information system (HIS) is an integrated, computer-assisted system that commonly reflects all hospital operations dimensions.It is used to store and retrieve information related to clinical, administrative, financial, and legal tasks to meet the needs of all authorized system users in the hospital (1).HIS is a potentially rich source usually used for research and educational purposes besides the quality of care improvement and managerial activities.A HIS with high-quality data can improve the quality of health care, monitor patient condition, and therapeutic response, reduce the frequency of errors, avoid adverse drug interactions, and reduce hospital costs (2).In recent years, there has been a growing interest in researching electronic medical records (EMRs) collected in HIS (3).Medical studies, epidemiological investigations, and analysis of disease progression are conducted based on EMRs collected from HIS.Furthermore, patients' EMRs can be used to train medical students and increase their clinical knowledge (4).
The validity of medical research and epidemiological studies done using EMRs strongly relies on the quality of recorded data (5).If the quality of data and reports recorded in HIS are not sufficiently high, the investment return is not guaranteed, and the health care system will not achieve the predefined objectives.Besides, high-quality EMRs can be considered a major educational tool in health care education (6) as it offers acknowledged benefits to medical students in academic environments (7).Therefore, the quality of patient EMRs is considered highly important in any hospital and should be frequently assessed to determine defective processes (8).
So far, studies have been done on HIS in Iran as a developing country, mainly focused on user satisfaction, financial issues (hospital costs, for example), and managerial and administrative functions (9).To our knowledge, a few studies have been performed to assess the HIS data quality by this study method.
Therefore, this study assessed the quality of electronic medical records especially those that possess educational and research value in terms of completeness, accuracy, and quality (completeness * accuracy), in academic hospitals in Mashhad, Iran.We also present the hospital information system recording ability (HISRA; the potential of the system to record the five necessary data groups) of the HIS.

Materials and methods
In this study, we assessed completeness and accuracy as the measures of data quality.We used "Data Source Agreement" as the strategy of assessment and paper medical records were regarded as the assessment "standard" to which the EMRs were compared (3).

Study location and sampling
We selected three academic hospitals, a 920-bed general hospital (A), a 144-bed pediatric hospital (B), and a 60-bed ophthalmic hospital (C) in Mashhad, which is Iran's secondlargest city with about 3 million inhabitants.In these three hospitals, data were mainly recorded in the form of hard copies and some electronic recordings were performed in parallel.
All hospitals use the same version of HIS designed in compliance with the international standard for the exchange of medical information (HL7).The HIS software was launched in 2001 in all hospitals belonging to Mashhad University of Medical Sciences.It is connected to the three ancillary subsystems (laboratory, pharmacy, and radiology) and to the databases including SNOMED, ICD.9.cm, ICD.10, and California tariff that feed patients' EMR.Furthermore, the system records some clinical documentation including medication administration, physician orders, and clinical and surgical procedures.
The study was conducted within 90 days, from September 2019 through November 2019.Over this period, 150, 100, and 50 cases were selected in the three hospitals, A, B, and C, respectively.We used stratified random sampling as the number of selected cases in each hospital was proportionate to that hospital's daily inpatient discharges.We included only inpatient discharges for them both paper and electronic records were available.
We carried out a retrospective comparison between a sample of electronic records and hard copies to examine the quality of electronic medical records.

Categorization of data elements and evaluation of HISRA
To collect data, we considered a checklist of entire data elements of a standard paper record.In the checklist, the data elements were categorized into five groups, namely demographics (D), care handler (CH); diagnosis and treatment (DT); administrative and financial (AF); and laboratory and Para clinic (LP).
HISRA, standing for Hospital information software recording ability, indicates the potential of the hospital information software to capture all data values from the paper records.For each data group, it was calculated as the ratio of the number of available data fields that can be filled in both paper charts and the HIS software, to the total number of data elements, within the paper charts.

Specification of research and educational data elements
25 physicians from the studied hospitals, who were all faculty members of University of Medical Sciences, Mashhad, Iran; they were asked to determine whether a given data element is "Necessary "or "Not necessary" concerning research and educational importance.Then, the content validity ratio (CVR) was calculated for each data element according to the collected answers.For a group size of 25 (the number of physicians included in this experiment), the critical number and critical CVR were calculated as 18 and 0.44, respectively.The data elements with CVR ≥ 0.44 were selected as elements with research and educational importance and called "research and educational data" (10).

Data collection, completeness, and accuracy
For every sampled discharge, a checklist was completed by comparing the electronic and paper medical records.In the checklist, the presence or absence of each entry in the paper and the electronic records was determined to evaluate data completeness.In addition, the exact concordance between the two values recorded for a specific element in electronic and paper medical records was considered to measure data accuracy.

Completeness
"Completeness" was defined as the existence of an entry for a given element in the electronic medical record that is present in the paper record, regardless of its accuracy.Completeness was calculated as the ratio of the number of data entries found in patients' electronic records to the total number of data entries that existed in patients' paper records.

Accuracy
"Accuracy" was defined as a strict concordance between electronic data value and the corresponding value in the paper record.Accuracy was calculated as the ratio of the number of accurate data values to the total number of data values (11,12).

Data analysis
Finally, for entire data together with the research and educational data, the quality of each data group was calculated by multiplying accuracy by completeness (11,12).The hospitals were statistically compared using P-value (<0.05), derived from the Pearson Chi-square test, to examine whether hospitals' difference considering data completeness and accuracy is statistically significant or the difference is by chance.

Data elements with research and educational importance
Based on their educational and research value, data elements were specified as "essential" and "not essential" by 25 physician selected from the three hospitals; then CVR was calculated for each data element.Table 1 shows the data elements that meet the criteria of CVR ≥ 0.44, and were selected as elements with research and educational importance.In this table, data elements of the same data group and the same CVR are shown together in a row.

HISRA of the HIS software
Concerning entire data elements, HISRA was 58.5%, while it was 47% for research and educational data elements (Tables 2,  3).Among entire data elements, HISRA of CH, and DT groups (75% and 43%, respectively) were lower than those of D, LP and AF groups (100%, 100%, and 91%, respectively).
Similarly, for research and educational data elements, HISRA of CH, and DT groups (75% and 39%, in order) were far lower than those of D, LP and AF groups (100%).

Data quality in hospitals A, B and C
In total, 300 discharged cases were assessed from September 2020 to November 2020.Table 4 shows the completeness, accuracy, and quality of five data groups in the three hospitals for entire data elements.Similarly, Table 5 shows them for research and educational data.
Concerning entire data, the total data quality of hospital B (70%) is higher than hospitals A and C (54% and 69%), and the quality of the D data group (more than 97%) is considerably higher than the other groups in the three hospitals.
Concerning research and educational data elements, hospital B has better total data quality compared to the others (70% vs. 53 and 66%).Moreover, the D data group has the highest quality (100%) in the three hospitals.It is equal to the AF in hospital C and followed by the LP data group in hospitals B and A (95 and 89%, in order).

Comparison of completeness and accuracy among the three hospitals
Table 6 shows the probability values of completeness and accuracy for entire data, which differ significantly among the three hospitals.The completeness of data groups but D were significantly different when comparing the three hospitals.There were significant differences in terms of accuracy of CH, DT, and LP data groups among the three hospitals (Table 7).

Discussion
We assessed the data quality of the HIS used in the hospitals belonging to the Mashhad University of Medical Sciences.To this, we performed a retrospective comparison between a sample of electronic and paper medical records in three academic hospitals.We calculated HISRA and five data groups' quality (completeness * accuracy), namely demographics, care handler, diagnosis and treatment; administrative and financial; and laboratory and Para clinic.
Assessment of the HIS Software used in these three hospitals indicated low HISRA for entire data elements (58.5%).It was even lower (47%) when considering research and educational data elements.The diagnosis and treatment data group had the poorest HISRA (less than 45%) among the five data groups, while HISRA for demographics, laboratory and Para clinic; and administrative and financial data groups was  For care handler; and diagnostic and treatment data, although the accuracy was relatively good (over 80% in all hospitals), incompleteness reduced the overall data quality to less than 70% for both entire data; and research and educational data.We observed that policies and regulations flaw has led to reduced completeness of diagnostic and treatment data in the three hospitals because caregivers are not required to record medical history, disease progress received consultations, and physician orders completely in electronic records.Poor policies and regulations also resulted in reduced completeness of the care handler group because clinical staff did not have to capture such information within the HIS.In the care handler group, inaccuracy commonly resulted from the wrong timestamps.Because the timestamp was created automatically by the computer and for a late data entry, which was not done at the point of medical action, it reflects the time of computerized data entry, not the real-time of the medical procedure.Moreover, EMRs sometimes included the user who entered the data instead of the doer of medical procedures.
According to our observations, the quality of laboratory and para clinic data in hospital B was the highest (over 90%).It was relatively good (over 80%) in hospital A, and unexpectedly poor in hospital C.This happened because, in the hospital C, a laboratory or para clinic report is recorded in both electronic and paper records, provided that it was done by the hospital's laboratory or para clinic section.Otherwise, it is held only in paper records.Whereas, in this hospital, it is common to refer patients to outside laboratories to do some physician-ordered tests.Similarly, the three hospitals, capture the medicine administration in the patient's EMR when the ordered medicine has been provided by the hospital's pharmacy.Otherwise, it was recorded only in the paper charts as this procedure flaw has increased inconsistency between paper and electronic records and has led to low completeness of laboratory and Para clinic data in hospital C and low completeness of diagnosis and treatment data in all three hospitals.
Several studies investigated the quality of electronic medical data used for research purposes; they employed various assessment strategies to report data quality.We found studies that referred to "data availability" as "completeness" and assessed it by three major strategies, (i) comparing with a  gold standard, (ii) checking the presence or absence of data value, and (iii) comparing with data from other data sources.We also found studies that defined "agreement among data elements" as "concordance, consistency, validity, correctness or accuracy".Considering the publications that we found, data agreement was assessed by two main strategies, including (i) Data Element Agreement, comparing the data with those within the EMR, and (ii) comparing the data with those from other sources that may be a gold standard.Other used data sources include paper records (11)(12)(13)(14)(15)(16)(17)(18) patient-reported data (19-21) or physician-reported data (22).Some studies reported the overall completeness and accuracy of the electronic data, comparing paper records as the "gold standard" (11)(12)(13)(14)23).Some others compared electronic medical data to paper records as the "data source" (15)(16)(17).
Compared to the mentioned studies, our study had a higher level of granularity of the elements measured and benefited from extensive assessment.Although most studies assessed data elements that serve the purpose of doing a determined study in the future (diagnosis and procedure codes related to a specific disease, for instance) (11-14, 16, 17), our study did not focus on the suitability of data elements for determining research.We tried to identify and assess essential data for research and education.We also expanded the assessment to entire recorded data (even financial ones) to investigate how to improve HIS to collect effective records.Our definitions of completeness, and accuracy, and the protocol we used to measure them were consistent with some previous studies.However, it was quite different from that employed by Whitelaw et al. (14) and Logan et al. (21) and also was somewhat different from the definition of Prins et al. (13).
In light of our findings, the following suggestions are offered to improve the HIS software used in these hospitals and as instructions to be given to the medical staff: Given the low HISRA of the software, particularly in terms of diagnosis and treatment data, the HIS used in these hospitals   Hospitals should set strict rules assuring the medical staff records the patient's history, physician's orders, disease progress notes, and consultations.HIS also must be enabled to record prescriptions and medications together with clinical information received from other medical centers outside the hospital to track medication and refill histories.This trend will certainly increase the completeness of the diagnosis and treatment; and laboratory and Para clinic data.
Since we frequently observed inaccuracy in date and time in the diagnosis and treatment data group, HIS must be redesigned to record the date and time for each entry based on the actual date and time.HIS must also provide the documenter with the possibility of manually entering the date and time of late entries.
If it is necessary to have both electronic and paper medical records in parallel, it is recommended to omit the hand-written paper records and print the computer-generated EMR.This procedure ensures legibility, completeness, validity, and precision of the paper record by utilizing software controls such as "input masks".It also increases the concordance between electronic and paper records.
Studies that use the "Data Source Agreement" strategy to assess the quality of electronic data prefer to compare data with a valid and reliable external data source.We used patients' paper records as the assessment standard even though they might have been illegible, incomplete, or unreliable; they were the only available documentation usually used in addition to electronic data by medical care providers in the mentioned hospitals.Thus, the absence of a reliable external data source formed a limitation of the present study.Furthermore, the result was not influenced by the transformation of the electronic record or the paper record, or both (because data values often had been recorded in the form of open text).However, accuracy assessment was influenced by the personal impression of the investigators as well as the performance of the persons who entered the data in to the EMRs.

Future work
As a possible future work, one could investigate the degree to which the data deviate from a truly "random sample" of the patients' data.For example, if normal laboratory values are more likely to be in the computer and abnormal values are more likely to be on paper, this could have a pronounced impact, and bias, on research that attempts to use only one or the other.

Conclusions
The findings demonstrated that the HISRA of the system was not good in general, and it was poor for diagnosis and treatment data in particular.It also showed the low quality of registering the care handler; and diagnosis and treatment data groups.
The high quality of administrative and financial data groups imply that in the three hospitals, the development of the software and data entry policies focus on managerial, administrative, and financial applications rather than potential academic and clinical applications.Thereby, it is suggested that the clinical and diagnostic aspects of those EMRs should be addressed.Easier data entry methods, better training of personnel, and changes in the HIS software can probably solve many obstacles.Furthermore, regulations and policies need to be revised to serve the data quality improvement.

TABLE 2
HISRA of the HIS concerning entire data elements.
D, Demographics; CH, Care Handler; DT, Diagnosis and Treatment; AF, Administrative and Financial; LP, Laboratory and Para clinic.Bold values are percent.

TABLE 1
Content validity ratio (CVR) for research and educational data elements.Blood Pressure After Using Anesthesia drugs, Urea, Hb and Hct, Blood sugar, Blood group,RH, Pulse, Vital Signs Before Operation, Time Of last Urination Before Operation, Pre-Operation Drugs & Time Of The Use, Pre-Anesthesia Drugs Effects, Anesthesia Time, Patient's Status In The Beginning / At The End Of the Operation, Fluids, Symbols, Name Of Operation, Type Of Operation, Direction, Pre/Post Operation Diagnosis, Procedure & Findings, General Condition, Awakens, Vital Signs, Skin color/ Temperature, Local/ General Cyanosis, Intra Venous Fluid, Fluid Absorption, Fluid Excretion, Plasma, Blood, Ordered Drugs (Name, Dose, Type, Frequency), Information Of Drug Administration, Medical procedures, Observations, Considerations And Sign Of Nurse, Vital Signs Control, fluid absorption/excretion; pre-operation; post-operation, and anesthesia care, and their timelines.HIS software flaw also has increased inaccuracy of the diagnosis and treatment data group due to the lack of data validity checks in open-text fields.

TABLE 3
HISRA of the HIS concerning research and educational data elements.

TABLE 4
Completeness, accuracy and quality of data groups concerning entire data elements.
D, Demographics; CH, Care Handler; DT, Diagnosis and Treatment; AF, Administrative and Financial; LP, Laboratory and Para clinic.N is the number of assessed cases.Bold values are percent.

TABLE 6
Comparison of completeness and accuracy for entire data values.

TABLE 7
Comparison of completeness and accuracy for research and educational data values.

TABLE 5
Completeness, accuracy and quality of research and educational data elements.Diagnosis and Treatment; AF, Administrative and Financial; LP, Laboratory and Para clinic.N is the number of assessed cases.Bold values are percent.to improve to record physical examinations, pre-operation, post-operation, anesthesia care, detailed nurse notes, vital signs, fluid absorption and excretion, and their time trends.Additionally, in these hospitals, a great amount of data are entered in an open text and non-structural form.By adding data-entry fields, software capabilities can facilitate input validation checking, which may increase the completeness and accuracy of the recording process.Considering the incompleteness of the care handler data group, the medical staff should be instructed to record the doers of the medical action.Moreover, digital signature and audit logging should be implemented to assure care handler data accuracy.