The main intent of our research was to examine the overall acceptance rate among medical professionals by comparing a number of records entered by medical and administrative staff members. To keep results more objective, we analyzed data collected from January 1st, 2012 until December 31st, 2015. Different acceptance rates were observed among different departments. Differences are explained in the scope of the technology acceptance model, based on the different influence of the external properties. In several departments and sub-departments, organizational structure and lack of IT infrastructure make administrative workers the only persons that can use information system. For these departments, a number of records registered by administrative workers can be assumed as potential false positives, thus they are presented separately. Thanks to this research, we are now able to restructure our deployment strategies and to work closely with our potential users to improve healthcare workflow within their departments. Medical information systems (MIS) based on Electronic Health Record (EHR) are nowadays a common component in medical care delivery. The meaningful use [Jones et al. 2014] and the overall healthcare delivery improvement [Wang et al. 2014] are the leading paradigms for successful MIS development. Once implemented in a healthcare institution, EHR based systems will have both positive and negative impacts on primary care medical practice [Holroyd-Leduc et al. 2011]. While structural and process benefits can be easily identified, the overall effect on clinical outcomes is less clear, and thus the usage and acceptance of MIS systems must be monitored. Having both potential positive and negative effects in mind, our research group worked on the MIS since 2008 [Rajković et al. 2009]. Pilot deployments started in the year 2010, and, today, our MIS is deployed in more than 25 different primary care centers [Rajković et al. 2013]. Since the primary care centers in the Republic of Serbia are organized on municipality level, installed MIS instances vary in overall complexity and collected data volume: starting with small municipalities having less than 15,000 inhabitants, and ending with Niš Primary and Ambulatory Care Center covering a city with almost 250,000 people and having more than 650 users. The main aim of the project was to develop EHR based systems that will ensure proper data collection (according to [WorldBank 2009]) as well as report to MoH. Now MoH supports the project that should ensure better communication between instances of different installed MIS [EU-IHIS 2009]. MIS developed by our research group was designed to satisfy both basic needs for registering medical services and to offer some advanced features that should make medical staffs' work easier. Basic functionalities are
defined to look like previously used paper-based documentation and within technology acceptance model (TAM) [Davis 1989] [Venkatesh and Davis 2000] they would be classified in perceived ease of use (PEOU) category. Today, after more than seven years of active use, we can analyze stored data and perform an analysis of our information system usage and overall acceptance rate by medical professionals.
The analysis of the MIS user behavior become an important topic in the last decade. The fact that MIS systems have been developed for more than half a century, but were often rejected by the end users, raised an interest in the investigation of user behavior and expectations. Dating back to the beginning of our project, the results of the analysis performed in North America [Leonard and Sittig 2007] stating that MIS should not be just acceptable on the first site, but also designed in the way that end-users change their habits and accept advanced features. This was also a point in the direction that so-called computer anxiety is maybe not the primary reason to reject MIS. The study from three clinics in the city of Skopje [Ketikidis et al. 2012], from similar cultural and organizational background, proved that the computer anxiety was rather low, but the overall MIS design is more important. Usability itself was the key issue for MIS acceptance in primary care both in the Netherlands [Meulendijk et al. 2013] and [Dünnebeil, et al, 2012]. It was even stated in [Meulendijk et al. 2013] that any newly develop and the advanced feature could be accepted only if “does not require extensive investments of time”.
The examined MIS was introduced during 2011, and since January 1st, 2012, the system is in full use. The period that we focused on started on January 1st, 2012, when most of the departments were equipped with the necessary IT infrastructure. Examined period ended on December 31st, 2015. Our goal was to check the system acceptance rate among medical doctors and analyze actual system use. Since in Serbian healthcare system, funding of primary care institution depends on reported medical services to MoH, the institution need to establish mechanisms ensuring all completed medical services are properly registered. Next, in the defined periods MIS generates reports and uploads them to the MoH server. Similarly, to the results presented in [Venkatesh and Davis 2000], if some action related to MIS is required, users will use it more often. It is important to state that in the period before the installation of MIS, Serbian primary care institutions used the dedicated reporting tool to send data to MoH. Users of the mentioned tool were administrative workers, or nurses working with medical records, but not medical doctors or physicians.
The material we used for our research was data collected within MIS. While entering data about the given medical service, the medical practitioner must set a potential diagnosis, enter anamneses and then, if needed, prescribe therapy and create a request for further specialist examinations or therapeutic treatments. The same set of data, excluding anamneses, must be entered also by administrative workers when they fill reports. We must keep in mind that seven years ago many medical professionals were not so willing to use MIS and that MoH still requires medical institutions to archive printed reports. Here we can immediately see two antagonistic requests: to still keep paper-based documentation and to electronically report all given services. Initially, we saw some resistance, especially among the older doctors.
One of the consequences of this behavior was that administrative workers or nurses sometimes
had to take doctors’ scripts and then enter them to MIS. Percentage of data entered by
administrative workers declined over time, and two major reasons were that data forms resemble on
paper documents and one-page-printed summary after the visit [Leonard and Sittig 2007]
[Guru
The Overview on Information System Acceptance in Serbian Primary Care – The Case of Regional Center • 15:3 extracting data from the database, each record storing data about a visit is connected to the person and its role when created. By inspecting this we can determine if the record was created by the medical or administrative worker.
The main identified entities are medical services. They are grouped in visits. Visits can be linked and form the history of the disease. At the same time, one medical service contains a list of generated documents. Three most important groups are medication prescriptions, requests for further medical services and reports for requesting medical service. Table 1 shows an overview of the volume of the collected data. During examined period MIS had total 669 active users (both medical and administrative workers). It is expected that data related to the medical services should be entered by medical staff. Sometimes there are objective problems that stop doctors to use MIS, and in these cases, administrative workers should take paper-based documents and enter the data into the MIS. Looking at the organization of work we identified possible false positives among the records entered by administrative staff. In this context, false positives are the records entered by the administrative staff members when a medical professional does not have access to MIS. In some departments, many records of this kind is a result of the healthcare workflow. This was apparent especially in the departments where various therapies are applied. Many of them have only MIS installed on the reception desk, where either administrative worker or nurse with administrative privileges is logged in. In this kind of departments, interaction with MIS is done by the person working at the reception desk. The percentage of these records is significantly lower in departments where MIS is installed near the therapeutic workplace. The most important indication of MIS usage by medical staff members is a comparison of a total number of registered medical services against the number of medical services registered by administrative workers. Table 2 shows statistics by year together with overall statistics. Percentage of medical services registered by administrative workers vary literally in a range from 0 to 100%, depending on department and their differences.
The best overall system acceptance is in the laboratory, diagnostic departments and in dental service. In the case of laboratory and diagnostic departments, the main cause for this fact is integration between their equipment and MIS. Diagnostic department and laboratory were already equipped with devices storing acquired data in digital format. Physicians and technicians from these two departments were already trained to use some form of medical software and integration with MIS was a logical step for them. A small number of entries provided by administrative staff mostly resulted in data exchange errors caused by software or network issues. On the other side, dental service was not previously equipped with any kind of software for patient registration. The special situation with dentists is that they are entirely paid to the base of reported medical services.
All the records within the community nurse department are entered by administrative staff. Community nurses visit their patients and after the visits they create reports. At the end of the day, they bring reports to administrative workers that enter data into the MIS. They are not equipped with mobile devices now, and the responsibility to enter data to MIS is on administrative workers. Similarly, to community nurses, visiting doctors do the most of their daily job outside of the office. They perform medical examinations and therapies in the patient’s home. Unlike to community nurses they are equipped with laptops having installed MIS clients, so they can immediately enter visit related data. There is only 1.54% of records handled by administrative workers within their department. For the specialist department, the percentage of data not entered by medical staff is mostly under 5%. Exceptions are sports medicine and department of physical medicine and rehabilitation, where many new pieces of equipment are connected to MIS, similarly as in Lab department. 15:4 •
General practice and pediatrics departments generate most of the records. They are on the level under one quarter with the overall slow increasing rate. The rate is higher for adults than for pediatrics, and the reason is similar as with sports medicine. General practice and pediatrics consist also of therapeutic units. Therapeutic units within pediatric departments are better equipped with computers and administrative worker on reception need only to register a new patient for a therapy.
With this large number of medical services, we identified several categories of potential false positives when looking for data entered by administrative workers. The first category of potential false positives is found in parts of GP departments when specific therapy is applied. When patients come for some therapy, they usually interact with the administrative worker or nurse located at reception. After therapy is applied, the usually same person that received patient enter the data about the therapy. From the MoH business workflow point of view, this practice seems correct. Service is registered, and it will be eventually reported back to MoH. The problem that can occur is when the error happened and when it should be traced back to find the responsible person. Table II Overview of overall acceptance rate. Columns marked as “Total” show total number of medical services given within some department; while column “Adm %” represents the percentage of records registered by administrative workers Specialists
Dermatology Gynecology Epidemiology General surgery Internal med.
Psychiatry Ophthalmology HEENT
Sports medicine Physical medicine Prevention
General prev.
General psych.
Adolesc. psych.
Children psych.
Biostatistics
Sociologist Diagnostics Laboratory Dental service General practice
Adults Pre-school age
School age Visiting doctors Comm. nurses The most common identified false positive is registered therapy applied by intramuscular or intravenous injection. Table 3 displays the statistics. For general practice and gynecology, almost all registered medical services of this kind are registered by the administration. For all the other department's trend is decreasing, resulting in six out of eight specialist departments not having “invalid” inputs any longer in 2015. In GP department, only receptionists have installed MIS software and they enter both administrative and medical data. This problem is usually not visible in specialist departments since they are significantly smaller – they have fewer patients and staff members. They usually do not have a reception, and it’s up to medical personnel who apply a therapy to use the MIS. Gynecology, like GP, has many patients daily. In the year 2015, this department got new equipment and one additional therapeutic place. In this new place, medical personnel has installed MIS. In the year 2015, there were almost 30% more injections applied, and the overall false positive rate dropped from almost 100 to 81%. The similar situation is with sports medicine. When therapeutic positions got equipped with MIS in mid-2014, the number of registered false positives dropped significantly. In 2015 only 17 of 1112 inputs were reported by the administrative worker.
Analyzing the acceptance rate by medical professionals, we are satisfied. Our development project
started in 2009 and many medical professionals were included [
The intention of implementation of such style of visual forms was to provide easier adoption of the software. Before installation of MIS, we started with basic IT courses for potential users. This should overcome the technology barrier for older doctors (computer self-efficacy). Mentioned forms were on the line with objective usability and helped in avoiding computer anxiety with some users. With the strong perception of external control, users of our system started using the system and its data collecting forms. All of the mentioned facts went in the direction of PEOU, thus we had a good ground for system acceptance in the light of PEOU. From the point of view of TPB, forms were designed to strongly support existing habits supporting belief that the outcome of the newly introduced system will be as expected. In the beginning, knowledge, and skills needed to effectively use the system were an issue for some medical professionals. Thanks to the proper training and incremental deployment strategy [Rajković et al. 2013], until the end of 2012 our system had more than 400 active users.
All the mentioned facts were in the line to the model the behavior expected from prospective users [Kim and Park 2012]. Depending on the department, the initial acceptance by medical professionals was on the expected level and in the most of departments, the percentage of records registered by administrative staff declined during the time. For example, this percentage among specialist departments was 17.59% in 2012 and it dropped to 5.05% by the end of 2015. But at the same time, this parameter got significantly increased for preventive and consultative departments. At first sight, this looks like a bad trend. The actual explanation is that this department changed the operation mode during the years. First, in 2012, they registered only activities within the institution. Starting in 2013, they start registering all medical services and at the same time increased the activity. They increased the number of visits to different companies (general prevention), schools (adolescent psychologists) and other externals in order to promote the significance of prevention. Unfortunately, they registered external visits in the same way as community nurses. For general medicine, the overall rate of the records entered by administrative workers slowly grown from 23.79 to 23.96. In 2013, therapeutic department extent working hours, so some percentage of patients that would report to an emergency got the therapy within primary care center. It is important to state here that due to the organization of therapeutic sub-department in general medicine, many of these records can be identified as false positives. In many of these cases, the only reception has a connection to MIS, and the administrative worker is the only one that can enter data. On the base of data collected during four years of system’s full-scale use we could identify within which departments; MIS functionalities are accepted by medical staff with a higher percentage. The average on the institutional level is 87.94%. Of some 18.2 million of entered medical service related records around 2.2 million are these entered by administrative workers. If we exclude from this calculation 1.8 million of records entered within therapeutic departments, when medical staff members did not have access to MIS, acceptance level can be assumed as even higher. With this exclusion, a total number of records entered by administrative workers will be only 394,777. Comparing with 16.4 million retained records, this results in an acceptance rate of 97.6%. The good example is on the other side is gynecology department. Gynecology department was extended at the end of 2012, and during the next three years, they registered around 50% more visits in comparison with 2012. With some changes in workflow for therapeutic sub-department and the improvement of IT infrastructure, the percentage of the records entered by the medical staff increased. General medicine is on almost the same level, while in preventive medicine percentage of services registered by medical staff decreased during the years. For general medicine, this is mostly due to the organizational changes, while preventive medicine made the field work more intensive. Unfortunately, the staff from preventive medicine is not equipped with computers as well as community nurses. The counter-example is visiting doctors having a proper electronic device for registering visits. Their rate is above 99%.
Thanks to this research, we are now able to restructure our deployment strategies and to work closely with our potential users to improve organization within their departments. Having in mind analysis of collected data and looking at the three most important part of primary care: medical examinations, laboratory analysis, and therapeutic treatments; we can estimate the volume of collected data for future deployments and update deployment guidelines. Ideally, each workplace should be equipped with proper IT infrastructure and equipment allowing medical professionals uninterrupted use of MIS. But, in cases when we face limitations regarding access to MIS, some priorities must be defined.
The Overview on Information System Acceptance in Serbian Primary Care – The Case of Regional Center • 15:7 Looking at the overall data volume (Table 1) and comparing it with data collected in separate departments (Table 2), we can conclude that general practice and pediatrics department generates the significant majority medical examinations. They have registered more than 35% of all medical services, which is almost three times more than all the specialist branches together. Since our representative institution is a regional center, it consists of many specialist departments that are not present in the smaller center. If data from some smaller center is analyzed, the ratio will be even more in favor of general practice and pediatrics. For this reason, equipping mentioned departments with MIS is the primary goal. In all of our deployments, we usually start with general practice and then pediatrics follows.
Usually, the next goal is introducing MIS in specialist departments. For specialist departments, there are many dedicated data collection forms required for their daily work. Specialist departments collect finely granulated data and tracking more different parameters than general practice. When our target institution does not have enough IT equipment this is the first point when the decision should be made. Looking at the actual usage statistics, and the department organization, next department that should have MIS introduced is gynecology. Gynecology generates a significant percentage of data (more than 30% of all specialist departments), and keep, according to Serbian primary care organization, specialist medical sub-record. Also, within the gynecologist examination, physicians can create the same set of entities as general practitioners and pediatricians. The rest of the specialist departments are then next in the line together with therapeutic departments. Therapeutic departments, like physical medicine and rehabilitation, can exist as separate or as subdepartments under general practice, pediatrics or gynecology. Commonly, the main tasks for mentioned sub-departments are the application of intramuscular injective therapy and inhalation. Therapeutic departments are in the most of cases the latest parts of the institution that got MIS installed. Beside they produce many records (Table 3), management usually chooses to install software in specialist departments before. In the scope of our research, we tend to identify these records as false positives. The reason for this is that medical professionals simply do not have access to MIS. In the cases when therapeutic departments have software running, the percentage of registered medical services is even higher than in specialist departments focused on medical examinations.
When is needed to choose whether to install MIS to support specialist departments or therapy, usual decision is to go first with specialist department. Many records are on the side of therapeutics, but data collection forms used there are much simpler and nurse or administrative worker on the reception desk are qualified to fill them properly. For example, the average neurologist examination contains more than 30 parameters, while inhalation report contains the only list of medication with respective quantities. The best results we get in integration with laboratory and diagnostics. Since mentioned departments use equipment that automatically collects and store data, the implementation of proper data exchange protocol is the main task. When this integration is running, automatic data exchange ensures that many records that needed to be entered by administrative workers are on the level of statistical error.
Analyzing the acceptance rate by medical professionals, we are satisfied. For the future work,
bringing experience together from more different MIS systems from Serbian primary care would be
interesting. Our analysis if focused only on one type of MIS used in Serbian primary health, and
more significant results could be obtained if other MIS systems used in Serbian primary health care
like [
As it has been stated before, our development process relies on a participation of medical professionals. Their involvement in the project helped in later system acceptance. Our challenges 15:8 • came from many personal and organization issues from target medical institutions, but, in our favor, we had user interface design, good communication and intensive training sessions with potential users. Another external property that helped in MIS introduction was the fact that institution funding depends on reporting through MIS, and medical professionals usually want to have control over their own inputs. We can conclude that intensive contacts with the clients during all phases of system development resulted in the later good response from the users. In the cases when our target institution was not able to initially provide enough IT equipment, analysis like this is crucial when deployment plan should be defined. Thanks to this analysis we could, in later deployments, to define the sequence of departments that will get the MIS installed [Rajković et al. 2013].
When started our analysis we tried to select a primary care center that can give us results that can be easily generalized. We had similar research at the end of 2012, after the initial deployment and one year of exploitation. We used the results of the analysis from 2012 for other deployments of our MIS system [Rajković et al. 2013]. Now, after almost seven years of intensive use, presented results can be helpful for other studies and professionals starting with MIS deployment projects. The key for initial acceptance are functionalities that conform to PEOU principle. Since EHR based systems have both positive and negative impacts on medical practice in primary care [Holroyd-Leduc et al. 2011], positive effects will be better visible if the system acceptance among the medical professionals is on a higher level. We can conclude that doctors are willing to use MIS if it is on the line with their needs and make their regular work easier.