Call for your view!!!

The Global Health Network undertook a comparative evaluation of 13 Clinical Data Management Systems to assess them for their suitability for use in Low resource settings. We now want to ask the Global Health Network users whether you could add to these findings and if you agree. We then will publish the results and share this information with funders and capacity development organisations.

Please read the following and then share your experience and views in the discussion area provided. Tell us what systems you use and what the strengths or limitations are. Tell us what you think of the conclusions made here. Is this an area that needs investment and capacity strengthening?

Thank you – this could make a real difference. If we can gather evidence on the lack of training, skills and suitable systems to enable good (and accessible! ) data management then we could get this changed…. So read on and have your say!!!

To join the discussion please click here. 

Background

The developments in IT for medical research over the last 3 decades have led to improved and more efficient methods of managing research data[11-13; 21]. Numerous benefits have been associated with the use of clinical data management systems (CDMS) for collection, storage and curation of data [5]. For instance, discrepancies in data have been shown to reduce by 60-80% when edit checks or on screen checks are preprogrammed in the CDMS [6, 7]. The use of electronic data capture systems has been found to reduce time, cost, reduce data errors and therefore enhance the quality of the data collected [9, 10]. [14]. Some advanced CDMS support multisite data management through remote web data entry, real time status reporting, subject tracking and on demand subject randomisation [8].  Along with adopting best practices to ensure data is complete, audit trails and security features in CDMS provide added reliability and confidence in the data whilst also fulfilling regulatory requirements such as ICH GCP and 21 CFR Part 11. [3, 4].

Despite this significant progress in the development and utilization of tools for data management, low numbers of researchers in low and middle income countries use data management systems that are fit for purpose. Rich, scalable and secure CDMS that are compliant with regulatory requirements are expensive, particularly for individual researchers or groups with limited resources [1]. This ultimately means that these researchers are left with the unavoidable choice of using data management systems that are less-than-fit for purpose, or submitting case report forms offsite for data processing. Offsite data processing results in increased time needed for data quality assurance and missed opportunities for capacity building at study sites. Use of systems that do not meet industry standards, may also compromise the quality of data and ultimately influence the reported study results. This undesirable set of circumstances has been acknowledged by the some funding groups such as the European and Developing Countries Clinical Trials Partnership and the African Malaria Network Trust [1]. The availability of inexpensive or an open source CDMS for individual researchers has the potential to improve clinical research and enhance academic standards by reaching a wider audience [5]. While these systems fulfil some key requirements of CDMS systems, the set up and maintenance also requires technical knowledge that may be lacking for some independent researchers or may not be worthwhile for researchers setting up relatively small or non-complicated studies.

The objectives of this research was to: perform a requirements analysis for clinical data management systems in low resource settings; Identify data management software currently used by researchers in low and middle income resource settings; Compare features and functionality of data management systems that exist in the market and lastly to identify data management systems that are suitable for researchers with limited resources.

Methods

A three stage sequential exploratory mixed methods design was used [24].

In the first phase, qualitative data was collected through focus group discussions and interviews with the aim of identifying data management systems that are used in low resource settings along with desirable system requirements of users based in these areas [15-16]. The output of the qualitative data was a listing of data management systems being used in LMIC settings and a list of ideal characteristics of a clinical data management system. This list informed the development of an evaluation criteria and scoring matrix.

In the second phase, supplementary data was collected from a search of electronic journals to identify which data management systems have been used for previous studies. This provided information from a larger population that would have otherwise not been reached through the surveys.

Finally, an Expert Opinion Assessment was performed in which quantitative data was collected through scoring of the data management system against the established evaluation criteria. The level of consensus was determined through computation of the Kendall’s Coefficient of Concordance.

Data collection

First phase: 14 interviews and 2 focus group discussions were held with investigators and data managers working in low resource settings in Africa and Asia.  The respondents were either independent investigators with no affiliation to an established research institution or investigators and data managers working in research institutions that are based in, and run studies in low and middle income countries. The focus group discussions were held with data managers and statisticians in established research institutions in Asia. Data variables from the interviews and discussions helped inform the compilation of the evaluation criteria. These variables are listed in Table 1.

Table 1: List of variables of interest from interviews

Second phase: A search was conducted on two open access journals. Search queries were based on the following key terms: "EDC System" and "Clinical Trial", "Electronic Data Capture System" and "Clinical Trial", "EDC System", "Electronic Data Capture System", "Data Management" and "Electronic Data Capture System", "Data Management" and "Low and Middle Income", "Data Management" and "Asia" and "South East Asia" and” Africa", "EDC system" and "Low and Middle Income", "EDC System" and "Asia" and "South East Asia" and " Africa" and findings as outlined in the Figure 2:

Figure 2: Flow diagram of electronic journal search

Based on data from the qualitative interview findings and electronic journal search, 13 candidate data systems were identified for evaluation: CS Pro, Excel, Epidata, EpiInfo, MsAccess , REDCap, ClinCapture, OpenClinica, MACRO EDC, In-Form, Oracle Clinical, ODK, ClinRes

Final phase: User requirements from the interviews were collated and findings summarized into 27 characteristics (Appendix 1). These were further grouped into 6 main domain areas: 1) CRF/Questionnaire Development, 2) Data Entry/Review, 3) Data Quality, 4) Standards and Compliance, 5) Reporting 6) Cost and Implementation (See Table 3).

Based on interview and discussion findings, an evaluation criteria was compiled and a scoring system  generated; scores were based on a weights that were generated based on the importance of the criterion to data quality and management. In the weighted decision matrix, all decisions summed to 1 as in figure 4:

Where n is the number of valuations and W is the weighting factor [17].  A sensitivity analysis was conducted to check robustness of the results by looking at the effects of changing the weights.

The scoring was performed by 7 independent data management experts with experience in running academic and industry studies. Experts were identified based on their level of experience in data management and experience in using the identified data management systems in low and middle income settings. Motivational and cognitive bias was detected through probability distribution models. Responses that showed a skew and/or shift in the distribution were identified. The results were shared with the expert and where necessary a second round of evaluation was conducted. Kendall’s coefficient of concordance was computed to measure the convergence of the expert’s opinions of the rankings.

where W is the Kendalls co-efficient and S is the sum of squared deviations, n is the total number of CDMS evaluated and n is the number of experts.

Results

First Phase:

Table 3: The evaluation criteria

Second Phase: 

The 13 Data Managements Systems 

Microsoft Access

Microsoft Access (MsAccess or Access) was developed and first released in 1992 by Microsoft as part of the Microsoft Office suite of applications. It consists of the Microsoft Jet Database Engine, a graphical user interface and tools for scripting. Access may be used as a front-end system in combination with a backend relational database such as Oracle, MS SQL server and MySQL. It may also be used as a back-end database or as a complete database solution hosting both front end and back-end in Access. Visual Basic for Applications (VBA) scripting allows developers to create Access applications that address unique study requirements, have enhanced data manipulation, and advanced user control features. An Access license costs approximately 160 USD as of October 2015. Access can be used to create applications for clinical trials and observational studies.

OpenClinica

OpenClinica is an open source clinical research software developed by Akaza Research (now OpenClinica LLC) in 2005. The system is built using Java programming language and uses either an Oracle or PostgreSQL database. It supports eCRF creation and clinical data management for clinical research studies including multi-site studies. With over 18,000 registered members in 2014, the system is widely used for clinical trials data management around the world including low and middle income countries. The system has been developed following the 21 CRF part 11 guidelines and is compatible with CDISC ODM standards. There is an active user community and software documentation is available online.  Although the software is available at no cost, funds are needed for infrastructure and user training. The application does not support offline data capture; however, offline data capture features can be created through additional programming and utilizing the inbuilt web services feature. OpenClinica is suitable for clinical trials and has been used to capture data for observational studies including longitudinal studies.

ClinCapture

ClinCapture is an open source system that was developed and released in 2009 by Clinovo Inc. ClinCapture was developed from OpenClinica source code (version 3.1.3).  ClinCapture contains all the features in OpenClinica version 3.1.3, but with a modified data entry workflow and integration with tools such as Randomize (allows patient randomization), ePro (an eClinical system), Clinical Trial Management System (CTMS), CDISC mapping tool and Pentaho (a reporting tool). As of July 2014, the user community had over 500 members.

Epidata

Epidata is an application developed by EpiData Association in 1999 using Borland Delphi Pascal programming language. The software is available for free for organizations in low-income settings. The software is widely used for collection of quantitative data for routine surveillance, outbreak investigation and research across the world. Organizations such as the World Health Organization and Médecins Sans Frontières have widely used Epidata in data collection and analysis in epidemiological studies and clinical trials respectively. The software has three modules: A form designer module called Epidata Manager, a data entry and data documentation module called Epidata Entry and an analysis module, Epidata Analysis. The software does not support multiple users on a network. It is suitable for collection of data for observational studies and can be used for data collection in areas with no internet connectivity.

 EpiInfo

EpiInfo is an open source application developed and maintained by Centers for Disease Control and Prevention. It was first released in 1985 and is currently built using Microsoft’s C# .NET programming language. It constitutes of three main parts; a Form Designer component, Enter (a data entry component) and Analysis (a data analysis component). Additional features include GIS mapping, a reporting dashboard and a report generation component. Epi Info™ is used worldwide for the rapid assessment of disease outbreaks; for the development of small to mid-sized disease surveillance systems; as ad hoc components integrated with other large scale or enterprise-wide public health information systems; and in the continuous education of public health professionals learning the science of epidemiology, tools, and techniques. It is suitable for observational studies.

Microsoft Excel

Microsoft Excel is a spreadsheet application developed by Microsoft as part of Microsoft Office suite of applications and is popularly known as Excel. It was first released in 1993 and is currently used on both Microsoft Windows and Mac Operating systems. Its key features include functions and calculations, graphing tools, pivot tables and charts, and a macro programming language called Visual Basic for Applications similar to the Ms Access VBA. Excel has been widely used to collect and store data in a flat file format. However, data integrity is difficult to maintain as changes to data is not easily tracked.

CS Pro

Census and Survey Processing System (CSPro) is a software package that allows users to enter, edit, tabulate, and disseminate census and survey data. CSPro is developed and supported by the U.S. Census Bureau and ICF Macro, the organization that implements the Demographic and Health Surveys (DHS) and has been actively used since 2000. CSPro is offered as an open-access software and is currently used in over 160 countries handling up to 235 million records in censuses and surveys. Through CSPro’s procedural programming language users can define data validation checks and create customized applications. In addition to features for developing questionnaires and the database, CSPro can be used to perform basic analysis using the tabulate function. The system documentation describes the programming language as well as CSPro’s features. The publicly accessible software is in binary form (the code is not accessible) and runs exclusively on Microsoft Windows. CSPro is suitable for handling census and survey data.

REDCap

REDCap is a web-based application designed to support electronic data capture for research studies. The system was developed by a multi-institutional consortium initiated at Vanderbilt University. The application is available at no cost; however the programming code is not open to the public. Questionnaires are created through an online designer or by uploading a metadata (data dictionary) file in Microsoft Excel. It contains an inbuilt audit trail, role-based access levels for users and ability to integrate with other information systems through use of web services. REDCap is designed to comply with HIPAA regulations, however it is not validated for 21 CFR Part 11 compliance yet. REDCap is web-based, indicating that a network connection is needed for multiple users to access it simultaneously. 

Open Data Kit

Open Data Kit (ODK) is a free and open-source set of tools which allows users to collect and manage data collected on mobile devices. There are three main components: ODK Build (allows users to create  data collection, XLSForm is recommended for larger forms); ODK Collect (facilitates collection on a mobile device); and ODK Aggregate (aggregates the collected data and extracts it for analysis).

ODK has been used in health surveys, decision support systems and for mapping mostly in Africa.

 In Form

In-form is a web-based electronic data capture (EDC) solution that collects and manages patient information in clinical trials. It was first developed by PhaseForward Inc which was acquired by Oracle Inc in 2010. It is widely used with more than 100,000 active sites worldwide with investigators trained in InForm. InForm allows for quick development of the study and provides an intuitive user interface. InForm is a proprietary data management system.

CliRes

CliRes is a data management system developed by the Oxford University Collaborative Research Unit in Vietnam. The software is Windows based and runs on SQL Server database. The system can be used for both intervention and observational studies. It is currently used for studies running in OUCRU and has over 300 users. The system is available as a web-based system and is intended to be made publicly accessible as an open access system. We selected CliRes as an example of institutionally owned and developed data management system.

Oracle Clinical

Oracle Clinical is a product of Oracle Corporation that is widely used in pharmaceutical, biotechnology, medical device, and contract research organizations. Oracle Clinical’s global library enforces standards and consistency throughout all studies developed in the system. It supports electronic data capture and hybrid studies by using the same study definitions and page layouts for both electronic and paper environments. Streamlined study management functionality allows staff to assign multiple patients to a study book, while also providing complete control over protocol amendments. It is designed to provide a seamless experience for sites to capture and process clinical data.

 MACRO EDC

MACRO is a web based clinical data management system developed by InferMed and released in 1999. MACRO has been used for clinical data management in commercial and not-for-profit clinical research. It is widely used in European academic research units. MACRO is designed to support compliance with the requirements of ICH Good Clinical Practice and FDA 21 CFR Part 11. It is based on a client-server architecture and runs on Windows Operating System.

Final Phase:

13 data management systems were selected for scoring according to the evaluation criteria. Results were only  produced for 11 of the data management systems. ODK and CliRes were excluded from the analysis as ODK did not fulfil the criteria for use as a clinical data management system in isolation, while CliRes is an institutional software that may not be shareable with the wider public yet. A summary of the scoring of software according to the evaluation criteria is given in table 2.

Table 2: Summary of scoring of software

Discussion

Cost and implementation

Data management systems costs are constituted of one-off expenses for software licensing, infrastructure costs, and running costs such as internet, electricity, user support/license renewals and staff salaries. Proprietary systems will cost from $100 to over $100,000 for the software license. Additionally, proprietary software may require periodic renewal of licenses to enable users to be able to gain access to support and training from the vendor, and to allow upgrade to new software releases. While open source and open access software eliminate the software costs, there is a risk of lack of free, consistent and centralized support; inadequate documentation and compatibility problems because of changes in technology [21]. The cost of training and support services may overshadow initial savings. To mitigate these risks, it is important to identify systems with a strong presence and an active user community. REDCap and OpenClinica have large and resourceful online communities, while ClinCapture is fairly new and with a smaller user community.  Epidata and EpiInfo have strong online user communities while the CSPro online community is smaller but active with about 300 members as at July 2014.

Of the 13 systems evaluated, 8 are built for deployment in a client-server architecture (See Figure 5). This requires extra hardware and the setup of a network. Web-based systems are ideal for multisite studies or studies requiring real time data entry and with good network connectivity. Researchers with limited or no access to internet can set up web-based systems on personal computers in a local network or utilise desktop-based systems such as Epidata, EpiInfo and CSPro. Several attempts with varying success rates have been made in setting up mobile data collection systems. However, implementation is expensive or does not meet regulatory requirements required for best practice standards for clinical research. EpiInfo can be set up on an enterprise database (SQL Server) and can support multiple users concurrently.

Among the systems evaluated, REDCap, MACRO and InForm support offline data collection. The REDCap Data Transfer Service (DTS) allows data to be pushed into a REDCap study database from external systems and provides an adjudication process for checking the quality of incoming data based on the defined data quality rules. The REDCap API can also be utilized to automate data import and validation into REDCap from other REDCap instances or from a CSV source file. Earlier versions of Macro EDC supported mobile a data collection; this was however discontinued. InForm supports offline data capture through the InForm Unplugged module. Although efforts have been made to create mechanisms for offline/mobile data collection for ClinCapture and OpenClinica, there is no officially validated module for offline/mobile data collection for these two systems. ClinCapture and OpenClinica have web services functionality; systems integration through web services requires programmatic knowledge and intensive testing for validation.

Figure 5: Desktop based software vs Client-Server architecture software

Functionality and usability

An ideal data management system should be user friendly for data entry users and for eCRF developers. Epidata, EpiInfo, CSPro feature intuitive questionnaire design features. These systems support drag-and-drop features for questionnaire/ eCRF design thus novice users can set up questionnaires and validation checks within a few hours of training. Although the MsAccess and CSPro drag-and-drop feature makes it appealing for form design, extra programming is required to extend functionality beyond basic data entry. REDcap and MACRO have an intuitive form design process with a drag-and-drop feature for creating data elements and forms.  The form design elements in REDCap are however limited and thus eCRFs may not resemble pCRF. ClinCapture and OpenClinica require definition of the CRF metadata in a spreadsheet for the form to be generated. OpenClinica and ClinCapture eCRFs can be modified to closely resemble the paper CRF through application of HTML formatting and use of Javascript.

The data entry workflow in OpenClinica is lengthy and requires considerable amount of user training. ClinCapture eCRFs are similar to OpenClinica; the data entry workflow is however shorter. MACRO is intuitive and user friendly. Usability of Epidata, EpiInfo, CSPro and Access is dependent on the eCRF design, but generally intuitive. Double data entry is well supported in OpenClinica, ClinCapture, EpiInfo, and CSPro.

Query management system was found to be inbuilt within OpenClinica, ClinCapture, MACRO, InForm and Oracle Clinical. Data extraction in text and delimited data formats such as CSV is supported in all the systems evaluated.

Limitations

Data was collected from South East Asia and Africa; more data from other LMICs such as Central/South America, Middle East Asia may provide further insight on challenges experienced. Further work on mobile data collection for clinical research will inform this initiative. The software products analyzed in this work are continually evolving thus continual analysis should be performed

Conclusions

For non-intervention studies, EpiInfo, CSPro, REDCap and Epidata were considered as potential candidates. EpiInfo and CSPro are designed to run exclusively on Windows operating systems while Epidata can run on both Windows and Linux platforms. Source code for Epidata and CSPro are not available in the public domain and therefore the systems cannot be independently customized to create or modify features. Provided the target user has a Windows operating system set up, EpiInfo is superior to Epidata and CSPro with regard to sustainability and future expansion of the system. REDCap’s statistical and analysis module is less sophisticated than EpiInfo; on the other hand, REDCap is compliant with HIPAA and supports electronic signatures, audit trails and role-based user access levels. In cases where a web-based system is required, REDCap offers the most comprehensive set of functionality coupled with ease of use. However, there is limited flexibility in design and appearance of the electronic questionnaires. Offline/ Mobile data collection and synchronization will need to be thoroughly tested and appropriate user training resources created. If a desktop-based solution is desired, EpiInfo is superior to other desktop-based systems; it will however require users to run a Windows operating system and have also Microsoft Access installed on their computers.

For clinical trials, REDCap, OpenClinica, ClinCapture, MACRO EDC, In-Form and Oracle Clinica were considered as potential candidates.  With the exception of  REDCap all systems are validated for compliance with FDA 21 CRF Part 11 whilst RedCap fulfils requirements of the 21 CRF Part 11, these requirements have not yet been validated for compliance. MS Access applications can also be developed and validated for compliance with regulatory requirements. However, this requires significant investment. Although MACRO EDC, In-Form and Oracle Clinical fulfill all the requirements for clinical trials data management, these applications are an exceptionally expensive consideration for use by researchers in low and middle income countries at the current market rate. ClinCapture and OpenClinica offer the same set of functionality as the proprietary counterparts and offer a more cost-effective solution for CDMS implementation.

While ClinCapture has more alluring features, it has a much smaller user base than OpenClinica. In 2014, OpenClinica had over 18,000 members of the community while ClinCapture had less than 500 members. In addition, ClinCapture was developed as a fork of OpenClinica code; this means that when new developments are made in OpenClinica, ClinCapture will need to rework all the enhancements into the new version of OpenClinica in order to enjoy new developments in OpenClinica.   OpenClinica’s user base, larger and active user community makes it preferable as a choice open source software for clinical trials.

This analysis shows that there are wide variety of CDMS and software available for clinical data management. While no one system fits all criteria, effort can be made to enhance existing systems to fit the requirements required for researchers in low resource settings.

Acknowledgements

Sincere thanks to InferMed Inc and Clinovo for their helpful correspondence regarding their software applications. A big thankyou to all the participants and respondents in the survey and The Global Health Network data management experts.

References

1. Fegan GW, Lang TA. Could an open-source clinical trial data management system be what we have all been looking for?. PLoS Medicine 2008;5;3: 347-349
2. Gerritsen MG, Sartorius OE, vd Veen FM Meester GT. Data Management in multi-centre clinical trials and the role of a nationwide computer network. A 5 year evaluation. Proc Annu Symp Comput Appl Med Care. 1993:659-62
3. Krishnankutty B, Bellary S, Kumar NBR, Moodahadu L. Data management in clinical research: An overview. Indian J Pharmacol. 2012; 44; 2: 168–172.
4. Food and drug (FDA) (2009) Cod of Federal Regulation Title 21 Part 11 Electronic Records Available http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/cfrsearch.cfm Accessed 24 July 2015

5. Shah J, Rajor D, Pradham S, McCready M, Zaveri A, Pietrobon R. Electronic data capture for registries and clinical trials in orthopedic surgery. Clin Orthop Relat Res 2010;468:2664-2671.
6. Velikova G, Wright EP, Smith AB, Cull A, Gould A, Forman D, Perren T, Stead M, Brown J, Selby PJ. Automated collection of quality of life data: a comparison of paper and computer touch-screen questionaires. J Clin Oncol. 1997;17:998-998
7. Prokscha S. Practical Guide to Clinical Data Management. 2nd ed. Boca Raton, FL: CRC Press; 2006.
8. El Eman K, Jonker E, Sampson M, Krleza-Jeric K, Neisa A. The use of electronic data capture in clinical trials: web survey of 259 Canadian trials. J Med Internet Res 2009;11;1: e8
9. Welker JA. Implementation of electronic data capture systems: Barriers and solutions. Contemp Clin Trials. 2007;28:329–36
10. Sujansky WV. The benefits and challenges of an electronic medical record: much more than a “word-processed” patient chart. West J Med. 1998; 169:176–183. 
11. R.A. Kronmal, K. Davis, L.D. Fisher, R.A. Jones, M.J. Gillespie, Data management for a large collaborative clinical trial (Cass: Coronary Artery Surgery Study), Comput. Biomed. Res. 11 (1978) 553–566. doi:10.1016/0010-4809(78)90034-4.
12. M.A. Hasson, R.M. Fagerstrom, D.C. Kahane, J.H. Walsh, M.H. Myers, C. Caughman, et al., Design and evolution of the data management systems in the prostate, lung, colorectal and ovarian (PLCO) cancer screening trial, Control. Clin. Trials. 21 (2000) 329S–348S. doi:10.1016/S0197-2456(00)00100-8.
13. C.A. Brandt, S. Argraves, R. Money, G. Ananth, N.M. Trocky, P.M. Nadkarni, Informatics tools to improve clinical research study implementation., Contemp. Clin. Trials. 27 (2006) 112–22. doi:10.1016/j.cct.2005.11.013.
14. J. a. Welker, Implementation of electronic data capture systems: Barriers and solutions, Contemp. Clin. Trials. 28 (2007) 329–336. doi:10.1016/j.cct.2007.01.001.
15. E.T. McFadden, F. LoPresti, L.R. Bailey, E. Clarke, P.C. Wilkins, Approaches to data management, Control. Clin. Trials. 16 (1995) 30–65. doi:10.1016/0197-2456(94)00093-I.
16. J.D. Franklin, A. Guidry, J.F. Brinkley, A partnership approach for Electronic Data Capture in small-scale clinical trials, J. Biomed. Inform. 44 (2011) S103–S108. doi:10.1016/j.jbi.2011.05.008.
17. George ED, Schmit LG. Engineering design. 4th ed. Mc Graw Hill; 2009
18. R.G. Duncan, D. Saperia, R. Dulbandzhyan, M.M. Shabot, J.X. Polaschek, D.T. Jones, Integrated web-based viewing and secure remote access to a clinical data repository and diverse clinical systems., Proc. AMIA Symp. (2001) 149–153. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2243282/.
19. B. Ene-Iordache, S. Carminati, L. Antiga, N. Rubis, P. Ruggenenti, G. Remuzzi, et al., Developing Regulatory-compliant Electronic Case Report Forms for Clinical Trials: Experience with The Demand Trial, J. Am. Med. Inform. Assoc. 16 (2009) 404–408. doi:10.1197/jamia.M2787.
20. M.M. Ngari, N. Waithira, R. Chilengi, P. Njuguna, T. Lang, G. Fegan, Experience of using an open source clinical trials data management software system in Kenya, BMC Res. Notes. 7 (2014) 845. doi:10.1186/1756-0500-7-845.

20. R.A. Kronmal, K. Davis, L.D. Fisher, R.A. Jones, M.J. Gillespie, Data management for a large collaborative clinical trial (Cass: Coronary Artery Surgery Study), Comput. Biomed. Res. 11 (1978) 553–566. doi:10.1016/0010-4809(78)90034-4.
21. Kappa statistics: https://books.google.co.th/books?id=YZIQO0acuWwC&pg=PA294&lpg=PA294&dq=agreement+rating+stata&source= bl&ots=YqA3wXHqZd&sig=9jWkun7mu6Na2XHr97xviHC0uxA&hl=en&sa=X&ved=0CEgQ6AEwB2oVChMI_8ihhOSgyAIV0VuOCh3ZpAC9#v=onepage&q=agreement%20rating%20stata&f=false
22. L. Morgan, P. Finnegan, Beyond free software: An exploration of the business value of strategic open source, J. Strateg. Inf. Syst. 23 (2014) 226–238. doi:10.1016/j.jsis.2014.07.001.
23. J.W. Creswell, Research design: Qualitative, quantitative, and mixed methods approaches, Sage publications, 2013.