By Author
  By Title
  By Keywords

November 2015, Volume 65, Issue 11

Review Article

OSATS for total knee replacement: Assessment of surgical competence in the operating room

Authors: Shahryar Noordin  ( Orthopaedic Surgery, Aga Khan University, Karachi, Pakistan. )
Salim Allana  ( Department of Epidemiology, Emory University, Rollins School of Public Health, Atlanta, GA. )

Abstract

Traditionally the surgical and technical competence of Residents has been assessed inadequately and has received little attention among the core competencies defined by the Royal College of Physicians and Surgeons of Canada\\\'s CanMEDS programme and the Accreditation Council for Graduate Medical Education (ACGME).
With the development of novel and advanced surgical techniques with different learning curves, time pressure in busy operating rooms, and increasing complexity of cases at university hospitals, acquiring technical skills for Residents has become more challenging. Over the last two decades, methods have been developed to assess technical competence objectively. In this paper we describe use of an Objective Structured Assessment of Technical Skills (OSATS) for total knee replacement (TKR).
Keywords: Knee Arthroplasty, Medical Education, Residency, Orthopaedics. (JPMA 65: S-52 (Suppl. 3); 2015)


Introduction

An orthopaedic surgical trainee requires a fine balance between knowledge acquisition and the development of technical skills. Despite the fact that these two aspects are closely intertwined, surgical technical experience is the essential step in training an orthopaedic surgeon. Traditionally the surgical and technical competence of Residents has been assessed inadequately and has received little attention among the core competencies defined by the Royal College of Physicians and Surgeons of Canada\\\'s CanMEDS program and the Accreditation Council for Graduate Medical Education (ACGME).1 Current licensure requirements include oral and written exams, but objective measures to assess technical skills are not yet routinely used.
The current orthopaedic surgical Resident confronts the challenge of acquiring the same skills as his antecedents, but in a shorter duration of training time.2 Over the years, a reduction in Resident working hours and transformed working patterns, in addition to increased patient expectations with a decrease in overall training years, have resulted in a reduction in surgical training hours from 30,000 to 6,000.3 Consequently, contemporary surgical Residents have a significant decrease in their caseload.4 Furthermore, with the development of novel and advanced surgical techniques with different learning curves, time pressure in busy operating rooms, and increasing complexity of cases at university hospitals, acquiring technical skills for Residents has become more challenging. The department of Postgraduate Medical Education at Aga Khan University, Karachi, Pakistan, in line with international norms, also adopted the 80-hour workweek regulations for Residents in the winter of 2007. Therefore, the traditional surgical apprenticeship model based on experiential training, increased volumes of surgical cases, and subjective, observational assessment of Residents\\\' surgical skills needs to be revisited.
Historically, technical skills have been assessed in the operating room by supervision and feedback. The traditional paradigm of surgical training has valued quantity over quality. Using logbooks to measure experience and subjective assessment by the consultant based on recollection is unreliable, invalid, does not measure the actual skill level, and is prone to recall bias as well as halo effect. Logbooks lack content validity regarding the trainee\\\'s operative capability. Ultimately, skill deficiencies translate into inadequate post-Residency performance, resulting in poor patient outcomes. Therefore, on one hand surgical skills training needs to be made more efficient, while on the other hand, assessment methods need to be optimised in order to achieve the training objectives.
This literature review describes the use of an Objective Structured Assessment of Technical Skills (OSATS) for total knee replacement (TKR).


Material and Methods

In order to analyse this problem, a literature search was conducted using Eric, Google Scholar, Medline and CINAHL Plus databases from 1985 to March 2013 using search terms \\\'OSATS\\\', \\\'surgical competence\\\', \\\'total knee replacement\\\' and \\\'total knee arthroplasty\\\'. In addition to databases, controlled vocabulary, synonyms and truncation were used to identify appropriate references. Boolean operators were used to combine search terms and references. All manuscripts that incorporated an educational perspective regarding medical education and knee replacement were included.


Results and Discussion

The main crux of surgical Residency programmes is to provide training for technical and psychomotor skills to future surgeons. By convention, consultants supervise Residents performing surgery and based on subjective, distant recall of unstructured observation; determine when their trainees have achieved surgical expertise. These assessments have been shown to have poor test-retest and inter-observer reliability.4 Therefore, based on contemporary concepts of surgical education, valid, reliable assessment tools are required to ensure the competency of future orthopaedic surgeons. Furthermore, these validated objective measures should also be used routinely to gauge Residents\\\' performance as well as for their promotion.
Resident logbooks used currently are often dependent on accurate completion by the trainee and do not allow for real-time feedback and evaluation. These logbooks therefore lack content validity and do not adequately reflect the Residents\\\' technical skills. Therefore, an objective assessment linked to the logged procedure that assesses the Resident\\\'s performance would enhance the value of the procedure log in defining the Residents\\\' technical skills.1



An ideal assessment instrument should provide constructive feedback on performance as well as benchmark standards for credentialing. Over the last two decades, methods have been developed to assess technical competence objectively. Characteristics of an ideal assessment tool are that it should be valid, reliable, feasible, inexpensive, acceptable, easy to implement and ethically appropriate. General surgery programmes have gauged performance by analysis of dexterity, \\\'blinded\\\' video ratings using structured rating scales and virtual reality surgical simulation.2 However, there are few validated methods in orthopaedic surgery including global rating scale and video feedback. A study in 19975 reported using OSATS for the assessment of surgical Residents in a clinical skills laboratory setting where they were shown to have construct validity and inter-rater reliability. Traditionally, OSATS consisted of three components, including a procedure-specific checklist, a global rating scale and a pass/fail judgment. These laboratory-based assessments are expensive and time consuming. The benefits of these laboratory-based OSATS is that they can be used in simulations, live animal models and provide the opportunity of repeated practice without harming patients. However, they split assessment from the operating room setting dichotomising technical and decision-making skills, separating the various aspects of a mature surgeon, with no correlation that the sum of the parts is equal to the whole.5


Intraoperative use of OSATS was reported5 to discriminate between novice and expert laparoscopic cholecystectomy in a video-based assessment. A 2008 study4 showed intraoperative OSATS to be a valid and valuable tool based on trainer and trainee feedback questionnaires. The problems with existing parameters of assessment, including duration of surgery and complication rate, are crude and indirect measures and depend upon the difficulty of individual surgical case, including patient comorbid conditions. Intraoperative OSATS not only overcomes these disadvantages, but can also be used to assess a surgical Resident\\\'s training over time. Plotting OSATS score against Resident\\\'s level of experience gauges the Resident\\\'s progression and can be used to identify Residents who need more training. Because of the intraoperative setting, the cost is nominal with minimal use of faculty time.
Our objective was to develop a valid and reliable tool for the assessment of total knee arthroplasty performed by Residents in the operating room. An OSATS form was developed for primary knee arthroplasty competency assessment (Appendix). The form was based on the work done reported in literature.5-7


Conclusion

The search for an ideal method continues to find assessment tools for competency-based training. We need to continue to develop tools with improved discriminatory power. Lot of work is still needed to further validate methods for surgical assessment.


References

1. Gofton WT, Dudek NL, Wood TJ, Balaa F, Hamstra SJ. The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE): a tool to assess surgical competence. Acad Med 2012; 87: 1401-1407.
2. Leong JJ, Leff DR, Das A, Aggarwal R, Reilly P, Atkinson HD, et al. Validation of orthopaedic bench models for trauma surgery. J Bone Joint Surg Br 2008; 90: 958-65.
3. Chikwe J, de Souza AC, Pepper JR. No time to train surgeons. BMJ 2004; 328: 418-419.
4. Bodle JF, Kaufmann SJ, Bisson D, Nathanson B, Binney DM. Value and face validity of objective structured assessment of technical skills (OSATS) for work based assessment of surgical skills in obstetrics and gynaecology. Med Teach 2008; 30: 212-6.
5. Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 1997; 84: 273-278.
6. Hiemstra E, Kolkman W, Wolterbeek R, Trimbos B, Jansen FW. Value of an objective assessment tool in the operating room. Can J Surg 2011; 54: 116-122.
7. Trajkovski T, Veillette C, Backstein D, Wadey VM, Kraemer B. Resident self-assessment of operative experience in primary total knee and total hip arthroplasty: is it accurate? Can J Surg 2012; 55(Suppl 2): S153-S157.

Journal of the Pakistan Medical Association has agreed to receive and publish manuscripts in accordance with the principles of the following committees: