Clinical prediction rules (CPRs) are intended to improve patient care, but critics have suggested that—for a number of reasons— the opposite may be true. Lower extremity experts interviewed by are in agreement that, ultimately, even the best CPRs should be only one piece of the clinical puzzle.
By Cary Groner
Clinical prediction rules (CPRs)—mathematical pattern-based tools intended to guide medical decision making—elicit both praise and skepticism from their intended users, primarily physicians and physical therapists.
At the 2011 annual meeting of the American Physical Therapy Association (APTA), an Oxford-style debate on the pros and cons of CPRs ended with most of the audience members siding with those arguing against the approach.1 One participant in the APTA debate further detailed the pitfalls of CPRs in published editorials,2,3 most recently in the March 2016 issue of the British Journal of Sports Medicine (BJSM).3
Part of the controversy lies in CPRs’ heterogeneity; some are well designed and validated, others less so. Others fear clinicians will develop an overreliance on CPRs, to the point where unique aspects of a particular case may be overlooked.
Even the name is problematic for some people. Jenny McConnell, AM, FACP, a physical therapist in Sydney, Australia, known for her eponymous patellar taping technique, told LER: “I don’t like the term ‘rule’; to me that suggests that people stop thinking, that if something doesn’t fit the rule they dismiss it and don’t accommodate the patients’ individual differences.”
But it’s worth noting that in the APTA Oxford debate, even those who argued in support of CPRs emphasized that they should be used as a starting point for clinical decision making, not as its sole determinant.2 Similarly, experts interviewed by LER were in agreement that, ultimately, even the best CPRs should be only one piece of the clinical puzzle.
In essence, CPRs—also known by less provocative names such as “probability assessments” and “prediction models”—use a medley of predictive variables, such as patient characteristics and investigation results, to estimate outcome probabilities and help determine the interventions most likely to be effective.4 The best ones are both highly sensitive and highly specific, meaning they identify most of the patients who will benefit from a given intervention and exclude most of those who won’t. This goal is much easier to state than to achieve, however, and coming up with useful CPRs remain a challenge.
Establishing CPRs require four phases:4
- development (the identification of predictors from observational studies, also called derivation);
- validation (reliability testing of the rule in a separate population);
- impact analysis (clinical assessment of the rule’s usefulness in terms of costs and benefits, patient satisfaction, time and resource allocation, etc); and
- implementation (widespread acceptance and adoption).
“Essentially, people are looking for patterns that could identify the best way to treat a specific problem,” said David Armstrong, DPM, MD, PhD, a professor of surgery and director of the Southwestern Academic Limb Salvage Alliance (SALSA) in the Keck School of Medicine at the University of Southern California, in Los Angeles. “It’s the promise of predictive analytics to try to find guidance, not only for clinicians, but also for administrators upstream.”
Armstrong acknowledged the aforementioned criticism that CPRs might make clinicians lazy. In response, he likened good CPRs to the autopilot on commercial airliners.
“Is flying safer than it used to be as a result?” he asked. “Yes. But is the pilot apt to be less able to fly the plane than if he or she wasn’t used to autopilot? Also yes.”
As an example, he noted the 2013 Asiana Airlines crash in San Francisco, a situation in which the pilots had grown so reliant on autopilot that when the airport’s system was temporarily down, they were unable to land the plane safely on their own.
Part of the problem with CPRs is that far more of them are developed than are validated or implemented.5 In fact, according to Daniel Riddle, PT, PhD, the Otto D. Payton Professor of Physical Therapy at Virginia Commonwealth University in Richmond, most of the CPR literature is based solely on derivation.
“There are far fewer that have been externally validated, but that’s where clinicians should spend their time,” Riddle said.
When validated, CPRs—like autopilot—are able to incorporate vastly more data in critical decision making than the human brain is able to. They aren’t intended to supersede existing clinical practice guidelines, but rather contribute to their evolution.
“There are times when decision rules could give the wrong advice, but I don’t think we should throw the baby out with the technological bathwater,” Armstrong said. “Clinicians pride themselves on their didactic knowledge, their experience, and their judgment, and they don’t want to cede these things to some list. But at the same time, CPRs can give you insight into conditions you may rarely see. If we can augment what we see with what we learn from having more data, I don’t see how that’s bad. Who would want to turn a blind eye to more information?”
Studies have shown that clinicians often respond differently to CPRs based on their age and level of experience, for that matter, with younger physicians and residents often more open to them.6,7 Emergency medicine providers are more likely to embrace CPRs than are internal medicine physicians, as well, possibly due to the increased pressure they face in making urgent diagnostic and treatment decisions.8
CPRs are often subdivided into three functional groups: diagnostic (just what it sounds like), prognostic (designed to predict an outcome such as success or failure); and prescriptive (designed to identify the most effective interventions).9 The clinicians LER spoke with were primarily—though not exclusively— interested in prescriptive rules, as these are most likely to affect clinical decision making and protocols regarding lower extremity issues.
Daniel Riddle explained how best to go about creating and validating a CPR.
“To develop a clinical prediction rule, whether prognostic, diagnostic, or to test an intervention, one has to first apply a series of examination procedures or medical history questions to a large group of people, apply statistical procedures to figure out which of those best differentiate between people who do and don’t end up with the outcome of interest, and then come up with a statistical model that includes all the important variables,” he said. “To validate the rule, you have to apply that statistical model to an independent set of patients—unrelated to the first set—to see if it applies to them as well.”
David Armstrong addressed the matter with regard to lower extremity issues in particular.
“I think we can look at a lot of things related to decision making in physical therapy and in orthopedics,” he said. “Surgeons are always trying to refine their craft and find better ways to treat patients. Right now, we’re trying to get better data on certain aspects of treating osteomyelitis. There are data from labs, data from imaging, the type of bacteria growing or the antibiotic used, certain symptoms—putting all that into a clinical prediction rule might be really helpful in guiding long-term therapy, both surgically and in cases where we suppress the bone infection.”
Armstrong’s team has also identified algorithms involving frequent measurements of temperature and inflammation that could identify the site of a future diabetic foot ulcer weeks before the wound appears. Earlier studies have already suggested the utility of CPRs in diabetes; for example, a 2004 paper in Diabetes Care found treatment algorithms led to substantially better A1C and lipid levels than did standard care.10 And a 2015 systematic review of 16 studies involving more than 16,000 participants suggested the creation of a CPR for foot-wound risk, including insensitivity to a 10-g monofilament or lack of pedal pulse, as well as a history of foot ulcers or lower extremity amputation,11 not that these risk factors would come as a surprise to many lower extremity specialists.
Studies have demonstrated the utility of CPRs for a variety of lower extremity conditions, including patellofemoral pain (PFP). In a 2010 paper published in BJSM, for example, researchers from the University of Queensland in Brisbane, Australia, reported that CPRs for treating patellofemoral pain with foot orthoses suggested four variables were most likely to predict success: age older than 25 years, height less than 165 cm (roughly 5′ 5″), worse pain on a visual analog scale, and a difference of more than about 11 mm in midfoot width from nonweightbearing to weightbearing. Patients with three of these four variables had a success rate of 86% with orthotic treatment, versus 40% in the study population as a whole (n = 42; 24 women).12
Natalie Collins, PT, PhD, a research fellow at the university and one of the paper’s authors, told LER the participants received off-the-shelf orthoses that were then individually customized with heat molding and the addition of wedges. She added that the study helped elucidate which risk factors were modifiable and which weren’t.
“Obviously age and height can’t be modified, but the change in midfoot width was very interesting,” Collins said. “An orthotic may help someone with a more mobile foot in giving some support and input to potentially better control their foot when under load. We’re now trying to validate that.”
When treating patients with PFP in her own clinical practice as a physical therapist, Collins said, if she sees some of these characteristics—particularly the change in midfoot width—she might alter her approach accordingly.
“I wouldn’t necessarily deviate from what we already know works well for patellofemoral pain— exercise, or patellar taping in some cases—but certainly if someone has a more mobile midfoot, I may try an orthotic earlier in the treatment program,” she said.
Collins did urge caution when using CPRs to predict outcomes, however.
“Ours is still a derivation study that hasn’t yet been validated,” she said. “We don’t know if it will be generalizable to all people with patellofemoral pain. A validation study has recently been completed by Mark Matthews and Bill Vicenzino [data are currently being analyzed], and we can’t really draw too many conclusions until we know the outcome of that study.”
Another Australian study, published in 2011, reported on 60 patients with PFP who were given prefabricated, noncustomized foot orthoses and assessed for improvement 12 weeks later.13 Although only 25% of the patients reported marked improvement, that number jumped to 78% if three of the following four criteria were met: less supportive footwear, lower pain levels, ankle dorsiflexion range of motion of less than 41° with the knee flexed, and less single-leg squat pain with the orthosis than without.
One of the paper’s coauthors, Christian Barton, PhD, a postdoctoral researcher at La Trobe University in Melbourne, started investigating approaches to PFP a decade ago.
“I wanted to try to create a bit more guidance for clinical practice,” Barton explained. “I would prescribe orthoses, and sometimes people would do really well with them; at other times, they didn’t really help. You don’t want to give them to people they won’t benefit. We created some functional tests to see if it was possible to change pain or function immediately with orthotics, and if that was a predictor.”
The findings, Barton said, may provide some clinical guidance.
“First, we know that the chances are slim of a patient getting better just by being given orthoses,” he said. “But we should still consider them in conjunction with other things. For one, we found that if patients have limitations in their ankle dorsiflexion range, they’re more likely to benefit from the orthosis—even though I’d also try to improve their mobility. They also benefit more if they wear less-supportive shoes, so that means you can give them orthoses or they can get better shoes. Finally, the biggest predictor was that if you have a pain reduction during a single-leg squat with the orthoses, your chance of improvement gets markedly better, from twenty-five percent to forty-five percent.”
These findings may have a variety of clinical implications, Barton said.
“The most important thing is managing patients’ loads; a lot of people have pain because they’re trying to do too much exercise,” he said. “Second, if they’ve had pain for a long time, they’re going to have quadriceps and gluteal weakness, so you’ll have to do rehab to address that. And if we find that they can do more squatting-based exercises with the orthotic in the shoe, then you can use that; it’s really a pain modifier rather than something that will cure the condition. But if the pain is better, they’re going to be more likely to do the other things we want them to do.”
One of the oldest and most widely accepted CPRs is the result of the Ottawa Ankle Rules (OAR) Study, first published in 1992, which was designed to help emergency department physicians decide when to order x-rays for ankle injuries.14 The rule, which emphasizes pain near the malleoli, the inability to bear weight, and patient age of 55 years or older (for malleolus fractures), as well as midfoot pain and several points of bone tenderness (for midfoot fractures), was reported to be 100% sensitive and 40% specific, and theoretically reduced the need for ankle x-rays by about 36%. The OARs were most recently validated in 2016, when a paper pronounced them “the best filter” for determining the need for radiographs, despite a tendency to overestimate that need if applied within an hour after injury.15
Clinicians have recently been working on CPRs related to treatments for ankle sprain injuries, as well, because the associated pain and disability can continue for six months or longer.16 In 2009, investigators developed a CPR comparing manual therapy (thrust and nonthrust manipulation) with general mobility exercises in patients after inversion ankle sprains. They reported that 75% of patients exhibited excellent short-term success with the manual approach—a number that rose to 95% if they met three of four criteria: symptoms worse when standing, symptoms worse in the evening, navicular drop of 5 mm or greater, and distal tibiofibular joint hypomobility.17
Paul Mintken, DPT, one of the study’s authors and an associate professor of physical therapy at the University of Colorado School of Medicine in Aurora, told LER that in the early 2000s, the study’s primary author, Julie Whitman, PT, had started treating ankle sprains with relatively aggressive “thrust” manual therapy—similar to the adjustments used by chiropractors—which was controversial for an acutely injured joint.
“We looked back and tried to determine statistically what the people with the best response had in common,” Mintken said. “We ran logistic progressions to distill it down to four or five variables that when combined give an increased likelihood that the individual will benefit.”
Mintken acknowledged the study’s primary weakness was the lack of a control group, which made it hard to say if patients treated differently might have done just as well. The researchers subsequently conducted a randomized controlled trial comparing manual therapy and exercise after ankle sprain with a supervised home exercise program, however, and reported that the former was the superior treatment.18
“We found statistically significant improvements in pain and disability at all the time points out to six months,” he said, adding that the rules have been included in some of his clinic’s practice guidelines. “Now we know that if a patient has an ankle sprain and there’s no reason not to do the hands-on work—such as a fracture, or a grade three sprain where they’ve completely torn the ligaments—then we’re probably going to be using these techniques.”
A number of CPRs have been proposed regarding knee osteoarthritis (OA), partly because it’s the primary cause of chronic disability in older people.19 In a 2007 study, for example, researchers developed a CPR to identify patients with clinical evidence of knee OA who had a favorable short-term response to hip mobilizations.20 In 2012, researchers at Brooke Army Medical Center in San Antonio, TX, studied predictive variables for knee OA patients to determine who was less likely to respond to the usual treatments of manual PT and exercise.21 In that prognostic study, those who had two of three characteristics—patellofemoral pain, anterior cruciate ligament laxity, and height taller than 5′ 7″—had an 88% probability of nonsuccess with those treatments.
In 2016, investigators at Virginia Commonwealth University developed and validated another prognostic CPR for estimating the likelihood of developing incident tibiofemoral OA with rapid progression in individuals with a baseline Kellgren-Lawrence (KL) grade of 0 or 1.22 Although frequent knee symptoms at baseline were not found to be predictive, other factors—contralateral knee OA, higher body mass index (BMI), baseline index KL grade of 1, and higher WOMAC (Western Ontario and McMasters Universities Osteoarthritis Index) scores—predicted those more likely to develop a KL grade of 3 or 4 within five years.
Daniel Riddle, who was the paper’s lead author, explained the importance of the investigation.
“The standard rate of progression in knee osteoarthritis is slow, but there’s a subset of patients in whom it’s very rapid [see “The clinical implications of accelerated knee OA,” April 2016, page 43], and we don’t understand why,” he said. “We’re talking about millions of people, and we wanted to know whether we might be able to predict ahead of time who is at risk for rapid progression. There have been studies that have looked at these variables in isolation, but none that attempted to combine predictors into a single rule.”
The goal, of course, is not just to identify the patients most at risk but ultimately to devise effective interventions. Success will depend partly on determining which risk factors are modifiable and which are not.
“The KL grade in the contralateral knee is predictive, but there’s nothing we can do to intervene,” Riddle said. “But BMI is modifiable, so we can intervene if we identify a person with that issue who is at high risk of developing end-stage knee OA in the subsequent five years. The same thing is true with WOMAC scores; if we reduce pain and improve function, we now know that we can reduce the risk of rapid progression. So of the four variables we’ve identified, two are modifiable.”
In a related paper also published in 2016, Riddle and his colleagues endeavored to validate a CPR for predicting poor response to total knee replacement, which occurred in about 19% of patients. They reported a weak correlation between predicted and actual factors affecting nonresponse.23
“We found that the patient’s pain and function scores, and their KL grade, prior to surgery, were the most powerful predictors of who didn’t do well afterward,” he said. “Patients who had minimal pain and loss of function—and those who had mild knee osteoarthritis—didn’t do as well after surgery compared to the rest of the sample. So in some ways that externally validated the prediction rule we were testing, but other scores—the mental health score and obesity measure—were not important in the prediction.”
Riddle generalized about the potential uses of CPRs based on his own experience.
“A patient considering knee replacement surgery and his surgeon will both want to consider the likely outcome,” he said. “The most powerful way to identify either a diagnosis or a prognosis is to use all available information and combine it into a single score. That’s what a CPR is designed to do, so it’s more evidence-based and powerful than more traditional approaches. The trick is to be a critical consumer of evidence.”
Cary Groner is a freelance writer in the San Francisco Bay Area.
- Cook C. The Oxford debate 2011: Clinical prediction rules dead or alive? Evidence in Motion website. http://www.evidenceinmotion.com/blog/2011/06/15/the-oxford-debate-2011-clinical-prediction-rules-dead-or-alive/. Published June 15, 2011. Accessed August 9, 2017.
- Cook CE. Potential pitfalls of clinical prediction rules. J Man Manip Ther 2008;16(2):69-71.
- Haskins R, Cook C. Enthusiasm for prescriptive clinical prediction rules (eg, back pain and more): a quick word of caution. Br J Sports Med 2016;50(16):960-961.
- Adams S, Leveson S. Clinical prediction rules. BMJ 2012;344:d8312
- Toll DB, Janssen KJ, Vergouwe Y, Moons KG. Validation, updating, and impact of clinical prediction rules: a review. J Clin Epidemiol 2008;61(11):1085-1094.
- McGinn TG, McCullagh L, Kannry J, et al. Efficacy of an evidence-based clinical decision support in primary care practices: a randomized clinical trial. JAMA Intern Med 2013;173(17):1584-1591.
- McCullagh LJ, Sofianou A, Kannry J, et al. User centered clinical decision support tools: adoption across clinical training level. Appl Clin Inform 2014;5(4):1015-1025.
- Richardson S, Khan S, McCullagh L, et al. Healthcare provider perceptions of clinical prediction rules. BMJ Open 2015;5(9):e008461.
- Editorial (unattributed), J Manual Manip Ther 2008;16(2):69-71.
- Fanning EL, Selwyn BJ, Larme AC, DeFronzo RA. Improving efficacy of diabetes management using treatment algorithms in a mainly Hispanic population. Diabetes Care 2004;27(7):1638-1646.
- Crawford F, Cezard G, Chappell FM, et al. A systematic review and individual patient data meta-analysis of prognostic factors for food ulceration in people with diabetes: the international research collaboration for the prediction of diabetic foot ulceration (PODUS). Health Technol Assess 2015;19(57):1-210.
- Vicenzino B, Collins N, Cleland J, McPoil T. A clinical prediction rule for identifying patients with patellofemoral pain who are likely to benefit from foot orthoses: a preliminary determination. Br J Sports Med 2010; 44(12):862-866.
- Barton CJ, Menz HB, Crossley KM. Clinical predictors of foot orthoses efficacy in individuals with patellofemoral pain. Med Sci Sports Exerc 2011;43(9):1603-1610.
- Stiell IG, Greenberg GH, McKnight RD, et al. A study to develop clinical decision rules for the use of radiography in acute ankle injuries. Ann Emerg Med 1992;21(4):384-390.
- David S., Gray K, Russell JA, Starkey C. Validation of the Ottawa ankle rules for acute foot and ankle injuries. J Sport Rehabil 2016;25(1):48-51.
- Braun BL. Effects of ankle sprain in a general clinic population 6 to 18 months after medical evaluation. Arch Fam Med 1999;8(2):143-148.
- Whitman JM, Cleland JA, Mintken PE, et al. Predicting short-term response to thrust and non-thrust manipulation and exercise in patients post-inversion ankle sprain. J Orthop Sports Phys Ther 2009;39(3):188-200.
- Cleland JA, Mintken PE, McDevitt A, et al. Manual physical therapy and exercise versus supervise home exercise in the management of patients with inversion ankle sprain: a multi-center randomized clinical trial. J Orthop Sports Phys Ther 2013;43(7):443-455.
- Guccione AA, Felson DT, Anderson JJ, et al. The effects of specific medical conditions on the functional limitations of elders in the Framingham Study. Am J Public Health 1994;84(3):351-358.
- Currier LL, Froehlich PJ, Carow SD, et al. Development of a clinical prediction rule to identify patients with knee pain and clinical evidence of knee osteoarthritis who demonstrate a favorable short-term response to hip mobilization. Phys Ther 2007;87(9):1106-1119.
- Deyle GD, Gill NW, Allison SC, et al. Knee OA: which patients are unlikely to benefit from manual PT and exercise? J Fam Pract 2012;61(1):E1-E8.
- Riddle DL, Stratford PW, Perera RA. The incident tibiofemoral osteoarthritis with rapid progression phenotype: development and validation of a prognostic prediction rule. Osteoarthritis Cartilage 2016;24(12):2100-2107.
- Riddle DL, Golladay GJ, Jiranek WA, Perera RA. External validation of a prognostic model for predicting nonresponse following knee arthroplasty. J Arthroplasty 2017;32(4):1153-1158.