Steven Bryan of Vigillo has long been engaged with efforts around improvement of the CSA Safety Measurement System to be a better tool for all involved. His data-mining company made its name early in the CSA era providing carriers with a window into their SMS percentile rankings/scores prior to the CSA SMS‘ official release. He was also one of the many parties on whom the National Academies of Science called to present on various aspects of the SMS during its evaluation of it.
Steve BryanI talked to Bryan a bit recently about just what he sees as likely to happen in the wake of the National Academies’ report on the SMS, which he in part summarized with his own analysis in this blog post on the Vigillo website last week. The NAS recommendations were to revamp the statistical model underpinning SMS with an “item response theory” (IRT) approach, which Bryan describes as one that would make the relevant adjustments in carrier scores to account for confounding factors like state enforcement departments’ known violation biases/predilections and the like. As Bryan lays it out in his blog, IRT applied to the CSA system could:
- Account for the probability of being selected for inspection
- Provide a basis with which to evaluate how data insufficiency could impact safety ratings
- Provide a basis to more rigorously and empirically evaluate the utility of individual violations
- Allow severity weights to change over time (e.g., as violations become more or less prevalent);
- Determine empirically whether severity weights should be different for trucks versus passenger carriers;
- Enable adjustment for factors that may be outside a carrier’s direct control
- Accommodate new violations over time
Bryan “gives credit to the National Academies” for the idea, he says, as he believes it could be a way to build a better tool out of the SMS. But “the question is, will it work when we actually start applying numbers to it? None of us know that yet.”
Another question, of course, is whether the “FMCSA will even accept this,” Bryan says. “We know their history. Everybody who’s been critical of CSA – all we’ve ever gotten back is ‘yeah, we disagree.’ Hopefully, that’s not where we are.”
Bryan says the response could hinge on what the next administrator thinks about the issue overall. The current FMCSA head is Deputy Administrator Daphne Jefferson, until Trump gets around to making a nomination, assuming he will. “Leadership that buys into this would be huge,” Bryan says. “If we don’t get that, it’s just an academic exercise.”
Whatever the case, if scores are to return to the public, the FAST Act mandates that FMCSA issue a corrective action plan relative to the SMS within 120 days, with which the agency has said it will comply. After that, the DOT OIG then has its own 120 days to report back to the same Congressional committee with its analysis of the plan and whether it addresses all that the FAST Act required.
Bryan sees the agency either doing the above or, on the flipside, letting the current state of the SMS, with scores hidden from public view, remain in perpetuity. “The FAST Act only requires that process [outlined in previous paragraph] if the agency insists on bringing the scores public again,” he notes. “If they throw up their hands and say we’re fine with them being them private, they don’t have to do anything.” In that case, Bryan believes, the CSA SMS is little more than a “half-baked solution sitting in the dark.”
The third way
But perhaps there’s a third way forward. I also spoke with Tom Sanderson of Transplace, one of the principals behind the Alliance for Safe, Efficident and Competitive Truck Transportation (ASECTT). His organization sees the NAS report as further evidence that the data-sufficiency arguments against the SMS approach are valid and insurmountable under the current roadside-inspection regime. While the NAS report seems to laud the intention behind the SMS, ASECTT disagrees, particularly when it comes to the data-sufficiency piece of the puzzle.
“Our thought is that given all of the problems with SMS methodology,” he says, including the fact that “it’s been tweaked and worked on for years without every really addressing the fundamental problem that such a small percentage of carriers are ever going to get that 20 or more inspections [every two years, recommended by GAO’s 2014 critical report], the group thought it might make more sense to build off of the methodology that the new-carrier certification is based on.”
ASECTT’s third way would have FMCSA abandon SMS but not carrier evaluation. Rather than have government try to ramp up roadside inspections, pursue quicker e-inspections or otherwise produce more data — Vigillo’s Steven Bryan notes that data sources pondered by the NAS report (including carriers’ ELD data, turnover rates, driver-pay information, etc.) are never going to materialize with industry cooperation — Sanderson and ASECTT propose an audit conducted every two years modeled on New Entrant remote, off-site “desktop” audits. In comments made to the June meeting of the Motor Carrier Safety Advisory Committee (MCSAC), ASECTT argued a biennial (every two years) audit could serve to rate more carriers and fully underpin a new safety-rating system that would provide an equitable look into the safety processes/records of all motor carriers:
The Agency’s current use of offsite audits for many new entrants (despite the reference to on-site audits in 49 C.F.R. 385.315) has affirmed that a remote or desktop audit is an accurate, cost-effective tool for determining carrier compliance. Petitioners propose that the similar desktop audit be conducted of every registered motor carrier, on at least a biennial schedule corresponding to its census update deadline. This audit would be conducted by federal and state officials or outside contractors at the cost of approximately $300 per audit, the expense to be borne by the registrant. Desktop audits would result in a finding that a carrier is “fit to operate and fit to use” – in accordance with the Agency’s certification responsibilities under 49 U.S.C. 31144 and with the directive from Congress in SAFETEA-LU that the Agency develop a method for reviewing and rating all registrants. If no such finding appears appropriate, the Part 385 safety rating process or existing “imminent hazard” procedures would be available to the Agency.
Sanderson envisions a cohort of “independent certified examiners looking at what the carriers are doing,” he says, perhaps aggregated in a national registry of certified auditors akin to what happens for certified DOT medical examiners today. “Having the carriers pay for that certification we think makes a lot of sense. And then we have the chance of evaluating far more carriers than will be captured by the roadside data.”
As it is, and as I’ve reported several times before, very few carriers, relative to years past, today are getting the benefit of a Satisfactory safety rating, the de facto safe-to-use certification. Changing the output of the audit system to fit/unfit and using a full compliance review to issue any adverse rating as exists in regulation today could make for a “fresh approach,” Sanderson believes. “The agency’s been messing around with [the SMS] for 14 years. And they’re really no closer to having a valid certification, having the FMCSA determine a thumbs-up or thumbs-down on whether the carrier’s safe. … This is something that could be implemented for the broadest set of carriers rather than trying to once again rebuild the SMS.”
