The Federal Motor Carrier Safety Administration’s plan to rework its Compliance, Safety, Accountability safety scoring program overall is sound, but potential major challenges could impede the agency’s ability to implement its plan to shore up CSA’s ability to accurately assess carriers’ safety performance.
Those are the conclusions of a report issued by the U.S. DOT’s Office of Inspector General (OIG), which on Friday announced it had completed its Congressionally required review of FMCSA’s corrective action plan for the CSA program. DOT’s OIG began its review in October 2018, following FMCSA’s unveiling of its reform plan in June 2018.
While the agency’s approach could indeed address industry-wide concerns about CSA’s ability to accurately gauge carriers’ safety risk, OIG said, the plan lacks important details in how the agency will implement the reforms and how those reforms will improve key areas of concern, such as transparency in how it scores motor carriers and the data that underpins those scores.
As part of its planned overhaul, FMCSA seeks to replace the existing CSA Safety Measurement System (the heart of the CSA program) with a new scoring system. It would be based on a so-called “item response theory” (IRT) method, intended to account for variables better than the previous scoring system. FMCSA also said it hopes to improve the quality of data used to score motor carriers and make it easier for carriers to understand their CSA scores. The agency also is considering adding an absolute scoring system to CSA, instead of relying solely on relative scores that compare motor carriers to their peers.
Congress, via the December 2015-passed FAST Act highway bill, required FMCSA to pull CSA Safety Measurement System (SMS) rankings from public view. It also called for the National Academies of Science to study the program and offer recommendations to FMCSA about how it could resolve concerns about the accuracy of carriers’ safety ratings. FMCSA’s corrective action plan was based on recommendations made by NAS in 2017.
In its report issued Friday, Sept. 27, DOT’s OIG agreed that FMCSA’s plan “mostly addresses” CSA’s deficiencies. But OIG admits the reforms “lack implementation details for improving SMS transparency and its assessment of carrier safety rankings,” such as “costs, benchmarks and other implementation details.” OIG also noted that FMCSA doesn’t plan to collect new data to feed CSA, which could limit the agency’s ability to truly reform the program — and to effectively implement the IRT methodology.
FMCSA began small-scale IRT testing last fall. That test will determine whether it will proceed with full-scale implementation of an IRT system. Yet, OIG’s report notes, “there are a limited number of individuals in the United States with experience in developing large-scale IRT models,” potentially a major hindrance in scaling the agency’s IRT system.
What’s more, FMCSA told OIG that it’s limited in its ability to feed new data into the scoring system, particularly in its small-scale IRT pilot. Thus, the IRT system could be plagued by the same questions of accuracy in rating motor carriers’ safety, OIG says.
OIG recommends that FMCSA estimate costs for implementing the reforms, including staffing, enforcement and data collection, as well as benchmarks for gauging success.
OIG notes that FMCSA plans to finish its small-scale IRT testing by Sept. 30. It will determine whether it can proceed with a full-scale IRT model for CSA by Sept. 30, 2020.