Devil’s in the data: Regulators seek sources, improvements for CSA revamp at public meeting

user-gravatar Headshot
Updated Sep 5, 2018

“The data is the data,” said David Yessen of the Federal Motor Carrier Safety Administration’s compliance office. Yessen, speaking about halfway through a hearing this morning in Washington, D.C., was reiterating the agency’s strategy toward reforming the Compliance, Safety, Accountability system. That strategy follows the National Academies of Science’s June 2017 recommendations to FMCSA, in which it urged the agency to build a new statistical model to underly the SMS, one based on so-called “item response theory,” to better prioritize at-risk carriers for investigation or other intervention. FMCSA has begun that process, and Wednesday’s meeting was part and parcel of its engagement with stakeholders to address concerns and solicit ideas.

Yessen outlined steps the agency has taken in recent times during the hearing, including building small-scale versions of a new statistical model to test and considering new ways to utilize the data it has — its MCMIS database of inspections and violations, crash data, etc. — as well as new data inputs.

As it identifies a potential new source of data to use over the coming year as it builds the system, the agency will be dropping the data into the IRT model to see if it has the appropriate effects. Yessen likened the model to “a funnel, and you can keep dropping information into that funnel, and we’re looking for opportunities to do that.” At today’s hearing, representatives from the regulated industry communities were asked to suggest improvements related specifically to Recommendations 2, 3 and 4 from the NAS’ report on the CSA SMS.

Having begun addressing No. 1 in these recommendations from the NAS report, Nos. 2, 3 and 4 were the subject of public comment at the hearing.Having begun addressing No. 1 in these recommendations from the NAS report, Nos. 2, 3 and 4 were the subject of public comment at the hearing.

New data under consideration include driver pay method, carriers’ driver turnover rates
Improving the quality of MCMIS data, part of recommendation No. 2, was the context for Yessen’s “data is the data” notion, a response to a commenter’s recommendation that the agency take into account geographical variations of carrier operations — and the attendant well-documented and widely varying violation priorities in different states — when normalizing the data to provide more accurate safety assessments.

Irwin Shires of Panther Expedited was quick to point out an example where the data might not be the data, so to speak, when the wrong assumptions are made. Shires is a member of the Motor Carriers for Regulatory Reform coalition, a group with CSA and other reforms at the top of their agenda. The MCRR, as member attorney Henry Seaton noted in conversation with Overdrive last week, continues to view an at least biennial “desktop audit” conducted offsite (akin to an offsite version of FMCSA’s new entrant audits in some ways) of every carrier in the nation as perhaps the best solution to safety assessment in light of the difficulty of accurately assessing small carriers with a data-based approach.

As the agency nonetheless examines bolstering data in the new system with different information, Shires cautioned that the agency to “be careful about assuming” a simple relationship to safety performance. Among new items the agency urged commentary on — driver method and level of compensation, a carrier’s driver-turnover rate, driver age and other factors — Shires noted turnover could easily suggest more than a single interpretation. For instance, “if a carrier has adopted a proactive culture of safety, which I’ve been told that’s what they want us to do, if I have methods in place to identify bad eggs … and I get them out of my fleet before they have an accident, I still have to replace the driver. That’s a kind of positive turnover.”

While Shires recognized “there are a number of carriers out there who churn through drivers” as a matter of course, he said, “take into account the exceptions to the rule before you implement them into an IRT model.”

“We’re going into this with an open mind,” Yessen said.

Though there were some commendations for the agency taking into account method of compensation and turnover as potential data points for safety assessment, getting carriers to disclose such information to the system would require rulemaking for the FMCSA to actually require that disclosure, Director of Enforcement Joe DeLorenzo acknowledged.

Dan Horvath, safety policy VP for the American Trucking Associations, provided a counterpoint when he noted that, in his association’s view, “from a compensation perspective, we don’t think it’s reliable to judge a carrier on that.”

Former ATA rep Dave Osiecki, now with the Scopelitus firm, underscored the variation that is fairly common today from a per-mile, hourly and/or percentage or per-trip basis all within the same company, to say nothing of the reality of an independent owner-operator’s varying contracts with brokers, shippers and other freight parties.

In addition, on turnover rates and other carrier-specific data, Horvath said, “there’s definite hesitation from carriers to provide those perspectives.”

FMCSA’s Barbara Baker, also of the compliance office, acknowledged as much: “We get into a catch 22,” she said. “If we have an IRT model based on new sources liked driver turnover, age,” and etc., “that ends up being company confidential and/or personal information” that needs to be protected from dissemination to the public. The raw information in the MCMIS database and its use by public and private research and other entities was another topic of discussion that received less attention at this morning’s session.

An alternate way to employ company compensation or other proprietary data might be in a kind of voluntary-share scenario, DeLorenzo suggested. “How might it be used on a voluntary basis to help the carriers?” he asked, adding that information on types of cargo, accurate and verifiable miles traveled and more could also fall under a similar framework in something akin to the past-proposed “beyond compliance” initiative to help carriers with issues potentially improve scores. “Those are the kinds of things that we want to get at.”

Catch in-depth state-by-state data and analysis via Overdrive‘s long-running CSA’s Data Trail series and its main page, recently updated to reflect year 2017 data for numerous metrics.Catch in-depth state-by-state data and analysis via Overdrive‘s long-running CSA’s Data Trail series and its main page, recently updated to reflect year 2017 data for numerous metrics.

Geographic disparity in enforcement and exposure
Well-documented issues of disparity in state-by-state enforcement and carrier risk exposures are problems Shires believed needed to be addressed before the agency went too far down the road developing its new statistical model. “State-by-state discrepancies have to be addressed,” he said, “and have to be normalized, before any system involving data is created. However you deal with it with state partners, it needs to be dealt with.”

He gave the example of the state of Indiana, the perennial leader in Overdrive‘s violation-priorities state rankings in the moving violations category, by a significantly large margin: “If my trucks did not have to drive through Indiana, my Unsafe Driving [SMS BASIC] score would be about a third of what it is now.”

Better normalizing crash rates for exposure, too, whether to account for the issue of team haulers having twice the per-power-unit exposure of solo drivers or of urban-delivery drivers’ much higher per-mile exposure to traffic over rural-OTR operations, “needs to be dealt with before anything else,” Shires added.

“When we start talking about what violations will trigger our prioritization system, [geographic variations are] something we’ll consider,” Yessen said. “But we need it to tell us what carriers to look at – it’s a prioritization tool to assign our investigators to cases.”

As in past arguments around how to make the CSA program overall a better mousetrap, the agency remained vocally focused in this meeting on use for internal prioritization for carrier interventions. FMCSA punted on issues of any future public display of CSA scores output from the new model. That public display is where the proverbial scoring system rubber hit the road in the past in terms of its problematic nature for companies of all sizes around the industry. FMCSA did note it aimed to have an IRT statistical model close to fully developed to share for industry and other stakeholder review by September of 2019.

Those interested in engaging on these issues with the agency can comment on the docket for this request for input at this link. The comment period will be open through most of the month of October, and the agency hoped to open lines of communication further with another public hearing early in the new year.