Repairer Driven News
« Back « PREV Article  |  NEXT Article »

Compare Tractable photo estimating AI vs. shops, insurers with new demo results

By on
Education | Insurance | Repair Operations | Technology
Share This:

In April, Repairer Driven News compared the artificial intelligence photo estimating system Tractable’s performance on a damaged Chevrolet Cruze to the insurer’s initial estimate and the shop’s final, paid estimate.

Following that demo, the tech company then offered us a chance to test the remaining packages of photos we’d prepared, and we took them up on it last month. Tractable said both demos were live.

“If it ain’t live tested, it ain’t real,” Tractable CEO Alex Dalyac wrote of photo AI in an email after the demo May 1.

Repairers provided us with images and both the insurer and shop estimates for two luxury vehicles and two more mainstream ones: an Audi Q7, Ford C-Max, Porsche Panamera and Subaru Crosstrek. As with the Cruze, we gave Tractable the images, but not the estimate results.

What follows are some of the results captured in the May 1 demo and the real-world estimates. As we noted in April, a few data points aren’t proof of anything, but they do provide a glimpse at the technology poised to disrupt auto claims handling, and we though readers would be interested.

Some of the differences between the AI and human estimates presented below might involve items the AI isn’t taught to or meant to consider, something to keep in mind when checking out the results. Whether such software should be expected to do so automatically or instead have these items added manually by a human estimator partner is something the industry can debate. Before passing judgment, note that the human estimators here missed some of the same items on their initial estimate too.

2014 Ford C-Max SEL hybrid

The Tractable AI calculated this 2014 Ford C-Max SEL hybrid’s rear bumper fascia as a replace, with 93 percent confidence from a side view and as a replace with 82 percent accuracy from a three-quarters rear view. Both angles saw it considering the trunk lid either intact or repairable, concluding the same for the quarter panel.

The AI also accurately concluded it was looking at a rear bumper from two extremely close-up photos, which we found interesting. One photo saw the software determine with 58 percent confidence it was looking at a rear bumper fascia, based merely upon a tight shot of the area around the bottom driver’s side of the tailgate containing the C-Max emblem. Tractable’s software declared the part a replace with 72 percent confidence.

 

Another shot focused tight on the first quarter or so of the bumper, including a parking sensor, and the AI expressed 66 percent confidence it was indeed looking at a rear bumper fascia. It called the part a replace with 92 percent confidence.

The system gave no indication it recognized the emblem or parking sensor on the C-Max, nor did it seem to notice the mirrors prominent in the Crosstrek photos.

“We do cover mirrors but that was not part of this demo,” spokesman Ahmed Zifzaf wrote in an email Thursday. “Sensors we leave for diagnostics and emblems we found usually a negligible part of total costs.”

The system also didn’t seem confused by the dirty C-Max liftgate, deeming the part intact/repairable with high confidence rather than misreading the grime as damage.

The human insurer’s $1,046.88 estimate called for replacing parts like the bumper assembly, valence and right rear bracket outer. The carrier wrote for an R&I of the parking sensor and six minutes of “Clean/polish scuff” on the quarter panel and liftgate.

The repairer’s final, paid estimate was for $3,761.44. They estimated 3.5 hours to repair the liftgate, including half an hour to fix the inner shell, and called for new “SEL,” “C-MAX” and “HYBRID” emblems for the part, which was to be painted. The shop instead ordered new or CAPA versions of the individual bumper parts — a fascia, energy absorber, parking sensor and bumper beam — instead of writing for a used assembly. A replacement roof molding, valence and left rear bracket outer (not a right one) were on the estimate as well.

The shop applied 0.5 hours to repair the quarter panel.

2017 Audi Q5 Premium Plus

A 2017 Audi Q5 Premium Plus’ battered passenger-side rear door was deemed a replace with 98 percent confidence from a three-quarters shot and with 100 percent confidence with a photo looking at the door straight on. The system didn’t mention the sill.

The initial $8,350.55 insurer estimate called for replacing parts like the outer rocker panel, the door shell, the lower door molding and door weatherstripping, as well as repairing the lower rocker panel.

The shop’s final paid estimate was $10,367.99 and also included replacing all of these parts but also components like labels, glass rivets, molding grommets, clips, the fuel filler housing, and quarter wheel molding. The shop also wrote to repair the lower rocker.

2017 Subaru Crosstrek (manual)

The software concluded the grille on a damaged 2017 Subaru Crosstrek with 95 confidence needed to be replaced, but deemed the bumper fascia merely intact/repairable — but with less confidence. On the other hand, the software didn’t appear to be thrown off by the large L-shaped shadow on the hood, deeming it intact/repairable with 84 percent confidence.

By switching to a three-quarters view of the front, the AI noticed the bumper fascia also needed to be replaced, calling itself 86 percent confident in that assessment.

The insurer wrote for a $1,309 repair, also calling for a bumper, grille, grille surround and molding. The shop wrote for a $2,485.91 repair in its final paid estimate, adding a clip kit and a lower bracket to the list of items needing replacement.

It’s also worth noting that the Tractable AI didn’t seem fooled about the white truck seen seen next to the Crosstrek in the photos. For example, two of the other vehicle’s doors are clearly visible and prominent in the head-on image of the Crosstrek, yet the AI didn’t start making decisions about the closures.

2014 Porsche Panamera

Another series of photos were lower-resolution shots of a 2014 Porsche Panamera. This didn’t seem to bother the AI. It concluded all the parts it saw were either fine or repairable.

The insurer, however wrote a $1,386.92 estimate calling for the repair of the bumper cover and replacing the lower cover. The shop wrote a $3,058.18 estimate including not just replacing the lower cover but also the bumper cover.

More information:

“Tractable: How to Evaluate Photo AI in 5 minutes”

Tractable, May 1, 2019

Images:

Tractable’s AI evaluates a 2017 Audi Q5 Premium Plus with damage. (Screenshot from Tractable demo; redaction by Repairer Driven News)

Tractable’s AI evaluates a 2014 Ford C-Max SEL hybrid with damage. (Screenshot from Tractable demo; redaction by Repairer Driven News)

Tractable’s AI evaluates a 2017 Audi Q5 Premium Plus with damage. (Screenshot from Tractable demo; redaction by Repairer Driven News)

Tractable’s AI evaluates a 2017 Subaru Crosstrek with damage. (Screenshot from Tractable demo; redaction by Repairer Driven News)

Tractable’s AI evaluates a 2014 Porsche Panamera with damage. (Screenshot from Tractable demo; redaction by Repairer Driven News)

Share This: