You know the story: In 2012, the US Department of Justice swooped into town and ordered the City of Portland do something about the city’s disproportionately high number of police officers shooting and killing people with a mental illness.

In a settlement agreement with the feds, Portland vowed to make substantial changes to police policy, training, and oversight to eliminate those violent interactions.

Seven years later, criminal justice expert Dennis Rosenbaum, an outside contractor tasked with evaluating the city’s compliance with the settlement, says Portland has finally met all the agreed-on improvements. In February, Rosenbaum will present his findings to a federal judge, who will have the final say on Portland’s compliance with the DOJ agreement.

But, as Portland police officers continue to disproportionately shoot and kill those experiencing mental health crises, the public’s not ready to call it a victory.

“Forty percent of people killed by police in the last year were having a mental health crisis,” said Elliott Young, a member of the public committee that oversees the city’s settlement process, at an October meeting. “How is that compliance?”

Members of that committee—Portland Committee on Community-Engaged Policing (PCCEP)—had just sat through a presentation by Rosenbaum in which he declared the city had complied with the terms of the DOJ settlement. Rosenbaum highlighted new mental health training requirements, more efficient police misconduct investigations, and a new “community engagement plan.”

“I want to point out the difference between outcomes and outputs,” said Lakayana Drury, co-chair of the PCCEP, in response. “The settlement agreement addresses outputs... but until we know what their outcomes are, how are we even able to assess what’s being done? Yeah, there’s new police training, but is it leading to officers using less force?”

Rosenbaum didn’t have an answer.

In an interview with the Mercury, Rosenbaum said the settlement agreement called for structural changes in order to spur reform but didn’t mandate improved outcomes.

“You can debate what should be included in the settlement agreement, but that’s how it was written,” said Rosenbaum. “Our hands are tied.”

Rosenbaum suggests Portlanders develop their own outcome-based goals for its police force that will outlive the federal agreement—and won’t require a federal judge or a pricey contractor to evaluate.

But can—and will—Portland take on that responsibility?

For nearly a decade, Portland has been forced to follow mandatory police reforms that are so significant and fundamental they had to be instituted and overseen by the federal government. As it checks the final boxes off its DOJ-mandated list, is the city prepared to continue holding itself accountable?