What happens when an algorithm cuts your health care

In home care, the problem of allocating help is particularly acute. The United States is inadequately prepared to care for a population that’s living longer, and the situation has caused problems for both the people who need care and the aides themselves, some of whom say they’re led into working unpaid hours. As needs increase, states have been prompted to look for new ways to contain costs and distribute what resources they have.

States have taken diverging routes to solve the problem, according to Vincent Mor, a Brown professor who studies health policy and is an InterRAI member. California, he says, has a sprawling, multilayered home care system, while some smaller states rely on personal assessments alone. Before using the algorithmic system, assessors in Arkansas had wide leeway to assign whatever hours they thought were necessary. In many states, “you meet eligibility requirements, a case manager or nurse or social worker will make an individualized plan for you,” Mor says.

Arkansas has said the previous, human-based system was ripe for favoritism and arbitrary decisions. “We knew there would be changes for some individuals because, again, this assessment is much more objective,” a spokesperson told the Arkansas Times after the system was implemented. Aid recipients have pointed to a lack of evidence showing such bias in the state. Arkansas officials also say a substantial percentage of people had their hours raised, while recipients argue the state has also been unable to produce data on the scope of the changes in either direction. The Arkansas Department of Human Services, which administers the program, declined to answer any questions for this story, citing a lawsuit unfolding in state court.

When similar health care systems have been automated, they have not always performed flawlessly, and their errors can be difficult to correct. The scholar Danielle Keats Citron cites the example of Colorado, where coders placed more than 900 incorrect rules into its public benefits system in the mid-2000s, resulting in problems like pregnant women being denied Medicaid. Similar issues in California, Citron writes in a paper, led to “overpayments, underpayments, and improper terminations of public benefits,” as foster children were incorrectly denied Medicaid. Citron writes about the need for “technological due process” — the importance of both understanding what’s happening in automated systems and being given meaningful ways to challenge them.

Critics point out that, when designing these programs, incentives are not always aligned with easy interfaces and intelligible processes. Virginia Eubanks, the author of Automating Inequality, says many programs in the United States are “premised on the idea that their first job is diversion,” increasing barriers to services and at times making the process so difficult to navigate “that it just means that people who really need these services aren’t able to get them.”

One of the most bizarre cases happened in Idaho, where the state made an attempt, like Arkansas, to institute an algorithm for allocating home care and community integration funds, but built it in-house. The state’s home care program calculated what it would cost to care for severely disabled people, then allotted funds to pay for help. But around 2011, when a new formula was instituted, those funds suddenly dropped precipitously for many people, by as much as 42 percent. When the people whose benefits were cut tried to determine how their benefits were determined, the state declined to disclose the formula it was using, saying that its math qualified as a trade secret.

In 2012, the local ACLU branch brought suit on behalf of the program’s beneficiaries, arguing that Idaho’s actions had deprived them of their rights to due process. In court, it was revealed that, when the state was building its tool, it relied on deeply flawed data, and threw away most of it immediately. Still, the state went ahead with the data that was left over. “It really, truly went wrong at every step of the process of developing this kind of formula,” ACLU of Idaho legal director Richard Eppink says.

Most importantly, when Idaho’s system went haywire, it was impossible for the average person to understand or challenge. A court wrote that “the participants receive no explanation for the denial, have no written standards to refer to for guidance, and often have no family member, guardian, or paid assistance to help them.” The appeals process was difficult to navigate, and Eppink says it was “really meaningless” anyway, as the people who received appeals couldn’t understand the formula, either. They would look at the system and say, “It’s beyond my authority and my expertise to question the quality of this result.”

Idaho has since agreed to improve the tool and create a system that Eppink says will be more “transparent, understandable, and fair.” He says there might be an ideal formula out there that, when the right variables are entered, has gears that turn without friction, allocating assistance in the perfect way. But if the system is so complex that it’s impossible to make intelligible for the people it’s affecting, it’s not doing its job, Eppink argues. “You have to be able to understand what a machine did.”

“That’s an argument,” says IHPI member, Brant Fries, Ph.D., M.S., Professor of Public Health at the University of Michigan. “I find that to be really strange.” He’s sympathetic to the people who had their hours cut in Arkansas. Whenever one of his systems is implemented, he says, he recommends that people under old programs be grandfathered in, or at least have their care adjusted gradually; the people in these programs are “not going to live that long, probably,” he says. He also suggests giving humans some room to adjust the results, and he acknowledges that moving rapidly from an “irrational” to a “rational” system, without properly explaining why, is painful. Arkansas officials, he says, didn’t listen to his advice. “What they did was, in my mind, really stupid,” he says. People who were used to a certain level of care were thrust into a new system, “and they screamed.”

Fries says he knows the assessment process — having a person come in, give an interview, feed numbers into a machine, and having it spit out a determination — is not necessarily comfortable. But, he says, the system provides a way to allocate care that’s backed by studies. “You could argue everybody ought to get a lot more care out there,” he says, but an algorithm allows state officials to do what they can with the resources they have.

As for the transparency of the system, he agrees that the algorithm is impossible for most to easily understand, but says that it’s not a problem. “It’s not simple,” he says. “My washing machine isn’t simple.” But if you can capture complexity in more detail, Fries argues, this will ultimately serve the public better, and at some point, “you’re going to have to trust me that a bunch of smart people determined this is the smart way to do it.”