The Think Family Database holds records on close to half a million people who live in the city of Bristol, England. For many years, few of them knew anything about it.
Launched in 2016 by the Bristol City Council and the regional Avon and Somerset Police, the database has stored all manner of sensitive information—police intelligence reports, housing status, mental health records, teenage pregnancies, enrollment in parenting courses, free school meals. On top of this sensitive data, officials built machine-learning models to assign scores to thousands of adults and children. They hoped to build what they called a “picture of threat, harm, and risk” in the region. At an event in early 2022 to help officials tackle child exploitation crimes, one police data scientist described part of the approach this way: “I essentially dump all that data in a big bucket and stir it with a data-science spatula, and we come out with a lovely risk score for everybody.”
WIRED has made this article free for all to read because it is primarily based on reporting from public records requests. Please consider subscribing to support our journalism.
This risk scoring inside the Think Family Database was just one part of Avon and Somerset Police’s sprawling predictive analytics program. Among at least 23 separate models the force created were algorithms to identify the risk that people would commit burglary, fail to turn up in court, go missing, or become a victim of domestic abuse. One senior officer described creating a “league table” of the area’s most dangerous criminals—an apparent reference to the Offender Management App, which was designed to hold data on around 300,000 people in the region.
How the police have developed and used their predictive tools hasn’t always been clear to the public. John Pegram, the leader of a local police accountability group in Bristol, says he didn’t hear about the Offender Management App until 2023, years after it had been created. When he did learn about it, he began to suspect he might be included. “I think I knew I was on the app,” Pegram says.
In early 2024, Pegram filed a request to find out how the police were using his data. The police refused to say. Months later, after Pegram had hired solicitors to work on his case, the police confirmed he was on the app but declined to elaborate further. Like others across Bristol, the UK, and, increasingly, around the world, Pegram didn’t know whether he had been scored by an algorithm, what that score might be, or how it could affect his interactions with the authorities.
WIRED, working in partnership with the nonprofit newsroom Liberty Investigates, plus the Bristol Cable and Lighthouse Reports, obtained hundreds of pages of documentation from public records requests to build the most comprehensive picture to date of Avon and Somerset’s regional experiment with data collection and predictive analytics. (Liberty, the parent organization of Liberty Investigates, had some early involvement in a potential legal challenge to the program and continues to support Pegram’s litigation.)
The investigation reveals that at least two of these risk-scoring models were quietly abandoned after Bristol City Council staff deemed they could no longer trust them. Previously unreported documents show government inspectors and independent reviewers highlighting a startling lack of transparency about some elements of the program and warning that the systems could undermine public trust. Police data disclosed to WIRED—comprising more than 36,000 model performance scores—appear in some cases to show “genuinely poor predictive performance,” according to an independent analyst who reviewed the data for WIRED.
These findings come as the UK appears poised to embrace predictive analytics and artificial intelligence across the criminal justice system. A familiar face is helping lead the charge: the former chief constable of Avon and Somerset, Andy Marsh, who now heads the national standard-setting body for forces across England and Wales. As CEO of the College of Policing, Marsh has said that effective AI should be “injected like heroin” to speed up British police work. In a recent interview, Marsh said his organization was examining around 100 currently deployed AI tools, including for predictive policing. “Our job is to test the ones that work properly, test them with rigorous evaluation, and then spread them like wildfire through policing.”
In 2014, Avon and Somerset Police was under pressure on multiple fronts. The force, like others across the UK, had seen its budgets slashed. Its chief constable had been suspended. An official report had highlighted its failure to stick to procedures to protect some victims of domestic abuse. After that report was published, the force’s head of performance said, “We believe predictive analytics is the solution.”
Gary Davies, a former police chief superintendent who had moved to a role at the Bristol City Council two years earlier, was thinking along similar lines. Davies led a team at the council supporting children and families. When families were in crisis, “it was blatantly obvious,” he says. It was much harder to spot those who were at the top of a downward spiral.
Davies believed the answer lay in data. A child’s school might hold a record of increasing absences, while the police might know if the child had recently witnessed domestic abuse for the first time. On their own, these might not be enough to trigger an intervention from social services. But together? “If you could see the whole picture, you would realize that the trajectory they were on was going in the wrong direction,” he says.
Starting in 2015, a small group of Bristol City Council and Avon and Somerset Police staff moved into one of the city’s police stations to work on a solution to that problem together. The Insight Bristol team, headed by Davies, started pulling together data from across the public sector to provide frontline workers with all the information they might need about children and families.
The Insight Bristol team didn’t seek residents’ consent to use their data in the Think Family Database. Instead, Davies explains, the team relied on “legal gateways”—a term that describes when data sharing is deemed necessary to meet an agency’s legal obligations, such as the need to protect children. “If you were to give the impression that people had consent, then it creates a false illusion, because, actually, as [a] local authority or police or whoever, we have to keep those records.” Initially, residents could not opt out of the database; later, the council included an opt-out option in its tax letters to residents.
Davies, who recently retired, believes the project did help protect children. “It improved the understanding of risk and vulnerability for children and families,” he says. “It provided that information in a far more efficient way.” When it came to communicating that to the public, Davies says, “it was fairly difficult to get any enthusiasm or interest from groups of people.” Those who did engage said they understood the need to use personal data, he recalls, summarizing the feedback as “We don’t mind you using it to support us, but we don’t want you to use it against us.”
While the Insight Bristol team was busy creating the Think Family Database, Avon and Somerset Police had begun exploring the potential of predictive analytics. In March 2016, the force’s ethics committee met to consider how the work should proceed. Members advised that “careful consideration had to be given to what data is used” and “the variables that are used in the process,” concluding: “The use of the system must be treated with some caution and it must be ensured that there is no bias.” The committee advised that, if the force’s predictive analytics work was to proceed, “the public must be informed as to why and how you are carrying out such processes.”
Once work to compile the Think Family Database was completed, a police data scientist spearheaded the development of predictive risk models for the project. One of those models aimed to identify children at risk of sexual exploitation. The CSE model, as it was known, drew on a wide range of datasets held by the police, council, and other public agencies. Barnardos, a child protection charity, provided anonymized data involving 1,000 children known to have been sexually abused. The model was designed to detect children with similar characteristics. The system analyzed children’s social connections to determine if they were linked to anyone else deemed vulnerable to, or a likely perpetrator of, exploitation. Being flagged as “in need,” “persistently” absent from school, or having mental health concerns would increase scores created by the CSE model, documents say.
The police’s efforts to draw on such a wide range of data raised early concerns. In 2018, researchers at Cardiff University’s Data Justice Lab reviewed several UK citizen scoring programs, including the work in Bristol, noting, “The variables being used can in practice be proxies for poverty.” Davies recalls that most of the children with the highest risk scores were already on the authority’s radar. “They were complicated children already being worked with by social workers and family workers,” he says. “Most of the output told you what you already knew.”
Still, the police’s enthusiasm for predictive analytics was undimmed. “We want to make choices today that will prevent the crime from happening in the first place,” a police business intelligence manager said in 2018. A model for predicting child criminal exploitation, which according to Bristol City Council was introduced in 2019, again drew on data from a wide range of public agencies, including whether a family was receiving housing support or in rent arrears and if a child was receiving free school meals. That year, chief constable Andy Marsh announced: “In 12 months every part of Avon and Somerset Constabulary will be driven through predictive analytics and visualization.”
Across the UK, other law enforcement agencies had been conducting their own experiments in predictive policing—with mixed results. Kent Police, the first UK force to test the technology, had recently canceled its contract with the US firm PredPol, saying it had been “challenging” to show the program reduced crime. Durham Constabulary had come under fire for using sociodemographic data to try to predict risk of reoffending. By comparison, Avon and Somerset Police looked like a success story.
Behind the scenes, though, the police’s predictive analytics work seemed “messy,” according to Elle Pearson, a researcher at Royal Holloway University of London who is completing a PhD studying the programs. “When I started, nobody could tell me what data they had or where it came from or what system was using which data,” says Pearson, who has interviewed more than a dozen staff members from the police and local councils.
Over time, Pearson observed a “function creep,” with systems becoming more expansive, combining more data, and spreading beyond their original purposes. While “conscientious” teams oversaw the development of data analytics, Pearson says, there was little transparency. “In some instances it might just be one person who’s creating these risk models that are making decisions affecting potentially hundreds of thousands of people,” Pearson says.
By 2021, according to a review of Insight Bristol in government records obtained by WIRED, officials from the Centre for Data Ethics and Innovation (since dissolved) were hearing of “ethical tensions” associated with the project. “Large amounts of sensitive data” had gone into the risk scores, the reviewers said, noting that it had been gathered using “legal gateways” rather than by building trust with local people. “Legality is not the same as legitimacy,” they said.
Two years later, the nonprofit organization Social Finance conducted an independent review of the Think Family Database and Insight Bristol’s data work. The 100-plus-page review, which appears to have been made public only after a records request from WIRED, was commissioned by Bristol City Council and the nearby Somerset Council, which was planning a similar system. The review found that the Think Family Database and its visualizations had been useful for child protection staff and could “lead to timelier responses.”
However, it described the risk-scoring models as the “weakest element” of the project, noting that a “lack of accuracy” had undermined their potential to be useful. Council staff had raised doubts about the models designed to assess risk of child sexual exploitation (CSE) and child criminal exploitation (CCE). The review was completed around the same time the council stopped using the models, which staff had recently described as “not fit for operational use.”
While previously the CSE and CCE models had largely been telling staff what they already knew, according to Gary Davies, the former head of the Insight Bristol team, those same social workers told the Social Finance reviewers they increasingly found the algorithms to be “inaccurate.” In an email about the CSE model, one staffer had written: “There are people who’ve been victims of sexual offenses in the last month scoring below those who have been perpetrators of burglaries.”
According to the Social Finance review, there was a reason for the sudden drop in perceived quality: Police had stopped using Bristol City Council data. Officials wanted to “profile children across the entire Avon and Somerset Police boundary”—five separate councils—with the same algorithmic approach, the review says. While the force tried to strike data-sharing agreements with other local authorities, those efforts stalled, meaning the only data it could use to develop the algorithm was its own. This would include perpetrators and victims of crime but not the array of sensitive social factors the model had drawn on before.
After the switch, Bristol City Council staff said children who should have been identified as vulnerable “were not listed” in the results. “Personally, I feel uncomfortable using it to guide our work, because of the lack of transparency on where the numbers come from and how it was developed,” one staff member told Social Finance.
Another said, “I wouldn’t go into a meeting saying I’ve seen this on TFD, because I wouldn’t be confident that that is accurate enough.” One told the reviewers, “We know there’s young girls that get criminally exploited, but they don’t come up, we don’t talk about them cause they don’t fit.” Another added, “I used to spend a lot of time methodically going through the 30 names, emailing people, and checking all the details, but it took up so much of my time that I kind of stopped doing that.”
When the Social Finance reviewers wanted to conduct their own tests on the risk-scoring models, they discovered a startling lack of available information. “Source code and variables that detail how these models were created was unable to be found, which prevented us completing this element of the evaluation,” the report says. (Social Finance declined to comment on its report.)
Likewise, responses to public records requests from the council and the police suggest neither authority kept records about the decision to stop using the CSE and CCE models by June 2023.
Rob Procter, professor of social informatics at the University of Warwick, acted as an expert consultant on the Social Finance review. “The process to build the models was not documented in anything like sufficient detail,” he says. For him, the work in Bristol illustrated the critical need for transparency and public debate about the merits of such an approach whenever it is being considered. “This really raises the question of how you involve the public in deciding to develop and deploy these kinds of tools, and addressing people’s rightful concerns that this could lead to people being wrongly targeted,” he says. “You have to consider the impact that even one false positive has on a family if a child is flagged as at risk of criminal or sexual exploitation.”
Others express similar concerns. Debbie Watson is a professor in child and family welfare at the University of Bristol, where she has been leading a team researching the Think Family project since 2022. Watson says she has concerns about “historic harms” that may have been caused by the risk-scoring models. “Whilst they may no longer be in operation, their use appears to have been significant in ways that have seriously impacted some young people in the city.”
Bristol City Council declined interview requests about the use of predictive risk-scoring systems and did not answer detailed questions sent by WIRED. “This administration does not use any predictive analytics apart from helping to identify children who are at risk of becoming not in education, employment, or training after finishing school (NEET),” councillor Christine Townsend, the chair of the Children and Young People Policy Committee, said in a written statement. “The use of analytics has never replaced professional human judgment or decision-making.”
Davies says any impact would have been minimal, because his staff never relied on the risk scores. “They weren’t really using it to support their judgments, because they didn’t understand it and they didn’t value it,” he says. But the lack of any records about how the risk-scoring models worked, or precisely why they were scrapped, makes it impossible to know for sure. Anyone who was affected would likely never know the reason. As one Bristol City Council worker told the Social Finance reviewers, “Something that’s always on my mind—do the people we’re talking about know we have this data?”
When John Pegram received confirmation he was in the Offender Management App in 2024, he remembers thinking, “I’ve been here before, and I know what you’re trying to do to me.”
As a teenager, Pegram had grown used to police attention. He recalls being stopped by the police dozens of times, something he attributes to being a mixed-race kid in a largely white town. It was less clear to him why he’d ended up in the app. At a 2017 anti-fascist protest in Bristol, he was arrested for striking a police officer in the face. While the officer conceded it appeared to be an accident, Pegram was convicted of assault. Seven years had passed, but did that incident mean he was still being flagged as a likely offender? He had little confidence in the accuracy of any predictions that would be made about him or others. “There’s a lot of bias in the police’s data,” he says. “There’s too many issues for it to be done ethically and fairly.”
In response to WIRED’s public records requests, Avon and Somerset Police provided a huge trove of performance data for 13 risk models used between 2017 and 2024—including those used to predict missing people, antisocial behavior, and who was most likely to commit or fall victim to crime. WIRED passed this data, along with other contextual information about Avon and Somerset Police’s data science program, to the independent AI auditing firm Eticas for review. The verdict was damning.
“Most of these models produce low precision scores, meaning a high proportion of the individuals they flag as risks are incorrectly identified,” the data review found. A model used to help predict burglars appeared to operate with a precision rating lower than 10 percent for more than three years, according to the police data. According to Eticas, that meant fewer than one in 10 flagged as high risk would actually offend. Other concerns included performance metrics for various models shifting sharply. “This is not typical of well-governed models in operational use,” the audit observed.
A spokesperson for the Avon and Somerset Police told WIRED that the force chose not to deploy some of the models it developed, including the one relating to burglaries. When asked why the force had years of audit and performance data for models it did not use, the spokesperson said the audit process was “automated” and used data from a “static file which was not deleted when the decision was made not to deploy the model.”
The police force declined requests for interviews about its data science work and did not respond fully to a detailed list of questions. “Each model is scored based on its performance, and where issues are identified, they will be updated or turned off,” the Avon and Somerset Police spokesperson said in a statement, adding that models are reviewed by a police subject expert before they are deployed.
It’s not clear what steps Avon and Somerset Police took to address the risks raised by its own ethics committee in the early days of its data science work. The committee did not appear to discuss predictive analytics again after 2017, according to records request disclosures. And while Avon and Somerset Police says on its website that “each product and project” pursued as part of its data science work is reviewed by a dedicated ethics group, the spokesperson told WIRED “there has so far been no meeting held,” because “no model has been produced for which potential ethical issues have been identified.”
In response to one public records request, Avon and Somerset Police supplied a screenshot of a “bias check app” that appeared to monitor and compare average risk scores for white individuals and people of color, concluding there was “no significant difference between the two.” The Eticas review said: “Simply including ethnicity as a monitoring variable is not equivalent to testing whether the model produces discriminatory outcomes,” describing the absence of more detailed testing by ethnicity, gender, and socioeconomic status as “a significant omission.”
Asked whether he believes predictive analytics has a role to play in policing or social work, Davies says further work is needed. “When we were trying to do it, we were trying to do it for the right reasons, in the right way, but we didn’t have the capacity that it probably needed.” Part of that work should look at how risk models can inform workers without nudging them into foregone conclusions, he says. “There is a risk that staff see the computer say something and then don’t use their own judgment.”
Predictive analytics continues to play a significant role in policing and public services in the region. Bristol City Council still uses a risk-scoring model to assess the likelihood of a child falling out of education, employment, or training. Avon and Somerset Police’s latest audit data, provided in July last year, indicates that the model used by the Offender Management App correctly predicts just one in three people who actually offend, while one in four people flagged as likely offenders do not.
Last year, Avon and Somerset Police told Pegram that, while he had a profile on the Offender Management App, he had not been given a risk score, because he had not been linked to any offense in the past two years. He still doesn’t know what other data is held or how it might affect his interactions with the police. In July 2025, Pegram’s lawyers wrote to Avon and Somerset Police again, notifying the force of his intention to mount a legal challenge. The Avon and Somerset Police spokesperson declined to provide any comment on Pegram’s case or legal proceedings, although they said the police force is now “identifying an independent party” to review its models.
Pegram wants his details removed from the app, but he also wants Avon and Somerset Police to scrap the program entirely. “It’s not just me,” he says. “I don’t think an AI model should have that kind of power over people’s lives.”
But the direction of travel seems clear. The UK government has just created PoliceAI, a £75 million-backed body that will help to roll out a variety of AI tools to 43 police forces across England and Wales. The group is hosted by the College of Policing, led by Andy Marsh. Launching the project earlier this month, the UK’s policing minister, Sarah Jones, said, “This is the future of policing—and it is happening now.”
Let us know what you think about this article. Submit a letter to the editor at [email protected].
The post British Police Built a Sprawling Crime-Prediction Machine. Some Results Couldn’t Be Trusted appeared first on Wired.
