AI + CYBER RISK · APPROX. 4 MIN READ
The information can be accurate. The technology can work exactly as designed. And the conclusion can still be wrong.
By Shauna Visser
First, a story.
Before we meet Jill: the story that follows is fictional. The property, residents, screening company and events are invented.
The direction of the technology isn’t.
Upper management had recently signed us up with a new resident-screening company.
AI was their big selling point.
“It’s like having a private investigator,” they told us. “Only better.”
It could supposedly differentiate between people with similar names, connect information across multiple sources and catch things a human reviewer might miss.
And it seemed to work.
Our approved renters were stand-up residents. Delinquency was down. Crime was down.
I was becoming a believer.
Then Jill Watkins walked into my office.
Jill had lived at the property for eight years, had three small children and possessed the supernatural ability to know what was happening approximately fifteen minutes before I did.
That morning she was carrying papers.
“Why have you moved six registered sex offenders into this property in the last two months?”
I stared at her.
“What?”
“Six. And one lives next door to me.”
“Jill, that’s not possible. Our screening wouldn’t approve someone if that was on their background.”
She slid the papers across my desk.
“Well, apparently your screening doesn’t have Google.”
That was unnecessary.
But fair.
I pulled up the first resident.
Approved. Green.
Second resident.
Green.
Third.
Green.
By number six, neither of us was talking.
I called the screening company.
They investigated.
Then called back.
The system had found all six records.
“Oh, thank God.”
Pause.
Apparently, that was the wrong response.
It had found the records and discarded them as false-positive identity matches.
A middle initial didn’t match. One record said Robert while the application said Bob. A birth date differed by a digit. An old address didn’t fit the recent history.
The system hadn’t missed the information.
It had found it, analyzed it and confidently decided it belonged to somebody else.
Except it didn’t.
It belonged to Bob.
“So your fancy computer found them?”
“Yes.”
“And decided they weren’t them?”
“Apparently.”
Jill nodded.
“Your computer needs Jesus.”
I couldn’t argue with that.
Then I asked:
“How many other records has it excluded?”
The answer arrived Friday afternoon, because catastrophic information apparently observes traditional business hours.
Hundreds of potential matches had been excluded across the portfolio.
Most were probably correct.
Probably.
There is no word in property management quite as terrifying as probably.
So we started looking.
Some records had been disconnected from the right people.
Others had been connected to the wrong ones.
Then we found my favorite.
According to the system, one twenty-two-year-old resident owned three houses, operated a landscaping company and had a commercial driver’s license.
He worked at Taco Bell.
And didn’t have a driver’s license.
We called him.
“Do you own a landscaping company?”
“No.”
“Three houses?”
“No.”
“Do you know anyone named Gerald?”
Long pause.
“My grandpa.”
Of course.
Gerald owned the houses, the landscaping company and the commercial driver’s license.
The system had found a real connection.
It had simply misunderstood what the connection meant.
Yet every answer looked exactly the same.
Green. Approved. Complete.
No:
I’m about 82% sure Bob is not that Bob.
No:
This one got weird. You might want to look at it.
Just an answer.
A confident one.
And because the system usually worked, we’d learned to trust it.
That was almost the problem.
If it had been terrible, we would have stopped using it.
Instead, it had been right often enough that we’d stopped asking when it might be wrong.
Success built trust. And as trust grew, verification dwindled.
Before Jill left, I told her we were going to start reading the fine print.
She shook her head.
“No. I think you’re going to start reading the people.”
Jill Watkins, unofficial mayor of Building C, had just explained the problem better than the screening company’s entire sales presentation.
And then came the question anyone making housing decisions should be willing to ask:
Who did we deny, and for what?
Jill isn’t real. The problem is.
AI in resident screening isn’t hypothetical.
RealPage introduced AI Screening in 2019, describing a machine-learning model built using more than 30 million lease outcomes. It continues to market AI Screening today as a predictive scoring model for multifamily resident screening. RealPage
The particular technology and failures in Jill’s story are fictional. The story is a thought experiment, not an account of an incident involving RealPage or any other screening company.
But screening errors involving identity and records are real. The CFPB has warned that inadequate matching procedures can associate another person’s information with an applicant. The FTC likewise tells renters that common background-check errors include information belonging to someone else, incomplete records and duplicate records. In 2026, the FTC alleged that RentGrow failed to prevent some duplicate criminal and eviction records from appearing in tenant-screening reports. CFPB; FTC
But this isn’t really an article about screening.
It’s about trust.
Humans get things wrong.
Software gets things wrong.
AI will get things wrong.
The interesting question is what happens when something gets good enough that we stop expecting it to be wrong.
Trust changes how closely we look.
We don’t see millions of calculations.
We see:
APPROVED.
DECLINED.
LOW RISK.
Or a lovely green light indicating things are good to go.
Every correct answer quietly teaches us:
You probably don’t need to check me quite as closely next time.
We do this with people too.
An employee proves herself. We check less.
An expert establishes a track record. We defer more.
Trust isn’t the problem.
We couldn’t function without it.
The problem is what trust can do to verification.
NIST describes automation bias as excessive deference to automated systems. More interestingly, it warns that as generative AI becomes increasingly reliable, people may over-rely on it or perceive its output as higher quality than information from other sources. NIST
That’s the paradox.
Not increasingly unreliable AI.
Increasingly reliable AI.
Success builds trust.
Trust reduces scrutiny.
Reduced scrutiny makes the unusual mistake harder to catch.
Trust, but verify.
Easy to say.
Harder when the system got the previous 999 decisions right.
Verification has a price.
This is something businesses racing toward AI efficiency may want to include in the math.
If AI can perform in five minutes what took an employee two hours, that’s easy to measure.
But who audits it?
Who reviews the unusual case?
Who investigates a disputed match?
Who understands why the system reached its conclusion?
Who handles an appeal?
Who recognizes that number 1,000 is different?
And what does correcting the mistake cost after the answer has already moved downstream?
The cost of AI isn’t just the technology.
It’s also the verification architecture around it.
That doesn’t erase the efficiency.
It tells us what the efficiency actually costs.
And now the information may start moving.
In Utah, Mountainlands Community Housing Trust is working toward a centralized affordable-housing application for Wasatch Back renters. In September 2026, the Summit County Housing Authority approved a one-time $25,000 grant toward the effort.
Mountainlands Executive Director Jason Glidden described the concept as a “digital backpack” containing documents applicants need to establish eligibility. KPCW
There’s a lot to like about that idea.
But after Bob, it raises another question:
What happens when something we’ve stopped verifying starts traveling?
We’ll come back to that one.
Human in the loop.
There’s a phrase we’re going to hear a lot:
Human in the loop.
It sounds reassuring.
But consider this:
AI finds the information.
Matches the identity.
Connects the relationships.
Identifies the anomaly.
Assigns the risk.
Explains the problem.
Recommends the action.
Steve clicks APPROVE.
Congratulations.
We have a human in the loop.
I’m not entirely convinced we still have human judgment in the loop.
The better question isn’t:
Did a human touch the decision?
It’s:
Did a human independently question it?
Because eventually there will be another Bob.
Maybe in housing.
Maybe banking.
Insurance.
Hiring.
Cybersecurity.
The technology may find legitimate facts.
The pattern may look beautiful.
The light may turn green.
And everyone may keep moving because the machine has earned their trust.
Until somebody asks:
Wait. How do we know that’s Bob?
Trust but verify only works if the verification survives the trust.
And perhaps one of the biggest risks of increasingly reliable AI isn’t that we’ll never trust it.
It’s that eventually, we will.
