Too Smart to Fleece
by Erik Bilicki

Years ago my buddies and I ran a football simulation league. This fall we were at Lambeau for the Bears game and talked ourselves into bringing
it back.
I didn’t rebuild the old game. The old software had static coaching. A computer team ran the same approach in Week 14 as in Week 1, and once you figured out what it would do, you beat it the same way forever.
Nothing was running a franchise unless a human was sitting there.
The new one uses current NFL rules, and a team with no human owner still runs like a real franchise. Its coaching staff learns, adjusts quarter to quarter, scouts the next opponent, and builds its own game plan. Its front office scouts, talks trades, and works the roster all week, not just on Sunday. The AI owners will talk trash, and they’ll take a side bet over who owes the first round. It’s an experience now, not a game.
A few things are regulated on purpose. Games play out live at a scheduled time instead of resolving the instant you click simulate. Lineups come due on a set clock, kickoff is kickoff, and you wait for Sunday like everybody else. The AI staffs also work from the same information we do, with no extra secrets.
The idea is that the machines compete with us instead of around us. I don’t mind losing to something that did all the reading, but losing to something that acts a thousand times faster needed a rule.
The other half was keeping us from running away with it. A league dies the week somebody figures out how to fleece the computer, and this one was supposed to be too smart to fleece.
My friends are also my beta testers. A couple of them work in data and BI, so breaking software is close to a professional skill. They play at completely different levels, some in the app most days, others only on Sunday, which is about what real users look like. Across all of them, they’ve found playability and data problems I’d never have caught testing my own app.
There was one hole in that: None of them trade. They don’t believe in trading. So I had a trade system that nobody was touching, let alone trying to break.
So I built somebody who would
Last week I added two kinds of agents. One is an owner persona that actually runs a franchise, making the signings, lineups, and trades a human owner would, which means the league now sees more moves in a week than most of the human owners attempt in a month. The other is a red team, and it doesn’t own a franchise at all. It looks at the league through one club’s account, prices deals, reads whatever any owner can read, and files a report on whatever looks wrong. It never completes a single move. Its only job is to find ways to cheat.
Day one, the red team ran a deal past another club: seven bench players, nobody rated above 64, for a first-round pick. The AI owner would have taken it. Then it ran ten practice-squad guys for the same pick. Would have taken that one, too.
For comparison, it ran one solid linebacker by himself for that pick. Rejected.
The AI owner was adding players up like a grocery receipt. Ten guys who’d never see the field outweighed one who’d actually play, and every practice-squad player was worth about the same no matter how bad he was.
Eleven findings in seven days. Two serious, five moderate. Players already out for the season still traded at full value. At one point the agent asked for a verdict on a trade with nothing in it, and the system came back with “close,” plus a suggestion to toss in some seventh-rounders.
So much for too smart to fleece.
What I’d have missed
I’d tested the trade system before the agent showed up. Those tests ran ordinary deals, the kind a reasonable owner proposes, and ordinary deals worked fine. Five of the eleven findings needed somebody actively trying to break the pricing, and nobody was.
One of them was my own fault. Two days before the agent started, I’d capped what a kicker could be worth so nobody could turn one into a first-round pick. The agent pointed out that the league’s best kicker, a 96, was now priced like a fourth-rounder. My fix for one problem created another one.
Six fixes are live. The next run will tell me whether I fixed anything or just moved the problem. I also haven’t audited the trades completed since early November, so I can’t swear nobody used these, though given that my friends don’t trade, I like my odds.
The part that followed me to work
I lead the AI and data capability at Logisolve, and this is the piece of the hobby that turned out to be useful at the office.
My trade system got the kind of testing most things get: somebody confirms that normal use works. That leaves a gap when the system makes judgment calls, because a bad call doesn’t crash anything. It goes through looking like a decision. A customer talks a support agent into a refund it shouldn’t give. A vendor words an invoice to slip past an automated check. Testing only the reasonable requests doesn’t prove much.
My friends and my agents find different things. The humans tell me what’s confusing and what doesn’t match what they expected. The agents tell me what’s exploitable, because that’s the only thing I asked them to look for. A rollout needs both, and it needs you to be honest about who your testers are. Mine are my friends. They want the league to work, so none of them will spend a Saturday hunting for a way to cheat it. Open the same system to the public and somebody will, and that person will quietly use what they find instead of reporting it. The red team stands in for the guy I haven’t met yet.
Five things from the league that carry over:
- Give something the job of attacking it. A red-team agent tries what a polite tester never would, every day, and it doesn’t get bored. Point it at wherever money, access, or eligibility get decided, and tell it to win, rather than verify.
- Let it look without touching. Mine prices deals and never makes one, so I can run it against the live league with nothing at risk. In a business, that means a read-only role, a scoped set of credentials, or a sandbox cut off from the real thing. Decide which one before you point an agent at anything that can move money or change a record.
- Re-test after every fix. The kicker problem came out of a fix. Every change is a new thing nobody has tried to break yet, so a findings list is only worth keeping if it gets re-run against each release.
- Pace the agent to the people. Mine can’t move faster than the game clock allows. If an agent can act faster than anyone can review it, the review is decorative. Throttles, approval queues, and batch windows aren’t friction for its own sake; they keep a person in the loop.
- Keep a person on the judgment calls. The agent files the report, and somebody who knows what you intended decides what’s actually a bug. A couple of mine turned out to be the rules working as designed. An agent can tell you what a system does, but whether that’s wrong is a question about intent, and intent lives with people.
That’s how we approach AI at Logisolve: build the thing, then go looking for the ways it breaks before somebody else does. If you’ve got an agent heading for production, that’s a good conversation to have early.
The agent runs again tomorrow. I expect to hear about my kickers.
Note: The league is a personal project I build and run on my own time, for a handful of friends. It isn’t a Logisolve product, and nothing in it represents our services, our client work, or anything we sell.
Logisolve is part of the V-Soft Consulting Family
After more than 27 years of building Logisolve, we’re excited to share that we have
joined V-Soft Consulting Group, creating new opportunities for our clients, consultants,
and partners. Our clients will continue
working with the same trusted teams,
now supported by V-Soft’s expanded global capabilities in platform engineering,
Data & AI, Salesforce, and ServiceNow. For our consultants, this next chapter
brings broader opportunities, stronger support, and access to a larger network
to help grow their careers.
Together with our partners, we look forward to delivering even more scalable, comprehensive solutions. We’re excited for what’s ahead!
Explore Career Opportunities with Logisolve!
Looking for your next opportunity? At Logisolve, we have a variety of contract and contract-to-hire roles available across all our capabilities. If you’re interested or know someone who is, please reach out to us at referrals@logisolve.com. We’d love to connect! Logisolve offers medical, dental, vision, life insurance, short-term disability, long-term disability, paid sick leave, and retirement benefits to eligible employees.
