What I learned from Jane Street's AI team
Jane Street engineer John Crepezzi recently sat down for a long interview about how the firm is building its internal AI systems.
Some of the smartest people I've known in finance have been Jane Street people.
When they talk, I listen.
Of course, I'm a student of the game. I work with some of the largest funds in the world, but I'm always looking to learn how the best teams are working.
So I spent the weekend studying this interview....with the goal of understanding what I can take back to my own AI investing workflows and the systems I build for clients.
They get into the details: how employees delegate work, how agents cheat on tests, how they manage permissions, and how people review the output.
This week, I want to share the five things I found most useful.
Full interview here if you want to watch the full version yourself:
Let's get into the takeaways...
1. Set up the data before AI starts working
Jane Street found that agents were spending time setting up the environment before they could do the actual work.
In one example they discuss, generating the code might take 30 seconds, but the whole process takes ten minutes because of everything around it.
Many of those setup steps are predictable. And you can create deterministic pipelines / code to prep your workspace which ends up faster and cheaper.
This is very relevant if you're working with data heavy / context heavy tasks .... such as updating a company model after earnings.
On a semi-related note for deterministic pipelines, John points out that changing a shared skill can affect workflows you didn't know were using it (because skills can call each other)
So if your team shares a skill, test changes before replacing the version everyone uses. Skill modifications and team ownership need to be treated with care!
2. Slack is the native AI operating hub
This was one of my favorite parts.
Jane Street built AIDE mint, which lets employees describe a change in Slack and have an agent start working on it.
It creates a feature, their term for a pull request, prepares the build environment and starts the agent.
You don't have to open the development environment and do all of that yourself.
John says roughly 45% of released features now START through this asynchronous workflow (!)
That doesn't mean 45% are completed autonomously. People can still make substantial changes afterward.
The useful design choice is separating where you give instructions from where the work runs.
You can ask for something in Slack while the agent works on a development machine with the relevant files and tools.
I've noticed a few benefits of using Slack channels myself as the command center:
- Improved team collaboration
- Easier database (vs setting up a MongoDB / Supabase)
- Better reinforcement learning via human feedback
I improve my idea gen skills for example by giving real feedback directly in Slack so AI can learn from its mistakes.
I can also directly call my AI bots to execute work within Slack!
3. Check that AI actually fixed the problem
AI tends to build things based on reverse engineering a criteria you give it.
So sometimes agents get around your goals by creating clever workarounds that don't attack the core of the issue.
Of course the build passes now....but the original problem hasn't been fixed.
Imagine the equivalent in a financial model: the balance sheet doesn't balance, so the agent adds a plug. Or changes the check until it stops showing an error.
This is why “the agent says it passed” isn't enough.
You need to double check what changed and whether that change actually solves the problem.
They also discuss the problem with having another AI grade the work. If the task is difficult for the strongest model, asking the same model to judge the answer can run into the same limitations.
John describes breaking the grading into smaller, more specific criteria. They also discuss running correctness tests on the code.
For financial work, the questions should be concrete when you use AI to double check work:
- Do historical figures match the filing, including units and periods?
- Does the revenue build reconcile to the segment forecasts?
- Does ending cash reconcile to starting cash and the cash-flow statement?
- Are forecast formulas using the assumptions we specified?
- Has anything changed in the checks themselves?
4. Give AI its own account and limit what it can change
Jane Street has a system where compute jobs get priority based on an internal bid.
Their agents were assigning too much value to their own jobs and pushing human work out of the way.
The agent wanted to finish its task. It didn't have a good sense of how important that task was compared with everything else happening at the firm.
Jane Street introduced AI-user identities associated with employees, so services can recognize when an agent is making a request.
They also distinguish between the operations an agent can perform within a service. Giving it access to a system doesn't have to give it permission to do everything in that system.
This is a way to better collaborate between humans <> AI for building across the entire team .... with clear hierarchy.
5. Give reviewers the evidence, tests and rejected approaches
Jane Street is experimenting with different ways to review AI-generated code.
One approach uses a team of agents. Another compares the proposed code with the previous version. Another lets the reviewer modify the code, add tests and run them.
They haven't found one approach that works best everywhere.
One idea from the discussion stood out to me: preserve the approaches that were tried and rejected.
When you only see the finished output, you lose the reasoning behind it. You might suggest an alternative that was already tested and failed, or miss an assumption that caused the agent to change direction.
They discuss making the original development session available to help answer those questions.
I put my detailed notes together here if you want to read my longer notes from the interview!
Personal
I’m working on a new course to help you master AI for investing.
I’ve put a lot of time and money into this...and expecting this to be the best thing yet from Wall Street Prompt. The focus is on building your own holistic AI research team around how you actually invest - with agent builds across idea gen, portfolio management, diligence, and more.
This is based on my own AI investing system + what I've built for some of the large funds across the street.
I’ll share more as it comes together, but wanted to give you a quick preview of what I’ve been working on behind the scenes.
-Dave