As corporate America races to measure how employees are using AI, Goldman Sachs is taking a different path.
Many firms have taken to tracking individuals. At JPMorgan, the firm monitors dashboards displaying tens of thousands of users’ AI-related activities, letting employees compare themselves with their peers. At Meta, the social media giant is installing software on US employees’ computers to track keystrokes and mouse movements in order to train its AI, Business Insider reported last month.
At Goldman Sachs, Chief Information Officer Marco Argenti is focusing on evaluating teams’ velocity with AI tools rather than zeroing in on the metrics of individual users, which he says can result in “missing the forest for the trees.”
Argenti, who oversees roughly 12,000 engineers, is steering the firm through a rapid shift as AI reshapes how developers create software. He’s focused on how quickly Goldman’s engineers move from idea to production, and whether their output is actually improving how long it takes to go from an innovative idea to a product that’s ready for rollout.
While Goldman can access data on individuals’ use of tools, including its AI products, the firm is more focused on taking a cross-team view to speed up project timelines, perform quality control assessments, and track AI token consumption for budgeting. The bank hasn’t built tracking dashboards to enforce AI usage for developers to actively compare their adoption rates to their colleagues.
I sat down with Argenti to discuss how Goldman is defining success for developers in the AI age, and why he says individual monitoring of developers’ activities risks missing the point.
Here is our conversation, edited for length and clarity.
There’s a debate over whether to track or not. As a manager, what’s your take? Is there one road that’s more effective than the other in promoting AI adoption?
Given that work is generally done by teams — and now teams that are hybrid agents and humans — we tend to look at team metrics. Mostly, it’s the velocity at which they develop a feature.
We look at flow — how long it takes to go from idea to production. You know it because you see that a team has a certain backlog, and all of a sudden it starts burning down the backlog.
Tell me the rationale behind why you say it’s more effective to look at things on a team level rather than a personal basis.
If you look at the individual, you are really missing the forest for the trees. It would be like looking at only one player on the field.
Fine, this player is doing more movements, but why am I not scoring more goals? Well, because they need to pass the ball.
What’s the right way to go about analyzing how productive AI is making your engineers?
Measuring developer productivity, as you know, has been something that companies have been chasing forever. And there is no single magic metric, because some companies say, “How many lines of code?” but that’s not really a great way to do it. At the end of the day, what constitutes useful output is not necessarily the number of lines of code.
Say you get into a fitness training program. It’s probably more effective to see the change in some of your vitals rather than look at numbers in isolation. If you’re starting to see your cholesterol going down or your sugars go to a better level relative to where you were, then maybe that’s an indicator that you’re on the right path.
Another big topic on everyone’s mind is the soaring costs of tokens. How do you measure whether your spending is creating demonstrable results?
If you have a lot of token usage and output doesn’t move, then at that point, it probably means that you’re still in the experimentation phase. We identified a threshold — below it there was no change in the output metrics, but once we surpassed that threshold, productivity started moving.
Upon investigation, that showed us people were going back and forth with the AI on the planning itself — using tokens to create implementation plans and business requirement documents before getting into coding. That preparatory work does not immediately yield coding output, because it happens before developers start writing code.
So you see acceleration in token usage, but no immediate change in output. Once the plan is created, the agent starts to build code, and then you see both further increases in token consumption and results in the form of coding output.
How do your engineers feel about AI’s utility in speeding up how quickly they complete tasks?
We’ve passed a critical mass where excitement has overtaken fear.
I actually just came out of a bit of a showcase — an innovation type of meeting that we do. The dominant sentiment is really a sense of empowerment. People feel almost liberated. A few weeks or months ago, of course, there was a real bit of skepticism and fear, but I correlate that to people that were not really using it.
How does that speed change the way they present work to you? Are you seeing a shift away from “PowerPoint culture” toward something more hands-on?
They come in with a very concrete problem they solved. They go into prototypes of new products almost immediately — sometimes before fully formalizing the idea. Today, you have almost real-time prototyping. Even during a meeting, you talk to them and they can change it in front of your eyes.
In the old days, they would have come in with a PowerPoint or a six-pager and I’d have to imagine it. Today, I saw an actual product. I can literally say, “How about this?” and in the meeting, they can make changes. There’s zero time between idea and prototype. You kind of “3D print” software.
Read the original article on Business Insider
The post Forget tracking AI use. Goldman’s tech boss cares about this instead. appeared first on Business Insider.
