While working on a column about frontline supervisors vanishing from the workforce even as back-office managers thrive, a fundamental question dogged us: Do frontline managers matter?
It’s a sneakily punishing problem. To get enough data to figure out which level of management is most responsible for a firm’s success, you’d have to follow millions of workers over several years and evaluate them on crystal-clear performance metrics. It felt like our only option was to apply at somewhere like Walmart and devote half a career squirming high enough up the org chart that we were given access to the Data Motherlode.
But before we sent in our application, we went looking for any research that had touched on the subject.
Most of these research searches turn into wild-goose chases. Much like wild geese themselves, they never get near the final column. Well, with one notable exception about bird pecking order. And now we’ve got another, because once again we got ludicrously lucky!
The University of Toronto’s Stephan Heblich — one of the geniuses behind the research into pollution and poverty we profiled previously — just tackled this question. He did so with help from his equally creative collaborators, Andreas Ferrara of the University of Pittsburgh and Christian Dippel of Western University (which of course sits astride the Thames River in London, Ontario).
On the assumption that the fundamentals of leadership probably have remained constant, they went to where the data was: the U.S. Civil War, one of the most well-studied conflicts of all time.
For their working paper, they rounded up state reports detailing the careers of 2.2 million soldiers, all from one side of the conflict.
“The Union Army was incredibly good at record keeping. We tried to do the same thing for the Confederate Army, and there, this is a very different story,” Ferrara said, adding that “as the war continued, they increasingly ran out of resources and manpower, hence record-keeping was not a key priority for them.”
The researchers could see when someone enlisted (or, more rarely, was conscripted), when he was wounded, even if and when he deserted. They were able to link many of these Union soldiers to census records from before and after the war, and to the Army companies in which they’d served.
Linking those companies to battles wasn’t as straightforward as it might sound. But after months grinding through historical and genealogical records, Ferrara and friends compiled historical movements and actions for about 99 percent of them.
Once they matched those movements to major battles, they could figure out how each soldier and officer fared, which allowed them to tabulate desertions, derelictions, decorations, incapacitations and other tribulations.
The desertions were key. They were fairly common in the Civil War — about 10 percent of soldiers bailed at some point — and thus provide a simple, universal measure of a captain’s success. A better officer keeps his company together — in intranecine conflict, as in professional basketball, the best ability is availability.
Even outside of conflict, some captains struggled to keep their companies complete. Soldiers tended to “run away because they’re unhappy with the hygiene conditions, the pay — maybe they’re homesick,” Ferrara told us.
If a captain could keep his company together in the long hours when their lives weren’t at stake, that means he “must have done something to maintain team cohesion and to motivate his soldiers,” Ferrara said.
And that basic facility for leadership translates onto the battlefield, where the captains who were best at out-of-conflict cohesion also perform best at dereliction prevention in high-pressure situations.
Focusing on 19 pivotal battles, the researchers found units with below-median leaders had about 1.7 times as many desertions as units with above-median leaders. This was true even when you account for the quality of the units themselves — it’s not just that better leaders have better soldiers. It really is something those leaders did on the fields of Shiloh or Chickamauga — and the marches leading up to them — that set them apart.
This method allowed the researchers to cast aside the contemporary cruft that could complicate modern attempts to study leadership qualities.
“In this particular context, we can isolate this specific ability of leaders, because that’s literally the only thing they do,” Ferrara explained. “If you think about a store manager, they also are in charge of the finances, the hiring, the firing, the strategy for the store, the marketing. Our guy in the Union Army just needs to make sure that the soldiers stick together and then follow orders.”
And once you evaluate all those Union guys along that one axis, you begin to see patterns in what makes a leader awesome or awful.
“Effective leaders were older, courageous, and socially proximate to their teams,” the economists wrote in their working paper, adding that “Acts of valor by leaders fostered cohesion like nothing else.”
That may be one reason the lowest-ranked officers had the greatest effect on soldiers. At least in terms of keeping its men from deserting, a captain was more than three times as influential as any general. That is to say that an army with excellent captains and a mediocre corps of generals would be several times more effective than one with a brilliant general and crummy captains.
Interestingly, nobody could predict who would become a good leader. Not the soldiers themselves — units that elected their leader would have done just as well if they’d drawn straws. Not the military brass — the best leaders weren’t promoted any faster than the worst ones. And not the civilian world — men who had been managers before the war performed no better than their peers with no such leadership experience.
“Ultimately, you can’t really predict who is going to do well under pressure,” Ferrara told us.
The quality of the best leaders wasn’t always clear from the outset, but they tended to improve rapidly in their first few months — if they survived, that is. The analysis found that “high-quality captains were more likely to die in battle, consistent with leadership-by-example or leading-from-the-front.”
Among those who lived, the people who led best in the war did take that experience with them afterward. After they entered civilian life, the best wartime leaders earned significantly more than their less-impressive comrades in arms.
We’d understand if you were slightly skeptical that these 160-year-old principles, fueled by hardtack and forged by bayonet charges, remained relevant for modern civilian leaders. We weren’t sure ourselves.
But in a recent analysis, Columbia University economist Robert Metcalfe and his collaborators analyzed store managers for two major American and British retailers — in exchange for data, they agreed to keep the firms anonymous — and found much the same thing.
It’s tough to predict who will be a good manager in advance, but once you measure their performance, it’s super evident that they’re responsible for a significant share of the differences in quality between stores.
“Good managers know how to make labor productive and know what type of labor they need,” Metcalfe told us.
Dear readers, the Department of Data’s quest for questions continues! What are the most common features of made-up languages? What parts of America have the most-educated couples? Which NFL positions are most Republican-leaning? Just ask!
If your question appears in a column, we’ll send you an official Department of Data button and ID card.
The post Frontline leaders are disappearing. The Civil War shows why that’s a problem. appeared first on Washington Post.
