Fact Or Fiction? Scouting Departments Must Make Tough Draft Calls Off Smaller Sample Sizes

That is the dilemma teams are facing in the run-up to the 2020 draft.

Discerning the signal from the noise is a challenge for scouts every year, but that challenge is amplified when teams have just a few weeks of spring evaluation time, and none at all for northern high school players whose seasons had not yet begun.

So when a player performs at a level entirely different in the spring than he had shown previously, is he showing better tools? How does a scout know which is real, the past or the present?

“That’s the issue that every team is facing,” an American League scouting director said. “I think the first step in approaching it is, we’ll weigh the sophomore year performance X amount and the junior year performance based on the sample Y amount . . .

“The important thing we all need to consider is you can’t treat the Misners and (Hunter) Bishops and (JJ) Bledays all the same.”

While analytical models that weight players’ performance are present in every draft room, this year a separating factor could be the extent to which area scouts know their players.

It’s not enough to say a pitcher is throwing harder or hitting better early in the season. But why is a pitcher throwing harder? What has changed to allow a pitcher to throw more strikes? Is it a mechanical tweak? Is it a change in body composition or muscle mass? Is it a mental adjustment? Is a player performing because the competition just isn’t great, or has the player made a legitimate, sustainable change?

And in the cases where a player is struggling out of the gate, the same questions are waiting to be answered.

Figuring out those details isn’t a one-man job. The teams who succeed most often will seamlessly incorporate reports from area scouts, secondary evaluations from crosscheckers and national evaluators, analytical information and data from sources like TrackMan and Rapsodo. Then, they’ll tie all of that into the strengths and weaknesses of their organization’s player-development system.

Technology has played a larger role in scouting over the years, and perhaps no draft will be more reliant on technology than 2020.

“It’s absolutely going to have a ripple effect on how teams operate in the future,” one American League executive said. “Technology is not only more useful this year, it’s paramount.”

Scouting departments have spent the last several months regularly talking through the class over Zoom calls, while scouts have been filing reports and endlessly re-evaluating players by going over video. Some scouts have gone as far as talking with NFL evaluators—whose process typically involves much more video scouting—to see how best to evaluate players over tape.

Started in 2004 with an initial focus on basketball, Synergy, video database service, offers a wealth of information for many college programs that allows teams to dive deeper than ever before on prospects. Now, Synergy has become as commonplace in scouting parlance as Stalker radar guns and Travis Mathew polo.

“I would have said this was crazy two years ago,” the AL scouting director said. “If we come across a pitch where the pitch data our analysts like it, we can zone in on exactly what that pitch was and which pitch against which batter to try and validate it and say, ‘We’re seeing what you’re seeing.’

“If you get a crappy look (in person), instead of searching through ESPN, you are going right on Synergy and logging 20 innings of a pitcher and 20 (at-bats) from a hitter . . . You can log those ABs and provide yourself with more security (in your evaluation).”

If a scout wanted to see how a hitter adjusted in 0-2 counts, they could filter all of those scenarios and watch them. Or they could filter a specific pitch type to get a handful of looks on an offering they might not have seen in person or simply aren’t sure how to grade in their live looks.

“We are finding out that Synergy is becoming important,” one NL scout said. “I don’t think it should ever replace scouting, but I think it’s a good supplemental piece.

“Now we have a guy make a jump and the scouts like it. So then you go to the crosschecker and he says, “Oh, the jump makes sense because of X, Y and Z.’ And then you take it to analytics and they are like, ‘Oh well this is backed up by X, Y and Z . . .’ Scouting is about trying to put all the pieces of the puzzle together—Synergy allows for that extra layer.”

There are a number of players in the 2020 draft whose stock has shifted in just four weeks. South Carolina righthander Carmen Mlodzinski entered the year as a potential top 10 pick after a dominant summer in the Cape Cod League. But he didn’t dominate at the same level over his first four weeks and never got to show his stuff against SEC hitters.

“I don’t think you ever take a small sample and totally throw it away,” the NL scout said. “You need to not become overzealous. Mlodzinski was a consensus first-round pick coming into the year, but you are sitting there and he struggled the first four weeks. He didn’t punch anybody out . . . He took a huge hit, and it’s only based on those first few weeks of performance.

“What happens if he made an adjustment in conference and got better? He could be an undervalued guy in the draft because he is going to fall because of his performance. But what if we know he has more in there than what he showed?”

On the other side of the spectrum, Duke righthander Bryce Jarvis wasn’t seen at all over the summer. Instead, he was working on improving his pitch mix, which now features a fastball that’s been up to 96-97 mph after previously being in the 89-92 range.

“This isn’t about the competition he’s facing,” the AL scouting director said. “We have guys putting on excellent control and command (compared to) previous years. That’s a noticeable difference. Play out that start to the season over the course of all of 2020.”

A scouting department’s success will be determined by how well it identifies and separates the real changes for the noise created by small sample sizes. That is true to some extent every year, but with just five rounds in 2020, each pick carries more weight.

That could lead to more teams simply punting on risky players of any capacity and putting a priority on players with a consistently strong résumé—short sample and long. Of course, that comes with its own risk: missing out on toolsier players with more upside.

At the end of the day, the shortened 2020 season doesn’t significantly alter the scouting process. Players have never had static scouting reports you could figure out with certainty after seeing a certain number of at-bats or innings.

“It’s tough for us to get these guys with 100 percent certainty,” the NL scout said. “It’s impossible . . . I think that at the end of the day, the big thing is just knowing your players. You have to filter and look in between and figure out what’s real and what’s fake.

“And that’s tough . . . I think at the end of the day you have to understand what’s real and what’s not.”