College Football Playoff: Lessons learned in evaluating teams as a sham selection committee member

GRAPEVINE, Texas — I was a journalism student at Baylor in 2014 when the last chart on ESPN hit No. 4 Ohio State turned over. The Bears, infamous co-champions of the Big 12 with TCU, sat on the outside of the first College Football Playoff. Eight years later, much to my chagrin, the College Football Playoffs sham selection committee did exactly the same thing.

But we’ll get to that.

Twelve members of the media, including myself, were invited to participate in the annual mock selection held by the College Football Playoffs. The exercise takes place at the Gaylord Texan, behind the doors of the same swanky meeting room as the actual committee that holds the fate of countless athletic departments in its hands.

That day, I played the role of Navy Director of Sport and current CFP Committee member, Chet Gladchuk. “Behave a little quirky to play Chet’s role,” joked a CFP staffer of the 72-year-old, whose seat I was based in. But if things were going to be quirky, an in-depth revisit of the controversial 2014 playoffs was the perfect place to start.

The challenge of ranking the four “best” teams

The process for ranking teams has been widely reported, but here’s a quick overview: The leaderboards are divided into seven groupings – all three spots 1-9 and all four spots 10-25. We vote to consider six teams at a time for each treble. The first group: Alabama, Ohio State, Oregon, Florida State, Baylor and TCU. As it should be.

Boo Corrigan, CFP chairman and NC State athletic director, tasked us with finding the top 25 teams in the country. From the start, however, it became clear that each of us had different definitions of this metric. Some valued season success, while others wanted the best team from Selection Sunday. Regardless, the resume was a far more important consideration than the team that would win on any given Saturday.

Debating criteria with some of the sport’s most famous names was perhaps the part I was most unprepared for. The ex-players in the room insisted Florida State’s unbeaten season was a trump card. I disagreed. Try to be a sucker like me telling Pro Bowl running back Deuce McAllister and former first-round quarterback EJ Manuel that winning isn’t everything because of the metrics.

The CFP features an analytics system developed by SportSource Analytics that can compare up to four teams at a time using a variety of metrics on giant screens in front of the group. When the Baylor vs. TCU vs. Ohio State vs. Florida State comparison came on screen to demarcate between the #3 and #4 spots, everything that happened in 2014 made sense.

Jeyarajah (left) and former NFL linebacker Kirk Morrison

Kevin Jairaj/College Football Playoff

Team schedules are highlighted on the page with a list of results and common opponents or head-to-heads. However, the most noticeable part of the page is a gradient with teams highlighted from green (good) to red (bad). When the TCU and Baylor schedules surfaced, the amount of red was garish compared to the rest of the field.

Never underestimate the power of data visualization!

Additionally, one of the only true criteria the committee lists is that preference is given to similar teams with head-to-head wins or conference championships. CFP Director Bill Hancock confirmed to the group that since Baylor and TCU were both submitted by the Big 12 as co-champions, teams should effectively be treated as if they each had 0.5 titles.

Compared to the absolute world champions Ohio State and Florida State, the decision was not difficult. We can argue about the specific strength of the schedule and evaluation metrics that the committee uses, but it gave us a complete picture of the committee’s decision.

Perhaps guided by Manuel’s Florida State hand in the room, the ‘Noles actually jumped to #2 in our sham rankings, behind only Alabama. I voted Florida State #5; the 12 metrics that the CFP has identified as most correlated with profits hated FSU and their tight win margins weren’t enough to convince me over other (better) conference champions.

With full disclosure, I tended to rely on efficiency numbers and game quality more than other voters. SportSource Analytics has neatly organized and color-coded stats that correlate most closely to winnings after extensive historical research, with relative scores and games per attack point leading the way. (A special team metric was added to the committee’s top 12 stats at the behest of former Nebraska coach Tom Osborne.) Other mock committee members spoke about the quality of their schedule. Some drew on the quarterback game or attempted to appeal to NFL prospects.

Ultimately, it is about this diversity of opinion. Seven of the 13 members of the CFP Committee are athletic directors. Seven are former college football players, including former NFL player and MIT mathematician John Urschel. Two are coaches. One, former USA Today columnist, Kelly Whiteside, is a journalist. Everyone sees the game differently.

The combination of ex-gamers, writers, and TV personalities gave our group similarly interesting context. San Diego State legend and NFL veteran Kirk Morrison wanted to make sure the Group of Five contenders received the attention they deserved. The Athletic’s Ari Wasserman asked us to consider recruiting gaps. AL.com editor John Talty implored us to remember that Katy Perry was at The Grove for Ole Miss’ matchup with Alabama. McAllister and former Michigan quarterback Devin Gardner urged us to think about winning first. I mentioned that Ole Miss was eclipsed 30-0 by a team from Bret Bielema, Arkansas, who achieved a so-called “borderline erotic” bowl win over Texas. Everyone has their role.

NC State Athletic Director and CFP Chairman Boo Corrigan

Kevin Jairaj/College Football Playoff

How to rank each other’s best

If you’re wondering why there can be moments of inconsistency in a ranking, that’s why. In every secret ballot, different voters weigh their own criteria while trying to convince other voters that their perspective is correct. And really, that made Talk #7-25 infinitely more interesting and competitive.

“When I was Army athletic director, we would have been thrilled to have made the top 25,” Corrigan told the group. “We have to pay as much attention to the tail lights as we do to the leaders.”

We did. The process demanded it.

Some participants tried to bring conference strength into the picture or turned down some programs because they were from the Group of Five. CFP Executive Director Bill Hancock and Corrigan quickly gunned them down. Frankly, after delving into the matchups and seeing the flow of planning from green to red again, it became clear that it wasn’t necessary.

The data visualization had records against teams over .500, against the previous top 25 and the previous committee top 10, but there were really very few of these high-end data points to choose from. Instead, green-filled schedules caught our attention. The ranked games – or borderline ranked games – were discussed on a case-by-case basis.

We drove to Boise State and some were shocked by the amount of high profile opponents in a Group of Five conference. It moved up. Conversely, Wisconsin’s “Power Five” schedule was filled with red. SEC contenders Ole Miss and Georgia were full of green wins and red losses. Michigan State and Kansas State didn’t have green wins, but their only blemishes came from the nation’s top teams.

After each round we chose new teams. At the end we had the chance to discuss all the major differences. UCLA originally backed an Arizona State team that defeated the Bruins in Tempe, Arizona. That has been fixed. Georgia continued to move down while Arizona climbed several places. Marshall came into the field after comparing well on the 12 factors with teams like Minnesota and Louisville.

By the way, this discussion lasted more than five hours. The real thing offers six weeks of talking, many hours of debate and countless weekends watching cutups of every relevant football game imaginable. Hancock noted that there was one committee member who decided to rate each player individually in every game he watched. Every year is different.

What will change (and what won’t)

This, of course, leads us to the 12-team playoffs. In many ways, the decisions the committee makes will change. With six automatic bids, there is less emphasis on figuring out which teams make up the field and more emphasis on ranking and seeding teams.

If the playoffs expand, the committee will remain at 13 members. They still plan on seeding 25 teams and feel confident that at least six conference champions will fall below that number. Instead of filling bowl pairings with teams in the bracket, those will now fill playoff spots at large. Basically, the process isn’t going to change much when the playoffs expand, which contestants in the room are hoping for sooner rather than later.

Ultimately, I gained a lot of respect for the process. It’s not perfect, it’s not unimpeachable, but it’s thorough. With the amount of information and videos available to committee members, I am more confident that they have all the tools needed to make the best decisions possible. More importantly, the secret voting process makes changing the leaderboard for branding or matchup purposes just a little more difficult. Transparency won’t appease conspiracy theories, but it does make me more confident in the process.

And to Hancock, whenever you need a new name to add to your list of committee members… I’m available.

Leave a Comment