The Seattle Seahawks have a relatively large offensive staff, though three of their coaching positions are dedicated to offensive line play. Who, among this staff, is in charge of not just looking at the X’s and O’s of the Seattle offense, but those of offenses league-wide? Does Brian Schottenheimer have a staff extensive enough to study every team in the league and extract every usable insight from both the collegiate and professional levels? Perhaps there is a new and innovative XFL or college coach trying to make a name for themselves; how does an organization find these individuals quickly? Does the team have enough analytical support to test the theories he may have?
In the first installment of this two-part series, we concluded that the vast majority of NFL staffs are not sufficiently designed to outperform a generational coach like Bill Belichick. Results have made this apparent for the past two decades.
It’s time for that to change.
Forming the Staff
The central tenet of building a new type of organization is that no one person can solve every problem alone. A good analyst will know that above all, humility is needed. There is no greater sin in the world of analysis than pride causing the rejection of evidence because of a prior held belief. No one analyst or coach knows all the answers; not everyone’s ideas can be implemented, but all are useful in the constant development of better and more thorough plans.
First and foremost, the staff should be split into three divisions with no member belonging to more than one: Gameday Operations, Training, and Future Operations. The integration of an in-house, fully-staffed analytical cell for each division is absolutely key to success. While the division is a separate entity, each group must work in concert to be most effective. Division chiefs must be fully empowered to not only focus on products needed by the other divisions; they shall have the authority to research on topics outside of their explicit scope. This helps ensure we do not form bureaucracy for its own sake, but rather allow mid-level leaders to direct their staffs in a meaningful way. This encourages personal investment in organizational direction, allowing for input that may otherwise not occur.
Gameday Operations is concerned exclusively with matchups in the upcoming game, meta-analysis of game script decisions, and evaluation of offensive and defensive trends. Duties would comprise of intensively testing coaches on situational fourth-down decisions, ensuring play calling attacks an opponent’s known weaknesses, and that play callers understand fundamental insights from both in-house researchers and the broader football research community. Specific examples of situational expertise may include field goal probability, expected points by play type, and the relationship between number of defenders in the box and run success rate.
Gameday Operations is the most balanced of the three divisions, integrating former players to provide insight on individual reactions to certain situations, quantitative analysts providing trend insights, and coaches who make the in-game calls. A strict hierarchy is required for effective conduct, where input is welcome from all levels and participants, but once a decision is made it must be followed to the letter. Charged with tape and data review after each contest, Gameday Operations provides feedback directly to the Training Division.
The Training Division
The Training Division is focused on player fundamentals, executing expected plays based on the upcoming matchup as defined by Gameday Operations, player tracking, feedback, and strength training. Player fundamentals coaches should be selected based on their overwhelming knowledge of a particular position. Candidates for this position will likely include former players within the position group in question. This provides an inlet for players looking to make the jump from the end of their careers to a decision-making role.
The player tracking analysis provided by this department will focus on exertion level during practice, injury prevention, and synthesizing input from the Future or Gameday Operations divisions. The Future Operations cell may discover, during the course of their research into both data and film, that a particular technique is now widely countered throughout the league. This feedback can be taken in by the Training Division and integrated with the existing scheme as defined by the coordinators and coaches. An example of this, pointed out by Mike Lombardi in his book Gridiron Genius, is the development of the 6-2 nickel goal line defense, the formation that led the New England Patriots to their Super Bowl XLIX win over the Seattle Seahawks.
[A fatal error has occurred. “SuperBowlXLIXinterception.jpg”: file not found. Do you want to continue? y/n]
Future Operations feeds into the other planning divisions. Concerned with scouting future opponents and feeding that information as a starting point to Gameday Operations, this division provides in-depth analysis and scouting reports to the Training Division upon the acquisition of new players. Future Operations is also involved in studying the fundamental truths of football, searching to discover not just when a play works, but why. Establishing causal relationships and directing experiments with associated minor league coaches will be legitimate possibilities in the coming years.
In addition, Future Operations shall evaluate other teams that are not scheduled opponents during the current season. The purpose of this exploration is to reverse engineer the methods of the more analytically versed teams to either counter their insights or use their methods to our advantage, should the situation arise. In either case, this research is passed to the Training Division for use after preliminary insights are discovered.
Minor League Input
Minor league input is critical to the establishment of a coaching pipeline. Few times in history has a league separate from the NFL stood up as an opportunity not just for play experimentation, but also coach and analyst development from the ground up. In a smaller, low-threat environment, teams have a laboratory to test creative play concepts, evaluate analytical insights to gain better samples, and explore many other opportunities.
For example: An NFL team wants to test the theory that coaches ought to attempt conversions more frequently on fourth down, confirm the idea that running plays are generally better than passing on fourth and short, or look into specific running play calls, various formation alignments, etc. Using a legitimate minor league to test the effectiveness of certain play types against certain defensive looks in certain situations is incredibly beneficial. If a team were to discover that the outside toss has been highly effective on fourth and three or less against a certain formation, that information could prove the difference between winning a division and a wild card; the difference between a postseason appearance and mediocrity. With the fold of the AAF, these efforts will be more difficult. But with the XFL just around the corner, the time to establish the plans for this pipeline is now.
Using a legitimate minor league to test the effectiveness of certain play types against certain defensive looks in certain situations is incredibly beneficial.
The success of the NFL may not be reliant on the success or failure of one minor league or another, but individual teams could find themselves at the forefront of coaching development and play calling theory by using such a league as their guinea pig and/or university. With strict NFL limits on offseason practices and how small practice squads are, teams are currently limited to scouting opponents and looking at play calls from other sources, such as college football. The problem with this approach is that it is a passive way of learning; our organization does not get to specifically seek out the edges we are interested in — we can only look at what someone tried. Instead, why not create a well-defined system designed for training, information, and feedback, all of which allow for direct synthesis and experimentation? In an age where edges appear and disappear incredibly quickly, a method of active learning could provide a sustainable method for generating not just one short term advantage, but a steady stream of them.
An evidence-based systems approach
The NFL draft is an exceptionally difficult closed market. Scouting alone does not seem to beat the drafting market and teams have been using computers for draft preparation since the 1960s, starting with the Dallas Cowboys. However, several organizations have had public breakups of personnel because of disagreement between those advocating more traditional methods versus those wanting to embrace the analytics revolution.
The success of our organization is driven by the ability to integrate disparate types of data into information easily synthesized by key decision makers. From the very beginning of the staff’s formation, the goal must be not just to analyze data, but present it in a meaningful way. There needs to be a way to cross check player performance from a scouting perspective versus an analytical one. In general, there seems little reason to overly examine the situation where analytics and scouting agree. In either the low or high potential case, there seems little reason to trouble ourselves when both methods agree. However, what is there to do when old world scouts are at odds with the new wave of analysts?
Specifics of exactly how to blend these two types of insight may vary as scouting methods improve, and player tracking brings NFL analysis into the next frontier. That is to say, today we may weight the input of scouts at 60 or 75 percent, as compared to analytics at 25 to 40 percent, but more research and data could change that balance. Though it seems obviously unlikely that scouts will ever be completely removed by automated processes, the balance of power can, may, and will shift to some extent in the minds of the league’s primary decision makers. In the case of disagreement between the two, much like a military, each side of the staff should give two perspectives of their evaluation.
First, the analyst or scout shall present their conclusion to the evaluation staff at large. Second, a measure of uncertainty should be introduced and acknowledged. Critical to communicating uncertainty is ensuring that all analysts are speaking in the same terms. What does it mean to be “sure” in your judgement? What does it mean to express “high risk/reward” in a draft pick? Whether a team uses a continuous scale, an odds ratio, or something adjacent to the Likert scale, the most important factor is to use a system well understood by all participants. The final decision, however, is left to the General Manager as now. That decision is final.
By establishing, explaining, and enforcing a common scale across evaluation types, executives can help root out the causes of disagreement between evaluation types and ultimately create a better decision. These disagreements could be based on flaws in evaluation methods, bias, or any number of other things. No one method will eliminate these negative influences, but having an established system in place to make decisions procedurally, rather than emotionally, can only help to improve organizational performance.
Other staffs will require adaptations. There is no time for consensus building meetings nor long planning sessions between snaps of an NFL game. These decisions are snap judgments made at either the coordinator or the Head Coach level, depending on organization. While NFL coaches have had tendency sheets on their opposition for a while, there is little doubt there are some edges in play calling not fully exploited. However, NFL rules limit the use of computer aid for those advising the coach over headset.
No matter the situation, there remains the ability for an assistant to quickly reference a pre-made chart on the latest available data and relay that information to the play caller in time to assist their decision.
Decisions such as these will be influenced by the personnel at hand, but one of the most often cited canards is lamenting the lack of game state being taken into account. The dirty little secret being: in many cases, this practice doesn’t change the decision materially, or accounting for it is trivially easy.
An analysis of opponents’ tendencies is generally one where game state matters a great deal. Intuitively, teams act differently while leading by 10 than when down by eight, as they should. This problem is easily tractable because of the nature of scoring in football. While separate tendency sheets may be required — depending on the evidence available — for a team up by three instead of seven, there is little doubt that a team down by one will react the same as one down by two. (Barring of course that we’re not talking about the time immediately after a touchdown when the decision is in regards to extra points.) The natural breaks in points being often referred to as “scores” offer us a guide to differentiate our charts in a natural way without binning in a manner unsuited to the game and of little value to the play caller.
Data storage and design should be as such that all applications run by the organization work off of a centralized database. Each play reference in play by play data, or within the radio frequency data provided by the NFL, is tagged with a unique play identification number. By ensuring our scout videos are tagged with that same ID number, any general insights of play type evaluations can be instantly tied to video evidence as well. Data feeds pattern insight tied directly to video examples. This provides a method of discussion between coaches, scouts, and analysts to find the underlying causes of gaps in performance.
Administration and the death of productivity
When installing this new organizational structure, it is absolutely key that personnel and coaches tasked with specific duties are not hindered by unneeded administrative requirements. The installation of a General Staff-esque system must learn from previous mistakes. There is no surer nor quicker way to destroy the productivity of a staff than to burden them with excessive bureaucratic tasks. To this end, all administrative duties must belong to a completely separate department, unobtrusive and built specifically to allow maximum concentration on research, coaching, and planning by the operational divisions. If this step is not taken, the defined system will surely fail, and the reformation will be for naught.
Innovation is the key to success, and the ability to singularly focus is the key to innovation.
Current football staffs are split into departments in which responsibilities overlap and the sheer amount of necessary analysis will inevitably overwhelm. What I’ve proposed is the sectioning of a football staff into three distinct divisions, all fully staffed with their own coaches and analytical department, in order to ensure all possible avenues of research are explored. Empowered mid-level leaders will allow for a more robust approach to research within an organization. Department heads will nest the vision of their cell within the overall vision of the organization, provide feedback for the other divisions, and overcome the deficits found in player talent or individual decision makers.
While a member of the team — coach or player — may have a weakness in a particular area, a quality staff should overcome any shortcomings. In order to maximize the performance of our organization, we must effectively research all possible performance edges, find those used by other teams, nullify them or at least counter them mindfully, and exploit our adversary at every possible point. Rather than sit back and passively learn from the lessons of other teams, even on the lower level, we must set out to establish a method for active experimentation and learning. This provides for a decisive intelligence advantage.
Our new organization will seize the initiative against our opponents before the coin is tossed and adapt to any changes in the playing landscape.
We need not only adapt to change.
We can drive it.
Listen to our newest episode of Tasteful Profanity right here!