Prediction Model (PM) 1.11

Last week I demonstrated that home field advantage is important and that teams that have won tend to win--not exactly breaking new ground. This week, I really wanted to stamp my name into the history books that win/loss record over the course of the season, and not just the previous game, can help us predict winners. Watch out, Copernicus.

First, I developed and quickly discarded PM 1.1. It used win/loss records from the season and adjusted the value of wins on the quality of the opponent using a basic Elo Chess format. I included a factor for home field advantage and gave the world PM 1.11.

PM 1.11

EP(w) = 1/(1+10^((opp rating-team rating+x*Home)/400))

where:
EP(w) = the estimated probability of a win
x = the home adjustment factor
Home = location (1=home, -1=away)

Each team starts the season with a rating of 1200 (Div 1AA teams enter each game at 800), and then are adjusted based on their performance in each game. The adjustment factor is:

team rating(t) = team rating(t-1) + k*(X(w)-EP(w))

where:
X(w) = the outcome of the game (1=win, 0=loss)
k = the adjustment factor

k can be adjusted to give greater emphasis to games. You might want to place more importance on games against better opponents, such that k(opp rating) = f(opp rating)*l. Or, you can put more emphasis on losses, or wins, or home games, or games late in the season, etc.

For this week, I have opted for a k of 300. Chess uses values of 32 and 16, but football seasons are not long enough to allow for that kind of adjustment. A larger value also places greater emphasis on the team's performance in recent games, and the k value actually had very little impact on the model's accuracy.

First, let's review its past performance. I didn't bother with week 1 because the rating's at that point were random.

week 2 = 59.09%
week 3 = 65.22%
week 4 = 60.55%
week 5 = 69.91%
week 6 = 61.82%

I have two major concerns with these results. First, they aren't very good. Second, they are consistent. Theoretically, the model's predictive power should increase through the season--unless team's performances vary more than I am assuming with this model. To the model's credit, it actually predicts a team's odds of winning and is, in this sense, more accurate than it is at actually picking the winner.

And the ratings for the first six weeks produce the following top 25:

1	Ohio State
2	Illinois
3	South Carolina
4	LSU
5	South Florida
6	Missouri
7	California
8	Boston College
9	Kansas
10	Arizona State
11	Connecticut
12	Auburn
13	Cincinnati
14	Tennessee
15	Brigham Young University
16	Texas A&M
17	Colorado
18	Oklahoma
19	Boise State
20	Virginia Tech
21	Hawaii
22	Maryland
23	Wisconsin
24	Virginia
25	Florida State

Illinois got a big boost and Wisconsin took a fall after the Illini completed the un-upset last week (did anyone outside of Badger nation think Wisconsin would win that game?). BYU, Texas A&M and Maryland are surprises in the top 25--though as a BYU grad, a U of Maryland student and lifelong Aggie I can't disagree--but the biggest shocker, in my opinion, is the Jayhawks breaking the top 10.

There are three prominent exclusions from this 25. West Virginia had been in the top 10 the two weeks prior to their loss to South Florida and Syracuse did not help them climb back into the top 25--the Elo Chess model adjusts only for differences in the odds of a win and the actually game outcome. This is, obviously, a flaw in the system.

But PM 1.11 does have the wisdom to recognize that a bad Texas team, who has lost two straight and barely escaped against Arkansas State and Central Florida, has no place in the top 25. PM 1.11 places them at a more deserved 64.

USC dropped 34 spots to 41. PM 1.11 had given them a 95.3% chance of winning. Anyone, even Harbaugh himself, who claims to have believed that Stanford would pull that off has got to be an idiot. But USC lost with an odds differential of -.953, which means their rating dropped almost 300 points in one week.

On a side note, its a bad sign for college football as a whole this year that Ohio State is contending for a national championship. No offense to the Buckeyes, of which I can name one or two at most, but they have half the team they had a year ago. LSU is a very good team, but they needed a heroic effort to beat a rebuilding Florida squad.

As for this weeks picks, PM 1.11 is pretty conservative. It favors the higher ranked team (in the national polls, not the PM 1.11 poll) in all cases except it gives Missouri a 70% chance against Oklahoma. Aggie fans might want to get excited. If they win (and they have a 92% chance according to PM 1.11 at Texas Tech) and Oklahoma does lose, they will have a two game lead on the rest of the South.

The Forward Lateral vs. Keepin' It on the Ground

This first half of Texas A&M's season has been primarily defined by a debate that sees coaches and players on one side and many fans and commentators on the other. The players' and coaches' position was spelled out by a post-game rant from quarterback Stephen McGee. After beating Fresno State is triple overtime, McGee fielded questions on the team's inability, or unwillingness, to throw the ball down field:

I'm so tired of hearing about that. When it comes down to it, we're a team. We're going to do whatever it takes to win. Today we did that. That's what it comes down to: getting the ball into the end zone. Offensively, we ran the ball every freakin' time, and they did not stop us. I wouldn't see us doing anything different. That's what we are about: getting the ball in the end zone.1

Getting the ball in the end zone is important, but A&M blew a 19 point lead at halftime and needed some heroics to stay alive until Fresno State failed on a two point conversion in the third overtime. The game also might have been over sooner, and less stressful, if the Aggies had put the ball in the end zone in the first overtime.

Despite what McGee might think, we no longer live in the days of the four horsemen and four yards and a cloud of dust. But is it possible that the rest of the nation has become too reliant on passing yards.

Aaron Schatz and the NFL stats pros at Football Outsiders2 point to the importance of net yards per pass play (including sacks). Some college teams consistently put up pass efficiency numbers that even the 2006 Colts didn't amass. Is the relationship yards per pass and offensive efficiency or, more important, winning the same as it is in the NFL?

My methods are simple and preliminary, based on data from all games including DI-A teams for the first five weeks of the 2007 college football season. But the results are worth noting.

I first ran simple correlations between same basic offensive statistics in a game and the Win/Loss outcome. Yards per pass play were highly correlated with success, with a pearsons r of .48. But total passing yards (TPY) was not significantly correlated with victory. More pronounced was the relationship between rushing and winning. Both total rushing yards (TRY) and yards per rush (YPR) had even higher correlations than YPP (yards per pass play), both exceeding .5. In other words, the initial analysis suggests that successfully running the ball is more important in winning football games than throwing the ball.

In fact, in turns out that achieving a higher YPP is, in large part, a product of not throwing incomplete passes. YPP and a teams completion percentage are highly correlated, but completion percentage is not strongly correlated with victory. It appears that passing can help a team win in two ways--producing first downs by consistent completions, leading to good field position or, perhaps more significantly, producing the occasional 90 yard bomb.

We also need to view TPY in context. Often teams will rack up meaningless yards in blowouts and in desperate efforts to come back from deficit in the last minutes. This is consistent with the result that TPY is positively correlated with number of points scored, even by the opposition. TPY is also generated by throwing more passes. Passes are also about twice as likely as rushes (4% of passes and 2% of rushes) to result in a turnover.

Successfully rushing the ball (TRY, YPR) and completing passes (comp, YPP) have the advantage of not only producing points (r=.50, .52, .35, and .58, respectively), but of preventing opponents from scoring (r=-.31, -.28, -.15, -.19, respectively). Not surprisingly, rushing is more successful at preventing opponents from scoring, because it uses more clock. Yards on the ground and completing passes also generate first downs and keep the opposing team off the field.

But the relationship between YPP, YPR, and TRY are not correlated with the opponent's TPY, YPP, and YPR and is only weakly correlated with TRY. Instead, the rushing the ball appears to be better correlated with better field position, so that opponents need to generate more yards to score.

Running teams have another advantage--it appears that production through the running game is more consistent and, therefore, more reliable. I used a simple model to predict a team's rushing yards by generating rating from past performances and subtracting from it an estimation of their opponents defense, again from past performances. By week 5, this model was better at predicting a team's rushing yards than a similar model at predicting passing yards. It seems rather intuitive that a rushing attack will vary less from game to game.

Not surprisingly, running the ball influences a teams ability to throw the ball. For example, if a team rushes for more yards they also tend to have higher yards per completion. This weekend, when Miami (FL) threw a 97 yard touchdown pass against North Carolina on a play action in a losing effort, they profited in yards per completion from the threat of running the ball.

It might not be appropriate, then, to focus on getting the right mix of running and passing. Easterbrook provided a nice discussion of the values of the spread offense, and efforts therein to find the correct run to pass ratio.3 The proper ratio for each team will vary with its ability to throw or run the ball--perhaps A&M should keep the ball on the ground. In reality, the proper call on any play is the one that has the best chance of accomplishing the primary objective--a first down, touchdown, or just getting a lot of yards very quickly. But, to appease my curiosity, I decided to ask--is there, generally speaking, an optimum pass to run ratio?

The values on the y-axis are the number of pass plays per run play. On the x-axis is the point margin. Overall, throwing fewer passes and rushing the ball more tends to result in a more favorable point margin. No team won in the week presented (week 4) while running twice as many pass plays are run plays (earlier weeks, when Texas Tech was winning by 50+, there are a few high-pass-ratio outliers; Hawaii, the other prolific passer, won by 56 with a pass to run ratio just under 2). A line of best fit, though, shows an upturn in the number of pass plays for those teams winning by 10 or more.

What, then, do we learn? First, in college football it is important, with a few exceptions, to establish a run game. Running the ball keeps your defense off the field, it forces your opponents to start deeper in their territory (in part because you are less likely to turn the ball over), and it sets up the pass (and vice versa). Passing the ball allows the occasional quick-strike touchdown, especially if the running game has sucked in the safeties, and can also be effective, like the running game, of eating yards and keeping the defense off the field if you are able to complete a lot of passes. Despite McGee's claim that Fresno State could not stop the run, if the Aggie offense had occasionally stretched the field with a play action pass down field, the might not have gone into triple overtime.

Finally, a target rate of between 2 and 1.5 run plays per pass play appears optimum for most teams in most scenarios. Obviously, a team should sacrifice ratios if they are losing by a couple of scores in the fourth quarter, but a powerful running game will prevent a team from finding itself in this situation.

PM 1.0 - Results and Conclusions

It was a good weekend for PM 1.0. It picked 4 meaningful upsets for Saturday (10/6)--a top 15 team losing to a lower or unranked team--and finished 3-1. It only misfired on Syracuse and West Virginia.

Prophetically, it also gave LSU the nod over Florida.

Texas and Oklahoma, despite a tight game from start to finish, did not even go into overtime, let alone violate the rules of the game and finish in a tie.

The model was guided by two basic principles. First, teams that won last week are typically better than teams that lost the week before. It doesn't control for the quality of the opponent in the past week, but, just maybe, we tend to over qualify results based on the strength of schedule, and I can think of two reasons why the quality of the opponent doesn't matter as much as we might assume.

First, football is as much psychological as physical. Mentally, good teams tend to play down to bad opponents and bad teams play up for good opponents--especially in conference play where intimidation is not as much of a factor.

Second, the result of a football game is really the product of the dozens of minute match-ups. I learned this lesson in Little League and have never forgotten it. At the time, our teams was in second place behind the Twins, but a head-to-head gave us one last chance to challenge for the league championship. The Twins were starting their ace, a junk baller named Mark Ramirez. He had a weak fastball, but his curveball had dominated the best hitters in the league.

The year before, though, I made a discovery--Mark tipped off his curveball by his facial expression when he pitched. I was a fastball hitter, so I just sat back and waited for him to groove me a fastball or hang the curve. I finished 3 for 4 (1 HR, 4 RBI) and we won the game--but still finished in second at the end of the year.

In football, the most visible battles are those between receivers and cornerbacks, but linemen are also going head to head. Domination by one, especially if it is the receiver or defensive end, can dramatically affect the outcome of the game. Just ask Winston Justice and Eagle's fans about that. If a team has a lineman that can slow down Glenn Dorsey, the performance of the LSU defense is drastically diminished (though still good). A rating of an offense makes assumptions not only about the typical performance of the unit, but also of each individual player against generic defenses--and no defense is generic.

The second principle guiding PM 1.0 is that there is an advantage of playing at home. Of the 4 upsets picked for this past weekend, the home team won all 4

Home field advantage varies both with the travel experience and the "rowdiness" of the crowd. In fact, I believe we often over emphasize the importance of a crowd. Just think about how you feel after a 4 hour flight--flying and bus travel have real, measurable physical consequences. Unfortunately, I don't have data on the length of the flight, the hotel accommodations of the team, or measures of the sodium levels of key players prior to game time so I will have to make generic assumptions about the effect of playing on the road. But it is obvious that playing at home helps teams win games.

The Best Possible Ranking (BPR) Explanation

All rankings in college football are subjective, so there is no right or wrong answer, but some systems are more right than others. The Best Possible Ranking (BPR) is, quite simply, the most right possible.

Why? The BPR ranks teams only on wins and losses and the difficulty of their schedule, but does so with minimal data loss. It achieves this through a two step process. First, teams are power-rated using wins and losses, margin of victory (MOV), total yards, turnovers, yards per play, etc. Second, power-ratings are used to evaluate a team's schedule and estimate the difficulty of achieving a team's win/loss record given that schedule. Consequently, each team is ranked exclusively on its wins and losses - that is, of course, the point of sport - but we can better assess the value of those wins. Unlike other non-MOV systems - I use all available data.

The first step in the BPR is important, but not unique. There are literally hundreds of systems for statistically rating teams. The key to the BPR is in the second step. It is motivated by this concept - if team B were to play team A's schedule, what is the probability that B would win more games, the same number of games or fewer games than A.

In the chart above, each line represents a max win frontier. Moving up the y-axis is the probability that the event will occur and across the bottom is the probability that a team will win each game on the schedule (for the sake of this example, we are assuming that the schedule is made up of 12 identical opponents). As we move to the right, the wins come easier, and as we move up the event is more likely. The "event" is the team winning that many games or more. Consequently, lines curve up as we slide to the right - the probability of winning at least that many games increases as the games get easier.

If we were to draw a line horizontally across the chart, any intersecting point would represent an equally likely event. This fact allows us to make some interesting comparisons. For example, as highlighted, it is equally difficult to go undefeated against a schedule of teams you would normally beat 90% of the time (favored by around17.4 points, see chart at bottom) as it is to win at least 11 games against a schedule of teams you would beat 80% of the time (favored by around 11.3). Likewise, it is equally likely that a team will win 1 or more of 12 games against teams it has a 15% chance of beating and that a team will win 4 or more of 12 games against teams it has a 45% chance of beating.

What does this mean? If we have a reliable system for generating power ratings (power ratings are an objective fact, though impossibly difficult to actually measure), and we reduce a team's accomplishments to its wins and losses given the strength of its schedule, we can mathematically, and objectively, deduce national rankings.

It is, quite simply, the Best Possible Ranking.

Prediction Model 1.0

A meteorologist and a statistician agreed on a bet. Before the start of the year they would each guess the weather conditions for each day of the next year, and whoever was closer would win. To the meteorologist's surprise, despite studying charts and trends and the position of the earth relative to Venus, he was beaten by the statistician. He asked the statistician how he was better able to guess the weather and the statistician replied -- I predicted that the weather each day would be just like that from the day before.

Prediction model 1.0 is based on this very simple assumption. It assumes that a team that won in their last game is more likely to win again, and a team that lost is more likely to lose, such that:

If G(t-1) = W and OG(t-1) = L then G(t) = W and
If G(t-1) = L and OG(t-1) = W then G(t) = L

where G(x) is the outcome of the given game.

Frequently, both teams will have won or lost in their last game, and in these circumstances I have given preference to the home team, such that:

If G(t-1) = OG(t-1) then If H(t) = H then G(t) = W and
If G(t-1) = OG(t-1) then If H(t) = A then G(t) = L

Results:

I applied the formula to six conferences in week 5 of the 2007 college football season. I counted every performance by a team in those conferences uniquely, such that if two teams in those conferences played, the results would be double counted. Of 61 games, PM 1.0 accurately predicted 39, or 63.9% of the outcomes. Not too shabby for an extremely simple model in a week filled with upsets.

It also picked upsets of Colorado over Oklahoma, Illinois over Penn State, Florida State over Alabama, and South Florida over West Virginia.

Notable picks from PM 1.0 for week 6:

Illinois over Wisconsin. This one seems perfectly reasonable to me. Illinois at home with tons of young talent coming off a win against Penn State against an overranked Wisconsin team that needs to lose.

Syracuse over West Virginia. Don't underestimate the Orange. If PM 1.0 proves prophetic they could run away with the Big East title and still be one of the worst teams in college football.

Tennessee over Georgia. The Vols are slow and Georgia is playing well, but don't underestimate the importance of home field.

South Carolina over Kentucky and LSU over Florida. Who would have thought that the second biggest game of the week would involve Kentucky? The SEC seems to have a monopoly recently on interesting match ups.

Texas/Oklahoma tie. No home field advantage to decide the tie breaker so, defying the regulations of the sport, Texas and Oklahoma are supposed to break even.

And the pick of the week:
Duke defeats Wake Forest

The Value of a Tournament

In the next few paragraphs, I will present the solution to the most pressing problem (in my opinion) facing college football. (How about that for an intro!)

The primary goal of a college football season is to recognize a national champion. Logically, there has been a lot of discussion about how this is best done. The focus of this discussion has been a lack of benchmarks: too few games and too many teams. But a tournament is not a magic solution.

There are two problems with a tournament. First, it places too much emphasis on a few games at the end of the season. Second, because a team's performance may vary from game to game the team that wins the tournament is often not the best team overall.

In a tournament, performance through the season is recognized by allowing the team to participate in the tournament and matching up better teams against worse teams as much as possible. If we believe that a college football season does not offer the necessary match-ups throughout the season, a short tournament that provides those match-ups may be the best available option.

Theoretically, the superior system would give entrance only to those teams that have a legitimate claim at the national championship after their performance during the season. The superior system should also be more likely to crown the superior team with a minimal number of games. A seven game series would be preferable if it didn't take two months to play out.

Including too many teams, therefore, is a double wammy. First, it increases the number of games to be played. With 8 teams there are seven elimination games and the finalists must play three extra games. Because teams vary in their performance from one week to the next, the more teams and the more games a team must win, the less likely it is that the best team will actually win.

This can be easily demonstrated with the equations developed for week 4.

Assume we have 8 teams, A through H, with ratings: A = 35, B = 34; C = 33, etc., such that A would be favored by 1 over B, by 2 over C, and by 3 over D. These are, in this case, objective ratings, such that A has been the best team over the course of the season and is currently the best team in the country. But the probability that A will win the national championship varies with the type of tournament employed.

In a single elimination tournament with 8 teams:

A	24.71%
D	12.33%
E	9.60%
H	4.39%
B	19.93%
C	15.80%
F	7.48%
G	5.76%

In a single elimination tournament with 4 teams:

A	32.19%
B	26.89%
C	22.37%
D	18.55%

In a double elimination tournament with 4 teams:

A	34.36%
B	27.30%
C	21.51%
D	16.83%

In a single championship game with 2 teams:

A	53.06%
B	46.94%

Therefore, in this scenario, the best predictor is, in fact, the single championship game, and logically so. This format, though, is flawed because we cannot always objectively identify the top two teams.

In recent years, three of the four top conferences (PAC 10, Big 12, SEC and Big 10) have produced a title contender. We have also seen challenges made by representatives of the Big East and WAC, and, before long, the ACC will begin to produce contenders again. Therefore, a four or five team tournament (with a play-in game) should be sufficiently inclusive to identify the nation's top team.

Another advantage of a four team tournament is that placement plays almost no role in deciding the winner.

This narrows our options to two potential formats, single or double elimination. The double elimination format is more likely to crown the superior team, but requires almost as many games and more games per team than the 8 team format.

But the double elimination format is only significantly more likely to produce the correct national champion if the best team is significantly better than the rest of the field. For example, if we reproduce the earlier experiment but with larger differences between teams (which is, admittedly, the less likely scenario), such that A=35, B=31, C=27, D=23, etc., we arrive at the following probabilities:

In a single elimination tournament with 8 teams:

A	54.34%
D	4.28%
E	1.31%
H	0.03%
B	27.76%
C	11.79%
F	0.38%
G	0.11%

In a single elimination tournament with 4 teams:

A	53.72%
B	27.58%
C	12.84%
D	5.86%

In a double elimination tournament with 4 teams:

A	61.22%
B	26.03%
C	9.52%
D	3.23%

In a single championship game with 2 teams:

A	62.01%
B	37.99%

In this case, the three or four extra games would definitely favor the better team at all in the end. But, assuming we would want a single championship game (such that the team emerging from the losers bracket would not need to defeat the winner bracket representative twice), the odds of A winning would fall to 56.24% from 61.22%. And, since the best team is still almost twice as likely to win the tournament as any other team using this format, it can still efficiently identify the champion.

A play-in game should increase the probability that the correct team is identified as the national champion (unless the best team is ranked 4th at the time) and could be played before the rest of the bowl season. As such, a system could be devised that the conference champions of the major conferences could be reserved a play-in spot if they do not otherwise qualify by ranking as long as, I believe, the top three teams are not forced to win more than two games to win the tournament.

Graphing Margin of Victory against Probability of Victory

As a cautionary note, this particular blog entry is not inherently interesting to the average college football fan. It is important, because it lays the groundwork for much of what I will be doing in later blog entries, but if you don't want to read it all I suggest that you skip to the implications--that's the fun part.

All teams have good games and bad games, but most will tend to fall somewhere in the middle. In fact, if a football team were able to play the same game over and over again, and if we could objectively measure the performance of the team and graph it, it would look something like this:

Two teams will have overlapping distributions, and the better team (which we will call A) will peak further to the right than the inferior team (which we will call B). The more they overlap, the more likely it is that B’s performance will land higher on its distribution than A’s performance and B will win.

Point margin is useful because it provides more information than win/loss records. If team A beats team B by 40 points and C beat D by 3 points, we can make the educated guess that, were they to play again, it is more likely that A will beat B than that C will beat D. In this way, the Margin of Victory (MOV) is related to the Probability of Victory (POV)

Trade Sports (tradesports.com) allows participants to buy stocks (that represent a prediction) that will pay out at $1 if the prediction is correct, so the price of the stock represents the probability the prediction will be correct. For college football, Trade Sports allows participants to buy stock on the outcome of the game and probability that a team will cover the spread. Therefore, if there is a 50% chance that a team will cover a 20 point spread and an 85% chance that they will win the football game (based on the price of the relevant stock), we can assume that the teams, based on their "averages", are 20 points different. More importantly, we can assume that if two teams are 20 points different on average then one will win 85% of the time and the other 15%.

Back to our previous example, if A beat B by 20 in week 1 and then they play in week 2, all else being equal, then we would assume that A has an 85% chance of winning the second game based on their performance in the first game.

Using scores from 47 games on Trade Sports, I came up with the following equation:

POV = 1/(1+10^(-MOV/18.8))

This equation represents a simplified version of the line of best fit. It is notable that the over/under had no effect—a 7 point win is a 7 point win, whether that win is 100 – 93 or 7 – 0.

Our best guess, then, if A beat B by 20 in game 1, is that A has a 92% chance of winning the rematch.

Implications:
On average, a team will be within in 12.7 points of their average performance 80% of the time.

.8 = 1-(2*(1/(1+10^((12.685-0)/(18.8*SQRT(2)))))

Also, a team has a one in five chance of making up a 12.6 point difference in ability if the opponent puts up an average performance.

A team that was favored by 20 (10 points per half), but losing by 14 points at halftime, still has a 38% chance of winning the game.

POV=1/(1+10^(((-expMOV(-10))-(currentMOV(-14)))/18.8)=.3799

Addendum: I've approached this problem several ways since I first wrote this entry, and every time I've come up with the same conclusion.

Link Exchange and Guest Blogging

If you are interested in exchanging links or being a guest blogger, send me an email with any necessary information. If the site has a similar topic to this blog and good content, I will include the link in the Sites Worth Seeing section. If it is less related, I will probably include the link in More Links.

If you have any questions, send them here.