On Thursday Andy Katz wrote this piece:
https://www.ncaa.com/news/basketball-men/article/2019-02-07/net-rankings-ncaa-tournament-what-they-mean
Since the NCAA has been rather opaque as to what the NET is, I decided to do a deep dive into Andy’s column. From the top:
AK: “The NET will be the most common word heard throughout selection week and on Selection Sunday.”
This might be true, but not necessarily in the way Andy meant it here. We will likely hear a lot about the NET quadrant 1 and quadrant 2 wins.
AK: “The purpose of the NCAA’s Evaluation Tool ranking (i.e. NET) is to sort teams into the four quadrants on the team sheets the men’s basketball selection committee uses for selection and seeding.”
Bingo.
AK: “It is not a deciding factor.”
Of course, but no one has suggested that it would be.
AK: “It is not going to determine if a team is in or out of the bracket.”
If this were true or false we’d have no way of ever finding out.
AK: “It is an organizational piece for the committee.”
OK.
AK: “And with a month left in the regular season this is what you need to know:”
AK: “The most important thing about the NET is that if you beat good teams, don’t lose to bad teams and have quality wins away from home on a road or neutral court, you’re going to have a solid net ranking.”
Is there a system that rewards losing to bad teams, or punishes you for beating good teams?
AK: “The primary component of the NET is the TVI, a results-based factor that considers the strength of the opponent and the location of the game. If you beat a team that you’re expected to beat, then it doesn’t do as much for your ranking. Losing to teams that you were expected to beat will hurt your ranking.”
Beating teams you are supposed to beat should boost your rating, and losing to teams you weren’t expected to beat should still hurt it. The question to ask is “How much” and the NCAA isn’t saying.
AK: “The committee has always, and will always, place emphasis on winning quality games, and the quadrant system introduced last season places greater importance on the location of the games. The formula of the average net efficiency (offensive efficiency minus the defensive efficiency) is a factor in computing the ranking. So, too, is the winning percentage, the adjusted winning percentage (based on where the game is played — home, neutral or road) and the scoring margin with a cap at 10 per game.”
We’ve had the quadrant system before, although switching it from the RPI can only help. For more details as to what Andy is referring to, you can check this out:
More on this graphic in a bit.
AK: “These analytical tools weren’t just created with no intention. The data scientists at Google Cloud looked at several years’ worth of data for late-season games played on neutral courts, including conference and NCAA tournament games. They used those results as test cases to measure the accuracy of the ranking system. The computer program considered many features, and it was determined that the aforementioned data points were the best ones to use for the ranking formula.”
Accuracy? Accuracy of what? Predictive accuracy? Is NET intended to be a predictive metric? Winning percentage and adjusted winning percentage are not predictive inputs. They’re descriptive. The Team Value Index (TVI) is also a descriptive metric. Thus, I ask: How did Google and the NCAA test the accuracy of a descriptive metric?
AK: “The NCAA tournament men’s selection committee and the NABC wanted to get rid of the old RPI. There was talk of at one point averaging a number of the other metrics available to the public. But the issue there would be a lack of control of each formula.”
The NET is an absolute black box as for as the public is concerned. Also, while I understand a desire to have full knowledge of the metrics you are using, averaging predictive and descriptive metrics is a bigger problem, and it’s one we’re still stuck with.
AK: “The NET treats all games the same, whether they are played in November or February. And that’s why teams that may have had strong non-conference schedules can continue to have a strong ranking even during a losing skid in conference.”
Ignoring when the games are played is a good thing.
AK: “The goal of the NET was to produce a true ranking/sorting system for the selection committee for Selection Sunday. And while the NET is being released throughout the regular season, the end result is what it will look like as a final product, not after each game.”
If they wanted us to evaluate it as a final product, they could easily have released the NET results from previous seasons.
AK: “Valuing when a game is played is an individual choice of each member of the selection committee. But where the game was played and who it was against is used in determining the NET ranking, regardless of the time of the season.”
We’ve been over this…
AK: “When the season ends — the hope is it will show — that when you played and beat good teams you got rewarded, especially if the games were away from home.”
Like Winning Points?
http://www.vegaswatch.org/2014/01/winning-points.html
AK: “Not all home games are created equal (see a Duke home win over Virginia versus a win over Boston College) but all road wins aren’t inherently valued more than a home win. Remember, the NET will help sort the teams for the selection committee to decide on selection and seeding. But it won’t be the ultimate factor in either decision.
Let’s break this down. No one was saying all road wins are greater than all home wins. No one was saying the NET was going to be decisive in selection or seeding.
Here are some of our questions about the NET:
- What are the various inputs weightings?
- How is quality of opponent weighted in the TVI
- What are the NET Ratings? (Not the rankings. The underlying ratings.)
- What is pushing Rice up into the top 200? No predictive or descriptive metric has them that high, so we cannot find any explanation for their ranking.
- What were the pre-tournament NET rankings (and ratings) for previous years. Not releasing them suggests that the knowledge could somehow be harmful to the perception of the NET, which is a bad sign in and of itself.
The RPI is dead. Now we need to learn a lot more about the new monarch.
sethburn 