Possibility for parameter for how ratings and win chances adjust for uneven teams #130
Comments
|
Duplicate #29 |
|
It's not a duplicate, this issue is about teams of different amounts, not about player performance during a game. |
|
I decided to multiply the mu of each player on the team with fewer players by a variable factor (I used 1.1) for the win_percent function. It seems to be working fine. |
|
Is your issue solved? |
|
Not exactly, I used a temporary fix that isn't very effective. If it's not possible (or no one is willing to implement it) this issue can be closed. |
Can you provide a reproducible example? |
|
Yeah, a concrete example would be helpful here. If you don't like the result of one of the functions, then what should the result be, and why should it be that? |
|
I apologize, I've been a little busy. I'll get some examples ready. Thanks for the help. |
model = PlackettLuce()
async def get_games_with_unbalanced_teams() -> None:
games = await SM5Game.filter(ranked=True).all()
print("Getting win chances for unbalanced games. Close game defined as: difference <= 5000 points")
for game in games:
red_entity_starts = await game.get_team_entity_starts(Team.RED)
green_entity_starts = await game.get_team_entity_starts(Team.GREEN)
red_players = []
green_players = []
if not (abs(len(red_entity_starts) - len(green_entity_starts))) > 0:
continue
for player in red_entity_starts:
red_players.append(await Player.filter(entity_id=player.entity_id).first())
for player in green_entity_starts:
green_players.append(await Player.filter(entity_id=player.entity_id).first())
win_chance = get_win_chance(red_players, green_players) # wraps PlackettLuce.predict_win([team1, team2])
score_diff = abs(await game.get_team_score(Team.RED) - await game.get_team_score(Team.GREEN))
#if score_diff > 5000:
# continue
print(
f"""Win chance for {game.id}: ({(win_chance[0]*100):.2f}%, {(win_chance[1]*100):.2f}%) \
red: {await game.get_team_score(Team.RED)}, \
green: {await game.get_team_score(Team.GREEN)}, \
difference: {score_diff}, \
close: {score_diff <= 5000}, \
team_lengths: {len(red_players)}, {len(green_players)}\
"""
)Here's an example from my code grabbing all games with uneven teams and displaying their win chances, scores, and team sizes. Output for close games only. The win percent chance is way off, even though these are only a few examples, even the games that weren't close still have wildly inaccurate win chances. It seems that when the team sizes change, it isn't able to predict the outcome anymore, though the amount each player adds to a team varies by game, so if a solution is implemented it should probably be one where the amount a player contributes to a team is variable (possibly exponentially). Here's an example that includes all games, not only the close ones. My ratings are defined well with small sigma values due to amount of games and it has been proven to work well with even sized teams. Let me know if more info is needed. Thanks guys. |
What should the win percent chance be, and why should it be that and not anything else? |
|
Since these games were so close, the teams were more even and that should be reflected in the win chances. It should be much closer to 50:50 for games that were that close (although the score doesn't always reflect the win chance, it often does) I've provided some control data below for what the win chances look like for close evenly matched teams. I can also provide data for evenly matched games that aren't close as well if that's needed. As you can see in the data, the win chances are much more reasonable for how close the score is (removing outliers), so it only makes sense for that to be the case for teams of uneven sizes. This is a pretty big difference compared to unevenly matched teams. Thanks again. |
|
I can see what you're saying, but I feel like the best path forward here would be if you wrote your own predict_win function, and then we look at how we can generalize that with a parameter. It's all open source; there are no secrets, everything's there for you to fork 😅 |
|
Of course, I just don't have any of the required experience in this field of math. This is the solution I'm using now (which is definitely not mathematically supported but works somewhat decently). This is built towards my use case (since it only supports two teams). But it could(?) be a good place to start. I do want to point out that this isn't a great solution as it doesn't solve the problem entirely, though I'm sure there's a better solution than what I'm doing. UNEVEN_TEAM_FACTOR = 0.09
class CustomPlackettLuce(PlackettLuce):
def predict_win(self, teams: List[List[PlackettLuceRating]]) -> List[Union[int, float]]:
# Check Arguments
self._check_teams(teams)
n = len(teams)
# uneven team adjustment is only implemented for 2 teams
# 2 Player Case
if n == 2:
# CUSTOM ADDITION
if len(teams[0]) > len(teams[1]):
logger.debug("Adjusting team ratings for uneven team count (team 1 has more players)")
# team 1 has more players than team 2
for player in teams[1]:
# multiply by 1 + 0.1 * the difference in player count
player.mu *= 1 + UNEVEN_TEAM_FACTOR * abs(len(teams[0]) - len(teams[1]))
elif len(teams[0]) < len(teams[1]):
logger.debug("Adjusting team ratings for uneven team count (team 2 has more players)")
# team 2 has more players than team 1
for player in teams[0]:
# multiply by 1 + 0.1 * the difference in player count
player.mu *= 1 + UNEVEN_TEAM_FACTOR * abs(len(teams[0]) - len(teams[1]))
total_player_count = len(teams[0]) + len(teams[1])
teams_ratings = self._calculate_team_ratings(teams)
a = teams_ratings[0]
b = teams_ratings[1]
result = phi_major(
(a.mu - b.mu)
/ math.sqrt(
total_player_count * self.beta**2
+ a.sigma_squared
+ b.sigma_squared
)
)
return [result, 1 - result]
return PlackettLuce.predict_win(self, teams) |
|
Is this implementation able to actually predict the outcome of real matches data? I can see why in some games, teams with fewer players might win in let's say traditional games where an extra player counts. If you could provide some match data where the predict_win function is failing for close matches, then it would be very helpful. It would also aid in making a new parameter let's call it |
|
The implementation in my comment before is shaky at predicting the outcome for uneven teams, that's why I'm looking for a better solution. It's more of a hacky fix than a real solution. The normal implementation, on the other hand, works great for matches of even teams but fails on games with uneven player amounts. Here's additional data for close games with team size differences only including ones that failed to predict the winner, I don't have many instances of games with uneven teams so this doesn't have much data. Here's the data with the CustomPlackettLuce class. And the same data using the normal PlackettLuce model Plus here's the data for even teams to compare (is not affected by the custom model) I have lots of data for this and I can provide more if needed (or if it needs to be displayed differently). I'm also available if you would like to test any additions on my codebase, I'd be happy to help out with that part. |
|
A few instances of data are unfortunately not sufficient. We require match counts in the tens of thousands (the bigger the better) from real and actually played games to verify the effectiveness of such changes. To show that changes work, we use these open-source datasets to perform data analysis and measure performance. You can see the datasets used in |
I'm using openskill for a game where sometimes we have teams of, for example, 6 vs 7. When making teams we put the better players on the team with the lesser amount of players. Openskill estimates are way off results when dealing with uneven teams. It seems that it values extra players much more than the specific game I'm using for it does.
Does anyone have any insight on how to tune a parameter that makes team disparity less important?
Thanks!
The text was updated successfully, but these errors were encountered: