{ "cells": [ { "cell_type": "markdown", "id": "cell00", "metadata": {}, "source": [ "# What wins a CTF game? \ud83c\udfc6\n", "\n", "**995 league games on one map. Every lever below is measured in the first\n", "25 seconds of play and compared only between teams that were playing the same\n", "matchup.**\n", "\n", "Those two constraints are the whole design, and they exist because the obvious\n", "way to answer this question fails twice.\n", "\n", "**Whole-game statistics describe the result.** 64% of wins are wipes, so \"the\n", "winner got more kills\" is close to a restatement of who won: over a whole\n", "game the winner is ahead on net kills in **92%** of decisive games, which\n", "predicts the result almost perfectly and explains none of it. Measuring from the\n", "opening of play instead puts the measurement before the outcome.\n", "\n", "**Strong policies do more of everything.** Any raw winner-versus-loser\n", "comparison is confounded by who was playing. This corpus fixes that for us: it\n", "is a complete round-robin, every ordered pairing replayed 8 to 10 times, so\n", "conditioning on the matchup holds both policies and both sides exactly fixed.\n", "\n", "> **The corpus.** 995 games: 923 decisive (593 wipes, 330 captures) and 72\n", "> lose-lose draws. 11 policies, all 55 pairs, both side assignments, eight\n", "> seats per team." ] }, { "cell_type": "markdown", "id": "cell01", "metadata": {}, "source": [ "## 0 \u00b7 Setup\n", "\n", "Install the wheel and fetch the bundle, both from `https://sw.gy/files`, so this\n", "notebook runs anywhere with a network connection. Everything below runs from\n", "three small tables on a laptop CPU in a few seconds." ] }, { "cell_type": "code", "id": "cell02", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "%pip install --quiet https://sw.gy/files/ctf_analysis-0.2.0-py3-none-any.whl" ] }, { "cell_type": "code", "id": "cell03", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "%matplotlib inline\n", "import tarfile\n", "import urllib.request\n", "from pathlib import Path\n", "\n", "FILES = \"https://sw.gy/files\"\n", "BUNDLE = Path(\"ctf-victory.tgz\") # ~1 MB; cached next to the notebook\n", "\n", "if not Path(\"data/pairs.parquet\").exists():\n", " if not BUNDLE.exists():\n", " print(f\"downloading {FILES}/{BUNDLE.name}...\")\n", " urllib.request.urlretrieve(f\"{FILES}/{BUNDLE.name}\", BUNDLE)\n", " with tarfile.open(BUNDLE) as t: # unpacks into data/\n", " try:\n", " t.extractall(\".\", filter=\"data\")\n", " except TypeError: # Python < 3.12\n", " t.extractall(\".\")\n", "print(sorted(p.name for p in Path(\"data\").glob(\"*\")))" ] }, { "cell_type": "code", "id": "cell04", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "import polars as pl\n", "import matplotlib.pyplot as plt\n", "from matplotlib.colors import LinearSegmentedColormap\n", "\n", "from ctf_analysis.stats import (benjamini_hochberg, bradley_terry, logistic_irls,\n", " midranks, paired_rank_biserial, rank_auc)\n", "\n", "PAGE, INK, MUTED, GRID = \"#14100c\", \"#e8dcc8\", \"#9d8f79\", \"#2b2119\"\n", "BLUE, ORANGE, AQUA, WARM = \"#3987e5\", \"#d95926\", \"#199e70\", \"#e66767\"\n", "plt.rcParams.update({\n", " \"figure.facecolor\": PAGE, \"savefig.facecolor\": PAGE, \"axes.facecolor\": PAGE,\n", " \"figure.dpi\": 110, \"font.size\": 10, \"text.color\": INK,\n", " \"axes.labelcolor\": MUTED, \"axes.titlecolor\": INK, \"axes.edgecolor\": \"#4a4038\",\n", " \"xtick.color\": MUTED, \"ytick.color\": MUTED, \"legend.frameon\": False,\n", "})\n", "\n", "def frame(ax, grid_axis=\"x\", title=None):\n", " # Recessive chrome: two spines, one grid direction, nothing competing with\n", " # the marks.\n", " for s in (\"top\", \"right\"):\n", " ax.spines[s].set_visible(False)\n", " ax.set_axisbelow(True)\n", " if grid_axis:\n", " ax.grid(axis=grid_axis, color=GRID, lw=0.8)\n", " if title:\n", " ax.set_title(title, loc=\"left\", pad=10, fontsize=11)\n", " return ax\n", "\n", "def spread(values, gap):\n", " # Nudge direct labels apart so a crowded line chart can skip the legend.\n", " out = np.array(values, float)\n", " order = np.argsort(out)[::-1]\n", " for a, b in zip(order, order[1:]):\n", " out[b] = min(out[b], out[a] - gap)\n", " return out\n", "\n", "pairs = pl.read_parquet(\"data/pairs.parquet\")\n", "features = pl.read_parquet(\"data/features.parquet\")\n", "episodes = pl.read_parquet(\"data/episodes.parquet\")\n", "\n", "DECISIVE = pairs.filter(~pl.col(\"is_draw\"))\n", "window = lambda w: DECISIVE.filter(pl.col(\"window\") == w)\n", "W = window(\"t600\") # the primary window: the opening 25 seconds\n", "WINDOWS = [\"t300\", \"t600\", \"t1200\", \"full\"]\n", "LABEL = {\"t300\": \"12.5 s\", \"t600\": \"25 s\", \"t1200\": \"50 s\", \"full\": \"whole game\"}\n", "\n", "print(f\"{episodes.height} games | {W.height} decisive | \"\n", " f\"{episodes['is_draw'].sum()} draws\")\n", "print(f\"endings {dict(W.group_by('ending').len().sort('len', descending=True).iter_rows())}\")\n", "print(f\"design {W['red_policy'].n_unique()} policies, {W['matchup'].n_unique()} \"\n", " f\"ordered matchups, {W['pair'].n_unique()} unordered pairs\")\n", "played = episodes.group_by(\"red_policy\", \"blue_policy\").len()[\"len\"]\n", "print(f\"replays per ordered matchup: {played.min()} to {played.max()}\")" ] }, { "cell_type": "markdown", "id": "cell05", "metadata": {}, "source": [ "## 1 \u00b7 The whole test, in one function\n", "\n", "Every result below comes out of one comparison: **inside a single matchup, did\n", "the side that did more of this win?**\n", "\n", "Ranks are computed inside each matchup, so map, seed, engine version, both\n", "policies and both side assignments are shared by everyone being compared and\n", "cancel out. The null hypothesis is that winning is unrelated to the metric,\n", "which means the winner labels can be shuffled inside each matchup without\n", "changing anything. Shuffling them a few thousand times gives the null\n", "distribution for this data directly, with no assumption about the shape of the\n", "metric.\n", "\n", "Every effect below is one number on [\u22121, +1], and it converts to a percentage:\n", "**+0.40 means the side ahead on that metric won about 70% of those games**, 0.00\n", "means 50% (a coin flip), and \u22120.40 means 30%. Halve the effect and add a half to\n", "convert. The textbook names for these two forms are the rank-biserial effect and\n", "the AUC, and the notebook quotes whichever one reads more plainly.\n", "\n", "Ranks never change when labels move, so the entire null is one array shuffle.\n", "The cell below also checks it against the reference implementation in the wheel,\n", "which does the same test with an explicit loop." ] }, { "cell_type": "code", "id": "cell06", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "def stratified_effect(v, y, strata, rounds=2000, seed=0):\n", " \"\"\"Within-stratum rank effect on [-1, 1], its permutation p, and the null band.\"\"\"\n", " v, y, s = np.asarray(v, float), np.asarray(y).astype(bool), np.asarray(strata)\n", " ok = np.isfinite(v)\n", " v, y, s = v[ok], y[ok], s[ok]\n", "\n", " codes = np.unique(s, return_inverse=True)[1]\n", " order = np.argsort(codes, kind=\"stable\")\n", " codes, v, y = codes[order], v[order], y[order]\n", " edges = np.r_[np.flatnonzero(np.r_[True, np.diff(codes) != 0]), codes.size]\n", "\n", " ranks, keep, weight, offset = np.empty(codes.size), np.zeros(codes.size, bool), 0.0, 0.0\n", " for a, b in zip(edges[:-1], edges[1:]):\n", " n1 = int(y[a:b].sum()); n0 = (b - a) - n1\n", " if n1 and n0: # a one-sided stratum carries no signal\n", " keep[a:b] = True\n", " ranks[a:b] = midranks(v[a:b])\n", " weight += n1 * n0\n", " offset += n1 * (n1 + 1) / 2.0\n", " ranks, y, codes = ranks[keep], y[keep], codes[keep]\n", "\n", " effect = lambda rank_sum: 2.0 * ((rank_sum - offset) / weight) - 1.0\n", " observed = effect(ranks[y].sum())\n", "\n", " rng = np.random.default_rng(seed)\n", " shuffled = np.argsort(codes + rng.random((rounds, codes.size)), axis=1)\n", " null = effect((ranks[shuffled] * y).sum(axis=1))\n", " p = (np.sum(np.abs(null) >= abs(observed)) + 1.0) / (rounds + 1.0)\n", " return float(observed), float(p), np.percentile(null, [2.5, 97.5])\n", "\n", "\n", "# Same answer as the wheel's reference implementation, about thirty times faster.\n", "from ctf_analysis.stats import stratified_permutation_test\n", "ref = stratified_permutation_test(W[\"net_kills\"].to_numpy().astype(float),\n", " W[\"red_won\"].to_numpy(), W[\"matchup\"].to_numpy(),\n", " rounds=2000)\n", "fast = stratified_effect(W[\"net_kills\"], W[\"red_won\"], W[\"matchup\"])\n", "print(f\"early net kills reference {ref[0]:+.4f} (p={ref[1]:.4f})\")\n", "print(f\" here {fast[0]:+.4f} (p={fast[1]:.4f})\")" ] }, { "cell_type": "markdown", "id": "cell07", "metadata": {}, "source": [ "## 2 \u00b7 The answer, in one figure\n", "\n", "Three panels, three findings, and the rest of the notebook is the argument for\n", "them.\n", "\n", "**Left.** Sort the 923 decisive games by how much more damage red dealt than\n", "blue in the opening 25 seconds. Win rate climbs from 7% in the bottom eighth to\n", "91% in the top eighth.\n", "\n", "**Middle.** The same test applied over four windows. Attrition is already worth\n", "something in the first 12.5 seconds and grows. Heart activity sits at exactly\n", "zero early and only becomes \"important\" at whole-game scale, which is what a\n", "statistic looks like when the result is driving it.\n", "\n", "**Right.** The 72 draws pay \u22121 to both sides, and they are visible 25 seconds\n", "in as games where nobody did anything." ] }, { "cell_type": "code", "id": "cell08", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "fig, axes = plt.subplots(1, 3, figsize=(16.5, 4.4))\n", "\n", "# -- left: win rate against the early damage differential --------------------\n", "dmg, won = W[\"damage_dealt\"].to_numpy().astype(float), W[\"red_won\"].to_numpy()\n", "edges = np.unique(np.quantile(dmg, np.linspace(0, 1, 9)))\n", "slot = np.clip(np.digitize(dmg, edges[1:-1]), 0, len(edges) - 2)\n", "rate = np.array([won[slot == k].mean() for k in range(len(edges) - 1)])\n", "mid = np.array([np.median(dmg[slot == k]) for k in range(len(edges) - 1)])\n", "\n", "ax = frame(axes[0], \"y\", \"win rate against the early damage differential\")\n", "ax.axhline(0.5, color=\"#4a4038\", lw=1)\n", "ax.plot(mid, rate, \"-o\", color=BLUE, lw=2, ms=8, mec=PAGE, mew=1.5)\n", "for i, dy in ((0, 12), (len(rate) - 1, -6)):\n", " ax.annotate(f\"{rate[i]:.0%}\", (mid[i], rate[i]), textcoords=\"offset points\",\n", " xytext=(9, dy), ha=\"left\", color=INK)\n", "ax.set_xlabel(\"red minus blue damage dealt, first 25 s (game octiles)\")\n", "ax.set_ylabel(\"red win rate\"); ax.set_ylim(0, 1)\n", "\n", "# -- middle: lever against symptom, across four windows ----------------------\n", "ax = frame(axes[1], \"y\", \"measured early, a lever separates from a symptom\")\n", "TRACK = [(\"damage_dealt\", BLUE, \"damage dealt\"), (\"net_kills\", AQUA, \"net kills\"),\n", " (\"carry_ticks\", ORANGE, \"heart carry time\"), (\"steals\", WARM, \"heart steals\")]\n", "tracked = {}\n", "for metric, colour, label in TRACK:\n", " tracked[metric] = [stratified_effect(window(w)[metric], window(w)[\"red_won\"],\n", " window(w)[\"matchup\"]) for w in WINDOWS]\n", " ax.plot(range(4), [e for e, _, _ in tracked[metric]], \"-o\", color=colour,\n", " lw=2, ms=7, mec=PAGE, mew=1.2)\n", "for (metric, colour, label), ly in zip(\n", " TRACK, spread([tracked[m][-1][0] for m, _, _ in TRACK], 0.075)):\n", " ax.text(3.1, ly, label, color=colour, fontsize=9, va=\"center\")\n", "ax.axhline(0, color=\"#4a4038\", lw=1)\n", "ax.set_xticks(range(4), [LABEL[w] for w in WINDOWS])\n", "ax.set_xlim(-0.15, 4.6); ax.set_ylim(-0.1, 1.0)\n", "ax.set_ylabel(\"within-matchup effect\"); ax.set_xlabel(\"window measured from the opening\")\n", "\n", "# -- right: what a draw looks like at 25 seconds -----------------------------\n", "early = (features.filter(pl.col(\"window\") == \"t600\").group_by(\"episode_id\")\n", " .agg([pl.col(c).sum().alias(c) for c in\n", " (\"kills\", \"damage_dealt\", \"shots\", \"forward_presence\")])\n", " .join(episodes.select(\"episode_id\", \"is_draw\"), on=\"episode_id\"))\n", "drawn = early[\"is_draw\"].to_numpy()\n", "names = [(\"kills\", \"kills\"), (\"damage_dealt\", \"damage\"), (\"shots\", \"shots\"),\n", " (\"forward_presence\", \"forward presence\")]\n", "\n", "ax = frame(axes[2], \"x\", \"at 25 seconds, the draws have already done less\")\n", "for i, (col, label) in enumerate(names):\n", " x = early[col].to_numpy().astype(float)\n", " a, b = np.median(x[drawn]), np.median(x[~drawn])\n", " ax.plot([a / b, 1.0], [i, i], color=\"#4a4038\", lw=1.5, zorder=1)\n", " ax.plot(a / b, i, \"o\", color=ORANGE, ms=9, mec=PAGE, mew=1.2, zorder=2)\n", " ax.plot(1.0, i, \"o\", color=BLUE, ms=9, mec=PAGE, mew=1.2, zorder=2)\n", " ax.annotate(f\"{a:,.0f}\", (a / b, i), textcoords=\"offset points\", xytext=(0, 11),\n", " ha=\"center\", color=MUTED, fontsize=9)\n", " ax.annotate(f\"{b:,.0f}\", (1.0, i), textcoords=\"offset points\", xytext=(0, 11),\n", " ha=\"center\", color=MUTED, fontsize=9)\n", "ax.set_yticks(range(len(names)), [n for _, n in names])\n", "ax.set_xlim(0.45, 1.3); ax.set_ylim(-0.5, len(names) + 0.25)\n", "ax.set_xlabel(\"median for both teams, relative to a decisive game\")\n", "ax.plot([], [], \"o\", color=ORANGE, ms=8, label=\"draws (n=72)\")\n", "ax.plot([], [], \"o\", color=BLUE, ms=8, label=\"decisive (n=923)\")\n", "ax.legend(loc=\"upper left\", labelcolor=MUTED, fontsize=9)\n", "\n", "plt.tight_layout(); plt.show()" ] }, { "cell_type": "markdown", "id": "cell09", "metadata": {}, "source": [ "## 3 \u00b7 Why the matchup has to be held fixed\n", "\n", "The single strongest predictor of who wins is who is playing. The league spans\n", "**4.3 log-odds** from the strongest policy to the weakest, and the win-rate\n", "matrix has visible structure: sort the policies by fitted strength and the upper\n", "triangle is warm.\n", "\n", "That is exactly the confound. A strong policy wins and also shoots more, pushes\n", "further and steals more hearts, so any raw comparison of winners against losers\n", "gives partial credit to everything the strong policy happens to do. Conditioning\n", "on the matchup removes it, because inside one cell of this matrix both policies\n", "and both side assignments are constant." ] }, { "cell_type": "code", "id": "cell10", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "y = W[\"red_won\"].to_numpy().astype(float)\n", "rp, bp = W[\"red_policy\"].to_numpy(), W[\"blue_policy\"].to_numpy()\n", "bt = bradley_terry(np.where(y == 1, rp, bp), np.where(y == 1, bp, rp))\n", "rank = sorted(bt, key=lambda k: -bt[k])\n", "pos = {p: i for i, p in enumerate(rank)}\n", "\n", "grid = np.full((len(rank), len(rank)), np.nan)\n", "for a, b, won in zip(rp, bp, y):\n", " i, j = pos[a], pos[b]\n", " grid[i, j] = won if np.isnan(grid[i, j]) else grid[i, j] + won\n", "counts = np.zeros_like(grid)\n", "for a, b in zip(rp, bp):\n", " counts[pos[a], pos[b]] += 1\n", "with np.errstate(invalid=\"ignore\"):\n", " winrate = np.where(counts > 0, grid / np.where(counts > 0, counts, 1), np.nan)\n", "\n", "fig, axes = plt.subplots(1, 2, figsize=(15, 5.4), width_ratios=[1.25, 1])\n", "\n", "cmap = LinearSegmentedColormap.from_list(\"bwr\", [BLUE, \"#383835\", WARM])\n", "cmap.set_bad(PAGE)\n", "ax = axes[0]\n", "im = ax.imshow(winrate, cmap=cmap, vmin=0, vmax=1)\n", "ax.set_xticks(range(len(rank)), rank, rotation=45, ha=\"right\", fontsize=8)\n", "ax.set_yticks(range(len(rank)), rank, fontsize=8)\n", "ax.set_xlabel(\"blue policy\"); ax.set_ylabel(\"red policy\")\n", "ax.set_title(\"win rate of the row policy, both ordered by fitted strength\",\n", " loc=\"left\", pad=10, fontsize=11)\n", "for s in ax.spines.values():\n", " s.set_visible(False)\n", "cb = fig.colorbar(im, ax=ax, fraction=0.045, pad=0.03)\n", "cb.set_label(\"row policy's win rate\", color=MUTED)\n", "cb.outline.set_visible(False)\n", "\n", "ax = frame(axes[1], \"x\", \"fitted policy strength (Bradley-Terry log-odds)\")\n", "vals = [bt[p] for p in rank][::-1]\n", "ax.barh(range(len(rank)), vals, color=[BLUE if v >= 0 else ORANGE for v in vals],\n", " height=0.62)\n", "ax.set_yticks(range(len(rank)), rank[::-1], fontsize=9)\n", "ax.axvline(0, color=\"#4a4038\", lw=1)\n", "ax.set_xlabel(\"log-odds against the field\")\n", "print(f\"strongest to weakest: {max(bt.values()) - min(bt.values()):.1f} log-odds\")\n", "plt.tight_layout(); plt.show()" ] }, { "cell_type": "markdown", "id": "cell11", "metadata": {}, "source": [ "## 4 \u00b7 What actually wins\n", "\n", "Fifteen metrics, all measured in the opening 25 seconds, all tested the same\n", "way. Read the chart in three moves:\n", "\n", "- **The grey bar** is where the effect would land if the metric had nothing to\n", " do with winning. Any dot inside it is noise.\n", "- **The blue dot** is the effect with the matchup held fixed. Filled dots\n", " survive a Benjamini-Hochberg correction across all fifteen tests.\n", "- **The orange tick** is the same comparison without conditioning. The gap\n", " between tick and dot is the part of the raw effect that was really policy\n", " strength." ] }, { "cell_type": "code", "id": "cell12", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "FAMILIES = [\n", " (\"attrition\", [\"damage_dealt\", \"net_kills\", \"trade_share\", \"deaths\"]),\n", " (\"efficiency\", [\"accuracy\", \"shots\", \"kill_per_damage\", \"heals\"]),\n", " (\"space\", [\"forward_presence\", \"enemy_half_share\", \"unwatched\", \"dispersion\"]),\n", " (\"objective\", [\"carry_ticks\", \"steals\", \"runs_started\"]),\n", "]\n", "sign = np.where(W[\"red_won\"].to_numpy(), 1.0, -1.0)\n", "\n", "rows = []\n", "for family, metrics in FAMILIES:\n", " for m in metrics:\n", " eff, p, band = stratified_effect(W[m], W[\"red_won\"], W[\"matchup\"])\n", " raw = paired_rank_biserial(W[m].to_numpy().astype(float) * sign)\n", " rows.append(dict(family=family, metric=m, eff=eff, p=p, band=band, raw=raw))\n", "for row, q in zip(rows, benjamini_hochberg([r[\"p\"] for r in rows])):\n", " row[\"q\"] = q\n", "\n", "fig, ax = plt.subplots(figsize=(11, 7.5))\n", "frame(ax, \"x\")\n", "ticks, labels, ypos = [], [], 0.0\n", "for family, metrics in FAMILIES:\n", " ypos -= 0.7\n", " ax.text(-0.87, ypos + 0.15, family, color=MUTED, fontsize=10, style=\"italic\")\n", " for m in metrics:\n", " r = next(r for r in rows if r[\"metric\"] == m)\n", " ax.hlines(ypos, r[\"band\"][0], r[\"band\"][1], color=\"#3b2f24\", lw=9, zorder=1)\n", " ax.plot(r[\"raw\"], ypos, \"|\", ms=15, mew=2.2, color=ORANGE, zorder=3)\n", " ax.plot(r[\"eff\"], ypos, \"o\", ms=9, color=BLUE, zorder=4,\n", " mfc=BLUE if r[\"q\"] < 0.05 else PAGE, mec=BLUE, mew=1.8)\n", " ax.text(0.80, ypos, f\"{r['eff']:+.2f}\", color=MUTED, fontsize=9, va=\"center\")\n", " ticks.append(ypos); labels.append(m)\n", " ypos -= 1.0\n", "\n", "ax.axvline(0, color=\"#4a4038\", lw=1)\n", "ax.set_yticks(ticks, labels, fontsize=10)\n", "ax.set_ylim(ypos + 0.4, 0.2); ax.set_xlim(-0.9, 0.9)\n", "ax.set_xlabel(\"effect on winning, first 25 seconds (rank effect on [\u22121, +1])\")\n", "ax.plot([], [], \"o\", color=BLUE, ms=9, label=\"matchup held fixed\")\n", "ax.plot([], [], \"|\", color=ORANGE, ms=13, mew=2.2, label=\"raw winner vs loser\")\n", "ax.plot([], [], \"s\", color=\"#3b2f24\", ms=9, label=\"null band (95% of shuffles)\")\n", "ax.legend(loc=\"lower right\", labelcolor=MUTED, fontsize=9, ncols=3,\n", " bbox_to_anchor=(1.0, -0.16))\n", "ax.set_title(\"fifteen levers, one test\", loc=\"left\", pad=12, fontsize=12)\n", "plt.tight_layout(); plt.show()" ] }, { "cell_type": "markdown", "id": "cell13", "metadata": {}, "source": [ "### Verdict: attrition, and one result that only conditioning finds\n", "\n", "**Damage decides the opening.** Damage dealt in the first 25 seconds is the\n", "largest single effect at **+0.48**, with net kills, trade quality and deaths\n", "avoided just behind it at about **\u00b10.41**. Roughly a third of the raw effect was\n", "policy strength, and the rest survives holding the matchup fixed.\n", "\n", "**Shot volume is a policy trait.** `shots` loses most of its raw effect under\n", "conditioning while `accuracy` keeps its own. Firing more is something particular\n", "policies do; hitting more is something a side does in the game it wins.\n", "\n", "**Forward presence is worth more than it looks.** It is the one metric whose\n", "effect *grows* when the matchup is held fixed, from +0.12 raw to **+0.30**.\n", "Across the league, pushing up the map is not a mark of a stronger policy, and\n", "inside a single pairing the side that pushed is the side that won. Skipping the\n", "conditioning would have called this weak and been wrong.\n", "\n", "**Heart play does nothing in the opening.** Carry time, steals and attempts all\n", "land inside the null band, at **+0.03 to +0.05**, against a design that resolves\n", "effects around 0.2. That is a measured absence rather than a shrug." ] }, { "cell_type": "markdown", "id": "cell14", "metadata": {}, "source": [ "## 5 \u00b7 The heart is a symptom\n", "\n", "Objective play landing at zero is the result most worth attacking, because\n", "capturing the heart is how a third of these games end. So watch what happens to\n", "it as the measurement window widens.\n", "\n", "A lever should be visible early and grow. A statistic that is flat until late\n", "and then explodes is measuring the result: by the time a game is won, the winner\n", "has had the time and the freedom to carry a heart home." ] }, { "cell_type": "code", "id": "cell15", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "fig, axes = plt.subplots(1, 2, figsize=(15, 4.8))\n", "\n", "print(f\"{'':18s}\" + \"\".join(f\"{LABEL[w]:>16s}\" for w in WINDOWS))\n", "ax = frame(axes[0], \"y\", \"effect on winning, by how much of the game you look at\")\n", "for metric, colour, label in TRACK:\n", " eff = [e for e, _, _ in tracked[metric]]\n", " print(f\"{label:18s}\" + \"\".join(f\"{e:+9.3f} (p{p:5.3f})\"\n", " for e, p, _ in tracked[metric]))\n", " ax.plot(range(4), eff, \"-o\", color=colour, lw=2, ms=7, mec=PAGE, mew=1.2)\n", "for (metric, colour, label), ly in zip(\n", " TRACK, spread([tracked[m][-1][0] for m, _, _ in TRACK], 0.075)):\n", " ax.text(3.1, ly, label, color=colour, fontsize=9, va=\"center\")\n", "ax.axhline(0, color=\"#4a4038\", lw=1)\n", "ax.set_xticks(range(4), [LABEL[w] for w in WINDOWS]); ax.set_xlim(-0.15, 4.6)\n", "ax.set_ylabel(\"within-matchup effect\"); ax.set_ylim(-0.08, 1.0)\n", "\n", "ax = frame(axes[1], \"y\", \"the same leakage, with no matchup conditioning at all\")\n", "final = []\n", "for metric, colour, label in TRACK:\n", " auc = []\n", " for w in WINDOWS:\n", " d = window(w); yy = d[\"red_won\"].to_numpy()\n", " v = d[metric].to_numpy().astype(float)\n", " auc.append(rank_auc(v[yy], v[~yy]))\n", " final.append(auc[-1])\n", " ax.plot(range(4), auc, \"-o\", color=colour, lw=2, ms=7, mec=PAGE, mew=1.2)\n", "for (metric, colour, label), ly, value in zip(TRACK, spread(final, 0.035), final):\n", " ax.text(3.12, ly, f\"{value:.3f}\", color=colour, fontsize=9, va=\"center\")\n", "ax.axhline(0.5, color=\"#4a4038\", lw=1)\n", "ax.set_xticks(range(4), [LABEL[w] for w in WINDOWS]); ax.set_xlim(-0.15, 3.9)\n", "ax.set_ylabel(\"chance the winner ranks above the loser\"); ax.set_ylim(0.45, 1.02)\n", "plt.tight_layout(); plt.show()" ] }, { "cell_type": "markdown", "id": "cell16", "metadata": {}, "source": [ "### Verdict: you win the fight, and then you carry\n", "\n", "Heart carry time is worth **exactly zero** in the first 12.5 seconds and\n", "**+0.05** at 25 seconds, then climbs to +0.16 at 50 seconds and **+0.73** over\n", "the whole game. Steals follow the same path. Attrition, by contrast, is already\n", "worth +0.20 and +0.26 in the first 12.5 seconds.\n", "\n", "The right panel is the same story with nothing held fixed: pick a winning team\n", "and a losing team at random and ask how often the winner is ahead. Whole-game\n", "net kills reach **97%**, which reads as a near-perfect predictor and is mostly\n", "the wipe defining itself.\n", "\n", "A policy tuned to maximise steal attempts is optimising the shadow of the\n", "scoreboard." ] }, { "cell_type": "markdown", "id": "cell17", "metadata": {}, "source": [ "## 6 \u00b7 How much is behaviour worth, next to knowing who is playing?\n", "\n", "Effect sizes rank levers one at a time. They do not say what each one *adds*\n", "once the others are known, so the last question is a prediction problem: hold\n", "out whole policy pairs, fit on the rest, and score the held-out games.\n", "\n", "Bradley-Terry strengths are refit inside each training fold, and folds are\n", "grouped by unordered pair so games of the same matchup never straddle a split.\n", "Both precautions matter: without them the model can score well by recognising\n", "the pairing instead of the play." ] }, { "cell_type": "code", "id": "cell18", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "def robust_z(v):\n", " # Median and IQR rather than mean and SD: several of these are zero-inflated\n", " # counts with long tails.\n", " v = np.asarray(v, float)\n", " spread = np.nanpercentile(v, 75) - np.nanpercentile(v, 25)\n", " return np.nan_to_num((v - np.nanmedian(v)) / (spread if spread > 0 else 1.0))\n", "\n", "grp = W[\"pair\"].to_numpy()\n", "keys = np.unique(grp)\n", "fold = np.array([{k: i % 5 for i, k in\n", " enumerate(np.random.default_rng(0).permutation(keys))}[k] for k in grp])\n", "\n", "def cv_auc(metrics=(), strength=True):\n", " cols = [robust_z(W[m].to_numpy()) for m in metrics]\n", " scores = np.empty(y.size)\n", " for f in range(5):\n", " tr, te = fold != f, fold == f\n", " use = list(cols)\n", " if strength:\n", " fit = bradley_terry(np.where(y[tr] == 1, rp[tr], bp[tr]),\n", " np.where(y[tr] == 1, bp[tr], rp[tr]))\n", " use = [np.array([fit.get(a, 0.0) - fit.get(b, 0.0)\n", " for a, b in zip(rp, bp)])] + use\n", " X = np.column_stack(use)\n", " scores[te] = X[te] @ logistic_irls(X[tr], y[tr])\n", " return rank_auc(scores[y == 1], scores[y == 0])\n", "\n", "BEST = [\"net_kills\", \"damage_dealt\", \"kill_per_damage\"]\n", "base = cv_auc()\n", "attr = cv_auc([\"net_kills\"])\n", "bars = [(\"policy strength alone\", base, \"#6b5b49\"),\n", " (\"+ early net kills\", attr, BLUE),\n", " (\"early behaviour alone, no idea who is playing\",\n", " cv_auc(BEST, strength=False), AQUA),\n", " (\"strength + three early behaviours\", cv_auc(BEST), BLUE)]\n", "inc = sorted(((r[\"metric\"], cv_auc([\"net_kills\", r[\"metric\"]]) - attr)\n", " for r in rows if r[\"metric\"] != \"net_kills\"), key=lambda t: -t[1])\n", "\n", "fig, axes = plt.subplots(1, 2, figsize=(15, 4.4), width_ratios=[1.15, 1])\n", "\n", "ax = frame(axes[0], \"x\", \"how often the model ranks the winner above the loser\")\n", "ax.barh(range(len(bars)), [b[1] for b in bars], color=[b[2] for b in bars], height=0.3)\n", "ax.set_yticks(range(len(bars)), [b[0] for b in bars], fontsize=9)\n", "ax.invert_yaxis(); ax.set_xlim(0.5, 0.92); ax.set_ylim(len(bars) - 0.45, -0.55)\n", "for i, (_, v, _) in enumerate(bars):\n", " ax.text(v + 0.004, i, f\"{v:.3f}\", va=\"center\", color=INK, fontsize=10)\n", "ax.set_xlabel(\"share of winner/loser pairs ranked correctly, out of fold \"\n", " \"(0.5 = a coin flip)\")\n", "\n", "ax = frame(axes[1], \"x\", \"what each lever adds on top of strength and net kills\")\n", "ax.barh(range(len(inc)), [d for _, d in inc], height=0.6,\n", " color=[BLUE if d > 0.005 else \"#4a4038\" for _, d in inc])\n", "ax.set_yticks(range(len(inc)), [m for m, _ in inc], fontsize=9)\n", "ax.invert_yaxis(); ax.axvline(0, color=\"#4a4038\", lw=1)\n", "ax.set_xlabel(\"gain in that share\")\n", "plt.tight_layout(); plt.show()" ] }, { "cell_type": "markdown", "id": "cell19", "metadata": {}, "source": [ "### Verdict: three early numbers beat knowing the names\n", "\n", "Knowing only which two policies are playing gets the winner right in **79%** of\n", "held-out comparisons, and that is the bar any behavioural story has to clear.\n", "\n", "Early net kills add **five points** on top of it. Damage dealt adds a further\n", "**1.8 points** even after net kills, because damage is the leading edge of the\n", "same process: chip damage that has not yet become a body is information about\n", "the next ten seconds. Every other lever adds **a fifth of a point or less**,\n", "including accuracy, which had a real standalone effect and turns out to matter\n", "through damage rather than beside it.\n", "\n", "The sharpest version: **three behavioural numbers from the first 25 seconds get\n", "the winner right 84% of the time with no idea who is playing, beating the 79%\n", "you get from both policy names and nothing else.**" ] }, { "cell_type": "markdown", "id": "cell20", "metadata": {}, "source": [ "## 7 \u00b7 Draws are a mutual failure to fight\n", "\n", "The 72 time-limit draws score \u22121 for both sides, so they are a joint loss rather\n", "than a tie. They sit outside everything above, because a winner-versus-loser\n", "design needs a winner. They deserve their own question: what does a game look\n", "like 25 seconds in, when it is heading nowhere?\n", "\n", "Here the comparison is the **total** of both teams, since a draw is a property of\n", "the game rather than of one side." ] }, { "cell_type": "code", "id": "cell21", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "fig, axes = plt.subplots(1, 2, figsize=(15, 4.2))\n", "\n", "ax = frame(axes[0], \"y\", \"total kills by both teams in the first 25 seconds\")\n", "for mask, colour, label in ((drawn, ORANGE, \"games that end in a draw\"),\n", " (~drawn, BLUE, \"games with a winner\")):\n", " x = np.sort(early[\"kills\"].to_numpy().astype(float)[mask])\n", " ax.step(x, np.arange(1, x.size + 1) / x.size, color=colour, lw=2, where=\"post\",\n", " label=f\"{label} (n={mask.sum()})\")\n", "ax.set_xlabel(\"kills, both teams\"); ax.set_ylabel(\"share of games at or below\")\n", "ax.set_xlim(0, 25); ax.legend(loc=\"lower right\", labelcolor=MUTED, fontsize=9)\n", "\n", "ax = frame(axes[1], \"x\", \"how well each early total flags a coming draw\")\n", "auc = [(label, rank_auc(early[col].to_numpy().astype(float)[drawn],\n", " early[col].to_numpy().astype(float)[~drawn]))\n", " for col, label in names]\n", "auc.sort(key=lambda t: -t[1])\n", "ax.barh(range(len(auc)), [0.5 - a for _, a in auc], left=[a for _, a in auc],\n", " color=ORANGE, height=0.3)\n", "ax.set_yticks(range(len(auc)), [n for n, _ in auc], fontsize=9)\n", "ax.invert_yaxis(); ax.set_ylim(len(auc) - 0.45, -0.55)\n", "ax.axvline(0.5, color=\"#4a4038\", lw=1)\n", "for i, (_, a) in enumerate(auc):\n", " ax.text(a - 0.008, i, f\"{a:.2f}\", va=\"center\", ha=\"right\", color=INK, fontsize=10)\n", "ax.set_xlim(0.15, 0.56)\n", "ax.set_xlabel(\"chance a coming draw shows more of it than a decisive game \"\n", " \"(0.5 = no signal)\")\n", "plt.tight_layout(); plt.show()" ] }, { "cell_type": "markdown", "id": "cell22", "metadata": {}, "source": [ "### Verdict: passivity is the most expensive habit in the game\n", "\n", "Every early activity measure is *lower* in games that end in a draw, and the\n", "separation is strong in the direction of doing nothing: a game heading for a\n", "draw has more early kills than a decisive one only **25%** of the time (a median\n", "of 6 against 10), more damage 24% of the time, and more forward presence 21%.\n", "\n", "Two well-matched teams cancelling out would not look like this. A draw is two\n", "teams that never came forward and never engaged, and it is visible before the\n", "first minute is up. The ruleset pays \u22121 to both sides for it." ] }, { "cell_type": "markdown", "id": "cell23", "metadata": {}, "source": [ "## 8 \u00b7 What to take away\n", "\n", "1. **Win the opening exchange.** Damage dealt and net kills in the first 25\n", " seconds are the only levers that move the outcome once you account for who is\n", " playing: five and 1.8 points of predictive accuracy, against a fifth of a\n", " point for everything else.\n", "2. **Damage leads kills.** Damage output predicts beyond the kills it has already\n", " produced, so chip damage is information about the next ten seconds. Accuracy\n", " matters through damage, and adding it separately buys nothing.\n", "3. **Stop optimising the heart.** Steals and carry time have zero predictive\n", " value early and only look important at whole-game scale, which is the\n", " signature of a symptom. You win the fight, then you carry.\n", "4. **Push forward.** The one lever that only shows up under proper conditioning:\n", " within a matchup, the side that held ground further up the map won.\n", "5. **Never stall.** Draws pay \u22121 to both sides and are visible as low activity\n", " in the first 25 seconds.\n", "\n", "### Hand your agent\n", "\n", "> *\"Using `pairs.parquet` from this notebook: my policy's early-window damage\n", "> differential and net kill differential are the only two features that move win\n", "> probability once Bradley-Terry strength is controlled. Profile my policy's\n", "> first 600 ticks against the corpus median for both. If I am behind, work out\n", "> whether it is engagement rate (am I not shooting) or conversion (am I shooting\n", "> and missing), and change the opening behaviour accordingly. Spend no effort on\n", "> heart-steal timing; it has a measured effect of +0.03 against a design that\n", "> resolves 0.2.\"*" ] }, { "cell_type": "markdown", "id": "cell24", "metadata": {}, "source": [ "## 9 \u00b7 Where the numbers come from\n", "\n", "Every table here was baked from replays by\n", "[`ctf-analysis`](https://github.com/Metta-AI/ctf-analysis). A `.replay` holds\n", "only inputs and a per-tick hash, so positions and events exist only after the\n", "engine re-simulates the game on the build that recorded it. See\n", "`AnalyzingCTFGeography.ipynb` \u00a79 for the mining commands.\n", "\n", "```sh\n", "ctf-mine ~/ctf-replays data --engines ~/Dev\n", "python -m ctf_analysis.outcomes --data data --assets assets\n", "```\n", "\n", "| file | one row per | contents |\n", "|---|---|---|\n", "| `data/pairs.parquet` | episode \u00d7 window | **red minus blue** for every metric, plus `red_won`, `matchup`, `pair`, `ending` |\n", "| `data/features.parquet` | episode \u00d7 team \u00d7 window | the same ~25 metrics per team |\n", "| `data/episodes.parquet` | episode | outcome, policies, seed, engine version |\n", "\n", "Windows are `t300`, `t600`, `t1200` (ticks from the opening of play at 24\n", "ticks/s) and `full`. Every metric is signed so that larger is the direction the\n", "hypothesis predicts helps; `deaths` and `damage_taken` are the exceptions and\n", "read the other way.\n", "\n", "**Checks that ran before any of this was believed.** Event-derived per-team kills\n", "match the simulator's own per-seat counters on all 1,990 team-episodes, and\n", "capture counts match `episodes.parquet` exactly. Shuffling outcomes yields\n", "uniform p-values. Red win rate and side bias in three metrics come back null, as\n", "they must on a map that is pixel-exact mirror-symmetric with sides balanced\n", "50/50.\n", "\n", "**What this does not license.** These are predictive contributions, not causal\n", "path coefficients. What the early window buys is temporal precedence: the metric\n", "came first, which rules out the reverse direction without ruling out a common\n", "cause. And policy-level claims have n = 11, not 923." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.12" } }, "nbformat": 4, "nbformat_minor": 5 }