README.md · Pitch Tracker CV

121 lines · markdown

# pitch-tracker-cv
 
A real-time computer-vision aim-assist for **offline, single-player** MLB The Show 26.
Built as an accessibility aid for players with motor disabilities who cannot
reliably execute the small left-stick corrections that the game's Zone hitting
interface demands, or the precise gesture inputs that Pinpoint pitching requires.
 
The system reads the game through a capture card, runs a vision pipeline that
tracks the ball and the PCI (Plate Coverage Indicator), predicts where the
pitch will cross the plate, and drives a Titan Two adapter to nudge the
controller stick so that the user can play vs-CPU modes they otherwise could
not.
 
This is not a cheat. The tool only operates in offline modes (Diamond Dynasty
vs CPU, Conquest, Moments, Showdown offline, Road to the Show, March to October,
Franchise). It refuses to arm if any online UI element is on screen. See
[ACCESSIBILITY_STATEMENT.md](ACCESSIBILITY_STATEMENT.md) and [NOTICE.md](NOTICE.md).
 
## What it is, in one sentence
 
A 1080p60 capture-card video stream goes in, controller stick deflections come
out, with a YOLO-trained ball detector and classical CV PCI tracker in between.
 
## Architecture
 
```
+----------------+        +-----------+      +--------------+      +-----------+
| Capture card   |  USB3  | ingest.py |  ZMQ | ball_tracker |  ZMQ | bridge.py |
| (1920x1080@60) +------->|           +----->|              +----->|           |
+----------------+        +-----------+      | pci_tracker  |      +-----+-----+
                                             +--------------+            |
                                                                         v
                                                                +-----------------+
                                                                | Titan Two (GPC) |
                                                                +--------+--------+
                                                                         |
                                                                         v
                                                                +-----------------+
                                                                | Xbox controller |
                                                                +-----------------+
```
 
Three independent processes connected by ZMQ on localhost:
 
1. **`capture/ingest.py`** opens the capture card via OpenCV/DirectShow and
   publishes raw frames as multipart `[meta_json, jpeg]` messages on
   `tcp://127.0.0.1:5555`. Measured at 59.88 FPS @ 1920x1080 with mean read
   latency 16.3 ms (p95 17.4 ms) on the reference hardware.
2. **`cv/ball_tracker.py`** subscribes to capture frames, applies an HSV +
   circularity filter (or an ONNX YOLO detector for harder lighting), fits a
   2D parabolic trajectory across a rolling 0.8 s window, and emits
   `pitch_pred` events with predicted plate-crossing `(x, eta_ms)`.
3. **`cv/pci_tracker.py`** template-matches or HSV-segments the PCI circle
   and emits `pci_track` events with the current PCI position.
4. **`io_titan/bridge.py`** subscribes to both tracker streams, runs the
   decision logic (aim error → stick deflection, residual error + count →
   swing decision), packs a 20-byte fixed-layout command frame, and writes
   it to the Titan Two via Gtuner IV's GCV (Game Computer Vision) interface.
5. **`io_titan/mlb26_bridge.gpc`** is the GPC script that runs on the Titan
   Two itself, reads the command frame from GCV memory, and applies the
   stick / button input to the Xbox controller passthrough.
 
## Hardware required
 
- Xbox Series X/S
- Monster 4K USB 3.0 capture card (or any 1080p60 DirectShow-compatible card)
- Titan Two adapter with Gtuner IV installed
- Wired Xbox controller
- Windows PC for the vision pipeline (Python 3.11+)
 
## Quick start
 
```bash
python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt
 
python -m capture.diagnose
python -m capture.ingest --smoke
 
python -m cv.ball_tracker
python -m cv.pci_tracker
python -m tools.viewer
```
 
YOLO ball detection (optional, more robust under HDR tone-mapping):
 
```bash
pip install -r requirements-yolo.txt
python -m tools.yolo_train_ball --data configs/ball_yolo.yaml
python -m io_titan.mlb26_gcv_yolo
```
 
## Project layout
 
```
capture/         capture card ingest, device probe
cv/              ball tracker (HSV + parabolic fit), PCI tracker
io_titan/        Python + GPC bridge to Titan Two
configs/         runtime tunables, YOLO data config, templates
tools/           one-shot utilities: viewer, snapshot, hsv_probe,
                 detect_on_frame, YOLO frame collection/labeling/training
docs/            architecture notes, ball-detector reference
```
 
## Safety model
 
The pipeline is designed to fail safe:
 
- `safety.online_mode_abort` in `configs/runtime.yaml` instructs the bridge
  to disarm if any online-mode UI element is detected.
- `safety.abort_on_menu_detected` disarms when the game leaves play.
- `safety.abort_on_capture_loss_ms` disarms if the capture stream stalls.
- A hardware deadman hotkey (default `F12`) forces full passthrough.
- The Titan Two script always passes raw controller input through. Aim assist
  is additive only and clamped to small stick deflections.
 
## License
 
MIT, with an explicit additional clause forbidding use in any online or
multiplayer context. See [LICENSE](LICENSE).

1	# pitch-tracker-cv
2
3	A real-time computer-vision aim-assist for offline, single-player MLB The Show 26.
4	Built as an accessibility aid for players with motor disabilities who cannot
5	reliably execute the small left-stick corrections that the game's Zone hitting
6	interface demands, or the precise gesture inputs that Pinpoint pitching requires.
7
8	The system reads the game through a capture card, runs a vision pipeline that
9	tracks the ball and the PCI (Plate Coverage Indicator), predicts where the
10	pitch will cross the plate, and drives a Titan Two adapter to nudge the
11	controller stick so that the user can play vs-CPU modes they otherwise could
12	not.
13
14	This is not a cheat. The tool only operates in offline modes (Diamond Dynasty
15	vs CPU, Conquest, Moments, Showdown offline, Road to the Show, March to October,
16	Franchise). It refuses to arm if any online UI element is on screen. See
17	[ACCESSIBILITY_STATEMENT.md](ACCESSIBILITY_STATEMENT.md) and [NOTICE.md](NOTICE.md).
18
19	## What it is, in one sentence
20
21	A 1080p60 capture-card video stream goes in, controller stick deflections come
22	out, with a YOLO-trained ball detector and classical CV PCI tracker in between.
23
24	## Architecture
25
26	```
27	+----------------+ +-----------+ +--------------+ +-----------+
28	\| Capture card \| USB3 \| ingest.py \| ZMQ \| ball_tracker \| ZMQ \| bridge.py \|
29	\| (1920x1080@60) +------->\| +----->\| +----->\| \|
30	+----------------+ +-----------+ \| pci_tracker \| +-----+-----+
31	+--------------+ \|
32	v
33	+-----------------+
34	\| Titan Two (GPC) \|
35	+--------+--------+
36	\|
37	v
38	+-----------------+
39	\| Xbox controller \|
40	+-----------------+
41	```
42
43	Three independent processes connected by ZMQ on localhost:
44
45	1. `capture/ingest.py` opens the capture card via OpenCV/DirectShow and
46	publishes raw frames as multipart `[meta_json, jpeg]` messages on
47	`tcp://127.0.0.1:5555`. Measured at 59.88 FPS @ 1920x1080 with mean read
48	latency 16.3 ms (p95 17.4 ms) on the reference hardware.
49	2. `cv/ball_tracker.py` subscribes to capture frames, applies an HSV +
50	circularity filter (or an ONNX YOLO detector for harder lighting), fits a
51	2D parabolic trajectory across a rolling 0.8 s window, and emits
52	`pitch_pred` events with predicted plate-crossing `(x, eta_ms)`.
53	3. `cv/pci_tracker.py` template-matches or HSV-segments the PCI circle
54	and emits `pci_track` events with the current PCI position.
55	4. `io_titan/bridge.py` subscribes to both tracker streams, runs the
56	decision logic (aim error → stick deflection, residual error + count →
57	swing decision), packs a 20-byte fixed-layout command frame, and writes
58	it to the Titan Two via Gtuner IV's GCV (Game Computer Vision) interface.
59	5. `io_titan/mlb26_bridge.gpc` is the GPC script that runs on the Titan
60	Two itself, reads the command frame from GCV memory, and applies the
61	stick / button input to the Xbox controller passthrough.
62
63	## Hardware required
64
65	- Xbox Series X/S
66	- Monster 4K USB 3.0 capture card (or any 1080p60 DirectShow-compatible card)
67	- Titan Two adapter with Gtuner IV installed
68	- Wired Xbox controller
69	- Windows PC for the vision pipeline (Python 3.11+)
70
71	## Quick start
72
73	```bash
74	python -m venv .venv
75	.venv\Scripts\activate
76	pip install -r requirements.txt
77
78	python -m capture.diagnose
79	python -m capture.ingest --smoke
80
81	python -m cv.ball_tracker
82	python -m cv.pci_tracker
83	python -m tools.viewer
84	```
85
86	YOLO ball detection (optional, more robust under HDR tone-mapping):
87
88	```bash
89	pip install -r requirements-yolo.txt
90	python -m tools.yolo_train_ball --data configs/ball_yolo.yaml
91	python -m io_titan.mlb26_gcv_yolo
92	```
93
94	## Project layout
95
96	```
97	capture/ capture card ingest, device probe
98	cv/ ball tracker (HSV + parabolic fit), PCI tracker
99	io_titan/ Python + GPC bridge to Titan Two
100	configs/ runtime tunables, YOLO data config, templates
101	tools/ one-shot utilities: viewer, snapshot, hsv_probe,
102	detect_on_frame, YOLO frame collection/labeling/training
103	docs/ architecture notes, ball-detector reference
104	```
105
106	## Safety model
107
108	The pipeline is designed to fail safe:
109
110	- `safety.online_mode_abort` in `configs/runtime.yaml` instructs the bridge
111	to disarm if any online-mode UI element is detected.
112	- `safety.abort_on_menu_detected` disarms when the game leaves play.
113	- `safety.abort_on_capture_loss_ms` disarms if the capture stream stalls.
114	- A hardware deadman hotkey (default `F12`) forces full passthrough.
115	- The Titan Two script always passes raw controller input through. Aim assist
116	is additive only and clamped to small stick deflections.
117
118	## License
119
120	MIT, with an explicit additional clause forbidding use in any online or
121	multiplayer context. See [LICENSE](LICENSE).