A 1080p60 capture-card stream goes in, controller stick deflections come out. Built as an offline accessibility aid for players who can't reliably execute the small precise inputs Zone hitting demands.
Accessibility tool, offline only. This runs exclusively against CPU difficulty in offline modes (Diamond Dynasty vs CPU, Conquest, Road to the Show, Franchise). It detects and disarms on any online-mode UI element. The LICENSE adds an explicit clause forbidding use in any online or multiplayer context.
Zone hitting in MLB The Show requires fast, precise left-stick corrections as the pitch approaches. For players with motor disabilities that make small stick movements unreliable, that interface is a genuine barrier to playing the game at all. This system closes that gap for offline play.
Three independent processes talk over ZMQ on localhost: ingest.py reads the capture card and publishes raw frames; ball_tracker.py and pci_tracker.py subscribe and emit detection events; bridge.py subscribes to both, computes the aim error, and writes a 20-byte command frame to the Titan Two adapter via Gtuner IV's GCV interface. The Titan Two's GPC script reads that frame and applies stick input to the Xbox controller passthrough.
Ball detection. Each frame is converted to HSV; a range mask isolates ball-colored pixels; connected components are tested against a circularity threshold (contour area / enclosing-circle area >= 0.6) and a radius range. The best candidate per frame is appended to a rolling 0.8-second trail. A static-detection ban zone suppresses false positives from on-screen UI elements: any cluster of detections with spatial spread under 3 pixels gets added to a timed exclusion list.
PCI tracking. The Plate Coverage Indicator is a multi-part shape (brackets + inner + center), so fitting a single contour circle does not work reliably. Instead the tracker masks the configured HSV green range, restricts to the central strike-zone region, and returns the centroid of all matching pixels as the PCI position. Radius is estimated from the 80th-percentile spread of matching pixels. When a pre-captured template exists, cv2.matchTemplate (TM_CCOEFF_NORMED) runs first and the HSV centroid approach is the fallback.
Frame delivery. ingest.py reads the DirectShow capture card via OpenCV's VideoCapture backend and publishes each frame as a ZMQ PUB message with a monotonic timestamp and sequence counter. Mean read latency is 16.3 ms at 1080p60 (p95: 17.4 ms). Subscribers use CONFLATE=1 so each worker always processes the newest frame and never falls behind under load.
def try_fit(trail: deque[Detection], plate_y_px: float) -> tuple[float, float] | None:
"""Return (plate_x_px, eta_ms_from_now) or None if not fittable."""
if len(trail) < MIN_FIT_POINTS:
return None
recent = list(trail)[-FIT_USE_LAST_N:]
ys = np.array([d.y for d in recent], dtype=np.float64)
if ys.max() - ys.min() < MIN_DOWN_PX:
return None
if (ys[-1] - ys[0]) < -30: # trajectory moving up -> not a live pitch
return None
t0 = recent[0].ts_ns
ts = np.array([(d.ts_ns - t0) / 1e9 for d in recent], dtype=np.float64)
xs = np.array([d.x for d in recent], dtype=np.float64)
ay, by, cy = np.polyfit(ts, ys, 2) # quadratic y(t) (gravity)
bx, cx = np.polyfit(ts, xs, 1) # linear x(t) (lateral drift)
disc = by * by - 4 * ay * (cy - plate_y_px)
if disc < 0 or abs(ay) < 1e-6:
return None
sqrt_d = float(np.sqrt(disc))
t_candidates = [(-by + sqrt_d) / (2 * ay), (-by - sqrt_d) / (2 * ay)]
t_cross = min((tc for tc in t_candidates if tc > ts[-1]), default=None)
if t_cross is None:
return None
now_ns = time.time_ns()
plate_ns = t0 + int(t_cross * 1e9)
eta_ms = (plate_ns - now_ns) / 1e6
if eta_ms < 0 or eta_ms > 2000:
return None
plate_x = bx * t_cross + cx
return float(plate_x), float(eta_ms)
Once the rolling trail has at least 4 points spanning enough vertical distance, try_fit fits two independent 1D polynomials against wall-clock time: a degree-2 polynomial over y (capturing the parabolic drop due to gravity rendering and camera perspective) and a degree-1 polynomial over x (lateral drift is close to linear for in-game pitches). The y polynomial is then solved for the plate crossing time by finding the root of ay*t^2 + by*t + (cy - plate_y) = 0 via the quadratic formula, taking the forward root only. The corresponding x position is evaluated from the linear fit at that time.
The result is a (plate_x_px, eta_ms) pair. bridge.py uses the eta to schedule the stick input: it applies the deflection early enough to account for the Titan Two's processing and the controller's USB poll, not at the moment of crossing. Fits that project a crossing more than 2 seconds out or in the past are discarded as noise.
The YOLO ball detector is trained on frames collected from actual gameplay captures using tools in tools/: a frame extractor (yolo_collect_frames.py), an HSV probe to verify ground-truth pixels (hsv_probe.py), a labeling helper (yolo_label_ball.py), and a Ultralytics training wrapper (yolo_split_dataset.py). The classical HSV tracker handles most frames; YOLO is the fallback for difficult lighting (HDR tone-mapping washes out the ball's color range).
Training results are included in the repo. The system is designed to fail safe: capture loss, menu detection, or a hardware F12 keypress all immediately disarm the assist and return full passthrough to the controller.
