Parameter Golf

A budget reproduction of an SP4096 Parameter Golf record under the 16 MB cap

Challenge entry · OpenAI Parameter Golf · Reproduction and experiments · May 2026

Overview

OpenAI's Parameter Golf challenge asks for the best small language model whose submission fits in 16 MB. I reproduced Kevin Clark's accepted SP4096 record on a small compute budget and tested one change on top of it.

What I did

I ran the record training script on a single H100 with a 60 minute cap and 86 train shards, instead of the official 10 minutes on 8 H100s. I added an artifact fit guard: if the compressed quantized model lands slightly over the 16,000,000 byte cap, it repeatedly coarsens a selected low impact tensor until it fits, which moved the score by only about 0.00012 in the best run.

Results

The baseline reproduction reached 1.11043 validation bits per byte with sliding window evaluation. Setting QK_GAIN_INIT=4.5 improved that to 1.10743, with a total artifact of 15,987,195 bytes.

This is a single seed, non-record reproduction, not a new leaderboard result. The record folder and its pull request to the challenge repo are public, and the pull request is still open.