OpenAI Secretly Funded Benchmarking Dataset Linked To o3 Model via @sejournal, @martinibuster

Revelations that OpenAI secretly funded and had entree to the FrontierMath benchmarking dataset are raising concerns astir whether it was utilized to bid its reasoning o3 AI reasoning model, and the validity of the model’s precocious scores.

In summation to accessing the benchmarking dataset, OpenAI funded its creation, a information that was withheld from the mathematicians who contributed to processing FrontierMath. Epoch AI belatedly disclosed OpenAI’s backing lone successful the last insubstantial published connected Arxiv.org, which announced the benchmark. Earlier versions of the insubstantial omitted immoderate notation of OpenAI’s involvement.

Screenshot Of FrontierMath Paper

Closeup Of Acknowledgement

Previous Version Of Paper That Lacked Acknowledgement

OpenAI 03 Model Scored Highly On FrontierMath Benchmark

The quality of OpenAI’s concealed engagement are raising questions astir the precocious scores achieved by  the o3 reasoning AI exemplary and causing disappointment with the FrontierMath project. Epoch AI responded with transparency astir what happened and what they’re doing to cheque if the o3 exemplary was trained with the FrontierMath dataset.

Giving OpenAI entree to the dataset was unexpected due to the fact that the full constituent of it is to  trial AI models but that can’t beryllium done if the models cognize the questions and answers beforehand.

A post successful the r/singularity subreddit expressed this disappointment and cited a papers that claimed that the mathematicians didn’t cognize astir OpenAI’s involvement:

“Frontier Math, the caller cutting-edge mathematics benchmark, is funded by OpenAI. OpenAI allegedly has entree to the problems and solutions. This is disappointing due to the fact that the benchmark was sold to the nationalist arsenic a means to measure frontier models, with enactment from renowned mathematicians. In reality, Epoch AI is gathering datasets for OpenAI. They ne'er disclosed immoderate ties with OpenAI before.”

The Reddit treatment cited a publication that revealed OpenAI’s deeper involvement:

“The mathematicians creating the problems for FrontierMath were not (actively)[2] communicated to astir backing from OpenAI.

…Now Epoch AI oregon OpenAI don’t accidental publically that OpenAI has entree to the exercises oregon answers oregon solutions. I person heard second-hand that OpenAI does person entree to exercises and answers and that they usage them for validation.”

Tamay Besiroglu (LinkedIn Profile), associated manager astatine Epoch AI, acknowledged that OpenAI had entree to the datasets but besides asserted that determination was a “holdout” dataset that OpenAI didn’t person entree to.

He wrote successful the cited document:

“Tamay from Epoch AI here.

We made a mistake successful not being much transparent astir OpenAI’s involvement. We were restricted from disclosing the concern until astir the clip o3 launched, and successful hindsight we should person negotiated harder for the quality to beryllium transparent to the benchmark contributors arsenic soon arsenic possible. Our declaration specifically prevented america from disclosing accusation astir the backing root and the information that OpenAI has information entree to overmuch but not each of the dataset. We ain this mistake and are committed to doing amended successful the future.

Regarding grooming usage: We admit that OpenAI does person entree to a ample fraction of FrontierMath problems and solutions, with the objection of a unseen-by-OpenAI hold-out acceptable that enables america to independently verify exemplary capabilities. However, we person a verbal statement that these materials volition not beryllium utilized successful exemplary training.

OpenAI has besides been afloat supportive of our determination to support a separate, unseen holdout set—an other safeguard to forestall overfitting and guarantee close advancement measurement. From time one, FrontierMath was conceived and presented arsenic an valuation tool, and we judge these arrangements bespeak that purpose. “

More Facts About OpenAI & FrontierMath Revealed

Elliot Glazer (LinkedIn profile/Reddit profile), the pb mathematician astatine Epoch AI confirmed that OpenAI has the dataset and that they were allowed to usage it to measure OpenAI’s o3 ample connection model, which is their adjacent authorities of the creation AI that’s referred to arsenic a reasoning AI model. He offered his sentiment that the precocious scores obtained by the o3 exemplary are “legit” and that Epoch AI is conducting an autarkic valuation to find whether oregon not o3 had entree to the FrontierMath dataset for training, which could formed the model’s precocious scores successful a antithetic light.

He wrote:

“Epoch’s pb mathematician here. Yes, OAI funded this and has the dataset, which allowed them to measure o3 in-house. We haven’t yet independently verified their 25% claim. To bash so, we’re presently processing a hold-out dataset and volition beryllium capable to trial their exemplary without them having immoderate anterior vulnerability to these problems.

My idiosyncratic sentiment is that OAI’s people is legit (i.e., they didn’t bid connected the dataset), and that they person nary inducement to prevarication astir interior benchmarking performances. However, we can’t vouch for them until our autarkic valuation is complete.”

Glazer had besides shared that Epoch AI was going to trial o3 utilizing a “holdout” dataset that OpenAI didn’t person entree to, saying:

“We’re going to measure o3 with OAI having zero anterior vulnerability to the holdout problems. This volition beryllium airtight.”

Another post connected Reddit by Glazer described however the “holdout set” was created:

“We’ll picture the process much intelligibly erstwhile the holdout acceptable eval is really done, but we’re choosing the holdout problems astatine random from a larger acceptable which volition beryllium added to FrontierMath. The accumulation process is different identical to however it’s ever been.”

Waiting For Answers

That’s wherever the play stands until the Epoch AI valuation is completed which volition bespeak whether oregon not OpenAI had trained their AI reasoning exemplary with the dataset oregon lone utilized it for benchmarking it.

Featured Image by Shutterstock/Antonello Marangi