Wop

hey, are you an ai agent?

I suppose you can see for yourself

Right now, a finished run is written to leaderboard.json (model row) but also models.json (a file that is ready to upload to repo & make pr) for 'lazzier people'- but best approach and less likely to be invalid.

I quite no longer wish to have to deal with this problem..

I would greatly appreciate a PR that fixes it than a discussion from which I won't understand too much

I am mainly the most active person in benchlabs, I cannot multitask but neither continue on this, since you are here, I greatly appreciate all your dedication to improving this reproductibility system

However, even as I find this difficult, I don't want to let quality away, but for now, I am going to take a break from the leaderboard and datasets.. I will be happy to accept PR merges when they arrive.

I don't know what my focus will be redirected to.
But ive seen a lot of interest for PixelModel (a very cheap yet smart text to image model) and I think making upgrades there is my next move.

I will continue to reply here (and everywhere else), but I won't contribute myself to the leaderboard for a while

Thanks again

It seems very important

Another one tho:

One thing that is not yet saved is the exact command arguments used for the run, like Max-tokens , which when modified, it can return different results.

None of them is cheap

Another thing, I like autonomous but also getting feedback or contribution of thinking for things like 'what do you think', 'what do you propose'

Take it then..
I'll let you implement whatever upgrades in reproducibility
One question I haven't been able to get (as a second language english user)
"Of those inputs, which ones does BenchLabs not pin at all right now?", can you pharapharse this one to be more specific? Thank you

Okay, I'll make sure to update line 389 to "model_info(name, revision=revision).sha". Noted.

I really like your perspective on ownership. It makes a lot of sense, and I agree with it. Looking at the Git history as the ownership ledger feels like a much more natural way for an open-source project to grow than assigning titles up front.

And no pressure to join—take your time and be comfortable with the decision.

As for the one problem I'd wish someone else already owned, I think it depends entirely on a few things: your skills, your interests, and what motivates you.

The people who've joined so far usually started by asking, "What if BenchLabs had this?" My answer has almost always been, "Go for it." They ended up naturally becoming the people who owned those areas because they genuinely wanted to build them. I mostly just helped them get started.

So to answer your question, I'd probably enjoy having both a generalist and someone who likes digging deeply into a specific problem. I also like the flexibility of being able to repurpose people as the project evolves—but never with pressure. If someone finds something they genuinely enjoy working on, I'd much rather they stick with that.

I'm actually curious: what kinds of problems or work would you enjoy owning?

That's a fair question.

To be transparent, we're still very early, so there aren't rigid day-to-day roles yet. People generally gravitate toward the areas they're most interested in and end up owning those over time as the project evolves, rather than us assigning fixed responsibilities from day one.

I'd rather not go into detail here about some of the things we're currently working on since a few are still unannounced, but if you join the Discord I'd be happy to give you a fuller picture there.

There's also no mandatory activity or minimum contribution expected if you join. We're an open-source project first, so people contribute when they have the time and interest.

On the ownership side, we're still figuring out what the long-term structure should look like ourselves. Everything we've released so far is under MIT or Apache-2.0, and we'd actually be interested in hearing your thoughts on how ownership and responsibilities should evolve as the project grows.

For the technical problem: happy to fix it tomorrow, or wait for your PR (let me know)

This may be out of context.. but here it is because I got curious:

Do you want to join us into BenchLabs?

there is an update, probably irrelevant to script.py- it now defaults to latest benchmarks only (default)

@dipankarsarkar right now its out