Suspected measurement error on the leader board

[sehe]

Disclaimer: I have no stake in this event (I'm not participating).

I strongly suspect there's a measurement error with the current top leader-board position:

anassajaanan	16106	14465	C	16.11 sec (≈ 16.11 sec)

Aside from the fact that it is an outlier (which could just be /awesome/ coding, local comparison of some of the other top contenders in C++ consistently shows it slower.

Can you please check that the test configuration was the same?

Perhaps unrelated, I suspect that that the anassajaanan has a bug in the number parsing code, leading to deviating totals for the "minimal cost city" in various test sets. Though it's possible that the actual test-set used on the server happens not to trigger the bug path, it's seems more likely something is going wrong with the measurements.

I'd like to also point out that I think anassajaanan's submission is the only one to use fork(1) which means it has child-processes. It make a little bit of sense this is throwing the judging scripts off?

[sehe]

Could webNeat's submission be affected by the same problem? This is more complicated since there's both a php and a c++ solution file.

The C++ solution isn't particularly fast (although with elegant code, runs about 2x as long as sqrt-minus-one's solution).

However, the PHP solution (not tested) forks (using pcntl_fork). As far as I can see https://github.com/geeksblabla/blarun/blob/5d81dacacd26056523c5a69b757d9059f85223d0/src/main.rs all solution files might be looped, conflating measurements from both.

If the PHP measurement is affected by a similar confusion due to it forking this could well mess with the standings. All in all, I think the measurements should be repeatable, and this is what I see with a random selection of CPP solutions on my machine:

submission	time
ibrahata8	21.373s
webNeat	41.005s
sqrt-minus-one	21.098s
anassajaanan	29.082s	(note: deviating value in first line of output.txt)

Of course those are single runs on an old machine, but I'd really expect them to at least resemble the order on the leaderboard roughly?

[aboullaite]

Calling the expert here @chermehdi

[sehe]

Oh lol. I see now (here #217 (comment)) that I was a little bit too trusting then

The approach was a compromise between having something that gives rough numbers and not having to wait ages for a verdict as we get more submissions, but more thought will be put into deciding who is the winner. (dropping the cache is something that we planned to do in the final run once the submissions are closed).

Regardless, the top entry is still inexplicably much faster than on any measurment I did - so something else must be at play as well.

[chermehdi]

I'm referring to the solution file being set in a loop: https://github.com/geeksblabla/blarun/blob/5d81dacacd26056523c5a69b757d9059f85223d0/src/main.rs#L1013 At least it makes it hard for me to establish whether solution.cpp or solution.php will be run. Possibly both?

I still don't see the issue here, the loop is supposed to run both one after another.

See my comment on Samir's other discussion thread, the current leaderboard is mainly for showing how is participating and who is "potentially" wining, we will determine the winner with more rigorous testing once we stop receiving submissions and have more server time to run more extensive tests.

Also the current measurements aren't made to be accurate, they are mainly there to be fair.

For reference, Manually testing (after droping the cache).

Anas:

$ time ./anas

real	0m37.664s
user	2m5.269s
sys	0m9.547s

Sqrt (another cpp solution)

$ time ./sqrt

real	1m18.825s
user	1m39.797s
sys	0m33.245s

Realtime roughly matches what we see in the scoreboard (magnitude wise).

[sehe]

@chermehdi Weird how I get consistently contradicting results. I can only assume that your test server has Very Slow(TM) disk access, which is going to be even more visible in your cold-cache comparison.

So that's

name	time
anas	29.576s
sqrt	24.935s

The screen recording (done with scriptreplay so it doesn't influence the measurements!) shows the exact methodology. I'm uploading my 22GiB input.txt (xz-compressed) in a little bit in case you want to see it (I have several with consistent output, but I'm uploading the one used in this screen recording). (I'm running linux 5.19 on ZFS by the way, which is not fast by any standard, the machine is literally 10 years old)

I also note that the outputs are not the same. This is in itself a red flag that I've been raising from the very start.

I understand your point about fair vs accurate.

I don't mind the approach of averaging n+2 runs removing the extremes (which need not be outliers, statistically). That makes a lot of sense and may even remove the need for cold caches (since the effect will be evened out by the first run which likely is the lower extreme when cold)

Although it's fair that a lot of the challenge here is to manage I/O wisely, I wonder whether it's fair when the leader-board can flip around just choosing a different HDD or filesystem.

[chermehdi]

I also note that the outputs are not the same. This is in itself a red flag that I've been raising from the very start.

That's one sad thing, we tested with a single output.txt if your solution is wrong but gets it right for that specific file it will be considered correct sadly (it will cost too much to change right now).

again, this might not be the best way to measure the time, but I don't think we will change it unless we have a better (fairer) way of doing this (When running, we keep getting consistent results from both solutions so I believe that's good enough for us).

What this actually means is that one solution is faster than the other on the particular server that we run on. (this might not be universally true, but I don't think the stakes here require more than this, the goal of the challenge is mainly for fun, and leaderboard is just a way of keeping score, everyone that tried is a winner IMHO).

If you have a suggestion on how to better measure this, please send us a PR on the blarun repository, would happily review :D

[sehe]

If you have a suggestion on how to better measure this, please send us a PR on the blarun repository, would happily review :D

Opened geeksblabla/blarun#11

[sehe]

I still don't see the issue here, the loop is supposed to run both one after another.

Fair enough. I never realized that submissions could contain multiple solutions. That's why I phrased it as a question:
"Possibly both?" - I never said it was a problem, I just wasn't sure the judging script was prepared to emit separate judgement results for separate solutions.

I just uploaded the input.txt I used in my measurements above: https://stackoverflow-sehe.s3.amazonaws.com/blanat/input.txt.xz (4.4GiB download). Here's the output using another input.txt newly generated from gen.py:

At least here the results are identical :) But the timings still completely contradict the leaderboard. In fact, I had a solution that just printed

Acorn_Squash 1.00
Apple 1.00
Apricot 1.00
Artichoke 1.00
Asparagus 1.00

Because statistically it's likely to occur (I haven't done the math, but a billion rows is a lot given 94x101 possible combinations: that's ~105329 lines for each prod x city. Each line a 1/9901 chance to have price = 1.00, meaning pretty close to 100% chance those happening for each prod x city combination?). It's possible people used this heuristic :)

I'm willing to give blarun a go. I completely anticipate the same results, but one never knows. Is it easy to setup? It look like it's designed to be running as a systemd service. I hope that's optional :)

[sehe]

@samir- Sure! I even figured out how to make blarun run locally, and can show you the video summary (with some timelapses): http://stackoverflow-sehe.s3.amazonaws.com/4f9c4daa-87d2-4438-bea4-4537dedd67b0/out-2024-02-28_22.09.55.mp4

Here's full video (using script-replay with speedup x2 and max-delay 1s) so you (or @chermehdi ) can inspect every single debug log statement in realtime (using the log timestamps), just in case:

out-2024-02-28_22.27.33.mp4

And that log in text form: https://gist.github.com/sehe/8d9194747740f0bd43c823904c9638f3

username	avg	median	lang
sehelocal-sqrt-minus-one	17363	16862	cpp
sehelocal-samir-	17844	17980	cpp
sehelocal-anassajaanan	22452	22387	c
sehelocal-webNeat	39956	39367	cpp

So, @chermehdi my hunch that something might be wrong with the measurement has been dispelled. The outcome cleanly reflects standalone runs. However, something is definitely off with the server config itself. I can hardly think of reasons, but it might have to do with swappiness (causing large process spaces to use disk IO) or, even more far-fetched, does the input actually live on a network volume (e.g. via automountd or similar?)

I can provide you with all the machine specs, or we can rent an AWS ec2 instance to get independent input. It seems important enough, because even though you personally think participating is winning (I AGREE, and it's why I don't actually participate), when there's prize money on the line you have a responsibility to make sure the judging system is actually fair.

[sehe]

@ibrataha8 added your entry back in the mix on my local machine just for reference

username	avg	median	lang
sehelocal-sqrt-minus-one	17363	16862	cpp
sehelocal-samir-	17844	17980	cpp
sehelocal-anassajaanan	22452	22387	c
sehelocal-ibrataha8	25170	25125	cpp
sehelocal-webNeat	39956	39367	cpp

[aouakki]

I'm also suspecting an issue with the runner machine type, I did some experiment runs of my golang solution and I got a lot of variance in the result between multiple runs.
I mentioned that here #208

[aboullaite]

Thank you all for your valuable feedback. As we stated in multiple occasions, this has been a learning journey for all of us and we are already aware of many issues with the curent evaluation process.

We have tried to be as transparent as possible and as fair as possible. You uncovered a lot of good points, that would take a bit of time for us to fix. We already invested a lot of time in this challenge and it took more effort that we initially planned. Full disclosure I see it a bit hard for us doing the measurements manually, or updating the runner & reconfiguring a whole new VM.

This was meant to be a fun community initiative, and community goes both ways. I am happy for you guys taking the lead on fixing the different issues mentioned and updating the leaderboard in a transparent, fair and collaborative way. If someone is up for it please reach out :)

Thank you all

[sehe]

We have tried to be as transparent as possible and as fair as possible.

Much appreciated. It's awesome that blarun is open, gen.py is open, the submissions are open.

[ibrataha8]

@sehe I did not understand how they calculate. They have a huge discrepancy in the results. I also tried it on my device and it gave me a discrepancy with the current results that they have in leaderboard

[sehe]

I understand full well how they calculate. Their blarun tool is transparent, and I even used it produce these independent results.

The question then remains, why is their server so inconsistent/different so that it completely favors some implementations.

[ibrataha8]

Even I didn't understand it, maybe there was something wrong while it was running at that time