Some Ubuntu mirrors are seriously out-of-date, should be ignored when speed benchmark

[martin68]

Hi,
When I am developing apt-smart, an automated Debian/Ubuntu/Linux Mint mirror selection tool, I found that some Ubuntu mirrors (Linux Mint calls them Base mirrors) listed in the Linux Mint Software Sources GUI are seriously out-of-date via Travis CI test logs. For example, in a log several days ago:

| 79 | http://mirror.atlantic.net/ubuntu | Yes | No | 6 weeks behind | 416.7 KB/s |

in another log a month ago:

| 80 | http://mirrors.arpnetworks.com/Ubuntu | Yes | No | 20 weeks behind | 425.78 KB/s |

(apt-smart downloads the Release file of a mirror to benchmark download speed and parse Date value in the Release file to detect the mirror's last update date)

Linux Mint used to check_mirror_up_to_date for all mirrors including Base mirrors, but since this commit
811a6b1 it only checks Linux Mint mirrors' age. @clefebvre , what was the background of this change?

Google search result shows that users complained the issue. Please keep an eye on it. Thanks!

[martin68]

@clefebvre By the way, this line

mintsources/usr/lib/linuxmint/mintSources/mintSources.py

Line 604 in b03bc9e

return True

in def check_mirror_up_to_date(self, url) seems weird, why just return True when (self.default_mirror_age is None or self.default_mirror_age < 2)? The comment above that line is not reasonable IMHO.

    def check_mirror_up_to_date(self, url):
        if (self.default_mirror_age is None or self.default_mirror_age < 2):
            # If the default server was updated recently, the age is irrelevant (it would measure the time between now and the last update)
            return True

[clefebvre]

Thanks Martin, I hate it when this happens... me looking at a commit from the past, wishing I had put more information in the commit message. I can't remember the rationale behind not testing the age of the base repos, and that case you pointed out also... well that also looks questionable. I'll add that to my todo and look into it during the BETA.

[clefebvre]

in def check_mirror_up_to_date(self, url) seems weird, why just return True when (self.default_mirror_age is None or self.default_mirror_age < 2)? The comment above that line is not reasonable IMHO.

OK, after talking with @mtwebster this is clear to me again :) Say we pushed an update 7 days ago, and we're pushing another one right now. All mirrors would be 7 days old. That information isn't relevant to classify them as outdated (even though they technically are). It doesn't mean their update mechanism isn't good, it just means they haven't had the opportunity to sync in what we deem to be a reasonable amount of time.

[martin68]

@clefebvre Thanks for reply, now I understand why you wrote return True. However, now let's continue your example: Say we pushed an update 7 days ago, and we're pushing another one right now, at the very moment, we cannot assume ALL mirrors would be 7 days old, some of them may be much older, say 1 month old, out-of-sync for a long time. We should take these very old mirrors out of users' reach even when (self.default_mirror_age is None or self.default_mirror_age < 2).
IMHO, in case we're pushing another update right now, we can sort the mirror_age of every mirror so that we can get rid of 1-month-old mirror compared with other non-default mirror_age.
Any other person can remind you why didn't you check Base mirrors' mirror_age?

[clefebvre]

It might have been down to performance, we'll test.

Regarding the return True.. well, the tool doesn't know it's been 7 days... for all it knows, it could have been 1 month, so within that time laps (2 days), the test simply isn't relevant.

Note that mintupdate also performs that check, so if the test isn't relevant at the particular moment the user is choosing a mirror, we'll still eventually warn him via the Update Manager.

[gm10]

Where is the problem? You do a HTTP HEAD request both to base and to the mirrors to get the Last-Modified header and you know exactly how out of date the mirrors are. For bonus points (since some servers don't send a Last-Modified) you could instead download the Release or InRelease file and parse the date in there, but probably overkill (I only had 2 online servers without a Last-Modified in the entire list, both Ubuntu mirrors).

FYI @clefebvre, a bunch of (currently) dead Mint mirrors:

E: Mirror unavailable: https://mirrors.layeronline.com/linuxmint-packages/dists/tricia/main/Contents-amd64.gz
E: Mirror unavailable: http://mirror.unlockforus.com/linuxmint/packages/dists/tricia/main/Contents-amd64.gz
E: Mirror unavailable: http://mirror.funkfreundelandshut.de/linuxmint/packages/dists/tricia/main/Contents-amd64.gz
E: Mirror unavailable: http://mirror.vutbr.cz/linuxmint/packages/dists/tricia/main/Contents-amd64.gz
E: Mirror unavailable: http://ftp.linux.edu.lv/mirrors/packages.linuxmint.com/dists/tricia/main/Contents-amd64.gz
E: Mirror unavailable: http://mirrors.serverhost.ro/mint/packages/dists/tricia/main/Contents-amd64.gz
E: Mirror unavailable: https://fourdots.com/mirror/LinuxMint/packages/dists/tricia/main/Contents-amd64.gz

[martin68]

It might have been down to performance, we'll test.

@clefebvre a thought suddenly comes into my mind: since the Ubuntu mirrors are updated very frequently (4 times a day) , self.default_mirror_age will be always < 2, so it will always return True. As a result, if we want to test, the logic has to be changed more than delete self.is_base from this line

mintsources/usr/lib/linuxmint/mintSources/mintSources.py

Line 664 in b03bc9e

if (self.is_base or self.check_mirror_up_to_date("%s/db/version" % url)):

Note that mintupdate also performs that check, so if the test isn't relevant at the particular moment the user is choosing a mirror, we'll still eventually warn him via the Update Manager.

So glad to hear mintupdate also performs that check and warns. Linux Mint deserves its reputation for good user experience, which I really appreciate. However, if I'm not wrong, this line

https://github.com/linuxmint/mintupdate/blob/f1327e15908f51d108b1ded2e3138de1a430aa85/usr/lib/linuxmint/mintUpdate/mintUpdate.py#L936

indicates that it only checks Linux Mint mirror, remaining Ubuntu (Base) mirrors unfortunately unchecked by the Update Manager.

[clefebvre]

Ah yes good point.

OK, so we now remember all the reasons why, and we can clearly do better. The checks look OK to me for the Mint repos, but we need to come up with something different for the base ones as pointed out by @gm10.

We could check if the delta was larger than some arbitrary number, irrelevant of how old the official server is. This would work for Ubuntu repos which are updated very often, and it would also work for old releases if they weren't updated for long periods of time. It wouldn't work in a very niche case, when the gap between two updates is longer than the arbitrary period... but although that's quite relevant for our repos, I'm not sure it is for Ubuntu's.

I didn't look at Debian's btw yet.

[martin68]

@clefebvre Yeah, so IMHO we need 2 different mechanisms to check Ubuntu/Debian mirrors (updated very often) and Linux Mint mirrors (updated rarely).

[alan-wint]

There are also large numbers of geographically distant mirrors shown - why generate traffic for mirrors on the other side of the planet? Also, what about a filter for HTTPS only?