[voidlinux/void-packages] xbps-src: update-check looks into version directories (#9196)

Piotr at Tue, 14 Nov 2017 20:44:10 +0000 (UTC)
Usual path for sources is like https://example.org/pub/${pkgname}-${version}.tar. But some projects, especially KDE, uses per-version directories like https://example.org/pub/${version}/${pkgname}-${version}.tar. This change makes possible to find updates on servers with such path layout. It looks into all higher version directory rather than the highest to not miss anything (ex. kea, phonon-backend-gstreamer). Here is a [list](https://gist.github.com/Chocimier/6d9dc3be795a24f10f3b28e3aa11a796) of updates of packages having `/${version}/` string in distfiles found with updated script. You can view, comment on, or merge this pull request online at: https://github.com/voidlinux/void-packages/pull/9196 -- Commit Summary -- * xbps-src: update-check looks into version directories -- File Changes -- M common/xbps-src/shutils/update_check.sh (21) -- Patch Links -- https://github.com/voidlinux/void-packages/pull/9196.patch https://github.com/voidlinux/void-packages/pull/9196.diff
yopito at Wed, 15 Nov 2017 07:13:02 +0000 (UTC)
nice idea. I was thinking about that too ;) Regarding KDE stuff: could you extend your code to deal with directories that contains only a subpart of `${version}` but which name is still with digits ? Example: package `extra-cmake-modules` (and all KDE framework packages) uses "`distfiles=http://download.kde.org/stable/frameworks/${version%.*}/${pkgname}-${version}.tar.xz`" But it costs much more curl checks (and code) since have to check several versioned-directories ?
Michael Gehring at Wed, 15 Nov 2017 07:21:34 +0000 (UTC)
The problem with this is that it makes a slow process even slower. So maybe this could be made optional by setting some flag in the update file.
Piotr at Wed, 15 Nov 2017 09:50:03 +0000 (UTC)
@yopito: Well, it already looks for numeric-named directories, as update-check work on processed distfiles value, so it can't easily check whether unprocessed value has `${version%.*}`, `${version}`, `5` or anything else. @ebfe: More requests are done only if there is numeric directory, so it is likely that newest packages are in another directory. For non-numeric path, slowdown comes only from additional shell code. Should this option be enabled by default? If not, should we have script that checkes whether flag is enabled for packages having `${version` in distfiles?
Michael Gehring at Wed, 15 Nov 2017 10:01:52 +0000 (UTC)
> @ebfe: More requests are done only if there is numeric directory, so > it is likely that newest packages are in another directory. Ok, i missed that part. > Should this option be enabled by default? If not, should we have > script that checkes whether flag is enabled for packages having > `${version` in distfiles? I'd just add an option the update file (https://github.com/voidlinux/void-packages/blob/master/Manual.md#checking-for-new-upstream-releases) that enable this. Not sure how many false positives you'd actually get with this enabled by default.
Michael Gehring at Wed, 15 Nov 2017 10:09:56 +0000 (UTC)
Regarding slowdown/false positives. Did you try running [void-updates](https://github.com/voidlinux/void-updates) on master vs. this change?
Piotr at Wed, 15 Nov 2017 11:23:56 +0000 (UTC)
Oh, I didn't know that script. I can run for timing, but do you have any idea to automatically spot false positives? ;)
Michael Gehring at Wed, 15 Nov 2017 11:36:52 +0000 (UTC)
> I can run for timing, but do you have any idea to automatically spot false positives? ;) Automatically no. But you get a rough idea by comparing void-updates output with/without .this.
Piotr at Wed, 15 Nov 2017 14:45:18 +0000 (UTC)
@Chocimier pushed 2 commits. b678fc5 xbps-src: update-check version directories speedup a112af9 xbps-src: update-check version directories disabled by default
Piotr at Wed, 15 Nov 2017 14:50:57 +0000 (UTC)
I can't get reliable time comparision for all packages now, sorry. Checking packages selected as described in first post takes about 1.5 more time after modification.
Piotr at Thu, 16 Nov 2017 09:34:20 +0000 (UTC)
Here is a [comparision](https://gist.github.com/Chocimier/cf2ab56e71bb1200a9ba223a94d1d024#file-updates-diff) of scanning with master and updated script. Lower time may be caused by declining traffic in all network during scanning.