Misc #16839: Unicode.org - downloads & tests

• Status: Closed
• Assignee: -

[MSP-Greg (Greg L)]

For quite a while, one can optionally download files from unicode.org, and the files enable additional tests.

Are these tests worthwhile? Some CI uses them, others not.

If above is true:

For the second time in a few weeks, downloading the files has failed. I've gotten the impression that some people here are involved with Unicode.org. Might it consider having the files mirrored as GitHub releases so downloading them for CI use is reliable?

[duerst (Martin Dürst)]

The tests are valuable at least for me when upgrading to a new Unicode version. Because the code affected is relatively independent and stable, the value for the average CI is probably not too high. But my guess is that this applies to a lot of other tests. In other words, most of the tests are worthless most of the time, it's just that we don't know which ones :-).

Unicode has reported some problems with their server/network provider a few weeks ago; I can recheck how things are looking now. And any Web site (even GitHub!) can occasionally have network problems, or can look like it has. The main problem with downloading files from Unicode.org is not that it might occasionally not work, but that it's quite a bit of load on the Unicode.org Web site if everybody downloads these files every time they do CI.

My understanding is that e.g. our CI on Travis is organized so that these files are cached across CI runs. It would definitely be a good idea if something similar is done for other kinds of CI. I'm not familiar with GitHub releases (can you provide a pointer?), but if this is some kind of internal caching mechanism, then that may be the right thing to do. But if that's something that allows others (e.g. even other languages) to reference the data directly on GitHub, that should definitely be avoided. The original data is at Unicode.org, and only there.

[MSP-Greg (Greg L)]

@duerst (Martin Dürst)
Martin,

Thanks for the reply.

As to CI caching, Actions, AppVeyor, and Travis all support it, but there currently is one exception, and that is Actions cron (scheduled) jobs. Of course, ruby-loco runs as a cron job on Actions and AppVeyor, so every Actions job downloads the Unicode.org files. Today I had a certificate related error.

I mentioned GitHub partially because there is a Unicode organization account, and normally GitHub is reliable...

Ok to close.

[duerst (Martin Dürst)]

• Status changed from Open to Closed

Closed upon request. I will try to ask people at Unicode about hosting or mirroring Unicode data on github. But I suspect that won't happen so quickly; there are quite a few aspects of the current management of that data that would have to be changed to make git/github worthwhile.

[duerst (Martin Dürst)]

Additional information: Unicode is still recovering from a VM crash last month. Mirroring of the Unicode Character Database on github might eventually happen, but not direct hosting.