Mirrors #38

Open
opened 1 year ago by alissonlauffer · 29 comments

Providing a rsync URL will allow creating new mirrors of the repo. I may provide one of them in the future.

Providing a rsync URL will allow creating new mirrors of the repo. I **may** provide one of them in the future.
anonfunc added the
enhancement
label 1 year ago

I would like to host a mirror too if possible!

I would like to host a mirror too if possible!
Owner

I'll see if I get that setup later today.

I'll see if I get that setup later today.

Thanks! I'm looking forward to it.

Thanks! I'm looking forward to it.
Owner

Mirroring via rsync

source: rsync://alhp.harting.dev/alhp
lastupdate: https://alhp.harting.dev/lastupdate
expected size: around 30GB per march (e.g. x86-64-v3) you want to sync.

I highly recommend using this script from Archlinux as base for a sync service. It checks if a sync needs to be done (by comparing lastupdate before syncing). It's also worth to read comments.

You are free to mirror only one march or exclude /logs, see line 90 of said script above.

If you have setup a mirror, let me know here in this issue (if possible with the location of the mirror; country/region is enough), then I'll add your mirror to the README for users to choose from.

### Mirroring via rsync **source**: `rsync://alhp.harting.dev/alhp` **lastupdate**: `https://alhp.harting.dev/lastupdate` **expected size**: around 30GB per `march` (e.g. *x86-64-v3*) you want to sync. I highly recommend using [this script from Archlinux](https://gitlab.archlinux.org/archlinux/infrastructure/-/blob/master/roles/syncrepo/files/syncrepo-template.sh) as base for a sync service. It checks if a sync needs to be done (by comparing *lastupdate* before syncing). It's also worth to read comments. You are free to mirror only one march or exclude `/logs`, [see line 90](https://gitlab.archlinux.org/archlinux/infrastructure/-/blob/a4a8a640d821acdb95e72770ea0632beffb28df9/roles/syncrepo/files/syncrepo-template.sh#L90) of said script above. If you have setup a mirror, let me know here in this issue (if possible with the location of the mirror; country/region is enough), then I'll add your mirror to the README for users to choose from.

ooo Thank you so much! I'll set this up ASAP

ooo Thank you so much! I'll set this up ASAP

Here's my mirror: https://www.gardling.com/alhp it syncs every 5 minutes (I did some modifications to the script you referenced).

Here's my mirror: `https://www.gardling.com/alhp` it syncs every 5 minutes (I did some modifications to the script you referenced).

If more people start mirroring this. Maybe you can setup a ahlp-mirrorlist package just like the chaoticaur unofficial repo with their chaoticaur-mirrorlist package. Just a thought 👍

If more people start mirroring this. Maybe you can setup a `ahlp-mirrorlist` package just like the `chaoticaur` unofficial repo with their `chaoticaur-mirrorlist` package. Just a thought 👍

If more people start mirroring this. Maybe you can setup a ahlp-mirrorlist package just like the chaoticaur unofficial repo with their chaoticaur-mirrorlist package. Just a thought 👍

I've had been thinking, I don't know if the build server is the same as the repo (I presume so), if we had two mirrors, we could use primarily those mirrors and offload the build server.

I don't know if it is really useful but what your thoughts?

> If more people start mirroring this. Maybe you can setup a `ahlp-mirrorlist` package just like the `chaoticaur` unofficial repo with their `chaoticaur-mirrorlist` package. Just a thought 👍 I've had been thinking, I don't know if the build server is the same as the repo (I presume so), if we had two mirrors, we could use primarily those mirrors and offload the build server. I don't know if it is really useful but what your thoughts?

I wouldn't be able to host a x86-64-v3 build server, but I can host a x86-64-v2 one (like I mentioned here: anonfunc/ALHP.GO#17). But yea, that is a good idea. That other server would have to be run by a trusted person as they're also now compiling the packages themselves too, so those packages can't just be verified by a checksum or something.

But I don't know how that would be organized.

I wouldn't be able to host a x86-64-v3 build server, but I can host a x86-64-v2 one (like I mentioned here: https://git.harting.dev/anonfunc/ALHP.GO/issues/17#issuecomment-746). But yea, that is a good idea. That other server would have to be run by a trusted person as they're also now compiling the packages themselves too, so those packages can't just be verified by a checksum or something. But I don't know how that would be organized.

I wouldn't be able to host a x86-64-v3 build server, but I can host a x86-64-v2 one (like I mentioned here: anonfunc/ALHP.GO#17). But yea, that is a good idea. That other server would have to be run by a trusted person as they're also now compiling the packages themselves too, so those packages can't just be verified by a checksum or something.

But I don't know how that would be organized.

No really what I've thought, let me try to explain it again.

At the moment the Alhp server is the build server and repo at the same time.

My thoughts is, the current Alhp server, just become the build server, then the repos mirror would retrieve the build package and distribute it.

I don't know if it is more clear that way

> I wouldn't be able to host a x86-64-v3 build server, but I can host a x86-64-v2 one (like I mentioned here: https://git.harting.dev/anonfunc/ALHP.GO/issues/17#issuecomment-746). But yea, that is a good idea. That other server would have to be run by a trusted person as they're also now compiling the packages themselves too, so those packages can't just be verified by a checksum or something. > > But I don't know how that would be organized. No really what I've thought, let me try to explain it again. At the moment the Alhp server is the build server and repo at the same time. My thoughts is, the current Alhp server, just become the build server, then the repos mirror would retrieve the build package and distribute it. I don't know if it is more clear that way

Oh ok, I get what you mean. Like kinda how the official arch repos are structured, right?

Oh ok, I get what you mean. Like kinda how the official arch repos are structured, right?

Oh ok, I get what you mean. Like kinda how the official arch repos are structured, right?

Yup! Just like the arch repos structure, sorry for my bad explanation it just super late where I live so ^^'

> Oh ok, I get what you mean. Like kinda how the official arch repos are structured, right? Yup! Just like the arch repos structure, sorry for my bad explanation it just super late where I live so ^^'

Ah ok. Yea I like that idea!

Ah ok. Yea I like that idea!
Owner

If more people start mirroring this. Maybe you can setup a ahlp-mirrorlist package just like the chaoticaur unofficial repo with their chaoticaur-mirrorlist package. Just a thought 👍

Probably not a bad move. I also want to bring the keyring into the AUR to prevent people getting trust or other stuff wrong, so a mirrorlist would not hurt either.

> If more people start mirroring this. Maybe you can setup a `ahlp-mirrorlist` package just like the `chaoticaur` unofficial repo with their `chaoticaur-mirrorlist` package. Just a thought 👍 Probably not a bad move. I also want to bring the keyring into the AUR to prevent people getting trust or other stuff wrong, so a mirrorlist would not hurt either.
Owner

At the moment the Alhp server is the build server and repo at the same time.

My thoughts is, the current Alhp server, just become the build server, then the repos mirror would retrieve the build package and distribute it.

I don't know if it is more clear that way

I want to move ALHP to a dedicated server if it ever reaches enough support. Sadly there is no way to contribute such a server directly, since trust issues would be unavoidable.

> At the moment the Alhp server is the build server and repo at the same time. > > My thoughts is, the current Alhp server, just become the build server, then the repos mirror would retrieve the build package and distribute it. > > I don't know if it is more clear that way I want to move ALHP to a dedicated server if it ever reaches enough support. Sadly there is no way to contribute such a server directly, since trust issues would be unavoidable.

At the moment the Alhp server is the build server and repo at the same time.

My thoughts is, the current Alhp server, just become the build server, then the repos mirror would retrieve the build package and distribute it.

I don't know if it is more clear that way

I want to move ALHP to a dedicated server if is ever reaches enough support. Sadly there is no way to contribute such a server directly, since trust issues would be unavoidable.

Yup that a big problem,and there is the fact that some mirrors could end-up like some outdated server on arch

> > At the moment the Alhp server is the build server and repo at the same time. > > > > My thoughts is, the current Alhp server, just become the build server, then the repos mirror would retrieve the build package and distribute it. > > > > I don't know if it is more clear that way > > I want to move ALHP to a dedicated server if is ever reaches enough support. Sadly there is no way to contribute such a server directly, since trust issues would be unavoidable. Yup that a big problem,and there is the fact that some mirrors could end-up like some outdated server on arch

If I wanted to setup something like this (a mirror) would there be a way to limit the amount of traffic I get? (without throwing annoying errors at users)?

If I wanted to setup something like this (a mirror) would there be a way to limit the amount of traffic I get? (without throwing annoying errors at users)?

Without trowing an error no, but you can limit the number of connections/bandwith

Without trowing an error no, but you can limit the number of connections/bandwith
Owner

If I wanted to setup something like this (a mirror) would there be a way to limit the amount of traffic I get? (without throwing annoying errors at users)?

You can always rate-limit your webserver. Here and here is the doc for nginx, there should be something similar available for apache.

> If I wanted to setup something like this (a mirror) would there be a way to limit the amount of traffic I get? (without throwing annoying errors at users)? You can always rate-limit your webserver. [Here](http://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate) and [here](http://nginx.org/en/docs/http/ngx_http_limit_conn_module.html) is the doc for nginx, there should be something similar available for apache.

I see, what kind of traffic can I expect? 100 gb per month? 1000?

I see, what kind of traffic can I expect? 100 gb per month? 1000?
Owner

Difficult to answer, since it really depends on many factors. I have no dedicated stats for ALHP, but considering the whole server traffic usage I would imagine <500GB/month.

But, like I said, it really depends on a lot of factors, like overall users, your mirror location, etc.

TL;DR: Don't mirror if you're on a tight traffic budget.

Difficult to answer, since it really depends on many factors. I have no dedicated stats for ALHP, but considering the whole server traffic usage I would imagine <500GB/month. But, like I said, it really depends on a lot of factors, like overall users, your mirror location, etc. **TL;DR**: Don't mirror if you're on a tight traffic budget.

Thank you very much for your response! I hope I can assist so I can alleviate some of the traffic to the main server. Just need to discuss it with my hosting company.

Thank you very much for your response! I hope I can assist so I can alleviate some of the traffic to the main server. Just need to discuss it with my hosting company.
anonfunc changed title from Provide a rsync URL for mirrors to Mirrors 11 months ago
anonfunc added
informational
and removed
enhancement
labels 10 months ago

Inspired by https://cloudflaremirrors.com/archlinux I've set up an experimental mirror: https://alhp.krautflare.de/

Usage:
Server = https://alhp.krautflare.de/$repo/os/$arch/

Backend server is only 400 Mbit/s (DE) but that should not matter. It is using cloudflare extensively (all their meme stuff like tiered caching, tunneling and whatnot). Means, the more people use it, the more content gets cached by cf and the faster it will be (hopefully).

I sync every 20m and it is currently configured to cache *.(zst(.sig)?) for 1 month and for any other files (this includes the directory listing) 20 min.

The only thing which won't work is if a package is rebuilt with the same version number / file name - the old version will be stuck in cf's cache. Question: Does this happen? If yes, how often? In case this occasionally happens, I could write a script which nukes them from the cf cache via their API after each rsync. The cache nuke via API thing is now implemented and working (comparing ctimes of files to the last run).

Feedback and suggestions welcome, and thanks for this project @anonfunc

Inspired by https://cloudflaremirrors.com/archlinux I've set up an experimental mirror: https://alhp.krautflare.de/ Usage: ```Server = https://alhp.krautflare.de/$repo/os/$arch/``` Backend server is only 400 Mbit/s (DE) but that should not matter. It is using cloudflare extensively (all their meme stuff like tiered caching, tunneling and whatnot). Means, the more people use it, the more content gets cached by cf and the faster it will be (hopefully). I sync every 20m and it is currently configured to cache `*.(zst(.sig)?)` for 1 month and for any other files (this includes the directory listing) 20 min. ~~The only thing which won't work is if a package is rebuilt with the same version number / file name - the old version will be stuck in cf's cache. Question: Does this happen? If yes, how often? In case this occasionally happens, I could write a script which nukes them from the cf cache via their API after each rsync.~~ The cache nuke via API thing is now implemented and working (comparing ctimes of files to the last run). Feedback and suggestions welcome, and thanks for this project @anonfunc
Owner

Sounds good. I'm almost done creating an alhp-{keyring,mirrorlist} package for ALHP, so I'm going to include that mirror in there (probably going to upload it soon).

The only thing which won't work is if a package is rebuilt with the same version number / file name - the old version will be stuck in cf's cache. Question: Does this happen? If yes, how often? In case this occasionally happens, I could write a script which nukes them from the cf cache via their API after each rsync.

Feedback and suggestions welcome, and thanks for this project @anonfunc

That is one issue I still like to tackle. ALHP should increase its buildversion (so package-X.Y-Z.1, the added part behind pgkrel) if a rebuild of the same version was done. I haven't had a chance to implement that (yet), but it's on the todo.
Regarding how often that happens currently: Overall it's a rare occurrence, but it does happen (as with nextcloud-client or linux-lts just recently).

ALHP now increases its build-version correctly.

Sounds good. I'm almost done creating an alhp-{keyring,mirrorlist} package for ALHP, so I'm going to include that mirror in there (probably going to upload it soon). > The only thing which won't work is if a package is rebuilt with the same version number / file name - the old version will be stuck in cf's cache. Question: Does this happen? If yes, how often? In case this occasionally happens, I could write a script which nukes them from the cf cache via their API after each rsync. > > Feedback and suggestions welcome, and thanks for this project @anonfunc ~~That is one issue I still like to tackle. ALHP should increase its buildversion (so package-X.Y-Z.**1**, the added part behind *pgkrel*) if a rebuild of the same version was done. I haven't had a chance to implement that (yet), but it's on the todo. Regarding how often that happens currently: Overall it's a rare occurrence, but it does happen (as with `nextcloud-client` or `linux-lts` just recently).~~ ALHP now increases its build-version correctly.
Owner

Both packages are now online on the AUR.

https://aur.archlinux.org/packages/alhp-mirrorlist/
https://aur.archlinux.org/packages/alhp-keyring/

The alhp-keyring package replaces manual key management, while alhp-mirrorlist provides an up-to-date selection of mirrors to use.

I'm going to edit the README shortly.

Besides this I moved all ALHP repos into a new organization ALHP, as you may have noticed.

The alhp-mirrorlist repo contains the mirrorlist used in the AUR package. All issues with mirrors and new-mirror-PRs should be opened there.

Both packages are now online on the AUR. https://aur.archlinux.org/packages/alhp-mirrorlist/ https://aur.archlinux.org/packages/alhp-keyring/ The `alhp-keyring` package replaces manual key management, while `alhp-mirrorlist` provides an up-to-date selection of mirrors to use. I'm going to edit the README shortly. Besides this I moved all ALHP repos into a [new organization ALHP](https://git.harting.dev/ALHP), as you may have noticed. The alhp-mirrorlist repo contains the mirrorlist used in the AUR package. All issues with mirrors and [new-mirror-PRs](https://git.harting.dev/ALHP/alhp-mirrorlist#how-to-become-a-mirror) should be opened there.

One thing that i think should be done, is to provide alhp-keyring in the repo, it could be added in the core repo aside of the archlinux-keyring.

The reason is that not everybody always update they're aur packages, and since the keyring is more crucial than the mirrorlist, it could prevent some frustration.

One thing that i think should be done, is to provide `alhp-keyring` in the repo, it could be added in the core repo aside of the `archlinux-keyring`. The reason is that not everybody always update they're aur packages, and since the keyring is more crucial than the mirrorlist, it could prevent some frustration.
Owner

One thing that i think should be done, is to provide alhp-keyring in the repo, it could be added in the core repo aside of the archlinux-keyring.

The reason is that not everybody always update they're aur packages, and since the keyring is more crucial than the mirrorlist, it could prevent some frustration.

That should be possible.

> One thing that i think should be done, is to provide `alhp-keyring` in the repo, it could be added in the core repo aside of the `archlinux-keyring`. > > The reason is that not everybody always update they're aur packages, and since the keyring is more crucial than the mirrorlist, it could prevent some frustration. That should be possible.

@incognico What's the server software you are currently using to host the files? Asking this because I tried two times to download a large package from your mirror with 5 concurrent connections and it killed the server process (I guess?) both times, making Cloudflare respond with timeout (524) errors.

Maybe using Nginx would be a better choice for resource managing while serving files (Just like ALHP's main server)?

@incognico What's the server software you are currently using to host the files? Asking this because I tried two times to download a large package from your mirror with 5 concurrent connections and it killed the server process (I guess?) both times, making Cloudflare respond with timeout (524) errors. Maybe using Nginx would be a better choice for resource managing while serving files (Just like ALHP's main server)?

@alissonlauffer Technically you are downloading form cloudflare servers, so the endpoint for the concurrent connections should be cloudflare (which fetches with plain http1 from my server). Maybe it was a coincidence as I was upgrading cloudflared yesterday. Could you test again and/or post me your XferCommand (or the command you have used) so I can try to reproduce it?

@alissonlauffer Technically you are downloading form cloudflare servers, so the endpoint for the concurrent connections should be cloudflare (which fetches with plain http1 from my server). Maybe it was a coincidence as I was upgrading cloudflared yesterday. Could you test again and/or post me your XferCommand (or the command you have used) so I can try to reproduce it?
Sign in to join this conversation.
Loading…
There is no content yet.