Skip to content

Commit 465964f

Browse files
annkotsannkots
committed
TCR-840 new mirror system to CL doc
1 parent eae2b8f commit 465964f

2 files changed

Lines changed: 221 additions & 0 deletions

File tree

‎docs/.vuepress/config-client/sidebar.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@ export default {
5353
collapsable: false,
5454
children: [
5555
"/cloudlinuxos/cloudlinux_installation/",
56+
"/cloudlinuxos/repositories_and_mirrors/",
5657
"/cloudlinuxos/elevate/",
5758
"/cloudlinuxos/limits/",
5859
"/cloudlinuxos/lve_manager/",

‎docs/cloudlinuxos/repositories_and_mirrors/README.md‎

Lines changed: 220 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
# Repositories & Mirrors
2+
3+
## Mirror Migration for New CloudLinux Versions
4+
5+
This section explains why the old mirror system will not support newer CloudLinux OS/package versions, and how to migrate to the new mirror configuration.
6+
7+
[[toc]]
8+
9+
## Summary
10+
11+
New CloudLinux versions are moving to the new mirror service and open HTTPS mirrors. The legacy mirror system (custom SSL/XMLRPC) will not support newer OS and package versions. Customers should reconfigure their mirrors to the new standard HTTPS layout and endpoint.
12+
13+
**Currently, only CloudLinux OS 10 is supported**, and support for older versions will be enabled gradually.
14+
15+
For customer-managed mirrors, the migration means **adding a new HTTPS mirror** (or **replacing** the legacy mirror endpoint) rather than "reconfiguring". In most cases you can **reuse existing storage** and just adjust the mirror layout/paths and transport (open HTTPS).
16+
17+
## What is changing
18+
19+
- Legacy mirrors used custom SSL and XMLRPC transport.
20+
- New mirrors use standard HTTPS and normal directory browsing.
21+
- The mirrorlist endpoint is now `cl-mirrors` (not `cln-mirrors`).
22+
- The new system supports partial mirrors (only the OS versions you need).
23+
24+
## Why the old system will not work for newer versions
25+
26+
- Newer CloudLinux tooling uses the **new mirrorlist endpoint** (`cl-mirrors`) and standard HTTPS mirrors.
27+
- The legacy XMLRPC/custom SSL flow is being phased out.
28+
- Mirrors that only support the old endpoint (`cln-mirrors`) may not be selected by updated systems.
29+
30+
## What you need to do (migration quick plan)
31+
32+
1. **Set up a new HTTPS mirror** that exposes SWNG content via standard HTTPS.
33+
2. **Use the new mirrorlist endpoint**:
34+
- `https://repo.cloudlinux.com/cloudlinux/mirrorlists/cl-mirrors`
35+
3. **Optionally mirror only the versions you need** (partial mirrors are supported).
36+
4. **Provide your mirror URL and scope** to CloudLinux support so it can be added to mirror rotation.
37+
38+
:::tip
39+
- SWNG is the main operational repository for updates.
40+
- `repo.cloudlinux.com` contains conversion and install assets.
41+
- Full `repo.cloudlinux.com` is large (>3 TB), so partial sync is recommended unless you need a fully autonomous environment.
42+
:::
43+
44+
## Creating a CloudLinux Mirror (SWNG)
45+
46+
### Minimum requirements for a supported mirror
47+
48+
Your mirror should meet these requirements to be eligible for inclusion in the mirror service:
49+
50+
- **HTTPS access** with a valid certificate from a public CA (e.g., Let's Encrypt)
51+
- **Correct base paths** (recommended layout):
52+
- `https://mirror.example.com/swng/` for SWNG
53+
- (optional) `https://mirror.example.com/cloudlinux/` for `repo.cloudlinux.com` content
54+
- **No authentication** for repository downloads
55+
- **Repository metadata is accessible** (e.g., `repodata/repomd.xml` for the mirrored trees)
56+
- **Regular sync schedule** (cron/systemd timer) and enough disk space for the chosen scope
57+
58+
### Step-by-step: mirror SWNG
59+
60+
**1. Prepare storage**
61+
62+
Use a dedicated disk or partition. Plan for approximately 500 GB for SWNG.
63+
64+
**2. Sync SWNG**
65+
66+
Use rsync from the upstream endpoint:
67+
68+
```
69+
# Example: mirror all SWNG
70+
rsync -avH --delete --numeric-ids --safe-links \
71+
rsync://rsync.upstream.cloudlinux.com/SWNG/ \
72+
/var/www/mirrors/swng/
73+
```
74+
75+
If you only need specific versions, mirror only those subpaths (partial mirrors):
76+
77+
```
78+
# Example: mirror CloudLinux 10 SWNG only
79+
mkdir -p /var/www/mirrors/swng/10
80+
rsync -avH --delete --numeric-ids --safe-links \
81+
rsync://rsync.upstream.cloudlinux.com/SWNG/10/ \
82+
/var/www/mirrors/swng/10/
83+
```
84+
85+
**3. Expose via HTTPS**
86+
87+
Configure Nginx (or another web server). Enable directory browsing and ensure HTTPS works with a valid certificate.
88+
89+
**4. Validate the public index**
90+
91+
Ensure the index matches the upstream structure. Mixed setups can use `<domain>/swng/`.
92+
93+
### Optional: mirror `repo.cloudlinux.com` content
94+
95+
Use this only if you need a fully autonomous install/conversion environment:
96+
97+
- Full sync is **3+ TB**.
98+
- In most cases, sync only the required sub-repositories.
99+
100+
If you decide to mirror it, use the upstream rsync module:
101+
102+
```
103+
rsync -avH --delete --numeric-ids --safe-links \
104+
rsync://rsync.upstream.cloudlinux.com/CLOUDLINUX/ \
105+
/var/www/mirrors/cloudlinux/
106+
```
107+
108+
### Information to provide to CloudLinux Support
109+
110+
When contacting CloudLinux Support to register your mirror, provide:
111+
112+
- Mirror URL (HTTPS), e.g. `https://mirror.example.com/swng/`
113+
- Whether the mirror is **public** or **private**
114+
- For private mirrors: list of IPs or networks
115+
- Geographic location (country/city)
116+
- Available bandwidth/capacity
117+
- Sync method and frequency
118+
- Whether the mirror is **complete** or **partial**
119+
- If partial: list the versions mirrored
120+
121+
## Self-diagnostics (mirror server)
122+
123+
This section helps you verify your mirror is healthy before contacting Support.
124+
125+
### Connectivity to upstream
126+
127+
```
128+
# DNS resolution
129+
nslookup upstream.cloudlinux.com
130+
131+
# HTTPS connectivity (for browsing)
132+
curl -fsSI https://upstream.cloudlinux.com/ | head
133+
134+
# RSync endpoint listing (should return module list)
135+
rsync rsync://rsync.upstream.cloudlinux.com/
136+
```
137+
138+
If you are behind a corporate proxy/firewall, ensure outbound access from the mirror host:
139+
- TCP **873** to `rsync.upstream.cloudlinux.com` (rsync)
140+
- TCP **443** to `upstream.cloudlinux.com` (HTTPS browsing/validation)
141+
142+
### RSync checks
143+
144+
```
145+
# Dry-run a small test sync (replace path as needed)
146+
mkdir -p /tmp/cl-mirror-test
147+
rsync -avv --delete rsync://rsync.upstream.cloudlinux.com/SWNG/10/ /tmp/cl-mirror-test/ | head -n 50
148+
```
149+
150+
Common rsync pitfalls:
151+
- `@ERROR: ...` typically indicates connectivity/firewall/proxy issues.
152+
- Slow sync is often disk I/O bottlenecks or too frequent `--delete` on huge trees; use a reasonable sync interval.
153+
154+
### HTTPS checks (your mirror)
155+
156+
Replace `mirror.example.com` with your mirror domain.
157+
158+
```
159+
# Mirror root must be reachable via HTTPS
160+
curl -fsSI "https://mirror.example.com/swng/" | head
161+
162+
# Metadata must exist for mirrored subtrees (example for CL10 x86_64)
163+
curl -fsSI "https://mirror.example.com/swng/10/x86_64/repodata/repomd.xml" | head
164+
```
165+
166+
If you use a different layout (e.g., you expose SWNG at `/`), adjust paths accordingly. Trailing slashes matter for some web server configurations.
167+
168+
### Storage, permissions, and sync hygiene
169+
170+
```
171+
# Disk space
172+
df -h
173+
174+
# Basic directory permissions (adjust path to your mirror root)
175+
ls -ld /var/www/mirrors /var/www/mirrors/swng
176+
177+
# Check that new files are appearing after a sync run (example)
178+
ls -lh /var/www/mirrors/swng/10/ | head
179+
```
180+
181+
Recommendations:
182+
- Run sync as a dedicated user that can write to the destination directory.
183+
- Ensure your HTTPS server can read the mirrored files.
184+
- Keep sync logs (cron output or systemd journal) so you can share errors with Support.
185+
186+
## Troubleshooting
187+
188+
| Issue | Fix |
189+
|-------|-----|
190+
| **Firewall blocks RSync/HTTPS** | Outbound from mirror host: TCP `873` (rsync) and `443` (HTTPS). Inbound to your mirror (for clients): TCP `443` (HTTPS). |
191+
| **DNS fails to resolve** | Verify `/etc/resolv.conf`, corporate DNS rules. Run `nslookup upstream.cloudlinux.com`. |
192+
| **RSync not installed** | Install `rsync` with your package manager, then retry the sync. |
193+
| **Mirror returns 403/404** | Confirm the exact base paths and trailing slashes. Confirm your web server maps URLs to the correct filesystem directories. Reference: `https://upstream.cloudlinux.com/swng/` and `https://upstream.cloudlinux.com/cloudlinux/` |
194+
| **Insufficient disk space** | Run `df -h` and sync only the required sub-repositories. |
195+
| **Permission errors** | Ensure the sync user can write to the destination directory. |
196+
| **Certificate problems** | Renew or replace the certificate (Let's Encrypt auto-renewal recommended). |
197+
| **Partial mirror not in mirrorlist** | Confirm the mirrored versions match the scope provided to Support. For private mirrors, confirm egress IPs match the scope. Changes may take time to propagate. |
198+
199+
### Connection issues
200+
201+
```
202+
# Test RSync connectivity
203+
rsync rsync://rsync.upstream.cloudlinux.com/
204+
205+
# Test HTTP connectivity
206+
curl -I https://upstream.cloudlinux.com/
207+
208+
# Check DNS resolution
209+
nslookup upstream.cloudlinux.com
210+
```
211+
212+
### Sync failures
213+
214+
```
215+
# Test with verbose output
216+
rsync -avv --delete rsync://rsync.upstream.cloudlinux.com/CLOUDLINUX/ /tmp/test-sync/
217+
218+
# Check disk space
219+
df -h
220+
```

0 commit comments

Comments
 (0)