|
1 | | -Arch Linux--- |
2 | | -description: Troubleshooting the Rootless mode |
3 | | -keywords: security, namespaces, rootless, troubleshooting |
4 | | -title: Troubleshooting |
5 | | -weight: 30 |
6 | | ---- |
7 | | - |
8 | | -### Distribution-specific hint |
9 | | - |
10 | | -{{< tabs > |
11 | | -}} |
12 | | -{{< tab name="Ubuntu" >}} |
13 | | -- Ubuntu 24.04 and later enables restricted unprivileged user namespaces by |
14 | | - default, which prevents unprivileged processes in creating user namespaces |
15 | | - unless an AppArmor profile is configured to allow programs to use |
16 | | - unprivileged user namespaces. |
17 | | - |
18 | | - If you install `docker-ce-rootless-extras` using the deb package (`apt-get |
19 | | - install docker-ce-rootless-extras`), then the AppArmor profile for |
20 | | - `rootlesskit` is already bundled with the `apparmor` deb package. With this |
21 | | - installation method, you don't need to add any manual the AppArmor |
22 | | - configuration. If you install the rootless extras using the [installation |
23 | | - script](https://get.docker.com/rootless), however, you must add an AppArmor |
24 | | - profile for `rootlesskit` manually: |
25 | | - |
26 | | - 1. Create and install the currently logged-in user's AppArmor profile: |
27 | | - |
28 | | - ```console |
29 | | - $ filename=$(echo $HOME/bin/rootlesskit | sed -e 's@^/@@' -e 's@/@.@g') |
30 | | - $ [ ! -z "${filename}" ] && sudo cat <<EOF > /etc/apparmor.d/${filename} |
31 | | - abi <abi/4.0>, |
32 | | - include <tunables/global> |
33 | | - |
34 | | - "$HOME/bin/rootlesskit" flags=(unconfined) { |
35 | | - userns, |
36 | | - |
37 | | - include if exists <local/${filename}> |
38 | | - } |
39 | | - EOF |
40 | | - ``` |
41 | | - 2. Restart AppArmor. |
42 | | - |
43 | | - ```console |
44 | | - $ systemctl restart apparmor.service |
45 | | - ``` |
46 | | - |
47 | | -{{< /tab >}} |
48 | 1 | {{< tab name="Arch Linux" >}} |
49 | | -- Add `kernel.unprivileged_userns_clone=1` to `/etc/sysctl.conf` (or |
50 | | - `/etc/sysctl.d`) and run `sudo sysctl --system` |
51 | | -{{< /tab >}} |
52 | | -{{< tab name="openSUSE and SLES" >}} |
53 | | -- `sudo modprobe ip_tables iptable_mangle iptable_nat iptable_filter` is required. |
54 | | - This might be required on other distributions as well depending on the configuration. |
55 | | - |
56 | | -- Known to work on openSUSE 15 and SLES 15. |
57 | | -{{< /tab >}} |
58 | | -{{< tab name="CentOS, RHEL, and Fedora" >}} |
59 | | -- For RHEL 8 and similar distributions, installing `fuse-overlayfs` is recommended. Run `sudo dnf install -y fuse-overlayfs`. |
60 | | - This step is not required on RHEL 9 and similar distributions. |
61 | | - |
62 | | -- You might need `sudo dnf install -y iptables`. |
63 | | -{{< /tab >}} |
64 | | -{{< /tabs >}} |
65 | | - |
66 | | -## Known limitations |
67 | | - |
68 | | -- Only the following storage drivers are supported: |
69 | | - - `overlay2` (only if running with kernel 5.11 or later) |
70 | | - - `fuse-overlayfs` (only if running with kernel 4.18 or later, and `fuse-overlayfs` is installed) |
71 | | - - `btrfs` (only if running with kernel 4.18 or later, or `~/.local/share/docker` is mounted with `user_subvol_rm_allowed` mount option) |
72 | | - - `vfs` |
73 | | -- cgroup is supported only when running with cgroup v2 and systemd. See [Limiting resources](./tips.md#limiting-resources). |
74 | | -- Following features are not supported: |
75 | | - - AppArmor |
76 | | - - Checkpoint |
77 | | - - Overlay network |
78 | | - - Exposing SCTP ports |
79 | | -- To use the `ping` command, see [Routing ping packets](./tips.md#routing-ping-packets). |
80 | | -- To expose privileged TCP/UDP ports (< 1024), see [Exposing privileged ports](./tips.md#exposing-privileged-ports). |
81 | | -- `IPAddress` shown in `docker inspect` is namespaced inside RootlessKit's network namespace. |
82 | | - This means the IP address is not reachable from the host without `nsenter`-ing into the network namespace. |
83 | | -- Host network (`docker run --net=host`) is also namespaced inside RootlessKit. |
84 | | -- NFS mounts as the docker "data-root" is not supported. This limitation is not specific to rootless mode. |
85 | | - |
86 | | -## Troubleshooting |
87 | | - |
88 | | -### Unable to install with systemd when systemd is present on the system |
89 | | - |
90 | | -``` console |
91 | | -$ dockerd-rootless-setuptool.sh install |
92 | | -[INFO] systemd not detected, dockerd-rootless.sh needs to be started manually: |
93 | | -... |
94 | | -``` |
95 | | -`rootlesskit` cannot detect systemd properly if you switch to your user via `sudo su`. For users which cannot be logged-in, you must use the `machinectl` command which is part of the `systemd-container` package. After installing `systemd-container` switch to `myuser` with the following command: |
96 | | -``` console |
97 | | -$ sudo machinectl shell myuser@ |
98 | | -``` |
99 | | -Where `myuser@` is your desired username and @ signifies this machine. |
100 | | - |
101 | | -### Errors when starting the Docker daemon |
102 | | - |
103 | | -**\[rootlesskit:parent\] error: failed to start the child: fork/exec /proc/self/exe: operation not permitted** |
104 | | - |
105 | | -This error occurs mostly when the value of `/proc/sys/kernel/unprivileged_userns_clone` is set to 0: |
106 | | - |
107 | | -```console |
108 | | -$ cat /proc/sys/kernel/unprivileged_userns_clone |
109 | | -0 |
110 | | -``` |
111 | | - |
112 | | -To fix this issue, add `kernel.unprivileged_userns_clone=1` to |
113 | | -`/etc/sysctl.conf` (or `/etc/sysctl.d`) and run `sudo sysctl --system`. |
114 | | - |
115 | | -**\[rootlesskit:parent\] error: failed to start the child: fork/exec /proc/self/exe: no space left on device** |
116 | | - |
117 | | -This error occurs mostly when the value of `/proc/sys/user/max_user_namespaces` is too small: |
118 | | - |
119 | | -```console |
120 | | -$ cat /proc/sys/user/max_user_namespaces |
121 | | -0 |
122 | | -``` |
123 | | - |
124 | | -To fix this issue, add `user.max_user_namespaces=28633` to |
125 | | -`/etc/sysctl.conf` (or `/etc/sysctl.d`) and run `sudo sysctl --system`. |
126 | | - |
127 | | -**\[rootlesskit:parent\] error: failed to setup UID/GID map: failed to compute uid/gid map: No subuid ranges found for user 1001 ("testuser")** |
128 | | - |
129 | | -This error occurs when `/etc/subuid` and `/etc/subgid` are not configured. See [Prerequisites](./_index.md#prerequisites). |
130 | | - |
131 | | -**could not get XDG_RUNTIME_DIR** |
132 | | - |
133 | | -This error occurs when `$XDG_RUNTIME_DIR` is not set. |
134 | | - |
135 | | -On a non-systemd host, you need to create a directory and then set the path: |
136 | | - |
137 | | -```console |
138 | | -$ export XDG_RUNTIME_DIR=$HOME/.docker/xrd |
139 | | -$ rm -rf $XDG_RUNTIME_DIR |
140 | | -$ mkdir -p $XDG_RUNTIME_DIR |
141 | | -$ dockerd-rootless.sh |
142 | | -``` |
143 | | - |
144 | | -> [!NOTE] |
145 | | -> |
146 | | -> You must remove the directory every time you log out. |
147 | | -
|
148 | | -On a systemd host, log into the host using `pam_systemd` (see below). |
149 | | -The value is automatically set to `/run/user/$UID` and cleaned up on every logout. |
150 | | - |
151 | | -**`systemctl --user` fails with "Failed to connect to bus: No such file or directory"** |
152 | | - |
153 | | -This error occurs mostly when you switch from the root user to a non-root user with `sudo`: |
154 | | - |
155 | | -```console |
156 | | -# sudo -iu testuser |
157 | | -$ systemctl --user start docker |
158 | | -Failed to connect to bus: No such file or directory |
159 | | -``` |
160 | | - |
161 | | -Instead of `sudo -iu <USERNAME>`, you need to log in using `pam_systemd`. For example: |
162 | | - |
163 | | -- Log in through the graphic console |
164 | | -- `ssh <USERNAME>@localhost` |
165 | | -- `machinectl shell <USERNAME>@` |
166 | | - |
167 | | -**The daemon does not start up automatically** |
168 | | - |
169 | | -You need `sudo loginctl enable-linger $(whoami)` to enable the daemon to start |
170 | | -up automatically. See [Advanced Usage](./tips.md/#advanced-usage). |
171 | | - |
172 | | -### `docker pull` errors |
173 | | - |
174 | | -**docker: failed to register layer: Error processing tar file(exit status 1): lchown <FILE>: invalid argument** |
175 | | - |
176 | | -This error occurs when the number of available entries in `/etc/subuid` or |
177 | | -`/etc/subgid` is not sufficient. The number of entries required vary across |
178 | | -images. However, 65,536 entries are sufficient for most images. See |
179 | | -[Prerequisites](./_index.md#prerequisites). |
180 | | - |
181 | | -**docker: failed to register layer: ApplyLayer exit status 1 stdout: stderr: lchown <FILE>: operation not permitted** |
182 | | - |
183 | | -This error occurs mostly when `~/.local/share/docker` is located on NFS. |
184 | | - |
185 | | -A workaround is to specify non-NFS `data-root` directory in `~/.config/docker/daemon.json` as follows: |
186 | | -```json |
187 | | -{"data-root":"/somewhere-out-of-nfs"} |
188 | | -``` |
189 | | - |
190 | | -### `docker run` errors |
191 | | - |
192 | | -**docker: Error response from daemon: OCI runtime create failed: ...: read unix @->/run/systemd/private: read: connection reset by peer: unknown.** |
193 | | - |
194 | | -This error occurs on cgroup v2 hosts mostly when the dbus daemon is not running for the user. |
195 | 2 |
|
196 | | -```console |
197 | | -$ systemctl --user is-active dbus |
198 | | -inactive |
| 3 | +Arch Linux defaults to the `iptables-nft` backend. This may cause networking |
| 4 | +issues when running Docker in rootless mode. |
199 | 5 |
|
200 | | -$ docker run hello-world |
201 | | -docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: error while starting unit "docker |
202 | | --931c15729b5a968ce803784d04c7421f791d87e5ca1891f34387bb9f694c488e.scope" with properties [{Name:Description Value:"libcontainer container 931c15729b5a968ce803784d04c7421f791d87e5ca1891f34387bb9f694c488e"} {Name:Slice Value:"use |
203 | | -r.slice"} {Name:PIDs Value:@au [4529]} {Name:Delegate Value:true} {Name:MemoryAccounting Value:true} {Name:CPUAccounting Value:true} {Name:IOAccounting Value:true} {Name:TasksAccounting Value:true} {Name:DefaultDependencies Val |
204 | | -ue:false}]: read unix @->/run/systemd/private: read: connection reset by peer: unknown. |
205 | | -``` |
| 6 | +If you encounter issues, consider switching to the legacy iptables backend or |
| 7 | +ensure nftables compatibility. |
206 | 8 |
|
207 | | -To fix the issue, run `sudo apt-get install -y dbus-user-session` or `sudo dnf install -y dbus-daemon`, and then relogin. |
| 9 | +See the Arch Linux announcement: |
| 10 | +https://archlinux.org/news/iptables-now-defaults-to-the-nft-backend/ |
208 | 11 |
|
209 | | -If the error still occurs, try running `systemctl --user enable --now dbus` (without sudo). |
210 | | - |
211 | | -**`--cpus`, `--memory`, and `--pids-limit` are ignored** |
212 | | - |
213 | | -This is an expected behavior on cgroup v1 mode. |
214 | | -To use these flags, the host needs to be configured for enabling cgroup v2. |
215 | | -For more information, see [Limiting resources](./tips.md#limiting-resources). |
216 | | - |
217 | | -### Networking errors |
218 | | - |
219 | | -This section provides troubleshooting tips for networking in rootless mode. |
220 | | - |
221 | | -Networking in rootless mode is supported via network and port drivers in |
222 | | -RootlessKit. Network performance and characteristics depend on the combination |
223 | | -of network and port driver you use. If you're experiencing unexpected behavior |
224 | | -or performance related to networking, review the following table which shows |
225 | | -the configurations supported by RootlessKit, and how they compare: |
226 | | - |
227 | | -| Network driver | Port driver | Net throughput | Port throughput | Source IP propagation | No SUID | Note | |
228 | | -| -------------- | -------------- | -------------- | --------------- | --------------------- | ------- | ---------------------------------------------------------------------------- | |
229 | | -| `slirp4netns` | `builtin` | Slow | Fast ✅ | ❌ | ✅ | Default in a typical setup | |
230 | | -| `vpnkit` | `builtin` | Slow | Fast ✅ | ❌ | ✅ | Default when `slirp4netns` isn't installed | |
231 | | -| `slirp4netns` | `slirp4netns` | Slow | Slow | ✅ | ✅ | | |
232 | | -| `pasta` | `implicit` | Slow | Fast ✅ | ✅ | ✅ | Experimental; Needs pasta version 2023_12_04 or later | |
233 | | -| `lxc-user-nic` | `builtin` | Fast ✅ | Fast ✅ | ❌ | ❌ | Experimental | |
234 | | -| `bypass4netns` | `bypass4netns` | Fast ✅ | Fast ✅ | ✅ | ✅ | **Note:** Not integrated to RootlessKit as it needs a custom seccomp profile | |
235 | | - |
236 | | -For information about troubleshooting specific networking issues, see: |
237 | | - |
238 | | -- [`docker run -p` fails with `cannot expose privileged port`](#docker-run--p-fails-with-cannot-expose-privileged-port) |
239 | | -- [Ping doesn't work](#ping-doesnt-work) |
240 | | -- [`IPAddress` shown in `docker inspect` is unreachable](#ipaddress-shown-in-docker-inspect-is-unreachable) |
241 | | -- [`--net=host` doesn't listen ports on the host network namespace](#--nethost-doesnt-listen-ports-on-the-host-network-namespace) |
242 | | -- [Network is slow](#network-is-slow) |
243 | | -- [`docker run -p` does not propagate source IP addresses](#docker-run--p-does-not-propagate-source-ip-addresses) |
244 | | - |
245 | | -#### `docker run -p` fails with `cannot expose privileged port` |
246 | | - |
247 | | -`docker run -p` fails with this error when a privileged port (< 1024) is specified as the host port. |
248 | | - |
249 | | -```console |
250 | | -$ docker run -p 80:80 nginx:alpine |
251 | | -docker: Error response from daemon: driver failed programming external connectivity on endpoint focused_swanson (9e2e139a9d8fc92b37c36edfa6214a6e986fa2028c0cc359812f685173fa6df7): Error starting userland proxy: error while calling PortManager.AddPort(): cannot expose privileged port 80, you might need to add "net.ipv4.ip_unprivileged_port_start=0" (currently 1024) to /etc/sysctl.conf, or set CAP_NET_BIND_SERVICE on rootlesskit binary, or choose a larger port number (>= 1024): listen tcp 0.0.0.0:80: bind: permission denied. |
252 | | -``` |
253 | | - |
254 | | -When you experience this error, consider using an unprivileged port instead. For example, 8080 instead of 80. |
255 | | - |
256 | | -```console |
257 | | -$ docker run -p 8080:80 nginx:alpine |
258 | | -``` |
259 | | - |
260 | | -To allow exposing privileged ports, see [Exposing privileged ports](./tips.md#exposing-privileged-ports). |
261 | | - |
262 | | -#### Ping doesn't work |
263 | | - |
264 | | -Ping does not work when `/proc/sys/net/ipv4/ping_group_range` is set to `1 0`: |
265 | | - |
266 | | -```console |
267 | | -$ cat /proc/sys/net/ipv4/ping_group_range |
268 | | -1 0 |
269 | | -``` |
270 | | - |
271 | | -For details, see [Routing ping packets](./tips.md#routing-ping-packets). |
272 | | - |
273 | | -#### `IPAddress` shown in `docker inspect` is unreachable |
274 | | - |
275 | | -This is an expected behavior, as the daemon is namespaced inside RootlessKit's |
276 | | -network namespace. Use `docker run -p` instead. |
277 | | - |
278 | | -#### `--net=host` doesn't listen ports on the host network namespace |
279 | | - |
280 | | -This is an expected behavior, as the daemon is namespaced inside RootlessKit's |
281 | | -network namespace. Use `docker run -p` instead. |
282 | | - |
283 | | -#### Network is slow |
284 | | - |
285 | | -Docker with rootless mode uses [slirp4netns](https://github.com/rootless-containers/slirp4netns) as the default network stack if slirp4netns v0.4.0 or later is installed. |
286 | | -If slirp4netns is not installed, Docker falls back to [VPNKit](https://github.com/moby/vpnkit). |
287 | | -Installing slirp4netns may improve the network throughput. |
288 | | - |
289 | | -For more information about network drivers for RootlessKit, see |
290 | | -[RootlessKit documentation](https://github.com/rootless-containers/rootlesskit/blob/v2.0.0/docs/network.md). |
291 | | - |
292 | | -Also, changing MTU value may improve the throughput. |
293 | | -The MTU value can be specified by creating `~/.config/systemd/user/docker.service.d/override.conf` with the following content: |
294 | | - |
295 | | -```systemd |
296 | | -[Service] |
297 | | -Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_MTU=<INTEGER>" |
298 | | -``` |
299 | | - |
300 | | -And then restart the daemon: |
301 | | -```console |
302 | | -$ systemctl --user daemon-reload |
303 | | -$ systemctl --user restart docker |
304 | | -``` |
305 | | - |
306 | | -#### `docker run -p` does not propagate source IP addresses |
307 | | - |
308 | | -This is because Docker in rootless mode uses RootlessKit's `builtin` port |
309 | | -driver by default, which doesn't support source IP propagation. To enable |
310 | | -source IP propagation, you can: |
311 | | - |
312 | | -- Use the `slirp4netns` RootlessKit port driver |
313 | | -- Use the `pasta` RootlessKit network driver, with the `implicit` port driver |
314 | | - |
315 | | -The `pasta` network driver is experimental, but provides improved throughput |
316 | | -performance compared to the `slirp4netns` port driver. The `pasta` driver |
317 | | -requires Docker Engine version 25.0 or later. |
318 | | - |
319 | | -To change the RootlessKit networking configuration: |
320 | | - |
321 | | -1. Create a file at `~/.config/systemd/user/docker.service.d/override.conf`. |
322 | | -2. Add the following contents, depending on which configuration you would like to use: |
323 | | - |
324 | | - - `slirp4netns` |
325 | | - |
326 | | - ```systemd |
327 | | - [Service] |
328 | | - Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_NET=slirp4netns" |
329 | | - Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER=slirp4netns" |
330 | | - ``` |
331 | | -
|
332 | | - - `pasta` network driver with `implicit` port driver |
333 | | -
|
334 | | - ```systemd |
335 | | - [Service] |
336 | | - Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_NET=pasta" |
337 | | - Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER=implicit" |
338 | | - ``` |
339 | | -
|
340 | | -3. Restart the daemon: |
341 | | -
|
342 | | - ```console |
343 | | - $ systemctl --user daemon-reload |
344 | | - $ systemctl --user restart docker |
345 | | - ``` |
346 | | - |
347 | | -For more information about networking options for RootlessKit, see: |
348 | | - |
349 | | -- [Network drivers](https://github.com/rootless-containers/rootlesskit/blob/v2.0.0/docs/network.md) |
350 | | -- [Port drivers](https://github.com/rootless-containers/rootlesskit/blob/v2.0.0/docs/port.md) |
351 | | - |
352 | | -### Tips for debugging |
353 | | - |
354 | | -**Entering into `dockerd` namespaces** |
355 | | - |
356 | | -The `dockerd-rootless.sh` script executes `dockerd` in its own user, mount, and network namespaces. |
357 | | - |
358 | | -For debugging, you can enter the namespaces by running |
359 | | -`nsenter -U --preserve-credentials -n -m -t $(cat $XDG_RUNTIME_DIR/docker.pid)`. |
360 | | - |
361 | | -## Uninstall |
362 | | - |
363 | | -To remove the systemd service of the Docker daemon, run `dockerd-rootless-setuptool.sh uninstall`: |
364 | | - |
365 | | -```console |
366 | | -$ dockerd-rootless-setuptool.sh uninstall |
367 | | -+ systemctl --user stop docker.service |
368 | | -+ systemctl --user disable docker.service |
369 | | -Removed /home/testuser/.config/systemd/user/default.target.wants/docker.service. |
370 | | -[INFO] Uninstalled docker.service |
371 | | -[INFO] This uninstallation tool does NOT remove Docker binaries and data. |
372 | | -[INFO] To remove data, run: `/usr/bin/rootlesskit rm -rf /home/testuser/.local/share/docker` |
373 | | -``` |
374 | | - |
375 | | -Unset environment variables PATH and DOCKER_HOST if you have added them to `~/.bashrc`. |
376 | | - |
377 | | -To remove the data directory, run `rootlesskit rm -rf ~/.local/share/docker`. |
378 | | - |
379 | | -To remove the binaries, remove `docker-ce-rootless-extras` package if you installed Docker with package managers. |
380 | | -If you installed Docker with https://get.docker.com/rootless ([Install without packages](./_index.md#install)), |
381 | | -remove the binary files under `~/bin`: |
382 | | -```console |
383 | | -$ cd ~/bin |
384 | | -$ rm -f containerd containerd-shim containerd-shim-runc-v2 ctr docker docker-init docker-proxy dockerd dockerd-rootless-setuptool.sh dockerd-rootless.sh rootlesskit rootlesskit-docker-proxy runc vpnkit |
385 | | -``` |
| 12 | +{{< /tab >}} |
0 commit comments