Repository navigation
Compiler doesn't work when using Docker #65662
Description
Activity
Nominating for libs team as this is a recent regression caused by #65094 -- cc @alexcrichton
Personally this leads me to believe we should either be happier to fallback to the previous code (e.g., for any error) or revert the PR entirely since breaking usage of Rust code in Docker is not really feasible (even if this is arguably a Docker bug).
(e.g., for any error)
Alternatively, we can try statx on something that should always succeed (maybe
/?)Cannot it match ENOSYS or EPERM just like getrandom syscall does?
rust/src/libstd/sys/unix/rand.rs
Line 51 in ad7c55e
Unlike for
getrandom, getting EPERM is not sufficient evidence thatstatxdoesn't work.Thanks for investigating this down to
statxbeing the culprit, definitely makes sense!I was searching around for other instances of this, and definitely turns out we're not the only ones running into this
- statx syscalls inside a docker container docker/for-linux#208 - another docker user reporting the same issue. It's pointed out that Docker 18.04 apparently allows
statxby default - https://bugreports.qt.io/browse/QTBUG-70557?focusedCommentId=419518&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel - another user with qt's build system finding that seccomp disallows a new syscall
- Looks like musl uses
statxforfstatat(called below) and only handles ENOSYS - Seccomp itself only somewhat recently got support for statx as a sycall in general
Overaall I don't think there's a lot of prior art for this to draw from, it seems that everyone's working around the seccomp issue rather than addressing it directly.
I think a reasonable solution might be to do something like try to stat
AT_CWDinitially and if that fails with EPERM or ENOSYS we disable the syscall entirely, otherwise it's cached as always good to use. @oxalica would you be interested in helping to implement this?- statx syscalls inside a docker container docker/for-linux#208 - another docker user reporting the same issue. It's pointed out that Docker 18.04 apparently allows
I think a reasonable solution might be to do something like try to stat
AT_CWDinitially and if that fails with EPERM or ENOSYS we disable the syscall entirely, otherwise it's cached as always good to use. @oxalica would you be interested in helping to implement this?@alexcrichton
Do you mean to check current working directory instead of the actual argument before the first call, and regard EPERM and ENOSYS asstatxnot implemented?
I'm glad to fix it.@oxalica yeah that's what I'm thinking, the idea being that if we don't know whether
statxworks we invoke it with a path which we know should work almost all the time (like the cwd) and if it returns EPERM in that situation we can be pretty certain we're seccomp blocked.We are encountering a similar issue while building an Docker image with rustc & cargo on CircleCI. We are basically doing:
rustup-init -y --no-modify-path --default-toolchain nightly rustup --version cargo --version rustc --version cargo fmt --version
and are getting:
rustup 1.20.0 (a7f257941 2019-10-14) cargo 1.40.0-nightly (3a9abe3f0 2019-10-15) rustc 1.40.0-nightly (7979016af 2019-10-20) error: no such subcommand: `fmt`This errors happens since yesterday. Our last successful build with the same script was 2 days ago, with these versions:
rustup 1.20.0 (a7f257941 2019-10-14) cargo 1.40.0-nightly (3a9abe3f0 2019-10-15) rustc 1.40.0-nightly (c23a7aa77 2019-10-19) rustfmt 1.4.9-nightly (33e3667 2019-10-07)So basically between
rustc c23a7aa77 2019-10-19andrustc 7979016af 2019-10-20, something changed that makescargounable to figure out thatrustfmtis installed.After many tests on different Docker hosts, we haved narrowed the issue to the following: if the Docker host is using the
overlay2orbtrfsstorage drivers, everything works perfectly; if it is usingaufsstorage driver, thencargo fmtbasically cannot see that the~/.cargo/bin/cargo-fixfile exists, and treats rustfmt as not installed.@mguillemot-tel On the hosts where things don't work, what is the docker version and host kernel version? Also, can you run your command with
straceto see if this is because ofstatx?I tried
ubuntu:xenialwith docker 18.09.2 on Linux kernel 4.14.118, it seems thatstatxworks well.root@86646b35b199:~# cat >test.c #include <stdio.h> #include <errno.h> #include <unistd.h> #include <sys/syscall.h> #define SYS_statx 332 // x86_64 #define STATX_ALL 0xFFF int main (void) { char buf[0x100] = {}; int ret = syscall(SYS_statx, 0, "/", 0, STATX_ALL, buf); if (ret == 0) puts("ok"); else { int e = errno; perror("err"); printf("errno = %d\n", e); } return 0; } root@86646b35b199:~# gcc test.c root@86646b35b199:~# ./a.out ok
Maybe need to update docker?
Related? podman-container-tools/buildah#1568
You need rather recent Docker and libseccomp packges.
if the Docker host is using the overlay2 or btrfs storage drivers, everything works perfectly; if it is using aufs storage driver, then cargo fmt basically cannot see that the ~/.cargo/bin/cargo-fix file exists
Old Docker versions (or new versions running on the old systems) default to aufs but in recent versions this has changed to overlay2. That's possible explanation of what you are seeing.
On CircleCI you cannot choose the storage driver, though you can opt in to newer Docker versions, which we've done.
docker infoon CircleCIContainers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 18.09.3 Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 7 Dirperm1 Supported: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: e6b3f5632f50dbc4e9cb6288d911bf4f5e95b18e runc version: 6635b4f0c6af3810594d2770f662f34ddc15b40d init version: fec3683 Security Options: apparmor seccomp Profile: default Kernel Version: 4.15.0-1027-gcp Operating System: Ubuntu 16.04.5 LTS OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 7.298GiB Name: default-ccdb4d48-4a1e-40b1-90af-0ccd64a89a94 ID: LYQL:2PXY:AHV3:PLE6:H4ZU:CQQR:IY7N:TE53:LP4R:4ECZ:ZUN5:OGKJ Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Labels: provider=generic Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false Product License: Community Engine WARNING: No swap limit support
Kernel and Docker versions are fairly recent, but aufs is indeed being used.
I also tried @oxalica 's snippet on a standard
ubuntu:xenialimage which produced the following output:err: Operation not permitted errno = 1Now, CircleCI may or may not be doing something unorthodox here, but I think it is clear that this change has broken things for potentially quite a few users who do not have the capability to simply update or reconfigure their docker environment by themselves.
@mguillemot-tel can you confirm your issue was solved with the latest nightly?
Reacted by mgI can confirm it seems to work well in docker again 👍 We were having these issues with
statxlast week, and added--privilegedto docker to solve it. Now we can remove it and it still works.@jethrogb Yes, it's working perfectly again! Thank you!
When I run this sequence of commands:
I get this error:
This happens due to the
statxsyscall failing with EPERM. I believe Docker uses seccomp to limit which system calls may be made, and thestatxcall is too new, so it's not whitelisted. Because the syscall fails with EPERM instead of ENOSYS, the fallback to regularstatdoesn't work.Host kernel: 4.15.0-65-generic
docker version