Containers power Coursera’s 2nd generation programming assignments infrastructure. Instructors package dependencies and test cases into docker container images, which are uploaded to Coursera. Then, for every submission we map the uploaded code into a fresh container instantiated from the corresponding image and kick off the instructor’s grading script.
Because we use a shared pool of resources and schedule submissions upon a unified cluster, this architecture is an order of magnitude more cost effective and reliable compared to our first generation infrastructure. Unfortunately, security and isolation is a huge challenge in this shared environment. Containers by themselves are not secure1, and so to ensure reliable operation of our clusters, we mix in some extra secret sauce. While much of our hardening is discussed elsewhere1 2 in greater detail, we perform a few operations within containers instantiated from the instructor-provided images. In order to these functions to operate securely, we have to take a few precautions. In particular, we must use native binaries that are statically linked to ensure that we only depend on the kernel application binary interface (ABI). This post will walk through a sample attack and the corresponding defense.
When compiling a C program on Linux, by default gcc/clang produce a dynamically linked ELF file. You can inspect the output file with the
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
When hello is executed, the kernel loads the binary into the address space of the new process. It then reads information in the ELF file and loads the referenced dynamic libraries from the file system into memory. The loader rewrites the addresses in all the loaded ELF files to point to the loaded addresses, and then transfers execution to the
main symbol in the loaded hello ELF file. In Linux (and Darwin), the loader is configurable through a few mechanisms. The first is an environment variable named
LD_PRELOAD, and the other on Linux is the file
/etc/ld.so.preload. The dynamic linker inspects these, and will preferentially link symbols defined in libraries ahead of those requested in the ELF header. By overriding bindings to key library functions, the preloaded library can intercept execution of unmodified native binaries.
Coursera uses Rust in our programming assignments infrastructure because in addition to being a safe, modern, and fast language, Rust compiles down to native binaries. Thanks to Cargo and Rustup, we can easily statically link these programs such that they only depend on the kernel’s ABI.
Our example rust program will be the simple hello-world program generated by
cargo new --bin hello. On Linux, Rust programs by default respect both the LD_PRELOAD environment variables and the
/etc/ld.so.preload configuration file. If we compile with
cargo build and then
strace the resulting binary we see the following:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 2 3 4 5 6 7 8 9 10 11 12 13
After compiling, we can run this as follows:
1 2 3
A malicious grading container could hook into the execution of our tools by setting the
/etc/ld.so.preload file (or even by inserting a malicious libc onto the filesystem). We can build such a container as follows:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
To build static binaries in Rust, we instead use the musl target triple. Using
rustup we can install it as follows:
rustup target add x86_64-unknown-linux-musl. To build our binary, we execute:
cargo build --target x86_64-unknown-linux-musl. When we run this statically built binary with the LD_PRELOAD flag set, no method interception occurs:
We can even run this under
strace and we can confirm that it makes no syscalls that read from the local filesystem2 (in addition to seeing substantially fewer syscalls than the default gnu-libc implementation).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
In this way, we can ensure consistent execution without making any assumptions on the filesystem of a container. Thank you to the following resources: