U.S. Air Force photo by Airman 1st Class Hrair H. Palyan
Let’s discuss CVE-2016-6811 now that it has been published. Freddie Rice (in the midst of reporting another hole) discovered that Apache Hadoop suffered from a security anti-pattern: trusting argv[0].
The fundamental problem with argv[0] is that it’s possible for a caller to modify its contents. This situation means that argv[0] can contain anything and everything. Using this value for any purpose without careful consideration can lead to all sorts of security problems.
Apache Hadoop utilizes a setuid C program called container-executor. This program’s original job was to switch to a different user when launching tasks. It has since been expanded upon to do all sorts of privileged operations. The ability to limit container-executor’s capabilities is through a file called container-executor.cfg. This file typically has strict permissions to prevent non-administrators from changing the configuration.
The problem is that container-executor was using argv[0] to locate the configuration file. Providing a custom location makes it possible for container-executor to read a container-executor.cfg file that enabled holes to allow an attacker to gain root.
The obvious and quickest fix is to hardcode the location of container-executor.cfg at compile. While that indeed works, it also dramatically reduces the ability to install a binary version of Apache Hadoop anywhere in the file system. Unfortunately, there is no POSIX way to do this safely.
Historically, container-executor came from the Hadoop 1.x task-controller. Despite this being used by the LinuxTaskController Java class, it ran and worked fine on nearly every OS that was POSIX compliant. When YARN was built, LinuxTaskController/task-controller was turned into LinuxContainerExecutor/container-executor. The rise of Hadoop vendors who effectively control the destiny of the Apache project also meant that portability took a back seat to features that could be issued a press release. Time to market matters and the operating systems the vendors didn’t support took a backseat.
When this security issue came up, I was already in the process of trying to repair container-executor’s ability to work on Mac OS X. Supporting OS X is essential due to the number of contributors that do all of their work on that operating system. If the code doesn’t work on it, then that code likely doesn’t get tested during development. Since I was already neck deep in portability, I figured I might as well see what it would take to replace the argv[0] functionality.
Wow. What a mess.
For those operating systems that supported this feature–and not all of them do!–they fell into three categories:
Group 1: /proc by default
Most modern Unix and Unix-like OSes support /proc. Unfortunately, not all operating systems mount /proc by default, and that makes it less likely to be available on those machines. Additionally, /proc’s layout tends to be OS-specific. Luckily for Linux and Solaris, my two targets where /proc is almost always available, it was just a matter of reading the target of two different symlinks (“/proc/self/exe” and “/proc/self/path/a.out” respectively).
Group 2: syscall
For the other operating systems, I was targetting, either /proc is an optional OS feature that isn’t always available (many of the various BSDs) or is not available at all (OS X). This circumstance means being required to take a different strategy. The next approach is to try using a system call to get the executable location.
For FreeBSD, sysctl with a simple mib is all that is required:
static int mib[] = {CTL_KERN, KERN_PROC, KERN_PROC_PATHNAME, -1};
sysctl will return a buffer with the location of the current executable. NetBSD uses a similar, but slightly different mib structure so it’s fundamentally the same code.
Mac OS X’s Mach kernel has a custom routine called proc_pidpath. This function allows a caller to find the location of the executable for a given process id. [I am unsure if this will enable one to look at processes for other
users.]
Group 3: Everything else
For those operating systems where there isn’t specific code to support relocation (e.g., AIX), the strategy is to also just hardcode the path at compile time.
A special mention about OpenBSD. This functionality is one of the places where it differs with a lot of the other open source BSD kernels. It doesn’t support the same syscall that FreeBSD and NetBSD do. In fact, at least when I wrote the code, it didn’t have any safe way for an executable to determine it’s location. As a result, on OpenBSD, the path to the configuration file is also hardcoded.
This research resulted in the creation of the get_executable.c in the container-executor’s codebase as part of the YARN-5121 and YARN-5456 patches. They fix this CVE as well help make Apache Hadoop correctly function on more operating systems.
I came across this issue during a pentest. I am not sure how it can be exploited because in configuration.c the is_only_root_writable function is checking if the file is owned by and writable only to root. As I see this is mitigating the issue with argv[0]. Am i missing something?
With relative paths under the old code base, you could cwd to something like /tmp, symlink configuration to /proc/self/environ (which is owned by root), then just create an environment with enough to get past the rest of the checks. Use something like the docker.binary with the docker command option to take over the box since it will run it as uid 0.