Jan 14, 2019

A Performance Pitfall in PHPs open_basedir option

Scrolling through strace logs showed an underdocumented behaviour of the interpreter when open_basedir is enabled. I present how an unrelated bug was magnified by open_basedir turning off the realpath cache. Underlying most filesystem functions, the cache can be critical to projects with many includes.

When the time difference of running the same code in two separate environments was an order of magnitude apart, I had a look with strace to both php-fpm processes. Each showed a different behaviour of accessing the file paths of identical class names. While the remote caused lstat directory-walks to the root with every file_exists() function, those didn't take place after a quick access locally. It turned out that I lived oblivious to the dated2 fact of open_basedir implicitly setting realpath_cache_size = 0. The number of syscalls was suspicious with the bulk having malformed arguments and though this contributed to the slowdown, it was a separate issue. The image illustrates the difference:

strace directory walk

The issue is laid out at length in PHP bug #52312, where a user stumbled on this behaviour and some debugging and discussion ensued. Interesting pointers to specifics and shortcomings of the realpath cache can be found in the comments.

Though I do understand the reasoning behind disabling the realpath cache, it was not obvious to be aware of it. I campaigned in bug trackers to have it mentioned more prominently and it's now included in the ini.core documentation, the php.ini itself of upcoming releases and Symfonys performance docs. Less suprises!

Broken autoloading

The enabled open_basedir setting amplified the actual bug in a 3rd-party package by contributing to high syscall count and consequently slower execution. Custom class autoloading didn't exclude classes outside its own prefix (see PSR-4 example that does) and checked with file_exists() before requiring them. With spl_autoload_register($prepend = true)(?) set in the code, thousands of unresolved namespaces first ran through the plugin path. And with open_basedir enabled this trailed lstat walks to the root directory for every file_exists()-call, not meeting a populated realpath_cache. Without prepend, the namespaces would've been resolved before entering the faulty class loading. So at least three conditions had to be met before bringing attention to this issue. The standard for (optimized) autoloading can be found in PHPs composer package.

How can I know the number of includes for a given code path reachable by a request beforehand? it would be interesting to compare to file_exists() invoked by autoloading then seen for the given request in a trace by a profiler.

Realpath Cache

To see how file_exists() translates down to system calls, I checked the gdb logfile of the bug reporter. To probe around yourself, you can paste this example into a terminal and look with strace -e trace=file php -d 'open_basedir=.' realpath.php and compare to the output without the option. For gdb backtrace, follow bugticket comments or set breakpoints for php_stat in ext/standard/filestat.c and the codepath main/fopen_wrappers.c expand_filepath to Zend/zend_virtual_cwd.c -> tsrm_realpath_r -> realpath_cache_find.

There's some content online that gives information on the workings of the cache. The suggestions in this 2013 user comment are interesting. Julian Pauli explains the basics of the cache at length, the process-bound, non-shared nature of it and how other disadvantages of the cache can relate to code deployments. The introduction by Benjamin Eberlei gives an overview too. He writes:

the realpath cache [..] triggers in many of the filesystem functions [with the] most important ones being fopen, file_get_contents, is_file, is_dir, require, require_once, include and include_once.

A cache underlying those functions is highly beneficial in frameworks heavily utilizing autoloading of many classes and an interpreter language that depending on configuration can have a short-lived process run time.

I haven't found much mention of the unfortunate open_basedir coupling when the realpath cache comes up, a 2013 post by Hayden James did though. Maybe this fact is widely known if you've been around in 2013.

What about open_basedir?

If the code is neither run in a shared environment or along other PHP applications, the argument for disabling open_basedir can be had, though the option would still contribute to defense-in-depth. In shared environments, it's more complex and even after precautions3 have been met, newly published vulnerabilities can be more severe in their impact.

After it is established to be expected behaviour in the bug, the comments mention a possible workaround: disabling functions related to the CVE at the base of the change2 and then patch source to have both open_basedir and realpath enabled. With realpath_turbo a module exists to do this without patching, but this can have other disadvantages (userbase, testing).

If I theorize about ways to take advantage of unset open_basedir settings in a non-shared environment a network facing software would need to have an exploit allowing file writing capabilities or application bugs that do and the path be reachable. Upload-functions or any user input are possible vectors. A set open_basedir could potentially mitigate this if the written file is outside the configured paths.

  1. pids=$(pgrep -f pool); pgroup=$(printf -- "-p %s\n" $pids); strace -e trace=file -s 10000 -o strace-prepend.log -tt ${pgroup} 

  2. the change was introduced in 2006 with PHP 5.2 

  3. even with a set chroot, bugs like php-src#69090 in opcache path validation show that shared environments are harder to secure. Some advise for one master process per fpm pool.