Making Your PHP 7 Faster, Part 1: HugePages中文EN

PHP 7 has just released RC4, which includes a number of bug fixes along with our latest performance win (see the NEWS): "HugePageFy the PHP TEXT segment." With this feature enabled, PHP 7 "moves" its own TEXT segment (the executable body) onto HugePages. In earlier testing we were able to consistently see a 2%–3% QPS improvement on WordPress.
What are HugePages, briefly? By default, memory is paged in 4 KB chunks. Translating a virtual address to a physical one requires a table lookup, and CPUs speed this up with an internal TLB (Translation Lookaside Buffer). The smaller the virtual pages, the more entries that table needs — and since a TLB is finite, more entries mean a higher TLB cache-miss rate. So enabling large pages indirectly brings down that TLB miss rate. A Google search will turn up plenty of detail, so I won't go deeper here — the point of this post is mainly how to turn this new feature on and get a meaningful performance gain.

Enabling HugePages is easy on a modern kernel. On my dev VM (Ubuntu Server 14.04, Kernel 3.13.0-45), if we look at the memory info:

$ cat /proc/meminfo | grep Huge
AnonHugePages:    444416 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

A single HugePage is 2 MB, and none are currently in use. Let's first build PHP RC4 — and please do not pass --disable-huge-code-pages (this feature is on by default, so that flag would turn it off).
Then configure opcache. Since PHP 5.5, Opcache has been built in by default, but as a shared extension, so we still need to load it from php.ini:

zend_extension=opcache.so

This feature lives inside Opcache, so it's enabled through Opcache as well (by setting opcache.huge_code_pages=1). Concretely:

opcache.huge_code_pages=1

Now configure the OS to reserve some HugePages:

$ sudo sysctl vm.nr_hugepages=128
vm.nr_hugepages = 128

Now check the memory info again:

$ cat /proc/meminfo | grep Huge
AnonHugePages:    444416 kB
HugePages_Total:     128
HugePages_Free:      128
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

The 128 HugePages we reserved are now in place. Let's start php-fpm:

$ /home/huixinchen/local/php7/sbin/php-fpm
[01-Oct-2015 09:33:27] NOTICE: [pool www] 'user' directive is ignored when FPM is not running as root
[01-Oct-2015 09:33:27] NOTICE: [pool www] 'group' directive is ignored when FPM is not running as root

Now, check the memory info once more:

$ cat /proc/meminfo | grep Huge
AnonHugePages:    411648 kB
HugePages_Total:     128
HugePages_Free:      113
HugePages_Rsvd:       27
HugePages_Surp:        0
Hugepagesize:       2048 kB

At this point, to be clear: whenever HugePages are available, Opcache will also use them to back its opcode cache. So to actually prove opcache.huge_code_pages is doing what we think, let's turn it off and restart, then look at the memory info:

$ cat /proc/meminfo | grep Huge
AnonHugePages:    436224 kB
HugePages_Total:     128
HugePages_Free:      117
HugePages_Rsvd:       27
HugePages_Surp:        0
Hugepagesize:       2048 kB

As you can see, with huge_code_pages enabled, fpm used 4 extra pages after startup. Now let's check the text size of php-fpm:

$ size /home/huixinchen/local/php7/sbin/php-fpm
   text   data    bss    dec     hex filename
10114565  695200  131528 10941293  a6f36d /home/huixinchen/local/php7/sbin/php-fpm

The text segment is 10114565 bytes — roughly 4.8 of the 2 MB pages' worth. Taking alignment into account (the trailing bits under one 2 MB page don't get moved), that comes out to 4 pages — exactly what we observed.
So the configuration is working! Enjoy 🙂
One caveat up front: once this feature is on, profiling with perf report/perf annotate will show lost symbols (valgrind and gdb are unaffected). The root cause is that perf's design listens on mmap and records address ranges in order to map instruction pointers back to symbols, but HugeTLB currently only supports MAP_ANON, so perf concludes that this address range carries no symbol information. I hope a future kernel fixes this limitation..

Finally: "Hey, why aren't you posting a performance comparison?" Heh — I'm leaving that for you all to try yourself. Feel free to comment on what you observe 🙂
(To be honest, the real reason is that it's a holiday and my company VPN is being useless right now, so I can't reach the beefy dev machine that Lianjia set up for me. I have no test environment, and on a VM the effect is too subtle to measure — so I'm skipping the test. Heh.)