It is common in an LAMP setup to setup APC with the apc.stat = 0 set so that once your opcodes are loaded into the cache they stay there – till you flush the cache or restart the web server. This makes sense if you have relatively infrequent updates. At jobX we are deploying dozens of times a day, but generally, each deploy consists of changes to just a small number of files. Despite this we send Apache a graceful after each deploy which flushes the APC cache. This isn’t causing any big problems but we can do this a bit better.
We actually want apc.stat = 1.
There were all sorts of fear and doubts about doing this. It seemed the exact opposite of what you should do on a large site. We feared that all that all those extra stat calls would slow things down or even cause the site to crash. But rather than rely on our intuition about this we decided to actually try it out and see what actually happens.
First we confirmed that apc.stat (and apc.stat_ctime) would behave the way we expected when we dropped new code on the server. Relatively easy and worked as expected. Next we needed to figure out how big our cache should be and if we wanted a ttl or not. In order to do that I needed a bit of a deeper understanding of how APC operates.
Its pretty well documented that when APC fills up, it dumps the whole cache and starts over. Generally you want to avoid this - for similar reasons as to why we wanted to get away from doing graceful restarts. What was less well documented was how apc.ttl really works. Most of what I read encouraged apc.ttl = 0 to avoid fragmentation.
If you have apc.ttl = 0 apc will not purge anything from your cache (unless the cache fills up, at which point APC will purge everything from the cache and start over). It may not be obvious but with apc.stat = 1 and apc.ttl = 0, that means that APC will hold onto older versions of files that have changed. If you’ve got lots of frequent changes, this could quickly lead to your cache filling up with old versions of your opcodes.
This behavior changes when setting apc.ttl to something larger than zero. APC will not do any cleanup unless it needs to. So if you set your ttl to 3600 that does not mean that apc will automatically dump all files older than an hour. APC will remove items older that ttl…
- if it needs the space for other new items.
- if a newer version of the same file gets inserted and an old version is older than the ttl.
For example, your config.php is cached and you make a change and deploy a new config.php. APC will cache the new version of the file and it will also hold onto the previous version of the file. However, if apc.ttl is > 0, apc will opportunistically check to see if it can clear out any of those old versions of that file. This helps keep cache size in check but it comes at a cost, namely fragmentation.
As old items are purged from the cache you start to get fragmentation. If fragmentation gets too high (more than 50%) you’re likely to start to see some performance impact. One way to keep fragmentation in check is to have a large enough cache. Some of the research I found suggests that a cache size about twice as large as the size of your files will go a long way towards keeping fragmentation low. I’m not sure but it sounds like APC might do some relocation of items in the cache which goes easier when it has large empty blocks to use for reshuffling.
Armed with this understanding I took a look at our historical graphs of APC (you do have graphs for everything, right?) to see how big the cache should be. Over the weekends, when there generally aren’t any deploys, the cache grows to just under 200M. Our cache was sized at 384M to start. I wanted to keep the cache comfortably large so I started ours at 500M.
To get a ttl to start with I looked at our deploy frequency, which files change most frequently and how many files change per deploy and went with 3600.
Here are a set of articles with details about apc: