Homepage loads but all permalinks are 404 when using nginx & PHP-FPM

I have been trying to resolve an odd problem with my nginx config, which seems to have stopped working shortly after applying a Debian backports upgrade.

The landing page of my blog (http://blog.balaji-dutt.name/) works fine, but clicking on any other permalink (Example 1, Example 2) throws a nginx 404 error.

Here is my config, which is now pretty much the stock config from the Codex:

server {
    listen 80;
    server_name blog.balaji-dutt.name;
    error_log /var/log/nginx/blog-error.log;
    access_log /var/log/nginx/blog-access.log combined;

    ## Your only path reference.
    root /var/www/wordpress;
    ## This should be in your http block and if it is, it's not needed here.
    index index.php;

    #Gzip
    include /etc/nginx/sites-available/gzip.conf;

    location / {
            # This is cool because no php is touched for static content. 
            # include the "?$args" part so non-default permalinks doesn't break when using query string
            try_files $uri $uri/ /index.php?$args;
    }

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    location ~ /\.ht {
            deny  all;
            access_log off;
    }

    location ~ \.php$ {
            try_files $uri =404;

            fastcgi_split_path_info ^(.+\.php)(/.+)$;
            include fastcgi_params;
            fastcgi_index index.php;
            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
            #fastcgi_intercept_errors on;
            fastcgi_pass   127.0.0.1:9010;
    }

}

This is what I see in the nginx error log when I try to access a permalink (http://blog.balaji-dutt.name/about/):

2014/10/19 03:35:09 [error] 21698#0: *76739 "/var/www/wordpress/about/index.php" is not found (2: No such file or directory), client: 174.36.241.151, server: blog.balaji-dutt.name, request: "GET /about/ HTTP/1.1", host: "blog.balaji-dutt.name", referrer: "http://blog.balaji-dutt.name/"

It looks like nginx is not reading the location directive for .php files, but I have not been able to figure out why this is the case. Any suggestions or help is very welcome!

Update 1 – Oct 20 2014

Directory listing for nginx root location:

    # ls -la /var/www/wordpress
total 256
drwxr-xr-x  5 blog   www-data  4096 Sep 28 03:09 .
drwxr-xr-x  8 www-data www-data  4096 Jan 15  2014 ..
-rw-r--r--  1 blog   www-data   418 Sep 25  2013 index.php
-rw-r--r--  1 blog   www-data 19930 May 16 22:23 license.txt
-rw-r--r--  1 blog   www-data  1764 Oct  5 01:17 nginx.conf
-rw-r--r--  1 blog   www-data  7192 Sep 28 03:09 readme.html
-rw-r--r--  1 blog   www-data 55174 Mar 27  2014 sitemap.backup.xml
-rw-r--r--  1 blog   www-data  7125 Mar 27  2014 sitemap.backup.xml.gz
-rw-r--r--  1 blog   www-data  4951 Sep 28 03:09 wp-activate.php
drwxr-xr-x  9 blog   www-data  4096 Dec 12  2013 wp-admin
-rw-r--r--  1 blog   www-data   271 Jan  8  2012 wp-blog-header.php
-rw-r--r--  1 blog   www-data  4946 Sep 28 03:09 wp-comments-post.php
-rw-r--r--  1 blog   www-data  2746 Sep 28 03:09 wp-config-sample.php
-rw-r--r--  1 blog   www-data   237 Dec 22  2013 wp-config.php
drwxr-x---  9 blog   www-data  4096 Sep 28 03:09 wp-content
-rw-r--r--  1 blog   www-data  2956 Sep 28 03:09 wp-cron.php
drwxr-xr-x 12 blog   www-data  4096 Sep 28 03:09 wp-includes
-rw-r--r--  1 blog   www-data  2380 Oct 24  2013 wp-links-opml.php
-rw-r--r--  1 blog   www-data  2714 Sep 28 03:09 wp-load.php
-rw-r--r--  1 blog   www-data 33043 Sep 28 03:09 wp-login.php
-rw-r--r--  1 blog   www-data  8252 Sep 28 03:09 wp-mail.php
-rw-r--r--  1 blog   www-data 11115 Sep 28 03:09 wp-settings.php
-rw-r--r--  1 blog   www-data 26256 Sep 28 03:09 wp-signup.php
-rw-r--r--  1 blog   www-data  4026 Oct 24  2013 wp-trackback.php
-rw-r--r--  1 blog   www-data  3032 May 16 22:23 xmlrpc.php

When I try to access the URL (http://blog.balaji-dutt.name/2013/481-nginx-apache-w3-total-cache-a-bad-combination/), here is what appears in the nginx error log

*14 "/var/www/wordpress/2013/481-nginx-apache-w3-total-cache-a-bad-combination/index.php" is not found (2: No such file or directory), client: 174.36.241.151, server: blog.balaji-dutt.name, request: "GET /2013/481-nginx-apache-w3-total-cache-a-bad-combination/ HTTP/1.1", host: "blog.balaji-dutt.name", referrer: "http://blog.balaji-dutt.name/"

The path in the error log is confusing, because the part after /var/www/wordpress/ is actually the permalink and I’m not sure why it’s getting added to the request.

Update 2 – Oct 25 2014 [SOLVED]

Turns out that the nginx config file created by W3 Total Cache for Disk Enhanced caching will break permalinks, but since the file is read only on nginx reload the site will work until you reload/restart nginx. Detailed answer and a working config posted in this answer.

It turns out that the nginx config that W3 Total Cache inserts if Disk Enhanced mode is enabled will break permalinks, but only if you restart nginx after W3 Total Cache injects the config for Disk Enhnaced mode.

Based on birgire’s suggestion, I turned off all plugins and checked the site, which started working correctly. I then turned on W3 Total Cache and surprise! the site continued to work correctly. It continued to work correctly until I restarted nginx at which point the nginx.conf file that W3 Total Cache created was loaded and the site broke. What’s causing this behavior is the following line:

if ($w3tc_rewrite = 1) {
rewrite .* "/wp-content/cache/page_enhanced/$http_host/$request_uri/_index$w3tc_rewrite$w3tc_ext" last;
}

When nginx is restarted, the above line gets loaded and all URLs are redirected to to cached HTML files in wp-content/cache/page_enhanced. On the restart though, the HTML files are purged and hence links become 404. The solution I came up with was first to change permissions on the nginx config file that W3 Total Cache normally writes to so that it could not be overwritten. I then changed the above config:

location / {
    rewrite ^(.*\/)?w3tc_rewrite_test/?$ $1?w3tc_rewrite_test=1 last;
    if ($w3tc_rewrite = 1) {
        rewrite .* "/wp-content/cache/page_enhanced/$http_host/$request_uri/_index$w3tc_rewrite$w3tc_ext" last;
    }
    try_files $uri $uri/ /index.php?$args;
} 

The config does a couple of things – the original rewrite line is wrapped inside a location block so that even if the HTML file is not located, it falls back to a regular index.php rendering model. In addition, the w3tc_rewrite_test line is to eliminate an error in the Dashboard. I’m doing this in a separate file to centralize my W3 TC config so there are two location \ directives for my site.

For extra credit, it turns out that the nginx config for W3 Total Config’s minify module is broken as well. Here is a working config¹:

#Test Rewrites
    location ~ ^/wp-content/cache/minify/[^/]+/(w3tc.*)$ {
                   try_files $uri /wp-content/plugins/w3-total-cache/pub/minify.php?w3tc_rewrite_test=$1;
           }
#End Test Rewrites
# BEGIN W3TC Minify core
    set $w3tc_enc "";
    location ~ ^/wp-content/cache/minify/(.+/[X]+\.css)$ {
                try_files $uri /wp-content/plugins/w3-total-cache/pub/minify.php?test_file=$1;
    }
    location ~ ^/wp-content/cache/minify/(.+\.(css|js))$ {
                try_files $uri /wp-content/plugins/w3-total-cache/pub/minify.php?file=$1;
    }

# END W3TC Minify core

Since a lot of WordPress/W3 Total Cache info is version dependent, here’s the version info for the above config: W3 Total Cache v0.9.4 / WordPress 4.0 / nginx 1.6.2-2~bpo70+1

¹ _{https://rtcamp.com/wordpress-nginx/tutorials/single-site/w3-total-cache/}