Last month, I was asked to investigate why content that a website owner thought was protected behind their website’s “Members Only” area was showing up on Google, much to their horror. There are several ways of fixing this problem, and I thought I’d share one with you.
In this particular case, the website’s “Members Only” area was being protected using the WP-Members plugin, although this problem would have happened even if using WordPress’ built-in “protected post” feature.
The Member’s Only page was corrected being “blocked” (WP-Members plugin terminology) – requiring a user to be logged in, but after that is where the problems started.
Although the Member’s Only page was correctly blocked, the pages deeper down into the site were not. I can only assume here, but it looks like the person setting up the pages assumed that since you couldn’t access these pages from anywhere on the site except for the protected page, that all was good. This is not the case. If you knew the direct URL to the sub-pages, you could get the content as an anonymous user. This also meant that the URL to these pages is published in sitemaps, which make it really easy for search engines, like Google, to find the content.
Lesson: Make sure all pages that should be hidden from public view are protected.
Even if your page/post is protected, any files you upload to the media library and attach to the post are still accessible if the address is known. In this case, there were lots of PDF documents on the site, that were thought to only be accessible by “members”, since they were linked to from the pages that were in the Member’s Only area.
Blocking content from being accessed is done in WordPress through PHP code that checks which user you are, if you have the right permissions, etc. When your web browser is loading images, PDFs, videos, and other content that isn’t a web page, WordPress does not get involved in the process. The web server (Apache in this case) serves up the file without ever getting the PHP engine involved. So calling http://example.com/wp-content/2015/01/my_private_file.pdf will get delivered to anyone who asks for it. Scary, right?
The solution to this problem is to have WordPress act as an intermediary between the request for the file, and the file itself, allowing permissions to be verified in the process.
Enter Download Monitor, a handy plugin that will do exactly that, by doing two things:
- All files that are uploaded to the site using the Download Monitor custom content type are stored in a separate folder in WordPress (/wp-content/uploads/dlm_uploads/ to be exact). This folder contains an .htaccess file (instructions to the Apache web server) do not allow anyone access to the files in it (“deny from all” is the rule).
- Links in your posts/pages point to a new path, for example: /download/my_private_file/ that is managed by the Download Monitor plugin. It checks to see if the user has the correct permissions to access the file. If so, PHP retrieves the file on the server (because it is accessing the file directly on the hard drive, and not through a URL, the Apache .htaccess rules do not apply), and delivers the file to the browser as a download.
In addition to providing this protection of files that need to be restricted to specific users, it also gives you some additional features:
- The ability to see how many times a file has been downloaded, right in the WordPress dashboard.
- The ability to have different versions of the same file downloaded (think software, where you might want to make versions 1.0 and 1.1 downloadable)