Back to Basics – Updating WordPress Strategies

This post was originally posted on TimNash.co.uk on 24th Jan 2020. Hi, I'm Tim, tend to talk about WordPress and Security on my site, I hope you enjoy this repost.

In a darkened room the hacker is sitting, he has been trying to gain access to his target for hours, green writing on a black terminal flashes by as he crafts by hand the payload. This attack will be the one that lets him get in. Just at the moment, the pinnacle of success his Mum comes in with his cup of tea.

For most people, this is what they think a bad actor looks like, a solitary individual or group out to target them. The reality, of course, is most hacks are done entirely automated with no human ever typing a single command into your now rooted server.

With the start of the new decade, I thought I would take a look at some of the basics of WordPress security that we often take for granted and consider how in 2020 we can achieve a minimum level of security that aims to remove 99% of automated attacks.

This article is less of how-to, rather a look at updating strategies for agencies and sites even ones that consider mission-critical. With the right strategy in place and how to test updating is easy and something that shouldn’t be keeping you up at night. However, while not a how-to it includes tools and techniques that should hopefully be used.

Why Update?

It seems trivial, so obvious and yet actually can if you are not careful to turn out to be almost impossible to do but the number one way to keep software secure is to keep it patched and up to date.

Most bad actors (malicious attackers) are not using hugely clever tooling to attack your sites, instead, they rely on known vulnerabilities and code written by others to exploit your site and thousands of others in an automated fashion.

Let’s assume you have blue widgets plugin and it has a vulnerability that allows uploading of a file to the webserver with no checks. This is pretty bad and gold dust for our bad actors. However several things have to have happened, number one our bad actor needs to know about your site and number two they need to know about the vulnerability as well as how to exploit it.

So how do bad actors find your sites? Well, the first way is that your site is already known to them and perhaps is being specifically targeted, this is incredibly rare even larger hacks of well-known brands don’t start off as targeted hacks however your site might be the target of a bad actor either for political, economical, state reasons or just malice.

The second far more likely reason is that your site was found during an automated scan, looking for conditions that might make it exploitable, for example, your site might be the target because a scan has revealed you have the blue widgets plugin. In this scenario your site is not the goal, it is simply being used to gain access to the server. Automated attacks can be fairly specific trying just one vulnerability having checked such conditions exist, or a total scattergun approach, accepting your site is up, therefore trying as many exploits as it can till it’s blocked hoping one succeeds.

Sites are rarely specifically targeted and it’s sad but important to understand they are often simply collateral damage on the way through to the server resources behind them. There are exceptions, especially towards e-commerce sites where credit card skimmers are becoming an increasing threat but on the whole, if your site has not been specifically targeted its just a way to get on to the server

So how do vulnerabilities come to light? For our Blue Plugin Widget, someone needs to know there is a vulnerability and write the code to exploit it. A bad actor may identify a vulnerability, they might wander across it or be looking at a given plugin as a potential target. As likely a security researcher or the plugin authors themselves might identify the issue and patch it. When a vulnerability is found, by a bad actor or published as information by the plugin author/security researcher that information tends to get disseminated quickly. What starts off as a “this is a potential issue”, quickly changes to this is proof of concept on how to exploit it, to here is a “weaponised” version of the exploit, to it deployed to exploitation framework publicly available.

The good news if a vulnerability is discovered developers normally take steps to patch their code and fixing the bug that allows the vulnerability they then release this fix as a new version. So how do we stop most attacks?

We update.

Security Updates vs Fixing things

In the perfect world, WordPress core would have some sort of long term security release cycles where security fixes are released independently of anything else. Certainly some minor releases are referred to as security fix releases but quite often security patches get bundled up with general bug fixing.

WordPress core technically only supports the latest version of WordPress.

However the security team backports critical security bugs, so when there is a major issue the latest version of WordPress gets a minor release, but so does all/relevant branches all the way back to 3.7. At least currently as WordPress releases more and more this process is becoming more burdensome and this policy is under review so it’s not safe to assume anything but the latest version is considered patched.

All of this is just WordPress core, themes and plugins each have their own review life cycles but most do not backport security releases. Worst many plugins and themes do not declare security fixes leaving them as fixed a couple of bugs, line item in changelog. Consequently only the latest version of a plugin or theme is likely to be the one with the latest security patches.

It’s not that easy to update

Or maybe it is, WordPress has a very simple manual update procedure, you login in to your Admin Panel and under Dashboard, there is Updates clicking this lists all the updates needed to be done, for WordPress core itself as well as Plugins and Themes. Clicking update on an individual component or update all will connect to WordPress.org and download the appropriate Theme/Plugin or WordPress itself.

A few issues present itself with our default approach, the first and foremost is it relies on a human to do it and well we suck as a race at doing things like this. It’s like we have a built in phobia of update buttons, perhaps a traumatic past or just far to use to Windows failing has made us wary.

Another issue with this approach is someone needs to be in charge of doing it, a process of when and how it’s done is needed and ideally, it’s recorded when things get updated. This means for the solo site operators, no holiday for you without your laptop, and I hope you’re healthy. In a larger organisation, we have even more issues, with who pressed the update button and when. The meaning site can have issues and we don’t even know that it’s an update because Bob hit the update button for WooCommerce but failed to update anything else.

So letting humans willy nilly clicking buttons doesn’t sound good, what other approaches could we take?

Fully Automated

Route one, is to accept that we as humans are not going to update things and we should just let the system do the updates. By default WordPress already updates the core with “minor” updates so there are systems in place and by adding a few additional filters plugins/themes and major updates of WordPress can be enabled.

add_filter('allow_major_auto_core_updates', '__return_true'); // Updates all versions of WordPress Core
add_filter( 'auto_update_plugin', '__return_true' ); // Updates Plugins
add_filter( 'auto_update_theme', '__return_true' ); // Updates Themes
add_filter( 'auto_update_translation', '__return_true' ); // Updates Translations

It’s important to understand the automatic updater works using the default WordPress updater so only plugins that it can update, will work with the WordPress update solution. Likewise most third party automatic updaters, or WP-CLI still also rely on this system. If you have plugins or themes, that use their own updater and don’t hook into this system, then you may need to do additional steps to achieve full automation.

A fully automated process looks something like this:

Once every 12-24 hours, the site checks for updates using the standard WordPress checker.
If updates are found, the site updates.

Um that’s it.

Now, this is pretty simple but we all know that when we update something breaks, and this is going to be at 2 in morning. There is no benefit to us being secured from bad actors if the site isn’t operable. The reality is failures happen rarely even though they are perceived as a high risk, however a better process is:

Once every 12-24 hours, the site checks for updates using the standard WordPress checker.
If updates are found, we update the staging site with updates.
The staging site gets checked and if no one stops it the main site updates 24 hours later.

In our first scenario we could do that with 3 lines of code, now we have added a staging site and all its dependencies, we have to also built in a delay for our live site. Also it has the problem it’s reliant on those squishy humans to check things. A fully automated shouldn’t have humans involved.

So a more reliable process would be:

Once every 12-24 hours, the site checks for updates using the standard WordPress checker.
If updates are found, we update a test site with updates.
The test site is then “tested” against in some automated fashion.
If the tests pass then the main site gets updated.

So again our complexity has rapidly increased, we need a test site, ideally one that is pulling from the live site. Running a series of tests, meaning we need some sort of test suite and then pushing or re-running the update on the live site.

However once set up this fully automated approach should be incredibly robust as long as our test coverage is robust. Indeed for all of updating approaches a lot relies on testing (see testing strategies).

Updating and not panicking of failures

While our third option is the most, ideal what if we don’t have the technical capabilities to implement a multistage deployment should we just automate and be damned? The answer is almost certainly yes. Most people who have failures during update process fit into the infrequent updaters or the cautious update camps.

In both cases, they either don’t update or worse selectively update and consequently not everything gets updated at the same time, the result is a plugin which might have a compatibility fix with the latest version of WordPress doesn’t get updated while the core does. The results are an error and a fear to update. If everything gets updated the failure rate drops significantly enough that many managed hosts are willing to force clients into update cycles. In addition while it’s not ideal for a site to be down, few sites are so mission critical that once every few years they can’t afford to be down overnight. If your site is mission critical then it should be treated as such, if it’s your personal blog, or brochure site then we can update and if there is an issue we fix it in the morning.

Pros of Fully Automated

Never have to worry again about updates
Patch cycle time is immediate
No compliance issues, always up to date.

Cons of Automated

Needs a system to handle testing and deployment
Site may break and no one will be available to fi
Huge amount of complexity

Who is it best for

The fully automated solution is probably suited for the vast majority of sites with very few exceptions. It does however rely on good plugin and theme choices, the system falls down if there are “troublesome” plugins or large chunks of third party code that isn’t within the automated system. Building the system is a reasonable investment, and does require building out testing for the best results. Ironically agencies and development teams with CI (continuous integration) based deployments are often most resistant to fully automated approach yet are in the perfect place for building out such systems, as much of their tooling is already in place. If your a development shop not using fully automated updates then it’s an opportunity to massively improve your deployment strategies.

Patch Tuesday

Route 2 we just tried to convince everyone that full automation is the way to go and the response was something akin to: Don’t be silly.

The various arguments came forward all flawed but you my friend are not in a position to fight them, you try to convince them with failure rates below 1% automatic updates are safe, that testing will cover edge cases. You have tried to explain the investment in systems and processes will improve deployments across the board. It all falls on deaf ears then you need a different strategy and this is the time honoured “Patch Tuesdays” how does it work:

You keep a local or staging site, and you update it once a week

Test
Then deploy

This whole process is manual, BUT TIM SQUISHY HUMANS. Yes, but at least squishy humans are not randomly clicking buttons. Patch Tuesday strategy works because we update once a week regardless. If we fail to update then we no longer have a strategy so we are substituting automation for the process.

Patch Tuesday strategy means we turn off and disable automatic and indeed all automated updates our Patch Tuesday is our source of updating, this way they are tested in a single process.

Patch Tuesday approach has the advantage issues are in theory found before deployment, they have a chain where there is a person responsible for the deployment so they should find issues. However, just like any of the routes, Patch Tuesday is reliant on testing.

The biggest problem with Patch Tuesday is when it’s missed or worse something is not updated, the second this happens this becomes a failing system and as the gap between times to update increase more things are likely to break meaning Patch Tuesday becomes a burden worse it becomes an excuse.

One option is to automated Patch Tuesday

So that the initial commit or command run is automated, and then the human is just responsible for testing and deployment. In this way it’s not dissimilar to option 2 within the fully automated approach only by it happening just once a week or a specific period it reduces that burden down.

The name Patch Tuesday comes from Microsoft’s regular schedule for patches (you know the ones that just install on your machine, but in enterprises they manually update because)

Patch Tuesday route, also potentially limits the way your store code, for the site as almost certainly to do it reliably ALL code will need to be stored in version control including vendor folders etc. Where as fully automated only custom code would need to be in there, as the live site would be automatically getting latest version (or could do depending on strategy)

Pros of Patch Tuesday

Issues are picked up early and fixed prior to production
Built into existing deployment options

Cons of Patch Tuesday

Humans everywhere
It’s process not code heavy, the first time someone fails to patch the system breaks
It is much slower deployment time
Almost certainly always lagging behind a truly updated system.
Limits how you store code.

Who is it best for

Incredibly mission critical sites, where the entire business is based around the site and has enough of a team built around it that Patch Tuesday strategy can work. Teams already used to working in patch cycles, so attached to more traditional IT will have easier buy-ins for this style of deployment.

Approved Release

Route 3 is another messy release strategy, everyone agrees they love automated releases but they have WooCommerce installed and that can’t be trusted (it can but I know you won’t believe me) instead with approved release we pretty much follow option two of the Automated Release but with a few modifications:

Our staging/testing site is updated, and any automated testing is done.
A notification is sent to the human, who double checks and “approves”
The update then gets deployed on the live site.

Basically the staging site stays always up to date, but the final push is done by a human to live this gives it some advantages of Patch Tuesday and fully automated but with the disadvantages of both.

We still have squishy humans and indeed a squish human with a flexible process. One of the biggest benefits of Patch Tuesday is a rigid update structure means updates are expected at given points. Approved release is adhoc, and worst potentially in the hands of just one person.

Like Patch Tuesday, this strategy means all things go through staging/testing and so turning off all automatic updates and even removing update options from the user interface all together. This also works best in strategies where the site is entirely controlled through the staging/testing cycle with dedicated deployment processes. Indeed both Patch Tuesday and Approved Release only work if there is a single deployment pipeline, otherwise things will get missed (though Approved release, each update could pull from live first).

Pros of Approved Release

Issues are picked up early and fixed prior to production
Issues can be pushed live quickly

Cons of Approved Release

Humans everywhere
Still needs a deployment solution for automatic updates on staging, a way to deploy to live & a way to notify of updates
Yet still process reliant, if no one pushes the button.
Almost certainly always lagging behind a truly updated system.
Limits how you store code.

Who is it best for

Very few people, on first glance a lot of solo developers, and tiny agencies with maintenance contracts will look at this and think this is perfect we keep control but have a nearly entirely automated solution. For e-commerce sites Patch Tuesday or Fully Automated are better strategies. From a security and engineering perspective this strategy might not fit any one, however I can see certain business cases where an agency might wish to adopt this approach not least because it enforces a one deployment strategy.

Testing

In all 3 of our strategies they all fall down on testing, if we don’t test either manually or automatically then they are going to fail. For testing we have 3 rough categories of testing:

Error Monitoring
Acceptance
Visual Regression

Error Monitoring

In its most simple form is hitting end points and seeing if they have a PHP or Javascript error, in addition seeing if the amount of errors within your error log increases. This can easily be monitored, both by processing the error log and navigating around the site. Often Error Monitoring can be done alongside acceptance testing (acceptance testing does the clicking) so error monitoring can then look for differences.

In addition simple HTTP monitoring such as uptime tools can check the site is not generating 500 error on given pages. At minimum consider checking your Homepage, Login page and any mission critical pages (checkout for example).

While HTTP monitoring should be fairly instant, ultimately this a few curl requests made from multiple locations and could easily be done with your normal uptime tool (you are running uptime monitoring right?). Error monitoring which is looking at statistical increases in errors might take longer to be noticed. By combining with a baseline and acceptance tests we can speed up at least the initial analysis.

Recommended tools

Acceptance Testing

We alright integration and acceptance tests right? When you develop new features, well now is the opportunity to reuse them and not just for testing your backups. By running the acceptance tests against your live/test sites you should be able to identify if there has been any breaking changes. In addition you can use these along side error monitoring to see if your testing is generating additional errors in the logs (or if your logs are noisy, this could be a percentage increase, though in our perfect world your PHP error log would be empty).

The downside to acceptance testing is they can be slow, even on a very uninteresting site like this my acceptance test suite takes about a minute to run, however most acceptance/integration test software allows you to group tests, so you can run just high priority for example post builds, a good example for this site my acceptance tests:

Check my homepage loads, including looking at the footer text
My login page, loads, can login and prompted for two factor auth
My RSS feed loads and counts how many entries are on the feed

Now my full test suite does things like testing the contact form, some backend bits and a bunch of little tests.

Things to think about before running acceptance tests on live:

You might have security in place to prevent such automation
Your acceptance tests might be destructive in nature checking the delete all posts functionality works might not be a good test for live.
They might create DB entries, for example one of my acceptances tests creates a post and publishes it (actually it doesn’t because Gutenberg broke it, thats a post for another day)

Picking tests that have broad coverage ultimately for updates you want to know if things are working.

Recommended Tools

Codeception

Interested in Acceptance testing I spoke at WordCamp London 2015 on introduction to acceptance testing.

Visual Regression Testing

Visual Regression testing sounds very fancy, but it literally means comparing the output of your site with a previous version you took. This is normally done by screenshotting your site before and after and then comparing them, if there is a percentage difference between the two then it is flagged. A “poor mans” visual regression is not to use the rendered output (i.e screenshots) but simply to evaluate the DOM this has a higher level of false positives as even a slight element change, for example, a plugin updating its DOM to clean it up might result in every other element appearing to have changed.

Visual regression testing can be either very simple or very fiddly to set up depending on use cases, there are third-party services like VisualPing that will compare screenshots from previous days.

Recommended Tools

UPDATE ALL THE THINGS!!!

Everything needs an update from time to time, and your goal is to make it as easy as possible to update. For the vast majority of people and companies going fully automated is the answer even without a full test coverage the failure rate is so low as to be acceptable for all but the most absolute critical sites. For those mission-critical sites that can’t be fully automated then rolling out Patch Tuesday strategies minimises potential problems of Adhoc updating.

In all cases updating fails, when it’s not done consistently, keeping things up to date might feel daunting if you’re looking at a screen with a hundred plugins all needing updating but once you are on top of it, the process can be seamless and you will never have to think about it again.

It’s important to remember though WordPress is just one part of the stack, and so its not just updating WordPress, plugins, themes but everything underneath if you manage the hosting yourselves then you will need strategies for updating the operating system and all the other applications on it.

Regardless of which route you take, know this you need to update and keep on top of updates if something is stopping you from doing so it is the problem and should be considered a blocker.

Want to learn more?

This post is from a series called Back to Basics, here is the complete series so far:

Help others find this post:

This post was written by Me, Tim Nash I write and talk about WordPress, Security & Performance.

If you enjoyed it, please do share it!

Source