Incident Affecting Site Uptime Data

Spending hours maintaining client sites?

WP Remote will streamline your WordPress maintenance process and save you at least 4 hours per site every week.

Summary

Here is a short summary of the incident:

  • Imunify 360 blocked requests from our plugin.
  • The blocked requests showed many sites as down, even when they were not.
  • We cleaned the data, removing the false alerts.

Timeline

  • 6th May – Issues started
  • 7th May – Changed servers
  • 8th May – Blocked requests peaked; we received 5x of these alerts
  • 10th May – Alerts started waning
  • 12th May – Cleaned all customer data to reflect accurate stats

What happened 

Between 6th to 12th of May, we experienced issues with external firewalls blocking our plugin requests. 

A standalone firewall, Imunify 360, had implemented an invalid rule that was blocking our requests. Imunify 360 is used directly by several web hosts, and therefore this was a widespread issue.

As a consequence of these blocked requests, many customers were receiving incorrect alerts about site downtime. 

The sites weren’t actually down, but our uptime requests were blocked and therefore reports have incorrectly shown downtime. 

How we dealt with the issue

As this is the nature of dynamic IPs on the internet, we have proactively built in failsafes to avoid these situations. Therefore, when our requests are blocked, we move to different servers. Usually this is sufficient to resolve any issues, and service carries on uninterrupted. 

However, in this particular instance, Imunify 360 quickly blocked even the new IPs. Therefore we reached out to their support team, and were assured that they would be rolling out a fix.

The fix itself took a few days to implement, and therefore the incident took longer to resolve, and more customers were affected as a result. 

Impact on uptime stats

Many customers were incorrectly alerted to site downtime because of the blocked requests. 

We identified the affected sites, and removed the incorrect data. Therefore, all reports currently have the correct information. 

However, as some automated reports had been triggered before the data was cleaned, there is a segment of customers who will receive incorrect reports about their site’s uptime.

We recommend regenerating client reports, so they show the correct data. Or sending this incident report as an explanation of what occurred. 

Conclusion

Please reach out to us for any clarification. We are happy to assist in any way. 

Tags:

You may also like


SSL Monitoring feature image
Automate SSL Checks with WP Remote

Have you ever had one or more of your managed websites showing an “insecure” warning to visitors? Or worse, has a client of yours come running to you with a…

How do you manage your websites?

Managing multiple WordPress websites can be time consuming and error-prone. WP Remote will save you hours every day while providing you complete peace of mind.

Managing everything yourself

But it’s too time-consuming, complicated and stops you from achieving your full potential. You don’t want to put your clients’ sites at risk with inefficient management.

Putting together multiple tools

But these tools don’t work together seamlessly and end up costing you a lot more time and money.