Incident Affecting Site Uptime Data
Spending hours maintaining client sites?
WP Remote will streamline your WordPress maintenance process and save you at least 4 hours per site every week.
Summary
Here is a short summary of the incident:
Timeline
- 6th May – Issues started
- 7th May – Changed servers
- 8th May – Blocked requests peaked; we received 5x of these alerts
- 10th May – Alerts started waning
- 12th May – Cleaned all customer data to reflect accurate stats
What happened
Between 6th to 12th of May, we experienced issues with external firewalls blocking our plugin requests.
A standalone firewall, Imunify 360, had implemented an invalid rule that was blocking our requests. Imunify 360 is used directly by several web hosts, and therefore this was a widespread issue.
As a consequence of these blocked requests, many customers were receiving incorrect alerts about site downtime.
The sites weren’t actually down, but our uptime requests were blocked and therefore reports have incorrectly shown downtime.
How we dealt with the issue
As this is the nature of dynamic IPs on the internet, we have proactively built in failsafes to avoid these situations. Therefore, when our requests are blocked, we move to different servers. Usually this is sufficient to resolve any issues, and service carries on uninterrupted.
However, in this particular instance, Imunify 360 quickly blocked even the new IPs. Therefore we reached out to their support team, and were assured that they would be rolling out a fix.
The fix itself took a few days to implement, and therefore the incident took longer to resolve, and more customers were affected as a result.
Impact on uptime stats
Many customers were incorrectly alerted to site downtime because of the blocked requests.
We identified the affected sites, and removed the incorrect data. Therefore, all reports currently have the correct information.
However, as some automated reports had been triggered before the data was cleaned, there is a segment of customers who will receive incorrect reports about their site’s uptime.
We recommend regenerating client reports, so they show the correct data. Or sending this incident report as an explanation of what occurred.
Conclusion
Please reach out to us for any clarification. We are happy to assist in any way.
Tags:
Share it:
You may also like
Introducing Customizable WP Remote Notifications: Stay Informed, Not Overwhelmed
Managing website notifications can be a headache. How do you stay on top of things without drowning in emails or Slack notifications? Equally, you don’t want to miss the forest…
WP Remote’s New APIs: Make Data Access Effortless
APIs are a great way of aggregating data from multiple sources into custom dashboards. We’ve created GET APIs for WP Remote, so you can integrate insights from your premier site…
Introducing WordPress Two-Factor Authentication (2FA) in WP Remote
The problem with login security is not the site; it is the users. Picture this: despite your best advice, your website users will reuse their passwords across multiple sites. If…
How do you manage your websites?
Managing multiple WordPress websites can be time consuming and error-prone. WP Remote will save you hours every day while providing you complete peace of mind.
Managing everything yourself
But it’s too time-consuming, complicated and stops you from achieving your full potential. You don’t want to put your clients’ sites at risk with inefficient management.
Putting together multiple tools
But these tools don’t work together seamlessly and end up costing you a lot more time and money.
