Incident Affecting Site Uptime Data
Thinking of a free trial? Don’t miss this…
This is our only sale in the entire year. Full refunds too, lock your savings now!
Summary
Here is a short summary of the incident:
Timeline
- 6th May – Issues started
- 7th May – Changed servers
- 8th May – Blocked requests peaked; we received 5x of these alerts
- 10th May – Alerts started waning
- 12th May – Cleaned all customer data to reflect accurate stats
What happened
Between 6th to 12th of May, we experienced issues with external firewalls blocking our plugin requests.
A standalone firewall, Imunify 360, had implemented an invalid rule that was blocking our requests. Imunify 360 is used directly by several web hosts, and therefore this was a widespread issue.
As a consequence of these blocked requests, many customers were receiving incorrect alerts about site downtime.
The sites weren’t actually down, but our uptime requests were blocked and therefore reports have incorrectly shown downtime.
How we dealt with the issue
As this is the nature of dynamic IPs on the internet, we have proactively built in failsafes to avoid these situations. Therefore, when our requests are blocked, we move to different servers. Usually this is sufficient to resolve any issues, and service carries on uninterrupted.
However, in this particular instance, Imunify 360 quickly blocked even the new IPs. Therefore we reached out to their support team, and were assured that they would be rolling out a fix.
The fix itself took a few days to implement, and therefore the incident took longer to resolve, and more customers were affected as a result.
Impact on uptime stats
Many customers were incorrectly alerted to site downtime because of the blocked requests.
We identified the affected sites, and removed the incorrect data. Therefore, all reports currently have the correct information.
However, as some automated reports had been triggered before the data was cleaned, there is a segment of customers who will receive incorrect reports about their site’s uptime.
We recommend regenerating client reports, so they show the correct data. Or sending this incident report as an explanation of what occurred.
Conclusion
Please reach out to us for any clarification. We are happy to assist in any way.
Tags:
Share it:
You may also like
Search Engine Blocking For Staging Sites: Keep Everything Quiet Till You’re Ready
You’ve built a staging site for your client’s site, to make changes. Maybe you’re redesigning it. Maybe you’re testing some updates. Either way, your staging site isn’t for search engines. …
One-Click Login For Staging Sites: Start Using Test Site Immediately
You just spun up a staging site to test a plugin update. It’s ready. Now you have to log in. But what’s the password? Is it the same as the…
Access Your Staging Site Database In One Click
How do you usually access your staging site database? It’s not in your client’s cPanel. Remember, your staging site runs on our cloud, not your client’s server. This means you…
How do you manage your websites?
Managing multiple WordPress websites can be time consuming and error-prone. WP Remote will save you hours every day while providing you complete peace of mind.
Managing everything yourself
But it’s too time-consuming, complicated and stops you from achieving your full potential. You don’t want to put your clients’ sites at risk with inefficient management.
Putting together multiple tools
But these tools don’t work together seamlessly and end up costing you a lot more time and money.
