Our data center provider will be conducting core network upgrade and maintenance during the above window. Our Managed Web and Mail Service will experience some packet loss and downtime during this period. We expect only a few minutes of downtime.
If you have any questions, please feel free to contact us.
On August 13 starting about 3:38PM CT, our Managed Web and Managed Mail Services went offline. The issue is with our data center provider, which started an investigation at 3:50PM CT. Our service came back online around 3:52PM CT.
We will continue to monitor the situation and update this post with additional detail when available.
Update: August 13, 2021 at 10:00PM
Services are currently stable, however our Data Center Provider will be doing maintenance on network equipment in the emergency maintenance window below:
August 14, 2021 12:00AM - August 14, 2021 4:00AM.
During this maintenance windows there may brief moment of packet loss and/or increase latency.
If you have any questions, please feel free to contact us.
Start: 2021-07-16 at 3:15PM CDT
End: 2021-07-16 at 11:50PM CDT
During a normal maintenance window our Software Defined Network (SDN) Controller went offline for upgrades, however it failed to start after the upgrade.
We are in the process of recovery the controller.
What does that mean we manage your network devices, we are unable to change settings on your devices. However, the devices do not need the SDN Controller available to continue function. Your equipment will continue to work, however any statistics will be lost.
We will update this post with additional information, when it becomes available.
Update
2021-07-16 at 3:50PM CDT We have found the cause of the problem and are attempting a fix now.
2021-07-16 at 11:50PM CDT We have resolved the issue with our SDN Controller and has returned to nominal state.
If you have questions or concerns, please feel free to contact us.
On June 8, 2021 starting at approximately 5:39PM CDT, our alerting system informed us that are primary web and database server became unresponsive from the greater Internet.
Upon investigation, we found that our Data Center (DC) provider is having a connectivity issue across there entire global footprint. The DC provider has started investigating this large scale issue.
As of 11:30PM on June 8, 2021 we do not have any further information to share.
UPDATE: June 8, 2021 @ 11:55PM: Web Sites & Services Started to be restored and coming back online. We are still waiting for additional information from the DC about these restorations. We are assuming this may be temporary and some or all services may become intermittent at times. Additional Updates when available.
Once we know more, will update this post and our twitter account.
Timeline of Events (Time in Central Time)
June 8, 2021
5:39PM: Monitoring system indicate multiple failures in our web services.
6:20PM: Investigation into cause, found possible DC Provider failure.
9:00PM: DC Provider investigating outage
11:55PM: Web Sites and Service are coming back online.
Effected Service
Managed Web Service
Managed Mail Service
Incoming mail is spooling on backup servers, will be delivered when back online
Server is accessible, but authentication and delivery failures.
Online Invoice & Billing System
When additional information is provided, we will publish updates here and on twitter.
If you have questions, please feel free to contact us.
On May 20, 2021 we were informed that our Managed Web Service Server has been blacklisted from sending emails directly. This was a change by Spamhaus by placing the server address on a Policy Based Blacklist due to bad neighbors in our data center provider.
We are looking into options to remove our IP address from Spamhaus - this is unlikely to happen.
However, we have decided to continue to implement our plans to switch to using our third-party mail delivery service for all web applications.
We will post updates on this situation as they come available.
If you have any questions, please feel free to contact us.