Webmail Upgrade

Start: 2022-08-17 @ 5:00PM CDT
End: 2022-08-17 @ 6:00PM CDT

We will be upgrading out webmail interface. This upgrade will not effect mail delivery to your mailboxes.

We expect about 30 minutes of downtime to the webmail interface during this upgrade process.

If you have questions, please feel free to contact us.

Server Maintenance

Start: 2022-01-01 @ 11:00PM CDT
End: 2022-01-02 @ 03:00AM CDT

We will be performing some server maintenance during the above maintenance window.

We expect less than 30 minutes of downtime for the following services:

Affected Service

  • Managed Web Services
  • Managed Mail Services

If you have questions or concerns, please feel free to contact us.

Web and Mail Service Outage

On August 13 starting about 3:38PM CT, our Managed Web and Managed Mail Services went offline. The issue is with our data center provider, which started an investigation at 3:50PM CT. Our service came back online around 3:52PM CT.

We will continue to monitor the situation and update this post with additional detail when available.


Update: August 13, 2021 at 10:00PM

Services are currently stable, however our Data Center Provider will be doing maintenance on network equipment in the emergency maintenance window below:

August 14, 2021 12:00AM - August 14, 2021 4:00AM.

During this maintenance windows there may brief moment of packet loss and/or increase latency.

If you have any questions, please feel free to contact us.

Outage: SDN Controller Unavailable

Start: 2021-07-16 at 3:15PM CDT End: 2021-07-16 at 11:50PM CDT

During a normal maintenance window our Software Defined Network (SDN) Controller went offline for upgrades, however it failed to start after the upgrade.

We are in the process of recovery the controller.

What does that mean we manage your network devices, we are unable to change settings on your devices. However, the devices do not need the SDN Controller available to continue function. Your equipment will continue to work, however any statistics will be lost.

We will update this post with additional information, when it becomes available.


Update

  • 2021-07-16 at 3:50PM CDT
    We have found the cause of the problem and are attempting a fix now.
  • 2021-07-16 at 11:50PM CDT
    We have resolved the issue with our SDN Controller and has returned to nominal state.

If you have questions or concerns, please feel free to contact us.

Outage on June 8, 2021

On June 8, 2021 starting at approximately 5:39PM CDT, our alerting system informed us that are primary web and database server became unresponsive from the greater Internet.

Upon investigation, we found that our Data Center (DC) provider is having a connectivity issue across there entire global footprint. The DC provider has started investigating this large scale issue.

As of 11:30PM on June 8, 2021 we do not have any further information to share.

UPDATE: June 8, 2021 @ 11:55PM: Web Sites & Services Started to be restored and coming back online. We are still waiting for additional information from the DC about these restorations. We are assuming this may be temporary and some or all services may become intermittent at times. Additional Updates when available.

Once we know more, will update this post and our twitter account.

Timeline of Events (Time in Central Time)

  • June 8, 2021
    • 5:39PM: Monitoring system indicate multiple failures in our web services.
    • 6:20PM: Investigation into cause, found possible DC Provider failure.
    • 9:00PM: DC Provider investigating outage
    • 11:55PM: Web Sites and Service are coming back online.

Effected Service

  • Managed Web Service
  • Managed Mail Service
  • Incoming mail is spooling on backup servers, will be delivered when back online
  • Server is accessible, but authentication and delivery failures.
  • Online Invoice & Billing System

When additional information is provided, we will publish updates here and on twitter.

If you have questions, please feel free to contact us.

— Mad Scientist Technologies Team

Web Server Mail Delivery

On May 20, 2021 we were informed that our Managed Web Service Server has been blacklisted from sending emails directly. This was a change by Spamhaus by placing the server address on a Policy Based Blacklist due to bad neighbors in our data center provider.

We are looking into options to remove our IP address from Spamhaus - this is unlikely to happen.

However, we have decided to continue to implement our plans to switch to using our third-party mail delivery service for all web applications.

We will post updates on this situation as they come available.

If you have any questions, please feel free to contact us.

— Mad Scientist Technologies’ Team

Server Maintenance

Start: 2021-01-02 @ 11:00PM CDT
End: 2021-01-03 @ 03:00AM CDT

We will be performing some server maintenance during the above maintenance window.

We expect less than 10 minutes of downtime for the following services:

Affected Service

  • Managed Web Services
  • Managed Mail Services

If you have questions or concerns, please feel free to contact us.

Internal Data Center Updates

Start: 2020-09-18 @ 03:30 UTC / September 17, 2020 @ 10:30 PM CDT
End: 2020-09-18 @ 04:30 UTC / September 17, 2020 @ 11:30 PM CDT

Our Data Center Provider will be updating internal tooling that will limit our ability to manage clients servers during the above maintenance window.

We expect no downtime from this maintenance window for all client and managed services.

If you have questions or concerns, please feel free to contact us.

Data Center Maintenance (Hypervisor)

Start: 2020-09-14 @ 00:00 UTC / September 13, 2020 @ 7:00PM CDT
End: 2020-10-02 @ 00:00 UTC / October 01, 2020 @ 7:00PM CDT

Our Data Center Provider will be rolling out updates across there entire server fleet to improve networking performance and stability on our servers. We do not have a more specific date or time for this maintenance window, we will update this notice when we know more.

This update should have limited to no downtime associated with the maintenance.

Affected Service

  • Managed Web Services
  • Managed Mail Services
    • Webmail
    • Send & Receiving

If you have questions or concerns, please feel free to contact us.

Managed Mail Service Outage - Post-Morten

First off, I would like to apologize for this unexpected outage to our valued clients.

Our recent managed mail service outage that was between 12:17A - 12:35A on September 10, 2020, was a result of a missed announcement of an upcoming maintenance window by our Data Center Provider to resolve an issue with the underlying server that our Managed Mail Service is hosted on. This maintenance window was missed due to multiple maintenance notices for the Month of September.

The maintenance window was scheduled for:

Start: 2020-09-10 @ 04:00 UTC / Sept 9, 2020 @ 11:00PM CDT End: 2020-09-10 @ 12:00 UTC / Sept 10, 2020 @ 7:00AM CDT

Live migration was attempted, however, failed and an offline migration had to be performed. An offline migration requires the Managed Mail Server to be powered off before the migration process. This offline migration caused the outage and extended the downtime.

Timeline of Events:

  • 12:17A - Data Center Provider Attempt Live Migration of Mail Server to the new host, but fails
  • 12:22A - System Monitoring: Notification that Mail Server is unavailable
  • 12:24A - Started investigating
  • 12:28A - Found that Data Center is performing offline migration of Server
  • 12:30A - Migration Completed
  • 12:35A - Server back online

As mentioned above, there are other upcoming maintenance windows scheduled for September 2020, below is the upcoming maintenance windows:

  • September 13, 2020 (CDT) - Hypervisor/Network Upgrades
  • September 17, 2020 (CDT) - General Data Center Maintenance

Again, I would like to apologize again for this outage and I take full responsibility for this error.

If you have any questions or concerns, please feel free to contact us.

Chris Holbrook Mad Scientist Technologies