DC Maintenance Window

Start: 2023-01-11 @ 18:00 CST
End: 2023-01-11 @ 22:00 CST

Our data center provider will be conducting core network upgrade and maintenance during the above window. Our Managed Web and Mail Service will experience some packet loss and downtime during this period. We expect only a few minutes of downtime.

If you have any questions, please feel free to contact us.

— Mad Scientist Technologies Team

Maintenance Window (Rescheduled)

Start: 2022-08-19 @ 12:01AM CDT
End: 2022-08-19 @ 1:00AM CDT

We will be performing server maintenance on the following services:

  • Managed Web Platform
  • Managed Mail Service

We expect no more than ten (10) minutes of downtime with this maintenance window.

If you have any questions, please feel free to contact us.

Webmail Upgrade

Start: 2022-08-17 @ 5:00PM CDT
End: 2022-08-17 @ 6:00PM CDT

We will be upgrading out webmail interface. This upgrade will not effect mail delivery to your mailboxes.

We expect about 30 minutes of downtime to the webmail interface during this upgrade process.

If you have questions, please feel free to contact us.

Server Maintenance

Start: 2022-01-01 @ 11:00PM CDT
End: 2022-01-02 @ 03:00AM CDT

We will be performing some server maintenance during the above maintenance window.

We expect less than 30 minutes of downtime for the following services:

Affected Service

  • Managed Web Services
  • Managed Mail Services

If you have questions or concerns, please feel free to contact us.

Web and Mail Service Outage

On August 13 starting about 3:38PM CT, our Managed Web and Managed Mail Services went offline. The issue is with our data center provider, which started an investigation at 3:50PM CT. Our service came back online around 3:52PM CT.

We will continue to monitor the situation and update this post with additional detail when available.


Update: August 13, 2021 at 10:00PM

Services are currently stable, however our Data Center Provider will be doing maintenance on network equipment in the emergency maintenance window below:

August 14, 2021 12:00AM - August 14, 2021 4:00AM.

During this maintenance windows there may brief moment of packet loss and/or increase latency.

If you have any questions, please feel free to contact us.

Outage: SDN Controller Unavailable

Start: 2021-07-16 at 3:15PM CDT End: 2021-07-16 at 11:50PM CDT

During a normal maintenance window our Software Defined Network (SDN) Controller went offline for upgrades, however it failed to start after the upgrade.

We are in the process of recovery the controller.

What does that mean we manage your network devices, we are unable to change settings on your devices. However, the devices do not need the SDN Controller available to continue function. Your equipment will continue to work, however any statistics will be lost.

We will update this post with additional information, when it becomes available.


Update

  • 2021-07-16 at 3:50PM CDT
    We have found the cause of the problem and are attempting a fix now.
  • 2021-07-16 at 11:50PM CDT
    We have resolved the issue with our SDN Controller and has returned to nominal state.

If you have questions or concerns, please feel free to contact us.

Outage on June 8, 2021

On June 8, 2021 starting at approximately 5:39PM CDT, our alerting system informed us that are primary web and database server became unresponsive from the greater Internet.

Upon investigation, we found that our Data Center (DC) provider is having a connectivity issue across there entire global footprint. The DC provider has started investigating this large scale issue.

As of 11:30PM on June 8, 2021 we do not have any further information to share.

UPDATE: June 8, 2021 @ 11:55PM: Web Sites & Services Started to be restored and coming back online. We are still waiting for additional information from the DC about these restorations. We are assuming this may be temporary and some or all services may become intermittent at times. Additional Updates when available.

Once we know more, will update this post and our twitter account.

Timeline of Events (Time in Central Time)

  • June 8, 2021
    • 5:39PM: Monitoring system indicate multiple failures in our web services.
    • 6:20PM: Investigation into cause, found possible DC Provider failure.
    • 9:00PM: DC Provider investigating outage
    • 11:55PM: Web Sites and Service are coming back online.

Effected Service

  • Managed Web Service
  • Managed Mail Service
  • Incoming mail is spooling on backup servers, will be delivered when back online
  • Server is accessible, but authentication and delivery failures.
  • Online Invoice & Billing System

When additional information is provided, we will publish updates here and on twitter.

If you have questions, please feel free to contact us.

— Mad Scientist Technologies Team

Web Server Mail Delivery

On May 20, 2021 we were informed that our Managed Web Service Server has been blacklisted from sending emails directly. This was a change by Spamhaus by placing the server address on a Policy Based Blacklist due to bad neighbors in our data center provider.

We are looking into options to remove our IP address from Spamhaus - this is unlikely to happen.

However, we have decided to continue to implement our plans to switch to using our third-party mail delivery service for all web applications.

We will post updates on this situation as they come available.

If you have any questions, please feel free to contact us.

— Mad Scientist Technologies’ Team

Server Maintenance

Start: 2021-01-02 @ 11:00PM CDT
End: 2021-01-03 @ 03:00AM CDT

We will be performing some server maintenance during the above maintenance window.

We expect less than 10 minutes of downtime for the following services:

Affected Service

  • Managed Web Services
  • Managed Mail Services

If you have questions or concerns, please feel free to contact us.

Internal Data Center Updates

Start: 2020-09-18 @ 03:30 UTC / September 17, 2020 @ 10:30 PM CDT
End: 2020-09-18 @ 04:30 UTC / September 17, 2020 @ 11:30 PM CDT

Our Data Center Provider will be updating internal tooling that will limit our ability to manage clients servers during the above maintenance window.

We expect no downtime from this maintenance window for all client and managed services.

If you have questions or concerns, please feel free to contact us.