We will be installing a critical security update on our servers during this maintenance window. There may be up to 5 minutes of downtime with this maintenance while servers restart.
On Sunday, April 25, 2016 @ 1:00AM, we will be doing system maintenance for our Managed Services - for system updates. We expect between 5-15 Minutes of downtime with this maintenance window. The managed services will be unavailable for a few minutes while the servers restart. The services provided by each server will be unavailable for a short period of time while they restart.
TL;DR: We setup stuff to make imposter mail accounts a thing of the past. Scroll to the end of the article for the rollout schedule for your service(s).
We are planning to implement DMARC (Domain-based Message Authentication, Reporting & Conformance) Records across all our own and managed mail domains in the coming weeks.
DMARC is an Internet Standard that is used to confirm that a email messages are truely from who it say it’s from. For example, when you a send email message to another mail server, the remote mail server can use DMARC records to confirm that your message is really from your mail server and not an imposter. If the message is not from the authorize mail server it will quarantine and send us reports on why it failed.
This implementation process will take a few weeks to implement and should not cause too many problems for our managed mail customers.
Our clients that have both Managed Mail and Web Presences on the other hand will have a slower rollout. The reason is with DMARC you must know and list all authorized hosts that can send email on behalf of your domain. If you use Email Service Providers (such as MailChimp, Amazon Web Services - Simple Email Service, Campaign Monitor, etc), web forms that send emails, and then your actual mail accounts - all these source must be included in the set of records that make DMARC.
We will reach out to each our Managed Web Presence Clients individually to confirm issues will not arise during this process.
The below is the tentative schedule for the implementation of DMARC on our managed services.
Managed Mail Clients Some properties may ramp faster, depending reports
April 18, 2016: Initial Records Set (Monitor Only)
May 2, 2016: Set to Quarantine 1% of Email that does not conform.
May 5, 2016: Quarantine 5%
May 9, 2016: Quarantine 10%
May 12, 2016: Quarantine 25%
May 23, 2016: Quarantine 50%
May 30, 2016: Quarantine 75%
June 13, 2016: Quarantine All - Mid Point
June 16, 2016: Reject 1%
June 23, 2016: Reject 5%
June 30, 2016: Reject 10%
July 5, 2016: Reject 25%
July 7, 2016: Reject 50%
July 11, 2016: Reject 75%
July 18, 2016: Reject All - Complete
Managed Web Presence & Mail Client The ramp for these properties will be slower, depends on reports
April 21, 2016: Initial Record Set (Monitor Only)
May 5, 2016: Set to Quarantine 1% of Email that does not conform.
May 12, 2016: Quarantine 5%
May 19, 2016: Quarantine 10%
May 26, 2016: Quarantine 25%
June 2, 2016: Quarantine 50%
June 6, 2016: Quarantine 75%
June 9, 2016: Quarantine All - Mid Point
June 20, 2016: Reject 1%
June 23, 2016: Reject 5%
June 27, 2016: Reject 10%
June 30, 2016: Reject 25%
July 7, 2016: Reject 50%
July 14, 2016: Reject 75%
July 21, 2016: Reject All - Complete
This schedule is tentative and may vary per client needs or results or the reporting mechanism of the DMARC Standard. We will report to each client when their implementation reach the following milestones:
Quarantine 50%
Quarantine All
Reject 25%
Reject 75%
Reject All
If you have questions about this process, implementation, or any other questions, please feel free to reach out to us.
As of April 17, 2016 at 10:15PM CDT. We started recieved reports that email messages destined for Hotmail/Outlook.com email addresses were not being delivered to Hotmail/Outlook.com email accounts.
We have reached out to Microsoft, but we must wait for them to gather and report data back to us. This process may take up to 72 hours. We would like to apologize for this issue and the inconvenience this has caused our valued clients.
We will continue to see if we can resolve the problem from our side.
We will share more information as we get it.
April 18 @ 6:00AM
We may have found the problem. It seems that our configuration for DKIM verification may have an error in it. We are currently seeing what can be done to fix the issue, without reissuing DKIM Keys.
April 18 @ 8:50AM
The cause of the problem is caused by an configuration error on our DKIM configuration. Apparently when the server signed outgoing mail the domains’ DKIM it put the wrong DNS name to verify the DKIM key.
We will be rolling out a fix, shortly.
We will update this post with more information as it comes in.
April 18 @ 11:30PM
We have rekeyed our DKIM implementation and updated SPF Records for each client that is effected by this change. If we manage your DNS Records you have nothing to worry about, however if you manage your own DNS Records, we will be in contact with you to update your DKIM and SPF records.
As soon as these new DNS Records have propagated, we will be able to confirm that this issue is cleaned up.
We would like to apologize again for this issue and any inconvenience that it has caused you.
April 19-20
Update: We have discovered that Microsoft’s Hotmail.com, Outlook.com, and Live.com email accounts do not accept mail from new or small email server and been a documented issue with Microsoft Email Service since 2010.
We will be reaching out to Microsoft to request an exception for our clients and our mail server. This process may take a few business days to sort out.
We will share more information as we receive it, stay tune.
Since 7:30AM CDT April 9, our data center provider has been experiencing network interruptions that has been causing connection issue to out managed services located in Los Angeles data center. We will update this post, when we have more information.
Update: As of April 9, 6:00PM CDT, The Data Center provider has not provided any status or update on this issue. We will update this post, as soon as we have more information to share.
Update: April 10 7:30AM CDT - Our Los Angeles data center provider still has not provided us any type of update. However, systems seem to be “stable” at this time. We will continue to closely monitor these services until we have an all clear notice. As always, we will update this post with the latest information concerning this issue.
Update: April 11 12:50PM CDT - Still no update from our Los Angeles data center provider. Our monitoring services have not detected any additional outages since April 9. This issue will remain open until our data center provider close it, however unofficially this issue is close. As always, we will continue to monitor the servers and update this site with updates.
If you have questions, please feel free to contact us at our website.
Our Datacenter provider has reported that they are having problems with connections to their DNS Servers and are current investigating the problem.
Our servers are still online, but with DNS unavailable connections/clients may be unable to connect to the site.
We will update this post with more details, as soon as we have more details.
March 24, 2016 @ 10:45AM
Our Datacenter provider has identity the failure and are working to resolve the connectivity issue their DNS Servers.
March 24, 2016 @ 11:45AM
DNS services have been partially restored, we will continue to monitor the situation and update.
March 24, 2016 - Postmortem
Our New York Data Center Provider has posted an postmortem of the cause of today’s DNS failure. The cause was due to a increase demand upon their DNS Servers Clusters - Distributed Denial of Service Attack upon the providers DNS Infrastructure. The provider will be changing their DNS Infrastructure to improve the time to mitigate DNS DDoS Attack in the future.
We are planning to do a very quick system reboot on March 23, 2016 at 11:59PM CDT March 25, 2016 at 11:59PM CDT. We are doing this to complete the installation of some security patches. We expect only 1-2 minutes of downtime while our servers restart.
March 23, 2016 at 11:59PM: 1-2 Minutes of Downtime
March 25, 2016 at 11:59PM: 1-2 Minuntes of Downtime
We will be doing some internal network changes to prepare for our new mail service. Our client sites located on our new server may experience a few minutes of down time during this period, we would like to apologize in advance for the inconvenience.
There will be some network maintenance by our data center provider to the edge routers. This maintenance is being done to resolve some bugs and to increase performance of the routers.
We expect minimal impact to services, there may be minor latency and packet loss while the connections are re-routed.
Maintenance Windows:
Start: 2016-03-08 03:00 UTC (2016-03-07 10:00 EST)