Maintenance should start promptly at 6PM EDT (UTC-4:00) and may last up to an hour. PPTP/L2TP/IPsec users may not be able to login during the maintenance. openVPN users will be unaffected.
We’ve been shamefully neglectful of our blog for far too long. And, truth be told, we’re still not quite ready to unleash the deliciously insightful and tantalizing topics we envision ourselves sharing with you. Encryption is gripping stuff after all.
We may even wait for our planned website refresh before we really get serious about it. Probably a good a reason as any to procrastinate a bit more. So, this is likely really just a little post to let you know that we’re still totally seriously committed to thinking about it.
You’ve been warned.
Our monitoring is showing most hosts coming back up as reachable. Hopefully, just “one of those Internet things” and it’s behind us.
Fiber cut or Internet outage? Major Internet-wide routing issue/outage to Europe affecting multiple carriers.
Our monitoring appears to have detected what is a MAJOR routing issue to Europe on the Internet backbone. We’re investigating but this appears to be an Internet-wide issue as it affects multiple carrier networks. The result for WiTopia customers, at a minimum, is that a number of our gateways are unreachable, or will be very slow to connect from the US. As you can see from Internet traffic report many hosts are showing 100% packet loss which, in practical terms, is an outage. Again, this affects all Internet users, not just WiTopia users, and we will continue to investigate the source as reports are scarce at this point. Could be a fiber cut, could be a routing issue, etc. Even if some routes exist, they will quickly become congested as the Internet traffic defaults to the open routes.
One of our providers in Los Angeles is having a major outage. Because we plan for such things (we have four separate and distinct data center providers in LA) most customers will be unaffected. If you were issued a static IP address from this provider, you will be unable to connect to this IP address and should try a different gateway close by.
We did some extensive systems upgrades earlier today and some customers are experiencing difficulties logging onto their VPN service. We’re all over it, but no ETA as yet.
4:05 am update is that they are still eliminating possible causes. Obviously, this is likely related to upgrades, but systems were behaving optimally after initial relaunch of systems, and there are many successfully logged in, so it’s more difficult to troubleshoot. will continue updating as news received.
4:45 am update — network engineers are readying a new config they will push out to network in an attempt to resolve intermittent authentication issues some customers continue to experience.
7:41 am update — we have discovered the issue (the first attempt earlier did not resolve the problem) and the fix is propagating throughout network. There may still be some occasional issues for some customers, but the majority of those affected should start to be able to authenticate normally again.
8:38 am update — solution has been pushed out to entire network for a bit and life is good again. A big apology to those affected. This was a particularly sneaky issue.
9:52 am update — unfortunately the issue has returned. network engineers are on it.
12:31 PM — We realize we have said it before, but we believe we may have identified the cause. Not certain as yet but it makes sense as to why the servers reverted back to not passing authentication traffic as they should after they were fixed and why openVPN service customers were seemingly never affected. Tragically, it may not be a lighting quick fix even if the cause, but we should know more very soon…as in minutes, not hours.
1:42 pm — we have consensus on the cause and a fix is being implemented. and, although nothing is 100%, we’re more sure than last time we were sure. And, we were mostly right last time, but we didn’t go far enough “upstream.” You can’t always just shoot the gator closest to the boat I suppose is the lesson here. Yes… we’re all a bit punchy and these types of old sayings make one feel less stressed no matter how ill-fitting to the situation they might be.
1:51 pm — the fix was working , but got stalled by a non-responsive server. we’re adjusting it so that doesn’t happen again so we can push it out to whole network without it being hung up.
1:53 — and, just in case it wasn’t clear from support…if you have an openVPN account with us..they are unaffected and should work fine. This issue is limited to PPTP, L2TP, IPsec authentication.
2:21 — ok..we hope for the beginning of significant improvement in next 15 minutes. It will take a bit for things to settle down.