We did some extensive systems upgrades earlier today and some customers are experiencing difficulties logging onto their VPN service. We’re all over it, but no ETA as yet.
4:05 am update is that they are still eliminating possible causes. Obviously, this is likely related to upgrades, but systems were behaving optimally after initial relaunch of systems, and there are many successfully logged in, so it’s more difficult to troubleshoot. will continue updating as news received.
4:45 am update — network engineers are readying a new config they will push out to network in an attempt to resolve intermittent authentication issues some customers continue to experience.
7:41 am update — we have discovered the issue (the first attempt earlier did not resolve the problem) and the fix is propagating throughout network. There may still be some occasional issues for some customers, but the majority of those affected should start to be able to authenticate normally again.
8:38 am update — solution has been pushed out to entire network for a bit and life is good again. A big apology to those affected. This was a particularly sneaky issue.
9:52 am update — unfortunately the issue has returned. network engineers are on it.
12:31 PM — We realize we have said it before, but we believe we may have identified the cause. Not certain as yet but it makes sense as to why the servers reverted back to not passing authentication traffic as they should after they were fixed and why openVPN service customers were seemingly never affected. Tragically, it may not be a lighting quick fix even if the cause, but we should know more very soon…as in minutes, not hours.
1:42 pm — we have consensus on the cause and a fix is being implemented. and, although nothing is 100%, we’re more sure than last time we were sure. And, we were mostly right last time, but we didn’t go far enough “upstream.” You can’t always just shoot the gator closest to the boat I suppose is the lesson here. Yes… we’re all a bit punchy and these types of old sayings make one feel less stressed no matter how ill-fitting to the situation they might be.
1:51 pm — the fix was working , but got stalled by a non-responsive server. we’re adjusting it so that doesn’t happen again so we can push it out to whole network without it being hung up.
1:53 — and, just in case it wasn’t clear from support…if you have an openVPN account with us..they are unaffected and should work fine. This issue is limited to PPTP, L2TP, IPsec authentication.
2:21 — ok..we hope for the beginning of significant improvement in next 15 minutes. It will take a bit for things to settle down.