We use Duo at work for 2fa. I am laughing my ass off, I literally can't do anything -- can't get on the VPN, can't get into my emails, can't access company services. They locked everything down so hard, they've literally chained themselves to a radiator and tossed the key out of reach. We didn't receive internal communications about the outage yet -- and my bet is it's because whoever's in charge of that, is locked out due to the outage.
This is also an unfortunate coincidence, as today is Georgia Tech's first day of classes, and the institution's Single Sign-On (SSO) for almost all its services, such as the Canvas LMS, registration, and the Bursar, is based on Duo's service. Right now it seems to be having some issues and is timing out / not logging in.
However, I'm not sure if GT's Duo service is self-hosted or is hooked into Duo's service "in the cloud".
Duo seems to have had lots of success, selling into higher education. To add to the other schools mentioned in this thread, Oregon State, and Stanford both use it too. And lots of schools are starting class today, which might be what contributed to their downtime.
To be honest, their useR experience is pretty slick compared to most two factor solutions. I am happy for them, (and their users) that they have been successful, selling into large organizations – good user experience usually isn’t found in large enterprise software due in part to the principal agent problem.
My duo workflow was figuring out I can request an SMS OTP and read the OTP code as a notification on my laptop instead of unlocking my phone and confirming via the app. Hopefully they get volume pricing on texts.
This kind of workflow is probably against your workplace's rules. This bypasses one of the protections meant through something like a Duo app locked behind a device password screen; which is that even if your laptop is logged in and your password manager is running, a bad actor still couldn't get into protected things if they don't know your phone passcode.
I am reading this, slacking off, because I couldn't get into Georgia Tech's Canvas. Judging by the status.gatech.edu message, it seems like changing the instance of Duo that responses is a quick fix.
I'm actually taking a course there right now, as part of my pointless masochistic love of continuing education in stuff I like knowing to be well-rounded but no one will ever give a crap about professionally, and I logged in fine. Duo's push notifications seemed to stop working, but it lets you fall back on the TOTP passcode, which continued working.
UC (Cincinnati, not California) first day of classes is today too, I always hate that out IT doesn't allow 2FA except Due. I always wanted to request an exception, but my previous experience with them refusing to allow IMAP access to my email because it is not secure!.
> Did you notice that 2FA went away around 10 am? I worked on the team that made that happen. Its back now that the outage is over.
Yes, altough I did not get affected that much. I was just trying to renew overdue book from the library system. But I'm sure ~45k student got affected harshly in their first day of the semester.
> We do have an app that lets faculty exempt students from 2FA, but it's mostly for students who need the exception when taking tests etc.
It would be better if there is an option to allow graduate students who do research (PhD candidates) to be treated differently as they are not students anyway. And Duo is annoying. I understand that this is not something that will happen specially with the reputition of UC IT department (sorry but you problably know)
> You are going to see a big change in the tech at UC very soon. The old guard is getting the boot.
This is something I have been hearing since I joined but without the old guard getting the boot part. Each year with increasing student enrollment, we can't even provide stable internet connection. I still remember two years ago the outage of the auth server for wifi system on the first day of classes (after covid) and this stayed the case for almost a week.
University of Kentucky too, first day of classes, no one can get into anything they aren't already logged in to with a valid cookie.
I spent 20 minutes trying to figure out what new cookie I needed to grey-list for the half dozen redirections in the M365 auth flow to not bork before I thought to check if it was generally broken.
This doesn’t mean GT is offline. There are a couple dozen Duo server #’s and this only affects DUO1. If GT is on DUO5 or DUO19 or DUO25 they’ll be fine today. It does raise the question of how to provide continuity of services if your server is offline though.
There are a lot more other problems that happen when each service manages their own authentication. The move to SSO has been in response to problems that existed then.
I was among the last to have to use dial-up from off-campus (which required a specialty ISP @ 26.6k or some awful speed). Fortunately, they upgraded their system while I was there, so I could use the “much” faster DSL. Still not as nice as on-campus internet.
Is it really discontinued yet? If anything it's an ongoing naming blunder with no end in sight.
There are two products called Google Meet now. The web interface for the former Google Duo is duo.google.com, just rebranded to Google Meet. The former Google Meet still exists on meet.google.com. Both also have their own Android apps - one called Meet and other Meet (original). Both products have a different set of features and neither completely replaces the other.
The original Duo app was named "Meet (original)" on Android phones, but the old Duo icon remained until you launched the app (so it could inform you about the name change). Then the Google Meet app was introduced. This resulted in phones having three icons ("Duo", "Meet", "Meet (original)") for the same service.
People really don't understand 2FA codes. Imagine trying to tell thousands of students to get the code from their 2FA app (Which app?). What happens when a student goes home over the summer and gets a new phone, but doesn't transfer the app info? Duo offers a level of management that other apps don't. If a student is struggling, you can send them a text with a direct link to the app they need to download. You can temporarily bypass 2FA from the Duo console. For the longest time, it was the only 2FA app that offered any kind of management. Okta has it now, too, but most higher ed already has a different SSO provider, so switching to Okta just to get 2FA management (And I'm not sure it's as good as Duo's) is probably an impossible task to get off the ground.
Okta offers a similar feature. So much easier to click a confirmation on my phone than to scroll through dozens of 2FA codes (some of which might be orphaned).
This implementation sounds better. Though for me I still have to manually input a code from the Duo app (that doesn't auto refresh after code entry since it's not time based).
Having the do the manual entry and the lack of refresh is a choice of your security team/administrator. Duo supports push notifications and auto-refreshing TOTPs.
Indeed. They are generally understaffed and salaries are very low so they're very lucky to get any "1x-5x" developers who stick around long enough to understand the infrastructure. Outsourcing as much as possible makes a lot of sense in that environment, it does create major single points of failure but "roll-your-own" would likely fail more often anyways.
And the proud Illinois tradition of some mission-critical service crashing on the first day of class continues.
In this case, it is an external service. However, I also suspect that the Duo outage is probably shielding other on-campus services from load surges that would probably be causing them to get crashy.
I guess I don't know how we could ever prevent such incidents. Given that the first day of classes is a well-kept secret /s.
I love that the status page is being updated regularly.
But I have no idea what the difference is between DUO1, DUO2, etc. through DUO73. I feel like they should have a better way to clarify which users are affected.
They're all identical deployments of their whole stack. They shard customers onto the deployments to reduce blast radius. In your Duo admin dashboard, look for "Deployment ID" in the left pane.
They published the postmortem, and the penetration into colleges is hinted at as being causal.
> Increased load on DUO1 due to significantly increased adoption and simultaneous peak usage across multiple larger customers led to authentication failures.
I work in post-sec and this is very common practice. There are few key players that tend to capture the majority of schools in the States/Canada for specific tech solutions. Blackboard/Canvas/D2L for LSMs, Shibboleth for SSO, Duo for 2FA, Cisco AnyConnect for VPNs.
tech solutions in the field tend to be incredibly low risk given the size and make-up of the anticipated users (enterprise services with thousands of employees and tens of thousands of students). For public institutions, there's the added element of public sector risk avoidance.
To be clear, Shibboleth is often self-hosted and usually the grey-beards understand how to maintain it. It's been around a long time and is very stable/robust and at least as unlikely to fail as Duo/Cisco (which are overall fairly robust with rare enough breaking failures). OTOH, rolling their own 2FA would likely create points of failure that rear their ugly head more often, not less often.
Shibboleth is kind of an outlier here, due to its age/maturity and position as a very old-school piece of foundational tech that got implemented when academic IT salaries were quite a bit easier to live on than they are today.
The disparity between tech salaries in academic institutions and FAANG/SaaS corps has grown immensely in the past 20 years. Most of the people who do the real work at academic institutions have been employed there for 25-40 years. Most of the young people can't stick around for long because they need to earn more money to build a stable life.
