Come sit by my me, boys and girls, and let me spin you a yarn.
October 24, 2006. Fedora Core 6. Zod conquered.
fedoraproject.org -- KNEEL BEFORE ZOD!
fedora.redhat.com -- KNEEL BEFORE ZOD!
www.redhat.com -- EVEN YOU, WITH YOUR MIGHTY BANDWIDTH, MUST KNEEL BEFORE ZOD!
Zod took down everything. It was a week before the websites were back up and functioning like they should.
And in the back offices of Red Hat, many questions were asked to your Fearless Leader.
"Fearless Leader, how did Fedora's traffic manage to crash redhat.com?"
"Fearless Leader, do you know how much money Red Hat loses if people cannot get to the store to buy subscriptions?"
"Fearless Leader, do you realize that some parts of Red Hat have service level agreements that do not contain an 'Zod melted the server' clause?"
"Fearless Leader, are you trying to put Red Hat out of business?"
"Fearless Leader, surely this will never happen again, right?"
Well, as it turns out, no, it didn't happen again.
Zod has stepped aside, or been distracted by a nice big jug of Moonshine, and here we are 7 hours after the release, and everything is still humming.
Red Hat, Fedora, Fedora Wiki, Fedora Docs, Fedora Planet, torrents, etc. Everything is still up, and (at least as far as I can tell) as fast as always. And the even better part is, that there was almost zero downtime along the way.
Credit for this falls completely into the hands of our very dedicated, and very skilled Fedora Infrastructure team.
Did you know that almost all of the people on that team are volunteers? Did you know that the volunteer group that we have is so geographically diversified that we almost have 24x7 sysadmin coverage of Fedora infrastructure machines?
Were you in #fedora-admin today? Did you see the nonstop work that these guys put in for several hours, tweaking firewall rules to help with connection throttling, managing Xen instances in order to take down build boxes (we're not compiling anything on release day) and using that infrastructure instead to bring up additional proxy servers to better manage the web load?
Did you know that the Fedora Infrastrucutre team has been working for months on the problems that we faced at the FC6 release in order to prevent anything like that from happening today? A new mirroring system, the death of fedora.redhat.com, more flexibile infrastructure as our Xen usage has been built out.
Do you know that Fedora Infrastructure is not only a world class example of how to manage a global system administration project, but that they are also one of the best examples of "eating our own dog food" based on the amount of Fedora and Xen that they use in their infrastructure?
Did you hug your sysadmin today?
-
- Max (
spevack) wrote,