ninjette.org

Go Back   ninjette.org > Forum Info > Forum Information

Notices

Reply
 
Thread Tools
Old August 11th, 2011, 12:37 AM   #1
Alex
ninjette.org dude
 
Alex's Avatar
 
Name: 1 guess :-)
Location: SF Bay Area
Join Date: Jun 2008

Motorcycle(s): '13 Ninja 300 (white, the fastest color!), '13 R1200RT, '14 CRF250L, '12 TT-R125LE, '15 CRF110F, '13 TT-R50E

Posts: Too much.
Blog Entries: 7
Well that sucked. (Outage)

As you may have noticed, the site was down almost all day today. We are hosted at a Tier 1 co-lo datacenter near Dallas. They have redundant power, generators with days of fuel, UPS's, and all of the other things that you'd expect a modern datacenter to have. But - that doesn't mean unexpected things don't happen. One of the key pieces of equipment between the power grid, those backup generators, and the UPS's, is evidently something called the Automated Transfer Switch (ATS). And while there are multiple ones on site, it turns out that they aren't truly redundant. When one failed catastrophically today, it took down a very large portion of the data center, and getting it back up involved some significant engineering and re-wiring.

Once the datacenter itself got its power back up and running to the cages, then the servers themselves needed to be confirmed good. For an unknown reason, the one that we were hosting ninjette.org on was behaving strangely when it came back up. It was "up", but not visible from the internet. After troubleshooting that for a few hours this evening, I was able to migrate all of our content (the entire VPS) onto a known good server, and voila, we're up.

Sorry for the trouble, and glad things like this happen as rarely as they do. If anything does look like it's behaving strangely here, please let me know, but I don't expect any issues. Time for bed...
__________________________________________________
Montgomery Street Motorcycle Club / cal24.com / crf250l.org / ninjette.org

ninjette.org Terms of Service

Shopping for motorcycle parts or equipment? Come here first.

The friendliest Ninja 250R/300/400 forum on the internet! (especially Unregistered)
Alex is offline   Reply With Quote




Old August 11th, 2011, 12:40 AM   #2
Alex
ninjette.org dude
 
Alex's Avatar
 
Name: 1 guess :-)
Location: SF Bay Area
Join Date: Jun 2008

Motorcycle(s): '13 Ninja 300 (white, the fastest color!), '13 R1200RT, '14 CRF250L, '12 TT-R125LE, '15 CRF110F, '13 TT-R50E

Posts: Too much.
Blog Entries: 7
For a more detailed analysis, check out this thread on the webhostingtalk forum.

I was glad that we had the facebook ninjette.org group to get some communications out during the day as well. If you're on facebook, consider joining that group for days just like these. Though hopefully they are few and far between...
__________________________________________________
Montgomery Street Motorcycle Club / cal24.com / crf250l.org / ninjette.org

ninjette.org Terms of Service

Shopping for motorcycle parts or equipment? Come here first.

The friendliest Ninja 250R/300/400 forum on the internet! (especially Unregistered)
Alex is offline   Reply With Quote


Old August 11th, 2011, 12:42 AM   #3
Skippii
Milkshake Drinker
 
Skippii's Avatar
 
Name: Skippii
Location: Richmond, Va
Join Date: Aug 2011

Motorcycle(s): 2007 Orange DRZ400-S, 2005 Ninja 250 & Custom Thundercunt Dirt Chopper

Posts: A lot.
Cool

Phew. I had JUST joined this site, so I figured you were taking extreme measures to keep me from posting...
__________________________________________________
"What a sweetheart."- ninja_sleeper to me
Skippii is offline   Reply With Quote


Old August 11th, 2011, 12:43 AM   #4
Alex
ninjette.org dude
 
Alex's Avatar
 
Name: 1 guess :-)
Location: SF Bay Area
Join Date: Jun 2008

Motorcycle(s): '13 Ninja 300 (white, the fastest color!), '13 R1200RT, '14 CRF250L, '12 TT-R125LE, '15 CRF110F, '13 TT-R50E

Posts: Too much.
Blog Entries: 7
You can never be too careful.
__________________________________________________
Montgomery Street Motorcycle Club / cal24.com / crf250l.org / ninjette.org

ninjette.org Terms of Service

Shopping for motorcycle parts or equipment? Come here first.

The friendliest Ninja 250R/300/400 forum on the internet! (especially Unregistered)
Alex is offline   Reply With Quote


Old August 11th, 2011, 01:43 AM   #5
akima
Nooblet
 
akima's Avatar
 
Name: Akima
Location: England
Join Date: Jul 2011

Motorcycle(s): 2011 Ninja 250R FI

Posts: A lot.
Blog Entries: 5
MOTM - Oct '13
The email company I use was down for the same period of time as Ninjette.org. They use colo4.com Dallas co-location . I suspected you were using them too. I bet the people in the co-location centre were running around like maniacs trying to get everything fixed
akima is offline   Reply With Quote


Old August 11th, 2011, 01:50 AM   #6
Snake
ninjette.org certified postwhore
 
Snake's Avatar
 
Name: Rick
Location: Alexandria, Louisiana
Join Date: Jan 2009

Motorcycle(s): 05 Blue Ninja 250

Posts: Too much.
MOTY - 2017, MOTM - Jan '19, Oct '16, May '14
I was starting to get the shakes. I felt like an addict needing to get a fix.
Snake is offline   Reply With Quote


Old August 11th, 2011, 02:58 AM   #7
ally99
Ninja chick
 
ally99's Avatar
 
Name: Allyson
Location: Athens, GA
Join Date: Jun 2009

Motorcycle(s): '13 Ninja 300

Posts: A lot.
Blog Entries: 4
MOTM - Dec '13, Feb '15
Yeah, Ninjette withdrawals SUCK!
__________________________________________________
Sometimes it's the journey that teaches you a lot about your destination. ~Drake

Check out my Appalachian Trail journal, 2015!

Postwhores are COOL! ~Allyson
ally99 is offline   Reply With Quote


Old August 11th, 2011, 03:05 AM   #8
Racer x
ninjette.org certified postwhore
 
Racer x's Avatar
 
Name: Eric
Location: Iowa City
Join Date: May 2009

Motorcycle(s): 2008 Kawmeracchi 350 2010 Project X

Posts: A lot.
Blog Entries: 54
MOTM - Sep '18, Feb '16
Quote:
Originally Posted by Skippii View Post
Phew. I had JUST joined this site, so I figured you were taking extreme measures to keep me from posting...
Hi Skippy welcome
Racer x is offline   Reply With Quote


Old August 11th, 2011, 03:07 AM   #9
the big mike
ninjette.org sage
 
the big mike's Avatar
 
Name: Michael
Location: Belgium
Join Date: Apr 2011

Motorcycle(s): 2008 Ninja 250r, Fazer8

Posts: 580
there is a facebook group? I'm getting there right now
__________________________________________________
I can not be a true player by playing around.
the big mike is offline   Reply With Quote


Old August 11th, 2011, 04:28 AM   #10
TnNinjaGirl
Ms. Personality
 
TnNinjaGirl's Avatar
 
Name: CB
Location: Murvill, TN
Join Date: Jan 2009

Motorcycle(s): Depends on the week you ask

Posts: A lot.
There needs to be an NA (Ninjetters Anonymous) group if you ask me.
TnNinjaGirl is offline   Reply With Quote


Old August 11th, 2011, 04:36 AM   #11
dubojr1
ninjette.org certified postwhore
 
dubojr1's Avatar
 
Name: Jason
Location: Jacksonville, Florida
Join Date: May 2010

Motorcycle(s): 09 Ninja 250R SE, 07 Honda CRF230F, 06 Honda CRF150F

Posts: A lot.
Quote:
Originally Posted by Snake View Post
I was starting to get the shakes. I felt like an addict needing to get a fix.
Holy cow.... I was thinking the same thing. I was checking to see if the site was back up like every 5 mins at work and even continued checking about every half hour at home. I generally spend my time here while at work but my wife even noticed my nervousness questioned what was happening when I was grabbing for my iPhone so often.

I'm glad we are back up!
__________________________________________________
09 250R SE: Paying it forward one post at a time!

Don't forget to add yourself to our Member Map
dubojr1 is offline   Reply With Quote


Old August 11th, 2011, 06:23 AM   #12
Azhyen
ninjette.org Pirate
 
Azhyen's Avatar
 
Name: Seph
Location: San Diego
Join Date: Nov 2010

Motorcycle(s): 2009 Ninja 250r

Posts: 301
I got SOOOOO much work done yesterday...not that I'm correlating or anything,
__________________________________________________
"She said I can't get a bike, so I dumped her."
Azhyen is offline   Reply With Quote


Old August 11th, 2011, 07:25 AM   #13
CC Cowboy
ninjette.org certified postwhore
 
CC Cowboy's Avatar
 
Name: Whodat
Location: Ware Is.,MA
Join Date: Jan 2009

Motorcycle(s): I pass the wind!

Posts: A lot.
MOTM - Mar '13, Jun '14
I thought Kelly had hacked in and shut down the site.
__________________________________________________
If everything seems under control; you're just not going fast enough!

Last futzed with by CC Cowboy; August 12th, 2011 at 02:08 PM.
CC Cowboy is offline   Reply With Quote


Old August 11th, 2011, 07:37 AM   #14
caps
Sexy Beast
 
caps's Avatar
 
Name: Mat
Location: Tampa
Join Date: Mar 2011

Motorcycle(s): 2010 Ninja 250R, 2006 ZX-14, 2012 Ducati Monster

Posts: A lot.
Quote:
Originally Posted by Snake View Post
I was starting to get the shakes. I felt like an addict needing to get a fix.
this is EXACTLY how I felt
__________________________________________________
2010 Ninja 250
2006 Ninja ZX-14
^Both Links
caps is offline   Reply With Quote


Old August 11th, 2011, 08:33 AM   #15
Linuss
ninjette.org guru
 
Name: Steve
Location: DFW
Join Date: Apr 2011

Motorcycle(s): 2010 Ninja 250R

Posts: 294
Quote:
Originally Posted by Alex View Post
running to the cages

Cagers... always ruining it for motorcyclist.
Linuss is offline   Reply With Quote


Old August 11th, 2011, 09:17 AM   #16
Palero
ninjette.org member
 
Palero's Avatar
 
Name: Mike
Location: Los Angeles, California
Join Date: Aug 2011

Motorcycle(s): 1988 Honda Hawk GT 1989 EX250F (Died from left-turning car) 2011 EX250J (Died from SAME left-turning car)

Posts: 51
Quote:
Originally Posted by Linuss View Post
Cagers... always ruining it for motorcyclist.
Heh, I work at at Data Center like the one Ninjette.org live in-- not all Data Center co-location workers strictly commute in cages
Palero is offline   Reply With Quote


Old August 11th, 2011, 09:39 AM   #17
nickjpass
#squid
 
nickjpass's Avatar
 
Name: nickypoo
Location: Five Guys
Join Date: Jul 2011

Motorcycle(s): Track dedicated 2008 ZX6R

Posts: A lot.
MOTM - Jul '16
So how did you guys deal with the outage?
__________________________________________________
nickjpass is offline   Reply With Quote


Old August 11th, 2011, 09:42 AM   #18
dubojr1
ninjette.org certified postwhore
 
dubojr1's Avatar
 
Name: Jason
Location: Jacksonville, Florida
Join Date: May 2010

Motorcycle(s): 09 Ninja 250R SE, 07 Honda CRF230F, 06 Honda CRF150F

Posts: A lot.
Quote:
Originally Posted by nickjpass View Post
So how did you guys deal with the outage?
My keyboard is recovering now...
__________________________________________________
09 250R SE: Paying it forward one post at a time!

Don't forget to add yourself to our Member Map
dubojr1 is offline   Reply With Quote


Old August 11th, 2011, 09:45 AM   #19
nickjpass
#squid
 
nickjpass's Avatar
 
Name: nickypoo
Location: Five Guys
Join Date: Jul 2011

Motorcycle(s): Track dedicated 2008 ZX6R

Posts: A lot.
MOTM - Jul '16
Quote:
Originally Posted by dubojr1 View Post
My keyboard is recovering now...
I just found out the problem and didn't worry about it
__________________________________________________
nickjpass is offline   Reply With Quote


Old August 11th, 2011, 10:52 AM   #20
Snake
ninjette.org certified postwhore
 
Snake's Avatar
 
Name: Rick
Location: Alexandria, Louisiana
Join Date: Jan 2009

Motorcycle(s): 05 Blue Ninja 250

Posts: Too much.
MOTY - 2017, MOTM - Jan '19, Oct '16, May '14
Quote:
Originally Posted by TnNinjaGirl View Post
There needs to be an NA (Ninjetters Anonymous) group if you ask me.
Hello my name is Snake...umm mmm I mean Rick and I am a ninjette.org addict. BTW is there a computer near by? I need to check on my...umm...stocks, yea my stocks, really.
Snake is offline   Reply With Quote


Old August 11th, 2011, 11:23 AM   #21
TnNinjaGirl
Ms. Personality
 
TnNinjaGirl's Avatar
 
Name: CB
Location: Murvill, TN
Join Date: Jan 2009

Motorcycle(s): Depends on the week you ask

Posts: A lot.
Lol :-)
TnNinjaGirl is offline   Reply With Quote


Old August 12th, 2011, 12:17 PM   #22
Alex
ninjette.org dude
 
Alex's Avatar
 
Name: 1 guess :-)
Location: SF Bay Area
Join Date: Jun 2008

Motorcycle(s): '13 Ninja 300 (white, the fastest color!), '13 R1200RT, '14 CRF250L, '12 TT-R125LE, '15 CRF110F, '13 TT-R50E

Posts: Too much.
Blog Entries: 7
A more complete post-mortem from Rimu's perspective:

link: http://rimuhosting.com/maintenance.j..._oid=181086391

Quote:
Summary

Colo4 lost power at their Dallas facility on 10 August CST. See http://accounts.colo4.com/status/ for the technical details.
RimuHosting's website was affected by the outage and was also unavailable.
We started pushing out information on the outage via Twitter and Facebook. We had a good number of customers join in on our live chat page. We brought up a temporary server in Auckland and used that to post a http://rimuhosting.com temporary page with information on the outage.
Colo4 brought some temporary electrical switching equipment online.
Power was restored.
Some of our core infrastructure servers did not come back up. We needed to replace some power supplies to get them back up.
We then needed to restart some network switches.
Then there were some physical servers that had not powered up, so we needed to go through server by server to get those back up and running.
And we also had a large number of niggly little issues. e.g. Some host servers that were misconfigured to not automatically restart their VPSs. e.g. one VLAN configuration issue. e.g. failed power supplies or blown hard drives or broken GRUB configs on physical servers.
We received a large number of support requests. e.g. Tomcat or Liferay not set to run on startup. e.g. SSL certificates installed with a passphrase on the private key requiring manual intervention for the server to complete boot up. e.g. MySQL databases that required repair table commands to be issued.
We have worked through most of the issues now. And if you are experiencing any problems that you'd like our help with please just pop in a support ticket and we can try to help.

Issues

Our core routers are on A+B power and should not have been impacted by the outage. They were. We will be investigating further.
We have some cabinets with A+B power feeds. Some servers in these cabinets did not lose power. But because of the core router issue they did not have network connectivity. About half our VPS host servers did not lose power at all.
Core RimuHosting servers were affected by the outage. Impacting our website, email, support tickets, and our equipment/location databases.
One of the things that was unavailable was our DNS management UI/API. This prevented customers from changing their DNS records from unresponsive servers to running servers.
After the power was restored some servers remained down and needed hands-on work done. e.g. restarts, re-grubbing, changing kernels, changing firewall settings via KVMoIPs, replacing power supplies. These required data center staff to act. And sometimes we were not getting the typically excellent and 'instant' responses we normally expect from the Colo4 staff.

Improvements

We need better fault tolerance and failover on our RimuHosting core infrastructure servers. So we are better able to communicate with customers. And so they can access key services like DNS.
We have had a number of requests from customers about how to reduce the impact of a fault on any one server or at one of the data centers they use. Our howto on this at http://rimuhosting.com/knowledgebase...g-and-failover is probably worth a refresh. I invite your suggestions and advice.
Some customers have requested more information about servers with redundant power. We can do that for dedicated server customers. Your server will need to be located in a cabinet with A+B power feeds. The server model will need to have a redundant dual power supply (currently only available on high end servers). Just let us know and we can work through the details with you.

SLA credits

We will be offering credits to all customers with Dallas-based servers. We will be offering a standard credit percentage or amount (to be determined). What do you think is appropriate in your case?
We know some customers were more impacted by the outage than others so we want to have the flexibility to ensure those customers get an appropriate credit.
To do that we will be letting customers claim the standard credit or apply for a different amount that best reflects the impact of the outage. We will be setting up a web page that will help us manage this. We want it up before the end of next week. To claim your credit then, please go to https://rimuhosting.com/cp/hostingcredit.jsp

Any questions?

Thank you to everyone who contacted us on http://twitter.com/rimuhosting and http://facebook.com/rimuhostingcom For your supportive comments or constructive advice. We appreciate your feedback. It really helped our sysadmin crew during a long and stressful day.
If you feel this notice is missing any information or you needed further details, email us and we can update this notice or provide you any further details you need. You are welcome to email us or join a public discussion on our blog at http://blog.rimuhosting.com/2011/08/...august-report/
I'm generally happy with their response. They do provide great service, great uptime right up until this incident (for years), and they provide this service at a good price. No intention now or in any foreseeable future to move us anywhere else.
__________________________________________________
Montgomery Street Motorcycle Club / cal24.com / crf250l.org / ninjette.org

ninjette.org Terms of Service

Shopping for motorcycle parts or equipment? Come here first.

The friendliest Ninja 250R/300/400 forum on the internet! (especially Unregistered)
Alex is offline   Reply With Quote


Old August 12th, 2011, 01:49 PM   #23
dubojr1
ninjette.org certified postwhore
 
dubojr1's Avatar
 
Name: Jason
Location: Jacksonville, Florida
Join Date: May 2010

Motorcycle(s): 09 Ninja 250R SE, 07 Honda CRF230F, 06 Honda CRF150F

Posts: A lot.
That just scrambled my brains...
__________________________________________________
09 250R SE: Paying it forward one post at a time!

Don't forget to add yourself to our Member Map
dubojr1 is offline   Reply With Quote


Old August 12th, 2011, 01:52 PM   #24
Alex
ninjette.org dude
 
Alex's Avatar
 
Name: 1 guess :-)
Location: SF Bay Area
Join Date: Jun 2008

Motorcycle(s): '13 Ninja 300 (white, the fastest color!), '13 R1200RT, '14 CRF250L, '12 TT-R125LE, '15 CRF110F, '13 TT-R50E

Posts: Too much.
Blog Entries: 7
Have you tried rebooting?
__________________________________________________
Montgomery Street Motorcycle Club / cal24.com / crf250l.org / ninjette.org

ninjette.org Terms of Service

Shopping for motorcycle parts or equipment? Come here first.

The friendliest Ninja 250R/300/400 forum on the internet! (especially Unregistered)
Alex is offline   Reply With Quote


Old August 12th, 2011, 01:58 PM   #25
dubojr1
ninjette.org certified postwhore
 
dubojr1's Avatar
 
Name: Jason
Location: Jacksonville, Florida
Join Date: May 2010

Motorcycle(s): 09 Ninja 250R SE, 07 Honda CRF230F, 06 Honda CRF150F

Posts: A lot.
Quote:
Originally Posted by Alex View Post
Have you tried rebooting?
Um... I think I had a B power failure.
__________________________________________________
09 250R SE: Paying it forward one post at a time!

Don't forget to add yourself to our Member Map
dubojr1 is offline   Reply With Quote


Reply




Similar Threads
Thread Thread Starter Forum Replies Last Post
Well that sucked LNasty Riding Skills 76 December 22nd, 2017 04:19 PM
Godaddy outage affects ninjette.org Alex Forum Information 5 September 11th, 2012 06:49 AM
so cal + arizona power outage alex.s Off-Topic 14 October 12th, 2011 04:15 PM
First ride in Rain today. IT SUCKED!!! untdavid General Motorcycling Discussion 33 May 4th, 2010 10:47 PM
[kropotkin thinks...] - MotoMatters.com Switching Hosting - Some Site Outage Possible Ninjette Newsbot Motorcycling News 0 March 15th, 2010 10:50 AM


Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


Motorcycle Safety Foundation

All times are GMT -7. The time now is 07:49 AM.


Website uptime monitoring Host-tracker.com
Powered by vBulletin®
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
User Alert System provided by Advanced User Tagging (Lite) - vBulletin Mods & Addons Copyright © 2024 DragonByte Technologies Ltd.
Except where otherwise noted, all site contents are © Copyright 2022 ninjette.org, All rights reserved.