Well, today could have gone better
Jul. 8th, 2004 01:00 amGot up on time, got to the paper. At least they got the RAID for the server back online, right?
Go to launch SQL, restart replication and... SQL Server Agent won't start. Crap. Go into Enterprise Manager, and my msdb is marked suspect. Double crap. Look for a backup of the msdb, and there isn't one. As
smurfchick would say, Holy crap on a crap cracker! We had backups of the DATA databases, but not the system databases. Not for the backup server. With no msdb database, I couldn't start replication, and the system was pretty unhappy.
At least I was never worried for the data - servers can always be rebuilt as long as the data is good.
Ponied up $245 to call Micro$oft, and spoke to a nice guy named Shane in Toronto who helped me fix it, after 90 minutes on the phone. I probably could have rebuilt the server in that amount of time, but I learned a neat command to rebuild a transaction log which may just come in handy in the future.
So, it's lunchtime, and the servers are back and happy and syncing up. Come back after lunchtime to do failover testing.
Hahaha. We failover the servers, and they FAIL. Spend three more hours scrambling to fix that, and finally get them going. Look out the window, and realize that although it's only 4:15, it is as dark as twilight out there. Then the thunder and lightening starts.
Drive to the airport through a spectacular torrential downpour. Get to the airport to find bedlam and chaos. Flights delayed, flight diverted, flights cancelled, people everywhere. My flight was supposed to leave at 7:30. It's 9:27 by my clock, and the plane still isn't here. Sat with a bunch of people who'd been trying to leave on a 3:30 flight, it finally left about 8:45, reminding me that it could always be worse. There's one guy here who was supposed to fly in from Frankfurt yesterday. That flight was cancelled, and today's was delayed and delayed and delayed. Finally get here, supposed to be on the 7:30 flight to Boston. So he can fly out in the morning to San Antonio. He can't fly straight to Texas tomorrow because his dog is already in Boston waiting for him. The real kicker is he's got his 7 year old son with him, who is remarkably well behaved, for having been up nearly 24 hours straight.
Watching people interact is always interesting. The Frankfurt guy is in the Air Force, and the woman sitting across from me has a son and daughter in law in the airforce. She mentions that her DiL has to choose a non-US posting, and is trying to choose between Korea, Turkey, Iceland and a few other places. Frankfurt guy was stationed in Iceland for two years, and raves about it. The woman sid her DiL hadn't really considered it, but she actually gets on her cell phone to call her and tell her that she needs to reconsider it, based on what Frankfurt guy has said. Amazing.
Other people are ranting and raving, spitting mad about the delay, about sitting in the airport, about wasting their time. Lots of things piss me off, but this just isn't one of them. Thunderstorms are beyond anyone's control, the airline people (at least the poor folks working at the desk) are doing everything they can to get people where they're supposed to be, and I just can't get worked up about it. Maybe if I had somewhere I needed to be by a certain time, but even then, it's not like I can do anything about it. Besides, I have my knitting. Yesterday, I had no sock at all. Today, I'm halfway up the foot of tthe sock, and it fits like a dream.
Now it's 1 am, I've been home for about 45 minutes, and I'm happy to be here. Yay!
Go to launch SQL, restart replication and... SQL Server Agent won't start. Crap. Go into Enterprise Manager, and my msdb is marked suspect. Double crap. Look for a backup of the msdb, and there isn't one. As
At least I was never worried for the data - servers can always be rebuilt as long as the data is good.
Ponied up $245 to call Micro$oft, and spoke to a nice guy named Shane in Toronto who helped me fix it, after 90 minutes on the phone. I probably could have rebuilt the server in that amount of time, but I learned a neat command to rebuild a transaction log which may just come in handy in the future.
So, it's lunchtime, and the servers are back and happy and syncing up. Come back after lunchtime to do failover testing.
Hahaha. We failover the servers, and they FAIL. Spend three more hours scrambling to fix that, and finally get them going. Look out the window, and realize that although it's only 4:15, it is as dark as twilight out there. Then the thunder and lightening starts.
Drive to the airport through a spectacular torrential downpour. Get to the airport to find bedlam and chaos. Flights delayed, flight diverted, flights cancelled, people everywhere. My flight was supposed to leave at 7:30. It's 9:27 by my clock, and the plane still isn't here. Sat with a bunch of people who'd been trying to leave on a 3:30 flight, it finally left about 8:45, reminding me that it could always be worse. There's one guy here who was supposed to fly in from Frankfurt yesterday. That flight was cancelled, and today's was delayed and delayed and delayed. Finally get here, supposed to be on the 7:30 flight to Boston. So he can fly out in the morning to San Antonio. He can't fly straight to Texas tomorrow because his dog is already in Boston waiting for him. The real kicker is he's got his 7 year old son with him, who is remarkably well behaved, for having been up nearly 24 hours straight.
Watching people interact is always interesting. The Frankfurt guy is in the Air Force, and the woman sitting across from me has a son and daughter in law in the airforce. She mentions that her DiL has to choose a non-US posting, and is trying to choose between Korea, Turkey, Iceland and a few other places. Frankfurt guy was stationed in Iceland for two years, and raves about it. The woman sid her DiL hadn't really considered it, but she actually gets on her cell phone to call her and tell her that she needs to reconsider it, based on what Frankfurt guy has said. Amazing.
Other people are ranting and raving, spitting mad about the delay, about sitting in the airport, about wasting their time. Lots of things piss me off, but this just isn't one of them. Thunderstorms are beyond anyone's control, the airline people (at least the poor folks working at the desk) are doing everything they can to get people where they're supposed to be, and I just can't get worked up about it. Maybe if I had somewhere I needed to be by a certain time, but even then, it's not like I can do anything about it. Besides, I have my knitting. Yesterday, I had no sock at all. Today, I'm halfway up the foot of tthe sock, and it fits like a dream.
Now it's 1 am, I've been home for about 45 minutes, and I'm happy to be here. Yay!
(no subject)
Date: 2004-07-08 07:28 am (UTC)(no subject)
Date: 2004-07-09 04:31 am (UTC)(no subject)
Date: 2004-07-08 02:43 pm (UTC)Yup, I really don't have a clue what you do. Not a bit of that made sense to me.
RAID - Kills bugs in the kitchen
DATA Databases - Are there non-data databases?
SQL - Sort for squirrel, the little critters my crazy neighbors feed by hand in the back yard.
Glad someone is out there fighting the good fight!
Oh, and glad you got home safe.
(no subject)
Date: 2004-07-08 03:22 pm (UTC)(no subject)
Date: 2004-07-09 04:30 am (UTC)And lesse -
RAID - literally stands for redundant array of inexpensive disks. It's a bunch of really big hard drives that the computer thinks is one REALLY REALLY big hard drives, put together in such a way that if one drive fails, you don't lose your data. Since we store hundreds of thousands of digital pictures, we use a lot of disk space.
DATA databases - There are non-data databases. Or rather databases used by the system for system things, but not as important to me as the database that hold the information our software stores. It's like the difference between the OS system files on your computer, and the files that hold your information. If an OS file goes bad, you can always re-install your OS (or you know, your local computer geek can for you), but if the file of the paper you've just slaved over for hours goes bad, you're pretty screwed.
And SQL is just a database language - in this case, Microsoft SQL, which is the engine behind the software my company makes.
There, how's that for more than you wanted to know?
(no subject)
Date: 2004-07-09 02:57 pm (UTC)Now you're talkin'