As a developer and part-time BroadVision administrator, our production website needs to be updated. Usually, we go through the normal change control processes, get scheduled, come up with backout plans, and then proceed onward. We set aside some time on the weekends, I get a little root access (w00t!), and then we apply patches, do upgrades, and all that good stuff. To add to the joy, I don’t have to do most OS upgrades, hardware upgrades, database maintenance, or even backups. We have dedicated teams that take care of all of GP and they rock. All I have to do is focus on BV and I love it.

This weekend, that love was challenged. We had a version upgrade to move BroadVision from 5.5 to 6.0. The big deal was to apply some schema changes to the database. Read on for the sordid story.

Saturday
11:00am We did a backup on the system right before I ran the scripts for the schema changes. That was our backout plan.
4:30pm I ran the upgrade scripts. Then the scripts puked on an extending rollback segment error. I said, “Maybe that’s not so bad.”
5:30pm Testing then revealed that our orders were not working properly, “That’s not good.”
6:00pm “Okay, let’s rollback”, I said. Confident that our rollback plan would work.
6:30pm J from Unix Support Services called to confirm which files to restore. He told me we’d restore “/oracle, /data1 – /data8, and some BV files”. I asked, “What about /data9 and /data10?” He said we didn’t have those in the backup. Uh-oh. In hindsight we should’ve stopped there. But I hoped that it would be okay. I didn’t know how wrong I would be.
11:00pm I look in on the system to see that the restore is still going on.

Sunday
7:40am I can’t sleep. I look in on the system. Start Oracle. Get an error. Shit. That’s not good. The database won’t mount. BroadVision can’t see the database. It’s too early to call the rest of the team. I go back to bed.
10:50am I call Chao (our dba) and ask if he can look at the database. He says he’ll be in the office at 1:00pm.
1:00pm I call a friend to see if she wants to catch a matinee movie at 3:00. Maybe I can salvage this afternoon.
2:00pm Chao gets in to look at the database.
3:00pm I get a conference call from DB Services. After about 30 minutes, we come up with a plan to use a much older backup and roll forward from there. Our backups have been incomplete for the past week. Damn, that’s definitely not good. I’ve missed the movie and I tell Yara to meet our friend without me. The tape backups will take about 3 1/2 hours to get. That frees me for a little while.
4:50pm I catch up with Yara and Gayle to do a little shopping. Yara and I grab a quick dinner before I have to be back at the house.
7:30pm The database team starts the long process of rebuilding our database. I’m shitting bricks. All our business data is at risk. About this time, I curse ever going into a job where I do maintenance. Consulting was always develop, deploy, disappear. There’s a reason why that works!
11:00pm I hear my friend Laughlin on a conference call. He sounds really pissed. The database has is ready to be restored.

Monday
1:30am DB Services calls me to tell me that the database will be ready in about an hour.
2:30am The database is restored and I start up BroadVision to test it. God smiles on me. It works. I let the rest of the guys know they done good. Then I go to sleep where I dream about working somewhere else. Really I did.

Sometimes, life gives you lemons, then it rubs ‘em in your eyes.