Friday, March 12, 2010

Perfect ending!

Just got home from work on the Friday night/Saturday Morning. The workday started at 9AM in the Morning. Today is the last day before our project is going to the UAT (user acceptance test) for the team high visibility project. Our company partner with one the the greatest technology company in the World to work on their rapid deployment project. The whole group was ready to finish the project and hand it to the user.
I went to check with our DBA to ensure our database is ready. To my surprise, I was told that the server had some issues and database was not going to be ready until after lunch. Once the whole team learned of the news, you could tell the tension out there. We couldn't do anything without our database. Some how everyone was still confident we would be able to get it done. We still worked on our own part to prepare for installation in the afternoon. As promised, we had our database ready after lunch.
We were in the good mood again and ready for the next task, testing firewall port was open. Firewall was setup the night before, however, since our database was not up and running, we could not test. I opened my telnet and tried to connect to the database, I watched on my screen “connecting …..”. The longer the …... going, the more nervous I got. Finally the bad news came, “unable to connect”. It was time to regroup. My of my team member got on the phone and reached out to the network team. In the mean time, another team member contacted vendor who was standing by to help us to install their software. An hour passed by, network group concluded that web server was using the wrong IP.
It was time to get server team involved. Server team spent the next hour to research why the server was using the wrong IP and suggested to add another network card to the server. The suggestion sounded reasonable and we agreed. Second network card added to the server. I started my telnet session again “connecting …..”. Bad news again! We went back to network and asked for help. Network told us to run a command to force server using the correct IP. What a great idea! It worked. My telnet session finally went through and we were golden. All of us relieved after all, we had our plans for the Friday. One was planning to go to girlfriend's big birthday dinner, one had a movie date with wife, and the other two had poker night planned weeks ago.
We were ready for roll and we had vendor ready to support us. We were happy again! Our supporting people went home since we had resolved all the problem. The rest of the install was just a piece of cake. We followed our installation instruction and ready to install. We ran the first command, “Error, unable connect to the database”. We could see the database, we could reach the database, but we just could not connect to the database. Vendor suggested we used different way to connect. No, it didn't work! After many tries, we concluded that something wrong with the account we used to connect to the database. DBA went home already. Time to page on-call DBA. It was raining outside and they just left work. We waited for awhile and got a call from our on-call DBA. After we told him what kind of problem we had, he said he just pulled to the side of the road and it would be another 30/40 minutes before he could get home, but there was another DBA would be able to help us in a few minutes.
DBA back from home and spent a few minutes to check the account we were using. There was nothing wrong with the account permission! But we still could not connect! Finally we found out that we read the password wrong. There was an exclamation point in the password “!”, not the letter “i”. Now you knew why I used so many exclamation points here!
We ran the command again and it worked. We were able to connect to the database, finally, yea! We finished installation. It was time to configure the software. “Go home” said our supervisor. It was going to be boring for you from now on. We had done this part before so it was going to be easy. Not long after he said “go home”, “Houston, we had problem”. The configuration person reported wizard crashing. No................., this could not happen. It did happened. It was time to go back to the drawing board again. Everyone one of us researched on the reason behind the crash. Vendor was doing the same. After many tries, one of the team member found the solution. We needed our colleagues from directory service to help. Our service account didn't have proper permission. Fortunately, it didn't take long to get us back to the business. Our service account was granted the permission to fix the problem. Wow, finally our server was serving up the page to ask our credentials that we didn't get before. We were excited! Enter user name, then password. The login page came back to ask user name and password again. We must type in the wrong password. We typed slowly to make sure we didn't miss typing it. Still didn't work. What else could be wrong! Then we remembered trouble shooting 101 in our favor operating system, reboot! Of course, it worked, actually it worked like a charm. We could continue configuration wizard again. “Running.....”, then “Error” again. Was this real? This could not be true. We would never be home, everyone was thinking the same. It was close to the midnight. This software was getting ready to be released to the public at the end of month. Let's run it again. To our surprise, it didn't give us the error this time. But why? At this point, no one would care to find the reason it worked the second time. We got it done and we finished up the work.
What amazing over the whole day was that no one gave up and we had a sense of humor among us the entire day. It was the team work that got us through and ended the night with an exclamation point!

No comments:

Post a Comment