iprev3 Failover Issues
On Feb 12, 2007 a disk failure on iprev2 (then primary) necessitated the failover to iprev3 as primary server.
The following is a list of issues that had to be addressed:
- Change dns records to point www.globalnetwatch.com to iprev3. After making this change the notify server was restarted on iprev3 which caused it to become primary based on a dns lookup of www.globalnetwatch.com
- The logout and login function required a fix to the Apache config file. The symptom was that a user had to login in twice in order to login and the logout sometimes didn't work. This had to do with Apache pointing to iprev3 instead of www.
- Snapshot - some of the tables and and data were missing from the snapshot feature. They were not being backed up and synced correctly. Symptoms were: an Oracle error message in the user interface when viewing the hour detail of a service when an error occurred during the hour requested. The snapshot button did not show in the interface. Steve imported the needed tables and data to fix the problem.
Perl Scripts - the Perl scripts (such as from the crontab and gnwtls) did not work. This was due to the OraPerl module being missing from iprev3. This module allows for an Oracle connection from Perl. When the necessary files were imported the Perl scripts worked.
- NTPD needed to be turned on to keep the time synced and accurate.
- The diagnostic tools (ping and traceroute) did not work. This is because the code had absolute paths hardwired in the code. After setting up links to mimic this path the tools worked.
- Stats data was missing - Steve had to import from backups.
- Compare 10 DNS and website had to be configured to work on the new server. This required changes to the DNS records and changes to the Appache httpd.conf on iprev3.
- Maintenance Window - XSL
- Kevin Stats were missing - created a script to allow him to regenerate past days data
Several features/screens displayed oracle errors or wouldn't work at all - this was caused by the Oracle sequences being out of sync. See Oracle Sequences for more info.
- Set up a keypair between iprev3 and binary4
- Set up passwords for data directories for Cabelas,West - Apache conf