Maintwindows

From CoolSolutionsWiki

-HA TODO
  1. Dec 8 Maint. window checklist
    1. Primary objective - Move two MTAs off NW cluster and retire that 4 node cluster.
      1. Check list
        1. Down mta agents and backup GW DB files
        2. Online mta agents and change paths, ensure syncs (connect to each Domain)
        3. Copy DB files to Linux servers
        4. configure/connect linux GW agents to DB files
        5. Configure HA load scripts for new MTAs
        6. Add HA monitor to GWIA and ensure each important HA sub-module has monitor enabled
        7. Verify each HA resource can go on each node
          1. chkconfig heartbeat off for tests, then turn back on after tests
    2. X Secondary Objective - Install and configure GW Monitor and alerting
    3. Third Objective - Move Webaccess and IM to gwmail4 (non-cluster enabled)
    4. MISC

Contents

[edit]

Outages

  1. Feb 3, POA migrated and did NOT startup on its own. Cause was related to this evms issue
  2. Feb 27, the poa stop'd listening, likely because of the permissions issue - self inflicted


[edit]

Maint. window 1

COMPLETED DEC 22

[edit]

correct small size cib.xml tasklist

  1. ensure customer has full backup
  2. correct the /etc/ha.d/ha.cf file so the private NIC is first - we suspect comms problem and it might be trying the public LAN first :(
    1. Backout plan: cp ha.cf ha.cf.back1 BEFORE modifying. So backout plan is: rcheartbeat stop, cp ha.cf.back1 ha.cf, rcheartbeat start
    2. Verification: Sniff the wire and ensure HB traffic is talking on the HB Lan
  3. bring the other two nodes into the cluster
    1. ensure gwmail1 and gwmail2 servers are in standby mode
    2. ensure all 3 nodes have chkconfig heartbeat off - to avoid any chance of multiple reboots.
    3. ensure stonith is OFF
    4. consider stopping all HB resources as hb_gui requests seem to be ignored currently and it would be nice to stop all the resources and rcheartbeat stop
    5. backup the cib.xml file on gwmail3 and snip the check file
    6. rcheartbeat start on gwmail3
    7. crm_mon -i2 to ensure he joins and takes on resources
    8. gwmail1 and gwmail3 - rcheartbeat start
      1. use crm_mon -i2 to ensure they join and say standby
      2. verification: cibadmin -Q to ensure they are getting the proper cib.xml
    9. bring gwmail1 and gwmail2 out of standby and into ONLINE mode
    10. verify migration of resources to gwmail1 and gwmail2 are successful
  4. correct pridom's need for manual start (Troy)
    1. ensure cib.xml for this resource is correct
    2. thing1
    3. thing2
  5. ensure pridom resource is able to move to each node
  6. enable HB monitor for appropriate resources (like GWIA)
    1. put non-DC nodes into standby mode
    2. use hb_gui to enable HB Monitor
    3. take nodes out of standby mode once hb_gui modifications are done.
  7. re-enable stonith
  8. Once everything checks out ok, then make sure all 3 nodes have the same GW version.
    1. gwmail1 has the hp1b/DST patch on it - so rpm -Uivh to back rev those to the same version as gwmail2, gwmail3
    2. ensure GroupWise startup script is HB aware and has the correct paths to 64bit
  9. VERIFY each resource can go successfully to each node.
  10. chkconfig heartbeat on on each node
  11. Time permitting move the secondary GW domain over to the Linux HA
[edit]

Customer requirements

  1. A list of names and expertise of who will be on-line during the maintenance windows.
    1. A list of names and expertise of who will be on-site for the maintenance windows.
      1. Thomas E., PSE comms, NOS(NW/Linux), eDir, GW, ZENworks
      2. Cameron C., PSE Linux/IDM
      3. Troy W., PSE former GW resources for PSEs
      4. Jason R., PSE resource for Linux NOS
  2. Trigger times for initiating a roll back for each major task.
    1. To be determined by customer
  3. The roll back steps for each major task.
  4. The testing verification steps that indicate a completed task.
[edit]

Maint. window 2 - move secondary MTA to HB Cluster, bind former webaccess and IM ip addr. to gwmail4

[edit]

Move secondary domain

COMPLETED DEC 29

  1. ensure chkconfig heartbeat off
  2. ensure stonith is disabled
  3. ssh into HB DC
  4. put other two nodes into standby mode
  5. offline all resources gracefully
  6. Create EVMS container for secondary Domain
  7. Create EVMS volume for secondary Domain
  8. Create hb_gui resource group and add the necessary hb_gui resources to mount the disk
  9. mount the new secondary domain partition and copy the DB files to it
  10. ensure /gwise/dom2 domain directory is samba enabled
  11. update the domain paths to the new location
  12. repair the domain so the paths get sync'd to the DB
  13. ensure the agent has the GW startup files - AKA install and configure the GW MTA agent with the correct domain name and paths
  14. add additional hb_gui group resource sub-modules to enable the agent to start properly.
  15. verify the agent is online and communicating
    1. ps aux | grep gwmta
    2. gw http monitor - check links
    3. send mail to diff. Post Office and to/from the Internet
  16. re-enable stonith
  17. Verify resources can migrate to each node
  18. chkconfig heartbeat on


[edit]

Bind former webaccess and IM ip addr. to gwmail4

COMPLETED DEC 22

  1. Customer to provide Public SSL Certificate
  2. unbind/offline former IP addr. holder
  3. add these two ip addresses to gwmail4
  4. restart IM/webaccess services and see if they bind to the new alias
    1. If not simply re-run the config script for IM and put the new IP addr
    2. Webaccess should bind to all addresses including alias addresses, but if not we can add it to the apache conf file
  5. verify IM clients can connect non-ssl and ssl
  6. verify webaccess users can connect
  7. ensure chkconfig grpwise on and IM services are on
  8. put in place scripts to auto-restart service if they go offline
  9. retire former Webaccess and IM servers
    1. check timesync
    2. check report sync
    3. check for obits - dsrepair -a | adv | ext ref
    4. load config.nlm /all put sys:\system\config.txt in a safe place
    5. dsrepair -rc put sys:system\dsr_dib in a safe place
    6. nwconfig | directory options | remove DS from this server
[edit]

Remove old NW Cluster

COMPLETED DEC 29

  1. Monitor period before removing NW Cluster (time between maint. window 1 and maint. window 2 should be long enough to know)
    1. Remove NW cluster
      1. Time in sync?
      2. Report sync clean?
      3. no obits with dsrepair -a | adv | check ext ref
      4. document all load/unload scripts
      5. offline each resource
      6. uldncs.ncf
      7. delete sub NCS objects - or just wait
      8. nwconfig | dir | remove directory services (CHECK other services like slpda/time provider)
[edit]

Misc tasks

  1. snip old GW library reference from DB
    1. COMPLETED DEC 29


[edit]

Maint window Feb 14 troubleshoot small cib.xml

[edit]

OUTCOME

  1. Successfully have the full/correct cib.xml on each server
  2. Was able to observe in greater deepth the EVMS worker thread issue
[edit]

Plan A: (less impact)

  1. Disable stonith (follow our usual plan of ensuring backup and chkconfig heartbeat off etc etc)
  2. command line - migrate the PO resource from gwmail3 to one of the other nodes
  3. Then rcheartbeat stop, which will stop the heartbeat on the DC and the DC will go to one of the other nodes - then we'll check its cib.xml file.

Our impression is that it will then have a full sized cib.xml

Pros: The PO is migrated elsewhere and clients will have a good chance of not getting any errors from migrating the PO to another server.

[edit]

Plan B: (more impact)

  1. Disable stonith (follow our usual plan of ensuring backup and chkconfig heartbeat off etc etc)
  2. command line - offline all resources
  3. stop heartbeat on all nodes
  4. bring up non-gwmail3 first and observe if it gets a large cib.xml
[edit]

Secondary Objectives

  1. Change directory permissions and verify GW agent restarts


[edit]

TO DO

  1. Partition off Root
    1. Health Check
      1. Check timesync
      2. check report sync
      3. check for obits - dsrepair -a | check ext ref
    2. Take dibs on multiple servers
    3. Split partition Root
  2. Virtual IP for GW consoleone
  3. investigate EVMS worker threads not cleaning up
  4. chkconfig lkcd off (get all debug off lkcd-utils in software management).
[edit]

March 12 maint window

[edit]

Primary objectives

  1. Get poa status to show properly in hb_gui, and thus have HB monitor restart/migrate resource when needed
    1. change permissions to log file /var/run/novell/groupwise
    2. find ownership of files - export to a file
    3. stop poa via clm
    4. start poa via hb_gui
  2. move LAN switch power supply
  3. evms
    1. enable debug forevms
    2. change evms timeout from 10 min. to 12 seconds
  4. when finished
    1. renable chkconfig heartbeat
    2. renable stonith
[edit]

Secondary objectives

  1. umount restore1 on gwmail1
  2. add https for webmail
  3. second spam box x.x.80.151 to outbound list in that file.
[edit]

Mar 29

[edit]

Primary Objective

  1. Get gwmta and gwpoa running as gwuser
    1. modify uid.run for each agent
    2. rcgrpwise stop
  2. chown all files to gwuser, including /var/log/novell/groupwise (or set the path to /gwise/agent/log
  3. Update the DOA UNC path to .87 and not .8
[edit]

Procedure

  1. modify the uid.run for each agent and replace root with gwuser
  2. chown -R gwuser:users /gwise
  3. rcgrpwise stop
  4. backup the wpdomain.db for primary and secondary domains
  5. rcgrpwise start
    1. rcgrpwise status and confirm all agents are running after "owner" change.
    2. Update proper log for MTAs as they currently go to /var/log/novell/groupwise
    3. update log level for all agents from 32 mb to 256mb
  6. Connect to primary and update UNC path for MTA
    1. May require "rebuild repair"
    2. Once resolved, connect to secondary and check paths
  7. Setup sysstat for server utilization stats
[edit]

Secondary Objective

  1. Disconnect link to Metro
  2. Update log limits to 256mb instead of 32 mb
  3. Added second "mail forward" to GWIA
[edit]

Procedure

  1. Go into link configuration and remove Metro
  2. Mail forward
    1. Use the GWIA object and not the startup file. Separate the mail forward1 and forward2 with a space.
[edit]

Next Maint window

[edit]

Primary Objective

  1. Troubleshoot gwia (Issue 1: startup files point to wrong binary and thus requires manual start of gwia Issue 2: Gwia stops working after 7 days exactly), which may include reinstalling gwia. If we are reinstalling, we may want to move the gwia to mail2.
    1. Backout plan
      1. Tape backup of GW DB
      2. backup binary and GroupWise configuration directories
[edit]

Secondary Objective

  1. Change daemon to run as gwuser instead root
  2. Set GW monitor to restart agents
[edit]

TO DO

  1. Power Path/EMC is being used instead of MPIO
  2. Consider FAN out of GW agents across multiple boxes. TE to send hard data, and pros/cons of each option, with snips of best practice.
  3. GWAVA Reload
  4. GWIA stop running after 7 days exactly
  5. set agents as gwuser
  6. PENDING sent email for Sam
    1. Internet outbound email with attachments. Not internal PO
  7. Large emails
    1. sent max at 10mb, but 8 mb are added 2 mb so an 8 mb is blocked. Change to 15 mb
    2. 9.1 PO->GWIA-> turns into 12 mb
[edit]

Oct 17, 2009 reload verification



For the purposes of making a TEST, I will recommend a procedure that will give you more downtime than actually doing it when needed - this just ensures a smoother return to normal production. What you're testing is the mechanism, after all.

  1. down the production POA
  2. make a Reload backup
  3. turn on DR mode
  4. run as long as you need to.
  5. turn OFF DR mode
  6. migrate back
  7. resume the production POA.

The downtime will be as long as it would take to run a standard backup.

In an actual unplanned downtime scenario, (production POA is down already)

If the file system of the production POA IS available, 1) do steps 2 and 3 as above.

IF the file system of the production POA is not available,

  1. turn ON DR mode. You will be as current as the last backup that ran. In other words, you will not have data that landed after the last backup.
  2. turn on DR mode
  3. run as long as you need to.
    1. When you're ready to switch back to the production POA
  4. do a pre-migrate-back
  5. down DR mode (there will be downtime now)
  6. run a final migrate back
  7. start the production POA.


The clients will be able to connect so long as either the production or DR POA is up. The idea is to do this to minimize downtime. There WILL be downtime, however.

[edit]

outcome

QUESTIONS

  1. automate ip addresss switch or do DNS
    1. automate the reload server binding to live server's ip address? no, but you can manually bind them, and then teach gwava reload
  2. get becfield own secondary ip addr
  3. GWIA automation
    1. no reload automation, just have a secondary gwia OR rsync data/config over and manually start gwia on reload as needed.

TIPS

  1. prior to going into DR mode, bind ip addresses so when the agent loads, it already binds to the correct address
  2. how to configure agent in DR mode to assume proper ip address
    1. go into | DR menu | select agent type | CONFIGURE DR mode main menu | enable DR mode settings | Go to the DR MTA config main menu | address |


TODO

  1. X document all ip addresses and ports for all agents
  2. get BT2 another IP address, and bind 79.5 as secondary...so 79.5 can move to the reload box.
  3. restore area - reload restore area configuration p. 61
  4. restore area IP addresses
[edit]

completed

  1. SUCCESSFULLY put becfield into DR mode and verified internal/external email. Incoming needs refinement to bind to production IP address
  2. Successfully put primary and secondary domains in DR mod
  3. Successfully put MIA PO into DR mode
  4. Found where to "automate" binding of production IP addrresses, further documentation for phase 2 verification.
  5. familiarized with process/methodology of implementing DR modes.
  6. Documented process for future reference.
[edit]

Nov 7, 2009

  1. Planned to move all GW agents to reload box and simulate system wide outage.
    1. change BT server so it'd have secondary IP address of 79.6 and give 79.6 primary and allow 79.5 to move to reload box
    2. turn off all agents so we can get all data
    3. use reload to backup all current data
    4. once all data sync'd, turn Reload DR agents on for all agents.
      1. all agents turned on and functioned except the primary. Contacted gwava support and got "OK" support
    5. Management decided to abort simulation due to the time window.
      1. resync'd all data back to live system
    6. loaded all agents up, but primary failed. Tried connecting with C1 to do repairs, that failed. Tried copying locally and do repairs, still could not connect. Thus corrupt primary DB, found that the primary DB had not changed since Oct 23, so we restored Oct 22 DB and did secondary to primary sync in C1 to get data that may have changed since Oct 22nd.
      1. This may be the cause of the corruption, or a multi-faceted cause of losing power. 7002459
        1. Prevention: run c1 off GW linux server through X
[edit]

TO DO

  1. Create a reload gwia dom profile so we can put the gwia dom in DR mode if needed.
    1. Once gwia dom is on reload, gwia should load up on reload server.
  2. Refine how BT uses its PO ip address
  3. setup and configure restore area
  4. Setup c1 to launch through X
  5. confirm sync secondary with primary completed, got memory error: memory function failure (likely workstation out of memory)
  6. Make sure all remote tools work effectively. Struggled with admin tools performing well.
Retrieved from "http://wiki.novell.com/index.php/Maintwindows"