NWmemoryProtectionRestart

From CoolSolutionsWiki

(Difference between revisions)
Revision as of 16:08, 2 November 2009
Terickson (Talk | contribs)

� Previous diff		 Revision as of 16:13, 2 November 2009
Terickson (Talk | contribs)

Next diff →
Line 77: Line 77:
# Oct 29 ftp://ftp.novell.com/incoming/sr10569161301-091029AA-lvo-02-swgas.zip # Oct 29 ftp://ftp.novell.com/incoming/sr10569161301-091029AA-lvo-02-swgas.zip
## This is a cluster abend, "At least one of the nodes is Alive in the old master's node partition. This node is NOT in the old master's node partition." The server is not busy. In fact, it isn't even hosting any Groupwise services at the moment. This is going to be a LAN channel issue though in a 2 node cluster. If this pops up more, then we'll need to troubleshoot that. ## This is a cluster abend, "At least one of the nodes is Alive in the old master's node partition. This node is NOT in the old master's node partition." The server is not busy. In fact, it isn't even hosting any Groupwise services at the moment. This is going to be a LAN channel issue though in a 2 node cluster. If this pops up more, then we'll need to troubleshoot that.
-# Nov 2nd sr10569161301-091101AA-lvo-02 Address in VSTIF6.NLM at code start +00012F20h AKA gwdcaw+# Nov 2nd ftp://ftp.novell.com/incoming/sr10569161301-091101AA-lvo-02 Address in VSTIF6.NLM at code start +00012F20h AKA gwdcaw
-# Nov 2nd sr10569161301-lvc-gw3-091101AA+# Nov 2nd ftp://ftp.novell.com/incoming/sr10569161301-lvc-gw3-091101AA
## Address in LIB0.NLM at code start +0001366Eh AKA gwpoa abend ## Address in LIB0.NLM at code start +0001366Eh AKA gwpoa abend

Revision as of 16:13, 2 November 2009

-my other wiki


Summary of Memory Protection Issue

Contents

Tracking

  1. SR 10504113741 GW protected memory faults Sept 21,2009
  2. SR 10523137371 GWSOAP-<post office name-Handler_5 Sept 1, 2009
    1. https://bugzilla.novell.com/show_bug.cgi?id=535749 Page Fault Processor Exception, GWSOAP-<post office name-Handler_5 Fixed GW 703 hp4, GW 801
  3. Defects
    1. GW8 515295 USERLIB.NLM abend. Unknown EIP - FIXED in 801
    2. GW8 491527 GWXIS12.NLM - SOAP Abend - FIXED in 801
    3. GW7 518385 USERLIB.NLM abend. Unknown EIP - FIXED in 703 HP4
    4. GW7 534927 Abend on GWSOAP process - FIXED in 703 hp4
    5. Bug 529767 - Abend in USERLIB.NLM|AllocSleepOK

Action Plan

  1. enable memory protection debug to take cores - sys:\system\memdbg.ncf and umemdbg.ncf
  2. 25% free on ALL PO volumes!!!
  3. Ensure Antivirus NOT scanning QUEUES - Customer changed this to comply
  4. Ensure Purge Immediate - Customer changed this to comply
  5. Fixed memory protection load/unload syntax to match NW 65 sp8 format
    1. syntax for sp8
      1. http://www.novell.com/communities/node/6863/consultants-corner-upgrading-groupwise-8-part-3-post-offices
      2. load restart address space=demopo sys:\system\gw8\gwpoa @demopo.poa
  6. Tried post gw 801 code: ftp://ftp.novell.com/outgoing/thomasgw8.0.1HP-88366_full_nlmwin_en.zip
  7. update to current bsdsock.nlm
  8. STOP PROCESSORS on the command line, or don't load the ACPIDRV.PSM driver. Either will do.
  9. Maximum Physical Receive Packet Size to 4224 (default) -- SWGAS already had at 4224
  10. keep sending CORES!!

Aggressive plans

  1. Let me install/configure SLES 10/Linux and put the PO there. 250+ days uptime
  2. decouple Memory Protection/NCS
  3. Replace hardware
  4. Back-rev

Customer cores

Accessing cores Internally: VNC 151.155.135.178

  1. August 31 LVC-GW3-SWGAS.ZIP (not in incoming anymore, but on VNC machine)
    1. EIP: 8FE5166E LIB0.NLM|NCPSendToLocalServerWithReplyLen+1EFA
  2. Sept 28 ftp://ftp.novell.com/incoming/090927-lvc-gw1.zip
  3. Sept 29 ftp://ftp.novell.com/incoming/090929-lvc-gw4-retry.zip
  4. Oct 8 ftp://ftp.novell.com/incoming/swgaszipfiles.zip
    1. swgaszipfiles/lvc-gw4-091007AA
      1. Current EIP: 8FE4466E LIB0.NLM|NCPSendToLocalServerWithReplyLen+1EFA
    2. swgaszipfiles/lvc-gw4-091008AA
      1. Address in USERLIB.NLM at code start +00002146h
  5. Oct 10 ftp://ftp.novell.com/incoming/lvc-gw4-10-5-09.zip
    1. Address in BSDSOCK.NLM at code start +000020E6h
    2. BSDSOCK Version 6.82.00 November 20, 2007
  6. Oct 13 ftp://ftp.novell.com/incoming/lvo-02-091013AA-retry.zip
    1. Address in SERVER.NLM at code start +00167858h
  7. Oct 28 ftp://ftp.novell.com/incoming/091028AA-lvc-gw1-swgas.zip
    1. gwcda abend - bug 550824
  8. Oct 28 ftp://ftp.novell.com/incoming/sr10569161301-lvc-gw2-091828aB.IMG
    1. server.nlm SERVER.NLM|TcoNewSystemThreadEntryPoint+40
  9. Oct 28 ftp://ftp.novell.com/incoming/SR10504113741-LVC-GW3-swgas.zip
    1. 091021AA.IMG
      1. Current EIP: F509507D CLIB.NLM|longjmp+2A
    2. 091021AB.IMG
      1. Current EIP: F509507D CLIB.NLM|longjmp+2A
      2. C18C3A80 F419F000 SCCUT.NLM|<Start of DATA segment>
  10. Oct 28 ftp://ftp.novell.com/incoming/SR10569161301-lvo-02-091028AA.zip
    1. Break caused by Abend: Deadlock detected waiting for spinlock currently owned by
    2. Current EIP: 0011367C LOADER.NLM|WaitForSpinLock+6C
  11. Oct 29 ftp://ftp.novell.com/incoming/sr10569161301-lvc-gw2-091029AA.zip
    1. Break caused by: Free detected corrupt preceeding redzone for node 0xCB730AE0
    2. This one abended with the message "Free detected corrupt preceeding redzone for node 0xCB730AE0". This means that memory has been corrupted. And it is usually by the owning application. It doesn't say who owns the memory currently, but only because it is in protected memory. So most likely it's going to be a Groupwise address space faulting.
    3. Address in SYSCALLS.NLM at code start +00000CF8h
  12. Oct 29 ftp://ftp.novell.com/incoming/sr10569161301-lvc-gw1-091029AA-retry.IMG
    1. gwcda
  13. Oct 29 ftp://ftp.novell.com/incoming/sr10569161301-091028AB-lvc-gw1-swgas.zip
    1. Current EIP: F3DCF64A EXHTML.NLM|EXHTMLGetLockCount+1E0B5
    2. gwcda
  14. Oct 29 ftp://ftp.novell.com/incoming/sr10569161301-091029AA-lvo-02-swgas.zip
    1. This is a cluster abend, "At least one of the nodes is Alive in the old master's node partition. This node is NOT in the old master's node partition." The server is not busy. In fact, it isn't even hosting any Groupwise services at the moment. This is going to be a LAN channel issue though in a 2 node cluster. If this pops up more, then we'll need to troubleshoot that.
  15. Nov 2nd ftp://ftp.novell.com/incoming/sr10569161301-091101AA-lvo-02 Address in VSTIF6.NLM at code start +00012F20h AKA gwdcaw
  16. Nov 2nd ftp://ftp.novell.com/incoming/sr10569161301-lvc-gw3-091101AA
    1. Address in LIB0.NLM at code start +0001366Eh AKA gwpoa abend


brief description of each core

  1. Oct 9 swgaszipfiles/lvc-gw4-091007AA
    1. P07# sw

Insert non-formatted text here Current EIP: 8FE4466E LIB0.NLM|NCPSendToLocalServerWithReplyLen+1EFA B3ED5EBC 00370003 SERVER.NLM|NewSystemCall+D3 Emulated 5000 and found no RET instruction Function may never return. P07# .m gwpoa.nlm GWPOA.NLM GroupWise Post Office Agent (Release version) 

  Loaded from [SYS:\SYSTEM\]
 [159]  PO3EMAIL address space
 Version 8.00.01   August 25, 2009
 Code Address: F463C000h  Length: 00215434h
 Data Address: F45BD000h  Length: 0007D989h

P07# .m userlib.nlm USERLIB.NLM NetWare Operating System Function Library 

  Loaded from [SYS:SYSTEM\]
 [169]  GWDCAW.LVCPO31 address space
 Version 5.60.00   September 29, 2008
 Code Address: F5353000h  Length: 000048DEh
 Data Address: F5350000h  Length: 000011F0h

USERLIB.NLM NetWare Operating System Function Library 

  Loaded from [SYS:SYSTEM\]
 [86]  PO3EMAIL address space
 Version 5.60.00   September 29, 2008
 Code Address: F5353000h  Length: 000048DEh
 Data Address: F5350000h  Length: 000011F0h

USERLIB.NLM NetWare Operating System Function Library 

  Loaded from [SYS:SYSTEM\]
 [15A]  GWDCA.LVCPO3 address space
 Version 5.60.00   September 29, 2008
 Code Address: F5353000h  Length: 000048DEh
 Data Address: F5350000h  Length: 000011F0h

Insert non-formatted text here

    1. Oct 9 swgaszipfiles/lvc-gw4-091008AA

Insert non-formatted text here P05# ? Address in USERLIB.NLM at code start +00002146h Previous: -0000009E F53550A8 USERLIB.NLM|AllocMemoryDefault Current: 00000000 F5355146 Next: +00000156 F535529C USERLIB.NLM|AllocateSlab P05#

USERLIB.NLM NetWare Operating System Function Library 

  Loaded from [SYS:SYSTEM\]
 [86]  PO3EMAIL address space
 Version 5.60.00   September 29, 2008
 Code Address: F5353000h  Length: 000048DEh
 Data Address: F5350000h  Length: 000011F0h

P05# .m gwpoa.nlm GWPOA.NLM GroupWise Post Office Agent (Release version) 

  Loaded from [SYS:\SYSTEM\]
 [157]  PO3EMAIL address space
 Version 8.00.01   September 27, 2009
 Code Address: F3C16000h  Length: 002155B4h
 Data Address: F3B97000h  Length: 0007D989h

P05# Insert non-formatted text here

  1. Oct 10 E:\cores\lvc-gw4-10-5-09
    1. Address in BSDSOCK.NLM at code start +000020E6h
    2. P01# .m bsdsock.nm

BSDSOCK.NLM Novell BSDSOCK Module 

  Loaded from [SYS:SYSTEM\]
 [79]  OS address space
 Version 6.82.00   November 20, 2007
 Code Address: A1631000h  Length: 00011BF9h
 Data Address: A1643000h  Length: 0000C0E0h

P01#


  1. Oct 13 cores\lvo-02-091013AA-retry
    1. Address in SERVER.NLM at code start +00167858h
    2. P00# sw Current EIP: 00371F18 SERVER.NLM|NewSystemCall+88 ESP invalid!
    3. USERLIB.NLM NetWare Operating System Function Library 
  Loaded from [SYS:SYSTEM\]
 [13C]  GWDCAW.LVOPO1 address space
 Version 5.60.00   September 29, 2008
 Code Address: F5353000h  Length: 000048DEh
 Data Address: F5350000h  Length: 000011F0h

P00# .m gwpoa.nlm GWPOA.NLM GroupWise Post Office Agent (Release version) 

  Loaded from [SYS:\SYSTEM\]
 [16F]  LVOEMAIL address space
 Version 8.00.01   September 27, 2009
 Code Address: F43B4000h  Length: 002155B4h
 Data Address: F4335000h  Length: 0007D989h

P00#

Retrieved from "http://wiki.novell.com/index.php/NWmemoryProtectionRestart"