Memory over 3GB on NetWare

From CoolSolutionsWiki

Contents

Summary

NetWare 6.5 can use memory at and above the 4GB 32-bit limit. It does this by using PAE, Physical Address Extension. The memory over the 4GB line is slower, thanks to the 32-bit x86 arcitecture, and this applies to all x86 platforms. Generally speaking, when planning for a server with more than 4GB RAM it is better to give it lots of 'extra' than a little. So spec a 6GB server, not a 5GB server, it'll be faster.

Memory over 4GB

How PAE works

With the Pentium PRO processor, in order to get around the 4GB memory limitation, Intel introduced PAE (Physical Address Extension). PAE is not just a mechanism to address memory above 4GB, but it is simply an overall different method to address memory, including the memory below 4GB. In fact, with PAE disabled, the processor uses so called paging tables to translate logical 32 bit addresses used by programs into 32 bit physical addresses used to access the memory. In PAE mode, the structure of these paging tables has been changed to map the 32 bit logical addresses to 36 bit physical addresses. These paging tables use bigger entries, and there is one more level of paging tables which causes a slight overhead (a few percent performance penalty) compared to the 32 bit tables for non PAE mode. Different operating systems have taken different approaches on the way how they decide whether to use PAE mode or not:

  • NetWare 6.5 automatically uses PAE mode when available, even if the server has less than 4GB. The initial release of NetWare 6.5 would actually not even run on a processor without PAE mode support, but support for non PAE mode was added again in SP1 to enable NetWare 6.5 to run on early Pentium-M processors which did not support PAE.
  • Linux allows you to compile the kernel in PAE or non PAE mode. For typical Linux distributions, you may already have precompiled PAE and non PAE kernels from which you can choose. Generally, the installation program will pick a kernel for you based on the specifications of the computer you install it on.
  • For Windows, the PAE mode is controlled by a startup switch. Most modern Windows versions default to using PAE mode because of dependencies such as DEP (data Execution Prevention), which only works in PAE mode.

So as you can see, most of the current operating systems default to enabling PAE, even on machines with less than 4GB of RAM.

How PAE can be used to address more than 4GB of RAM

Applications running in protected memory

Each application running in protected memory typically has its own memory space. This means that each application has its own paging table mapping the 32 bit address space available to the application to the 36 bit physical address space of the machine. This means each application is limited to 4GB (actually less because of the memory space reserved for the OS), but all applications together can exceed 4GB of memory. This method of using memory above 4GB has no additional overhead except for the general small overhead of enabling PAE.

Dynamically changing the page tables to map memory into the 4GB address space

In this case, the application reserves a certain address range within its 4GB address space, and then as needed maps different memory pages to that address space. This method has an important performance penalty because not only does it add overhead to change the paging tables each time, but also the TLB (translation lookaside buffer, a special fast cache for paging tables) needs to be flushed each time you change the page table. This is the method similar to what we had in the old DOS days with EMS memory.


Real world use of PAE by different operating systems

Linux and Windows

Linux and Windows are fully protected operation systems. That means that application programs automatically run in protected memory space, and each application has its own memory space. The typical memory map for Linux and Windows for a logical 32 bit address space is that 1GB is reserved for the OS and the OS is visible in the address space of each address space. The other 3GB are private address space for the application.. This means that Linux and Windows can make relative good use of memory above 4GB provided no single application needs more than 3GB of RAM. In the case of an OES/Linux server running a big eDirectory and NSS, this means for instance that you could have up to 3GB of RAM for eDirectory caching and at the same time still have up to 3GB of RAM for NSS caching (assuming of course that enough physical RAM is available).

NetWare

NetWare severely suffers from the fact that it is still mostly based on the mode of NetWare 3.0 where for performance reasons, Novell decided not to take advantage of the protected memory features of the processor, but instead run all programs in OS memory space. While it is now possible to load programs in protected memory, this is limited to programs that are nicely written to be compatible with protected memory space. Unfortunately some of the most memory hungry core services like the NSS file system or eDirectory are not written to take advantage of protected memory and therefore they have to share the OS memory space with each other and with many other services thus not directly taking advantage of the extra memory above 4GB. Because of the fact that most programs are loaded in OS memory rather than protected memory, the memory model for NetWare is quite different from Linux and Windows in that it reserves less than 1GB of memory space for protected mode applications (the so called “user” space) and more than 3GB for the OS. Among the programs loaded in OS memory, only NSS has been “enhanced” to be able to use memory above 4GB, but this is done using the expensive method of mapping small chunks of memory on demand. That’s why NSS limits itself to only use memory above 4GB for caching big files.

All in all, the choices Novell made with NetWare 3.0 that give it its performance edge are now becoming more and more a problem and largely contribute to the memory management problems from which NetWare 6.5 has suffered from time to time.


Memory types in NetWare

NetWare has three types of memory. Physical, Logical, and Extended. Physical memory is the memory BIOS reports to the OS as in the 0-4GB range. Extended memory is the memory the BIOS reports to the OS as in the >4GB range. Logical memory is how the OS works Extended and Physical together. NetWare is aware of the difference between Physical and Extended, and handles the two differently.

Physical Memory

This is the workhorse memory of the system. Anything can run in here, so it is prime real-estate.

Extended Memory

Extended memory is slower than Physical memory because data in this range has to be paged to Physical before operations can be performed on it. Because of this, Extended Memory is the location where seldom accessed data resides. NLMs can't execute from here, but they may store data in this range. By default NetWare uses a certain percentage of Extended Memory, if present, as part of the NSS file-cache. It is also used as Virtual Memory for applications running in protected memory spaces; Java is one application that always runs in protected memory.

Logical Memory

Logical memory is how the OS views memory as a whole. The details of this particular type of memory are beyond the scope of this article.

Memory between 3GB and 4GB

Another feature of the x86 architecture is device memory. PCI devices, such as network cards and SCSI and FibreChannel cards, reserve memory at the top of the 4GB range for their own use. This is done in hardware to facilitate I/O to the devices. The side-effect of this is that not all of the 4GB space is usable by the operating system for running programs.

MEMORY command on NetWare

If you type "MEMORY" at the NetWare console, you will get a display of what kind of memory you have in the system. Here is an example:

OFFSRV1:memory
Total server memory:   3,669,577 Kilobytes
Total Extended Server memory: 458,748 Kilobytes
OFFSRV1:

"Extended Server memory" is the memory that NetWare uses PAE to access. The server this example was drawn from has 4GB of RAM in it, yet 458,748 KB are in extended. This is because that amount of memory is reserved for PCI devices. The BIOS then presents this memory to the OS as Extended rather than Physical. Note that not all bioses will remap the PCI address range to extended memory - in many case the memory in the PCI range will be simply unusable.

BIOS versions and device memory

Older BIOS revisions may not present the 'lost' device memory back to the OS as Extended memory. This should only be a problem in older servers that were manufactured when servers with 4GB of RAM were uncommon. If you have one, look for newer BIOS revisions that fix this.