ZDM ZENworks Best Practice Middle Tier
ZENworks Best Practice Middle Tier
In NovellÃ‚Â® ZENworksÃ‚Â® 6 Desktop Management and above, Novell includes an architectural element known as the ZENworks Middle Tier Server. The middle-tier server is a Web server with ZENworks Web components built on Apache or Internet Information Services (IIS) that provide functions to the workstation agents, allowing them to deliver ZENworks Desktop Management features. When a ZENworks Middle Tier Server is part of the infrastructure, many workstation agents might be directed to the same middle-tier server, making it a resource that needs to be understood and managed properly in order to provide scalability and availability to all of the clients in the organization. This paper discusses the expectations that can be placed on a ZENworks Middle Tier Server. It also discusses methods for enhancing a middle-tier server configuration to provide greater scalability and availability.
Scalability versus Performance
Scalability is the ability to increase processing by adding more resources. Performance refers to the response times presented under a typical load. Increasing system scalability does not necessarily increase system performance. The goal of scalability is to provide constant, acceptable performance even as the load increases. Scaling can be achieved by adding additional hardware (such as CPUs and memory) to the server (known as scaling up), or by adding additional servers to a server farm to share the load (known as scaling out). It is vital to understand the ramifications of implementing both scaling methods so that you can be prepared to choose the one that best meets your needs and to address the bottlenecks that might occur in a middle-tier implementation. The following pages discuss these issues and the best ways to improve scalability.
Scalability versus Availability
Availability is the ability for a resource to be accessed. High availability is achieved when the resource is always available and users can utilize it at any time. Although providing a greater scale can provide some perceived increases in availability, a potential single point of failure still exists if resources are stored on only a single server. Availability can be achieved only through the use of additional resources such as server farms. This document specifies the methods that can be used to provide high availability to your middletier architecture.
Middle Tier functionality
In order to understand the potential bottlenecks and the best scaling and availability architecture, it is important to understand how the middle-tier server interacts with the agents on the devices and with Novell eDirectoryÃ¢â€žÂ¢ services.
Agent Requests to the Middle-tier Server
The ZENworks Middle Tier Server provides a support function for the ZENworks Desktop Management agents running on the user workstation. These agents must authenticate to eDirectory and retrieve policies and applications for both the workstation and the user. The Desktop Management agent accomplishes this by sending XML requests for information to the middle-tier server. In return, the middle-tier Web components retrieve the informationÃ¢â‚¬â€either from eDirectory or from the file systemÃ¢â‚¬â€and return it to the workstation agents either through XML or with WebDav or HTTP file transmissions.
Agent File Transfers Through the Middle-tier Server
The agent resolves file references using one of three alternative mediums:
- The Novell ClientÃ¢â€žÂ¢, if it is present on the workstation (not required for ZENworks)
- The Microsoft* Client, if a drive mapping or an accessible UNC share is referenced
- The ZENworks Middle Tier Server
As shown in Connection 2 in the figure, when the Desktop Management agent goes to retrieve a file from the network, it uses the client first if the file reference is a mapped drive or an accessible UNC share. If the file reference is not a mapped drive or if the UNC share is inaccessible, the Desktop Management agent requests a file and passes the UNC to the middle-tier server over Connection 1. The middle-tier server will then use the UNC path to retrieve the file and send it back to the workstation over Connection 1. If administrators want to provide MSI-based applications to users who do not have the ability to connect directly to the file server holding the MSI file, they must administer the MSI application object as force-cache. The force-cache flag causes the Novell Application LauncherÃ¢â€žÂ¢ to immediately request the MSI files through the middle-tier server and to store the retrieved MSI files in the local workstation cache. When the application is launched, the Microsoft Installer process is called, passing the local path to the MSI files in the cache.
Middle-tier Connectivity to eDirectory
The following figure represents communication conduits that can be created during normal activity among a Desktop Management agent, the middletier server and eDirectory.
When a Desktop Management agent communicates to the middle-tier server, it opens an HTTP connection to the Web server. This is a standard connection that is terminated when the communication between the workstation and server is complete; however, to provide continuity between HTTP connections, the Web server creates a session cookie that is subsequently used between the client and server. The session terminates when no communication has occurred between the workstation and the server, based on configured parameters on the Web server (traditionally 15 minutes). If the user logs out of eDirectory, the middle-tier server terminates the session.
When the middle-tier server needs to communicate with eDirectory to authenticate the user and workstation, it creates a connection between the middle-tier server and the eDirectory server. Because the cost of creating this connection is high, the middle-tier server maintains it for the length of the session rather than of the HTTP connection. When the session is terminated, either because it timed out or because the user explicitly logged off, the eDirectory connection is put into an available pool for use by another user. If the connection is not used after five additional minutes, it closes.
Middle Tier Performance
There are several factors that contribute to what scale is achieved on a middle-tier server. The factors having the greatest impact include the following:
- Speed and number of CPU processors on the middle-tier server
- Connectivity speed between the middle-tier server and eDirectory
- Amount of data to push to each user and workstation, particularly force-cache applications and policies
- Amount of RAM available on the middle-tier server
- Staggered login times and launcher refresh intervals
Considering these factors, you can approach the architecture for middle-tier installation by first improving the hardware speeds on the middle-tier server. When the middle-tier server is no longer experiencing high utilization, the next bottleneck is the connection between the middle-tier server and the eDirectory server. This can be addressed with a higher-speed connection between these two machines. Additionally, IIS and Apache can provide a significant improvement in performance if configuration modifications are applied. It is recommended, to improve performance, to modify the ThreadsPerChild configuration in the Apache adminserv.conf file. Additionally, improvements in IIS can be accomplished by following the suggestions from a Microsoft published whitepaper titled Ã¢â‚¬Å“The Art and Science of Web Server Tuning with Internet Information Services 5.0Ã¯Â¿Â½? by George Reilly, published March 9, 2001
Performance Research in a Novell Test Lab
Performance tests conducted in a Novell test lab implemented Novell NetWareÃ‚Â® 6.0 servers, NetWare 6.5 servers, Windows* 2000 servers and Windows Server 2003 machines with the following specifications:
- Dual Intel* Xeon* 2.8 GHz Processors
- Ultra-wide 320 SCSI hard drives
- Gigabit NIC
- 2 GB RAM
- Hyperthreading enabled on Windows servers
- Hyperthreading disabled on NetWare servers
After an initial application distribution, day-today performance normally consists of routine traffic from the Application Launcher through the middletier server to eDirectory as ZENworks checks for updates to its currently installed application base. Three critical tests were performed on middletier and back-end servers with this configuration, with ZENworks 6.5 SP1 installed. ZENworks 6.5 SP1 represents almost a 20% performance improvement over previous releases. The following information describes the results of those tests.
Performance Test 1 The first performance test was an authentication-only test of workstations authenticating simultaneously to a single middle-tier server. The results of this test on the supported server platforms are shown in the table below:
|OPERATING SYSTEM||1 CLIENT||240 CLIENTS||460 CLIENTS|
|NetWare 6||5 seconds||36 seconds||51 seconds|
|Windows 2000||9 seconds||50 seconds||48 seconds|
|NetWare 6.5||6 seconds||50 seconds||50 seconds|
|Windows Server 2003||7 seconds||26 seconds||50 seconds|
Performance Test 2 The second test shows the results of an Application Launcher non-initial refreshes of 40 applications:
|OPERATING SYSTEM||1 CLIENT||240 CLIENTS||460 CLIENTS|
|NetWare 6||7 seconds||1 minute, 14 seconds||2 minutes, 18 seconds|
|Windows 2000||21 seconds||1 minute, 10 seconds||1 minute, 6 seconds|
|NetWare 6.5||6 seconds||28 seconds||1 minute, 5 seconds|
|Windows Server 2003||6 seconds||1 minute, 2 seconds||1 minute, 3 seconds|
Applications used for this distribution test ranged in size from 2 MB to 17 MB. Applications were randomly associated to containers, groups and users. NOTE: It is uncommon for 460 users to log in simultaneously every day and then for each to trigger the distribution of 40 applications, therefore individual performance may be better than shown. Initial refreshes will take longer in order to fill the data cached on the device.
Performance Test 3 The third test shows the results of an Application Launcher non-initial refreshes of 100 applications:
|OPERATING SYSTEM||1 CLIENT||240 CLIENTS||460 CLIENTS|
|NetWare 6||13 seconds||3 minutes, 45 seconds||7 minutes, 18 seconds|
|Windows 2000||46 seconds||3 minutes, 17 seconds||4 minutes, 21 seconds|
|NetWare 6.5||10 seconds||1 minute, 26 seconds||3 minutes, 32 seconds|
|Windows Server 2003||13 seconds||2 minutes, 53 seconds||2 minutes, 35 seconds|
Applications used for this distribution test ranged in size from 2 MB to 300 MB. Applications were randomly associated with containers, groups and users.
NOTE: It is uncommon for 460 users to log in simultaneously every day and then for each to trigger the distribution of 100 applications, therefore individual performance may be better than shown. Initial refreshes will take longer in order to fill the data cached on the device.
A Customer Environment Tuned for Performance
The following diagram represents the ZENworks Desktop Management architecture used by a Novell customer for a large, Windows-centric production environment.
The details of this setup are as follows:
- Six ZENworks Middle Tier Servers are installed on Windows 2000 servers, all located in a Microsoft load-balanced cluster. The cluster is configured to allow 900-1200 concurrent connections per middle-tier server. The number of connections varies depending on the size of the data set being transferred. (Each server is an HP/Compaq* DL360 with a dual Xeon 4.0 GHz processor, 2 GB RAM and a single 1-GB network card.)
- Three Windows 2000 servers have Novell eDirectory and the ZENworks Desktop Management Server installed. (Each server is an HP/Compaq* DL360 with a dual Xeon 4.0 GHz processor, 2 GB RAM and a single 1-GB network card.)
- Middle-tier servers 1 and 2 point to eDirectory Server A. Middle-tier server 1 fails over to eDirectory Server B and then to eDirectory Server C. Middle-tier server 2 fails over to eDirectory Server C and then to eDirectory Server B. Middle-tier server 3 fails over to eDirectory Server A and then to eDirectory Server C.
- Desktop Management agents receive a DNS name for the middle-tier server from a local DHCP server. At headquarters, the middle-tier server DNS name points to the IP address that is assigned to the Microsoft cluster.
Although this round-robin DNS addressing configuration works, we recommend using an L4 switch. For more information, see the sections on Ã¢â‚¬Å“Round-robin DNS AddressingÃ¯Â¿Â½? and Ã¢â‚¬Å“Web Switching.Ã¯Â¿Â½?
Summary of Expected Performance
The day-to-day performance of ZENworks 6.5 Desktop Management (that is, the performance after an initial application distribution) depends on the configuration of your network environment (for example, the hardware you are running and the amount of data being pushed through the network). However, given a configuration similar to the one described in this paper, you can expect each ZENworks Middle Tier Server to handle 900 to 1200 concurrent connections with good performance.
Windows Server TCP/IP tuning
Tuning the TCP/IP configuration on a Windows Middle Tier is needed if a large amount of workstations will be connecting through this Middle Tier server. In the Microsoft Windows Server 2003 TCP/IP Implementation Details there is some interesting information that can be used to optimize the TCP/IP configuration for the Middle Tier and the back-end eDirectory servers.
When a TCP connection is closed, the socket-pair is placed into a state known as TIME-WAIT. This is done so that a new connection does not use the same protocol, source IP address, destination IP address, source port, and destination port until enough time has passed to ensure that any segments that may have been misrouted or delayed are not delivered unexpectedly. RFC 793 specifies the length of time that the socket-pair should not be reused as two maximum segment lifetimes (2 MSL), or four minutes. This is the default setting for Windows Server 2003 TCP/IP. However, with this default setting, some network applications that perform many outbound connections in a short time may use up all available ports before the ports can be recycled. Windows Server 2003 TCP/IP offers two methods of controlling this behavior. First, the TcpTimedWaitDelay registry parameter can be used to alter this value. Windows Server 2003 TCP/IP allows it to be set as low as 30 seconds, which should not cause problems in most environments. Second, the number of user-accessible ephemeral ports that can be used to source outbound connections is configurable using the MaxUserPort registry parameter. By default, when an application requests any socket from the system to use for an outbound call, a port between the values of 1024 and 5000 is supplied. The MaxUserPort parameter can be used to set the value of the uppermost port that the administrator chooses to allow for outbound connections. For instance, setting this value to 10,000 (decimal) would make approximately 9000 user ports available for outbound connections.
For servers with a large amount of workstations connected use the following settings:
MaxUserPort - Use the regedit command, access the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Services\TCPIP\Parameters registry subkey, and create a new REG_DWORD value named MaxUserPort - Set this value to decimal 32768 (Hex 0x00008000).
TcpTimedWaitDelay - Use the regedit command, access the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Services\TCPIP\Parameters registry subkey, and create a new REG_DWORD value named TcpTimedWaitDelay - Set the value to decimal 30 (Hex 0x0000001e). This value sets the wait time to 30 seconds.
After adding these settings to the registry the server will need to be restarted.
Increasing Middle Tier Scalability and Availability
In addition to tuning your hardware and software parameters to provide the best performance of your middle-tier server in your environment, there are additional configurations and architectures that you can use to increase its scalability and availability.
Delivering Data through File Servers
The load on an individual middle-tier server can be greatly reduced if applications and other required data can be retrieved through a local file server. This option can be accomplished by first configuring a mapped drive on the local workstation or referencing files via a UNC share to a local server. Then the application objects delivered to that workstation or workstation users can reference the mapped drive or share in order to retrieve the application files and other data. This option is available only to those who have a LAN connection. Workstations that retrieve information through the Internet but are not connected to the LAN must receive their distributions through the middle-tier server. It is advisable to use the randomizing features of the Novell Application Launcher to direct the workstations to connect at random intervals during a specified time frame. This precaution spreads the number of users connecting and retrieving files across the defined time frame rather than allowing all users to connect at the same time (when the workday begins, for example).
Multiple Web Servers
Performance increases when fewer users are sent to a single middle-tier server. This can be accomplished by directing Desktop Management agents on different workstations to different middle-tier servers. When the agent is installed, a registry key is set on the workstation to maintain the DNS/IP address of the middle-tier server to be contacted by the agent. Using the installation parameters for various agent installations, you can set the middle-tier server address for different sets of users. Even so, the application objects accessed by various middle-tier servers can still refer to the same storage server.
Round-robin DNS Addressing
If you want to enhance the availability of the middle-tier servers within your environment, you can introduce more middle-tier servers into the network and use DNS addressing techniques to provide a set of several IP addresses corresponding to one DNS name. Using this addressing method, when a workstation requests the IP address of a given DNS name, the DNS server returns several different IP addresses. This spreads the load across multiple middle-tier servers. When a workstation is successfully connected to a middle-tier server, it uses that middle-tier server until the session is terminated. (See Ã¢â‚¬Å“Connection 2Ã¯Â¿Â½? on page 6.)
Round-robin DNS addressing does not detect a disabled server within the group of addresses. This limitation introduces the potential for degradation in system performance because the Desktop Management agent can continually attempt to establish connections to a disabled server before attempting to connect to the next server on the list.
Another technique for obtaining increased scalability and availability is using an L4 (or higher) switch. You can administer such a switch to provide access to several servers that are referenced by the same IP address. This way, your agents can be given the same IP address yet can receive the benefits of multiple middle-tier servers. This configuration provides all of the benefits of round-robin DNS addressing described above. A load-balancing switch is aware of the loads of each of the individual servers and can use that information can also quickly detect a disabled device and then route requests away from it.
to distribute requests. Load-balancing switches If you decide to use a load-balancing switch, select one that allows the agent to establish and maintain a session with the middle-tier server.
For configutation info have a look at Implementing a L4 switch with Middle Tier servers
ZENworks Middle Tier Servers can be scaled out or up to provide services to a larger audience. Even with server hardware improvements, administrators must configure the system properly in order to provide the greatest potential performance for their users. Administrators should be aware of the following items:
Ã¢â‚¬Â¢ Connectivity speed between middle-tier servers and eDirectory. Separating these server functions among different servers provides maximum support. Ã¢â‚¬Â¢ The amount of data being pushed to each user and workstation through the middle tier, particularly force-cache applications and policies. Using other file servers can off-load the middle tier and improve performance at login. Ã¢â‚¬Â¢ Staggered login times and launcher refresh intervals. Better performance is possible if users do not simultaneously log in and refresh their agents. Using these guidelines, the ZENworks Middle Tier Server can support many users with a robust and positive performance experience.