NetWare to OES2
Migrating from NetWare to OES2 SP1
The remainder of this document refers to the state of the migration tools at the release of OES2SP1. In the mean time, Novell has fixed many bugs in the migration tools. While some areas (like iPrint and DHCP migration) still have issues left, other areas (like file migration) have become quite stable.
Given the important updates after the release of OES2SP1, it is essential that you fully patch the server from the update channel before tempting a migration.
There are a number of documents and articles that explain how to use the OES2 migration tools and especially the new GUI migration tool from OES2 SP1. However all these articles just tell you how to use the tools in Ã¢â‚¬Å“normalÃ¢â‚¬Â situations, and there is very little troubleshooting information available to help you when things donÃ¢â‚¬â„¢t go as expected. Now the truth is that the migration tools as included with the release of OES2 SP1 are still far form robust and you are quite likely to encounter problems when using them.
So the scope of this document is not to give you a step by step guide on how to do migrations. There is the official documentation and a number of other documents that tell cover the various migration steps more than adequately. Instead, this document tries to give you more background information how the migration tools work and interact with each other and what kind of problems you might encounter and how to deal with these problems. It also gives you a number of recommendations that might help you avoid some problems and allow things to go smoother. Be aware that this document is based on the state of the migration tools as they are with the release of OES2SP1. A number of bugs have already been fixed in the migration tools, but these fixes are not yet available to the public at the moment of writing this document. Of course, the recommendations do not hurt in case where bugs are fixed later on, and it is always better to be safe than sorry.
This document mainly targets the use of the migration tools to migrate from NetWare to Linux, but most of the document should also be relevant for Linux to Linux migrations.
A bit of history
NetWare to NetWare Migration tools
Long ago, Novell developed the NetWare Migration Wizard (MW) to allow simple upgrade/migration from a server to new hardware. Later, around the release of NetWare 6, Novell added the Server Consolidation Utility (SCU) which was mainly a tool to move printers or data from one server to a different one without migrating the server as a whole. The Server Consolidation Utility was not actually a separate tool, but was en enhancement to the Migration Wizard. By launching the Migration Wizard with the appropriate command line options, you actually got the SCU user interface. Later on, Novell acknowledged that the 2 tools were just one and the same and distributed it under the name Ã¢â‚¬Å“Novell Server Consolidation and Migration ToolkitÃ¢â‚¬Â (SCMT). While the SCMT may still have its quirks, it is a tool that is simple to use and generally works quite well.
When OES was released in March 2005, it came without any specific migration tools. The only tool that existed was the Server Consolidation Utility and it was only slightly updated for compatibility with OES. However the tool could only be used to copy files to an OES server. No other services could be migrated nor was there any kind of ID swap procedure. So customers that decided to move to OES1 had a lot of work because they need to recreate the configuration for all other services themselves (printers, DHCP, DNS ... )
After the customers complained a lot about the lack of migration tools to go from NetWare to OES, Novell proudly announced that with the release of OES2, the migration tools that the customers had been asking for had finally been included. Alas, what Novell delivered was not really what customers had been asking for. Customers just wanted a tool like the SCMT which they were used to with NetWare. What Novell delivered with OES2 was a collection of scripts and tools that could migrate individual services form NetWare to OES2/Linux. Furthermore, for a few of the tools, there was a quick and dirty Yast interface to run the tool from a graphical user interface.
"This time we have really listened to you and provided you with an integrated GUI migration tool" said Novell proudly to their customers when they released OES2SP1 in December 2008, almost 4 years after the release of the first OES version. Indeed, from the functionally point of view, it seems that the new migration GUI indeed provides an ease of use similar to the old SCMT for NetWare. Unfortunately, the poorly designed architecture of the migration tools, the lack of experience of the developers and the lack of testing in real world scenarios makes the new migration GUI very fragile and often unusable.
The Architecture of the migration tools
If you want to characterize the OES2SP1 migration tools architecture in one sentence, you can simply say that there is no architecture. It is just a collection of independent tools tied together by a common GUI. It seems that each team responsible for one of the OES2 services has written a tool to migrate their service. The tools are written in many different programming languages, mostly in script languages. Only very few tools are binary tools, and in those cases, the binary tools are not directly called by the user, but rather they are helper tools called by the main tool written in some script languages. So far, I have been able to identify tools written as shell scripts or in Perl or Ruby. Other tools have been written in Mono or Java. The very few binary tools have probably been written in C.
Overall, there seems to be a severe lack of overall guidelines on how these tools work, how they should be called and how they provide status and error information. The migration GUI has the hard task to correctly call these various tools and get meaningful status information back from the tools, in practice, it often miserably fails in distinguishing errors from information messages.
How the migration tools communicate with the servers
The migration tools use various protocols to communicate with the source and destination servers. Here is a list of protocols used and how they are used. In a later section, we discuss what kind of problems can be encountered with these protocols.
All eDirectory access by the migration tools is done through LDAP. The verification of the authentication information entered in the GUI migration tool is also done through LDAP.
There are 2 kinds of NCP access done using the NCP protocol. ncpfs is a third party NCP client implementation of the NCP protocol for Linux and is used by the migration GUI to model the file copy operations. ncpshell is a small command line NCP client that is included with the OES2 NCP server. It does not mount a NetWare volume as file system, but rather allows the performing simple NCP commands use a command line. The commands that can be performed by ncpshell are for example the possibility to copy a single file to or from a NetWare server, execute a NetWare console command, mount or dismount a NetWare volume or list the NLMs loaded on the server.
The Storage Management Services is an architecture designed by Novell to backup servers. It has initially been developed for NetWare but has been ported to OES as well. It consists of the SMDR service and a number of TSA services to access different kinds of data types to be backed up. The migration tools mainly use SMS for bulk data copy using the nbackup tool, but they sometimes also use SMS to copy single files (instead of the simpler to use ncpshell).
SSH is not used on NetWare servers, but only on OES/Linux servers in order to execute commands remotely (a bit like ncpshell is used to execute commands on NetWare servers).
Potential problem sources for the migration tools
Special character handling
The migration tools, and above all the migration GUI have terrible problems handling anything that is not standard alphanumeric characters, whether it is in file or directory names, in eDirectory object names or attributes and in passwords. Characters that can cause problems are blanks (spaces or tabs), non alphanumeric ASCII characters and extended (non ASCII) characters. One source of these problems is there is not a single migration tool that handles all strings internally, but that there are multiple tools which call each other with file names, object names or passwords on the command line. Now spaces or special characters can negatively interfere with the command line decoding and can cause various migration tools to be called incorrectly. There are plenty of places where, for example, spaces in object or container names cause the migration tools to fail. Novell has already fixed a number of these problems, but so far, these fixes are not released to the public yet. Beside these character problems that are the result of command line parsing issues, there is also the issue of extended characters. Here, the situation is very bad. The migration tools simply seem to be completely untested with non English languages and extended characters in file or object names, or even just in the descriptive text for printers are very often fatal to the migration process.
On NetWare servers, especially on older versions (5.1 or 6.0), LDAP is often hardly used and therefore does often not work correctly because of expired certificates or because it was never configured correctly. This becomes an issue with the migration tools as the migration tools require LDAP to verify the userÃ¢â‚¬â„¢s credentials and because LDAP is used for any kind of eDirectory access. IT is even more an issue in case of ID swap migrations because after the migration, the destination server will inherit the certificates and the LDAP configuration and for OES2 itself, a fully working LDAP server is essential. Note that different migration tools are more or less permissive on expired certificates. So for instance, you might be able to log into the main migration GUI when your LDAP certificate has expired, but the printer migration will fail with expired certificates. This can sometime make this issue very hard to detect.
Unfortunately, Novell decided to not support their own Linux client on OES2, but instead only use ncpfs, an open source client that is not written by Novell and that is based on reverse engineering of the NCP protocol. This client has many limitations, the most important in the context of migrations being the lack of UTF8 support and the default use of UDP instead of TCP. Given ncpfs is used to mount the volume of the source server in order to model the file copy and given the migration tools do not specify a code page when mounting an NCP volume, all directory and file names that have extended characters will not be displayed in the migration tool. Furthermore, if you have NCP over UDP disabled, or if your source volume is a cluster volume, then the connection to the source server will fail because of lack of NCP over UDP support. Novell is working on the ability to specify or detect the code page to be used by the migration tools and that would fix the missing directories problem. However, currently this fix is currently not yet available to the public.
There are two kinds of issues you might encounter with the file copies over SMS. The first issue is a connection issue. This is the more likely to occur if the server is an older NetWare version. However it has also been seen with the source server being a NetWare 6.5 server. Provided your SMDR and TSA configuration is OK on the source server, this problem can generally be avoided by updating the hosts files both on the source and destination servers to make sure that both source and destination servers are listed in both hosts files, and by using the host name instead of the IP address when specifying the source server in the migration tool. Be warned however that this connection by name should only be used for the file system migration. If you plan to do an ID swap, make sure you change back to using an IP address before you run the ID swap itself. Otherwise, the ID swap procedure will replace all occurrences of your destination serverÃ¢â‚¬â„¢s IP address with the string Ã¢â‚¬Å“nullÃ¢â‚¬Â. The second potential SMS issue is the copy of data from a non UTF8 enabled source. This can be a NetWare 5.1 server for any file system, or a NetWare 6.x server where the source file system is not NSS. In this case, the names of the files are transferred according to the source serverÃ¢â‚¬â„¢s code page and before doing the file copy, you have to make sure the same code page is configured in the destination serverÃ¢â‚¬â„¢s /etc/opt/novell/sms/tsafs.conf file.
The error handling in the migration GUI is very poor. This is due to the heterogeneous nature of the migration tools. Whenever the GUI runs a tool to do a specific migration task, it does at the same time start a thread that watches all output from the tool to determine whether there are errors or not. Unfortunately, given the possibly huge variety of messages, the migration GUI very often fails to correctly distinguish errors from purely informational messages. The result of this is not only that the migration GUI often fails to detect that a migration step has gone wrong, but the error messages are often not even recorded in normal log files.
Where to look for errors
The migration GUI records messages in a number of different log files. These log files are stored in the log directory below the main project directory. There are 3 different categories of log files generated by the migration GUI:
- the service specific log. For each type of service that is migrated, there is a service specific log file that just records the messages related to the migration of that service. The corresponding log files are afp.log, av.log, cifs.log, dhcp.log, dns.log, edirectory.log, filesystem.log, ftp.log, ifolder.log, iprint.log, ntp.log, serveridswap.log. When you view the log at the bottom of the migrations screen, you will see the content of the log file for the last migrated service. Note that the log display is not automatically updated. You may need to do a refresh to see the latest messages
- the global migration log. The same messages (more or less) that are recorded in the service specific log files are also stored in the file migration.log. This file can be accessed using the log icon in the left margin of the migration screen.
- The debug log. This log was probably only intended for Novell use. The service specific logs and the global migration log are filtered and show only messages that the migration GUI considers useful for the user. The debug.log file however contains very detailed messages enumerating each individual step done by a migration tool and records all messages output by the migration tools. Because of this, the file can become huge in size, especially when doing file system migrations. Each time the file reaches a size of 1M the old file is renamed to debug.log.<n> where <n> is an ongoing number and a new debug.log file is generated. For big migration projects, hundreds of megabytes worth of debug files can thus be generated. The debug.log file is very important because it is often the only place where actual error messages can be seen. However because of the huge size of the debug.log files, it can be very hard to find the important messages. The debug.log files are not accessible from the migration GUI. You have to browse the file system and use an editor or file viewer of your choice to view the debug files.
General migration tips and tricks
- Make sure you always have the migration documentation available. Best is to download the Migration Tool Administration Guide as PDF from Novell's OES2 documentation web site and print it out. Make sure you use the current version as the documentation is continuously getting updated, and especially the migration documentation has received a number of corrections since the release of OES2 SP1.
- Before starting the migration, update the hosts files on both source and destination servers and make sure that both the source and the destination servers can be found in both hosts files. Double check that you get the IP address and name right and that there are no other rogue entries in the hosts files with the same IP addresses. Wrong entries in the hosts files can cause difficult to debug migration problems.
- Make sure your source server has no expired SSL certificates. Just to be safe, run PKIDIAG.NLM and let it fix the certificates. If there was any change to the certificates, unload and reload NLDAP or restart the server to make sure the new certificates are being used.
- Check whether your servers are recorded in DNS. It is not required that your servers have valid DNS entries, but if there are DNS entries for your server names or server IP addresses, then they must be correct. Incorrect DNS entries corresponding to your server names or server addresses can cause migration problems that are difficult to debug.
- Make sure your destination server is fully patched prior to the migration. Install the recommended updates available from the subscription channel.
- Update the SMS modules on the source server. This is especially the case for NetWare 5.1 and 6.0 servers. While not officially supported, it seems that in practice, the updated modules from TSA5UP23.ZIP work better on NetWare 5.1 and 6.0 compared to the last versions supported on these platforms. For NetWare 5.1, you should also get the following updated TSA500.NLM.
- For file system migrations, if you get connection failures, use the name of the source server instead of its IP address in the migration project. This avoids connection failures that may otherwise occur. Note that connection errors are common with older SMS modules on the source server, but if you use the latest SMS modules as described in the previous paragraph, they should be rare.
- For ID swap migrations, make sure you use the IP address and not the name of the source server. Otherwise, the migration tool will wipe out the IP address from all configuration files and give you a lot of work manually fixing all services. If you do both file system and ID swap migration in one project, change your project configuration before doing the actual ID swap.
- When you have authentication problems with LDAP or when you are in doubt over your LDAP configuration on the source server, best is to recreate your LDAP configuration. This can be easily done by deleting the LDAP group and server objects for the source server and by recreating them. Having a healthy LDAP configuration is especially important for a server ID swap migration because after the migration, the destination server will inherit the LDAP configuration of the source server, and without a fully working LDAP configuration, the repair operations at the end of the ID swap procedure itself may already not be able to fully perform their task.
- If you are not doing an ID swap migration and you have LDAP problems specific to secure LDAP, then you can try disabling SSL for LDAP. For this to work, you do however need to change the configuration of your LDAP server on the source server to allow authentication on non secure connections. This is explained in the eDirectory configuration.
- For ID swap migrations, install the destination server into the same context as the source server. Do not forget to use the "premigration" pattern when configuring OES2 during the installation. In case you forget to install the destination server in premigration mode, remove all replicas from the destination server before doing the ID swap part of the migration.
- The ncpshell command (used by the iPrint and ID swap migrations) must have the reject ncp packets with bad components and reject ncp packets with bad lengths set parameters set to OFF on source NetWare servers. Otherwise, it may give a return code of 35198.
- Where possible, avoid spaces and special characters. This includes the migration project name, file or directory names, NDS object names and passwords. Of course, you canÃ¢â‚¬â„¢t change your tree or file system structure. However you generally can control how you call your migration project, what admin user you use for the migration and what password you assign to your admin and root users.
- After each migration step, check the debug.log file to see if there hasnÃ¢â‚¬â„¢t been any hidden failure. When a step fails and you canÃ¢â‚¬â„¢t get it to work from the migration GUI, find the command being used for that migration step in the debug.log file and try to manually executing it from a shell window.
The official migration tools documentation: http://www.novell.com/documentation/oes2/mig_tools_lx/data/bookinfo.html
Some tips for nbackup problems and an updated TSA500.NLM: http://www.novell.com/coolsolutions/tip/19709.html
The lastest SMS update. While officially only supported on NetWare 6.5, the update also works on NetWare 5.1 and 6.0: http://download.novell.com/Download?buildid=hnMn6vIgSyQ~
A troubleshooting TID for migration issues. It mainly concentrates on SMS related issues: http://www.novell.com/support/viewContent.do?externalId=7001767