Performance

In this chapter the performace of the different directory servers presented in chapters , and will be evaluated and compared. For reliable operation of a directory service, it must be ensured, that the server can meet the expected requirements. As the needs of the various applications can be very different, they should be analysed and tested, before a general judgement of the suitability of a product for various applications can be given.

The DirectoryMark^11.1benchmark by Mindcraft has been designed to evaluate the performance of directory servers. This benchmark can be tuned to test a specific combination of LDAP operations. It will therefore allow an exact simulation of the prevailing usage patterns in the planned service.

Mindcraft has identified three common scenarios for directory services: loading, messaging and address look-up. [70] An additional scenario was identified during the work on this thesis, i.e. the use of an LDAP server as an authentication back-end.

Test Environment

All tests were performed on an AMD Athlon 750 MHz with 256 MB of RAM. SuSE Linux 7.0 (Kernel 2.2.16) was used as Linux distribution. The Active Directory test was done under Windows 2000 Server (Servcie Pack 1, Build 5.00.2195). The client machine, on which the DirectoryMark 1.2 software was run, was an Intel PIII 500 with 128 MB of RAM runnig Windows 2000.

The DirectoryMark suite was used to actually perform the tests. It consisted of three components:

dbgen.pl generated the sample entries on which the tests were subsequently run. It created ``inetOrgPerson'' entries which were arranged below one of four ``organizationalUnit'' entries. The data generated was output into an LDIF file by dbgen.pl which was then imported into the different directory servers (see Section ). LDIF (LDAP Data Interchange Format, [15]) is a textual representation of a DIT. It is the standard format for exchanging directory data and is supported by all software packages in this test.

scriptgen.pl created scripts that instructed the simulator what to do. The behaviour of this script could be modified by a number of parameters. Among other things it was possible to specify the number of threads and operations per thread to create, as well as the weighing factor of the different LDAP operations.

DirectoryMark.exe simulated LDAP clients. It processed the scripts generated by scriptgen.pl and gave a statistic result as output.

For use in this thesis, the dbgen.pl utility was customized:

The length of the generated passwords was reduced from 10 to 8 as Unix systems will only take the first 8 characters into account.

Seven attributes, which were not included in the test data for Active Directory but are available in Active Directory's schema, were now also generated for Active Directory.

The ``unicodePwd'' attribute was used instead of ``userPassword'' in the test data for Active Directory. This allowed for setting a user's password in Active Directory via LDAP.

As distributed, the script chose a random name for the ``manager'' and ``secretary'' fields. This behaviour was altered so that these fields were now filled with DNs of previously generated entries.

The main reason for the last modification was that NDS does a referential integrity check on attributes that contain DNs, i.e. it will only allow such values that point to existing entries.^11.2

Finally, support for the authentication scenario was added to scriptgen.pl.

Tests

For the tests performed in this analysis, 20,000 sample entries were generated. This number was chosen to reflect the user population at Tübingen University Computing Centre. (see Section )

The messaging, address look-up, and authentication tests modeled a maximum load scenario. For a period of five minutes the test client sent request upon request to the server. In the address look-up and authentication scenario the client spawned four parallel threads to simulate concurrent access. The messaging test used only one thread. Each thread processed a script generated by scriptgen.pl with 10,000 operations. If a thread completed the execution of a script during the five minutes period, it restarted again.

Each test was consecutively run four times. The first run was intended to allow the server to populate its caches. Its results were therefore discarded. The numbers in the tables show the average of the remaining three runs.

Loading - Initial directory population

Before a test on a directory server could be performed, the test data had to be loaded first. Where available, vendor specific tools were been used to import the LDIF file generated by dbgen.pl. These ``bulkload'' tools assumed that the syntax of the data was correct and therefore would bypass any consistency checks. Some of the utilities could also write directly to the underlying database, in which case the LDAP server had to be stopped. Others used framing technology (e.g. eDirectory) to add multiple records at a time or allowed delayed writing to disk (e.g. Active Directory).

Figure: Bulk-load times for 20,000 accounts

[IMAGE png]

(128)

Microsoft's bulk-load tool ldifde could not be used to import passwords. Therefore OpenLDAP's ldapmodify with an SSL connection and GSSAPI authentication was used.

(129)

On import eDirectory generated a public/private key pair for each user, which was very CPU intensive. The ice utility had an option (``-l'') to turn this off. However, this lead to intermittent failures (``unknown error''), so this option could not be used.

(130)

With OpenLDAP, the best performance was achieved in a two step process. Entries were first added through slapadd without any indices defined (0:44). Index settings were then enabled in the config file and the indices generated with slapindex (3:38). Data was imported with schema check and dbsync disabled since the data was correct and the database did not need to be flushed after each operation.

(131)

Use of the provided tools (bulkload, ldif2db) only resulted in an error: ``Error code -1 from odbc string: 'SQLAllocEnv'``. OpenLDAP's ldapmodify was used instead.

Messaging

Figure: Results for the messaging scenario

[IMAGE png]

The messaging scenario simulated a mail transfer agent (MTA) that queried a directory for mail routing information. After initialising itself the MTA opened a connection, which stayed open while the MTA was running. For each incoming email, it then issued a query for the local part of the email address. The result then indicated to which host the mail should be forwarded and if the ``to'' field should be changed.

Address Look-up

Figure: Results for the address look-up scenario

[IMAGE png]

In the address setup the directory servers acted as an email look-up service for an organization. Users would use the directory to search for email addresses. The search pattern was a combination of substring and exact filters on given name, surname, common name and userid. After every fifth search, the connection was closed and reopened.

Authentication

This test was setup to simulate an LDAP authentication back-end. A client first opened a connection to the server and searched for the username, i.e. an exact subtree search on the ``uid'' attribute. If the user existed in the directory, it would then try to bind as the user and finally retrieve the information contained in the user's entry for further use, i.e. perform a base search on the user's DN.

Due to a limitation in DirectoryMark.exe, the operations had to be reordered. The bind operation was now executed first. As DirectoryMark.exe only counted the number of search requests, the figures reported had to be halved, in order to get a correct estimate for the maximum capacity of the authentication back-end.

Figure: Results for the authentication scenario

[IMAGE png]

Results

Figure: Results overview

[IMAGE png]

Figure shows the results from the query tests. Overall, Active Directory and the Netscape Directory Server stand out with regard to performance. If it was not for the bad result in the authentication test, eDirectory would be in joint third place along with OpenLDAP. Although M-Vault offers better authentication performance than eDirectory, it falls into fifth place overall.

Addtinional to its last place in the performance comparison, the Linux ``Technology Preview'' version of SecureWay Directory suffers from stability problems. The directory server silently died in the authentication tests with multiple concurrent clients. These errors will hopefully be fixed in the production release.

Comparison with real-life requirements

Figure: Daily amount of emails and login attempts at the Computing Centre

[IMAGE png]

The Computing Centre at Tübingen University handles email and computer accounts for students and academic staff and has a user population of about 19,000.

As displayed in Figure , the daily number of emails handled by the MTA goes up to 70,000. Historically the peak value has been 220,000 emails on one day. This was caused by a misconfigured client, which provoked an email loop. These figures represend the total numbers of emails handeled by the MTA. However, a directory look-up will only be performed for emails addressed to local recipients. When compared with the performance numbers, it is obvious that leading products can easily handle this load. It takes Active Directory and the Netscape Directory Server under 10 minutes to process a full day's peak load. OpenLDAP and eDirectory would handle this load in about a quarter of an hour.

The number of client logins to retrieve emails is quite constant over the respective three month period. A clear distinction is visible between the load on work days and weekends. There are about 25,000 IMAP and 55,000 POP3 logins on an average day, yielding a total of 80,000 authentication attemps per day. As in the messaging scenario, 80,000 login attempts to not pose any substantial requirements on the host running the directory server.

Combining the numbers for email routing and authentication, a directory server would have to handle an average number of 150,000 search requests a day. On the other hand, there are only 30 to 40 changes on an average day. Since the modification rate is less than 0.001 of the total number of operations, no tests dealing with operations, which change the data in the directory, were performed.