LPI Exam 302 Prep (Mixed environments): Performance tuning

Prerequisites

To get the most from the articles in this series, you should have an advanced knowledge of Linux and a working Linux system on which you can practice the commands covered in this article. You should also have a good understanding of TCP/IP networking.

Measuring Samba performance

Before you can improve something, you must be able to reliably measure it. Measure something, make changes, measure again, and compare results. In the case of Samba, you need to measure:

◉ Response time and throughput of a client under idle server conditions

◉ Response time and throughput of a client under a given server load

◉ Server characteristics under a given load

◉ Maximum server capacity in terms of clients or throughput

Measuring response times gives you an idea of what a typical client would expect under the test conditions. For this to work, your test cases should approximate actual client load. For example, seeing how fast you can repeatedly request the same file is not the same as copying a directory containing a mix of small and large files.

Measuring server characteristics under a given load gives you some idea of how much room you have to grow. If, under load, your server is struggling to keep up, then you know you don't have much capacity beyond the test load. The same techniques are used to measure a production server's load and extrapolate capacity.

Finally, the so-called "torture tests," where you throw all you can at a server and see where it breaks, provide interesting information but are not as useful as the other types of tests. If the goal is to prove that your server can handle a certain load, then this type of test will do. Such tests are often more helpful for measuring the lower-level characteristics, such as the maximum disk I/O capacity of a server.

Samba is a network server, so it is important to reasonably simulate the network being used. If your clients are 50 milliseconds away from the server, the effects of network latency will be more pronounced than with local clients. This information directs your tuning priorities accordingly.

Designing a test

You could buy a fairly expensive device that simulates client traffic and takes precise performance measurements. And if you're publishing benchmarks or developing the server hardware, such a device might be a good option. However, if you're interested in tuning your own server, and are faced with the usual constraints of time and money, you're probably not looking for an expensive tool that you'll have to learn how to use.

As an example, the first test assesses random read performance by downloading a large number of files with the Samba command-line client. The primary concern is, "how fast can this directory be downloaded?"

Like any well-behaved UNIX® utility, the smbclient tool can read its list of instructions from the standard input. The following snippet shows a series of commands for downloading the contents of a directory:

prompt
recurse
mget smbtest

The prompt command suppresses the client from asking you for confirmation of downloads, and recurse indicates that you want to descend into directories when you download multiple files. Finally, mget smbtest instructs the client to start downloading the smbtest directory. Fill that directory with a few hundred megabytes of test files, and you have all you need to test performance.

To run the test, connect to your share with smbclient, and redirect standard input to your file. Listing 1 shows how you do it.

Listing 1. Running the test

$ time smbclient '\\192.168.1.1\test' password < instructions
Domain=[BOB] OS=[Unix] Server=[Samba 3.5.8-75.fc13]
getting file \smbtest\file2 of size 524288000 as file2 (5323.2 kb/s) (average 5323.2 kb/s)
getting file \smbtest\file1 of size 139460608 as file1 (5275.3 kb/s) (average 5313.0 kb/s)

real    2m2.289s
user    0m0.509s
sys 0m4.580s

The command in Listing 1 starts with the time command, which times how long the command specified in the rest of the arguments takes to run. The command itself is smbclient, and the arguments to that are the name of the share and the user's password. You can add the typical arguments, such as -U to pass an alternate user name if your environment requires it. Finally, < instructions redirects the standard input of smbclient to the file called instructions, which contains the instructions from the first code snippet. The result of this command is a timed batch copy of several files.

The result of the command is a list of the files transferred, along with an average transfer rate per file. The output of time is added at the end, showing that copying roughly 664MB of files took 2 minutes and 2.289 wall-clock seconds. This is now the benchmark. If you make any changes and the test takes longer than 2 minutes, 2 seconds, then you're making things worse with your changes.

If you want to test just Samba parameters and negate the effects of local disk and caching, you can run the test several times and take the last measurement. Doing so ensures that the operating system caches as much as possible and minimizes disk access. As you test your changes, make sure you have a similar amount of free memory on your server; otherwise, the differences in the amount of cached data might alter the results of your experiment.

Viewing Samba's status

Looking at a server's CPU, memory, disk, and network information gives you some good information about the health of the server itself but provides no context for what the application is doing. Samba comes with a helpful utility called smbstatus that shows current connections and file activity; it's also good for both performance tuning and troubleshooting.

Listing 2. The smbstatus command

$ smbstatus
lp_load_ex: refreshing parameters
Initialising global parameters
params.c:pm_process() - Processing configuration file "/etc/samba/smb.conf"
Processing section "[global]"
Processing section "[homes]"
Processing section "[printers]"
Processing section "[extdrive]"

Samba version 3.5.8-75.fc13
PID     Username      Group         Machine                     
-------------------------------------------------------------------
17456     fred         fred       macbookpro-d0cd (::ffff:192.168.1.167)

Service      pid     machine       Connected at
-------------------------------------------------------
fred         17456   macbookpro-d0cd  Mon Jul 18 07:36:46 2011
extdrive     17456   macbookpro-d0cd  Mon Jul 18 07:36:46 2011

Locked files:
Pid    Uid   DenyMode   Access      R/W        Oplock      SharePath   Name   Time
----------------------------------------------------------------------------------
17456  505   DENY_NONE  0x100081    RDONLY     NONE        /home/fred   .
17456  505   DENY_NONE  0x100081    RDONLY     NONE        /home/fred   Documents

Listing 2 shows the current activity of the Samba server. One user, fred, is connected, and he has two shares mounted (fred and extdrive). There are also two locked directories.

Network tuning

Samba is a daemon that primarily sends and receives packets over the network. Several parameters alter how the packets are sent and how Samba interacts with the underlying operating system; these parameters can have a drastic effect on performance.

The basic options

Many daemons provide the best possible level of service to everyone and do not discriminate among clients. If the service is overloaded, everyone gets the same bad service. Contrast this to a telephone network: if congestion occurs, people aren't allowed to make new calls, but existing calls continue as though nothing's wrong. If you artificially limit certain options in Samba, you can prevent getting to a resource starvation situation.

The max connections parameter limits the number of connections you can have to a particular Samba server. Each connection takes memory and CPU resources, so it's possible to overload a server by having too many connections. By default, unlimited connections are allowed. If you need to limit a busy server, you can specify a hard limit with max connections.

The max smbd processes parameter controls the maximum number of processes that can be run. This parameter is similar to max connections but controls the number of processes resulting from the connections.

The more you log with log level, the more server resources are spent writing logs to disk. Keeping this value at 1 or 2 limits the amount of logs written to disk and saves more resources for serving clients.

Setting socket options

When an application asks the operating system to open a network connection, the application can also ask that the operating system treat the packets a certain way using socket options. Socket options can enable or disable network performance tweaks, set particular quality-of-service bits on the packets, or set kernel-level options on the socket.

The socket options command can control the type of service (TOS) bits on the packets. The TOS bits tell routers along the way how the traffic should be treated. If the routers are configured to respect these bits, the traffic can be handled in the manner the application asked for. The IPTOS_LOWDELAY keyword is most appropriate for low-delay networks such as LANs, while IPTOS_THROUGHPUT is for higher-latency WAN links. Your network may be configured differently, so it's possible that using the options might have the opposite effect.

The TCP_NODELAY option disables the Nagle algorithm, which is more appropriate for chatty protocols, at a cost of more packets being sent.

If you have firewalls or other devices that keep state in your network, you may be interested in the SO_KEEPALIVE option, which turns on TCP keepalives. These periodic packets keep the connection open and keep the state fresh inside firewalls. Otherwise, the firewall will drop the packets, and it will take some time for the client to realize it has to reconnect to the server.

Beyond tuning

Although playing with different settings and trying to come up with optimal configurations may be fun, some items outside the Samba configuration can really hamper performance. Any time the server is doing something other than reading data from disk and sending it to a client or receiving incoming network traffic and writing it to disk, you're making the operation take longer.

Ethernet errors

If a packet is lost in transmission, the kernel has to notice that the packet has gone missing and request a retransmission. This process can slow down a fast conversation, especially if the two sides are separated by high latency. One common source of packet loss is mismatched settings between the switch and the server. Either auto-negotiation fails to come up with a correct match or one side is set statically and the other side is still auto-negotiating. This discrepancy results in errors on both sides. You can look for such errors with the netstat command:

# netstat -d -i 2
Kernel Interface table
Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth1       1500   0  5258404      0      0      0  3024340      0      0      0 BMRU
eth1       1500   0  5258409      0      0      0  3024341      0      0      0 BMRU
eth1       1500   0  5258411      0      0      0  3024342      0      0      0 BMRU

This code shows the multipurpose netstat command looking for interface errors every 2 seconds with the -d -i 2 parameters. Every 2 seconds, the status of the available interfaces is shown. In the example above, the RX-OK and TX-OK columns show that packets are flowing in and out. The errors, drops, and overruns are all 0 in both directions, showing that there has been no packet loss.

If you see errors, determine what speed/duplex is being used with mii-tool or mii-diag. Listing 3 shows how to check your network settings.

Listing 3. Verifying network settings

# mii-tool
eth1: negotiated 100baseTx-FD, link ok
# mii-diag eth1
Basic registers of MII PHY #24:  3000 782d 0040 6177 05e1 41e1 0003 0000.
The autonegotiated capability is 01e0.
The autonegotiated media type is 100baseTx-FD.
Basic mode control register 0x3000: Auto-negotiation enabled.
You have link beat, and everything is working OK.
Your link partner advertised 41e1: 100baseTx-FD 100baseTx 10baseT-FD 10baseT.
End of basic transceiver information.

Listing 3 starts with the mii-tool command to give a short summary of the active interfaces. The results show that the interface has negotiated at 100/Full. mii-diag shows more detailed information and might be more suitable for sending to your network team if you have to escalate the issue. The mii-tool command can be used to fix the speed and duplex, though in practice it's better to have the link auto-negotiated.

Maintaining TDB files

If something is wrong with these databases, your server will have to perform more work, such as unnecessary disk seeks, or fail to cache information from remote services. Fortunately, you can check the TDB files for corruption and fix them if there's a problem. You can delete the temporary TDB files after shutting down Samba, and they'll be re-created on startup. For others, be sure to perform a backup, and then verify the files with tdbbackup -v.

Look at the client

Sometimes, the client may be the cause of slow performance. The client might have duplex problems, it might have malware, or some other problem may be occurring. Using a consistent client for your testing can help you eliminate the server as a potential source of slowness.