Using the rpcinfo Command
This example gathers information on the RPC services that are running on a server. The text that is generated by the command is filtered by the sort command to make it more readable. Several lines that list RPC services have been deleted from the example.
% rpcinfo -s bee |sort -n
program version(s) netid(s) service owner
100000 2,3,4 udp6,tcp6,udp,tcp,ticlts,ticotsord,ticots rpcbind superuser
100001 4,3,2 ticlts,udp,udp6 rstatd superuser
100002 3,2 ticots,ticotsord,tcp,tcp6,ticlts,udp,udp6 rusersd superuser
100003 3,2 tcp,udp,tcp6,udp6 nfs superuser
100005 3,2,1 ticots,ticotsord,tcp,tcp6,ticlts,udp,udp6 mountd superuser
100007 1,2,3 ticots,ticotsord,ticlts,tcp,udp,tcp6,udp6 ypbind superuser
100008 1 ticlts,udp,udp6 walld superuser
100011 1 ticlts,udp,udp6 rquotad superuser
100012 1 ticlts,udp,udp6 sprayd superuser
100021 4,3,2,1 tcp,udp,tcp6,udp6 nlockmgr superuser
100024 1 ticots,ticotsord,ticlts,tcp,udp,tcp6,udp6 status superuser
100029 3,2,1 ticots,ticotsord,ticlts keyserv superuser
100068 5 tcp,udp cmsd superuser
100083 1 tcp,tcp6 ttdbserverd superuser
100099 3 ticotsord autofs superuser
100133 1 ticots,ticotsord,ticlts,tcp,udp,tcp6,udp6 - superuser
100134 1 ticotsord tokenring superuser
100155 1 ticots,ticotsord,tcp,tcp6 smserverd superuser
100221 1 tcp,tcp6 - superuser
100227 3,2 tcp,udp,tcp6,udp6 nfs_acl superuser
100229 1 tcp,tcp6 metad superuser
100230 1 tcp,tcp6 metamhd superuser
100231 1 ticots,ticotsord,ticlts - superuser
100232 10 udp,udp6 sadmind superuser
100234 1 ticotsord gssd superuser
100235 1 tcp,tcp6 - superuser
100242 1 tcp,tcp6 metamedd superuser
100249 1 ticots,ticotsord,ticlts,tcp,udp,tcp6,udp6 - superuser
300326 4 tcp,tcp6 - superuser
300598 1 ticots,ticotsord,ticlts,tcp,udp,tcp6,udp6 - superuser
390113 1 tcp - unknown
805306368 1 ticots,ticotsord,ticlts,tcp,udp,tcp6,udp6 - superuser
1289637086 1,5 tcp - 26069
|
This example shows how to gather information about a particular RPC service by selecting a particular transport on a server.
% rpcinfo -t bee mountd program 100005 version 1 ready and waiting program 100005 version 2 ready and waiting program 100005 version 3 ready and waiting % rpcinfo -u bee nfs program 100003 version 2 ready and waiting program 100003 version 3 ready and waiting |
The first example checks the mountd service that is running over TCP. The second example checks the NFS service that is running over UDP.
snoop
This command is often used to watch for packets on the network. The snoop command must be run as root. The use of this command is a good way to ensure that the network hardware is functioning on both the client and the server. Many options are available (see the snoop(1M) man page). A shortened synopsis of the command follows:
snoop [ -d device ] [ -o filename ] [ host hostname ]
-d device | Specifies the local network interface |
-o filename | Stores all the captured packets into the named file |
hostname | Displays packets going to and from a specific host only |
The -d device option is useful on those servers that have multiple network interfaces. You can use many other expressions besides setting the host. A combination of command expressions with grep can often generate data that is specific enough to be useful.
When troubleshooting, make sure that packets are going to and from the proper host. Also, look for error messages. Saving the packets to a file can simplify the review of the data.
truss
You can use this command to see if a process is hung. The truss command must be run by the owner of the process or by root. You can use many options with this command (see the truss(1) man page). A shortened syntax of the command follows:
truss [ -t syscall ] -p pid
-t syscall | Selects system calls to trace |
-p pid | Indicates the PID of the process to be traced |
The syscall can be a comma-separated list of system calls to be traced. Also, starting syscall with a ! selects to exclude the listed system calls from the trace.
This example shows that the process is waiting for another connection request from a new client.
# /usr/bin/truss -p 243 poll(0x00024D50, 2, -1) (sleeping...) |
This is a normal response. If the response does not change after a new connection request has been made, the process could be hung. Follow the instructions in "How to Restart NFS Services" to fix the hung program. Review the instructions in "NFS Troubleshooting Procedures" to fully verify that your problem is a hung program.
How the NFS Service Works
The following sections describe some of the complex functions of the NFS software.
Version 2 and Version 3 Negotiation
Because NFS servers might be supporting clients that are not using the NFS version 3 software, part of the initiation procedure includes negotiation of the protocol level. If both the client and the server can support version 3, that version is used. If either the client or the server can only support version 2, that version is selected.
You can override the values that are determined by the negotiation by using the -vers option with the mount command (see the mount_nfs(1M) man page). Under most circumstances, you should not have to specify the version level, as the best level is selected by default.
UDP and TCP Negotiation
During initiation, the transport protocol is also negotiated. By default, the first connection-oriented transport that is supported on both the client and the server is selected. If this selection does not succeed, the first available connectionless transport protocol is used. The transport protocols that are supported on a system are listed in /etc/netconfig. TCP is the connection-oriented transport protocol that is supported by the release. UDP is the connectionless transport protocol.
When both the NFS protocol version and the transport protocol are determined by negotiation, the NFS protocol version is given precedence over the transport protocol. The NFS version 3 protocol that uses UDP is given higher precedence than the NFS version 2 protocol using TCP. You can manually select both the NFS protocol version and the transport protocol with the mount command (see the mount_nfs(1M) man page). Under most conditions, allow the negotiation to select the best options.
File Transfer Size Negotiation
The file transfer size establishes the size of the buffers that are used when transferring data between the client and the server. In general, larger transfer sizes are better. The NFS version 3 protocol has an unlimited transfer size, but starting with the Solaris 2.6 release, the software bids a default buffer size of 32 Kbytes. The client can bid a smaller transfer size at mount time if needed, but under most conditions this bid is not necessary.
The transfer size is not negotiated with systems that use the NFS version 2 protocol. Under this condition, the maximum transfer size is set to 8 Kbytes.
You can use the -rsize and -wsize options to set the transfer size manually with the mount command. You might need to reduce the transfer size for some PC clients. Also, you can increase the transfer size if the NFS server is configured to use larger transfer sizes.
How File Systems Are Mounted
When a client needs to mount a file system from a server, it must obtain a file handle from the server that corresponds to the file system. This process requires that several transactions occur between the client and the server. In this example, the client is attempting to mount /home/terry from the server. A snoop trace for this transaction follows.
client -> server PORTMAP C GETPORT prog=100005 (MOUNT) vers=3 proto=UDP server -> client PORTMAP R GETPORT port=33492 client -> server MOUNT3 C Null server -> client MOUNT3 R Null client -> server MOUNT3 C Mount /export/home9/terry server -> client MOUNT3 R Mount OK FH=9000 Auth=unix client -> server PORTMAP C GETPORT prog=100003 (NFS) vers=3 proto=TCP server -> client PORTMAP R GETPORT port=2049 client -> server NFS C NULL3 server -> client NFS R NULL3 client -> server NFS C FSINFO3 FH=9000 server -> client NFS R FSINFO3 OK client -> server NFS C GETATTR3 FH=9000 server -> client NFS R GETATTR3 OK |
In this trace, the client first requests the mount port number from the portmap service on the NFS server. After the client received the mount port number (33492), that number is used to ping the service on the server. After the client has determined that a service is running on that port number, the client then makes a mount request. When the server responds to this request, it includes the file handle for the file system (9000) that is being mounted. The client then sends a request for the NFS port number. When the client receives the number from the server, it pings the NFS service (nfsd), and requests NFS information about the file system that uses the file handle.
In the following trace, the client is mounting the file system with the public option.
client -> server NFS C LOOKUP3 FH=0000 /export/home9/terry server -> client NFS R LOOKUP3 OK FH=9000 client -> server NFS C FSINFO3 FH=9000 server -> client NFS R FSINFO3 OK client -> server NFS C GETATTR3 FH=9000 server -> client NFS R GETATTR3 OK |
By using the default public file handle (which is 0000), all of the transactions to obtain information from the portmap service and to determine the NFS port number are skipped.