Scalability Blog

Scaling tips, insights, updates, culture, and more from our Server Experts.
 

Debugging NFS File Access on Server and Client Side

In an environment where a lot of servers share same NFS mount, keeping track of which server created or deleted a file is important.  By default, NFS client configuration does not have an option for access logging in Linux distributions.  To mitigate this situation, you can use a utility called rpcdebug.
To see a list of all supported modules and flags:

# rpcdebug -vusage: rpcdebug [-v] [-h] [-m module] [-s flags...|-c flags...]
set or cancel debug flags.
Module     Valid flags

rpc        xprt call debug nfs auth bind sched trans svcsock svcdsp misc cache all

nfs        vfs dircache lookupcache pagecache proc xdr file root callback client mount fscache pnfs pnfs_ld all

nfsd       sock fh export svc proc fileop auth repcache xdr lockd all

nlm        svc client clntlock svclock monitor clntsubs svcsubs hostcache xdr all

In our case we are interested in modules nfsd for the server side, and nfs for client side.

The vfs flag would be more descriptive than proc flag, and using all could potentially overflow your logs and slow down the system on a busy NFS server or client.

To enable access logging on server side:

[root@nfs-server nfs]# rpcdebug -m nfsd -s proc

You can now test logging on server by creating and deleting a file from an NFS client:

[root@nfs-client ~]# cd /mnt/nfs && touch test && rm -rf test

The debug messages will be logged on the NFS server to /var/log/messages by default.  Now that you have enabled the debug logging, you can find when a particular file was deleted, since information on when the file was created could be obtained via stat command, but recovering information on a deleted file could be impossible after a while, especially if there is no way of knowing whether a file was deleted in the first place.

For our particular example, the filename is test :

[root@nfs-server nfs]# grep test /var/log/messages
Nov 17 16:26:24 nfs-server kernel: [8908851.761009] NFSD: nfsd4_open filename test op_stateowner           (null)

Nov 17 16:26:24 nfs-server kernel: [8908851.793629] NFSD: nfsd4_open_confirm on file test

Nov 17 16:26:24 nfs-server kernel: [8908851.821583] NFSD: nfsd4_close on file test

Access logging on NFS client side requires use of nfs module :

[root@nfs-client ~]# rpcdebug -m nfs -s proc
[root@nfs-client ~]# cd /mnt/nfs && touch test && rm -rf test
[root@nfs-client nfs]# grep test /var/log/messages
Nov 17 21:34:51 nfs-client kernel: NFS call  lookup test

Nov 17 21:34:51 nfs-client kernel: NFS call  create test

Nov 17 21:34:53 nfs-client kernel: NFS call  remove test

 

Using vfs flag, you would be searching for instances of unlink or safe_remove in log files.

If you would like to keep your debug messages in a separate file, add the following line to /etc/syslog.conf ( /etc/rsyslog.conf if you are running rsyslog on Fedora / openSUSE / Debian / Ubuntu  / *BSD ):

*.debug                                 /var/log/debug.log

To disable all debugging and stop logging:

On server side:

rpcdebug -m nfsd -c

On client side:

rpcdebug -m nfs -c

As simple as these tools are, they are an important part of auditing in an enterprise environment, where multiple applications could be accessing same network storage mount from different servers.