NOTE: archive.org link. Relevant info:

NFS behind a PF firewall

In the setup, there are 2 Linux-based NAS devices that export a 1Tb share over
NFS. Both NAS are in sync so, if one fails, we can mount the other one and
users would be able to keep working while the first NAS is replaced.

Both NAS devices are in a Gigabit LAN, connected with one of the network
interfaces on a FreeBSD server (subnet 192.168.10.0/24).

On the other side of the server, there was another LAN, where workstations are
connected to the server (subnet 192.168.1.0/24).

The plan was to mount the /nfs/shares share in the FreeBSD server and then
export it again from the server, allowing the workstations to mount it. It
didn't work. After some reading I found out that NFS does not like to re-export
NFS shares, that is, if you mount an NFS share from server A on server B and
then you try to export that share from the mount point in server B to server C
(for example) you will get all kinds of nasty errors.

It was time for a change in the plan. I didn't want to give full-access to the
NAS devices and I didn't want the workstations to mount the share directly from
the NAS devices either, because if one fails, I would have to modify the mounts
in every workstation so they use the backup NAS device.

Perhaps I could use PF in the FreeBSD server to redirect NFS traffic from the
workstations directly to the NAS devices...

Well, it took me some time to think about how to do it (and some help from viq
at #pf in freenode). The best approach could be some kind of binat that could
map the NFS requests for a given ip address to the ip address of the selected
NAS device. Anyway, you can't map a whole network (or any to use a wildcard),
so binat would not work. I had to think about a different way to do it.

Let's begin modifying the setup a little bit:

NFS behind a PF firewall (final setup)

As you can see in the picture, my idea was to add an alias to the nic connected
to the 192.168.1.0/24 subnet, then I would be able to use PF to redirect all
incoming traffic for that alias (192.168.1.100) to a given NAS device. As I've
a local DNS server in the network I could add an A entry for nfs.mydomain.com
that points to 192.168.1.100, this way I would be able to use that internal
subdomain instead the ip address to access the NFS shares. Nice, the new plan
was finished, let's start it! (I'll omit the DNS Zone modification, as it is
out of the scope of this post)

First, I added the alias definition to /etc/rc.conf:

ifconfig_bge0_alias0="inet 192.168.1.100 netmask 255.255.255.0"

This will create the alias properly after a reboot. Then I created the alias
manually:

ifconfig bge0 alias 192.168.1.100

Once I've created the alias, I added PF support in /etc/rc.conf:

pf_enable="YES"
pflog_enable="YES"
gateway_enable="YES"

This will load the needed stuff on boot (kernel modules, sysctl knobs, etc).
This will work if you've the GENERIC kernel, if not, you will have to check if
your custom kernel has pf support enabled within the kernel or as a KLD.

Then I created the /etc/pf.conf file. This is a stripped version of it (It
shows only the needed stuff for the NFS redirection to work):

lan_if="bge0" # NIC connected to the Intranet LAN
nas_if="bge1" # NIC connected to the NAS-storage LAN

nfsalias="192.168.1.100" # Local alias to manage requests for NFS storage
mainnas="192.168.10.2" # Main storage NAS
backupnas="192.168.10.3" # Backup storage NAS

rdr on $lan_if proto { tcp, udp } from $lan_if:network to $nfsalias port 111:65535 -> $mainnas

pass in log quick on $lan_if proto { tcp, udp } from $lan_if:network to $mainnas keep state

pass in all
pass out all

What I do with this pf.conf configuration file is set a redirection so all tcp
and udp traffic from the workstations LAN to the alias ip on any port between
111 and 65535 will get redirected to the main NAS device.

The reason for such redirect is that NFS uses random ports for connections (in
a similar way to what FTP does) so it is easier to open a full-range of ports.
The first port is 111 as it is the first port needed by NFS (sunrpc/rpcbind)
and, this way, I block requests for the NAS device web interface (port 80) and
SSH (port 22) and users from the workstations LAN will not be able to access
the management interfaces of the NAS device.

You should have noticed that the following line is not needed at all:

pass in log quick on $lan_if proto { tcp, udp } from $lan_if:network to $mainnas keep state

It is not (because I do a pass all later) but it allows me to log only NFS
traffic (useful for debugging).

Ok, once I had everything in place, I started PF:

/etc/rc.d/pf start

(This will load the kernel module if needed, set the proper sysctl options and
will enable pf too)

If my plan was right (and it was ;D) I should be able to access the NFS shares
of the main NAS device from any workstation, so I did some checks using
showmount:

$ showmount -d nfs.mydomain.com
Directories on nfs.mydomain.com:
*,192.168.1.0/24
*,192.168.10.1
/mnt/soho_storage/samba/shares/shares

It worked!. that soho_storage path you see there is the full path of the shares
share within the NAS device filesystem. The Iomega NAS stores all the data
within a directory called samba/shares and then it creates symlinks of the data
within the /nfs directory.

Ok, as showmount worked, it was time for the final test, to mount the share. In
FreeBSD workstations I modified /etc/fstab, adding:

nfs.mydomain.com:/nfs/shares /mnt/nfsstuff  nfs     rw,tcp            0       0

In the Linux workstations I added a line like:

nfs.mydomain.com:/nfs/shares /mnt/nfsstuff nfs rw,tcp,rsize=32768,wsize=32768,hard,intr,timeo=14,bg       0       0

Of course /mnt/nfsstuff must exist. With the proper fstab lines in place, I
just tried to mount the share:

# mount /mnt/nfsstuff

Which worked as expected:

$ df -h
Filesystem               Size    Used   Avail Capacity  Mounted on

nfs.mydomain.com:/nfs/shares    929G    269G    660G    29%    /mnt/nfsstuff

Conclusion:

It was really disgusting to find out that NFS does not support re-export of
shares. This is the first time in many years I find something I don't like
about NFS. Anyway, I found a great solution, perhaps even better than
re-exporting it. With this setup, if the main NAS device fails, I only have to
change one PF rule and all the requests for nfs.mydomain.com will go to the
backup NAS. If needed, I could even split NFS traffic, sending some
workstations to the main NAS and some other to the backup NAS or I could block
NFS traffic from certain ip addresses, adding some security to the whole thing.
IMHO this is a great solution.