I run a pair of GNU/Linux VPS's, one running Plex as a client and the other as a storage server for media. They both run on "the cloud" hosted by different providers. I've been noticing lately that transfer rates between them over SSHFS have been lagging, causing buffering and drop issues for some of my media streams. I tested directly between the hosts using Iperf3 and while not spectacular between two large hosting providers, it's fine for streaming SDTV content.

me@plex:~$ iperf3 -c 10.10.10.100 -p 5201 --bytes 1000000000 
Connecting to host 10.10.10.100 , port 5201
[  4] local 10.10.10.99 port 56634 connected to 10.10.10.100 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  57.0 MBytes   478 Mbits/sec  165   3.94 MBytes       
[  4]   1.00-2.00   sec  45.0 MBytes   378 Mbits/sec   17   1.99 MBytes       
[  4]   2.00-3.00   sec  43.8 MBytes   367 Mbits/sec    0   2.02 MBytes       
[  4]   3.00-4.00   sec  42.5 MBytes   356 Mbits/sec    0   2.15 MBytes       
[  4]   4.00-5.00   sec  48.8 MBytes   409 Mbits/sec    0   2.41 MBytes       
[  4]   5.00-6.00   sec  52.5 MBytes   440 Mbits/sec    0   2.76 MBytes       
[  4]   6.00-7.00   sec  60.0 MBytes   503 Mbits/sec    0   3.02 MByte

I then moved on and tested the mount point. I mount the media storage filesystem from plex client to storage server using SSHFS. So I performed a crude check on the SFTP mount point using cat and a standard utility called pipe viewer (pv).

me@plex:/mnt/media/tv/show/S01$ 
cat show.mkv | pv -rb --progress >/dev/null
4.88MiB [ 300KiB/s]          

indeed, file read hovers around ~300KB/s (with multiple stalls) were just bad. Now there are a slew of tweaks that can be done, via the ssh protocol lighter MACs, Ciphers, via SSHFS buffer cache, VFS tweaks etc.. and the network, buffer, window sizes etc.. I decided to sidestep all the tweaks and try a different SFTP implementation entirely. I remembered a hacker news post not too long ago and decided download a binary release of SFTPGo.

SFTPGo is a standalone SFTP server written in Go, that does not rely on the system authentication mechanisms and can utilize multiple SQL authentication back-ends such as SQLite, Mysql etc.. but I wanted a quick way to use public/private key authentication and good thing it supports this via its "portable" option.

First we'll create an ssh public/private key pair on the Plex client VPS

me@plex:~ ssh-keygen -t ecdsa -f ~/.ssh/id_ecdsa_sftpgo

We then add a section in ~/.ssh/config on the client that describes the storage server

Host storage_server
   Hostname 10.10.10.100
   Port 4444
   User sftpdude
   IdentityFile ~/.ssh/sftpgo_ecdsa
   Ciphers aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
   MACs umac-128-etm@openssh.com,umac-128@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha1,hmac-sha2-512-etm@openssh.com,hmac-sha2-512

Now we'll move to the storage server and create a dedicated directory and download the latest release of SFTPGo and install it manually

me@storage:/ sudo mkdir /opt/sftpgo
me@storage:/ sudo chown me:me !$
me@storage:/ cd !$
me@storage:/opt/sftpgo wget -O - https://github.com/drakkan/sftpgo/releases/download/0.9.6/sftpgo_0.9.6_linux_x86_64.tar.xz | tar Jxf -

Next create a wrapper script on the storage server and copy in the client's public key that was created earlier in ~/.ssh/id_ecdsa_sftpgo.pub

me@storage:/opt/sftpgo/ cat sftpgo_start.sh
#!/bin/sh
sftpgo=/opt/sftpgo/sftpgo
port=4444
pubkey='ecdsa-sha2-nistp256 AAAAE2V....3YPCHWceWD2QcQFG='
dir=/mnt/media

$sftpgo portable --username sftpdude --public-key "$pubkey" --sftpd-port $port --directory $dir --permissions '*'

We run the newly created script on the storage server

me@storage:/opt/sftpgo/ /opt/sftpgo/ssftpgo_start.sh

We connect to the storage_server via sshfs

me@plex:~ /usr/bin/sshfs -f storage_server:/tv /mnt/media/tv2 -o ServerAliveInterval=15 -o rw,reconnect,idmap=user

and retry via the crude cat | pv

me@plex:/mnt/media/tv2/show/S01$ 
cat show.mkv | pv -rb --progress >/dev/null
23.8MiB [2.84MiB/s]          

and lo and behold .. much faster transfer rates without tweaking and just switching implementations. Of course it's far slower than a raw byte transfers (iperf3) but we are utilizing FUSE which in itself is a bottleneck traversing user-space/kernel boundaries, but it's good enough for this use case.

Securing things

OpenSSH has decades of security under its belt and comes from the iron clad OpenBSD folks and we want to protect the system as much as possible from any weird side effects, bugs, and/or exploits typical of long running programs. While it is written in a garbage collected language (Go) which eliminates many classes of memory induced bugs, it is better to be safe than sorry. So if you're running a popular Linux distro today you most likely have systemd installed by default. Systemd is an initd/framework for starting/managing the system and it's services. It is what we'll use to to monitor, manage and secure the SFTPGo service. First thing we do is think about that what the service needs to do to perform its job; it starts up, creates a pub/priv key file if /opt/sftpgo/id_ecdsa doesn't exist, listens on a specified port, and reads/writes to /mnt/media/. So let's create a systemd service file that incorporates this behavior.

So here's the systemd service file

me@storage:/etc/systemd/system cat sftpgo@me.service
[Unit]
Description=SFTP Go
AssertDirectoryNotEmpty=/mnt/media/

[Service]
User=%i
ExecStart=/opt/sftpgo/sftpgo_start.sh
WorkingDirectory=/opt/sftpgo
ExecReload=/bin/kill -s HUP $MAINPID
KillMode=mixed
Type=simple
Restart=on-failure
RestartSec=10
StartLimitInterval=20s
StartLimitBurst=3
NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
DevicePolicy=closed
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/opt/sftpgo /mnt/media/tv 
RestrictAddressFamilies=AF_INET
#requires systemd 235+ and kernel 4.11+
IPAccounting=yes
IPAddressDeny=any
IPAddressAllow=10.10.10.0/24

[Install]
WantedBy=multi-user.target
Alias=sg.service

Basically we prevented read/write access anywhere except /opt/sftpgo and /mnt/media/tv. We restricted access to /home or any other filesystems via the ProtectHome, ProtectSystem stanzas. We also restricted socket creation to the AF_INET protocols while only accepting connections from Plex client ip address range. Systemd provides a host of facilities one can use to secure their services without that much effort in tandem with the kernel 4.x+ namespacing, eBPF, cgroup controls. Most of systemd's lock down parameters are explained here, there's even more we can do such as system call, namespace filtering, but we'll need to profile the binary some more before implementing that.

Now we enable and start our new service

me@storage:~ sudo systemctl enable sftpgo@me
me@storage:~ sudo systemctl start sftpgo@me
me@storage:~ sudo systemctl status sftpgo@me 

And check some transfer stats after transferring some files (since we enabled IPAccounting=yes)

me@storage:~ sudo systemctl show sftpgo@me -p IPIngressBytes -p IPEgressBytes 
IPIngressBytes=20382896
IPEgressBytes=677305852 

99% of the time I have no need to replace the standard SFTP server implementation on my Linux/*BSD hosts, but it was worth it this time for some flexibility and the bump in speed. Though the OpenSSH SFTP implementation is actually faster than SFTPGo, so clearly something is wrong between these hosts that I need to troubleshoot down the road. But, just the fact that I can decouple the standard GNU/Linux user authentication mechanism from SFTP and name the user whatever I want as long as the script has the correct read/write permissions is a plus for me.