Example: File Integrity Verification with VShell

[jdev]

Introduction
In file transfer scenarious, there are times where both the sender and the receiver desire some evidence or "receipt" that the file being transferred has been received and written to the recipient's file system without any corruption or tampering.

The SFTP protcol is implemented over an SSH2 transport, which ensures data integrity of a file while in transit between client and server.

But what if you desired further validation -- to have some way of providing both the client and the server with another means of verification?

This post describes a solution that uses VShell as a file transfer server configured with a trigger that will:

• automatically create a hash of a received file,
• compare it with a hash of the same file provided by the client, and
• provide email notification if the server-generated hash doesn't match the client-generated hash.

Design Overview
The goal is to provide both the client and the server with a way to verify the integrity of a data file that has been uploaded from the client to the server.
The design decisions outlined below provide one example of how to achieve this goal:

Diving into the details a little more...

1. Before uploading any data file, the client will first generate a hash file containing a hash value of the data file's contents.
   This hash file has the following format:
   

   hash_algorithm (file_name) = file_hash_result

   Conveniently, the sha256sum command line utility available on a stock Unbuntu 16.10 machine provides this format (using the SHA2-256 hashing algorithm) when the -tag command line option is supplied. For example:
   

   $ sha256sum -tag DataFile.dat > DataFile.dat(client_hash).txt
   $ cat DataFile.dat(client_hash).txt
   SHA256 (DataFile.dat) = 179dcc5ace4b7f4b1ffd02c65d33fb01b9ae7053e8e294e3a9a30b7244c1411d

2. After the client has generated the hash file, it will first upload the data file, DataFile.dat.
   This upload is accomplished using the scp utility with publickey authentication (to facilitate automation, the private key file does not have a passphrase).
   
3. Once the data file has been uploaded, the client will upload the hash file, DataFile.dat(client_hash).txt, to the same location on the remote file transfer server.
   
4. After uploading the hash file, the client will wait for the server to generate its own version of the hash file...
   
5. Once the server (VShell) receives both the data file and the hash file, the VShell server will generate a hash file of its own, using the same algorithm indicated in the hash file the client just uploaded: DataFile.dat(server_hash).txt
   
6. After VShell has generated its own hash file, it compares this hash value with the one provided by the client (as read from the client's hash file).
   If the values do not match, an email is sent with notification of the mismatch.
   
7. Once the hash file has been generated on the server's side, the client will download the server-generated hash file, DataFile.dat(server_hash).txt, and compare its contents with the hash it already generated, displaying information to the client-side end user indicating match/mismatch.

Configuration Overview
The configuration in this scenario involves two machines:

• A Windows machine on which VShell has been installed. VShell is configured with a file upload trigger.
  This trigger fires off a command whenever a file upload event completes.
  This trigger "command" launches a powershell script. The powershell script accepts arguments that control specific script behaviors.
  
  
• A stock Ubuntu 16.10 machine where the scp and sha256sum command line utilities are available by default.
  A shell script facilitates the client-side steps of generating the hash file, uploading both the data file and the hash file, downloading the server-generated hash file, and comparing the two hash values.

Windows VShell Configuration
Here are the steps for getting set up with VShell on your Windows machine.

1. Download VShell for Windows. It's a free download, and will provide you with an initial evaluation period.
   If it turns out you like it and want to continue to use it, you simply purchase a license, apply your license data to your existing installation, and you're all set.
2. Install VShell.
   This is a straightforward process that involves you clicking a few [Next] buttons, followed by [Install], then a [Finish].
   Administrator access is required to install VShell.
3. Reboot the Windows machine following your install so that Windows can load the public-key authentication module that will allow publickey authentication.
   If you only want to provide password authentication initially, you can defer the reboot until you need publickey authentication.
4. Configure VShell.
   Configuration of VShell is done through the VShell control panel applet.
   VShell places this in the Windows Control Panel area, but you can also launch it directly by running VShellCP.exe (found in VShell's program files location).

   

   1. Required (for this exercise). Set up a file upload trigger to launch the powershell script that will do the work.
      
      • For 'A' in the far right of the graphic above, the Command value is entered as follows:
        C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
        
      • For 'B' in the graphic above, the Parameters value is as follows:
        -executionpolicy bypass -file "C:\Program Files\VanDyke Software\VShell\ValidateFileReceivedByHashComparison.ps1"
        -file %P -admin_email admin@example.com -client_email client@example.com -client_ip %I -client_username %U
        • VShell dynamically replaces the %P, %I and %U substitutions with the path to the uploaded file, IP address of the connected client, and username the connected client used for authentication.
        • Intead of admin@example.com, put in your own adminstrator email address.
        • Instead of client@example.com, put in the client-organization's email address.
      
      
   2. Optional. Configure Access Control to restrict access to just Login and file transfer functionality.
   3. Optional. Configure file system-level access to lock users down to their own folders.Use %USER% to make this easy on you as an administrator.
   4. Optional. Enable Debug log file messages so that you can see what's going on behind the scenes and have information you can use to troubleshoot should things unexcpectedly fail.

Ubuntu 16.10 Client-side Configuration
Ubuntu 16.10 comes with the scp client, as well as the sha256sum command line utility as part of a default Ubuntu 16.10 installation.
Suggested configuration steps on the client side involve niceties to facility ease of use.

• Use a shell script to do all the client-side work of creating the hash file, uploading the data file, uploading the hash file, downloading the server's hash file, comparing the values, etc.
  An example shell script that does this work can be downloaded here. Here's the example shell script code (upload-file-with-hash):
  

  #!/bin/bash
  
  # Parameters for the SFTP connection. Public-key authentication is
  # used to prevent plain-text passwords from being embedded in this
  # file.
  sftp_host="192.168.232.138"
  sftp_user="user"
  sftp_remote_path="./"
  
  # File to upload is passed in as an argument
  strFileToUpload="$1"
  
  # A hash file path is composed
  strHashFile="$strFileToUpload(client_hash).txt"
  
  # The hash file is generated
  echo "Creating hash file..."
  sha256sum --tag $strFileToUpload > $strHashFile
  
  # upload the File using the scp command
  echo "Uploading '$strFileToUpload'..."
  scp $strFileToUpload $sftp_user@$sftp_host:$sftp_remote_path
  
  err=$?
  if [ "$err" != "0" ]; then
    echo "  Failed to upload '$strFileToUpload' to remote host: $err"
    exit $err
  fi
  
  # upload the hash file
  echo "Uploading hash file..."
  scp $strHashFile $sftp_user@$sftp_host:$sftp_remote_path
  
  err=$?
  if [ "$err" != "0" ]; then
    echo " Failed to upload hash file '$strHashFile' to remote host: $err"
    exit $err
  fi
  
  # Now download the server's hash file that it generated
  # It may take some time for the hash file to exist, which is why
  # we do this in a loop, exiting with an error only if we exhaust
  # the retry_max
  server_hash_file="$strFileToUpload(server_hash).txt"
  
  echo "Downloading server-generated hash file..."
  # Enter a retry loop that will make 10 attempts to download the
  # server-generated hash file.
  for i in {1..10}
  do
    # Give the server a few seconds to generate the hash file before
    # we try to download it... we'll retry if it's still not yet available,
    # but we'd like to increase the probability that the first time will
    # be a success!
    sleep 2
  
    # Now, attempt to download it locally...
    scp $sftp_user@$sftp_host:$sftp_remote_path/$server_hash_file ./
    err=$?
    if [ "$err" != "0" ]; then
      echo "  Download of server hash-file failed: $err (attempt #$i)"
      echo "    Waiting 2 seconds to try again..."
      sleep 2
    else
      echo "  Successfully downloaded server hash-file."
  
      # Now let's run a diff on the two files... a quick way to
      # see if the client and server hashes match:
      diff --strip-trailing-cr $server_hash_file $strHashFile
      diff=$?
      if [ "$diff" == "0" ]; then
        echo "Hashes match. Non-repudiation complete."
        exit 0
      else
        echo "WARNING: Hashes do not match."
        echo "Client hash file:"
        echo "------------------------------------------------------------"
        cat $strHashFile
        echo ""
        echo "Server hash file:"
        echo "------------------------------------------------------------"
        cat $server_hash_file
        echo ""
        exit 255
      fi
      exit 0
    fi
  done
  
  # if we ever get to this point, it indicates that none
  # of the retry attempts were successful in downloading
  # the server-generated hash file. Thus, we must exit
  # with the last error code returned from that attempt:
  exit $err

• Use publickey authentication to facilitate prompt-free (AKA "automated"), connectivity to the VShell server.
  1. If you already have a public/private key pair generated, skip to step 3.
  2. Generate a public/private key pair on the Ubuntu machine using the following command sequence (see green text below) entered at the terminal shell prompt.
     Note: When prompted for a passphrase for your private key, leave it blank and press {Enter}.
     

     $ cd ~/.ssh
     $ ssh-keygen -t ecdsa -b 521
     Generating public/private ecdsa key pair.
     Enter file in which to save the key (/home/user/.ssh/id_ecdsa):
     Enter passphrase (empty for no passphrase):
     Enter same passphrase again:
     Your identification has been saved in /home/user/.ssh/id_ecdsa.
     Your public key has been saved in /home/user/.ssh/id_ecdsa/pub.
     The key fingerprint is:
     SHA256:t/Fk9eR6OGdH6ovFVyn4tRKryw62q80fpLsZRNnzjL9
     $ cat id_ecdsa.pub
     ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAElGn8qCSi7nOLR6hljvqXg+JjmctwEQW1cwAMGZTHikafersFiB6eAkySpjYCPZGaE0VgVD/9LEhEgIS3NkeOfdQD2ru7T7J6+wi0yM+hOrZ4RgCj9x5kbXijibXCLvTVupHwhOm68wHk+XOO5ArtlDIu33DwSjCncJbvQBsd5fvw2Tg== user@linuxtwo

  3. Copy/email the id_ecdsa.pub file to the VShell machine/administrator.
     The VShell administrator will either
     • Manually copy the .pub file into VShell's PublicKey subfolder matching your user account name, or
     • Use the VShell control panel to configure your account there to allow publickey authentication for your user name.
       

Testing and Troubleshooting
If you've got your Windows VShell server machine configured, and your Ubuntu client machine all set up, you're ready to begin testing.

On your Ubuntu machine, run the command to launch your script, passing in as an argument the name of the file you want to hash-then-upload.
For example: ./upload-file-with-hash DataFile.dat

Once you've seen success in action, modify the data file contents, and upload it -- along with the original hash file -- to the VShell server.
Then go take a look at the VShell log folder on the Windows machine and see what's recorded.

I'll put together a video soon so you can see an example of this testing in action.

--Jake