In The Works

File transfer over multiple network paths

In data centres as well as enterprise networks, it is quite common to have multiple network paths between any two hosts. These paths are added into the network design for redundancy and for load-balancing, and could be at Layer 2 (e.g. with aggregate links, aka etherchannels) or at Layer 3 (more than one equal-cost next-hop for a given destination).

When transferring a file between two hosts using scp or sftp (or pretty much any file transfer protocol) in such a network, the protocol packets will only take one of the available paths.

For example, consider a single scp session between Hosts A and B in the figure below:

mtpftp

All packets for this scp session will only use one of the eight available links in the etherchannel. This is because network switches and routers will never distribute packets that belong to a single flow (a flow being a combination of source IP address + source TCP port + destination IP address + destination TCP port) across multiple paths.

One way to get around this is to split the file to be transferred into, say, four pieces and then start four separate scp/sftp sessions to transfer the pieces. Then reassemble the pieces at the receiver. However this only increases the chances that all paths will be used – this is not guaranteed because it depends upon the exact values of the TCP ephemeral ports chosen by the client.

It is certainly possible to write software and manually tune the system to use multiple paths in a predictable way.

 

Enable real-time cleartext packet capture of TLS sessions (using modified TLS libraries)

OpenSSL and other TLS libraries provide a callback function facility to log the session key used for a TLS session. The application can invoke this API (for OpenSSL):

typedef void (*SSL_CTX_keylog_cb_func)(const SSL *ssl, const char *line);

void SSL_CTX_set_keylog_callback(SSL_CTX *ctx, SSL_CTX_keylog_cb_func cb);

to setup a callback function which is then invoked with the TLS session key during session establishment. line is the key material.

This mechanism is used for debug purposes using the well-known SSLKEYLOGFILE mechanism, where TLS programs (like browsers) can be asked to save session keys to a file which can then be imported into Wireshark.

The SSLKEYLOGFILE method is not real-time, however, because the file containing the session keys has to be manually imported into a Wireshark session which already has the packet capture file loaded.

In environments where it is feasible to run modified (patched) versions of the TLS libraries (like development environments, sandboxes, co-operative users) it is possible to enable real-time packet capture of TLS sessions using a scheme that sends session keys to directly to the sniffer. Something like this:

openssl-idea

 

Test Framework for verifying network configuration changes

  • Look at an ACL and derive test traffic patterns that will exercise every ACE in the ACL. Send traffic and monitor output to verify correctness.
  • Ditto for firewall rules
  • Verify availability of services over all paths between servers and clients
  • Verify QoS policy correctness by sending traffic and monitoring egress interfaces for marked/shaped/policed packets
  • Integrate with automation/ci-cd (“netdevops”) frameworks