Multipath routing with MimoMax and RX1500

Recently I have been working on a project that involved adding redundant paths to a couple of existing comms links, using RuggedCom RX1500 routers at the end of each link, deploying OSPF to manage the routing.  The links themselves are RF, using Mimomax NDL radios which are also routers running OSPF. The paths are equal cost so I was expecting the RX1500 to load balance the paths more or less equally.  This was indeed the case, but when we plugged it all in, our DNP communications to all of the RTUs on those links appeared to get worse, with lots of failed polls occurring.  Upon further inspection it was mostly class 0 polls that failed, and the reason was that the packets were arriving out of order.  It is important to note here that were using DNP over UDP at the time. Below is an approximate diagram of the network (subnets are represented by the circles).

Network diagram

Network diagram

So the question was why are the packets arriving out of order when there are redundant paths available?  It seemed to me that the RX1500 was sending some packets down one path, and some down the other and when they arrived at the other end they didn't always land up in the correct order.  After some google, I discovered that there are two common types of multipath forwarding used in such cases, per-flow and per-packet.

Per-flow forwarding is where the router takes a hash of the source address, destination address, port number, protocol and perhaps some other information (implementations seem to differ quite a bit on what is included here), and the path is always consistent for the same hash.

Per-packet forwarding is basically a round robin or pseudo random process - arriving packets simply take turns at going to the next destination.

Per-packet results in evenly shared links, but causes issues with tcp due to the fast-transmit triggered by the out of order packets, and many other protocols that expect packets to arrive in order.  Per-flow is the one we want to use.

After some back and forth with my local distributor and Siemens it appears that the RX1500 balances UDP traffic per-packet, and TCP traffic per-flow.  This is an interesting decision, and means that using UDP is not practical over ECMP routes unless the protocol is mostly response-reply-response such as SNMP.  For streaming data such as webcams or voip I would not expect good results.

Of course TCP and UDP are not the only two protocols used, for example we use IPSec VPNs, which use the ESP protocol for wrapping up the UDP/L2TP packets.  I asked about these protocols as well and was told that they also use per-packet balancing.  However in testing no degradation of service was noticed at all, so I will test to see if this is the case in due course.

The second part of this equation is that the radios themselves are routers, and don't do any load balancing at all, instead always choosing the next hop with the lowest IP address cases where there are two options with equal cost.  This results in an interesting behavior when sending packets from R1-R3 because on the first link the load is balanced across the two RF links, but the second RF link all the traffic will go over one RF path, because the next hop decision is not made by R2, but by W3 or W4 .  Traffic going the other direction will do the same in reverse.  We decided that for our purposes this wasn't an issue, as the links are for redundancy and not throughput.

So the moral of the story here is to choose your network equipment wisely, especially if it is linux based, as it may not behave like you think it might!

| October 30th, 2016 | Posted in Networking |

Leave a Reply