• Hans Petter Selasky's avatar
    Add optimizing LRO wrapper: · e936121d
    Hans Petter Selasky authored
    - Add optimizing LRO wrapper which pre-sorts all incoming packets
      according to the hash type and flowid. This prevents exhaustion of
      the LRO entries due to too many connections at the same time.
      Testing using a larger number of higher bandwidth TCP connections
      showed that the incoming ACK packet aggregation rate increased from
      ~1.3:1 to almost 3:1. Another test showed that for a number of TCP
      connections greater than 16 per hardware receive ring, where 8 TCP
      connections was the LRO active entry limit, there was a significant
      improvement in throughput due to being able to fully aggregate more
      than 8 TCP stream. For very few very high bandwidth TCP streams, the
      optimizing LRO wrapper will add CPU usage instead of reducing CPU
      usage. This is expected. Network drivers which want to use the
      optimizing LRO wrapper needs to call "tcp_lro_queue_mbuf()" instead
      of "tcp_lro_rx()" and "tcp_lro_flush_all()" instead of
      "tcp_lro_flush()". Further the LRO control structure must be
      initialized using "tcp_lro_init_args()" passing a non-zero number
      into the "lro_mbufs" argument.
    - Make LRO statistics 64-bit. Previously 32-bit integers were used for
      statistics which can be prone to wrap-around. Fix this while at it
      and update all SYSCTL's which expose LRO statistics.
    - Ensure all data is freed when destroying a LRO control structures,
      especially leftover LRO entries.
    - Reduce number of memory allocations needed when setting up a LRO
      control structure by precomputing the total amount of memory needed.
    - Add own memory allocation counter for LRO.
    - Bump the FreeBSD version to force recompilation of all KLDs due to
      change of the LRO control structure size.
    Sponsored by:	Mellanox Technologies
    Reviewed by:	gallatin, sbruno, rrs, gnn, transport
    Tested by:	Netflix
    Differential Revision:	https://reviews.freebsd.org/D4914