1. 02 Dec, 2021 4 commits
    • Gleb Smirnoff's avatar
      tcp_hpts: rename input queue to drop queue and trim dead code · f971e791
      Gleb Smirnoff authored
      The HPTS input queue is in reality used only for "delayed drops".
      When a TCP stack decides to drop a connection on the output path
      it can't do that due to locking protocol between main tcp_output()
      and stacks.  So, rack/bbr utilize HPTS to drop the connection in
      a different context.
      
      In the past the queue could also process input packets in context
      of HPTS thread, but now no stack uses this, so remove this
      functionality.
      
      Reviewed by:		rrs
      Differential revision:	https://reviews.freebsd.org/D33025
      f971e791
    • Gleb Smirnoff's avatar
      tcp_hpts: make struct tcp_hpts_entry private to the module. · b0a7c008
      Gleb Smirnoff authored
      Also, make some of the functions also private to the module. Remove
      unused functions discovered after that.
      
      Reviewed by:		rrs
      Differential revision:	https://reviews.freebsd.org/D33024
      b0a7c008
    • Gleb Smirnoff's avatar
      tcp_hpts: provide tcp_in_hpts(). · 50f081ec
      Gleb Smirnoff authored
      It will hide some internal HPTS knowledge from the consumers.
      
      Reviewed by:		rrs
      Differential revision:	https://reviews.freebsd.org/D33023
      50f081ec
    • Gleb Smirnoff's avatar
      SMR protection for inpcbs · de2d4784
      Gleb Smirnoff authored
      With introduction of epoch(9) synchronization to network stack the
      inpcb database became protected by the network epoch together with
      static network data (interfaces, addresses, etc).  However, inpcb
      aren't static in nature, they are created and destroyed all the
      time, which creates some traffic on the epoch(9) garbage collector.
      
      Fairly new feature of uma(9) - Safe Memory Reclamation allows to
      safely free memory in page-sized batches, with virtually zero
      overhead compared to uma_zfree().  However, unlike epoch(9), it
      puts stricter requirement on the access to the protected memory,
      needing the critical(9) section to access it.  Details:
      
      - The database is already build on CK lists, thanks to epoch(9).
      - For write access nothing is changed.
      - For a lookup in the database SMR section is now required.
        Once the desired inpcb is found we need to transition from SMR
        section to r/w lock on the inpcb itself, with a check that inpcb
        isn't yet freed.  This requires some compexity, since SMR section
        itself is a critical(9) section.  The complexity is hidden from
        KPI users in inp_smr_lock().
      - For a inpcb list traversal (a pcblist sysctl, or broadcast
        notification) also a new KPI is provided, that hides internals of
        the database - inp_next(struct inp_iterator *).
      
      Reviewed by:		rrs
      Differential revision:	https://reviews.freebsd.org/D33022
      de2d4784
  2. 08 Jul, 2021 1 commit
  3. 07 Jul, 2021 1 commit
    • Randall Stewart's avatar
      tcp: HPTS performance enhancements · d7955cc0
      Randall Stewart authored
      HPTS drives both rack and bbr, and yet there have been many complaints
      about performance. This bit of work restructures hpts to help reduce CPU
      overhead. It does this by now instead of relying on the timer/callout to
      drive it instead use user return from a system call as well as lro flushes
      to drive hpts. The timer becomes a backstop that dynamically adjusts
      based on how "late" we are.
      
      Reviewed by: tuexen, glebius
      Sponsored by: Netflix Inc.
      Differential Revision: https://reviews.freebsd.org/D31083
      d7955cc0
  4. 01 Sep, 2020 1 commit
  5. 03 Mar, 2020 1 commit
  6. 26 Feb, 2020 1 commit
    • Pawel Biernacki's avatar
      Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) · 7029da5c
      Pawel Biernacki authored
      r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are
      still not MPSAFE (or already are but aren’t properly marked).
      Use it in preparation for a general review of all nodes.
      
      This is non-functional change that adds annotations to SYSCTL_NODE and
      SYSCTL_PROC nodes using one of the soon-to-be-required flags.
      
      Mark all obvious cases as MPSAFE.  All entries that haven't been marked
      as MPSAFE before are by default marked as NEEDGIANT
      
      Approved by:	kib (mentor, blanket)
      Commented by:	kib, gallatin, melifaro
      Differential Revision:	https://reviews.freebsd.org/D23718
      7029da5c
  7. 12 Feb, 2020 1 commit
  8. 07 Nov, 2019 1 commit
  9. 10 Jul, 2019 1 commit
  10. 10 May, 2019 1 commit
    • Andrew Gallatin's avatar
      Bind TCP HPTS (pacer) threads to NUMA domains · 4e255d74
      Andrew Gallatin authored
      Bind the TCP pacer threads to NUMA domains and build per-domain
      pacer-thread lookup tables. These tables allow us to use the
      inpcb's NUMA domain information to match an inpcb with a pacer
      thread on the same domain.
      
      The motivation for this is to keep the TCP connection local to a
      NUMA domain as much as possible.
      
      Thanks to jhb for pre-reviewing an earlier version of the patch.
      
      Reviewed by:	rrs
      Sponsored by:	Netflix
      Differential Revision:	https://reviews.freebsd.org/D20134
      4e255d74
  11. 04 Feb, 2019 1 commit
    • Warner Losh's avatar
      Regularize the Netflix copyright · 52467047
      Warner Losh authored
      Use recent best practices for Copyright form at the top of
      the license:
      1. Remove all the All Rights Reserved clauses on our stuff. Where we
         piggybacked others, use a separate line to make things clear.
      2. Use "Netflix, Inc." everywhere.
      3. Use a single line for the copyright for grep friendliness.
      4. Use date ranges in all places for our stuff.
      
      Approved by: Netflix Legal (who gave me the form), adrian@ (pmc files)
      52467047
  12. 01 Oct, 2018 1 commit
  13. 06 Sep, 2018 1 commit
  14. 10 Aug, 2018 1 commit
  15. 04 Jul, 2018 1 commit
    • Matt Macy's avatar
      epoch(9): allow preemptible epochs to compose · 6573d758
      Matt Macy authored
      - Add tracker argument to preemptible epochs
      - Inline epoch read path in kernel and tied modules
      - Change in_epoch to take an epoch as argument
      - Simplify tfb_tcp_do_segment to not take a ti_locked argument,
        there's no longer any benefit to dropping the pcbinfo lock
        and trying to do so just adds an error prone branchfest to
        these functions
      - Remove cases of same function recursion on the epoch as
        recursing is no longer free.
      - Remove the the TAILQ_ENTRY and epoch_section from struct
        thread as the tracker field is now stack or heap allocated
        as appropriate.
      
      Tested by: pho and Limelight Networks
      Reviewed by: kbowling at llnw dot com
      Sponsored by: Limelight Networks
      Differential Revision: https://reviews.freebsd.org/D16066
      6573d758
  16. 19 Jun, 2018 2 commits
    • Randall Stewart's avatar
      Move the tp set back to where it was before · f923a734
      Randall Stewart authored
      we started playing with the VNET sets. This
      way we have verified the INP settings before
      we go to the trouble of de-referencing it.
      
      Reviewed by:	and suggested by lstewart
      Sponsored by:	Netflix Inc.
      f923a734
    • Matt Macy's avatar
      convert inpcbinfo hash and info rwlocks to epoch + mutex · 9e58ff6f
      Matt Macy authored
      - Convert inpcbinfo info & hash locks to epoch for read and mutex for write
      - Garbage collect code that handled INP_INFO_TRY_RLOCK failures as
        INP_INFO_RLOCK which can no longer fail
      
      When running 64 netperfs sending minimal sized packets on a 2x8x2 reduces
      unhalted core cycles samples in rwlock rlock/runlock in udp_send from 51% to
      3%.
      
      Overall packet throughput rate limited by CPU affinity and NIC driver design
      choices.
      
      On the receiver unhalted core cycles samples in in_pcblookup_hash went from
      13% to to 1.6%
      
      Tested by LLNW and pho@
      
      Reviewed by: jtl
      Sponsored by: Limelight Networks
      Differential Revision: https://reviews.freebsd.org/D15686
      9e58ff6f
  17. 18 Jun, 2018 1 commit
  18. 14 Jun, 2018 1 commit
  19. 12 Jun, 2018 1 commit
  20. 11 Jun, 2018 1 commit
  21. 08 Jun, 2018 1 commit
  22. 09 May, 2018 1 commit
    • Warner Losh's avatar
      Minor style nits · 603bbd06
      Warner Losh authored
      Use full copyright year.
      Remove 'All Rights Reserved' from new file (rights holder OK'd)
      Minor #ifdef motion and #endif tagging
      Remove __FBSDID macro from comments
      
      Sponsored by: Netflix
      OK'd by: rrs@
      603bbd06
  23. 19 Apr, 2018 1 commit