Snabb 2016.11 “Babaco”

This release introduces a new library for managing NUMA pinning from within Snabb, a new design for more efficient shifting and appending to packets, and re-synchronization functionality for our ESP implementation, meaning its conceptually complete. Babaco also synchronizes the lwAFTR application with Igalia, and Snabb NFV has received some love as well. As usual this release contains many miscellaneous bug fixes and improvements.

Major Changes

• #1046 #1047 Synchronized lwAFTR program with Igalia
• #1038 Introduced lib.numa for managing NUMA pinning from within Snabb
• #1032 Introduced new design for shifts and appends in core.packet: instead of moving the packet data, it now re-positions the pointer
• #1000 Implemented re-synchronization in lib.ipsec.esp

Minor Changes

• #911 Added YANG model for Snabb NFV configuration
• #1051 Fixed bug in checksum method of lib.protocol.ipv4
• #1054 Polished and fixed critical bug in ESP integration for Snabb NFV
• #1039 Fixed a bug in Snabb NFV where redundant reconfiguration occurred
• #1034 Virtio: use packet shifting to tag packets
• #1031 Included various missing sections in the Snabb manual

Contributors

Andy Wingo (85):

• 4540e60: Add missing lwaftr end-to-end-vlan conf file
• c91d3a9: RateLimitedRepeater operates real line rate
• e114ec9: loadtest prints out bitrates with overhead
• 8260923: PodHashMap: Be more like CTable
• 36780f5: Add PodHashMap:iterate
• 7f5cca8: RangeMap: Add :iterate method
• 231c59e: Add BindingTable:iterate_psid_map
• 697c560: Add BindingTable:iterate_br_addresses
• 6d97550: Add BindingTable:iterate_softwires
• 033e746: Add documentation.
• 9278eb0: Add BindingTable:dump
• a051092: Hook up binding table dumping to the lwAFTR
• dd5e5b3: Fix promise bug with more than one argument.
• 2c118cd: Loadtest supports different programs
• 3b1dfb4: Forward-port CHANGELOG.
• 163e611: Disable JIT flush/drop monitor on loadtest; print ingress drops.
• e382b30: Update documentation
• f0d75a3: Add get_mempolicy, set_mempolicy NUMA syscalls
• af8f015: Add beginnings of NUMA module for Snabb
• fc0bb6d: Add cpu_get_numa_node
• 0cb8bd8: Add more NUMA methods
• a3d8fa4: Add choose_numa_node_for_pci_addresses
• 449490d: pci_get_numa_node can accept canonical addresses
• 85f0319: Add numa.check_affinity_for_pci_addresses
• 806746d: Work around a broken `getcpu'
• e062328: Use lib.numa in lwaftr run
• d9390f3: ljsyscall: Fix getcpu()
• fabb961: ljsyscall: Add migrate_pages.
• 50a920b: numa: Use ljsyscall's fixed "getcpu()" implementation
• d802b76: numa: Migrate existing pages when binding to NUMA node.
• 976b5df: lwaftr loadtest uses lib.numa
• 0f2fa19: More performance documentation updates
• b792e57: v2.7 changelog
• e45397a: More helpful jit.flush message
• d87de59: Fix time printout.
• eb4aac4: Print troubleshooting link at every jit.flush.
• 841266d: Change --cpu mentions to taskset/numactl
• 0c89a4d: v2.8 release notes
• c9676f6: virtio-net driver: Use C header files
• e25a6bb: Add some headroom before a packet's data.
• 0650c04: Disable backpressure on intel driver
• d6087e5: virtio-net driver: Use direct descriptors
• da583ac: Fix assumptions that packet == packet.data
• 30ddd81: Indirect access to packet data
• fbec950: Bring length back to beginning of struct packet
• ae121ca: lwaftr is resilient to changes in p.data
• dfe46a2: Refactor packet headroom mechanism
• 8c15b78: Add assertions in packet.lua
• 0e619fe: Adapt virtio to use packet.shift for its headers
• 57177b2: Bump packet headroom to 64 bytes.
• 1b401ba: Default headroom in Lua, not C
• 814b208: v2.9 changelog
• 780e46a: Refactor ingress drop monitor to have configurable actions
• c249604: Enable warning ingress drop monitor on the NFV
• 5540648: snabbnfv traffic: Only re-start engine when configuration changes
• a2a6713: Add v2.10 changelog.
• 938594d: Fix assumptions that packet == packet.data
• 5416f9c: lwaftr is resilient to changes in p.data
• 71679a5: Implement shiftleft/shiftright by moving packet around
• c2097d2: Add minimum packet alignment.
• 998427e: virtio-net driver: Use C header files
• a5541e5: virtio-net driver: Use direct descriptors
• 3e61642: Convert lwaftr V4V6, generator, nh_fwd apps to new config
• c09e62d: Fix name of Join output link.
• 3f8f337: Supply nh_fwd app with minimal configuration.
• 93629e0: Add tests.
• 12f77a3: Fix allocate/new_packet mixup.
• 7bfbba4: Add get_mempolicy, set_mempolicy NUMA syscalls
• 9a0abb2: ljsyscall: Fix getcpu()
• c2f2d3d: ljsyscall: Add migrate_pages.
• 3b5fe60: Add NUMA module for Snabb
• 243480d: Remove stale timers code
• 4b8930d: Revert message change relative to uptream ctable
• 2567ddc: snabbnfv traffic: Only re-start engine when configuration changes
• ade4b3e: Reformat performance-tuning.md. No change.
• b5a2cde: Merge lwAFTR performance notes to performance-tuning.md
• eef390a: Tweaks
• 6c1f585: Refactor ingress drop monitor to have configurable actions
• de93573: Enable warning ingress drop monitor on the NFV
• 3489e4b: Address feedback
• 6bb456f: Address feedback bis
• 21759e2: Add README.numa.md
• e115542: Update README.md for packet.shiftleft et al
• 4b9becf: Fix typos in performance-tuning.md
• 876addc: Fix shm.exists() check in ingress drop monitor

Diego Pino Garcia (136):

• 11ff29f: Update and clean up documentation
• 7ecb418: Monitor ingress packet drops and jit flush if threshold exceeded
• f9d6289: Remove invalid NDP NS test
• 6846a3f: NDP: Match Neighbour Solicitation target address to lwAFTR IPv6 address
• 740f29b: Rename arguments v4-pci and v6-pci to v4 and v6
• cf46993: Update documentation regarding v4 and v6 arguments
• c1758a2: Remove short options for arguments v4 and v6
• 377b121: Add warning message about deprecated arguments '--v4-pci' and '--v6-pci'
• 018b086: Several improvements in virt/lwaftrctl
• 9e86b23: Handle NS requests and retries without timers
• f04e111: Implement next-hop forwarder
• 7ca5e52: Add nh_fwd unit tests
• f5eaada: Dump packet in 'od' format
• 000b99e: Return source ethernet if Target Link is not present
• 08525dd: Assert destination ethernet address in selftest
• abe6584: Add Neighbor Advertisement unit test
• e2bf05c: Clear out Flags in ICMPv4 echo-reply
• 0b5c313: Add method timer.deactivate(name)
• 0cae549: Retry Neighbour Solicitations during time limit
• 5edd4a2: Send NS packet inmediately and retry after every second
• 21d6145: Refactor dump lwAFTR configuration
• 481f2b5: Dump binding table to text file
• 4bbe3a9: Change dump file names format
• 164dc62: Redo lwAFTR's conf dump selftest
• 2f829bd: Use conf parser to read conf file in dump.lua's selftest
• c6788af: Update performance-tunning.md
• a461a48: Reintroduce v4-pci and v6-pci for backwards compatibility
• cc677cc: Implement ARP requests retries
• 223da17: Cleanup of lwaftrctl script
• 4ad3379: Fix IPSec's esp selftest
• 06b3483: Adapt lwAFTR counters to new counters API
• c1ed420: Name parameters in 'snabb_run_and_cmp'
• 5b8895d: Fix test 'from-to-b4 IPv6 packet NOT found in the binding table'
• 85299c2: Print error if couldn't find file
• 7e03587: Add test 'Testing sending non-IPv6 traffic to the IPv6 interface'
• b727704: Add test 'Testing sending non-IPv4 traffic to the IPv4 interface'
• dc381f6: Add on-a-stick mode
• 10331e2: Add lwAFTR monitor program
• 2b35a95: Add V4V6 selftest
• c1471a0: Refactor load_check
• d74c72c: Tag ARP packets as IPv4 packets
• f8250d9: Add on-a-stick support for 'lwAFTR check'
• 8aca7e6: Run end-to-end tests in on-a-stick mode too
• 2959333: Run end-to-end-vlan tests in on-a-stick mode too
• 719b3ab: Rename 'v4v6' app to 'V4V6'
• 4fe25a3: Add support for icmpv4_rate_limiter_n_packets and icmpv4_rate_limiter_n_seconds configuration parameters
• 45a66d4: Refactor transmit_icmpv6_with_rate_limit
• ad87150: Rate-limiting of ICMPv4 packets
• 98519bf: Remove 'ingress_drop_monitor' dead code
• 1119dfb: Import 'lwaftr query' program to lwaftr branch
• a8353a7: Format numbers with lib.comma_value
• f20229a: Filter counters to print
• 30a83b2: Sort counters
• c880ee8: Align counter values
• 6de1d4a: Add option to print out all available counter names
• f915586: Add a flag to allow monitor all packets in on-a-stick mode
• 696c33e: Add lwaftr nexthop program
• f4c48d7: Import ipv6_equals
• a26698d: Fix self.debug comparison
• 4182e16: Add method 'exists' to core/shm
• 83dd337: Print out multiple snabb instance pids
• 32f81df: Move parsing of arguments to parse_args
• 58691a2: Fetch PID by lwAFTR ID
• ec6223f: Remove unused variables
• b69925f: Update lwAFTR query documentation
• 60aaa9c: Fix counters path
• 6e5f416: Assert counters were initialized
• 4132426: Fix initialization of Reassembly/Fragmentation apps
• b023195: Remove empty counter files
• ed3d6a1: Rename V4V6 input and output links
• 6c29222: Import Snabbvmx
• 4d404dc: Initialize counters and reassembly/fragmentation apps
• e1e0de4: Add hairpinning queue
• 29156d0: Add parameter --mirror to Snabbvmx
• 379b33d: Store lwAFTR ID in shared memory
• 5ee7afb: Default MTU value 9500 in snabbvmx
• 4d86ace: Create lwAFTR counters in apps/lwaftr/counters
• 236cffe: Add documentation for SnabbVMX and fix code issues
• de87bce: Fix fetch lwAFTR instance by name
• 0af0f68: Add support of VLAN tagging in on-a-stick mode
• 41f300f: Pass load_on_a_stick arguments as hash
• adf5060: Load two virtual NICs if using two different VLAN tags
• 6b01e0d: Flush ingress-packet-drops to a counter
• b65a362: Monitor ingress-packet-drops in SnabbVMX
• 318116e: Use discard attributes to setup ingress-drop-monitor
• e03efe4: Read ingress-packet-monitor values from SnabbVMX conf file
• dac8774: Add script tcpreplay
• 53ba35f: Add Snabbvmx test
• a2b1fb4: Increase time interval
• 0f2824e: Clean up tcpreplay.lua script
• 482a196: Fix vlan assignment
• 331b40e: Refactor SnabbVMX selftest
• f23359e: Add VLAN tag testing to SnabbVMX selftest
• 8bf708f: Add nexthop test datasets and configuration files
• 9c29680: Add nexthop selftest
• 24e41f7: Remove unused app 'Repeater'
• 07426fe: Append .counter suffix if necessary
• fb2fc13: Import snabbvmx top
• b2969df: Adapt snabbvmx top
• e0ce047: Update README
• 1f62c93: Remove top/util.lua
• 36cd34b: Select snabb instance per id
• b0ebb75: Adjust columns width
• 12a942d: Update README
• bd5456c: Tidy up top.lua
• ea2f572: Remove ipv6_address parameter in snabbvmx config file
• f75c7cb: Add missing file
• 1b2611b: Fix v4-only and v6-only options
• 7c695c5: Define LWAFTR_DEBUG env var
• 6395d10: Centralize packet deallocation in transmit_icmpv4_reply
• eae3759: Separate test-data from core-end-to-end.sh
• 2cd9a92: Document test data array entry
• bd29d04: Call flush_haipin if link is PKT_HAIRPINNED
• 5aec24e: Add test hairpin queue gets flushed
• aa70cd5: Add VLAN test
• 43f4c33: Add program soaktest
• f9f8639: Add soaktest script for CI testing
• 65804a8: Move soaktest to own folder
• a971f6d: Refactored lwaftr check on-a-stick code
• 4d74ca3: Update README
• 05674f7: Create counters file if it doesn't exist
• 96cbe55: Check counters file is a table
• 0a4bb2e: Append path to conf file
• 0b2afc4: Fix nexthop test
• cb527ac: Fix update of ingress-packet-drops counter
• 7fc9533: Allow empty binding table
• e421d19: Use lwAFTR inet MAC address as ARP source
• a6c60ae: Move lwaftr/nexthop to snabbvmx/nexthop
• c211921: Move apps/nh_fwd/nh_fwd.lua to apps/lwaftr/nh_fwd.lua
• aba846c: fix nexthop test
• 21bc5bb: lwAFTR check on-a-stick mode was ignored
• 7c77ac6: Remove skipped test
• 111a6b4: Extract common functions from lwaftr check
• f044d9e: Add SnabbVMX check
• 5518db2: Add SnabbVMX check test case
• 08ad8ae: Move fatal, file_exists, dir_exists and nic_exists to lwutil
• f007d1f: Import missing function fatal
• d100475: Allow to set monitor by id.
• d92abf4: Fix print error message
• 2729c6b: Exit snabbvmx top if selected non-lwaftr instance
• 4f88d34: Fix reporting of incoming ipv6 packets in packetblaster lwaftr mode
• 0fb961a: Calculate effective VLAN
• e0424bd: Use intel10g.ring_buffer_size setter
• b4a383d: Remove tcpreplay script and use packetblaster --no-loop instead
• c0ee520: Remove old benchmarking tools
• b759464: Use ethernet:ntop and ipv6:ntop
• 4c395fc: Define SKIPPED_CODE in nexthop/selftest.sh
• 3a36e95: Reset IPv4 checksum before recalculating it

Katerina Barone-Adesi (36):

• 04b3d8f: Added ARP support to the lwaftr.
• 7cdd3a5: Added documentation of filter performance
• cb79190: Fix ICMP error encapsulation to hosts behind B4s
• 8692f21: Remove phantom IPv4 packets while hairpinning
• 9e525a6: Added random ejection to ctables
• 0db5e19: Hardened IPv6 reassembly
• ee00591: Addressed nits
• e67046b: Normalize the vlan and non-vlan test suite
• 6aa512f: Addressed nits
• 6b522ae: Fixed an erroneous comment (bytes -> bits).
• a25cfd0: Removed unused global
• 1ae5334: Hardened IPv4 reassembly. All tests pass.
• eb31f54: Nits, halved default packet reassembly table size
• 7334ad8: Remove obsolete l2_size approach
• 16102b3: Fragmentation counters, for IPv4 and IPv6
• 1bf66c0: Use lwutil.keys
• 2d25d80: Move the line-oriented counters towards the edges of the app network
• 65fc13b: Move ctable changes to a wrapper
• dbc715a: Document 'check -r'
• e2d0cba: Got rid of packet.clone_to_memory
• 168e27a: Removed unused parameter
• 3461a48: Documented fragment counters and troubleshooting
• 01eb36c: Rename: needsreassembly -> needs-reassembly
• 9fe6b79: Rename: randomevicted -> random-evicted
• 4a0047e: Reverted moving the wire-oriented counters to the edges
• ec0b434: Dashes strike again
• 3149a39: Update counter path in docs
• 3ff3f6d: If there is a br address for a bad softwire, use it
• fceff6a: Make ntohl/htonl unsigned
• 6f5a2dc: Renamed var, removed comment about it, per review
• ccb1222: Added a regression test for issue 453
• af2ed2e: lwutil refactorings
• 5f74716: Make the core lwAFTR drop fragments
• 37bc4f0: Add -D: duration option to check
• 8448ccd: Fix variable name: self->lwaftr
• b847569: Addressed review comments

Marcel Wiget (17):

• 3c3c41b: check and use linux interface via rawSocket
• 96aeaee: add snabbvmx query (to xml)
• de05632: snabbvmx query to xml
• aea673f: fixed path to counters_dir in lwcounter
• a3d0dc4: no more special treatment for lwaftr app required
• a6cd1ec: remove unused function read_lwaftr_counters
• 8a4d9d6: remove reference to unused lwcounter
• d26fa9e: remove unused variables
• 592f0cc: fix ipv4 checksum in cache refresh packet
• f984b76: increment only after all packet sizes are sent
• 1194c25: ok to have no v4/v6 input/output ports in passthru mode
• 91c5bdc: sanitize xml tags and include links pci and engine stats
• 2fbf247: use the correct readme
• bb521e8: use integer for instance id and add name tag for the nic id
• 5fb1ad4: using :match() for readability and check if name exists
• b8953d9: set interface mtu based on lwaftr v4/v6 mtu
• 025fe10: use lwaftr.constants and adjust mtu in case of vlan, cover interface.mtu

Max Rottenkolber (19):

• 5c4d5dc: snabbnfv: initial YANG model for configuration.
• adde897: snabb-nfvconfig.yang: s/gigabytes/gigabits.
• 3909d47: snabb-nfvconfig.yang: Fix port.tunnel.type description.
• 4b275a4: snabb-nfvconfig.yang: Add initial revision.
• 1b08428: lib.ipsec.esp: minor formatting/style.
• f35f4e9: lib.ipsec.esp: refactor and fixes - move all resync logic into resync method - use self.aes_128_gcm.auth_buf for packet regeneration - remember to free temporary packet used in resync - move auditing code into audit method - make auditing configurable as specified in RFC 4303 - only log audits for packets that were actually rejected - update docs and comments
• dcd1a45: doc/genbook.sh: add missing apps to documentation.
• 7cc37b4: documentation: minor formatting fixes.
• 1bdd98d: README.md: link to documentation.
• 16fba64: snabbnfv: do not include gbps unit in QoS config options.
• 9c4a4d1: Rename snabb-nfvconfig.yang to snabb-nfvconfig-v1.yang (version model).
• 62d5a6f: apps.ipsec.esp: do not keymat for both streams, accepts two key/salt pairs instead. lib.ipsec: get rid of confusing “keymat” term.
• 8e1461b: lib.ipsec.esp: automatically pad window_size to implementation req.
• d1012d4: apps.ipsec.esp: refuse to operate with transmit_salt == receive_salt. See https://tools.ietf.org/html/rfc4106#section-10
• 7598462: …ipsec: fix documentation typos.
• baabc08: ipsec documentation: use $mdroot, include AES128gcm app docs.
• 9772042: snabb-nfvconfig-v1.yang: add crypto node.
• b1b60d2: neutron2snabb: update test fixtures (ignoring the sillyness for now).
• 1ada400: snabbnfv: support deprecated rx_police_gbps and tx_police_gbps.

Nicola Larosa (30):

• 4dd9553: Cleanup of the lwaftr app code
• 2902ff5: Several changes reverted after review
• dbf5a83: Add loading pf filter definitions from file (#317)
• 34c401b: Add ingress, egress and hairpin counters, with preliminary testing support
• 0de9d59: Add documentation for counters, including diagrams and descriptions
• 6dcac11: Updated text and images
• a1abc8f: Add drop counters and update all end-to-end tests to check them too
• ce9a564: Update all end-to-end tests, vlan ones included, so that they check the counters' values too
• c1d09a5: Moved counters' declaration to a single table, used dashes in all names, no underscores anymore
• 50a0330: Fix test data filenames, start adding drop counters' tests
• 8b5260f: Add drop counters' data to the tests, complete counters' renaming
• 7d39e1d: Implement drop-all counters
• a78d0c9: Remove TODOs for drop-all counters
• 1c90f57: Shorten long lines
• aafa4d9: Remove obsolete tests
• fd0bfe1: Count hairpinned packets in out-ipv4 too
• f64f07e: Do not count internally generated ICMPv4 packets in out-ipv4
• 39cc6e0: Extract the lookup queue from the binding table
• f78bd20: Add drop counters, with tests
• fec3c27: Reenable disabled test
• 92f8886: Add missing drop policy code section, fix one test and add two new ones
• bbad242: Add docs for the counters
• ce4edd1: Add a 'snabb lwaftr query' subcommand
• 92d6f52: Remove conflicts from the counters doc. (#404)
• cda6ca8: Add Hydra mode and filename support to bench CSV output (#468)
• 69825fe: Update the 'encapsulate queue to b4' diagram (#474)
• 7277bec: Make the run, run_nowh and transient lwAftr subcommands output CSV data to a file (#472)
• 3e4fef1: Add docs for packet reassembly config settings (#478)
• 0994e88: Remove useless column from benchmark CSV output in Hydra mode (#498)
• fe28653: Misc. cleanup of the program/lwaftr/ subtree (#501)

Timo Buhrmester (5):

• ef2b402: lib.ipsec.esp: Anti-replay - resynchronization (RFC 4303 App. A3), more sophisticated approach without duplicating packets/copying data in the normal (non-resync) code paths
• a6b7714: Eliminate a variable
• b0ac831: lib.ipsec.esp: Get rid of awkwardly named dual-use variable, slightly improve clarity
• 38ac409: lib.ipsec.esp: Add another test to verify we do not resync with very old, replayed packets
• f140d1c: lib.ipsec.esp: Make the resync threshold and depth configurable, add preliminary defaults; document it.