Listen on 0.0.0.0 and :: vs individual IPs
I'm currently playing around with different listening strategies.
Listen only on 0.0.0.0 (IPv4) and :: (IPv6): Advantage is that local IP change should not matter and packets still arrive. Disadvantage is that local peer discovery may be not working if there are multiple adapters. Windows is (understandably) not good at picking the right interface when sending out packets from a socket bound to IP ::.- Listen on each IP on each network adapter. Highest chance of receiving/sending because it has unique bindings on each network adapter which means Windows does not have to choose one when sending. Big downside is that local IP change will break sockets. (which would include changing Wifi networks)
Both: Listening on 0.0.0.0 and :: as well as on a separate port: We would end up with a lot of sockets/ports. It can get confusing when peers connect on different ports (especially one used only for multicast).
A good start could be checking what Bitcoin, Ethereum, Cosmos, & friends are doing. [see Appendix 1]
One possible solution for fixing the downside of option 2 could be running a background routine to check if the list of adapters (and reported IPs) change (every minute?) and if they do, send termination signals to ones that were removed and spawn new ones. That background routine could run more often in case specific errors are returned when sending/receiving that indicate that the adapter changed.
Update: After multiple tests it really looks like option 1 is off the table, since delivery can be quite bad in multi-adapter setups. In my test I have on one adapter public a IPv6, public temporary IPv6, link-local IPv6, private IPv4, then another adapter with a VPN with multiple IPv6, private IPv4, then another adapter for VirtualBox with an inactive link-local IPv6, then another adapter for Wifi, etc.
We'll go with option 2 - binding on each IP, and developing an algorithm to detect network adapter and IP change.
Because of that setup, however, another problem appears: Subscribing for IPv6 Multicast on multiple IPs of the same adapter might result in duplicate incoming UDP messages. I will do testing to confirm the existence of this problem.
Update 2: Duplicate incoming IPv6 Multicast messages is confirmed. The source IP is the same, but the destination IP is different based on the multicast subscription socket. Possible solutions:
- Low-level: Store hash of incoming multicast messages for X seconds (max of N hashes to prevent flood)
- Bad solution: Subscribe only to first IP in list, but that would mean less receival of multicast messages and "first" IP might change.
- High-level: Multicast/broadcast messages are only used for initial discovery. If they have their own command number, it can be easy ignoring duplicates out based on the fact that the sending peer is already in the peer list. If the sending peer is not in the list, an appropriate Response (command 1) is sent and it is added to the peer list.
Another question is how often should multicast/broadcast message be sent out? Proposal:
- During bootstrap: Immediately at the beginning, then every 10 seconds until there is at least 1 peer.
- Every 10 minutes during regular operation.
- Each time a network adapter / IP change is detected.
Appendix 1
- Bitcoin client btcd: They push it off to the end-user? https://github.com/btcsuite/btcd/blob/master/docs/configuration.md#peer-server-listen-interface
btcd allows you to bind to specific interfaces which enables you to setup configurations with varying levels of complexity. The listen parameter can be specified on the command line as shown below with the -- prefix or in the configuration file without the -- prefix (as can all long command line options). The configuration file takes one entry per line.
NOTE: The listen flag can be specified multiple times to listen on multiple interfaces as a couple of the examples below illustrate.
Howdy, Stranger!