Joining DN42

• 4 min read
Tags: Web  
⬇Download as Markdown ⬇Download as Gemtext

Last summer I joined the DN42 virtual network. Here is at last my experience in DN42.

DN42 is a community experiment that operates a large virtual network aimed at simulating the Internet. Optic fibers are replaced by Wireguard tunnels, and expensive routers are replace by cheap servers running Bird/FRRouting.

The bandwith is lower, the latency is high, there’s less things to download, but everything uses real-world protocols. The main goal of DN42 is to learn and experiment with Internet routing technologies.

Services

The DN42 has setup several of the base technologies that underpin the modern Internet:

  • IPv4 and IPv6
  • DNS, with a dummy DNSSEC implementation
  • a Certificate Authority for HTTPS, with ACME support
  • some websites like the Wiki
  • IRC (hackint)
  • a Git registry simulates the Local Internet Registry but any paperwork
  • some Internet Exchanges
  • BGP/IP Looking Glass, where you can run pings/traceroutes on other people’s hosts.

What can you actually do there ?

You can experiment in relative safety and with little consequences. You can route your prefixes as anycast, unicast, and some people even run a multicast network.

Most people have several routers, so you can setup your IGP however you like. I have used IS-IS.

The “default” setup

Most people choose Bird as the routing software, Wireguard for tunnels. They use IPv6 link-local peerings with extended next-hop and Multiprocol BGP. This means that no transfer networks are required, and IPv4 can flow over IPv6 peerings (provided your router has an IPv4).

Several networks in DN42 have an “auto-peer” service, where you can setup a peering without talking to a human ! This is great for starting, as you can test against “known good” configs. Some auto-peer service have monitoring.

Below is an example of interconnection between several peers.

    graph LR
subgraph AutoPeer1
R1a(R1)---|iBGP|R2a(R2)
end
subgraph You
R1y(R1)---|OSPF|R2y(R2)
end

R1a---|eBGP peeering|R1y
R2y---|eBGP peeering|R2a

R2y---|eBGP transit|Rb(another peer)

Rules and unwritten rules

DN42 is a community project for learning, so keep in mind that you aren’t owed anything, and people may leave the project for any reason.

On IPv4, keep the lenght of your prefix large, e.g. don’t request a /24 because like the Internet, IPv4 is a scarce resource in DN42.

On IPv6, many network will refuse prefixes larger than /64, and de-aggregating your prefixes isn’t well liked.

When you are peering with an auto-peer service, avoid providing transit (only send your prefixes), because the auto-peer is likely better connected than you.

Your routers should be reachable 24/7, to avoid causing network instability. If everyone aroud the earth were to shutdown their servers at night, it would cause a continuous disruption that would follow the sunset …

The most important thing to watch for is flapping: this is when a peering disconnects and reconnects frequently. A flapping link causes a flood of BGP updates that propagate to most of the network.

With lots of flapping, DN42 ends up being mostly about BGP updates …

My (old) network

I already had two machines in Oracle Cloud, so I added a third one to be my main DN42 router. Another router followed soon after. After that I expanded my network in Poland & the Netherlands, with two Scaleway VPS.

Here is my old network at its peak:

    flowchart TB
dn((DN42))---in((Internet))

subgraph SCW Warszawie
waw1[PL-WAR1]
end
waw1---|IS-IS l2tp|r19

r42---|IS-IS l2tp|waw1

subgraph OCI Paris
r42[FR-PAR1]---|IS-IS vxlan|vx
r43[FR-PAR2]---|IS-IS vxlan|vx
vx((VXLAN)):::hidden
vx---|IS-IS vxlan|r21[Internal R21]
vx---|IS-IS vxlan|r19[Internal R19]
end

in---|BGP wireguard|r42
in---|BGP wireguard|r43
in---|BGP wireguard|waw1
in---|BGP wireguard|ams1

subgraph SCW Amsterdam
ams1[NL-AMS1]
end
r43---|IS-IS l2tp|ams1
ams1---|IS-IS l2tp|r21

subgraph Home Lab
netgear[OpenWRT]
end
r19---|IS-IS l2tp|netgear
r21---|IS-IS l2tp|netgear

Because of some filtering in Oracle, I deployed a VXLAN mesh between my Oracle machines.

I chose IS-IS to run as my IGP, because like BGP it is multi-protocol: it can route both IPv4 and IPv6. For instance, OSPF or RIP require two separate processes to router IPv4 and IPv6. One OSPF process proved to be almost too much for my little OpenWRT home router (netgear).

IS-IS is not without some downsides though: it runs over Ethernet instead of pure IP, so I had to setup (unencrypted) L2TPv3 tunnels between my hosts.

My (new) network

One’s DN42 network is never really finished … My plan is to containerize the BGP routers, so that I have a clear boundary between my own network and the rest of DN42. This will be achieved by moving the Wireguard interfaces into the BGP routing namespace. In the end, my DN42 routers will not even have Internet access !

My IGP will switch from IS-IS to OSPFv3 (for IPv6) and RIP (for IPv4). Yes I know, RIP is not very cool but my OpenWRT router does not have a lot of memory, and it is cool to have two different procols for IPv4 and IPv6, so one mistake does not necesserily impact both protocols.

My internal links will be all Wireguard tunnels, so everything will be encrypted and I will be able to provide secure transit.

Read more