Fang-Pen Lin Eschews Multi-Gig Ethernet for a USB4 Mesh Network Hitting Nearly 12Gbps Throughput

Fang-Pen Lin, self-described "one-person-army software engineer," has put together a low-cost high-performance >10Gbps mesh network — by connecting mini-PCs together in a cluster with USB4 interconnections, requiring under $50 in cables.

"After some research, trial, and error, I finally built and ran a relatively low-cost cluster with a high-speed full-mesh interconnected network," Lin explains of the project. "The most interesting part is that the networking is based on a USB4 Ethernet bridge instead of a conventional Ethernet switch and cables. I tested the network speed, and it can hit 11Gbps. The cost of making the network is only $47.98!"

Initially, Lin had looked at using traditional 1U servers and 2.5-gigabit-Ethernet networking for the project. The servers were discounted in favor of compact miniaturized desktop PCs, offering quieter operation and lower running costs, but the cost of the hardware required for the 2.5-gig-Ethernet backbone — which would require a new high-speed Ethernet switch — was off-putting.

"While I was debating which switch to buy, doing my research regarding the brand, price, and my requirements, I realized, wait a minute, there are two USB4 ports on the machine," Lin explains. "In theory, it could provide up to 40Gbps speed. Who cares about 2.5G[bps]? That's 40Gbps we are talking about here! Considering the money spent on a 2.5G Ethernet router plus some Cat6 Ethernet cable, why not just make a full-mesh network with USB4 cables?"

Using $50 in cables total, Lin set about connecting the machines together over their USB4 ports — taking advantage of the ability to configure an Ethernet bridge over a USB port, making it appear to the system as a direct Ethernet connection to a second computer. The results fell short of the theoretical 40Gbps maximum available over USB4, but were more than four times faster than 2.5-gig-Ethernet — benchmarking at 11.8Gbps sustained throughput.

Lin notes that the mesh network only works because there are two USB4 ports per system and three systems total. "What if there are more than three nodes," he asks. "I recall reading some networking books that mentioned interesting ancient network structures a long time ago, such as ring topology networks or daisy-chain networks.

"There are many drawbacks to those network structures, and network equipment is pretty cheap, so those are rare nowadays. With a limited number of USB4/Thunderbolt ports and relatively expensive cable, maybe it makes sense to construct a network like the ancient ones."

Lin's full write-up is available on his blog.