The 100gig Adventure

"...so I ended up upgrading everything to 100Gbps, that's what you do, right?" No, Wendell, most of us don't have the kind of equipment to do that 😂

I'm a network engineer at an ISP. We're starting to install 100g internet connections. Pretty wild.

Wendel waves around a connectX-5 calling it old... I'm gonna go cuddle my beloved connectX-3...

Off to convince my wife of how this will significantly improve our quality of life. Wish me luck! :P

"I ended up upgrading everything to 100Gb, because that's what you do, right?" Yes. That is what you do.

God the FEC nightmare is real. I spend days trying to figure out why I couldn't get a Mikrotik router to talk to a Ubiquiti switch at 25 gig and the answer was FEC.

"Experimental RDMA capable SMB Server that's floating around out there on the internet" ... GO ON...

Network engineer here, we’re already on 100G between Metro POP sites and intercapital links for several months already, and now standing up multiple 100G link bundles on intracapital core links. Also our colleagues in the internet peering team are running multiple 100G link bundles on our internet borders.

Wendell is my Mr. Wizard for computers.

Nah, you don’t need ASIC for deep trafic analysis on 100Gb/s network. At my place, we do DPI at 100Gb/s with only one CPU (32-cores, though) and 64GB of RAM. Full line-rate certified by Spirent, at 14 millions new sessions per second and 140 mpps. But to do that, we had to redevelop the drivers of the e810 from scratch in Rust for everything to work in userspace (DPDK is… too limited for that). So it’s possible, took us 3 years of R&D, though ;-)

You should do a review of QNAP's 4x100GbE + 8x25GbE switch. It's reasonably priced and uses current gen tech, so much lower power/fan noise and has more ports than the Mikrotik 100GbE switch. It won't have all the fancy layer 3 capabilities of the used switches, but I'd like to see how it compares for those of us who care about noise.

In a previous job, I worked with 2x400G cards using DPDK. It was glorious to run TRex on the other side of the wire and see 800G flowing through our program.

My first corporate job had 4Mb/s Token ring networking. What a leap since.

I got used 100GbE-CWDM4 transceivers for $5 each off ebay. Those run with LC terminated duplex single mode fiber, which is much easier to deal with than 8 fiber MPO.

Fair to say I didn't understand most of what he was talking about but it was fun to listen to.

From a consumer perspective, 10 gig hardware (be it NICs, transceivers, or switches) are dirt cheap now. You can get any of those for under a hundred dollars now. But 25 gig and 100 gig, while the NICs/transceivers are affordable or even cheap, the switches are still in the high hundreds to low thousands of dollars. And those switches will probably need a secondary switch to connect 1 or 2.5 gig devices to your network, you can't really get an all-in-one switch like you can for mixing 1/2.5 and 10 gig clients. The costs add up fast and your minimum investment ends up in the thousands of dollars.

We use Xinnor for NVMe RAID in film post production where bandwidth is more important than data integrity of something like ZFS.

I just upgraded part of my home network from 1 gbe to 10 gbe and it was a huge quality of life improvement. Moving large files to/from my NAS is fast! Upgrading to 100 gbe sounds insane to me.

Video/content suggestion: Boot Windows over such a 100 GbE adapter from a ZFS Server and how to get the most performance out of it.

Remember the intel cards are 100g full duplex, while the Mellanox could push line rate per port if the pcie bus wouldn't limit it. The cx4 is still supported, as it uses the same driver as the cx7. If one does not need the new features like 200g or 400g the old cards are almost as capable. That slcould however not be said for 100g cards from qpogoc which are a pain in the ass compared to mlx and intel I would love to see some stuff with dpdk and vpp. A 100G router in x86 is very cool