Strange network performance behavior troubleshooting

[Handruin]

I've been trying to debug this odd performance behavior on my new NAS and I'm curious if anyone has more ideas I can try before I write off the NIC as a bad part. The short of it is, if lots of incoming traffic is received to this system, it appears to throttle the bandwidth. However if this system sends lots of outbound traffic, it does not appear to throttle. I've been using iper3 for testing server-to-server performance as a way to troubleshoot.

Some notes

• All three systems are running Ubuntu Server 24.04.2 LTS
• All three are cabled to the same 10Gb switch using DAC cables into SFP+ ports
• All three are using a version of a Mellanox Connect-X 3 NIC
• All three are on a flat network with no routing involved

System A (the problem machine)
System B
System C

When I run iperf to send from System B to System A I see performance drops after a few seconds.

iperf3 -c 192.168.1.231 -p 3000 -t 20
Connecting to host 192.168.1.231, port 3000
[  5] local 192.168.1.226 port 58434 connected to 192.168.1.231 port 3000
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   823 MBytes  6.90 Gbits/sec    0   1.58 MBytes
[  5]   1.00-2.00   sec   484 MBytes  4.06 Gbits/sec    0   1.58 MBytes
[  5]   2.00-3.00   sec   484 MBytes  4.06 Gbits/sec    0   1.58 MBytes
[  5]   3.00-4.00   sec   482 MBytes  4.05 Gbits/sec    0   1.58 MBytes
[  5]   4.00-5.00   sec   482 MBytes  4.05 Gbits/sec    0   1.58 MBytes
[  5]   5.00-6.00   sec   485 MBytes  4.07 Gbits/sec    0   1.58 MBytes
[  5]   6.00-7.00   sec   564 MBytes  4.73 Gbits/sec    9   1.20 MBytes
[  5]   7.00-8.00   sec   546 MBytes  4.58 Gbits/sec    0   1.23 MBytes
[  5]   8.00-9.00   sec   469 MBytes  3.93 Gbits/sec    0   1.25 MBytes

However if when I run iperf to send from System A to System B things look much better (but there are still retries I don't like seeing)

iperf3 -c 192.168.1.226 -p 3000 -t 20
Connecting to host 192.168.1.226, port 3000
[  5] local 192.168.1.231 port 46358 connected to 192.168.1.226 port 3000
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.40 MBytes
[  5]   1.00-2.00   sec  1.09 GBytes  9.39 Gbits/sec    0   1.47 MBytes
[  5]   2.00-3.00   sec  1.09 GBytes  9.39 Gbits/sec    0   1.47 MBytes
[  5]   3.00-4.00   sec  1.09 GBytes  9.38 Gbits/sec    0   1.68 MBytes
[  5]   4.00-5.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.76 MBytes
[  5]   5.00-6.00   sec  1.09 GBytes  9.39 Gbits/sec    0   1.76 MBytes
[  5]   6.00-7.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.76 MBytes
[  5]   7.00-8.00   sec  1.09 GBytes  9.35 Gbits/sec  254   1.28 MBytes
[  5]   8.00-9.00   sec  1.09 GBytes  9.36 Gbits/sec    0   1.44 MBytes
[  5]   9.00-10.00  sec  1.09 GBytes  9.38 Gbits/sec    0   1.44 MBytes

I did a similar test using System C to System A and I get very similar results. The majority of the performance drop is shown when a system is sending traffic to the problem System A.

Followup troubleshooting:

1. I changed to a new SFP+ switch port for System A - no change in behavior
2. I swapped the DAC cable for a new one for System A - no change in behavior
3. I painstakingly updated the firmware on the Mellanox Connectx-3 (MCX311A-XCAT) to the latest for System A - no change in behavior
   1. I made sure to power cycle after firmware update so it got loaded

I'm out of troubleshooting ideas. I tried using the Mellanox mst software to try and probe temperatures to see if when other systems are pushing traffic to it if the ASIC chip may be overheating but this specific NIC isn't known to run hot. I couldn't get the temperature readings unfortunetly. Still doesn't explain why the opposite traffic direction doesn't throttle when System A is sending data to other machines.

I paid $25 for the NIC, I may just replace it and see if that solves my issue but wanted to ask if there are other things I haven't considered.

 

[Mercutio]

Can you direct connect the two NICs without involving the switch? It seems like your switch is ruled out of involvement but it's worth a try.
What if you switch the two cards to IB mode instead of ETH?
Are you writing to equivalent storage on both sides? 4Gbit sounds weirdly close to SATA SSD top speeds.

 

[ddrueding]

Another thing to help rule out the switch is doing a test from System B to System C, but I think Merc might be right on the system transfer limit (bursting through the cache and then limiting).

 

[Handruin]

The iperf3 test does not write to any storage, it just goes to RAM so this shouldn't be a limiting factor here. I also did test performance from system B > C and those tests run near 100% bandwidth in each direction (B > C and C > B).

However, I tested a new hypothesis just now and I ran the linux utility named "stress" on the problematic system to load up all 8 cores on this CPU (Intel Atom processor C3758 8-Core). I was curious if this was some kind of C-state idling issue and I think it may be.

When I load up all 8 cores and rerun the test to have this problematic system receive traffic, it gets near ~99% the 10Gb bandwidth. If I kill the "stress" utility mid performance test, the bandwidth in the test also drops near 50-60% at the same time.

Now I need to figure out how adjust this or even if it will be a real world issue. Maybe it's just related to how the CPU schedules time for the "iperf3" utility.

 

[Mercutio]

I've seen mini PC systems get stuck at their lowest Speedstep state as a misbegotten firmware default. Turning off some of the C state options has helped me deal with that.

 

[Handruin]

I'll look into making changes there. I have read a bit and see there are some linux utilities "cpupower" that may be able to play around with this more in software but I need to do more reading.

I appreciate the suggestions on this, was definitely odd with only a single direction having issues.

 

[Santilli]

Sort of the same issue:
I've tested, and concluded the motherboard has Bluetooth issues.
Does anyone have any suggestions for a PCIe card that would work with 11, and is
capable of picking up my speaker system's Bluetooth signal?
The speaker system is NOT the issue. Connects VERY quickly to my phone, etc.

 

[Handruin]

You may want to make a new thread asking about bluetooth adapters, it's not at all related to my 10Gb networking issues.