Linux to Windows network performance

[timwhit]

So, I finally got fed up with having a crappy network at home and decided to do something about it.

I ran cat 6 solid cable through my ceiling (Rotozip into the drywall twice, push cables into resulting trough). I bought a Netgear GS108 8 port Gigabit switch and hooked it up when I got home from work tonight.

I tested everything was working between my Windows box and my Linux box using iperf. It reported speeds between 885mbps and 915mbps, which seems pretty good.

I have a Windows share mounted on my Linux box using fstab like this:

//192.168.1.103/desktop /media/htpc cifs credentials=/home/user/sambapass,uid=user,gid=users,file_mode=0777,dir_mode=0777

Here are the approximate performance numbers, both tests were run from the Linux box:
Copy file from Linux to Windows using cp: 26.6 MB/s
Copy file from Windows to Linux using cp: 6.69 MB/s

What I can't figure out is why the performance is so slow copying from the Windows box to the Linux box. I haven't tried to tune Samba or make any other changes yet. That will be my next step though. Any other ideas?

 

[timwhit]

I added the following to smb.conf:

socket options = IPTOS_LOWDELAY TCP_NODELAY

Copy file from Windows to Linux using cp is now: 9.34 MB/s

Still too slow.

 

[Chewy509]

Out of interest, what version of Windows and Linux? I know some Ubuntu 9.x had CIFS performance issues, but newer kernels have fixed most of those.

Also is the firewall on in Windows? Try turning it off.

If using Windows Vista or Win7, try to use SMB2 from your Linux box if you can, or alternatively turn SMB2 OFF on Windows if your linux box can't utilise SMB2.

Also what's the CPU utilisation on both when doing the copy operations? (may indicate PIO rather than DMA operations for the LAN adapters and/or HDDs).

Other options are to use a different file sharing protocol, eg NFS or FTP...

PS. I just tried it with my Arch Linux netbook (with kernel 2.6.37) to a Win7 laptop, and I got 2.6MB/sec with a 100Mbit connection for cp windows to linux. But copying from the same file from the same Win7 laptop to my Solaris 10 box on the same 100Mbit LAN, I got 5.5MB/s ?

 

[Bozo]

I have noticed the same thing on our network at work. Using a removable drive so the hardware was the same, going to and from the same XP computer, the Linux install was glacial. The data traveled through 3 Cisco Gb switches.
We screwed with this for a day and never could get the Linux install (it was some flavor of Redhat/Fedora) to move any faster.

 

[Mercutio]

What NICs are involved here?

 

[time]

{ Ubuntu }

cd /etc
sudo gedit hdparm.conf

Look for: write_cache = off

Change off to on, save and reboot.

 

[timwhit]

Out of interest, what version of Windows and Linux? I know some Ubuntu 9.x had CIFS performance issues, but newer kernels have fixed most of those.

It's Windows XP and OpenSuse 11.4.

Also is the firewall on in Windows? Try turning it off.

Both system's firewall are disabled.

If using Windows Vista or Win7, try to use SMB2 from your Linux box if you can, or alternatively turn SMB2 OFF on Windows if your linux box can't utilise SMB2.

I don't think this will be an issue with XP, correct?

Also what's the CPU utilisation on both when doing the copy operations? (may indicate PIO rather than DMA operations for the LAN adapters and/or HDDs).

I will need to test this one.

Other options are to use a different file sharing protocol, eg NFS or FTP...

NFS is out for Windows as far as I know. I have tried SFTP with similar results. SFTP isn't really FTP though and the encryption/decryption overhead crushes my CPU. I have not tried regular FTP yet.

 

[timwhit]

What NICs are involved here?

NICs are both onboard.

The Windows box uses this: http://www.newegg.com/Product/Product.aspx?Item=N82E16813128363

The Linux box uses this: http://www.newegg.com/Product/Product.aspx?Item=N82E16813128050

 

[timwhit]

I tested the CPU utilization on the Linux side and it's around 4-6%. I will test the Windows CPU utilization when I get home from work.

 

[Mercutio]

Do you have any way of looking at Windows to Windows or Linux to Linux throughput?

 

[timwhit]

Do you have any way of looking at Windows to Windows or Linux to Linux throughput?

I think I can test Linux to Linux by booting the Windows box from a Live image and using a NFS mount.

For Windows to Windows I can try connecting a laptop to the gigabit switch, I'm not sure if the laptop has a gigabit NIC though.

My other idea was to just scrap Windows and install Linux on the other machine. This is a lot more work, but I may still consider it.

 

[Chewy509]

My other idea was to just scrap Windows and install Linux on the other machine. This is a lot more work, but I may still consider it.

Sounds like the best plan so far! Then just run NFS across the board, avoiding CIFS (which searching the interweb, sees lots of performance issues with CIFS and Linux).

As a side not, I just copied 15GB of music from my Solaris box to my Linux Netbook via NFS, and was getting 11.1MB/s on a 100Mbit LAN. (Could never get that performance when using CIFS).

PS. I'm using my ADSL modem/router as the switch, which only has 100Mbit LAN ports, will upgrade the router once I can find a modem/router that supports IPv6 natively and has gigabit ports.

Disclaimer: I don't like Windows as a product, and try to avoid it where possible.

 

[timwhit]

PS. I'm using my ADSL modem/router as the switch, which only has 100Mbit LAN ports, will upgrade the router once I can find a modem/router that supports IPv6 natively and has gigabit ports.

Any reason not to just keep your current router and buy a gigabit switch?

 

[timwhit]

{ Ubuntu }

cd /etc
sudo gedit hdparm.conf

Look for: write_cache = off

Change off to on, save and reboot.

I believe the file is /etc/sysconfig/ide for OpenSuse.
The file was essentially empty so I added "write_cache = on"

I ran the test again and the results were essentially the same as before.

 

[Chewy509]

Any reason not to just keep your current router and buy a gigabit switch?

lack of physical room, lack of power outlets (already have filled a 8 point surge protector with stuff), and like to keep it simple. And 1 less device means less power and heat.

 

[timwhit]

I tested the CPU utilization on the Linux side and it's around 4-6%. I will test the Windows CPU utilization when I get home from work.

CPU Utilization on the Windows side is around 17-14%. That box has a C2D, don't remember what model though.

 

[LiamC]

Packet fragmentation? different frame sizes? Have you updated the drivers on the Windows box? See if the Realtek site has a later version than on the Gigabyte site.

On the Linux side for CIFS, see if Asynchronous I/O is enabled or disabled. Whatever it is, toggle it and retest? Some Linux/BSD flavours shipped NIC configs with Device polling (rather than interrupt driven) enabled. Check, toggle and retest.

 

[timwhit]

I decided to scrap Windows and installed OpenSuse on my HTPC. I now am able to transfer files over NFS at around 60 MB/s.

Thanks for all the help.

Now to figure out why I can't run a non-standard resolution with nvidia-settings.

 

[Chewy509]

Now to figure out why I can't run a non-standard resolution with nvidia-settings.

Maybe poor/bad EDID data from your display? Using HDMI, DVD or VGA?

 

[timwhit]

Maybe poor/bad EDID data from your display? Using HDMI, DVD or VGA?

HDMI.

I've already tried adding the resolution I want (1440x810) to xorg.conf, but no luck.

Here's an article that explains how to add a custom resolution: http://sathyasays.com/2008/10/26/how-to-tackle-screen-resolution-problems-in-linux/
I'll try this tonight.

 

[timwhit]

Here's another guide from the Ubuntu forums that looks decent:
http://ubuntuforums.org/showthread.php?t=83973

 

[timwhit]

I've tried everything possible to get the display to run at the resolution I want (1440x810) with no luck. This should not be that difficult to achieve, as it worked fine in Windows.

 

[Stereodude]

The joys of Linux...

 

[timwhit]

The joys of Linux...

There's upsides and downsides to anything.

Upside: 60MB/s transfer rate, downside: xorg.conf.

For now, I have given up on trying to force a custom resolution and have just increased the font sizes to be readable at 1920x1080. Firefox also does a good job of scaling websites when zooming in.

 

[time]

So an order of magnitude difference between Windblows and Linux? Is it too far fetched to wonder if Microsoft deliberately nobbled this setup?

Whatever, great result, Timwhit. Sorry to hear about your EDID woes, too much stuff assumes the values are always perfect.

 

[timwhit]

I was having trouble with NFS4 locking up my desktop when transferring files. I found similar problems that others were having with no real solution. I decided to order a real NIC rather than screwing around with the crappy onboard Realtek 8111B trying to get it to work. I ordered an Intel PWLA8391GT, which seems to have fixed the problem. The added benefit is that transfer rates now hover around 80MB/s.

 

[timwhit]

I ran a few timed tests to actually get real numbers rather than just eyeballing the copy dialog in KDE. I ran the tests exactly the same was as in the first post in this thread. The Windows box is now referred to as the HTPC and the Linux box is now the Desktop. The file is ~4476MB if anyone cares.

Here's the real numbers:
Desktop to HTPC: 67MB/s
HTPC to Desktop: 61MB/s

Still a ton better than CIFS. I wonder what kind of performance two Windows machines would get on this same setup, but I am not masochistic enough to attempt that now that everything is working the way I want.

 

[Handruin]

I'm curious what the difference is in a configuration that I'm running at home which is essentially Win 7 to rpath linux (my openfiler NAS). I get fantastic CIFS performance when writing to the linux-based NAS. Maybe there are some tweaked CIFS configurations to overcome some of the issues you had seen so far? Granted it's rpath vs your opensuse...maybe that's enough of a difference?

 

[timwhit]

Maybe OpenFiler has a better standard Samba configuration than OpenSuse. I'm not a Samba expert, nor did I want to waste the time fiddling with the settings. I wonder if OpenFiler uses standard Samba or if they modify it?

 

[LiamC]

I also get great CIFS performance from FreeNAS—details here: http://www.storageforum.net/forum/showpost.php?p=140322&postcount=36

I just copied a 4770MB VOB file for 48-49MB/s write and around 61MB/s read (when copied from the FreeNAS).

 

[Bookmage]

heh, the wonder of SMB and CIFS... I had something similar, but only with Windows XP and only when copying from windows to linux via a mounted share on windows.
when i mounted the windows shares in ubuntu, it was slower, but not that slow...
I used the netgear GS108T
Windows Vista was faster than XP, but ubuntu/centos to ubuntu via cifs and nfs was faster and i think win 7 to wni 7 was also faster than vista to ubuntu... but I at least got 40-60 MB/s depending on files and what not. ubuntu raid to ubuntu raid gets me 80+ MB/s

I vaguely remember reading somewhere about XP having max bw issues with gigabit nics... like it hit a max and couldn't go any faster... but I've since stopped paying attention to XP news

 

[blakerwry]

I've found that NIC drivers play the biggest role in network performance (moreso than the NIC itself). I tend to stick to intel NICs because they always seem to work well in whatever OS I choose (DOS, Linux, Windows) because of the broad and tested driver support.

Between different drivers I've seen performance go from ~6 to ~11MB/sec on the same NIC using SMB. I haven't done any real testing recently, but I routinely get ~60MB/sec on file transfers using SMB between my WHS server and workstations. The point being that CIFS/SMB is certainly capable of the speeds you're looking for, with some variation between implementations (SAMBA version, Windows version, etc).

If you do decide to go back to Windows, you might compare NIC drivers from Microsoft, the motherboard manuf, and the NIC chipset manuf to see if one resolves the problem.

 

[timwhit]

I doubt I will go back to Windows. I find Linux to just work a lot better than Windows. Plus, I like how much it costs.