So recently I dealt with a rather aggravating problem. Basically I had a couple of Poweredge R515’s to be built with NexentaStor Enterprise, which would not get a valid network configuration.
The Problem
During the initial setup via the console… I selected bnx0 as the primary and set it to use DHCP, after a bit of chewing I received the error below…
IOError: can not safely set interface bnx0 via DHCP.
After the wizard I went into the shell to look at the configuration.
nmc@myhost:/$ show network interface bnx0 ==== Interface: bnx0 ==== bnx0: flages=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1496 index 5 inet 0.0.0.0 netmask 00000000 broadcast 255.255.255.255 ether e8:9a:8f:be:25:ab
The only real important points to note from this output is that the IP address is 0.0.0.0, which is expected since it threw an error during the initial configuration. Additionally the MTU is set for 1496.
Configure Static IP with MTU, Gateway, and Nameserver
So perhaps the problem is specific with DHCP, so I go through the process of configuring the IP address statically.
nmc@myhost:/$ setup network interface bnx0 Option ? static bnx0 IP address : 10.0.8.41 bnx0 netmask : 255.255.255.0 bnx0 mtu : 1500 Gateway IP address : 10.0.8.1 Name Server #1 : 8.8.8.8 Name Server #2 : Name Server #3 : SystemCallError: failed to configure bnx0 with ip 10.0.8.41 netmask 255.255.255.0 mtu 1500 broadcast + up: ifconfig: setifmtu: SIOCSLIFMTU: bnx0: invalid argument
Configure Static IP with MTU and Gateway
So the error above says invalid argument, so lets remove stuff from the bottom up, no more name server.
nmc@myhost:/$ setup network interface bnx0 Option ? static bnx0 IP address : 10.0.8.41 bnx0 netmask : 255.255.255.0 bnx0 mtu : 1500 Gateway IP address : 10.0.8.1 Name Server #1 : Name Server #2 : Name Server #3 : SystemCallError: add net default: gateway 10.0.8.1: Network is unreachable
Configure Static IP with MTU
So above we have an issue with being unable to reach the gateway so Nexenta won’t accept the configuration, fine lets remove that.
nmc@myhost:/$ setup network interface bnx0 Option ? static bnx0 IP address : 10.0.8.41 bnx0 netmask : 255.255.255.0 bnx0 mtu : 1500 Gateway IP address : Name Server #1 : Name Server #2 : Name Server #3 : OK.
Check Interface Configuration for MTU
So above this responds with an OK. As in alright everything worked, but not so fast, we haven’t done anything to fix it and by definition our configuration would still be broken, no gateway and no name server. So lets look at our configuration.
nmc@myhost:/$ show network interface bnx0 ==== Interface: bnx0 ==== bnx0: flages=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1496 index 5 inet 10.0.8.41 netmask ffffff00 broadcast 10.0.8.255 ether e8:9a:8f:be:25:ab
Configure Static IP with MTU of 1496
Interesting, so it didn’t actually accept our MTU of 1500, like it said it did. Well lets see what happens if we play along and try and set the MTU to 1496 – since Nexenta is so adamant that is the correct way.
nmc@myhost:/$ setup network interface bnx0 Option ? static bnx0 IP address : 10.0.8.41 bnx0 netmask : 255.255.255.0 bnx0 mtu : 1496 Invalid mtu '1496' please enter value in range [1500:8982]
How contradictory, apparently Nexenta will only accept 1500-8982 as a valid MTU. Looking at our configuration we can see that nothing changed, and our configuration is still broken.
nmc@myhost:/$ show network interface bnx0 ==== Interface: bnx0 ==== bnx0: flages=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1496 index 5 inet 10.0.8.41 netmask ffffff00 broadcast 10.0.8.255 ether e8:9a:8f:be:25:ab
The Fix
Now the problem here is that the Broadcom NICs are using an invalid MTU setting and Nexenta is not smart enough to properly handle it, so we need to tell the bnx driver how to use the Broadcom NICs.
Enter expert mode, this allows us to use an actual command shell, instead of the incredibly limited NMC.
nmc@myhost:/# option expert_mode = 1
Now that we are in expert mode, we can do silly things like launch bash. I have no idea why we need to say not-bash to launch bash, but we do.
nmc@myhost:/# !bash
You are about to enter the Unix ("raw") shell and execute low-level Unix command(s). Warning: using low-level Unix commands is not recommended! Execute? (y/n)
Below is a little sed snippet which will allow you to uncomment the line without actually entering the file yourself.
nmc@myhost:/# sed -i '/mtu=/s/^#//' /kernel/drv/bnx.conf
Quick explanation on what this sed command does, -i tells it to make changes “in-place” or directly on the file in question, without this the file is sent to stdout where the change is made. The apostrophes annotate the full search string, separated by forward slashes. The first section /mtu=/ is our search string, so basically any line that matches this search string will have the next part run against it. The second section s/^#// is actually two sections, but it is a substitution section so we will treat it as one, since wherever it finds “^#” it will replace with “” so what is “^#” simple, it is a comment which occurs as the first character in a line. This simply removes that comment.
A simpler but much more verbose method would be to substitute the whole line (with the comment) for the whole line without the comment.
nmc@myhost:/# sed -i 's/#mtu=1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500;/mtu=1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500;/g' /kernel/drv/bnx.conf
Or if you’d prefer to handle the change manually what we are doing is uncommenting the following line from /kernel/drv/bnx.conf
#mtu=1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500;
Verify the change, then reboot.
# cat /kernel/drv/bnx.conf | grep mtu= mtu=1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500,1500;
# reboot
After the reboot, we can finally appropriately configure our networking.
nmc@myhost:/$ setup network interface bnx0 Option ? static bnx0 IP address : 10.0.8.41 bnx0 netmask : 255.255.255.0 bnx0 mtu : 1500 Gateway IP address : 10.0.8.1 Name Server #1 : 8.8.8.8 Name Server #2 : Name Server #3 : Enabling bnx0 as 10.0.8.41/255.255.255.0 mtu 1500 ... OK.
Bottom Line
Once I implemented this change I was able to successfully configure my networking with either DHCP or static, with the appropriate MTU for my purpose. I have no idea why this is required, if a simple uncommenting of the line is required then why would Nexenta as a Solution Builder perform that uncommenting for me to spare me from the trouble of doing it myself. I personally prefer “pure” ZFS solutions without the pretty interface, especially considering that I spent more time trying to learn how to use the Nexenta Management Console (command line) then I should have had to based on my experience. Add into this the fact that I have to do low level changes like this just to get basic networking so that I can run through the wizard.