Solaris 10 zone network configuration

I am testing out zones on Solaris 10, release 10/09 (s10x_u8wos_08a) with Generic_142901-04 kernel.

The host server has two physical NIC's (bnx0 and bnx1), and since I am intending to run several (three or more) zones, dedicating a NIC to a zone is out of the question. Presently, I do not have any "dedicated" IP-addresses assigned to these zones, so I'm obtaining leases from DHCP.

My questions, which we'll get back to once I've shown you my current setup, revolve mainly around this network configuration.

Here are the steps (with some additional information) I took to set up the zone:

erikn@global:~ $ zpool status zpool1
  pool: zpool1
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zpool1      ONLINE       0     0     0
          c1t1d0    ONLINE       0     0     0

errors: No known data errors

erikn@global:~ $ zfs list zpool1
NAME     USED  AVAIL  REFER  MOUNTPOINT
zpool1  84.1G   183G  6.14G  /zpool1

erikn@global:~ $ pfexec zfs create zpool1/zones

erikn@global:~ $ pfexec zfs set mountpoint=/zones zpool1/zones

erikn@global:~ $ pfexec zonecfg -z test \
> 'create -b ;
> set zonepath=/zones/test ;
> verify ;
> commit'

erikn@global:~ $ zoneadm list -vic
  ID NAME             STATUS     PATH                           BRAND    IP
   0 global           running    /                              native   shared
   - test             configured /zones/test                    native   shared

I proceeded to install the zone

erikn@global:~ $ pfexec zoneadm -z test install
A ZFS file system has been created for this zone.
Preparing to install zone <test>.
Creating list of files to copy from the global zone.
Copying <196614> files to the zone.
Initializing zone product registry.
Determining zone package initialization order.
Preparing to initialize <1595> packages on the zone.
Initialized <1595> packages on zone.
Zone <test> is initialized.
Installation of these packages generated errors: <SUNWvbox SUNWpostgr-83-server-data-root CSWcacertificates CSKmysql32 SUNWpostgr-82-libs SUNWpostgr-82-server SUNWpostgr-82-client SUNWpostgr-82-devel CSKphp5 SUNWpostgr-82-contrib CSKapache2 SUNWpostgr-82-server-data-root>
Installation of <1> packages was skipped.
The file </zones/test/root/var/sadm/system/logs/install_log> contains a log of the zone installation.

So some packages generated errors, but that doesn't concern me much since I do not need them inside this zone anyway.

The install exited with zero anyway.

erikn@global:~ $ echo $?
0

I booted up the zone

erikn@global:~ $ pfexec zoneadm -z test boot

And connected to the "console".

erikn@global:~ $ pfexec zlogin -C test

I followed the installation; selecting a language (English), setting the locale (U.S.A. (en_US.ISO8859-1)), terminal type (DEC VT100), giving it a hostname (test), setting time zone (Europe/Oslo), setting a root password ("password123" – no, just kidding).

Changed the root shell to bash:

# usermod -s /usr/bin/bash root
UX: usermod: root is currently logged in, some changes may not take effect until next login.

Logged out and in again.

Since there were a lot of enabled services (including telnet and finger), I figured it'd be easier to disable everything and then re-enable services untill I had a working system.

-bash-3.00# svcs | grep " svc:" | sed "s/ \{1,\}/ /g" | cut -d' ' -f3 | \
> xargs svcadm disable

Doing this resulted, as expected, in the zone entering System Maintenance Mode.

I re-enabled all the milstone services:

-bash-3.00# svcs -a | grep " svc:/milestone/" | sed "s/ \{1,\}/ /g" | \
> cut -d' ' -f3 | xargs svcadm enable

Enabled sshd and syslogd.

-bash-3.00# svcadm enable svc:/network/ssh:default
-bash-3.00# svcadm enable svc:/system/system-log:default

Looped over dependency services, enabling them:

-bash-3.00# while [ $( svcs -xv | wc -l ) -gt 0 ] ; do
> svcs -xv | grep "^svc" | cut -d' ' -f1 | xargs svcadm enable
> done
Hostname: test
Reading ZFS config: done.

Enabled console-login and checked the status.

-bash-3.00# svcadm enable svc:/system/console-login:default

-bash-3.00# svcs -xv
svc:/system/console-login:default (Console login)
 State: offline since August 27, 2012 11:38:04 AM CEST
Reason: Unknown.
   See: http://sun.com/msg/SMF-8000-AR
   See: man -M /usr/share/man -s 1M ttymon
Impact: This service is not running.

Rebooted the zone.

-bash-3.00# reboot

Was greeted by the console login prompt. Logged in. Checked service status:

-bash-3.00# svcs -xv

No output is good output.

Halted the zone and took a snapshot from the global zone for good measure.

-bash-3.00# halt
Aug 27 12:10:08 test halt: halted by root
Aug 27 12:10:09 test syslogd: going down on signal 15

[NOTICE: Zone halted]
~.
[Connection to zone 'test' console closed]

erikn@global:~ $ pfexec zfs snapshot zpool1/zones/test@neat

After this, I booted up the zone again, figured out the dependency order of the packages which were installed (using this the "pkgdep" script found here), and removed some of the stuff that I didn't need.

Next, it was time to configure the network, which is what this question is about.

From the global zone, I created a logical interface and assigned it to the zone "test".

erikn@global:~ $ pfexec ifconfig bnx1:1 plumb zone test

We can verify that this was assigned to the zone:

-bash-3.00# ifconfig bnx1:1
bnx1:1: flags=1100842<BROADCAST,RUNNING,MULTICAST,ROUTER,IPv4> mtu 1500 index 5
        inet 0.0.0.0 netmask 0

Now, since the zone is configured with IP-type "shared", it won't be able to send DHCP requests on the interface. Therefore, we do it from the global zone:

erikn@global:~ $ pfexec ifconfig bnx1:1 dhcp start
ifconfig: bnx1:1: wait timed out, operation still pending...

That didn't go too well, now did it?

erikn@global:~ $ echo $?
4

Back to the zone.

-bash-3.00# ifconfig bnx1:1
bnx1:1: flags=1104843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,ROUTER,IPv4> mtu 1500 index 5
        inet 10.0.6.92 netmask fffffc00 broadcast 10.0.7.255

Well, we have an IP-address… (Here, I have replaced the network with 10.0.4.0/22, but the netmask is the same).

We modify our /etc/nsswitch.conf and /etc/resolv.conf to use our name servers.

Since we are using a HTTP proxy on our network, we must remember to export that (i usually put it in ~/.bash_profile)

-bash-3.00# export http_proxy=http://proxy.example.com:8080

-bash-3.00# /usr/sfw/bin/wget http://www.google.com
--2012-08-27 13:03:30--  http://www.google.com/
Resolving proxy.example.com... 10.0.7.17
Connecting to proxy.example.com|10.0.7.17|:8080... connected.
Proxy request sent, awaiting response... 302 Found
Location: http://www.google.no/ [following]
--2012-08-27 13:03:40--  http://www.google.no/
Connecting to proxy.example.com|10.0.7.17|:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `index.html'

    [ <=>                                   ] 11,343      --.-K/s   in 0s

2012-08-27 13:03:40 (65.3 MB/s) - `index.html' saved [11343]

Although things appear to be working, it's not perfect. Let's go back to the global zone.

First we have the real bnx1 interface, which is also using DHCP.

erikn@global:~ $ ifconfig bnx1 dhcp status
Interface  State         Sent  Recv  Declined  Flags
bnx1       BOUND         2357  1558       778
(Began, Expires, Renew) = (08/27/2012 12:09, 08/31/2012 12:09, 08/29/2012 12:09)

Then we have our logical interface:

erikn@global:~ $ ifconfig bnx1:1 dhcp status
Interface  State         Sent  Recv  Declined  Flags
bnx1:1     PRE_BOUND       20    14         6

Let's create another logical interface, but we'll not give this one to any zone, so it'll belong to the global zone:

erikn@global:~ $ pfexec ifconfig bnx1:2 plumb

erikn@global:~ $ pfexec ifconfig bnx1:2 dhcp start

erikn@global:~ $ echo $?
0

erikn@global:~ $ ifconfig bnx1:2
bnx1:2: flags=1104843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,ROUTER,IPv4> mtu 1500 index 5
        inet 10.0.6.123 netmask fffffc00 broadcast 10.0.7.255

erikn@global:~ $ ifconfig bnx1:2 dhcp status
Interface  State         Sent  Recv  Declined  Flags
bnx1:2     BOUND            5     4         1
(Began, Expires, Renew) = (08/27/2012 13:11, 08/31/2012 13:11, 08/29/2012 13:11)

So that worked pretty flawlessly. Alas, it doesn't help us much.

I attempted to hand that interface over to the zone "test":

erikn@global:~ $ pfexec ifconfig bnx1:2 zone test
erikn@global:~ $ echo $?
0

It wasn't too happy about that:

erikn@global:~ $ ifconfig bnx1:2 dhcp status
Interface  State         Sent  Recv  Declined  Flags
bnx1:2     DECLINING        6     4         2

And releasing and starting DHCP client anew on the interface leads to the same result as that which we saw last time:

erikn@global:~ $ pfexec ifconfig bnx1:2 dhcp release

erikn@global:~ $ pfexec ifconfig bnx1:2 dhcp start
ifconfig: bnx1:2: wait timed out, operation still pending...

As a last, desperate attempt, I tried:

erikn@global:~ $ pfexec ifconfig bnx1:2 plumb \
> netmask 0xfffffc00 \
> broadcast 10.0.7.255 \
> zone test \
> up

No such luck.

erikn@global:~ $ pfexec ifconfig bnx1:2 dhcp start
ifconfig: bnx1:2: wait timed out, operation still pending...

My questions are basically:

• Is it at all possible to have DHCP assigned IP's for a Solaris 10/09 zone without giving it a physical NIC of its own?

• Am I going about this the wrong way? Should I instead have private IP's for the zones and let the host computer do NAT? Is that possible?

The end goal for me, is to have different network-connected services running in each their zone.

• There will be some network communication accross the zones, and

• they are also going to be running network accessible services which will be used by other hosts on our network.

pfexec zonecfg -z test
> 'create -b ;
> set zonepath=/zones/test ;
> add net ;
> set physica=bnx1 ;
> set address=10.0.6.92 ;
> end ;
> verify ;
> commit'