EX3300: Fixing Voice VLAN Issues After Upgrade to 15.1R6.7

juniperjuniper-exjuniper-junosvlanvoip

We have 50+ Juniper EX3300 switches with access points configured to work with dot1x and voice vlan.

We also have a variety of Yealink T23G, T42G, T26G and T48G phones that worked perfectly in our configuration when the switches were operating on JunOS version 12.3R6.6.

After upgrade to 15.1R6.7, the voice vlan function degraded. While the ports are still being served the correct tagged vlan, phones are no longer attempting to communicate over it and are not able to get a DHCP reply for the address.

We are in no luck so far in finding a solution to this problem, or even a fair diagnosis.

Configuration

interfaces {
    interface-range access-dot1x {
        member "ge-0/0/[0-47]";
        member "ge-1/0/[0-47]";
        member "ge-2/0/[0-47]";
        description access-dot1x;
        unit 0 {
            family ethernet-switching;
        }
    }
}
protocols {
    dot1x {
        authenticator {
            authentication-profile-name DOT1X-NPS;
            interface {
                access-dot1x {
                    supplicant multiple;
                    retries 2;
                    quiet-period 2;
                    transmit-period 2;
                    mac-radius;
                    reauthentication 32400;
                    supplicant-timeout 2;
                    server-timeout 10;
                    maximum-requests 2;
                    guest-vlan ZEROCONFIG;
                    server-reject-vlan ZEROCONFIG;
                    server-fail use-cache;
                }
            }
        }
    }
    lldp {
        interface all;
    }
    lldp-med {
        interface all;
    }
}
ethernet-switching-options {
    voip {
        interface access-dot1x {
            vlan DEV-VOIP;
            forwarding-class assured-forwarding;
        }
    }
}
ZEROCONFIG {
    vlan-id 2;
}
DEV-VOIP {
    vlan-id 14;
}

Diagnostics

After discovering the issue we have built an additional test environment to test two firmware versions with all kinds of Yealink phones. We have no phones from other vendors, but within this vendor, all phones were affected, including a variety of firmware versions.

Simmilarities:

Both versions return identical output on following commands:

> show ethernet-switching interface ge-0/0/6
Interface    State  VLAN members        Tag   Tagging  Blocking
ge-0/0/6.0   up     ADM-CSI             130   untagged unblocked
                    ZEROCONFIG          2     untagged unblocked
                    default                   untagged unblocked
                    DEV-VOIP            14    tagged   unblocked

Also with this command, but only regarding the interface list:

> show lldp detail
Interface      Parent Interface    Vlan-id     Vlan-name
ge-0/0/6.0     -                   130         ADM-CSI
ge-0/0/6.0     -                   14          DEV-VOIP
ge-0/0/6.0     -                   2           ZEROCONFIG

Differences:

A proper connection on 12.3R6.6 results in a following output:

> show ethernet-switching table interface ge-0/0/6
Ethernet-switching table: 3 unicast entries
  VLAN              MAC address       Type         Age Interfaces
  ADM-CSI           *                 Flood          - All-members
  ADM-CSI           98:5a:eb:xx:xx:4d Learn          0 ge-0/0/6.0   # computer connected via phone, dot1x assigned VLAN
  DEV-VOIP          *                 Flood          - All-members
  DEV-VOIP          00:15:65:yy:yy:0d Learn          0 ge-0/0/6.0   # phone after voice-vlan was assigned
  ZEROCONFIG        *                 Flood          - All-members
  ZEROCONFIG        00:15:65:yy:yy:0d Learn         50 ge-0/0/6.0   # phone before voice-vlan was assigned
  default           *                 Flood          - All-members


> show lldp neighbors
Local Interface    Parent Interface    Chassis Id          Port info          System Name
ge-0/0/6.0         -                   10.14.100.2         WAN PORT           SIP-T42G

A failed connection on 15.1R6.7 results in a following output:

> show ethernet-switching table interface ge-0/0/6
Ethernet-switching table: 3 unicast entries
  VLAN              MAC address       Type         Age Interfaces
  ADM-CSI           *                 Flood          - All-members
  ADM-CSI           98:5a:eb:xx:xx:4d Learn          0 ge-0/0/6.0   # computer connected via phone, dot1x assigned VLAN
  DEV-VOIP          *                 Flood          - All-members
  ZEROCONFIG        *                 Flood          - All-members
  ZEROCONFIG        00:15:65:yy:yy:0d Learn         50 ge-0/0/6.0   # phone before voice-vlan was assigned
  default           *                 Flood          - All-members

> show lldp neighbors
Local Interface    Parent Interface    Chassis Id          Port info          System Name
ge-0/0/6.0         -                   0.0.0.0             WAN PORT           SIP-T42G

There is however a difference in notification interval (lldp-configuration-notification-interval) value:

on 12.3R6.6 the value (default) is 0s, configurable in 0..3600 range,
on 15.1R6.7 the value (default) is 5s, configurable in 5..3600 range.

I was hoping this difference in notification interval will be the cause of our problem, yet setting the value to 5 on earlier switch did not cause it to degrade the functionality.

Yet another, and maybe the most significant difference appears in the LLDP and EAPOL packages exchanged by the party.

This begins with following differences in the first package send from device to phone:

Proper session (12.3R6.6):

14:13:50.002710 Out LLDP, length 266
    [...]
    System Description TLV (6), length 89
      Juniper Networks, Inc. ex3300-48t , version 12.3R6.6 Build date: 2014-03-13 07:02:54 UTC
    [...]
    Organization specific TLV (127), length 9: OUI IEEE 802.3 Private (0x00120f)
      MAC/PHY configuration/status Subtype (1)
        autonegotiation [supported, enabled] (0x03)
        # ==== HERE =====
        PMD autoneg capability [Sym PAUSE for fdx] (0x0400) 
        MAU type Unknown (0x0000)
    [...]
    Organization specific TLV (127), length 6: OUI Ethernet bridged (0x0080c2)
      Port Vlan-ID Subtype (1)
        # ==== HERE =====
        Vlan Id: 14
    [...]
    Organization specific TLV (127), length 15: OUI Ethernet bridged (0x0080c2)
      Vlan Name Subtype (3)
        Vlan Id: 14
        Vlan Name: DEV-VOIP
    Organization specific TLV (127), length 14: OUI Ethernet bridged (0x0080c2)
      Vlan Name Subtype (3)
        Vlan Id: 0
        Vlan Name: default
    Organization specific TLV (127), length 12: OUI Ethernet bridged (0x0080c2)
      Vlan Name Subtype (3)
        Vlan Id: 14
        Vlan Name: voice
    [...]
    Organization specific TLV (127), length 8: OUI ANSI/TIA (0x0012bb)
      Network policy Subtype (2)
        Application type [voice] (0x01), Flags [Tagged]
        Vlan id 14, L2 priority 0, DSCP value 0
    End TLV (0), length 0

Failed session (15.1R6.7):

14:04:52.612890 Out LLDP, length 333
    [...]
    System Description TLV (6), length 156
      Juniper Networks, Inc. ex3300-48t Ethernet Switch, kernel JUNOS 15.1R6.7, Build date: 2017-04-23 00:39:39 UTC Copyright (c) 1996-2017 Juniper Networks, Inc.
    [...]
    Organization specific TLV (127), length 9: OUI IEEE 802.3 Private (0x00120f)
      MAC/PHY configuration/status Subtype (1)
        autonegotiation [supported, enabled] (0x03)
        # ==== HERE =====
        PMD autoneg capability [1000BASE-T fdx] (0x0001)
        MAU type Unknown (0x0000)
    [...]   
    Organization specific TLV (127), length 6: OUI Ethernet bridged (0x0080c2)
      Port Vlan-ID Subtype (1)
        # ==== HERE =====
        Vlan Id: 0
    [...]
    Organization specific TLV (127), length 15: OUI Ethernet bridged (0x0080c2)
      Vlan Name Subtype (3)
        Vlan Id: 14 
        Vlan Name: DEV-VOIP
    Organization specific TLV (127), length 14: OUI Ethernet bridged (0x0080c2)
      Vlan Name Subtype (3)
        Vlan Id: 0
        Vlan Name: default
    Organization specific TLV (127), length 12: OUI Ethernet bridged (0x0080c2)
      Vlan Name Subtype (3)
        Vlan Id: 14
        Vlan Name: voice
    [...]
    Organization specific TLV (127), length 8: OUI ANSI/TIA (0x0012bb)
      Network policy Subtype (2)
        Application type [voice] (0x01), Flags [Tagged]
        Vlan id 14, L2 priority 0, DSCP value 0
    End TLV (0), length 0

Any ideas?

Best Answer

Hinted by our reseller, I've found this article hidden quite deep in Juniper docs, which states that in versions above 14.1X53-D40 / 15.1R4, due to security considerations, switch behavior has changed.

With older firmware, switches would allow access to voice VLAN regardless of 802.1x authentication status on the port.
In new firmware, switches will block access to voice VLAN:
- if the 802.1x authentication server fails (this behavior can be overriden with server-fail-voip statement in dot1x profile), or:
- if the 802.1x authentication server rejects the authentication (this behavior cannot be overriden).

As of now, there seems to be no opt-out for the latter case, but I will update this answer if this ever changes.

There seems to be no feasible workaround for the problem, other than perhaps authenticating the unknown devices to guest VLAN instead of rejecting them. This might be done either by:

populating authentication database with phone MAC addresses (with mac-radius),
configuring phone credentials for 802.1x authentication, or
creating a tightly scoped finishing RADIUS rule that would accept all unmatched devices without authentication and assign them to GUEST VLAN.

Relevant fragment of Juniper article:

Server fail fallback is supported for voice traffic starting in Release 14.1X53-D40 and Release 15.1R4. To configure server fail fallback actions for VoIP clients sending voice traffic, use the server-fail-voip statement. For all data traffic, use the server-fail statement. The switch determines the fallback method to use based on the type of traffic sent by the client. Untagged data frames are subject to the action configured with server-fail, even if they are sent by a VoIP client. Tagged VoIP VLAN frames are subject to the action configured with server-fail-voip. If server-fail-voip is not configured, the voice traffic is dropped.

Note: Server reject fallback is not supported for VoIP VLAN tagged traffic. If a VoIP client starts authentication by sending untagged data traffic to a VLAN while server reject fallback is in effect, the VoIP client is allowed to access the fallback VLAN. If the same client subsequently sends tagged voice traffic, the voice traffic is dropped. If a VoIP client starts authentication by sending tagged voice traffic while server reject fallback is in effect, the VoIP client is denied access to the fallback VLAN.

Related Solutions

Cisco – Voice VLAN information

First, a caveat: I'm not sure why you specified the following (emphasis mine),

Any suggestions for commands to specifically show the voice vlan assigned to a switch port without using too much reg-ex or seeking through the running configuration?

However, even the lengthiest reg-ex command in a Cisco device can be shortened by the alias command. In fact one of my frequently used aliases is to show the exact information you're looking for. Which I will include below.

Now, there are several ways to get the information you're looking for, and it depends on what exactly you know, and what you're trying to find.

If you know what the voice VLAN(s) is/are on that particular switch, and you're looking to find out what ports they're assigned to, you could simply issue the command:

show vlan id <voice-vlan-number>

This would give you a list of all ports utilizing that VLAN:

ATR4506-A1A-1#show vlan id 210 

VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
210  ATRIUM-IP-PHONES                 active    Gi2/2, Gi2/3, Gi2/4, Gi2/5, Gi2/6,

If you know the port of interest (or want to see all ports) and just want to see which voice VLAN is in use on that port, you are looking for something like the following:

show interfaces switchport | include Name|Voice

I have this command aliased to svv (for show voice vlan) like so:

conf t
alias exec svv show interfaces switchport | include Name|Voice

This is the command I most frequently use to gather this information, and it gives the output:

ATR4506-A1A-1#svv                                      
Name: Te1/1
Voice VLAN: none
Name: Te1/2
Voice VLAN: none
Name: Gi2/2
Voice VLAN: 210 (ATRIUM-IP-PHONES)
Name: Gi2/3
Voice VLAN: 210 (ATRIUM-IP-PHONES)
Name: Gi2/4
Voice VLAN: 210 (ATRIUM-IP-PHONES)
Name: Gi2/5
Voice VLAN: 210 (ATRIUM-IP-PHONES)

Another alternative would be to filter the show run output if you need the exact interface names and switchport info (for scripting purposes for example):

show running-config | include interface GigabitEthernet|switchport voice vlan

This gives:

ATR4506-A1A-1#show running-config | include interface GigabitEthernet|switchport voice vlan 
interface GigabitEthernet1/3
interface GigabitEthernet1/4
interface GigabitEthernet1/5
interface GigabitEthernet1/6
interface GigabitEthernet2/1
interface GigabitEthernet2/2
 switchport voice vlan 210
interface GigabitEthernet2/3
 switchport voice vlan 210

Juniper – Troubleshooting Unstable Uplink on SRX240 with VPN

With help from our service provider we've been able to localise the problem. The idea I had in my last comment proved correct: the difference between fast ethernet and gigabit ethernet was the source of the problem.

The reason for dropping connections, undeliverable packets and segmented packets was a mismatch in link mode between the fiber switch and the SRX - they hardcoded their port to 100m full duplex, but the SRX had automatic negotiation, which resolved to 100m half duplex.

The problem was further complicated by Juniper's odd configuration of speed and link mode. Just setting these configuration options wasn't enough:

speed 100m;
link-mode full-duplex;

This configuration was accepted, but when running show interfaces ge-0/0/0, it still showed:

Link-level type: Ethernet, MTU: 1514, Link-mode: Half-duplex, Speed: 100mbps,

Apparently, you have to explicitly disable auto-negotiation:

speed 100m;
link-mode full-duplex;
gigether-options {
    no-auto-negotiation;
}

In the end, I found that this guy had the exact same problem (and the solution!)

Thanks for all your help.