Prevent Empty Subelement List Error in Ansible Loop

When I try to loop (with Ansible 2.6 loop) with subelements through all public keys of a list of users and encounter a user which has no public keys defined:

- authorized_key:
    user: "{{ item.0.username }}"
    state: present
    key: "{{ item.1.pub_key }}"
  loop: "{{ users | subelements('ssh_pub_keys') | default ([]) }}"
  loop_control:
    label: "{{ item.username }}"

I'm getting the following error: the key 'ssh_pub_keys' should point to a list, got None

When I try to use skip_missing like this:

- authorized_key:
    user: "{{ item.0.username }}"
    state: present
    key: "{{ item.1.pub_key }}"
    loop: "{{ lookup('subelements', users, 'ssh_pub_keys', {'skip_missing': True})}}"
  loop_control:
    label: "{{ item.username }}"

I get this error: 'list object' has no attribute 'username'

This could be the users list:

users:
  - username: usera
    ssh_pub_keys:
      - from: home
        pub_key: kdzadizajdiazjd
      - from: work
        pub_key: dzadadazdzadzad
  - username: userb
    ssh_pub_keys:
      - from: home
        pub_key: kdzadizajdiazjd
      - from: work
        pub_key: dzadadazdzadzad
  - username: userc
  - username: userd
    ssh_pub_keys:
      - from: home
        pub_key: kdzadizajdiazjd
      - from: work
        pub_key: dzadadazdzadzad

How can I make the loop with subelements to go to the next user without throwing an error when a user is encountered which has no ssh_pub_keys list?