Nginx default_server not working

I'm running nginx 1.2.1 with a couple of services. They all use their on domain and listen on 443 only.

I want to lock out people who IP scan my server and don't know the domain. If there is no domain no site should reply. I'm not sure if that's possible because a domain is just a human readable IP?

I've tried a lot but did not get it working.

# Default server
server {
    listen [::]:443 ssl default_server;
    server_name _; # also tried to leave this out
    return 404;
}

If I enable this, none of my other services will reply. Regardless if this is the first or last site (alphabetically). They don't reply neither via IP nor via domain name.

What would be the correct way to reach this?

Edit: More Info

I've six sites defined, including the default catch all site which does not work:

etherpad, nagios, openmediavault-nginx, openmediavault-webgui, owncloud9, zzz-default. 

openmediavault-nginx contains only on site, a Piwik instance.

Each site has a server_name assigned. Two sites (nagios and webgui) share an internal server name but have different port numbers. All sites reachable from outsite have te same port (443) but different domain names.

I use dynDNS and CNAME to get my domains working.

When I enter https://<my ip>, I get my owncloud instance. There is a warning, that this domain is not trusted , but it's owncloud.

Here my owncloud listen line

listen [::]:443 ssl ipv6only=off deferred;

The rest of the config is mainly default as linked in the comments.

The question is: Why is owncloud responding even if there is no default or default_site option in it's config?

I've disabled everything except default-catch-all-site and owncloud.
Behavior is the same, if ddefault site is enabled, ownclud would not work.

Default site config is listed above, I added the ssl option.

Owncloud config is:

upstream php-handler {
 # server 127.0.0.1:9000;
  server unix:/var/run/php5-fpm-owncloud.sock;
}

server {
  listen [::]:443 ssl;
  server_name owncloud.mydomain.com
  ssl_certificate /etc/ssl/certs/openmediavault-7e8ef610-4ac4-4e47-9774-453fee8878bf.crt;
  ssl_certificate_key /etc/ssl/private/openmediavault-7e8ef610-4ac4-4e47-9774-453fee8878bf.key;

  # Add headers to serve security related headers
  add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
  add_header X-Content-Type-Options nosniff;
  add_header X-Frame-Options "SAMEORIGIN";
  add_header X-XSS-Protection "1; mode=block";
  add_header X-Robots-Tag none;
  add_header X-Download-Options noopen;
  add_header X-Permitted-Cross-Domain-Policies none;

  # Path to the root of your installation
  root /var/www/owncloud9/;
  # set max upload size
  client_max_body_size 10G;
  fastcgi_buffers 64 4K;

  # Disable gzip to avoid the removal of the ETag header
  gzip off;

  # Uncomment if your server is build with the ngx_pagespeed module
  # This module is currently not supported.
  #pagespeed off;

  #  security settings to reach A+
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_prefer_server_ciphers on;
  ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
  ssl_dhparam /etc/ssl/private/dhparams.pem;
  server_tokens off;

  index index.php;
  error_page 403 /core/templates/403.php;
  error_page 404 /core/templates/404.php;

  rewrite ^/.well-known/carddav /remote.php/dav/ permanent;
  rewrite ^/.well-known/caldav /remote.php/dav/ permanent;

  # The following 2 rules are only needed for the user_webfinger app.
  # Uncomment it if you're planning to use this app.
  #rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
  #rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json last;

  # Logging - important for fail2ban
 error_log /var/log/nginx/owncloud_error.log error;
  access_log /var/log/nginx/owncloud_access.log combined;


  location = /robots.txt {
    allow all;
    log_not_found off;
    access_log off;
  }

  location ~ ^/(build|tests|config|lib|3rdparty|templates|data)/ {
    deny all;
  }

  location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) {
    deny all;
  }

  location / {

    rewrite ^/remote/(.*) /remote.php last;

    rewrite ^(/core/doc/[^\/]+/)$ $1/index.html;

    try_files $uri $uri/ =404;
  }
location ~ \.php(?:$|/) {
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_param PATH_INFO $fastcgi_path_info;
    fastcgi_param HTTPS on;
    fastcgi_param modHeadersAvailable true; #Avoid sending the security headers twice
    fastcgi_pass php-handler;
    fastcgi_intercept_errors on;
  }

  # Adding the cache control header for js and css files
  # Make sure it is BELOW the location ~ \.php(?:$|/) { block
  location ~* \.(?:css|js)$ {
    add_header Cache-Control "public, max-age=7200";
    # Add headers to serve security related headers
    add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
    add_header X-Content-Type-Options nosniff;
    add_header X-Frame-Options "SAMEORIGIN";
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Robots-Tag none;
    add_header X-Download-Options noopen;
    add_header X-Permitted-Cross-Domain-Policies none;
    # Optional: Don't log access to assets
    access_log off;
  }