Varnish force cache everything

req.hash_always_miss should do the trick.

Don't do a full cache flush at the start of the spider run. Instead, just set the spider to work - and in your vcl_recv, set the spider's requests to always miss the cache lookup; they'll fetch a new copy from the backend.

acl spider {
  "127.0.0.1";
  /* or whereever the spider comes from */
}

sub vcl_recv {
  if (client.ip ~ spider) {
    set req.hash_always_miss = true;
  }
  /* ... and continue as normal with the rest of the config */
}

While that's happening and until the new response is in the cache, clients will continue to seamlessly get the older cache served to them (as long as it's still within its TTL).

The reason pass is occurring is because the default VCL logic refuses to cache anything with an Authorization or Cookie in the request headers, and your request has a Cookie.

if (req.http.Authorization || req.http.Cookie) {
    /* Not cacheable by default */
    return (pass);
}

That default behavior is out of an abundance of caution, so that servers where different content is returned for a page in response to the user's session; you can modify this behavior in your vcl_recv. Implement yours as a copy of the default, but strip out the req.http.Cookie check:

if (req.http.Authorization) {
    /* Not cacheable by default */
    return (pass);
}

In terms of the redirect behavior, sounds like it's getting a 30x response, which is redirecting the client browser - can you provide the logs for one of those hits?