It turns out this was due to an internal rewrite by mod_rewrite, due to running this in the context of a Symfony2 app, which rewrites all requests to its app.php controller. The same thing would happen in a CMS like wordpress that rewrites everything to index.php, or in other url prettification schemes that rewrite urls internally. (Note, this isn't an issue with browser redirects, since those result in an entire new request, just with internal rewrites.) Basically after your last rewrite finishes, the apache htaccess and configuration logic runs through again, and its on that run that your final environment variables and such are set. So if a variable was set prior to a rewrite, it won't be available afterward (like when I'm trying to set the header in the question here).
Even though these two lines in the question are adjacent...
SecAction "pass,setenv:TESTPAGE=1,nolog,id:10001001"
Header always set X-Debug "IsTest" env=TESTPAGE
The first line is running during the request body phase (modsec Phase 2) as per defaults since I haven't specified a phase. The rewrite to app.php (due to a .htaccess rule not shown) occurs afterward. And so when the Header is set the variable is no longer present.
However, all the pre-rewrite variables are still available with a 'redirect_' prefix. So to fix this, I need to write it like this:
SecAction "pass,setenv:TESTPAGE=1,nolog,id:10001001"
Header always set X-Debug "IsTest" env=REDIRECT_TESTPAGE
Or if I'm not sure whether the request will be redirected I could use both versions:
SecAction "pass,setenv:TESTPAGE=1,nolog,id:10001001"
Header always set X-Debug "IsTest" env=TESTPAGE
Header always set X-Debug "IsTest" env=REDIRECT_TESTPAGE
(There may be a way to combine those; not sure if you can use OR logic in apache 'env' statements; please comment if so!)
Edit: Regarding 'deny' no longer working, that was also due to redirects in a way. Mod Security was throwing a 500 error by default, but the default 500 error page, 500.shtml did not exist. Therefore the framework was intercepting that request and sending it back to app.php, which then looked at the request URL and proceeded to load the originally requested page despite the error. If the configured ErrorDocument exists, deny works properly and that document is shown.
Best Answer
Have you tried the combination of the two?:
Note you can different default actions per phase like above.
Note also that some rule sets (e.g. OWASP CRS) also set these default actions, and Kay also need then set a certain way if using anomaly scoring or immediate blocking.