How to deploy an exploded web app through WebLogic 11g administrative console

invalidateAll is documented as part of WLS 10.3 (aka 11g) so it should work in your logout servlet. Have you tried it and faced any error?

It also has a logout servlet that calls session.invalidate(). After calling this, i can't access protected pages in the main application, but can access protected pages in the other applications

This indicates to me, that the sessions are not shared across web applications, which is the default behaviour. If the sessions were shared the user would be logged out of all the applications. For the sessions to be shared, you would have to set sharing-enabled to true

As the docs state,

By default, Web applications do not share the same session. If you would like Web applications to share the same session, you can configure the session descriptor at the application level in the weblogic-application.xml deployment descriptor. To enable Web applications to share the same session, set the sharing-enabled attribute in the session descriptor to true in the weblogic-application.xml deployment descriptor.

This error occurs either because your project is structured incorrectly else because your deployment descriptor is incorrect/weblogic incompatible. Check your web.xml - the most probable cause.