This is a very broad question but I will try to give you an answer.
...there is some ambiguity with regard to the use of domain models and application services
There is no ambiguity if you design well your bounded contexts, the domain models and the relationships between them.
However, what happens if there are multiple database hits that depend on business logic?
In DDD, all the operations go through the Aggregate root (AR). The application services load the ARs from the persistence, send commands to them, then persist those ARs back. ARs don't need to hit the database at all. In fact a good designed AR does not even know that databases exists at all. All they touch/see/smell is their internal state and the immutable arguments that they receive in their command methods. If an AR needs something from the database then the Application service pass that thing as argument.
ARs should be pure, side effects free objects/functions. One reason is that commands applied on them must be retry-able, in case of concurrent modifications.
As an example: ARs don't send emails, they return a value object that holds the email data (from, to, subject and body) and the Application service takes that value object and passes it to a infrastructure service that does the actual sending of the email.
For example, I have a domain model "Order" that has a method "IsValid". To determine whether an order is valid, a database read must be performed.
You don't need an isValid method, as an Order cannot get into an invalid state anyway because any modifications are done through it's methods. If you are referring to the existence of an Order then this kind of validation is done by the Application service: if it does not find the Order in the persistence then it does not exist, as simple as that. Maybe you are referring to the ShoppingCart as being valid, not the Order. What about then? Well, you could try to create an Order from a ShoppingCart and if you succeed then a Cart is ready to be ordered. As the Order is side effect free, no order will actually be created. Just an example of how you might think in DDD.
DDD seems like it provides a lot of nice value, but I'm worried about these kinds of issues leading to design "rot".
If you follow the DDD approach well, your design will never rot. Never.
As a foot note: In the beginning I was a little miss-leaded by the Layered architecture. Don't make the same mistake. Take a look at some newer architectures like CQRS that fits very well with DDD.
Best Answer
Application. The magic search term you want is anti corruption layer
Typically, the message received by your application will be some flavor of DTO. Your anti corruption layer will typically create value types that the domain will recognize. The actual command dispatched to the domain model will be expressed in terms of validated value types.
Example: a DepositMoney command would likely include an amount, and a currency type. The DTO representation would probably express the amount as an integer, and the currency code as a string. The anti corruption layer would convert the DTO into a Deposit value type, which would include a validated Amount (which must be non negative) and a validated CurrencyCode (which must be one of the supported codes in the domain).
Having successfully parsed the command into types the domain model understands, the command is executed in the domain, which may still reject the command on the grounds that it would violate the business invariant (the account doesn't exist yet, the account is blocked, this particular account isn't allowed to use that currency? etc).
In other words, the business validation is going to happen in the domain model, after the anti corruption layer validates the inputs.
The implementation of the validation rules will normally live either in the constructor of the value type, or within the factory method used to construct the value type. Basically, you restrict the construction of the objects so that they are guaranteed to be valid, isolating the logic in one place, and invoking it at the process boundaries.