I don't think that your examples are really equivalent. There are three distinct groups, each with it's own rationale for its behaviour.

1. Magic value is a good option when there is an "until" condition such as StreamReader.Read or when there is a simple to use value that will never be a valid answer (-1 for IndexOf).
2. Throw exception when the semantics of the function is that the caller is sure that it will work. In this case a non-existing key or a bad double format is truly exceptional.
3. Use an out parameter and return a bool if the semantics is to probe if the operation is possible or not.

The examples you provide are perfectly clear for cases 2 and 3. For the magic values, it can be argued if this is a good design decision or not in all cases.

The NaN returned by Math.Sqrt is a special case - it follows the floating point standard.

A strong type system is supposed to help you

Using a strongly typed language is meant to prevent errors during runtime by catching them at compile time. Using an anonymous object (or dictionary for example) defeats this purpose and allows bugs to be compiled into the program that will only surface during runtime.

As an example take a look at the code below:

public object GetUsers() {
    var users = MyRepo.GetAll<User>();
    return new {
        count: users.Count(),
        items: users
    }
}

public object GetNewUsers() {
    var users = MyRepo.GetNewest<User>();
    return new {
        Count: users.Count(),
        items: users
    }
}

The first response uses a lowercase count parameter where the second response contains an uppercase Count parameter. In a class where these methods may be separated by a few blocks of code it is very easy to miss this typo. Even worse this will happily compile and run. If we were to use a typed DTO instead this situation would not be possible (in fact it wouldn't even compile).

Or how about this example:

public object GetSystemInstallDate(){
    var date = AppHost.GetInstallDate();
    return new { 
        date
    };
}

What happens if we refactor to change the GetInstallDate method to return a DateTimeOffset instead of a DateTime object? The code will happily compile but the output to the client will have changed. This is a weak example but there's many other instances were things like this can happen. Numerical types, refactoring enums, etc...

Refactoring

If your anonymous response is used in more than one location you won't be able to refactor using the fancy tools like resharper. You will have to manually refactor which is never a fun task. If it is used in a single location the refactoring is easier but still more work than defining the class in the first place.

DRY

I would argue that by using anonymous types you are not protecting yourself against violating DRY; rather, you are making yourself more likely to violate it. An anonymous type can only be used in one place; however, if what that type represents is ever required elsewhere the entire type will have to be recreated.

By using a strongly typed object you only have to define the type in a single place and you then use that type where it is needed. This prevents you from having to repeat writing out all the property names, formatting values, etc...

public object GetTop10Movies() {
    return new { lastmodified = DateTime.Now, items }
}

public object GetNewest10Movies() {
    // We now have to repeat the previous anonymous object
}

YAGNI

This is a guideline and not a rule. If it was a rule it would boil down to "ignore the future" which is always a recipe for disaster. At the same time trying to cover all possible future scenarios is a horrible idea and will lead to overly abstract and complicated codebase. The trick it to strike the right balance between thinking ahead and solving the problems at hand.

Another way of thinking about YAGNI in this situation is: Do you need compile time type safety?

I would argue that the work involved in creating a class to represent your response is only slightly more than the work involved in typing out an anonymous object. The complexity difference is essentially none (in fact with the C# world an anonymous object would most likely be considered a more complex route to take).

For a rather minimal time investment you get quite a few benefits that are real and usable to the developer. They may not be needed in the sense that without them the application won't function, but if you didn't need a strongly typed language I would question the use of C# in the first place.

Maintenance

The team convention was to return only DTOs and the DTOs were managed in a completely different project (among about 50 projects in the solution), I felt that in this case perpetuating that pattern was excessive and difficult to maintain.

I'm not sure where the difficulty in maintaining this pattern would come from except for the idea of having 50 projects all bundled into a single solution. You will have one of two situations:

1. A new response type is needed.

Add a new file to the DTO project representing your type and then use it in your application (a reference should already exist between the projects).

2. An existing response type is needed.

Start typing the namespace of the DTO assembly and select the type that you need.

I suppose a case can be made for DTOs as specification documentation, but when it becomes more of a maintenance chore to declare them and to look them up in their special folder in a project I would argue that there are more appropriate places and means to document the interface; indeed the anonymous object declaration seems to be documentation enough.

There's no better place to document your code than to have your code document itself. The name of your response DTO should describe what it is. Your anonymous declaration says nothing about what it represents, it only tells you what it contains. Having to look up external documentation (ala a website or help file) is even more cumbersome.

A box that consists of two bolts, two washers, and a nut doesn't describe what it is for at all. A box with the sticker "monitor wall mount" lets you know exactly what it is simply by looking at the packaging.

new { id, model }
// What is this? A vehicle? An economic model? Representation of DNA?

new Vehicle { id = id, model = model }
// Oh! A vehicle!