It looks like you took some fancy items from agile development, put them to waterfall process and now you call it agile.
The product is developed for a customer who will re-sell it while
paying us royalty.
This is OK.
The team does not get to talk directly to the end user. Only to
the reseller.
This is OK. Product owner talk to reseller and collects requirements.
A product requirements document was created before starting
development.
This is not OK. I haven't seen the project where definite requirement set can be defined upfront. Change your product requirements document to product vision (short) with some initial set of requirements which are subject to change.
The requirements are rigid and do not change.
This is not OK and you will see in the future that it is also not true.
A delivery schedule was agreed on with milestones such as "alpha",
"beta" etc. and features/times attached to those milestones.
This is not OK. The real schedule will be visible from the team progress. You can make general milestones but assigning exact set of features which will be implemented in these milestones is not agile. This can change during development.
All developers on the Scrum team report to the product owner, a
software manager.
This is not OK. I would not say that developers report to product owner. Scrum process keeps visibility of the process but developers do not report anything except regular meetings. It is responsibility of product owner to be in contact with a team and as active participant see the progress himself.
Testers on the team report to a QA manager.
This is not OK. Testers should be part of development team because user story is not done until it is tested (there should be automated test to validate acceptance criteria). There can be separate QA but it is additional level of complex testing and it is usually done on customer side (but doesn't have to be) to validate that SW does what customer expects and the feedback is collected as new backlog items or bugs to existing completed backlog items.
Separating complete QA outside of development team leads to breaking the whole purpose of definition of done. Some QA must be part of the team and that part is not related to any QA manager - that part is doing commitment with development team.
The product owner has directed the team towards certain high risk
technical tasks. The output of those tasks is not usable by the end
user but rather some technology/code that will eventually be used in
the product.
This happens in every project but it should be part of some product backlog item targeting end user. It can be included directly in backlog item implemented in current iteration or it can be included as a spike (proof-of-concept) to clarify complexity of some backlog item which should be implemented in the future.
The product owner has created a backlog based on the requirements.
This is a must.
The product owner is unable to answer some questions regarding the
product. He refers to others or to the documented requirements.
This is not OK. It is job of the product owner to know answers. He has a responsibility and he must do decisions. If he doesn't know answer he must find it asap.
The team goes through the motions of Scrum. Daily Scrum, Sprint
Planning, Retrospective etc. There is a ScrumMaster.
This is OK but it doesn't mean that team is doing Scrum.
Every sprint the product owner and management decide what backlog
items the team works on.
This is definitely not OK. The product owner and management can make priorities but commitment (selection of most prioritized items) is teams responsibility.
There is a burndown chart. Scrum board with stories and tasks. The
estimates on those come from the team.
This is OK.
The team sits in an open floor "bull pen" shared with other teams,
all visible and audible. There is cross-team noise and there is foot
traffic around the team area.
It is Scrum master's responsibility to make end of this if team feels like it reduces their productivity.
The team may be required to attend various meetings not directly
related to the goals of the sprint.
It is OK, the time wasted on these meetings will result in smaller commitment (team will do less real work). It is up to Scrum master / management to reduce these meetings to increase team's velocity.
There are pressures to select certain technical solutions. Some
tools and processes are mandated.
This is partially OK. There can be non-functional requirements for tools and architecture and there can be defined processes but still final implementation is up to the team.
As Robert noted in a comment, I really don't think you can fully encompass all your design constraints in one place. Since you mention that you're intending to document your design constraints to assist with your design decisions, I'm assuming you are working in a waterfall-style development environment. See the Criticism section on Wikipedia regarding waterfall development - while waterfall development may be justified in some cases, there's a reason why agile development is catching on. Since you are free to start developing a product in small increments, as soon as you find that something doesn't work, you throw out that piece and then try a different approach. It's also much more reactive to changing requirements over the course of a project.
Waterfall development often wastes lots of administrative time up front - how many of us have had to deal with a requirements document that is many dozens of pages long, only for it to be essentially useless towards the end of the project, since the requirements have changed over the course of its development?
So far, I haven't directly answer your question regarding how to document a design constraint; rather, I'm suggesting that you don't do it all up front. You don't describe what your development environment is like (meaning, if it's in a large company, you're self-employeed, etc.), but if you can change the expectations of what needs to be delivered up front before any development is done, I think that will help you enormously. If I can twist your question into, "What should I document up front?", then I would suggest you look into formulating user stories instead, which should better document the true needs that should be known up front.
(By the way, if you are working on a solo or small team, you would also benefit from reading up on lean development, which emerged from the Agile movement, and further focuses on removing waste from the development process).
Edit, based on Nicholas' comment:
It sounds like you are probably in a tough spot. The above links in my answer cover each of those areas that you're not familiar with (jump to the top of the Criticism link to see the full scoop on Waterfall development - if you're not familiar with Waterfall versus Agile development, it's almost certain you are doing the former).
Do you have a feeling for how flexible your management's expectations are for what a development project entails? If you feel comfortable bringing the idea up to them, I think you would really benefit from a lean technique, though admittedly, without having a person experienced in lean development there to guide you along, you might find it daunting at first.
If, by chance, you have a Pluralsight subscription (or don't mind signing up for the free trial), their course called "Best Practices for Software Startups" covers the principles behind lean development and would give you arguments to bring to your management as to why you shouldn't try to plan all these design constraints right from the start of a project. (Alternatively, there are a number of presentations you can watch on YouTube).
Best Answer
I would say no. Considering that the PO should focus on defining business value. Having the PO creating prototypes kind of influences the teams development in technical terms. I think it is important that the business value is well defined, so that the team can implement it. Also, the act of creating a prototype is a learning experience, to figure out how to implement the business value, maybe even what the business value is. It would be a missed opportunity for the team not to do the prototyping. The interface between the team and the PO is the user story and the defined business value. Having the PO do the prototyping kind of softens that up.
This sounds to me like the PO used to be a developer, I've had that experience.
It is my believe as a scrum master, that the PO should NOT develop. Focusing on defining business value and maintaining a backlog is the product owners way of communicating with the team. (of course verbally all the time too, after all we are using agile principles)