Is it true to say that the product manager and the QA should be responsible for taking the user-story and breaking it down to acceptance tests?
Mostly. They may not actually write the actual acceptance test. They may approve something you wrote. But they approve the acceptance tests. Yes.
the acceptance tests need to be formal, so they can be used as tests, but also human readable so that the product can approve they are the requirements, right?
Irrelevant. They may be formalized as automated tests. Or they may be informal and it may be your job to create automated tests from the informal acceptance test criteria.
Also. The "requirements" are the user story. There's no real need to create yet another version of the story called "requirements." Some folks to like to elaborate on the story before they code. You can call this requirements, but "design" is a better word. "Elaboration" is the best word.
Is it also true that I later take these acceptance tests and use them as my requirements, i.e. they are a set of use-cases which I implement (through TDD)?
Yes. The story leads to acceptance tests. The story is required behavior (i.e., "requirements"). The story leads to tests which drive software design and development.
that's the current flow I have in mind right now.
There isn't really a lot of "flow" to this.
Story -> acceptance tests.
Story -> elaboration ("design", "requirements") -> unit tests -> code.
Story -> User being able to do something of value.
Story -> Story points -> velocity calculation.
Note the pattern. The story largely drives everything.
When, by whom and to which format are the requirements compiled?
First. Define "requirements". How are the different from the story itself?
What I had in mind was that the product and QA define the requirements via acceptance tests
Not usually.
during the iteration then the developer might get stuck waiting for them.
Incorrect. The developer can (and often does) help write these. That's the point of "development": elaborate the story to a well-defined "done".
Again. When you have doubts or questions, you must actually read the Agile Manifesto. The Manifesto is quite clear: developers must talk with product owners, users, QA, and everyone else who is a stakeholder. The interaction is actually the most important thing that can happen.
I suspect your question will be closed for being too broad which is a shame as any developer who has been round the track a few times will have encountered this a lot.
But just to share my experiences - there are a number of aspects to this:
Unit tests
Paying down debt will no doubt involve refactoring legacy code. This is all very well, but you will soon become the focus of unwelcome attention if you start breaking things. A good unit test suite should be in place anyway, but if it isn't, get started on it, as these can give you an early warning that your refactorings are going awry.
Approaching legacy code
This is a topic that people literally write books about - the most famous of which can be found summarised here.
The nature of the debt
Not all debt is the same - how your tackle it depends on the nature of the debt. If it is simply an obsolete technology then it could simply be replaced rather than refactored. Hard to give much more advice without knowing specifics...
Getting buy-in
I've worked in places where management are all about features and bugs and simply won't entertain any time being devoted to "polishing". This of course is very short sighted. If you don't get buy-in to fix the problems, you're simply going to have to attend any problems while you're in the code doing other things.
Understand also, that it isn't just management buy-in, you need the buy-in of your peers too. They will have different ideas about how the various problems should be solved. Involve everyone and you'll find the whole process is a lot easier.
What are your rituals?
Some say you should never have technical debt. This is a pretty naive stance if I may say so. Perhaps what they mean is that your debt should be as low as feasibly possible (like household debt).
That being said, how much debt you have is a function of how you view it collectively. If it is always at the front of your mind to pay down debt wherever you find it, you'll find it coming down quite steeply. If you're looking to fix it in your free time (whenever that is), it is a long hard road - not least as you could simply accrue more debt in the time it took to pay another debt.
As for tracking it in the first place, it should be recorded it your usual work tracking tool. How you separate this out from the rest of the work will vary from tool to tool.
Best Answer
The roadblock you're running into is there by design, and you're running into it for the correct reasons.
User stories are things the user cares about. That's why it's written from the point of view of the user. If you can't express your task as a user concern, then it likely isn't a user concern, which is likely the case here. As far as the user can see, before and after the technical debt task, the same functionality will be maintained, so the user doesn't care what happens behind the screen.
Technical debt is a developer concern. It's something you care about, but which doesn't really add functionality that the application otherwise wouldn't have.
User stories, and the enforcement of describing tasks as such, forces the workload to be user oriented, which is where the real business value lies for most companies (functionality/features leads to user happiness, which leads to customer acquisition). In this system, technical upgrades can only exist if they allow for adding additional features/business value.
And this is where the crux of the issue lies: what is the business value of your proposed technical upgrade?
But the biggest bullet point, for which this roadblock was almost explicitly created, is developers doing something for themselves, e.g. wanting to play with the newest toys. While you can argue that there are indirect benefits (developer morale, keeping developers up to date, ...), there is usually no real added business value. If there were, you'd be able to easily express the business value as a user story.
In short, your inability to describe the change in function of its business value (= user story) suggests that there is no added business value. In a user story driven environment, this is the exact scenario that is intentionally being blocked to not waste development resources. Asking how to phrase it as a user story would be telling you how to circumvent the intentional checks and balances that your company chooses to operate under, which is not a good way to approach this.
If you need to circumvent the usual precedings (user stories), you need to talk to the project management about the upgrade and its indirect benefits that are not related to business value.
Note that this isn't exclusive to user stories. This format can be used to express developer concerns as well, e.g.
If this format is the only requirement, you don't actually need to use user stories for this. But if your company restricts this to user-driven business value, then your technical debt is sadly ignored.
I'm not a fan of companies who don't look at developer concerns and are only interested in user concerns; but as a developer I'm not the one who makes the executive decisions and if that's what the company chooses to do, I can't overrule that.