You've outlined some requirements which are typically catered for by workflow engines, such as following some process, task allocation, escalations and notifications. From this point of view, it would make a lot of sense to go with an actual workflow engine. There are two areas which make me question whether it will actually prove to be beneficial to use one in your case:
- Workflow engines can involve a significant investment of your time to learn. They also tend to complicate the environment in which your application runs. If your requirement is a "once-off", and you are unlikely to implement similar workflow-related features in the future, you might want to reconsider using one. E.g. I wouldn't use a full-blown document management system if all I wanted to do was upload an attachment on one screen, and something similar applies here. I would normally advise against "rolling your own", but in this case I'm not so hesitant; I have been faced with your exact requirements in the past, and chosen to write my own (due to insane deadlines, don't ask) - it was a resounding success.
- Your requirement about the user modifying the process, I have several issues with. Firstly, it's the kind of wishy-washy requirement that will grow in time. "Oh, we forgot, the user needs to specify who is going to action that extra step. Oh yeah, it can be someone outside of the company. Oh yeah, they need to be able to specify that at least 2 of the 5 targeted managers must approve", etc. Also, workflows can be complex beasts, and it's usually difficult to allow an actual user to modify them without breaking them, and to do so in a way which is quick and intuitive to the user. Beyond that, many workflow engines will not easily allow the dynamic modification of the process, so you may be limited to some kind of fixed logic like: "repeat this generic step until we run out of user-defined steps". My experience lies outside of open-source engines, but this would have been a difficult requirement to implement in the 3 or 4 proprietary ones I've worked with over the years.
To summarise, with the exception of the user-modifiability, your requirements are a pretty good match for a workflow engine. My suggestion: figure out whether there's an engine out there which gives you everything that you need (perhaps someone else can suggest specifics), and do some R&D with it. You will then be in a better position to weigh up your options.
EDIT: Sorry, my connection went down as I made that post, and seems to have been truncated. I've recovered it now.
Let the developer who went for a couple of months without merging fix it. Maybe they can get one big chunk of code to merge, maybe they can get a bunch of little chunks to merge one at a time. In any case, they should be doing the legwork to fix the problem, since they caused it.
What is the best way to deal with such a situation where one branch is really far away from the others?
In general, don't worry about it: it's the other developer's problem. If two branches are really too far apart to merge, then they aren't really part of the same project anymore and you have a defacto fork. If it's an open source project, that may not even be a problem.
If this developer is really brilliant, and their code is better/smarter/more important than the rest of the team combined, then it's worth making it your problem instead of theirs. Otherwise, it isn't.
To answer the literal question: the best way to deal with this kind of situation is to not get in that kind of situation.
What ways can we avoid branches getting a very large number of commits away from master in the future?
Make sure everyone notices that the developer who went for months without merging is having to fix the problem they caused. Make sure everyone knows that it's easier to commit to master frequently than infrequently, since fewer changes means fewer opportunities for conflicts.
Make sure that people know that they can pull from master to stay up-to-date with other people's changes.
"If you merge every day, suddenly you never get to the point where you have huge conflicts that are hard to resolve." --Linus Torvalds
That quote is from a talk he gave at Google, here's the transcript, and here's the video.
Best Answer
A workflow engine is useful when you need to go from a start to a finish but there are many different paths/logic/rules to get there.
For example, let's say I write a program that publishes content. So, in my case, the publishing goes through a review process, legal, and then final approval. I write up the program implementing my process logic and steps. Now this work great for me and my company. So, I decide others should use my program.
Unfortunately, not everyone publishes content using the same process, so instead of writing separate processes for each different case, we would implement a work flow process so the program is flexible to accomodate everyone. No matter how many steps or rules or logic are between those two points the result is the same.
So, if you have processes that are variable from start to end, use a workflow. If the same process can be used by everyone, then you don't need a workflow.