Make estimating easier
Break your sprint planning down.
Do you need to estimate the individual tasks? I've done sprint planning two ways:
- Stories are estimated in story points and then tasks are estimated in hours
- Stories are estimated in story points and tasks simply fall under that with no estimate
Of the two, I prefer the second option. I find that not estimating tasks gives more freedom to developers to cope with changes. If a task no longer makes sense (how many times have you found out that a task isn't applicable or was already done in a previous sprint) you simply throw it out without any penalty, or you may have to change a current task into something new, possibly breaking it up. You're really being redundant if you estimate both, as the sum of the tasks should represent the story points and vice versa. What value do you really gain by this other than knowing how much time individual tasks will take? If you find yourself with task sizes that really vary enough to make a difference, I would suggest breaking those tasks down into smaller, more homogeneous chunks.
By doing this, you can cut down on the time you spend in sprint planning. Stories get estimated during sprint planning, and when you start the sprint you can put down all the tasks you can think of that make up that story. Obviously if there are points that you come across in estimating the story that you know will have to be dealt with in a task, you can add that onto the story information and put it as a task.
Estimate in Ideal units
Story points are meant to be in ideal units such as ideal man hours or ideal work days. These are meant to mean that given the perfect day every day, where you had no interruptions, no meetings, and everything went according to plan, you could accomplish the task in X days. Now everyone knows that this simply isn't true, but the wonderful thing about statistics is that it doesn't have to be.
What I mean by this is that after a while of estimating these in ideal days, you realize that maybe it takes an extra 25% of the time you estimate on average to complete a story. Lets say you had estimated 4 ideal work days, and instead it took you 5. Over time, you keep track of this and then you have a rough idea of the conversion from ideal days to real days. Your first instinct would be to try and compensate for this by over estimating, and you would likely be wrong. The main thing here is to stay consistent. That way, your long term average remains the same. Sure sometimes, it'll be under and sometimes it'll be over, but the more you estimate, the better off you are. If you find that you still can't get a decent estimate, maybe that means you don't know enough about the story to estimate it properly.
Talk about the stories
When you estimate, everyone should have a rough idea of what will need to be done, from start to finish, of what it will take for this story to be complete. You don't need to know every detail, but enough that you think you, yourself, could undertake the story. If you don't have that level of confidence, you probably shouldn't be estimating it. If you say "Well this story is too big for us to know most of the details" then that's an indication that the story is too big, and should be broken down. Stories, at least in my experience, have been small enough that one person, if need be, could work on it alone and accomplish it within a week or two.
This also will help to solve your second point in the edit, which is too much estimation. Instead of estimating every single task for every single story, you simply estimate the story as a whole, which should help to remove a lot of the estimating. As for making the meetings less monotonous, I would suggest planning poker, which you can see more information about above.
Make planning more engaging
Estimate using Planning Poker
As far as making estimation more fun, have you tried planning poker? It's the way that I've always done planning for all my sprints on multiple teams, and it's a good way to keep everyone involved, as every person has to at least pick SOMETHING. There's also a fair amount of fun involved when everyone on the team picks 3, and someone puts down a 20 and has to explain themselves, or when everyone on the team puts down a 5 but the manager puts down an 8 (who's gonna argue with the boss when he wants to give you more time!).
To do this, all you need are some planning poker cards, which we often make on the back side of index cards, or using normal playing cards with values attached to face cards. Nothing fancy, and it keeps everyone focused. Just remember that trying to do any task for an entire day (including planning poker) takes a toll on productivity. Many sets of cards come with a coffee card for a reason; if someone is feeling burnt out, give the team a break to recharge and pick it up when everyone is fresh!
As an alternative to physical cards, you could also look at electronic cards. The real benefits here are automated tracking of results, tracking user stories to be estimated and allowing everyone to show their cards at once to avoid "cheating" (where one persons estimate is influenced by another's due to being able to see their card). Obviously this requires everyone have a computer and the ability to focus on the task at hand though, so use it at your own discretion.
Best Answer
Schedule a recurring "grooming" meeting where you have these design discussions. The team I'm on has them once per sprint, on the day before planning. The goal there is to have the design far enough nailed down that the team can agree on the time estimates for the overall story. You could consider doing task breakdowns in this meeting, so that planning becomes purely an exercise in deciding how much to pick up. In other words you should be doing the design in the sprints BEFORE you start doing the actual work.
Consider using planning poker, i.e. points/units of "effort" rather than man-days to estimate tasks. Also try to break down the stories as much as you can. The longer/more complex a story is, the less likely your estimate is going to be accurate.
In planning, the scrum master should keep the planning on track by calling a halt to any discussions that get too far into "solving." At that point the team members are required to quickly come to an agreement on the estimate, usually giving an upper-bounds/worst case number. It's much easier to pick up more work if the tasks end up being easier than you plan, than it is to deal with schedules slipping due to tasks taking longer than planned and stories rolling over into multiple sprints.
Talk about how the estimates panned out in the retrospective at the end of the sprint. Particularly if there were any that were remarkably far off. Did the team learn anything from how the story went versus how they expected it to go? The scrum master should keep the focus on actionable changes that can be made to your design/estimation process.