While it is relatively possible for an experienced developer to estimate how long it will take to implement code when the pattern and problem the code is solving is well understood, how can you make a good estimate when, while the end goal is well understood, the implementation is 95% theoretical/problem solving and has very small amounts of implementation?
My work often consists of tasks to accomplish well defined goals, however I have to find the way to reach that goal and until I understand the solution, it i6sn't clear what additional barriers there may be. More specifically, I'm often working on code generation or automated code manipulation tools. Once the solution is fully solved and the tool perfected, it will directly do 95% of the actual changes very quickly. I don't, however, have any way to estimate how many additional issues may have to be resolved to make the generation or analysis tool deal with unforeseeable edge cases.
For planning purposes, my company wants a better idea of how long it will take, but since I don't know how many additional problems may come up while working through solving each step of the solution. I'm not sure how I can approach giving a better estimate.
Best Answer
Before I get too far, let me say that Software Estimation: Demystifying the Black Art is an excellent resource for people looking at and thinking about estimations. Both of the images below are from that book as are the core if the ideas presented following.
As you've noted, estimations are an important part in being able to accurately predict and plan work. Not having estimations makes the business blind about how long something will take. It is not uncommon for business to have a completely mistaken idea on how long things will take - what they think is easy takes six to eight weeks and what is thought to be hard is a friday afternoon hack.
The first thing is to give an estimate. An estimate itself is not a single number - thats a commitment. "How long will ABC take" --> "About 5 days" means that its about 5 days. However, a good estimate is a range where you are 90% confident that you will have it in that range. If you mean to say "I am 90% confident that it will take betwene 1 and 5 days" then say that. Don't work from "I think it will take between 1 and 10 days, so 5 days is probably about average" - thats not an estimate and you'll be wrong 50% of the time.
Well, 50% or more of the time, programmers are notorious underestimators for task times.
Consider the Cone of Uncertainty:
Image from http://www.construx.com - full article at http://www.construx.com/Thought_Leadership/Books/The_Cone_of_Uncertainty/
Realize that the first estimate in that range is 16x. This is akin to saying "I think it will take between an afternoon and two weeks" - but you don't know yet. As you go forward with the design a bit, the range narrows down to 4x. This does not mean that it will take one week, it means that you would instead be saying "after looking at this a bit, it will take between three weeks" - yes, the estimate went up, but also the range of the estimate went down.
With each estimate you give, you need to be 90% sure that the estimate is within that range. You can be wrong - 10% of the time it will fall out of that range.
There are many ways to estimate the size of projects. Comparing it to past projects, using a proxy (I think it would take 1000 lines of code which would take this long to write), using function points (to convert into LOC...), getting estimates from a number of people and then refining it iteratively... some work for some projects, some work for other projects.
A very important chapter in this book that I mentioned at the top is #23 that deals with the politics of estimation and dealing with managers and executives.
The key to an estimation is the iterative process of refining it after working on it a bit.
Giving a too precise of an estimate too early in the process can be very error prone. If you aren't sure about it, give the wide estimate and then come back with another estimate after some period of time for more introspection into the problem and possibly sketching out how you will do it, looking at how much code you wrote it for the last similar problem and other factors that will impact the estimate.
Estimates require some thought - don't give off the cuff estimates. These often have huge errors associated with them compared to what it takes when you think about it a bit.
From How to respond when you are asked for an estimate?
From Chapter 4 of Software Estimation:
Note that in this, the estimates after a bit of review are systematically less wild and error prone than the off the cuff estimates. Don't do off the cuff estimates. Sit down and think about the task and estimate it after a bit of thought on it.