Is there a mathematical formula between wire size, current, temperature and wire type(Or resistance)?
The resistance of the wire will be 12/10 = 1.2 ohms
R = rho * length / Area (where rho is the resistivity of the material chosen, remembering to convert length etc. to appropriate unit)
then Area = rho * length /1.2 this will give you the cross sectional area. Look up the X sectional area to give a wire gauge.
All this will do is give you a wire of the correct length and resistance for a given material.
The temperature the wire will 'get up to' depends on how quickly energy is lost to the environment. (Power out = power in @ equilibrium temperature).
There is no set formula for the final temperature of the wire because there are so many variables e.g. how well insulated the wire is, how it is wound (large or small coils), in air or vacuum and so on and so on ...
You have two basic options.
Experiment: Get your wire (as calculated above), feed it with 12V and see what happens temperature wise.
Devise a circuit to limit how hot the wire gets (within your desired range) by controlling the amount of power dissipated in the wire. (e.g. - fit a thermostat)
The part you are missing is the thermal resistance from the wire to ambient air (or whatever medium you have it in). This is commonly specified in °C/W for semiconductor devices. For wires, of course the total power depends on the length, so you are looking for figure like °C/(W m).
For example, let's say you want the wire temperature to be 100 °C and that you've found its thermal resistance to ambient air is 10 °C/(W m). Ambient air in this example is at 20 °C, so the temperature rise required is 80 °C. That means that you need to dump 8 W into every meter of wire. If your wire segment is 200 mm long, then you need to dump 1.6 W into it.
Once you know the power required, you can use the electrical resistance to compute the voltage and current that will attain that power.
Keep in mind the resistivity of many substances changes significantly over temperature. Old LEBs (light emitting bulbs) were a good example. The filament gets so hot it gives off significant light. It is several times more resistive at that temperature than at room temperature. LEBs therefore had large currents for a short time after being switched on until they got to glowing temperature.
One way to deal with varying resistivity is to measure both voltage and current in a microcontroller, do the multiply to find the power, and adjust the power supply accordingly to maintain the desired power. The power supply is likely a switcher controlled by a micro anyway, so this doesn't add much complexity.
A even better way is to regulate the temperature directly. This scheme makes use of the fact that the wire resistance changes with temperature. You measure voltage and current, but this time compute the resistance. The power supply is then regulated to keep this resistance constant. This method is tricky if the material doesn't have a lot of resistivity change per temperature at the operating point you care about.