I believe your problem is that you changed the circuit from the one provided with the thermistor but did not alter the equation. The circuit on the website shows that the thermistor is driven from a 2000 mv (2 volts) source that is adjusted by a pot on the board. That accounts for the 2V factor in the equation relating output voltage to thermistor resistance. You are now driving the thermistor with 5 volts so the equation must be modified to change the 2V to 5V. You cannot get a Vout of more then 2 volts from the original board. That is why you are calculating a negative resistance.
Looks like there's some simple issue with the code.
To assist with getting the Steinhart-Hart equation parameters right, you can refer to the part specifications.
That refers to the "M" curve.
From the website, we get:
Which gives you the resistance values for various temperatures (multiply by 1000 for the values of your part in ohms).
Then, you can calculate the Steinhart-Hart parameters directly (from the Wikipedia page)
Or you can try this online calculator (I have not verified it is accurate)
In either case, optimize the thermistor linearization by picking three temperatures well spaced apart covering your range of interest.
The fourth column of the datasheet table, indicated as "B25/85 Value", is key: that is the Beta value measured between two temperature endpoints (25C and 85C). Second column has the resistance for the first temperature endpoint.
Given for device TTF-103: B25/85 = 3435, R25 = 10, R85 can be solved by using the formula:
R2 = R1 / (exp( B*(1/ T1 - 1/ T2) ))
Thus, R85 = 1.4513 and now you have two points. Beta can be assumed to be constant between the two test endpoints. Using the same formula and chosen temperature, you can establish the third point. Choosing T = 60, R60 = 2.9809 for example