If I understand you correctly, you have a function \$I\mapsto \omega(I) \$ (the mechanical simulation) and you are looking for \$I\$ that solves
\$ I = { V - G I \omega(I) \over R } \$.
This is a steady state solution, there are no system dynamics involved, so the issues are with the technique you are using to solve the equation.
There is no reason a priori that the 'relaxation' technique will work.
One hack you could try is to update the current \$I\$ incrementally, which
introduces some numerical 'damping'.
You could try a binary search to find the satisfying value.
You could implement a secant method (essentially estimate the slope by finite differences).
Just like it's simplest to learn about a lossless inductor first, so let's start with more or less lossless motor. We'll take account of losses when we have to, but they are not essential for basic understanding.
A motor is also a generator. Spin it, and it generates volts on the armature. It doesn't matter whether it's spinning because it's a motor, or spinning because you're driving it as a generator, speed = armature_volts/k.
Pass a current through it and it generates a torque. Torque = armature_current.k
You can think of a motor as a mechanical transformer. Power in = power out. Volts x amps in = speed x torque out. When equating power, that pesky k has cancelled out. If you change the value of k, the torque constant, then the motor gets faster and delivers less torque, or vice versa, but the power balance is the same. If you run the same machine as a generator, then speed x torque in = volts x amps out.
What happens if you apply a voltage source to a motor at rest?
Two things happen, at different speeds, the first so quickly you may not notice, the second rather more slowly.
A motor armature has inductance, Larm, and resistance Rarm. At the moment of switch on, we apply V to the armature. The current starts to increase. It initially increases at such a rate that Larm x dI/dt generates a back EMF equal to the terminal voltage. The current flowing through Rarm generates a voltage IRarm which opposes V, so there is less voltage across the inductance to drive an increase of current, so the rate of current increase slows down. Eventually, the current has increased to settle at V/Rarm, with a time constant of Larm/Rarm, typically in a matter of mS.
With a 'good' motor with a low Rarm, this current will typically be very large. It's known as the 'starting' current, for obvious reasons. Small motors are rated to be started like this safely. Big motors cannot be started like this, and need some sort of soft start controller.
So far, the motor still hasn't moved, the mechanical inertia means it's not rotating yet, or has barely started. But now there is a big armature current flowing, which generates a torque, and the motor accelerates.
Once it is turning, at any speed, it generates a back EMF proportional to its speed. This back EMF reduces the effective terminal voltage available to drive current through Rarm. The armature current therefore falls (with a time constant of Larm/Rarm), and so generates less torque.
Eventually the motor reaches a balance, where it's at a speed where the generated back EMF balances off most of the input voltage, and the small difference in voltage that remains drives an armature current through Rarm, which generates enough torque in the motor to match the load torque, plus loss torques like friction and air resistance.
Best Answer
This question seems to have caused a lot of confusion because of the description of the power supply. As revised, we can assume that the power supply may be adequate for the test described.
The equivalent circuit for the motor is shown below. The back emf, e(t) is a voltage generated by the armature that opposes the current from the power supply. The emf is directly proportional to the rotational speed of the armature. If the motor shaft is prevented from turning, e(t) = 0. That means that the armature current is determined by the armature resistance Ra. Ra is quite small. Since the power dissipated in Ra is responsible for most of the losses in the motor its value is inversely proportional to motor efficiency. If the motor is switched on with rated voltage Va applied, the initial current may be on the order of ten times the rated motor current.
Since the current allowed by Ra can damage the motor very quickly, the blocked rotor test described in the question should only be performed using a supply voltage that is much less than the rated motor voltage, 10% for example. The objective of the blocked rotor test is to determine the values of Ra and La. Since the power supply output impedance can have an effect on the determination, the power supply used should have an output impedance that is mush lower than the motor impedance. The output voltage should be monitored to note any drop in voltage.
With the rotor blocked, when Va(t) is applied as a step increase from zero, the current is limited by Ra and La. The rise in voltage will follow an inverse exponential curve with the time constant RxL.
If the rotor is free to turn, the same limitation in current applies initially, but as the motor accelerates, the back emf will further limit the current. If the shaft is not connected to a load, acceleration will be determined by the motor torque, the inertia of the motor rotor, the bearing and brush friction and the aerodynamic drag of the air on the rotor. The effect on motor current will be a slightly slower rise in current, and a slightly lower peak current followed by the current decreasing to a value that represents the torque required to overcome the friction and drag.