Wrt excursion you'll find that any enclosure will affect this at a given input level. The closed enclosure will provide damping of the excursion all way down, while the ported version will provide little or no damping below the resonance frequency of the port.
I would suggest to do a frequency sweep with the driver mounted in the actual box to be used, starting at reduced input level. You should test for power as you have suggested, but also for excursion.
You will need a way to determine when Xmax is reached, and one way to do it is to use light weight pin. The pin should have a rounded head and be fixed loosely in e.g. expanded polystyrene or the like. You need to jerk the pin in and out some times to get it loose enough. The idea is that the pin is fixed at a distance from the dust cap, and when the desired excursion is reached, the pin will be displaced. Remember that polystyrene will easily move with the displaced air, so it should be a little as possible and held in position by a device made of high density material.
If you do this together with measuring the power, you will get the info you need for safe operation.
The key part in the question is that it's an "average reading (with full scale rectification)" rather than a true RMS calculation. What this means is that the input signal is rectified and the average value is then effectively multiplied by a constant with the assumption the input waveform was a pure sinusoid.
The definition of RMS for a periodic waveform is:
$$ V_{RMS} = \sqrt{\frac{1}{T}\int_0^T v^2(t) dt}$$
The average value with full scale rectification would be:
$$ V_{avg} = \frac{1}{T}\int_0^T |v(t)| dt$$
So the ratio of these for a pure sine wave would be the multiplication factor \$k\$ built into the meter. That is:
$$\begin{eqnarray}
k &=& \frac{V_{RMS}}{V_{avg}}\\
k &=& \frac{\sqrt{\frac{1}{T}\int_0^T v^2(t) dt}}{\frac{1}{T}\int_0^T v(t) dt}\\
k &=& \frac{\sqrt{\frac{1}{2\pi}\int_0^{2\pi} \sin^2(t) dt}}{\frac{1}{2\pi}\int_0^{2\pi} |\sin(t)| dt}\\
k &=& \frac{\frac{1}{\sqrt{2}}}{\frac{2}{\pi}}\\
k &=& \frac{\pi}{\sqrt{8}}\\
k &\approx& 1.11
\end{eqnarray}$$
In other words, the measured rectified average voltage will be multiplied by about 1.11 to get the indicated value.
To answer the first part of the question, you need to find \$V_{avg}\$ for the sawtooth waveform, which you have already correctly done and found it to be 5V.
What the meter will then do is multiply that by 1.11 and it will indicate 5.55V (not 6.5V -- your given answer is not correct, which was probably the source of your problem).
To determine the percentage of error is simply to compare that value with the true RMS value, which again, you have correctly calculated as 5.77V.
$$err = \frac{5.55 - 5.77}{5.77} = -0.0381 = -3.81\%$$
So the percentage of error is -3.81%, with the negative sign meaning that the indicated value is lower than the actual. Here again, the given answer was simply not correct.
If this is from a textbook, you might want to see if there are published errata that corrects that. If not, you might send the author a note -- although it's too late for you, it could save countless hours of frustration for future students.
Best Answer
Please see this link :http://www.testequipmentdepot.com/uei/digital-multimeters/dm200.htm
The meter is a discontinued model also available from Ebay. It is NOT a true RMS meter so non-sine waves will give a ambiguous reading at best. Usually a non-RMS meter will see only the most peak voltage present and not the true average.
If you want an excellent true RMS DVM stay with a known good manufacture like Fluke. The Fluke 87 III series is $400 USD, but it is top dog for accuracy.