If someone asked for the average power dissipated in a device, what
would that mean?
The average power is the time average of the instantaneous power. In the case you describe, the instantaneous power is a 1W peak square wave and, as you point out, the average over a period is zero.
But, consider the case of (in phase) sinusoidal voltage and current:
$$v(t) = V \cos \omega t $$
$$i(t) = I \cos \omega t $$
The instantaneous and average power are:
$$p(t) = v(t) \cdot i(t) = V_m \cos\omega t \cdot I_m \cos\omega t = \dfrac{V_m \cdot I_m}{2}(1 + \cos2\omega t) $$
$$p_{avg} = \dfrac{V_m \cdot I_m}{2}$$
(since the time average of sinusoid over a period is zero.)
In the above, we evaluated the time average of the instantaneous power. This will always give the correct result.
You link to the Wiki article on AC power which is analyzed in the phasor domain. Phasor analysis assumes sinusoidal excitation so it would be a mistake to apply the AC power results to your square wave example.
The product of the rms phasor voltage \$\vec V \$ and current \$\vec I \$ gives the complex power S:
$$S = \vec V \cdot \vec I = P + jQ$$
where P, the real part of S, is the average power.
The rms phasor voltage and current for the time domain voltage and current above are:
$$\vec V = \dfrac{V_m}{\sqrt{2}} $$
$$\vec I = \dfrac{I_m}{\sqrt{2}} $$
The complex power is then:
$$S = \dfrac{V_m}{\sqrt{2}}\dfrac{I_m}{\sqrt{2}} = \dfrac{V_m \cdot I_m}{2}$$
Since, in this case, S is purely real, the average power is:
$$P = \dfrac{V_m \cdot I_m}{2}$$
which agrees with the time domain calculation.
The key part in the question is that it's an "average reading (with full scale rectification)" rather than a true RMS calculation. What this means is that the input signal is rectified and the average value is then effectively multiplied by a constant with the assumption the input waveform was a pure sinusoid.
The definition of RMS for a periodic waveform is:
$$ V_{RMS} = \sqrt{\frac{1}{T}\int_0^T v^2(t) dt}$$
The average value with full scale rectification would be:
$$ V_{avg} = \frac{1}{T}\int_0^T |v(t)| dt$$
So the ratio of these for a pure sine wave would be the multiplication factor \$k\$ built into the meter. That is:
$$\begin{eqnarray}
k &=& \frac{V_{RMS}}{V_{avg}}\\
k &=& \frac{\sqrt{\frac{1}{T}\int_0^T v^2(t) dt}}{\frac{1}{T}\int_0^T v(t) dt}\\
k &=& \frac{\sqrt{\frac{1}{2\pi}\int_0^{2\pi} \sin^2(t) dt}}{\frac{1}{2\pi}\int_0^{2\pi} |\sin(t)| dt}\\
k &=& \frac{\frac{1}{\sqrt{2}}}{\frac{2}{\pi}}\\
k &=& \frac{\pi}{\sqrt{8}}\\
k &\approx& 1.11
\end{eqnarray}$$
In other words, the measured rectified average voltage will be multiplied by about 1.11 to get the indicated value.
To answer the first part of the question, you need to find \$V_{avg}\$ for the sawtooth waveform, which you have already correctly done and found it to be 5V.
What the meter will then do is multiply that by 1.11 and it will indicate 5.55V (not 6.5V -- your given answer is not correct, which was probably the source of your problem).
To determine the percentage of error is simply to compare that value with the true RMS value, which again, you have correctly calculated as 5.77V.
$$err = \frac{5.55 - 5.77}{5.77} = -0.0381 = -3.81\%$$
So the percentage of error is -3.81%, with the negative sign meaning that the indicated value is lower than the actual. Here again, the given answer was simply not correct.
If this is from a textbook, you might want to see if there are published errata that corrects that. If not, you might send the author a note -- although it's too late for you, it could save countless hours of frustration for future students.
Best Answer
Simple: the average of a sine is zero.
Power is proportional to voltage squared:
\$ P = \dfrac{V^2}{R} \$
so to get average power you calculate average voltage squared. That's what the RMS refers to: Root Mean Square: take the square root of the average (mean) of the squared voltage. You have to take the square root to get the dimension of a voltage again, since you first squared it.
This graph shows the difference between the two. The purple curve is the sine squared, the yellowish line the absolute value. The RMS value is \$\sqrt{2}/2\$, or about 0.71, the average value is \$2/\pi\$, or about 0.64, a difference of 10 %.
RMS gives you the equivalent DC voltage for the same power. If you would measure the resistor's temperature as a measure of dissipated energy you'll see that it's the same as for a DC voltage of 0.71 V, not 0.64 V.
edit
Measuring average voltage is cheaper than measuring RMS voltage however, and that's what cheaper DMMs do. They presume the signal is a sine wave, measure the rectified average and multiply the result by 1.11 (0.71/0.64) to get the RMS value. But the factor 1.11 is only valid for sinewaves. For other signals the ratio will be different. That ratio got a name: it's called the signal's form factor. For a 10 % duty cycle PWM signal the form factor will be \$1/\sqrt{10}\$, or about 0.316. That's a lot less than the sine's 1.11. DMMs which are not "True RMS" will give large errors for non-sinusoidal waveforms.