Usually, if the modulation percentage is relatively high, say 30% or greater, the time domain approach will give the best accuracy. For low modulation percentages, 30% and less, the frequency domain approach has greater potential accuracy.

Let's work through a couple of examples.

First, let's consider a signal modulated 30% with a 1 KHz sine wave, using the signal generator's modulation percentage meter to set the level. In this case, the signal source is a Boonton 102D signal generator.

In the time domain, using an oscilloscope we see a signal similar to that shown below.
 

We then calculate the modulation percentage:

The measured modulation is quite close to the signal generator's 30% setting.

 

This is not nearly as accurate as our envelope (time domain) measurement. Why? Because an error of only one or two dB in measuring the difference between the carrier and side band levels causes a relatively large error. 

Let's look at a more difficult case, 5% modulation. As you can see, accurately measuring the peak and trough of the modulated waveform will be difficult, particularly with an analog oscilloscope without on-screen measurement cursors.
 

Looking at the same signal in the time domain with the Z90 shows the modulating sidebands are considerably lower than the carrier.

One final observation that we may make much easier in the frequency domain is to determine if the signal generator has simultaneous AM and FM modulation. Almost all signal generators before the era of phase lock and DDS techniques have some degree of incidental frequency modulation that occurs simultaneously with the desired amplitude modulation. Simultaneous AM and low level incidental FM will reveal itself in different amplitude of the upper and lower sidebands.

To illustrate this, I intentionally simultaneously modulated my Boonton 102D signal generator with 1 KHz, setting the AM level at 30% and FM level corresponding to a deviation of  100 Hz. This is a low amount of FM, but it shows up as a quite noticeable difference between the upper and lower sidebands. You will note a very small difference between the two sidebands in the 5% AM modulation as well, perhaps 1 dB or so. This is the result of a very small unwanted simultaneous AM and FM modulation.