Methods of Determining the Accuracy of a Watch

[South Pender]

Artec has kindly sent me his Casio EX-F20 high speed video camera to see if it would help get quicker results with the video method. As a reminder, with a 25fps camera, the resolution is 0.04 seconds and therefore the accuracy of the method over 24 hours is of 14spy and over a week of 2spy.

Not quite true. If the error associated with a single timing at the beginning of the 24-hour period is .04 seconds, as you have stated, it follows that there is also a similar error associated with the single timing at the end of the 24-hour period. The change (or drift) estimate relies on both measurements. Thus, the total error in the assessment of drift--by which you estimate accuracy--is about .06 seconds, or about 22 spy. This is a perfect illustration of why attempting to estimate spy using a one-day timing period will necessarily be far too imprecise.

However, your assertion of an error bandwidth of .04 seconds accounts only for camera error. What about clock error? This must be factored in, as noted in my just-preceding post. Once we do this, we get a single-measurement bandwidth of about ± .06 seconds, and this applies at each measurement point. Since two measurements are required to evaluate drift (that is to get a spy value), the actual error bandwidth of the change, or drift, calculation will be about ± .08 - .09 seconds. Given this fact, any estimate of spy using a single observation at each time point with the Video Method will produce a total error bandwidth of about ± 31 seconds for a one-day assessment of spy, and about ± 4.5 seconds for a spy estimate based on a one-week time period--that is, taking the offset from the atomic clock at Time 1 and then again at Time 1 + 7 days and using the change between the two as an estimate of drift, prorated to an annualized spy value.

[Catalin]

Artec has kindly sent me his Casio EX-F20 high speed video camera to see if it would help get quicker results with the video method. As a reminder, with a 25fps camera, the resolution is 0.04 seconds and therefore the accuracy of the method over 24 hours is of 14spy and over a week of 2spy.

Tried it immediately but unfortunately the high speed mode requires a lot of light so I wasn't able to get a reading during my usual night-testing.

Tried again outdoors and while it's pretty fascinating to see the backlask on the seconds had of my 8F35 at 210fps (480x360 resolution), it's really hard to read the time display on my laptop as it seems to move in increments of 0.02s, i.e. a refresh rate of 50fps. Unfortunately the setting below that is a 30-210fps. Can't fine-tune it, probably an auto fps mode, will check the manual to see if it can be forced to 100fps.

Very interesting!!! However please also note my original reference to the refresh rate of the monitor - since I have an ordinary camera the way I am trying to use the video method is based more on the frame-rate of the monitor (than on the frame-rate of the camera), and even with my pretty old camera at 30 fps in 2-3 consecutive videos I can very often get enough cases where I can observe the 60 fps intervals from the monitor - meaning intervals of around 17 milliseconds

[webvan]

Yes same here with my 25fps camera, except it's 20ms. I was mostly interested in seeing what a high speed camera could do to improve the video method for observations over a shorter period of time but it seems it's not much, if anything. The monitor being the limiting factor and also the requirement for light at these frame rates. Still, these cameras create impressive slow motion videos so it was time well spent

[Catalin]

Yes same here with my 25fps camera, except it's 20ms. I was mostly interested in seeing what a high speed camera could do to improve the video method for observations over a shorter period of time but it seems it's not much, if anything. The monitor being the limiting factor and also the requirement for light at these frame rates. Still, these cameras create impressive slow motion videos so it was time well spent

Yes, those slow-motion videos can be indeed spectacular

But returning to the video method - now I realize that for plenty of people things might not be as easy as in my example - using a 30fps camera and a 60fps monitor could generate a slightly different kind of 'frame hunt' than when using a 25fps camera and a 60fps monitor - my old camera can only do 15 or 30 fps so I can't easily test the 25fps results but that sounds as an interesting test some day (or eventually a test with 30fps camera and 50fps monitor rate)

[South Pender]

Catalin, why do you not just take several video readings (say 8-10), and use their average reading as your data point. The error component is reduced by √n, where n is the number of readings you would take. In other words, if your error from a single observation is 20 ms., your error associated with the average of n = 10 readings would be 6.32 ms. That way, you'd overcome most of the frame-speed limitations. Or perhaps you do average (I can't remember), in which case, how many readings do you take?

[Catalin]

Catalin, why do you not just take several video readings (say 8-10), and use their average reading as your data point. The error component is reduced by √n, where n is the number of readings you would take. In other words, if your error from a single observation is 20 ms., your error associated with the average of n = 10 readings would be 6.32 ms. That way, you'd overcome most of the frame-speed limitations. Or perhaps you do average (I can't remember), in which case, how many readings do you take?

I believe the assumption above is ONLY valid when you deal in 'continuous values' where the errors are pretty random and probably smaller than any of the known sources of non-random errors - with the video method (and decent 'hardware') the values are not continuous but instead have certain 'quantified' steps in multiples of about 16.66 ms for the monitor and 33.33 ms for the camera.

I am generally making 2-3 videos of around 10 seconds (which provides about 20-30 pairs of relevant frames) each but not in order to compute clear averages but instead in order to be able to get to the higher degree of accuracy from the monitor steps (which is obviously twice better than the 'default one' from the camera).

It does not make enough sense at this point to try anything more than that - given the fact that the error from the initial Internet time-sync is still in the range of 10-20 ms (maybe closer to 5 ms. now that I use ntpdate with multiple servers) and that the error on the computer clock seems to be in the range of +1 ms/minute ...

Also from what I have seen many of the watches in my tests seem to 'move' the seconds hand in something around 10 ms but I have also seen a few clearly going from one position to the next in an interval higher than 33ms.

[South Pender]

I believe the assumption above is ONLY valid when you deal in 'continuous values' where the errors are pretty random and probably smaller than any of the known sources of non-random errors - with the video method (and decent 'hardware') the values are not continuous but instead have certain 'quantified' steps in multiples of about 16.66 ms for the monitor and 33.33 ms for the camera.

I am generally making 2-3 videos of around 10 seconds (which provides about 20-30 pairs of relevant frames) each but not in order to compute clear averages but instead in order to be able to get to the higher degree of accuracy from the monitor steps (which is obviously twice better than the 'default one' from the camera).

It does not make enough sense at this point to try anything more than that - given the fact that the error from the initial Internet time-sync is still in the range of 10-20 ms (maybe closer to 5 ms. now that I use ntpdate with multiple servers) and that the error on the computer clock seems to be in the range of +1 ms/minute ...

Also from what I have seen many of the watches in my tests seem to 'move' the seconds hand in something around 10 ms but I have also seen a few clearly going from one position to the next in an interval higher than 33ms.

Good points. However, your point about averages only making sense with continuously-distributed random variables is not quite correct. Most of the data that we analyze in any branch of research is in the form of a discrete (in steps) variable overlaying a continuous variable. The numbers are discrete steps because of the impossibility of capturing the infinite number of observations associated with the underlying continuous random variable. This fact, however, doesn't stop us from computing means, standard deviations, and many other statistics from such discrete variables.

[Catalin]

Good points. However, your point about averages only making sense with continuously-distributed random variables is not quite correct. Most of the data that we analyze in any branch of research is in the form of a discrete (in steps) variable overlaying a continuous variable. The numbers are discrete steps because of the impossibility of capturing the infinite number of observations associated with the underlying continuous random variable. This fact, however, doesn't stop us from computing means, standard deviations, and many other statistics from such discrete variables.

That is correct in certain conditions but not in all - for instance if you can only measure the height of a person in integer multiples of let's say feet (meters would be a little too extreme ) and you calculate the means over a REALLY large population the final result might still be surprisingly close to the means that you get from measurements at 1cm increments; however the standard deviation could be quite different (and that could also suggest a measurement limitation). On the other hand if you measure the height of a SINGLE person in increments of 1 feet no matter how many times you measure/average it the result will be (for like 99% of the people) a very constant number which will also be quite far away from the actual height measured in increments of 1cm!

[South Pender]

Quite true. All that is required really, with a discrete random variable, is that the intervals be of the same size at all points of the scale. So, are you saying that your interval size is something on the order of 16.66 ms.? If so, would your measurements be like: -33.33, -16.66, 0 +.166, +33.33? That is pretty coarse all right, but wouldn't a mean of, say, 10 of these produce far better data points than just one observation?

[Catalin]

Quite true. All that is required really, with a discrete random variable, is that the intervals be of the same size at all points of the scale. So, are you saying that your interval size is something on the order of 16.66 ms.? If so, would your measurements be like: -33.33, -16.66, 0 +.166, +33.33? That is pretty coarse all right, but wouldn't a mean of, say, 10 of these produce far better data points than just one observation?

At some point with a very, very large amount of extra data plus using the extra fact that I can estimate the (huge) drift of the computer clock and I can force a large number of NTP syncs plus eventually knowing the inhibition period and eventually assuming that we only talk about the calibers where the seconds-hand moves very fast (in under 10 ms.) we MIGHT be able to cut the errors to maybe 3-5 ms - the point however remains that it will be INCREDIBLY time-consuming - don't forget that already (with 17 watches) I was losing over 1 hour and we are now maybe talking about losing 1 hour for a single watch ... definitely in the current economy