Trying to update too many bugs at once causes a spurious TooMuchTimeSkew error with DebBugs watches

I'm unclear as to why this should happen unless there's bad data in
Launchpad's DebBugs mirror. I'll investigate and report back.

 status incomplete

This was apparently caused by OOPS-1206CCW762, which resulted from a TooMuchTimeSkew error.

What's weird is that the DebBugs database that Launchpad imports from is actually a local mirror, so the TooMuchTimeSkew error is essentially the checkwatches process saying that the server its on is too far out of sync with... itself. Which makes no sense.

The watches that failed to be updated should be checked some time in the next 24 hours, though it may be worth doing a full checkwatches run against DebBugs in the meantime to update everything just to be sure.

I'll mark this as low priority since I'm fairly certain it's a spurious failure. If it happens again we can bump it up.

Okay, so in an example of spurious behaviour suddenly not being quite so spurious, it's happened again: OOPS-1207CCW717.

A couple of questions spring to mind:

 1. What's going on at the same time as this (it failed this time at ~13:00; last time at ~13:51; maybe there's a process on loganberry chewing up CPU at about that time).
 2. How long is it taking to get from the time that the expected server time is declared (i.e. server_time = datetime.now()) to where it's checked against the *actual* server time (also datetime.now())? Are we getting stuck in a stupidly long-running loop at some point?

I'm going to bump this up to high for investigation purposes.

It's happened again: OOPS-1208CCW674, this time at 12:11. I'll see if I can investigate today with the help of a LOSA. At the very least we could add some debugging code for subsequent runs.

This is happening with depressing regularity now - once a day at least. This needs fixing sooner rather than later.

This is happening because there are 9000+ DebBugs watches all being checked at the same time. I'll write a script to fix this. This isn't a bug in checkwatches so much as a failure to deal with an abnormally large volume of data.

Er. I didn't mean to invalidate the bug. I meant to bump it to low priority and un-milestone it. Like this.

Fixing this looks to be as simple as moving the check for a time skew so that it isn't likely to be impacted by long-running loops.

Fixed in devel r8623.