afs: Avoid panics in afs_InvalidateAllSegments

afs: Avoid panics in afs_InvalidateAllSegments

Currently, afs_InvalidateAllSegments panics when afs_GetValidDSlot
fails. We panic in these cases because afs_InvalidateAllSegments
cannot simply return an error to its callers; we must invalidate all
segments for the given vcache, or we risk serving incorrect data to
userspace as explained in the comments.

Instead of panicing, though, we could simply sleep and retry the
operation until it succeeds. Implement this, retrying every 10
seconds, and logging a message every hour that we're stuck (in case
we're stuck for a long time).

When we retry the operation, do so in a background request, to avoid a
somewhat common situation on Linux where we always get I/O errors from
the cache when the calling process has a SIGKILL pending. Create a new
background op for this, BOP_INVALIDATE_SEGMENTS.

With this, the relevant vcache will be effectively unusable for the
entire time we're stuck in this situation (avc->lock will be
write-locked), but this is at least better than panicing the whole
machine.

Reviewed-on: https://gerrit.openafs.org/13677
Reviewed-by: Benjamin Kaduk <kaduk@mit.edu>
Reviewed-by: Michael Meffie <mmeffie@sinenomine.net>
Reviewed-by: Cheyenne Wills <cwills@sinenomine.net>
Tested-by: BuildBot <buildbot@rampaginggeek.com>
(cherry picked from commit 3be5880d1d2a0aef6600047ed43d602949cd5f4d)

Change-Id: Iba1cde70a4d5e919fedfe27d0540878113a369e4
Reviewed-on: https://gerrit.openafs.org/13847
Tested-by: Andrew Deason <adeason@sinenomine.net>
Reviewed-by: Michael Meffie <mmeffie@sinenomine.net>
Reviewed-by: Cheyenne Wills <cwills@sinenomine.net>
Reviewed-by: Marcio Brito Barbosa <mbarbosa@sinenomine.net>
Reviewed-by: Mark Vitale <mvitale@sinenomine.net>
Reviewed-by: Stephan Wiesand <stephan.wiesand@desy.de>