]> git.michaelhowe.org Git - packages/o/openafs.git/commit
viced: consistently enforce host thread quota for ICBS(3)
authorMark Vitale <mvitale@sinenomine.net>
Tue, 17 Sep 2019 19:14:44 +0000 (15:14 -0400)
committerStephan Wiesand <stephan.wiesand@desy.de>
Sat, 25 Jan 2020 20:59:08 +0000 (15:59 -0500)
commit9f38cbdc544d6659c60dbd578e86110df86457b6
tree706a307bd27c567b3f2fedff807f091dae4249ce
parent0922caa3be5df220aa3d0a0d828c21b279defad1
viced: consistently enforce host thread quota for ICBS(3)

From time to time, the fileserver may issue potentially long-running
RXAFSCB_* RPCs back to a host (client).  If these are holding h_Lock_r
(host->lock) while running, they may cause other service threads for the
same host (client) to block.

In order to prevent a given host from tying up too many service threads
in this way, the fileserver enforces a quota limiting how many threads
can be waiting for h_Lock_r on a particular host while waiting for one
of the following RPCs to complete:
- RXAFSCB_TellMeABoutYourself (TMAY)
- RXAFSCB_WhoAreYou
- RXAFSCB_ProbeUuid
- RXAFSCB_InitCallBackState (ICBS)
- RXAFSCB_InitCallBackState3 (ICBS3)

Note: Although some of these RPCs are relatively lightweight, they may
still experience network delays.

This quota is enforced by calling h_threadquota() in h_Lookup_r and
h_GetHost_r.  The quota check is enabled for a given host by turning on
host->hostFlags HWHO_INPROGRESS for the duration of the RXAFSCB_* RPC.
The quota check is only needed, and should only be enabled, when the RPC
is issued while h_Lock_r is held.

However, there are a few paths to ICBS(3) where h_Lock_r is held but
HWHO_INPROGRESS is not set.  A delay in those paths may allow a host to
consume an unlimited number of fileserver threads.  One such path
observed in a field report was SRXAFS_FetchStatus -> CallPreamble ->
BreakDelayedCallBacks_r -> RXAFSCB_ICBS3.

Instead, enable host thread quotas for all remaining unregulated ICBS(3)
RPCs.

Reviewed-on: https://gerrit.openafs.org/13873
Reviewed-by: Benjamin Kaduk <kaduk@mit.edu>
Tested-by: BuildBot <buildbot@rampaginggeek.com>
(cherry picked from commit aefc4c4f46e13f59b4cbe043e1a2a6f4ed99e076)

Change-Id: If3883a152078bba9995e0c8a13ab31788db6347f
Reviewed-on: https://gerrit.openafs.org/13893
Reviewed-by: Michael Meffie <mmeffie@sinenomine.net>
Reviewed-by: Mark Vitale <mvitale@sinenomine.net>
Reviewed-by: Andrew Deason <adeason@sinenomine.net>
Reviewed-by: Stephan Wiesand <stephan.wiesand@desy.de>
Tested-by: Stephan Wiesand <stephan.wiesand@desy.de>
src/viced/callback.c