sock_ops: calling bpf_sock_ops_cb_flags_set() for already established sockets

[Alan Maguire <alan.maguire@oracle.com>]

hi folks

I've run into a few cases where users have wanted to enable additional
tcp-bpf sock_ops events for a socket _after_ connection establishment.
The problem is that to set the flags to enable additional events, we
have to be in the context of a sock_ops program, and as I understand it,
by default only events early in the socket lifetime are enabled by
default (such as connection established/accepted). As a consequence, if
we do not catch one of those early events, the sock_ops program will not
run and we miss the opportunity to enable more sock_ops events. This can
be a problem for boot-time connections like iSCSI where we are too late
to catch connection establishment.

I can see a few possibilities:

- support setting sock_ops event flags via a socket iterator. This would
mean that the user can always set per-socket flags on
already-established sockets by iterating over existing sockets,
selecting those of interest.
- supporting setting event flags via setsockopt(). In fact we wouldn't
need to fully support setting event flags via "real" setsockopt(); we
could simply use a cgroup/setsockopt program and allow
bpf_sock_ops_cb_flags_set() to run in the cgroup/setsockopt context
(with additional checks to ensure it is indeed a tcp socket).

Do either/both of these seem reasonable, or is there a better way to
tackle this? Thanks!

Alan