Re: [PATCH] iotests: fix copy-before-write for macOS and FreeBSD

[Thomas Huth <thuth@redhat.com>]

On 05/07/2022 19.22, Richard Henderson wrote:
> On 7/5/22 21:07, Vladimir Sementsov-Ogievskiy wrote:
>> strerror() represents ETIMEDOUT a bit different in Linux and macOS /
>> FreeBSD. Let's support the latter too.
>>
>> Fixes: 9d05a87b77 ("iotests: copy-before-write: add cases for cbw-timeout 
>> option")
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
>> ---
>>
>> As John and Thomas noted, the new iotests fails for FreeBSD and maxOS.
>> Here is a fix. Would be great if someone can test it.
>>
>> I tried to push it by
>>
>>    git push --force  -o ci.variable="QEMU_CI=1"
>>
>> to my block branch, I get a blocked pipeline
>>    https://gitlab.com/vsementsov/qemu/-/pipelines/580573238
>> but it doesn't have neither freebsd nor macos jobs.. How to get them?
> 
> You'd have to have an attached cirrus token.
> Better to just use 'make vm-build-freebsd'.
> 
>>       def test_timeout_break_guest(self):
>>           log = self.do_cbw_timeout('break-guest-write')
>> +        # macOS and FreeBSD tend to represent ETIMEDOUT as
>> +        # "Operation timed out", when Linux prefer
>> +        # "Connection timed out"
>> +        log = log.replace('Operation timed out',
>> +                          'Connection timed out')
> 
> Um, really?  Matching strerror text?  This is ultra-fragile.
> Dare I say broken already.

Many of the iotests rely on output text matching. It's very fragile, always 
has been and always will be (unless we rewrite the whole test suite to not 
use output text matching anymore). For example, the iotests also do not work 
with the libc from Alpine Linux (where one of the error messages is "I/O 
error" instead of "Input/Output error"), so we only run check-unit and 
check-qtest in the CI there. It's a pity, but that's the way it is 
currently. I think it's still better to tweak some of the text strings here 
instead of not running the tests at all.

  Thomas