systemd-resolved and single-label DNS entries

systemd-resolved and single-label DNS entries

[Jeff Layton]

I'm hitting a problem with NFS testing on kdevops guests. Recently when
I bring up a guest, it's unable to resolve the name of the NFS server
via DNS.

When guests are brought up, kdevops currently registers their names in
DNS without any domain. For instance, the hostname "kdevops-nfsd" gets
a hostname in DNS of "kdevops-nfsd." (with no domain).

The DNS server on the KVM host will resolve a name like that just fine,
but systemd-resolved will not forward requests for those names via DNS.
It will only attempt a LLMNR request (which gets no response). I think
this is by design:

    https://github.com/systemd/systemd/issues/2514

I'm not sure of the best way to fix this. I can see a couple of
options:

1/ We could create a bogus domain name (.example.com? .kdevops?) and
put it in all of the search paths for the guests.

2/ Alternately, maybe we could go all-in on LLMNR and get that working.
I have zero experience with that, so I'm not sure what it entails. This
might also be problematic for guests that have no LLMNR lookup
capability.

-- 
Jeff Layton <jlayton@kernel.org>

Re: systemd-resolved and single-label DNS entries

[Daniel Gomez]

On 26/08/2025 12.58, Jeff Layton wrote:
> I'm hitting a problem with NFS testing on kdevops guests. Recently when
> I bring up a guest, it's unable to resolve the name of the NFS server
> via DNS.

FYI, nfstests works fine with Debian testing (forky). Both hosts and guests.

Re: systemd-resolved and single-label DNS entries

[Jeff Layton]

On Tue, 2025-08-26 at 14:53 +0200, Daniel Gomez wrote:
> On 26/08/2025 12.58, Jeff Layton wrote:
> > I'm hitting a problem with NFS testing on kdevops guests. Recently when
> > I bring up a guest, it's unable to resolve the name of the NFS server
> > via DNS.
> 
> FYI, nfstests works fine with Debian testing (forky). Both hosts and guests.
> 
> 


Thanks. Some questions:

1/ did your setup get the entries in /etc/hosts? That seems to be what
has changed recently. That's not working for some reason.

2/ Does forky use systemd-resolved? The problem is clearly there with
Fedora 42. The DNS server on the virtual bridge works fine:

----------------8<-----------------
$ dig @192.168.122.1 kdevops-nfsd.

; <<>> DiG 9.18.33 <<>> @192.168.122.1 kdevops-nfsd.
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24955
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;kdevops-nfsd.                  IN      A

;; ANSWER SECTION:
kdevops-nfsd.           0       IN      A       192.168.122.85

;; Query time: 0 msec
;; SERVER: 192.168.122.1#53(192.168.122.1) (UDP)
;; WHEN: Tue Aug 26 10:18:47 EDT 2025
;; MSG SIZE  rcvd: 57

$ resolvectl query kdevops-nfsd.
kdevops-nfsd.: 'kdevops-nfsd.' not found
----------------8<-----------------

It seems like we're doing something "unseemly" by putting bare names in
DNS like this, given that systemd-resolved won't pass them along.

Since systemd-resolved will only resolve bare names without a domain
using LLMNR, we should either take steps to make that work, or fix it
so that systemd-resolved will forward the requests (by setting up a
proper domain name and setting up the search path in the resolver).

-- 
Jeff Layton <jlayton@kernel.org>

Re: systemd-resolved and single-label DNS entries

[Chuck Lever]

On 8/26/25 10:21 AM, Jeff Layton wrote:
> On Tue, 2025-08-26 at 14:53 +0200, Daniel Gomez wrote:
>> On 26/08/2025 12.58, Jeff Layton wrote:
>>> I'm hitting a problem with NFS testing on kdevops guests. Recently when
>>> I bring up a guest, it's unable to resolve the name of the NFS server
>>> via DNS.
>>
>> FYI, nfstests works fine with Debian testing (forky). Both hosts and guests.
>>
>>
> 
> 
> Thanks. Some questions:
> 
> 1/ did your setup get the entries in /etc/hosts? That seems to be what
> has changed recently. That's not working for some reason.
> 
> 2/ Does forky use systemd-resolved? The problem is clearly there with
> Fedora 42. The DNS server on the virtual bridge works fine:
> 
> ----------------8<-----------------
> $ dig @192.168.122.1 kdevops-nfsd.
> 
> ; <<>> DiG 9.18.33 <<>> @192.168.122.1 kdevops-nfsd.
> ; (1 server found)
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24955
> ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
> 
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 1232
> ;; QUESTION SECTION:
> ;kdevops-nfsd.                  IN      A
> 
> ;; ANSWER SECTION:
> kdevops-nfsd.           0       IN      A       192.168.122.85
> 
> ;; Query time: 0 msec
> ;; SERVER: 192.168.122.1#53(192.168.122.1) (UDP)
> ;; WHEN: Tue Aug 26 10:18:47 EDT 2025
> ;; MSG SIZE  rcvd: 57
> 
> $ resolvectl query kdevops-nfsd.
> kdevops-nfsd.: 'kdevops-nfsd.' not found
> ----------------8<-----------------
> 
> It seems like we're doing something "unseemly" by putting bare names in
> DNS like this, given that systemd-resolved won't pass them along.
> 
> Since systemd-resolved will only resolve bare names without a domain
> using LLMNR, we should either take steps to make that work, or fix it
> so that systemd-resolved will forward the requests (by setting up a
> proper domain name and setting up the search path in the resolver).
> 

"won't pass them along" -- does that depend on the settings in
/etc/nsswitch.conf ?


-- 
Chuck Lever

Re: systemd-resolved and single-label DNS entries

[Jeff Layton]

On Tue, 2025-08-26 at 11:07 -0400, Chuck Lever wrote:
> On 8/26/25 10:21 AM, Jeff Layton wrote:
> > On Tue, 2025-08-26 at 14:53 +0200, Daniel Gomez wrote:
> > > On 26/08/2025 12.58, Jeff Layton wrote:
> > > > I'm hitting a problem with NFS testing on kdevops guests. Recently when
> > > > I bring up a guest, it's unable to resolve the name of the NFS server
> > > > via DNS.
> > > 
> > > FYI, nfstests works fine with Debian testing (forky). Both hosts and guests.
> > > 
> > > 
> > 
> > 
> > Thanks. Some questions:
> > 
> > 1/ did your setup get the entries in /etc/hosts? That seems to be what
> > has changed recently. That's not working for some reason.
> > 
> > 2/ Does forky use systemd-resolved? The problem is clearly there with
> > Fedora 42. The DNS server on the virtual bridge works fine:
> > 
> > ----------------8<-----------------
> > $ dig @192.168.122.1 kdevops-nfsd.
> > 
> > ; <<>> DiG 9.18.33 <<>> @192.168.122.1 kdevops-nfsd.
> > ; (1 server found)
> > ;; global options: +cmd
> > ;; Got answer:
> > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24955
> > ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
> > 
> > ;; OPT PSEUDOSECTION:
> > ; EDNS: version: 0, flags:; udp: 1232
> > ;; QUESTION SECTION:
> > ;kdevops-nfsd.                  IN      A
> > 
> > ;; ANSWER SECTION:
> > kdevops-nfsd.           0       IN      A       192.168.122.85
> > 
> > ;; Query time: 0 msec
> > ;; SERVER: 192.168.122.1#53(192.168.122.1) (UDP)
> > ;; WHEN: Tue Aug 26 10:18:47 EDT 2025
> > ;; MSG SIZE  rcvd: 57
> > 
> > $ resolvectl query kdevops-nfsd.
> > kdevops-nfsd.: 'kdevops-nfsd.' not found
> > ----------------8<-----------------
> > 
> > It seems like we're doing something "unseemly" by putting bare names in
> > DNS like this, given that systemd-resolved won't pass them along.
> > 
> > Since systemd-resolved will only resolve bare names without a domain
> > using LLMNR, we should either take steps to make that work, or fix it
> > so that systemd-resolved will forward the requests (by setting up a
> > proper domain name and setting up the search path in the resolver).
> > 
> 
> "won't pass them along" -- does that depend on the settings in
> /etc/nsswitch.conf ?
> 

No. In particular, see Lennart's comment here:

  https://github.com/systemd/systemd/issues/2514#issuecomment-334532052

I haven't looked though to see how to make systemd-resolved apply a
search list.
-- 
Jeff Layton <jlayton@kernel.org>

Re: systemd-resolved and single-label DNS entries

[Jeff Layton]

On Tue, 2025-08-26 at 11:07 -0400, Chuck Lever wrote:
> On 8/26/25 10:21 AM, Jeff Layton wrote:
> > On Tue, 2025-08-26 at 14:53 +0200, Daniel Gomez wrote:
> > > On 26/08/2025 12.58, Jeff Layton wrote:
> > > > I'm hitting a problem with NFS testing on kdevops guests. Recently when
> > > > I bring up a guest, it's unable to resolve the name of the NFS server
> > > > via DNS.
> > > 
> > > FYI, nfstests works fine with Debian testing (forky). Both hosts and guests.
> > > 
> > > 
> > 
> > 
> > Thanks. Some questions:
> > 
> > 1/ did your setup get the entries in /etc/hosts? That seems to be what
> > has changed recently. That's not working for some reason.
> > 
> > 2/ Does forky use systemd-resolved? The problem is clearly there with
> > Fedora 42. The DNS server on the virtual bridge works fine:
> > 
> > ----------------8<-----------------
> > $ dig @192.168.122.1 kdevops-nfsd.
> > 
> > ; <<>> DiG 9.18.33 <<>> @192.168.122.1 kdevops-nfsd.
> > ; (1 server found)
> > ;; global options: +cmd
> > ;; Got answer:
> > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24955
> > ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
> > 
> > ;; OPT PSEUDOSECTION:
> > ; EDNS: version: 0, flags:; udp: 1232
> > ;; QUESTION SECTION:
> > ;kdevops-nfsd.                  IN      A
> > 
> > ;; ANSWER SECTION:
> > kdevops-nfsd.           0       IN      A       192.168.122.85
> > 
> > ;; Query time: 0 msec
> > ;; SERVER: 192.168.122.1#53(192.168.122.1) (UDP)
> > ;; WHEN: Tue Aug 26 10:18:47 EDT 2025
> > ;; MSG SIZE  rcvd: 57
> > 
> > $ resolvectl query kdevops-nfsd.
> > kdevops-nfsd.: 'kdevops-nfsd.' not found
> > ----------------8<-----------------
> > 
> > It seems like we're doing something "unseemly" by putting bare names in
> > DNS like this, given that systemd-resolved won't pass them along.
> > 
> > Since systemd-resolved will only resolve bare names without a domain
> > using LLMNR, we should either take steps to make that work, or fix it
> > so that systemd-resolved will forward the requests (by setting up a
> > proper domain name and setting up the search path in the resolver).
> > 
> 
> "won't pass them along" -- does that depend on the settings in
> /etc/nsswitch.conf ?
> 

What's happening in my environment is that this list in the
update_etc_hosts playbook comes up empty:

- name: Build list of hosts                                                                    
  set_fact:       
    ueh_hosts: "{{ ansible_play_hosts_all | difference([inventory_hostname]) }}"
                       
That recently changed in this commit:

commit 1cf0800c9ffced98da393e5d2b1594302d23eb78
Author: Daniel Gomez <da.gomez@samsung.com>
Date:   Thu Jul 24 21:36:14 2025 +0200

    gen_hosts: templates: include localhost in the all group

I couldn't revert that commit cleanly, so I tried out the commit just
before it, and /etc/hosts got updated correctly. I'll have to look at
proper fixes for this, but if anyone has ideas I'm happy to try them
out.
-- 
Jeff Layton <jlayton@kernel.org>

Re: systemd-resolved and single-label DNS entries

[Daniel Gomez]

On 26/08/2025 18.32, Jeff Layton wrote:
> On Tue, 2025-08-26 at 11:07 -0400, Chuck Lever wrote:
>> On 8/26/25 10:21 AM, Jeff Layton wrote:
>>> On Tue, 2025-08-26 at 14:53 +0200, Daniel Gomez wrote:
>>>> On 26/08/2025 12.58, Jeff Layton wrote:
>>>>> I'm hitting a problem with NFS testing on kdevops guests. Recently when
>>>>> I bring up a guest, it's unable to resolve the name of the NFS server
>>>>> via DNS.
>>>>
>>>> FYI, nfstests works fine with Debian testing (forky). Both hosts and guests.
>>>>
>>>>
>>>
>>>
>>> Thanks. Some questions:
>>>
>>> 1/ did your setup get the entries in /etc/hosts? That seems to be what
>>> has changed recently. That's not working for some reason.
>>>
>>> 2/ Does forky use systemd-resolved? The problem is clearly there with
>>> Fedora 42. The DNS server on the virtual bridge works fine:
>>>
>>> ----------------8<-----------------
>>> $ dig @192.168.122.1 kdevops-nfsd.
>>>
>>> ; <<>> DiG 9.18.33 <<>> @192.168.122.1 kdevops-nfsd.
>>> ; (1 server found)
>>> ;; global options: +cmd
>>> ;; Got answer:
>>> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24955
>>> ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
>>>
>>> ;; OPT PSEUDOSECTION:
>>> ; EDNS: version: 0, flags:; udp: 1232
>>> ;; QUESTION SECTION:
>>> ;kdevops-nfsd.                  IN      A
>>>
>>> ;; ANSWER SECTION:
>>> kdevops-nfsd.           0       IN      A       192.168.122.85
>>>
>>> ;; Query time: 0 msec
>>> ;; SERVER: 192.168.122.1#53(192.168.122.1) (UDP)
>>> ;; WHEN: Tue Aug 26 10:18:47 EDT 2025
>>> ;; MSG SIZE  rcvd: 57
>>>
>>> $ resolvectl query kdevops-nfsd.
>>> kdevops-nfsd.: 'kdevops-nfsd.' not found
>>> ----------------8<-----------------
>>>
>>> It seems like we're doing something "unseemly" by putting bare names in
>>> DNS like this, given that systemd-resolved won't pass them along.
>>>
>>> Since systemd-resolved will only resolve bare names without a domain
>>> using LLMNR, we should either take steps to make that work, or fix it
>>> so that systemd-resolved will forward the requests (by setting up a
>>> proper domain name and setting up the search path in the resolver).
>>>
>>
>> "won't pass them along" -- does that depend on the settings in
>> /etc/nsswitch.conf ?
>>
> 
> What's happening in my environment is that this list in the
> update_etc_hosts playbook comes up empty:

Can you check if your ansible.cfg has the inventory field?

grep inventory ansible.cfg
inventory = /xfs1/dagomez/kdevops/hosts

> 
> - name: Build list of hosts                                                                    
>   set_fact:       
>     ueh_hosts: "{{ ansible_play_hosts_all | difference([inventory_hostname]) }}"
>                        
> That recently changed in this commit:
> 
> commit 1cf0800c9ffced98da393e5d2b1594302d23eb78
> Author: Daniel Gomez <da.gomez@samsung.com>
> Date:   Thu Jul 24 21:36:14 2025 +0200
> 
>     gen_hosts: templates: include localhost in the all group
> 
> I couldn't revert that commit cleanly, so I tried out the commit just
> before it, and /etc/hosts got updated correctly. I'll have to look at
> proper fixes for this, but if anyone has ideas I'm happy to try them
> out.

My issue was that the list included the localhosts as it was now part of hosts
file. In the commit log I added:

    Finally, update ueh_hosts in the update_etc_hosts role to include only
    the hosts targeted in the current play, rather than all hosts in the
    inventory. This avoids the error below when the inventory includes
    localhost in the all group:

    "The task includes an option with an undefined variable. The error
    was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute
    'ansible_all_ipv4_addresses'. 'ansible.vars.hostvars.HostVarsVars
    object' has no attribute 'ansible_all_ipv4_addresses'"

And this is the commit change related to the ueh_hosts:

diff --git a/playbooks/roles/update_etc_hosts/tasks/main.yml b/playbooks/roles/update_etc_hosts/tasks/main.yml
index 4c1307fb..2b9af91d 100644
--- a/playbooks/roles/update_etc_hosts/tasks/main.yml
+++ b/playbooks/roles/update_etc_hosts/tasks/main.yml
@@ -26,7 +26,7 @@

 - name: Build list of hosts
   set_fact:
-    ueh_hosts: "{{ groups.all | flatten(levels=1) | reject('eq', inventory_hostname) }}"
+    ueh_hosts: "{{ ansible_play_hosts_all | difference([inventory_hostname]) }}"


Below are the logs with current vanilla kdevops (1.) and logs reverting the
changes above (2.):

1. make bringup AV=2 V=1
...

TASK [update_etc_hosts : Gather network facts] *********************************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:16
ok: [debian13-posix]
ok: [debian13-dio]
ok: [debian13-interop]
 [started TASK: update_etc_hosts : Build network address on debian13-dio]
 [started TASK: update_etc_hosts : Build network address on debian13-interop]
 [started TASK: update_etc_hosts : Build network address on debian13-posix]
 [started TASK: update_etc_hosts : Build list of hosts on debian13-dio]
 [started TASK: update_etc_hosts : Build list of hosts on debian13-interop]
 [started TASK: update_etc_hosts : Build list of hosts on debian13-posix]

TASK [update_etc_hosts : Build list of hosts] **********************************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:27
ok: [debian13-dio] => {"ansible_facts": {"ueh_hosts": ["debian13-interop", "debian13-posix"]}, "changed": false}
ok: [debian13-interop] => {"ansible_facts": {"ueh_hosts": ["debian13-dio", "debian13-posix"]}, "changed": false}
ok: [debian13-posix] => {"ansible_facts": {"ueh_hosts": ["debian13-dio", "debian13-interop"]}, "changed": false}
 [started TASK: update_etc_hosts : ansible.builtin.debug on debian13-dio]
 [started TASK: update_etc_hosts : ansible.builtin.debug on debian13-interop]
 [started TASK: update_etc_hosts : ansible.builtin.debug on debian13-posix]

TASK [update_etc_hosts : ansible.builtin.debug] ********************************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:31
ok: [debian13-dio] => {
    "ueh_hosts": [
        "debian13-interop",
        "debian13-posix"
    ]
}
ok: [debian13-interop] => {
    "ueh_hosts": [
        "debian13-dio",
        "debian13-posix"
    ]
}
ok: [debian13-posix] => {
    "ueh_hosts": [
        "debian13-dio",
        "debian13-interop"
    ]
}
 [started TASK: update_etc_hosts : Stat /etc/cloud/cloud.cfg.d on debian13-dio]
 [started TASK: update_etc_hosts : Stat /etc/cloud/cloud.cfg.d on debian13-interop]
 [started TASK: update_etc_hosts : Stat /etc/cloud/cloud.cfg.d on debian13-posix]

TASK [update_etc_hosts : Stat /etc/cloud/cloud.cfg.d] **************************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:34
ok: [debian13-dio] => {"changed": false, "stat": {"atime": 1756234164.14, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "binary", "ctime": 1756234453.4407628, "dev": 65025, "device_type": 0, "executable": true, "exists": true, "gid": 0, "gr_name": "root", "inode": 16529, "isblk": false, "ischr": false, "isdir": true, "isfifo": false, "isgid": false, "islnk": false, "isreg": false, "issock": false, "isuid": false, "mimetype": "inode/directory", "mode": "0755", "mtime": 1756234453.4407628, "nlink": 2, "path": "/etc/cloud/cloud.cfg.d", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 4096, "uid": 0, "version": "612645937", "wgrp": false, "woth": false, "writeable": false, "wusr": true, "xgrp": true, "xoth": true, "xusr": true}}
ok: [debian13-posix] => {"changed": false, "stat": {"atime": 1756234164.664, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "binary", "ctime": 1756234451.0282118, "dev": 65025, "device_type": 0, "executable": true, "exists": true, "gid": 0, "gr_name": "root", "inode": 16529, "isblk": false, "ischr": false, "isdir": true, "isfifo": false, "isgid": false, "islnk": false, "isreg": false, "issock": false, "isuid": false, "mimetype": "inode/directory", "mode": "0755", "mtime": 1756234451.0282118, "nlink": 2, "path": "/etc/cloud/cloud.cfg.d", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 4096, "uid": 0, "version": "612645937", "wgrp": false, "woth": false, "writeable": false, "wusr": true, "xgrp": true, "xoth": true, "xusr": true}}
ok: [debian13-interop] => {"changed": false, "stat": {"atime": 1756234164.104, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "binary", "ctime": 1756234449.885939, "dev": 65025, "device_type": 0, "executable": true, "exists": true, "gid": 0, "gr_name": "root", "inode": 16529, "isblk": false, "ischr": false, "isdir": true, "isfifo": false, "isgid": false, "islnk": false, "isreg": false, "issock": false, "isuid": false, "mimetype": "inode/directory", "mode": "0755", "mtime": 1756234449.885939, "nlink": 2, "path": "/etc/cloud/cloud.cfg.d", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 4096, "uid": 0, "version": "612645937", "wgrp": false, "woth": false, "writeable": false, "wusr": true, "xgrp": true, "xoth": true, "xusr": true}}
 [started TASK: update_etc_hosts : Disable cloud-init hosts management on debian13-dio]
 [started TASK: update_etc_hosts : Disable cloud-init hosts management on debian13-interop]
 [started TASK: update_etc_hosts : Disable cloud-init hosts management on debian13-posix]

TASK [update_etc_hosts : Disable cloud-init hosts management] ******************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:39
changed: [debian13-dio] => {"backup": "", "changed": true, "msg": "line added"}
changed: [debian13-posix] => {"backup": "", "changed": true, "msg": "line added"}
changed: [debian13-interop] => {"backup": "", "changed": true, "msg": "line added"}
 [started TASK: update_etc_hosts : Add IP address of all hosts to all hosts on debian13-dio]
 [started TASK: update_etc_hosts : Add IP address of all hosts to all hosts on debian13-interop]
 [started TASK: update_etc_hosts : Add IP address of all hosts to all hosts on debian13-posix]
 [started TASK: update_etc_hosts : Add IP address of all hosts to all hosts on debian13-dio]
 [started TASK: update_etc_hosts : Add IP address of all hosts to all hosts on debian13-interop]
 [started TASK: update_etc_hosts : Add IP address of all hosts to all hosts on debian13-posix]

TASK [update_etc_hosts : Add IP address of all hosts to all hosts] *************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:61
changed: [debian13-dio] => (item=debian13-interop) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-interop", "msg": "line added"}
changed: [debian13-interop] => (item=debian13-dio) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-dio", "msg": "line added"}
changed: [debian13-posix] => (item=debian13-dio) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-dio", "msg": "line added"}
changed: [debian13-dio] => (item=debian13-posix) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-posix", "msg": "line added"}
changed: [debian13-interop] => (item=debian13-posix) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-posix", "msg": "line added"}
changed: [debian13-posix] => (item=debian13-interop) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-interop", "msg": "line added"}
 [started TASK: update_etc_hosts : Fix up hostname on Debian guestfs hosts on debian13-dio]
 [started TASK: update_etc_hosts : Fix up hostname on Debian guestfs hosts on debian13-interop]
 [started TASK: update_etc_hosts : Fix up hostname on Debian guestfs hosts on debian13-posix]

TASK [update_etc_hosts : Fix up hostname on Debian guestfs hosts] **************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:73
ok: [debian13-posix] => {"backup": "", "changed": false, "msg": ""}
ok: [debian13-dio] => {"backup": "", "changed": false, "msg": ""}
ok: [debian13-interop] => {"backup": "", "changed": false, "msg": ""}

PLAY RECAP *********************************************************************
debian13-dio               : ok=9    changed=2    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
debian13-interop           : ok=9    changed=2    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
debian13-posix             : ok=9    changed=2    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   


2. After reverting that change on ueh_hosts, I can replicate my original issue:

TASK [update_etc_hosts : Gather network facts] *******************************************************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:16
ok: [debian13-dio]
ok: [debian13-posix]
ok: [debian13-interop]
 [started TASK: update_etc_hosts : Build network address on debian13-dio]
 [started TASK: update_etc_hosts : Build network address on debian13-interop]
 [started TASK: update_etc_hosts : Build network address on debian13-posix]
 [started TASK: update_etc_hosts : Build list of hosts on debian13-dio]
 [started TASK: update_etc_hosts : Build list of hosts on debian13-interop]
 [started TASK: update_etc_hosts : Build list of hosts on debian13-posix]

TASK [update_etc_hosts : Build list of hosts] ********************************************************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:27
ok: [debian13-dio] => {"ansible_facts": {"ueh_hosts": ["localhost", "debian13-interop", "debian13-posix", "debian13-iscsi", "debian13-nfsd"]}, "changed": false}
ok: [debian13-posix] => {"ansible_facts": {"ueh_hosts": ["localhost", "debian13-dio", "debian13-interop", "debian13-iscsi", "debian13-nfsd"]}, "changed": false}
ok: [debian13-interop] => {"ansible_facts": {"ueh_hosts": ["localhost", "debian13-dio", "debian13-posix", "debian13-iscsi", "debian13-nfsd"]}, "changed": false}
 [started TASK: update_etc_hosts : ansible.builtin.debug on debian13-dio]
 [started TASK: update_etc_hosts : ansible.builtin.debug on debian13-interop]
 [started TASK: update_etc_hosts : ansible.builtin.debug on debian13-posix]

TASK [update_etc_hosts : ansible.builtin.debug] ******************************************************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:32
ok: [debian13-dio] => {
    "ueh_hosts": [
        "localhost",
        "debian13-interop",
        "debian13-posix",
        "debian13-iscsi",
        "debian13-nfsd"
    ]
}
ok: [debian13-interop] => {
    "ueh_hosts": [
        "localhost",
        "debian13-dio",
        "debian13-posix",
        "debian13-iscsi",
        "debian13-nfsd"
    ]
}
ok: [debian13-posix] => {
    "ueh_hosts": [
        "localhost",
        "debian13-dio",
        "debian13-interop",
        "debian13-iscsi",
        "debian13-nfsd"
    ]
}
 [started TASK: update_etc_hosts : Stat /etc/cloud/cloud.cfg.d on debian13-dio]
 [started TASK: update_etc_hosts : Stat /etc/cloud/cloud.cfg.d on debian13-interop]
 [started TASK: update_etc_hosts : Stat /etc/cloud/cloud.cfg.d on debian13-posix]

...

TASK [update_etc_hosts : Add IP address of all hosts to all hosts] ***********************************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:62
[ERROR]: Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'

Task failed.
Origin: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:62:3

60     - terraform_private_net_enabled
61
62 - name: Add IP address of all hosts to all hosts
     ^ column 3

<<< caused by >>>

Finalization of task args for 'ansible.builtin.lineinfile' failed.
Origin: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:65:3

63   become: true
64   become_method: sudo
65   ansible.builtin.lineinfile:
     ^ column 3

<<< caused by >>>

Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'
Origin: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:68:11

66     dest: /etc/hosts
67     regexp: ".*{{ item }}$"
68     line: "{{ hostvars[item].ansible_all_ipv4_addresses | first }} {{ item }}"
             ^ column 11

failed: [debian13-dio] (item=localhost) => {"ansible_loop_var": "item", "changed": false, "item": "localhost", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
failed: [debian13-interop] (item=localhost) => {"ansible_loop_var": "item", "changed": false, "item": "localhost", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
failed: [debian13-posix] (item=localhost) => {"ansible_loop_var": "item", "changed": false, "item": "localhost", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
changed: [debian13-dio] => (item=debian13-interop) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-interop", "msg": "line added"}
changed: [debian13-interop] => (item=debian13-dio) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-dio", "msg": "line added"}
changed: [debian13-posix] => (item=debian13-dio) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-dio", "msg": "line added"}
changed: [debian13-dio] => (item=debian13-posix) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-posix", "msg": "line added"}
failed: [debian13-dio] (item=debian13-iscsi) => {"ansible_loop_var": "item", "changed": false, "item": "debian13-iscsi", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
changed: [debian13-interop] => (item=debian13-posix) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-posix", "msg": "line added"}
changed: [debian13-posix] => (item=debian13-interop) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-interop", "msg": "line added"}
failed: [debian13-dio] (item=debian13-nfsd) => {"ansible_loop_var": "item", "changed": false, "item": "debian13-nfsd", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
failed: [debian13-interop] (item=debian13-iscsi) => {"ansible_loop_var": "item", "changed": false, "item": "debian13-iscsi", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
failed: [debian13-posix] (item=debian13-iscsi) => {"ansible_loop_var": "item", "changed": false, "item": "debian13-iscsi", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
failed: [debian13-interop] (item=debian13-nfsd) => {"ansible_loop_var": "item", "changed": false, "item": "debian13-nfsd", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
failed: [debian13-posix] (item=debian13-nfsd) => {"ansible_loop_var": "item", "changed": false, "item": "debian13-nfsd", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}


Let me know if any of this helps.

Re: systemd-resolved and single-label DNS entries

[Jeff Layton]

On Tue, 2025-08-26 at 21:04 +0200, Daniel Gomez wrote:
> 
> On 26/08/2025 18.32, Jeff Layton wrote:
> > On Tue, 2025-08-26 at 11:07 -0400, Chuck Lever wrote:
> > > On 8/26/25 10:21 AM, Jeff Layton wrote:
> > > > On Tue, 2025-08-26 at 14:53 +0200, Daniel Gomez wrote:
> > > > > On 26/08/2025 12.58, Jeff Layton wrote:
> > > > > > I'm hitting a problem with NFS testing on kdevops guests. Recently when
> > > > > > I bring up a guest, it's unable to resolve the name of the NFS server
> > > > > > via DNS.
> > > > > 
> > > > > FYI, nfstests works fine with Debian testing (forky). Both hosts and guests.
> > > > > 
> > > > > 
> > > > 
> > > > 
> > > > Thanks. Some questions:
> > > > 
> > > > 1/ did your setup get the entries in /etc/hosts? That seems to be what
> > > > has changed recently. That's not working for some reason.
> > > > 
> > > > 2/ Does forky use systemd-resolved? The problem is clearly there with
> > > > Fedora 42. The DNS server on the virtual bridge works fine:
> > > > 
> > > > ----------------8<-----------------
> > > > $ dig @192.168.122.1 kdevops-nfsd.
> > > > 
> > > > ; <<>> DiG 9.18.33 <<>> @192.168.122.1 kdevops-nfsd.
> > > > ; (1 server found)
> > > > ;; global options: +cmd
> > > > ;; Got answer:
> > > > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24955
> > > > ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
> > > > 
> > > > ;; OPT PSEUDOSECTION:
> > > > ; EDNS: version: 0, flags:; udp: 1232
> > > > ;; QUESTION SECTION:
> > > > ;kdevops-nfsd.                  IN      A
> > > > 
> > > > ;; ANSWER SECTION:
> > > > kdevops-nfsd.           0       IN      A       192.168.122.85
> > > > 
> > > > ;; Query time: 0 msec
> > > > ;; SERVER: 192.168.122.1#53(192.168.122.1) (UDP)
> > > > ;; WHEN: Tue Aug 26 10:18:47 EDT 2025
> > > > ;; MSG SIZE  rcvd: 57
> > > > 
> > > > $ resolvectl query kdevops-nfsd.
> > > > kdevops-nfsd.: 'kdevops-nfsd.' not found
> > > > ----------------8<-----------------
> > > > 
> > > > It seems like we're doing something "unseemly" by putting bare names in
> > > > DNS like this, given that systemd-resolved won't pass them along.
> > > > 
> > > > Since systemd-resolved will only resolve bare names without a domain
> > > > using LLMNR, we should either take steps to make that work, or fix it
> > > > so that systemd-resolved will forward the requests (by setting up a
> > > > proper domain name and setting up the search path in the resolver).
> > > > 
> > > 
> > > "won't pass them along" -- does that depend on the settings in
> > > /etc/nsswitch.conf ?
> > > 
> > 
> > What's happening in my environment is that this list in the
> > update_etc_hosts playbook comes up empty:
> 
> Can you check if your ansible.cfg has the inventory field?
> 
> grep inventory ansible.cfg
> inventory = /xfs1/dagomez/kdevops/hosts
> 

Yep, it's there:

inventory = /home/jlayton/git/kdevops/hosts

Here's the file itself for my current (2 client) setup:

-------------8<--------------
[all]
localhost ansible_connection=local
kdevops-nfs-v42
kdevops-nfs-v3
[all:vars]
ansible_python_interpreter =  "/usr/bin/python3"

[baseline]
kdevops-nfs-v42
kdevops-nfs-v3
[baseline:vars]
ansible_python_interpreter =  "/usr/bin/python3"

[dev]
[dev:vars]
ansible_python_interpreter =  "/usr/bin/python3"

[nfsd]
kdevops-nfsd
[nfsd:vars]
ansible_python_interpreter =  "/usr/bin/python3"
[service]
kdevops-nfsd
[service:vars]
ansible_python_interpreter =  "/usr/bin/python3"
-------------8<--------------

I have a bit more info too: If I spin up 2 client hosts and a single
server, then the clients each get an entry for the other clients. They
just don't get one for the server itself.

For instance:

$ ssh kdevops-nfs-v3 grep -v '^#' /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.122.160 kdevops-nfs-v42

...so I think we just need to make sure that the server hosts are also
included in the list. I'll tinker around some more today, and will
hopefully spin up a patch.

> > 
> > - name: Build list of hosts                                                                    
> >   set_fact:       
> >     ueh_hosts: "{{ ansible_play_hosts_all | difference([inventory_hostname]) }}"
> >                        
> > That recently changed in this commit:
> > 
> > commit 1cf0800c9ffced98da393e5d2b1594302d23eb78
> > Author: Daniel Gomez <da.gomez@samsung.com>
> > Date:   Thu Jul 24 21:36:14 2025 +0200
> > 
> >     gen_hosts: templates: include localhost in the all group
> > 
> > I couldn't revert that commit cleanly, so I tried out the commit just
> > before it, and /etc/hosts got updated correctly. I'll have to look at
> > proper fixes for this, but if anyone has ideas I'm happy to try them
> > out.
> 
> My issue was that the list included the localhosts as it was now part of hosts
> file. In the commit log I added:
> 
>     Finally, update ueh_hosts in the update_etc_hosts role to include only
>     the hosts targeted in the current play, rather than all hosts in the
>     inventory. This avoids the error below when the inventory includes
>     localhost in the all group:
> 
>     "The task includes an option with an undefined variable. The error
>     was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute
>     'ansible_all_ipv4_addresses'. 'ansible.vars.hostvars.HostVarsVars
>     object' has no attribute 'ansible_all_ipv4_addresses'"
> 
> And this is the commit change related to the ueh_hosts:
> 
> diff --git a/playbooks/roles/update_etc_hosts/tasks/main.yml b/playbooks/roles/update_etc_hosts/tasks/main.yml
> index 4c1307fb..2b9af91d 100644
> --- a/playbooks/roles/update_etc_hosts/tasks/main.yml
> +++ b/playbooks/roles/update_etc_hosts/tasks/main.yml
> @@ -26,7 +26,7 @@
> 
>  - name: Build list of hosts
>    set_fact:
> -    ueh_hosts: "{{ groups.all | flatten(levels=1) | reject('eq', inventory_hostname) }}"
> +    ueh_hosts: "{{ ansible_play_hosts_all | difference([inventory_hostname]) }}"
> 
> 
> Below are the logs with current vanilla kdevops (1.) and logs reverting the
> changes above (2.):
> 
> 1. make bringup AV=2 V=1
> ...
> 
> TASK [update_etc_hosts : Gather network facts] *********************************
> task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:16
> ok: [debian13-posix]
> ok: [debian13-dio]
> ok: [debian13-interop]
>  [started TASK: update_etc_hosts : Build network address on debian13-dio]
>  [started TASK: update_etc_hosts : Build network address on debian13-interop]
>  [started TASK: update_etc_hosts : Build network address on debian13-posix]
>  [started TASK: update_etc_hosts : Build list of hosts on debian13-dio]
>  [started TASK: update_etc_hosts : Build list of hosts on debian13-interop]
>  [started TASK: update_etc_hosts : Build list of hosts on debian13-posix]
> 
> TASK [update_etc_hosts : Build list of hosts] **********************************
> task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:27
> ok: [debian13-dio] => {"ansible_facts": {"ueh_hosts": ["debian13-interop", "debian13-posix"]}, "changed": false}
> ok: [debian13-interop] => {"ansible_facts": {"ueh_hosts": ["debian13-dio", "debian13-posix"]}, "changed": false}
> ok: [debian13-posix] => {"ansible_facts": {"ueh_hosts": ["debian13-dio", "debian13-interop"]}, "changed": false}
>  [started TASK: update_etc_hosts : ansible.builtin.debug on debian13-dio]
>  [started TASK: update_etc_hosts : ansible.builtin.debug on debian13-interop]
>  [started TASK: update_etc_hosts : ansible.builtin.debug on debian13-posix]
> 
> TASK [update_etc_hosts : ansible.builtin.debug] ********************************
> task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:31
> ok: [debian13-dio] => {
>     "ueh_hosts": [
>         "debian13-interop",
>         "debian13-posix"
>     ]
> }
> ok: [debian13-interop] => {
>     "ueh_hosts": [
>         "debian13-dio",
>         "debian13-posix"
>     ]
> }
> ok: [debian13-posix] => {
>     "ueh_hosts": [
>         "debian13-dio",
>         "debian13-interop"
>     ]
> }
>  [started TASK: update_etc_hosts : Stat /etc/cloud/cloud.cfg.d on debian13-dio]
>  [started TASK: update_etc_hosts : Stat /etc/cloud/cloud.cfg.d on debian13-interop]
>  [started TASK: update_etc_hosts : Stat /etc/cloud/cloud.cfg.d on debian13-posix]
> 
> TASK [update_etc_hosts : Stat /etc/cloud/cloud.cfg.d] **************************
> task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:34
> ok: [debian13-dio] => {"changed": false, "stat": {"atime": 1756234164.14, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "binary", "ctime": 1756234453.4407628, "dev": 65025, "device_type": 0, "executable": true, "exists": true, "gid": 0, "gr_name": "root", "inode": 16529, "isblk": false, "ischr": false, "isdir": true, "isfifo": false, "isgid": false, "islnk": false, "isreg": false, "issock": false, "isuid": false, "mimetype": "inode/directory", "mode": "0755", "mtime": 1756234453.4407628, "nlink": 2, "path": "/etc/cloud/cloud.cfg.d", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 4096, "uid": 0, "version": "612645937", "wgrp": false, "woth": false, "writeable": false, "wusr": true, "xgrp": true, "xoth": true, "xusr": true}}
> ok: [debian13-posix] => {"changed": false, "stat": {"atime": 1756234164.664, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "binary", "ctime": 1756234451.0282118, "dev": 65025, "device_type": 0, "executable": true, "exists": true, "gid": 0, "gr_name": "root", "inode": 16529, "isblk": false, "ischr": false, "isdir": true, "isfifo": false, "isgid": false, "islnk": false, "isreg": false, "issock": false, "isuid": false, "mimetype": "inode/directory", "mode": "0755", "mtime": 1756234451.0282118, "nlink": 2, "path": "/etc/cloud/cloud.cfg.d", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 4096, "uid": 0, "version": "612645937", "wgrp": false, "woth": false, "writeable": false, "wusr": true, "xgrp": true, "xoth": true, "xusr": true}}
> ok: [debian13-interop] => {"changed": false, "stat": {"atime": 1756234164.104, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "binary", "ctime": 1756234449.885939, "dev": 65025, "device_type": 0, "executable": true, "exists": true, "gid": 0, "gr_name": "root", "inode": 16529, "isblk": false, "ischr": false, "isdir": true, "isfifo": false, "isgid": false, "islnk": false, "isreg": false, "issock": false, "isuid": false, "mimetype": "inode/directory", "mode": "0755", "mtime": 1756234449.885939, "nlink": 2, "path": "/etc/cloud/cloud.cfg.d", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 4096, "uid": 0, "version": "612645937", "wgrp": false, "woth": false, "writeable": false, "wusr": true, "xgrp": true, "xoth": true, "xusr": true}}
>  [started TASK: update_etc_hosts : Disable cloud-init hosts management on debian13-dio]
>  [started TASK: update_etc_hosts : Disable cloud-init hosts management on debian13-interop]
>  [started TASK: update_etc_hosts : Disable cloud-init hosts management on debian13-posix]
> 
> TASK [update_etc_hosts : Disable cloud-init hosts management] ******************
> task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:39
> changed: [debian13-dio] => {"backup": "", "changed": true, "msg": "line added"}
> changed: [debian13-posix] => {"backup": "", "changed": true, "msg": "line added"}
> changed: [debian13-interop] => {"backup": "", "changed": true, "msg": "line added"}
>  [started TASK: update_etc_hosts : Add IP address of all hosts to all hosts on debian13-dio]
>  [started TASK: update_etc_hosts : Add IP address of all hosts to all hosts on debian13-interop]
>  [started TASK: update_etc_hosts : Add IP address of all hosts to all hosts on debian13-posix]
>  [started TASK: update_etc_hosts : Add IP address of all hosts to all hosts on debian13-dio]
>  [started TASK: update_etc_hosts : Add IP address of all hosts to all hosts on debian13-interop]
>  [started TASK: update_etc_hosts : Add IP address of all hosts to all hosts on debian13-posix]
> 
> TASK [update_etc_hosts : Add IP address of all hosts to all hosts] *************
> task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:61
> changed: [debian13-dio] => (item=debian13-interop) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-interop", "msg": "line added"}
> changed: [debian13-interop] => (item=debian13-dio) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-dio", "msg": "line added"}
> changed: [debian13-posix] => (item=debian13-dio) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-dio", "msg": "line added"}
> changed: [debian13-dio] => (item=debian13-posix) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-posix", "msg": "line added"}
> changed: [debian13-interop] => (item=debian13-posix) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-posix", "msg": "line added"}
> changed: [debian13-posix] => (item=debian13-interop) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-interop", "msg": "line added"}
>  [started TASK: update_etc_hosts : Fix up hostname on Debian guestfs hosts on debian13-dio]
>  [started TASK: update_etc_hosts : Fix up hostname on Debian guestfs hosts on debian13-interop]
>  [started TASK: update_etc_hosts : Fix up hostname on Debian guestfs hosts on debian13-posix]
> 
> TASK [update_etc_hosts : Fix up hostname on Debian guestfs hosts] **************
> task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:73
> ok: [debian13-posix] => {"backup": "", "changed": false, "msg": ""}
> ok: [debian13-dio] => {"backup": "", "changed": false, "msg": ""}
> ok: [debian13-interop] => {"backup": "", "changed": false, "msg": ""}
> 
> PLAY RECAP *********************************************************************
> debian13-dio               : ok=9    changed=2    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
> debian13-interop           : ok=9    changed=2    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
> debian13-posix             : ok=9    changed=2    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
> 
> 
> 2. After reverting that change on ueh_hosts, I can replicate my original issue:
> 
> TASK [update_etc_hosts : Gather network facts] *******************************************************
> task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:16
> ok: [debian13-dio]
> ok: [debian13-posix]
> ok: [debian13-interop]
>  [started TASK: update_etc_hosts : Build network address on debian13-dio]
>  [started TASK: update_etc_hosts : Build network address on debian13-interop]
>  [started TASK: update_etc_hosts : Build network address on debian13-posix]
>  [started TASK: update_etc_hosts : Build list of hosts on debian13-dio]
>  [started TASK: update_etc_hosts : Build list of hosts on debian13-interop]
>  [started TASK: update_etc_hosts : Build list of hosts on debian13-posix]
> 
> TASK [update_etc_hosts : Build list of hosts] ********************************************************
> task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:27
> ok: [debian13-dio] => {"ansible_facts": {"ueh_hosts": ["localhost", "debian13-interop", "debian13-posix", "debian13-iscsi", "debian13-nfsd"]}, "changed": false}
> ok: [debian13-posix] => {"ansible_facts": {"ueh_hosts": ["localhost", "debian13-dio", "debian13-interop", "debian13-iscsi", "debian13-nfsd"]}, "changed": false}
> ok: [debian13-interop] => {"ansible_facts": {"ueh_hosts": ["localhost", "debian13-dio", "debian13-posix", "debian13-iscsi", "debian13-nfsd"]}, "changed": false}
>  [started TASK: update_etc_hosts : ansible.builtin.debug on debian13-dio]
>  [started TASK: update_etc_hosts : ansible.builtin.debug on debian13-interop]
>  [started TASK: update_etc_hosts : ansible.builtin.debug on debian13-posix]
> 
> TASK [update_etc_hosts : ansible.builtin.debug] ******************************************************
> task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:32
> ok: [debian13-dio] => {
>     "ueh_hosts": [
>         "localhost",
>         "debian13-interop",
>         "debian13-posix",
>         "debian13-iscsi",
>         "debian13-nfsd"
>     ]
> }
> ok: [debian13-interop] => {
>     "ueh_hosts": [
>         "localhost",
>         "debian13-dio",
>         "debian13-posix",
>         "debian13-iscsi",
>         "debian13-nfsd"
>     ]
> }
> ok: [debian13-posix] => {
>     "ueh_hosts": [
>         "localhost",
>         "debian13-dio",
>         "debian13-interop",
>         "debian13-iscsi",
>         "debian13-nfsd"
>     ]
> }
>  [started TASK: update_etc_hosts : Stat /etc/cloud/cloud.cfg.d on debian13-dio]
>  [started TASK: update_etc_hosts : Stat /etc/cloud/cloud.cfg.d on debian13-interop]
>  [started TASK: update_etc_hosts : Stat /etc/cloud/cloud.cfg.d on debian13-posix]
> 
> ...
> 
> TASK [update_etc_hosts : Add IP address of all hosts to all hosts] ***********************************
> task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:62
> [ERROR]: Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'
> 
> Task failed.
> Origin: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:62:3
> 
> 60     - terraform_private_net_enabled
> 61
> 62 - name: Add IP address of all hosts to all hosts
>      ^ column 3
> 
> <<< caused by >>>
> 
> Finalization of task args for 'ansible.builtin.lineinfile' failed.
> Origin: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:65:3
> 
> 63   become: true
> 64   become_method: sudo
> 65   ansible.builtin.lineinfile:
>      ^ column 3
> 
> <<< caused by >>>
> 
> Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'
> Origin: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:68:11
> 
> 66     dest: /etc/hosts
> 67     regexp: ".*{{ item }}$"
> 68     line: "{{ hostvars[item].ansible_all_ipv4_addresses | first }} {{ item }}"
>              ^ column 11
> 
> failed: [debian13-dio] (item=localhost) => {"ansible_loop_var": "item", "changed": false, "item": "localhost", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
> failed: [debian13-interop] (item=localhost) => {"ansible_loop_var": "item", "changed": false, "item": "localhost", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
> failed: [debian13-posix] (item=localhost) => {"ansible_loop_var": "item", "changed": false, "item": "localhost", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
> changed: [debian13-dio] => (item=debian13-interop) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-interop", "msg": "line added"}
> changed: [debian13-interop] => (item=debian13-dio) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-dio", "msg": "line added"}
> changed: [debian13-posix] => (item=debian13-dio) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-dio", "msg": "line added"}
> changed: [debian13-dio] => (item=debian13-posix) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-posix", "msg": "line added"}
> failed: [debian13-dio] (item=debian13-iscsi) => {"ansible_loop_var": "item", "changed": false, "item": "debian13-iscsi", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
> changed: [debian13-interop] => (item=debian13-posix) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-posix", "msg": "line added"}
> changed: [debian13-posix] => (item=debian13-interop) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-interop", "msg": "line added"}
> failed: [debian13-dio] (item=debian13-nfsd) => {"ansible_loop_var": "item", "changed": false, "item": "debian13-nfsd", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
> failed: [debian13-interop] (item=debian13-iscsi) => {"ansible_loop_var": "item", "changed": false, "item": "debian13-iscsi", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
> failed: [debian13-posix] (item=debian13-iscsi) => {"ansible_loop_var": "item", "changed": false, "item": "debian13-iscsi", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
> failed: [debian13-interop] (item=debian13-nfsd) => {"ansible_loop_var": "item", "changed": false, "item": "debian13-nfsd", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
> failed: [debian13-posix] (item=debian13-nfsd) => {"ansible_loop_var": "item", "changed": false, "item": "debian13-nfsd", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
> 
> 
> Let me know if any of this helps.

I wasn't advocating that we revert anything. I'm sure you had good
reason to make the change you did. We just need to make sure that we
get the NFS server included in /etc/hosts update. I'll see what I can
figure out.

Thanks for the help!
-- 
Jeff Layton <jlayton@kernel.org>

Re: systemd-resolved and single-label DNS entries

[Daniel Gomez]

On 26/08/2025 16.21, Jeff Layton wrote:
> On Tue, 2025-08-26 at 14:53 +0200, Daniel Gomez wrote:
>> On 26/08/2025 12.58, Jeff Layton wrote:
>>> I'm hitting a problem with NFS testing on kdevops guests. Recently when
>>> I bring up a guest, it's unable to resolve the name of the NFS server
>>> via DNS.
>>
>> FYI, nfstests works fine with Debian testing (forky). Both hosts and guests.
>>
>>
> 
> 
> Thanks. Some questions:
> 
> 1/ did your setup get the entries in /etc/hosts? That seems to be what

No. AFAIK, we don't set up anything in the /etc/hosts. At least, the task
below does not run during my bringup.

- name: Fix up hostname on Debian guestfs hosts


> has changed recently. That's not working for some reason.
> 
> 2/ Does forky use systemd-resolved? The problem is clearly there with
> Fedora 42. The DNS server on the virtual bridge works fine:

It does not use systemd-resolved. At least, my system has been updated from
trixie to forky and did not get that installed.

> 
> ----------------8<-----------------
> $ dig @192.168.122.1 kdevops-nfsd.
> 
> ; <<>> DiG 9.18.33 <<>> @192.168.122.1 kdevops-nfsd.
> ; (1 server found)
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24955
> ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
> 
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 1232
> ;; QUESTION SECTION:
> ;kdevops-nfsd.                  IN      A
> 
> ;; ANSWER SECTION:
> kdevops-nfsd.           0       IN      A       192.168.122.85
> 
> ;; Query time: 0 msec
> ;; SERVER: 192.168.122.1#53(192.168.122.1) (UDP)
> ;; WHEN: Tue Aug 26 10:18:47 EDT 2025
> ;; MSG SIZE  rcvd: 57


sudo virsh list --all | grep "dio\|interop\|posix\|iscsi\|nfsd"
 13   debian13-interop                    running
 14   debian13-nfsd                       running
 15   debian13-dio                        running
 16   debian13-posix                      running
 17   debian13-iscsi                      running

dig @192.168.122.1 debian13-nfsd

; <<>> DiG 9.20.11-4-Debian <<>> @192.168.122.1 debian13-nfsd
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4579
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;debian13-nfsd.                 IN      A

;; ANSWER SECTION:
debian13-nfsd.          0       IN      A       192.168.122.193

;; Query time: 0 msec
;; SERVER: 192.168.122.1#53(192.168.122.1) (UDP)
;; WHEN: Tue Aug 26 11:03:33 PDT 2025
;; MSG SIZE  rcvd: 58

> 
> $ resolvectl query kdevops-nfsd.
> kdevops-nfsd.: 'kdevops-nfsd.' not found
> ----------------8<-----------------

dig debian13-nfsd

; <<>> DiG 9.20.11-4-Debian <<>> debian13-nfsd
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 60809
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;debian13-nfsd.                 IN      A

;; Query time: 3 msec
;; SERVER: 192.168.64.5#53(192.168.64.5) (UDP)
;; WHEN: Tue Aug 26 11:09:44 PDT 2025
;; MSG SIZE  rcvd: 42

nslookup debian13-nfsd
;; Got SERVFAIL reply from 192.168.64.5, trying next server
;; Got SERVFAIL reply from 192.168.64.6
Server:         192.168.64.6
Address:        192.168.64.6#53

** server can't find debian13-nfsd: SERVFAIL

nslookup debian13-nfsd 192.168.122.1
Server:         192.168.122.1
Address:        192.168.122.1#53

Name:   debian13-nfsd
Address: 192.168.122.193
;; communications error to 192.168.122.1#53: timed out
;; communications error to 192.168.122.1#53: timed out
;; communications error to 192.168.122.1#53: timed out
;; no servers could be reached


> 
> It seems like we're doing something "unseemly" by putting bare names in
> DNS like this, given that systemd-resolved won't pass them along.
> 
> Since systemd-resolved will only resolve bare names without a domain
> using LLMNR, we should either take steps to make that work, or fix it
> so that systemd-resolved will forward the requests (by setting up a
> proper domain name and setting up the search path in the resolver).
>