Re: systemd-resolved and single-label DNS entries

[Daniel Gomez <da.gomez@kernel.org>]

On 26/08/2025 18.32, Jeff Layton wrote:
> On Tue, 2025-08-26 at 11:07 -0400, Chuck Lever wrote:
>> On 8/26/25 10:21 AM, Jeff Layton wrote:
>>> On Tue, 2025-08-26 at 14:53 +0200, Daniel Gomez wrote:
>>>> On 26/08/2025 12.58, Jeff Layton wrote:
>>>>> I'm hitting a problem with NFS testing on kdevops guests. Recently when
>>>>> I bring up a guest, it's unable to resolve the name of the NFS server
>>>>> via DNS.
>>>>
>>>> FYI, nfstests works fine with Debian testing (forky). Both hosts and guests.
>>>>
>>>>
>>>
>>>
>>> Thanks. Some questions:
>>>
>>> 1/ did your setup get the entries in /etc/hosts? That seems to be what
>>> has changed recently. That's not working for some reason.
>>>
>>> 2/ Does forky use systemd-resolved? The problem is clearly there with
>>> Fedora 42. The DNS server on the virtual bridge works fine:
>>>
>>> ----------------8<-----------------
>>> $ dig @192.168.122.1 kdevops-nfsd.
>>>
>>> ; <<>> DiG 9.18.33 <<>> @192.168.122.1 kdevops-nfsd.
>>> ; (1 server found)
>>> ;; global options: +cmd
>>> ;; Got answer:
>>> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24955
>>> ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
>>>
>>> ;; OPT PSEUDOSECTION:
>>> ; EDNS: version: 0, flags:; udp: 1232
>>> ;; QUESTION SECTION:
>>> ;kdevops-nfsd.                  IN      A
>>>
>>> ;; ANSWER SECTION:
>>> kdevops-nfsd.           0       IN      A       192.168.122.85
>>>
>>> ;; Query time: 0 msec
>>> ;; SERVER: 192.168.122.1#53(192.168.122.1) (UDP)
>>> ;; WHEN: Tue Aug 26 10:18:47 EDT 2025
>>> ;; MSG SIZE  rcvd: 57
>>>
>>> $ resolvectl query kdevops-nfsd.
>>> kdevops-nfsd.: 'kdevops-nfsd.' not found
>>> ----------------8<-----------------
>>>
>>> It seems like we're doing something "unseemly" by putting bare names in
>>> DNS like this, given that systemd-resolved won't pass them along.
>>>
>>> Since systemd-resolved will only resolve bare names without a domain
>>> using LLMNR, we should either take steps to make that work, or fix it
>>> so that systemd-resolved will forward the requests (by setting up a
>>> proper domain name and setting up the search path in the resolver).
>>>
>>
>> "won't pass them along" -- does that depend on the settings in
>> /etc/nsswitch.conf ?
>>
> 
> What's happening in my environment is that this list in the
> update_etc_hosts playbook comes up empty:

Can you check if your ansible.cfg has the inventory field?

grep inventory ansible.cfg
inventory = /xfs1/dagomez/kdevops/hosts

> 
> - name: Build list of hosts                                                                    
>   set_fact:       
>     ueh_hosts: "{{ ansible_play_hosts_all | difference([inventory_hostname]) }}"
>                        
> That recently changed in this commit:
> 
> commit 1cf0800c9ffced98da393e5d2b1594302d23eb78
> Author: Daniel Gomez <da.gomez@samsung.com>
> Date:   Thu Jul 24 21:36:14 2025 +0200
> 
>     gen_hosts: templates: include localhost in the all group
> 
> I couldn't revert that commit cleanly, so I tried out the commit just
> before it, and /etc/hosts got updated correctly. I'll have to look at
> proper fixes for this, but if anyone has ideas I'm happy to try them
> out.

My issue was that the list included the localhosts as it was now part of hosts
file. In the commit log I added:

    Finally, update ueh_hosts in the update_etc_hosts role to include only
    the hosts targeted in the current play, rather than all hosts in the
    inventory. This avoids the error below when the inventory includes
    localhost in the all group:

    "The task includes an option with an undefined variable. The error
    was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute
    'ansible_all_ipv4_addresses'. 'ansible.vars.hostvars.HostVarsVars
    object' has no attribute 'ansible_all_ipv4_addresses'"

And this is the commit change related to the ueh_hosts:

diff --git a/playbooks/roles/update_etc_hosts/tasks/main.yml b/playbooks/roles/update_etc_hosts/tasks/main.yml
index 4c1307fb..2b9af91d 100644
--- a/playbooks/roles/update_etc_hosts/tasks/main.yml
+++ b/playbooks/roles/update_etc_hosts/tasks/main.yml
@@ -26,7 +26,7 @@

 - name: Build list of hosts
   set_fact:
-    ueh_hosts: "{{ groups.all | flatten(levels=1) | reject('eq', inventory_hostname) }}"
+    ueh_hosts: "{{ ansible_play_hosts_all | difference([inventory_hostname]) }}"


Below are the logs with current vanilla kdevops (1.) and logs reverting the
changes above (2.):

1. make bringup AV=2 V=1
...

TASK [update_etc_hosts : Gather network facts] *********************************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:16
ok: [debian13-posix]
ok: [debian13-dio]
ok: [debian13-interop]
 [started TASK: update_etc_hosts : Build network address on debian13-dio]
 [started TASK: update_etc_hosts : Build network address on debian13-interop]
 [started TASK: update_etc_hosts : Build network address on debian13-posix]
 [started TASK: update_etc_hosts : Build list of hosts on debian13-dio]
 [started TASK: update_etc_hosts : Build list of hosts on debian13-interop]
 [started TASK: update_etc_hosts : Build list of hosts on debian13-posix]

TASK [update_etc_hosts : Build list of hosts] **********************************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:27
ok: [debian13-dio] => {"ansible_facts": {"ueh_hosts": ["debian13-interop", "debian13-posix"]}, "changed": false}
ok: [debian13-interop] => {"ansible_facts": {"ueh_hosts": ["debian13-dio", "debian13-posix"]}, "changed": false}
ok: [debian13-posix] => {"ansible_facts": {"ueh_hosts": ["debian13-dio", "debian13-interop"]}, "changed": false}
 [started TASK: update_etc_hosts : ansible.builtin.debug on debian13-dio]
 [started TASK: update_etc_hosts : ansible.builtin.debug on debian13-interop]
 [started TASK: update_etc_hosts : ansible.builtin.debug on debian13-posix]

TASK [update_etc_hosts : ansible.builtin.debug] ********************************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:31
ok: [debian13-dio] => {
    "ueh_hosts": [
        "debian13-interop",
        "debian13-posix"
    ]
}
ok: [debian13-interop] => {
    "ueh_hosts": [
        "debian13-dio",
        "debian13-posix"
    ]
}
ok: [debian13-posix] => {
    "ueh_hosts": [
        "debian13-dio",
        "debian13-interop"
    ]
}
 [started TASK: update_etc_hosts : Stat /etc/cloud/cloud.cfg.d on debian13-dio]
 [started TASK: update_etc_hosts : Stat /etc/cloud/cloud.cfg.d on debian13-interop]
 [started TASK: update_etc_hosts : Stat /etc/cloud/cloud.cfg.d on debian13-posix]

TASK [update_etc_hosts : Stat /etc/cloud/cloud.cfg.d] **************************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:34
ok: [debian13-dio] => {"changed": false, "stat": {"atime": 1756234164.14, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "binary", "ctime": 1756234453.4407628, "dev": 65025, "device_type": 0, "executable": true, "exists": true, "gid": 0, "gr_name": "root", "inode": 16529, "isblk": false, "ischr": false, "isdir": true, "isfifo": false, "isgid": false, "islnk": false, "isreg": false, "issock": false, "isuid": false, "mimetype": "inode/directory", "mode": "0755", "mtime": 1756234453.4407628, "nlink": 2, "path": "/etc/cloud/cloud.cfg.d", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 4096, "uid": 0, "version": "612645937", "wgrp": false, "woth": false, "writeable": false, "wusr": true, "xgrp": true, "xoth": true, "xusr": true}}
ok: [debian13-posix] => {"changed": false, "stat": {"atime": 1756234164.664, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "binary", "ctime": 1756234451.0282118, "dev": 65025, "device_type": 0, "executable": true, "exists": true, "gid": 0, "gr_name": "root", "inode": 16529, "isblk": false, "ischr": false, "isdir": true, "isfifo": false, "isgid": false, "islnk": false, "isreg": false, "issock": false, "isuid": false, "mimetype": "inode/directory", "mode": "0755", "mtime": 1756234451.0282118, "nlink": 2, "path": "/etc/cloud/cloud.cfg.d", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 4096, "uid": 0, "version": "612645937", "wgrp": false, "woth": false, "writeable": false, "wusr": true, "xgrp": true, "xoth": true, "xusr": true}}
ok: [debian13-interop] => {"changed": false, "stat": {"atime": 1756234164.104, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "binary", "ctime": 1756234449.885939, "dev": 65025, "device_type": 0, "executable": true, "exists": true, "gid": 0, "gr_name": "root", "inode": 16529, "isblk": false, "ischr": false, "isdir": true, "isfifo": false, "isgid": false, "islnk": false, "isreg": false, "issock": false, "isuid": false, "mimetype": "inode/directory", "mode": "0755", "mtime": 1756234449.885939, "nlink": 2, "path": "/etc/cloud/cloud.cfg.d", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 4096, "uid": 0, "version": "612645937", "wgrp": false, "woth": false, "writeable": false, "wusr": true, "xgrp": true, "xoth": true, "xusr": true}}
 [started TASK: update_etc_hosts : Disable cloud-init hosts management on debian13-dio]
 [started TASK: update_etc_hosts : Disable cloud-init hosts management on debian13-interop]
 [started TASK: update_etc_hosts : Disable cloud-init hosts management on debian13-posix]

TASK [update_etc_hosts : Disable cloud-init hosts management] ******************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:39
changed: [debian13-dio] => {"backup": "", "changed": true, "msg": "line added"}
changed: [debian13-posix] => {"backup": "", "changed": true, "msg": "line added"}
changed: [debian13-interop] => {"backup": "", "changed": true, "msg": "line added"}
 [started TASK: update_etc_hosts : Add IP address of all hosts to all hosts on debian13-dio]
 [started TASK: update_etc_hosts : Add IP address of all hosts to all hosts on debian13-interop]
 [started TASK: update_etc_hosts : Add IP address of all hosts to all hosts on debian13-posix]
 [started TASK: update_etc_hosts : Add IP address of all hosts to all hosts on debian13-dio]
 [started TASK: update_etc_hosts : Add IP address of all hosts to all hosts on debian13-interop]
 [started TASK: update_etc_hosts : Add IP address of all hosts to all hosts on debian13-posix]

TASK [update_etc_hosts : Add IP address of all hosts to all hosts] *************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:61
changed: [debian13-dio] => (item=debian13-interop) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-interop", "msg": "line added"}
changed: [debian13-interop] => (item=debian13-dio) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-dio", "msg": "line added"}
changed: [debian13-posix] => (item=debian13-dio) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-dio", "msg": "line added"}
changed: [debian13-dio] => (item=debian13-posix) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-posix", "msg": "line added"}
changed: [debian13-interop] => (item=debian13-posix) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-posix", "msg": "line added"}
changed: [debian13-posix] => (item=debian13-interop) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-interop", "msg": "line added"}
 [started TASK: update_etc_hosts : Fix up hostname on Debian guestfs hosts on debian13-dio]
 [started TASK: update_etc_hosts : Fix up hostname on Debian guestfs hosts on debian13-interop]
 [started TASK: update_etc_hosts : Fix up hostname on Debian guestfs hosts on debian13-posix]

TASK [update_etc_hosts : Fix up hostname on Debian guestfs hosts] **************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:73
ok: [debian13-posix] => {"backup": "", "changed": false, "msg": ""}
ok: [debian13-dio] => {"backup": "", "changed": false, "msg": ""}
ok: [debian13-interop] => {"backup": "", "changed": false, "msg": ""}

PLAY RECAP *********************************************************************
debian13-dio               : ok=9    changed=2    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
debian13-interop           : ok=9    changed=2    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
debian13-posix             : ok=9    changed=2    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   


2. After reverting that change on ueh_hosts, I can replicate my original issue:

TASK [update_etc_hosts : Gather network facts] *******************************************************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:16
ok: [debian13-dio]
ok: [debian13-posix]
ok: [debian13-interop]
 [started TASK: update_etc_hosts : Build network address on debian13-dio]
 [started TASK: update_etc_hosts : Build network address on debian13-interop]
 [started TASK: update_etc_hosts : Build network address on debian13-posix]
 [started TASK: update_etc_hosts : Build list of hosts on debian13-dio]
 [started TASK: update_etc_hosts : Build list of hosts on debian13-interop]
 [started TASK: update_etc_hosts : Build list of hosts on debian13-posix]

TASK [update_etc_hosts : Build list of hosts] ********************************************************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:27
ok: [debian13-dio] => {"ansible_facts": {"ueh_hosts": ["localhost", "debian13-interop", "debian13-posix", "debian13-iscsi", "debian13-nfsd"]}, "changed": false}
ok: [debian13-posix] => {"ansible_facts": {"ueh_hosts": ["localhost", "debian13-dio", "debian13-interop", "debian13-iscsi", "debian13-nfsd"]}, "changed": false}
ok: [debian13-interop] => {"ansible_facts": {"ueh_hosts": ["localhost", "debian13-dio", "debian13-posix", "debian13-iscsi", "debian13-nfsd"]}, "changed": false}
 [started TASK: update_etc_hosts : ansible.builtin.debug on debian13-dio]
 [started TASK: update_etc_hosts : ansible.builtin.debug on debian13-interop]
 [started TASK: update_etc_hosts : ansible.builtin.debug on debian13-posix]

TASK [update_etc_hosts : ansible.builtin.debug] ******************************************************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:32
ok: [debian13-dio] => {
    "ueh_hosts": [
        "localhost",
        "debian13-interop",
        "debian13-posix",
        "debian13-iscsi",
        "debian13-nfsd"
    ]
}
ok: [debian13-interop] => {
    "ueh_hosts": [
        "localhost",
        "debian13-dio",
        "debian13-posix",
        "debian13-iscsi",
        "debian13-nfsd"
    ]
}
ok: [debian13-posix] => {
    "ueh_hosts": [
        "localhost",
        "debian13-dio",
        "debian13-interop",
        "debian13-iscsi",
        "debian13-nfsd"
    ]
}
 [started TASK: update_etc_hosts : Stat /etc/cloud/cloud.cfg.d on debian13-dio]
 [started TASK: update_etc_hosts : Stat /etc/cloud/cloud.cfg.d on debian13-interop]
 [started TASK: update_etc_hosts : Stat /etc/cloud/cloud.cfg.d on debian13-posix]

...

TASK [update_etc_hosts : Add IP address of all hosts to all hosts] ***********************************
task path: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:62
[ERROR]: Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'

Task failed.
Origin: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:62:3

60     - terraform_private_net_enabled
61
62 - name: Add IP address of all hosts to all hosts
     ^ column 3

<<< caused by >>>

Finalization of task args for 'ansible.builtin.lineinfile' failed.
Origin: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:65:3

63   become: true
64   become_method: sudo
65   ansible.builtin.lineinfile:
     ^ column 3

<<< caused by >>>

Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'
Origin: /xfs1/dagomez/kdevops/playbooks/roles/update_etc_hosts/tasks/main.yml:68:11

66     dest: /etc/hosts
67     regexp: ".*{{ item }}$"
68     line: "{{ hostvars[item].ansible_all_ipv4_addresses | first }} {{ item }}"
             ^ column 11

failed: [debian13-dio] (item=localhost) => {"ansible_loop_var": "item", "changed": false, "item": "localhost", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
failed: [debian13-interop] (item=localhost) => {"ansible_loop_var": "item", "changed": false, "item": "localhost", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
failed: [debian13-posix] (item=localhost) => {"ansible_loop_var": "item", "changed": false, "item": "localhost", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
changed: [debian13-dio] => (item=debian13-interop) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-interop", "msg": "line added"}
changed: [debian13-interop] => (item=debian13-dio) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-dio", "msg": "line added"}
changed: [debian13-posix] => (item=debian13-dio) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-dio", "msg": "line added"}
changed: [debian13-dio] => (item=debian13-posix) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-posix", "msg": "line added"}
failed: [debian13-dio] (item=debian13-iscsi) => {"ansible_loop_var": "item", "changed": false, "item": "debian13-iscsi", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
changed: [debian13-interop] => (item=debian13-posix) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-posix", "msg": "line added"}
changed: [debian13-posix] => (item=debian13-interop) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": "debian13-interop", "msg": "line added"}
failed: [debian13-dio] (item=debian13-nfsd) => {"ansible_loop_var": "item", "changed": false, "item": "debian13-nfsd", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
failed: [debian13-interop] (item=debian13-iscsi) => {"ansible_loop_var": "item", "changed": false, "item": "debian13-iscsi", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
failed: [debian13-posix] (item=debian13-iscsi) => {"ansible_loop_var": "item", "changed": false, "item": "debian13-iscsi", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
failed: [debian13-interop] (item=debian13-nfsd) => {"ansible_loop_var": "item", "changed": false, "item": "debian13-nfsd", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}
failed: [debian13-posix] (item=debian13-nfsd) => {"ansible_loop_var": "item", "changed": false, "item": "debian13-nfsd", "msg": "Task failed: Finalization of task args for 'ansible.builtin.lineinfile' failed: Error while resolving value for 'line': object of type 'HostVarsVars' has no attribute 'ansible_all_ipv4_addresses'"}


Let me know if any of this helps.