#yocto systemd not able to start sshd after a reboot

#yocto systemd not able to start sshd after a reboot

[srijan.nandi]

[-- Attachment #1: Type: text/plain, Size: 1603 bytes --]

I am facing a peculiar problem with openssh. I have built openssh_8.0p1on zeus.

The sshd.service file is as follows:
[Unit]
Description=OpenSSH server daemon
Documentation=man:sshd(8) man:sshd_config(5)
After=sshdgenkeys.service
Wants=sshdgenkeys.service

[Service]
Type=simple
PIDFile=/var/run/sshd.pid
EnvironmentFile=-/etc/default/sshd
ExecStart=/usr/sbin/sshd -D $OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/kill $MAINPID
PermissionsStartOnly=true
KillMode=process
Restart=on-failure
StandardError=syslog

[Install]
WantedBy=multi-user.target

It starts without issues, if i do a systemctl start sshd.service. If I do a test of the config file it does not come up with any errors:

genericx86-64:~$ sudo /usr/sbin/sshd -t
genericx86-64:~$

Problem:
If I reboot the server, sshd does not start. There is no error on syslog. I have enabled debug logging, still no logs in syslog.

# Logging
SyslogFacility DAEMON
LogLevel DEBUG3

Even systemctl is-enabled sshd shows as enabled.

After a reboot, if I do a systemctl status sshd it shows:
Loaded: loaded  (/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
Active: inactive (dead)

If I manually run systemctl start sshd.service, everything works perfectly well without issues. sshd start on 0.0.0.0:2224 and I am able to ssh in as well.

It's just that systemctl is not able to start sshd after a reboot and there is no error that i can find or debug. Absolutely run out of ideas to resolve this. Any help will be greatly appreciated.

Thanks and Regards,
-=Srijan Nandi

[-- Attachment #2: Type: text/html, Size: 2020 bytes --]

Re: [yocto] #yocto systemd not able to start sshd after a reboot

[Khem Raj]

does something like this work?

[Unit]
Description=OpenSSH Daemon
Wants=sshdgenkeys.service
After=sshdgenkeys.service
After=network.target

[Service]
ExecStart=/usr/bin/sshd -D
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=always

[Install]
WantedBy=multi-user.target

On Fri, Sep 18, 2020 at 8:29 AM <srijan.nandi@gmail.com> wrote:
>
> I am facing a peculiar problem with openssh. I have built openssh_8.0p1on zeus.
>
> The sshd.service file is as follows:
> [Unit]
> Description=OpenSSH server daemon
> Documentation=man:sshd(8) man:sshd_config(5)
> After=sshdgenkeys.service
> Wants=sshdgenkeys.service
>
> [Service]
> Type=simple
> PIDFile=/var/run/sshd.pid
> EnvironmentFile=-/etc/default/sshd
> ExecStart=/usr/sbin/sshd -D $OPTIONS
> ExecReload=/bin/kill -HUP $MAINPID
> ExecStop=/bin/kill $MAINPID
> PermissionsStartOnly=true
> KillMode=process
> Restart=on-failure
> StandardError=syslog
>
> [Install]
> WantedBy=multi-user.target
>
> It starts without issues, if i do a systemctl start sshd.service. If I do a test of the config file it does not come up with any errors:
>
> genericx86-64:~$ sudo /usr/sbin/sshd -t
> genericx86-64:~$
>
> Problem:
> If I reboot the server, sshd does not start. There is no error on syslog. I have enabled debug logging, still no logs in syslog.
>
> # Logging
> SyslogFacility DAEMON
> LogLevel DEBUG3
>
> Even systemctl is-enabled sshd shows as enabled.
>
> After a reboot, if I do a systemctl status sshd it shows:
> Loaded: loaded  (/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
> Active: inactive (dead)
>
> If I manually run systemctl start sshd.service, everything works perfectly well without issues. sshd start on 0.0.0.0:2224 and I am able to ssh in as well.
>
> It's just that systemctl is not able to start sshd after a reboot and there is no error that i can find or debug. Absolutely run out of ideas to resolve this. Any help will be greatly appreciated.
>
> Thanks and Regards,
> -=Srijan Nandi
>
> 
>

Re: #yocto systemd not able to start sshd after a reboot

[srijan.nandi]

[-- Attachment #1: Type: text/plain, Size: 276 bytes --]

Hello Khem,

With the above sshd.service file the sshd daemon fails to start. It gives an error "(code=exited, status=203/EXEC)".

Cannot figure out as to why the systemd for sshd fails to work, while the other systemd files are working perfectly fine.

-=Srijan Nandi

[-- Attachment #2: Type: text/html, Size: 300 bytes --]

Re: [yocto] #yocto systemd not able to start sshd after a reboot

[Zoran]

Hello Srijan,

Did you recap/look into this sshd.service file?
https://lists.yoctoproject.org/g/yocto/message/49993

Zoran
_______

On Fri, Sep 18, 2020 at 8:07 PM <srijan.nandi@gmail.com> wrote:
>
> Hello Khem,
>
> With the above sshd.service file the sshd daemon fails to start. It gives an error "(code=exited, status=203/EXEC)".
>
> Cannot figure out as to why the systemd for sshd fails to work, while the other systemd files are working perfectly fine.
>
> -=Srijan Nandi
> 
>

Re: #yocto systemd not able to start sshd after a reboot

[srijan.nandi]

[-- Attachment #1: Type: text/plain, Size: 348 bytes --]

Hello All,

I finally got it to work!!!

There was a sshd.socket file in /lib/systemd/system which had the following line in it.

Conflicts=sshd.service

I remove it and added the following two lines:

After=network.target
Before=sshd.service

And that did the trick. Now sshd service starts on every boot.

Thanks,
-=Srijan Nandi

[-- Attachment #2: Type: text/html, Size: 412 bytes --]

Re: [yocto] #yocto systemd not able to start sshd after a reboot

[Zoran]

[-- Attachment #1: Type: text/plain, Size: 1207 bytes --]

Interesting... Here is what I have on Fedora32:

[root@fedora32-ssd system]# pwd
/lib/systemd/system
[root@fedora32-ssd system]# cat /lib/systemd/system/sshd.service
[Unit]
Description=OpenSSH server daemon
Documentation=man:sshd(8) man:sshd_config(5)
*After=network.target sshd-keygen.target*
Wants=sshd-keygen.target

[Service]
Type=notify
EnvironmentFile=-/etc/sysconfig/sshd-permitrootlogin
EnvironmentFile=-/etc/sysconfig/sshd
ExecStart=/usr/sbin/sshd -D $OPTIONS $PERMITROOTLOGIN
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target

*Seems that some leftovers from System V still reside in YOCTO...
Correct???*

Best Regards,
Zoran
_______


On Sat, Sep 19, 2020 at 3:37 PM <srijan.nandi@gmail.com> wrote:

> Hello All,
>
> I finally got it to work!!!
>
> There was a sshd.socket file in /lib/systemd/system which had the
> following line in it.
>
> Conflicts=sshd.service
>
> I remove it and added the following two lines:
>
> After=network.target
> Before=sshd.service
>
> And that did the trick. Now sshd service starts on every boot.
>
> Thanks,
> -=Srijan Nandi
> 
>
>

[-- Attachment #2: Type: text/html, Size: 1674 bytes --]

Re: #yocto systemd not able to start sshd after a reboot

[srijan.nandi]

[-- Attachment #1: Type: text/plain, Size: 572 bytes --]

> 
> Seems that some leftovers from System V still reside in YOCTO...
> Correct???
> 

Not sure about that.

The problem I faced was because there was a sshd.socket that had the following line in it. The sshd.socket comes with openssh.

Conflicts=sshd.service

So I had two options. either to add the ExecStartPre in the sshd.service file or to remove the Conflicts line in sshd.socket.

I choose to remove the Conflicts line and add the following in the sshd.socket file.

After=network.target
Before=sshd.service

Thanks and Regards,
-=Srijan Nandi

[-- Attachment #2: Type: text/html, Size: 719 bytes --]

Re: [yocto] #yocto systemd not able to start sshd after a reboot

[Khem Raj]

On 9/21/20 5:17 AM, srijan.nandi@gmail.com wrote:
>     /Seems that some leftovers from System V still reside in YOCTO...
>     Correct???

Yocto project supports sysvinit as init system option as well so no 
there are no leftovers.

>     /
> 
> Not sure about that.
> 
> The problem I faced was because there was a sshd.socket that had the 
> following line in it. The sshd.socket comes with openssh.
> 
> Conflicts=sshd.service
> 
> So I had two options. either to add the ExecStartPre in the sshd.service 
> file or to remove the Conflicts line in sshd.socket.
> 
> I choose to remove the Conflicts line and add the following in the 
> sshd.socket file.
> 
> After=network.target
> Before=sshd.service

Usually you use socket activation for sshd then you would enable 
sshd.socket and not sshd.service, socket will be listening on incoming 
connections on ssh port ( 22  by defaault ) and launch sshd.service 
whenever there is incoming ssh connection request. I suggest you should 
perhaps follow this approach as well, its also efficient due to its 
on-demand launch nature.

> 
> Thanks and Regards,
> -=Srijan Nandi
> 
> 
> 
> 
> 

Re: [yocto] #yocto systemd not able to start sshd after a reboot

[Zoran]

> There was a sshd.socket file in /lib/systemd/system which had the following line in it.

Interesting... Pushed/forced me to think.

There is no formal dependency between sshd.service and sshd.socket!

[vuser@fedora32-ssd systemd]$ systemctl list-dependencies sshd.service
| grep ssh
sshd.service
● ├─sshd-keygen.target
● │ ├─sshd-keygen@ecdsa.service
● │ ├─sshd-keygen@ed25519.service
● │ └─sshd-keygen@rsa.service
[vuser@fedora32-ssd systemd]$ systemctl list-dependencies sshd.service
| grep socket
●   ├─lvm2-lvmetad.socket
●   ├─lvm2-lvmpolld.socket
[vuser@fedora32-ssd systemd]$ systemctl list-dependencies sshd.socket
| grep sshd
sshd.socket

Strange... Isn't it?!

Zoran
_______

On Sat, Sep 19, 2020 at 3:37 PM <srijan.nandi@gmail.com> wrote:
>
> Hello All,
>
> I finally got it to work!!!
>
> There was a sshd.socket file in /lib/systemd/system which had the following line in it.
>
> Conflicts=sshd.service
>
> I remove it and added the following two lines:
>
> After=network.target
> Before=sshd.service
>
> And that did the trick. Now sshd service starts on every boot.
>
> Thanks,
> -=Srijan Nandi
> 
>

Re: #yocto systemd not able to start sshd after a reboot

[srijan.nandi]

[-- Attachment #1: Type: text/plain, Size: 780 bytes --]

Seems strange to me too...I had been troubleshooting the sshd.service issue for two days. There was no logs, nothing. I was just hitting the wall. I tried a lot of combinations to get it to work but all failed.

The sshd.service was starting, if I manually did a systemctl start sshd.service. But always failed at startup. At times it would start and then get a signal 15 terminating and would close the daemon.

After not able to resolve the issue, I started checking all the other services. Every other service was starting at bootup just fine except sshd.

Not finding anything else to troubleshoot. I happened to stumble upon the sshd.socket and the Conflicts part of it. Did a hit and trial and it worked. Technically I still am not sure as to why..

-=Srijan Nandi

[-- Attachment #2: Type: text/html, Size: 812 bytes --]

Re: [yocto] #yocto systemd not able to start sshd after a reboot

[Zoran]

[-- Attachment #1: Type: text/plain, Size: 2117 bytes --]

Maybe this should be added to sshd.service [unit] part
(to have added dependency) to solve this problem:

> The sshd.service file is as follows:
> [Unit]
> Description=OpenSSH server daemon
> Documentation=man:sshd(8) man:sshd_config(5)
> After=sshdgenkeys.service
> Wants=sshdgenkeys.service
*Requires=**sshd.socket*

Zoran
_______
On Fri, Sep 18, 2020 at 5:29 PM <srijan.nandi@gmail.com> wrote:
>
> I am facing a peculiar problem with openssh. I have built openssh_8.0p1on
zeus.
>
> The sshd.service file is as follows:
> [Unit]
> Description=OpenSSH server daemon
> Documentation=man:sshd(8) man:sshd_config(5)
> After=sshdgenkeys.service
> Wants=sshdgenkeys.service
>
> [Service]
> Type=simple
> PIDFile=/var/run/sshd.pid
> EnvironmentFile=-/etc/default/sshd
> ExecStart=/usr/sbin/sshd -D $OPTIONS
> ExecReload=/bin/kill -HUP $MAINPID
> ExecStop=/bin/kill $MAINPID
> PermissionsStartOnly=true
> KillMode=process
> Restart=on-failure
> StandardError=syslog
>
> [Install]
> WantedBy=multi-user.target
>
> It starts without issues, if i do a systemctl start sshd.service. If I do
a test of the config file it does not come up with any errors:
>
> genericx86-64:~$ sudo /usr/sbin/sshd -t
> genericx86-64:~$
>
> Problem:
> If I reboot the server, sshd does not start. There is no error on syslog.
I have enabled debug logging, still no logs in syslog.
>
> # Logging
> SyslogFacility DAEMON
> LogLevel DEBUG3
>
> Even systemctl is-enabled sshd shows as enabled.
>
> After a reboot, if I do a systemctl status sshd it shows:
> Loaded: loaded  (/lib/systemd/system/sshd.service; enabled; vendor
preset: enabled)
> Active: inactive (dead)
>
> If I manually run systemctl start sshd.service, everything works
perfectly well without issues. sshd start on 0.0.0.0:2224 and I am able to
ssh in as well.
>
> It's just that systemctl is not able to start sshd after a reboot and
there is no error that i can find or debug. Absolutely run out of ideas to
resolve this. Any help will be greatly appreciated.
>
> Thanks and Regards,
> -=Srijan Nandi
>
> 
>

[-- Attachment #2: Type: text/html, Size: 2636 bytes --]