WALinuxAgent fails to start after update to latest version 2.7.3.0-1_ol001.el8

Known Issues Azure
1

Affected Products:

  • Red Hat Enterprise Linux (RHEL) 8 images built before 2022-10-04
  • AlmaLinux 8 images built before 2022-10-04
  • Rocky Linux 8 images built before 2022-10-04
  • CentOS Stream 8 images built before 2022-10-04
  • CentOS 8 images built before 2022-10-04

To find out the image built timestamp, please read this article.

Opened: 2022-11-04

Severity: 2-Minor

Symptoms:

Version 2.7.3.0-1_ol001.el8 of the WALinuxAgent software package is now available on the OpenLogic repository.

In all Azure images built before 2022-10-04, ProComputers is using the openlogic repo as instructed in step 6 of this Microsoft article.

After updating WALinuxAgent package to the latest available version 2.7.3.0-1_ol001.el8 hosted on the openlogic repo, the waagent service does not start correctly anymore, and throws the following errors in the logs:

[azureuser@test ~]$ sudo journalctl -b | grep waagent
[…]
Nov 04 20:59:34 test systemd[1882]: waagent.service: Failed to execute command: No such file or directory
Nov 04 20:59:34 test systemd[1882]: waagent.service: Failed at step EXEC spawning /usr/bin/python: No such file or directory
[…]

[azureuser@test ~]$ sudo systemctl status waagent
● waagent.service - Azure Linux Agent
Loaded: loaded (/usr/lib/systemd/system/waagent.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/waagent.service.d
└─12-CPUQuota.conf
Active: activating (auto-restart)* (Result: exit-code) since Mon 2022-11-07 21:00:52 UTC; 4s ago
Process: 1938 ExecStart=/usr/bin/python -u /usr/sbin/waagent -daemon (code=exited, status=203/EXEC)
Main PID: 1938 (code=exited, status=203/EXEC)
CPU: 1ms
[…]

Solution:

The reason for this issue is that the WALinuxAgent package version 2.7.3.0-1_ol001.el8 as built by OpenLogic is using the /usr/bin/python binary, which does not exist by default in an RHEL 8 environment (and all binary compatible operating systems like AlmaLinux and Rocky Linux) . Previous version of the OpenLogic built WALinuxAgent package used the /usr/bin/python3.6 binary.

[azureuser@test ~]$ ls -la /usr/bin/python*
lrwxrwxrwx. 1 root root 9 Feb 10 2022 /usr/bin/python2 → python2.7
-rwxr-xr-x. 1 root root 8024 Feb 10 2022 /usr/bin/python2.7
lrwxrwxrwx. 1 root root 25 May 17 11:18 /usr/bin/python3 → /etc/alternatives/python3
lrwxrwxrwx. 1 root root 31 Aug 11 2021 /usr/bin/python3.6 → /usr/libexec/platform-python3.6
lrwxrwxrwx. 1 root root 32 Aug 11 2021 /usr/bin/python3.6m → /usr/libexec/platform-python3.6m

We could use the alternatives command and create an /usr/bin/python binary, however our preferred solution is to reconfigure the WALinuxAgent to use the correct /usr/bin/python3.6 binary, as below:

  1. Create the /etc/systemd/system/waagent.service.d folder:

[azureuser@test ~]$ sudo mkdir -p /etc/systemd/system/waagent.service.d

  1. Using sudo and your favorite editor, create the /etc/systemd/system/waagent.service.d/local.conf file with the following content:

[Service]
ExecStart=
ExecStart=/usr/bin/python3.6 -u /usr/sbin/waagent -daemon

NOTE: we have to explicitly clear ExecStart before setting it again, as it is an additive setting.

  1. Reload the systemd manager configuration and restart the waagent service:

[azureuser@test ~]$ sudo systemctl daemon-reload
[azureuser@test ~]$
[azureuser@test ~]$ sudo systemctl stop waagent
[azureuser@test ~]$
[azureuser@test ~]$ sudo systemctl start waagent
[azureuser@test ~]$
[azureuser@test ~]$ sudo systemctl status waagent
● waagent.service - Azure Linux Agent
Loaded: loaded (/usr/lib/systemd/system/waagent.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/waagent.service.d
└─12-CPUQuota.conf
/etc/systemd/system/waagent.service.d
└─local.conf
Active: active (running) since Mon 2022-11-07 21:37:41 UTC; 6s ago
Main PID: 3431 (python3.6)
Tasks: 7 (limit: 100875)
Memory: 32.1M
CPU: 1.549s
CGroup: /azure.slice/waagent.service
├─3431 /usr/bin/python3.6 -u /usr/sbin/waagent -daemon
└─3443 /usr/bin/python3.6 -u bin/WALinuxAgent-2.8.0.11-py2.7.egg -run-exthandlers
[…]

If you still face the same issue, please contact ProComputers Support as instructed in this article.

References: