510 Commits

Author SHA1 Message Date
Dmitry Tantsur
0d4ae976c2
Support several API and Inspector URLs
Allows nodes with a single IP stack to be deployed from a dual-stack
Ironic.

Detecting advertised address and usable Ironic URLs are done completely
independently which does open some space for a misconfiguration. I hope
it's not likely in the reality, especially since this feature is
targetting advanced standalone users.

Change-Id: Ifa506c58caebe00b37167d329b81c166cdb323f2
Closes-Bug: #2045548
2024-01-09 16:43:23 +01:00
Dmitry Tantsur
2bb74523ae
Add missing headers to the inspection callback
Somehow, it has worked correctly for years, but now I've discovered that
the new inspection is (no longer?) tolerant to the missing header.

While here, copy all headers from the heartbeat code.

Change-Id: I9e5c609eb4435e520bc225dea08aedfdf169744b
2024-01-09 16:38:46 +01:00
Jay Faulkner
36e5993a04 [codespell] Fix spelling issues in IPA
This fixes several spelling issues identified by codepsell. In some
cases, I may have manually modified a line to make the output more clear
or to correct grammatical issues which were obvious in the codespell
output.

Later changes in this chain will provide the codespell config used to
generate this, as well as adding this commit's SHA, once landed, to a
.git-blame-ignore-revs file to ensure it will not pollute git historys
for modern clients.

Related-Bug: 2047654
Change-Id: I240cf8484865c9b748ceb51f3c7b9fd973cb5ada
2023-12-28 10:54:46 -08:00
Dmitry Tantsur
2ab8364649
Add a jitter to heartbeat retries
Currently, if heartbeat fails, we reschedule it after 5 seconds.
This is fine for the first retry, but it can cause a thundering herd
problem when a lot of nodes fail to heartbeat at once.

This change adds jitter to the minimum wait of 5 seconds. The jitter is
not applied for forced heartbeats: they still have a minimum wait of
exactly 5 seconds from the last heartbeat.

The code is re-ordered to move the interval calculation to one place.
Bonus: correctly logging the next interval.

The unit tests have been rewritten to test the heartbeat process step by
step and not rely on the exact sequence of the calls.

Closes-Bug: #2038438
Change-Id: I4c4207b15fb3d48b55e340b7b3b54af833f92cb5
2023-12-13 17:34:24 +01:00
Zuul
62041d6d9e Merge "Fix referencing to the raid_device var which is not set" 2023-12-12 17:01:32 +00:00
Iury Gregory Melo Ferreira
801da9ec1f Retry in ProxyError during post inspector data
* ProxyError is derived from ConnectionError, but it's necessary
to check the Response object to identify.

- Added ProxyError in retry_if_exception_type
- Updated _post_to_inspector to proper handle ProxyError
- Updated the wait to use wait_exponential instead of wait_fixed.

Closes-Bug: 2045429
Change-Id: Iefe3fe581cd4e7c91a0da708e6f6d0fdaacab6fe
2023-12-06 12:01:35 -03:00
Zuul
beccfe8c92 Merge "Revert "Fix vmedia network config drive handling"" 2023-11-30 15:14:20 +00:00
Dmitry Tantsur
c57deb7e76 Revert "Fix vmedia network config drive handling"
This reverts commit 33f01fa3c2f32f447ed36f00fea68321c3991c2e.

There are a few issues with the patch - see my comments there.
The most pressing and the reasons to revert are:
1) It breaks deployments when the vmedia is present but does not
   have a network_data.json (the case for Metal3).
2) It assumes the presence of Glean which may not be the case.

Neither Julia nor myself have time to thoroughly fix the issue,
leaving a revert as the only option to unblock Metal3.

Change-Id: I3f1a18a4910308699ca8f88d8e814c5efa78baee
Closes-Bug: #2045255
2023-11-30 10:33:29 +00:00
Maryna Savchenko
f80330839d Fix referencing to the raid_device var which is not set
Change-Id: I11180e5d61d893a78583ace555f6e90ba8845950
2023-11-29 12:40:29 +01:00
Zuul
eea9917023 Merge "Fix vmedia network config drive handling" 2023-11-29 01:10:25 +00:00
Zuul
768aa17442 Merge "Add mlnx deploy_step entry to enable deploy time firmware" 2023-11-23 00:12:13 +00:00
Zuul
7a4114512c Merge "Handle different device outputs for multipath" 2023-11-22 21:36:40 +00:00
Iury Gregory Melo Ferreira
0a29206b8d Handle different device outputs for multipath
In some cases the output of the multipath can differ
and we would return a wrong parent device.

Closes-Bug: 2043992
Change-Id: I848d7df798cc736bd5a55eed8fa46110caea1dc3
2023-11-20 22:51:41 -03:00
Adam Rozman
7a52314695 fix multipathd error handling release notes
This commit:
  - fixes some "multipathd error handling improvement"
    release notes
  - fixes a related comment in the code

Related launchpad issue https://bugs.launchpad.net/ironic-python-agent/+bug/2031092

Change-Id: Ie3ba0601fa117b053cb8db6284e47249ca9c9134
Signed-off-by: Adam Rozman <adam.rozman@est.tech>
2023-11-10 09:54:20 +02:00
Zuul
845df338f8 Merge "improve multipathd error handling" 2023-11-09 17:31:32 +00:00
Julia Kreger
33f01fa3c2 Fix vmedia network config drive handling
When performing DHCP-less deployments, the agent can start and
discover more than one configuration drive present on a host.

For example, a host was previously deployed using Ironic, and
is now being re-deployed again.

If Glean was present in the ramdisk, the glean-early.sh would end
mounting the folder based upon label.

If cloud-init, somehow is still in the ramdisk, the other folder
could somehow get mounted.

This patch, which is intended to be backportable, causes the agent
to unmount any configuration drive folders, mount the most likely
candidate based upon device type, partition, and overall state of
the machine, and then utilize that configuration, if present,
to re-configure and reload networking.

Thus allowing dhcp-less re-deployments to be fixed without
forcing any breaking changes.

It should also be noted that this fix was generated in concert
with an additional tempest test case, because this overall failure
case needed to be reproduced to ensure we had a workable non-breaking
path forward.

Closes-Bug: 2032377
Change-Id: I9a3b3dbb9ca98771ce2decf893eba7a4c1890eee
2023-11-08 12:11:06 -08:00
Zuul
9d9568ba23 Merge "Get numa_node info when collecting pci devices info" 2023-11-06 18:15:33 +00:00
Jay Faulkner
3d42298619 Remove standby.cache_image support
Image caching was never fully supported in Ironic or IPA; this is vestigal
code leftover from a partial implementation.

Even if we implemetented it today, we'd likely use a completely different
methodology.

Change-Id: Id4ab7b3c4f106b209585dbd090cdcb229b1daa73
2023-10-24 15:02:44 -07:00
Zhou Ya
76ad06225a Get numa_node info when collecting pci devices info
IPA now includes information about numa node id when collecting
information about PCI devices.

Closes-bug: #1622940
Co-Authored-By: Jay Faulkner <jay@jvf.cc>
Change-Id: I70b0cb3eff66d67bb8168982acbbf335de0599cd
2023-10-24 14:27:21 -07:00
Adam Rozman
13537db293 improve multipathd error handling
This commit:
  - Adds the ability to ignore inconsequential OS error caused
    by starting the multipathd service when an instance of the
    service is already running.

Related launchpad issue https://bugs.launchpad.net/ironic-python-agent/+bug/2031092

Change-Id: Iebf486915bfdc2546451e6b38a450b4c241e43a8
2023-10-23 16:33:03 +03:00
Zuul
b42f0be422 Merge "implement basic-auth support for user-image download process" 2023-10-13 17:08:28 +00:00
Julia Kreger
cb61a8d6c0 Retry on checksum failures
HTTP is a fun protocol.

Size is basically optional. And clients implicitly trust the server
and socket has transferred all the bytes. Which *really* means you
should always checksum.

But... previously we didn't checksum as part of retrying.

So if anything happened with python-requests, or lower level
library code or the system itself causing bytes to be lost off the
buffer, creating an incomplete transfer situation, then we wouldn't
know until the checksum.

So now, we checksum and re-trigger the download if there is a
failure of the checksum.

This involved a minor shift in the download logic, and resulted in
a needful minor fix to an image checksum test as it would loop for
90 seconds as well.

Closes-Bug: 2038934
Change-Id: I543a60555a2621b49dd7b6564bd0654a46db2e9a
2023-10-10 09:15:31 -07:00
Adam Rozman
70961789a6 implement basic-auth support for user-image download process
This feature was proposed in https://bugs.launchpad.net/ironic-python-agent/+bug/2021947

Change-Id: I9dbfc1402240beb75b6736214753fd86dccae676
2023-10-10 16:25:51 +03:00
Zuul
89be7bd420 Merge "Conditional creation of RAIDed ESP for UEFI Software RAID" 2023-10-10 11:07:25 +00:00
Zuul
23c8427224 Merge "Extend the lookup timeout to 600 seconds" 2023-09-22 12:35:00 +00:00
db9545eeec Update master for stable/2023.2
Add file to the reno documentation build to show release notes for
stable/2023.2.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2023.2.

Sem-Ver: feature
Change-Id: I8150eb8f35a444ef5a2bc7a648ec301e5094e52d
2023-09-21 11:18:18 +00:00
Zuul
73b76da5fe Merge "Add get_service_steps logic to the agent" 2023-09-15 22:29:59 +00:00
Julia Kreger
f86975d53c Add mlnx deploy_step entry to enable deploy time firmware
Follow-up from service steps addition change to add a deploy steps
alias for the Nvidia Mellanox network device firmware update clean
steps. This allows deploy time firmware updates to be codified as
part of a deployment with custom steps.

Change-Id: I9d80447dee7cfde4d3f8d81d9d39e738916b7824
2023-08-31 06:35:39 -07:00
Julia Kreger
eb95273ffb Add get_service_steps logic to the agent
Initial code patches for service steps have merged in
ironic, and it is now time to add support into the
agent which allows service steps to be raised to
the service.

Updates the default hardware manager version to 1.2,
which has *rarely* been incremented due to oversight.

Change-Id: Iabd2c6c551389ec3c24e94b71245b1250345f7a7
2023-08-31 06:22:22 -07:00
Julia Kreger
4efcce5310 Extend the lookup timeout to 600 seconds
Changes the default lookup timeout to be 600 seconds which
reduces the risk of lookup failing as a write operation
to the backing database is performed upon lookup thanks to
generation of an agent token.

Overall, this is fairly harmless since by default ramdisks
restart the agent if they were not able to successfully
start.

Change-Id: I35c64c0b4f9b3b607df1bc0c4c2a852aa3595cbd
2023-08-24 08:29:07 -07:00
Julia Kreger
b6c263a5dc preserve/handle config drives on 4k block devices
When an underlying block device (or driver) only supports 4KB IO,
this can cause some issues with aspects like using an ISO9660 filesystem
which can only support a maximum of 2KB IO.

The agent will now attempt to mount the filesystem *before* deleting the
supplied file, and should that fail it will mount the configuration drive
file from the ramdisk utilizing a loopback, and then extract the contents
of the ramdisk into a newly created VFAT filesystem which supports 4KB
block IO.

Closes-Bug: #2028002
Change-Id: I336acb8e8eb5a02dde2f5e24c258e23797d200ee
2023-08-24 08:10:22 -07:00
Julia Kreger
5ed520df89 Handle the node being locked
If the node is locked, a lookup cannot be performed when an agent
token needs to be generated, which tends to error like this:

  ironic_python_agent.ironic_api_client [-] Failed looking up node
  with addresses '00:6f:bb:34:b3:4d,00:6f:bb:34:b3:4b' at
  https://172.22.0.2:6385. Error 409: Node
  c25e451b-d2fb-4168-b690-f15bc8365520 is locked by host 172.22.0.2,
  please retry after the current operation is completed..
  Check if inspection has completed.

Problem is, if we keep pounding on the door, we can actually worsen
the situation, and previously we would just just let tenacity
retry.

We will now hold for 30 seconds before proceeding, so we have
hopefully allowed the operation to complete.

Also fixes the error logging to help human's sanity.

Change-Id: I97d3e27e2adb731794a7746737d3788c6e7977a0
2023-08-22 16:47:28 -07:00
Arne Wiebalck
286d66709a Conditional creation of RAIDed ESP for UEFI Software RAID
Rebuilding an instance on a RAIDed ESPs will fail due to sgdisk
running against an non-clean disk and bailing out. Check if there
is a RAIDed ESP already and skip creation if it exists.

Change-Id: I13617ae77515a9d34bc4bb3caf9fae73d5e4e578
2023-08-16 17:39:04 +02:00
Julia Kreger
b68a4c8a92 minor: fix release notes file path
Change-Id: I458d88bf14b55253179488cb771ae42e7b8c84d7
2023-08-07 12:57:34 -07:00
Zuul
119981a818 Merge "Fix nvidia hardware manager url parser to permit https" 2023-06-26 10:11:55 +00:00
Zuul
bb156aad6c Merge "Fix Bandit errors" 2023-06-26 09:25:09 +00:00
Julia Kreger
b83678c968 Fix nvidia hardware manager url parser to permit https
Change-Id: I9a10e543d3256ceaa78c6fbdb01fc0d88c0ee6e6
2023-06-06 15:35:16 +00:00
Julia Kreger
78c1343a54 Fix Bandit errors
Bandit 1.7.5 released with a timeout check for all requests and
urllib calls.

Fixed those.

In the process, then exposed a bandit b310 issue, which was already
covered by the code, but explicitly marked it as such.

Also, enables bandit checks to be voting for CI..

Change-Id: If0e87790191f5f3648366d571e1d85dd7393a548
2023-06-06 08:34:55 -07:00
Jay Faulkner
6098747ec5 Ironic (and IPA) use launchpad now
Correct links to point to launchpad bug tracker, correct docs config

Change-Id: I5d46af2a9d94f3b2e05e4f937e0619a89fe04d4c
2023-05-17 15:38:57 -07:00
Zuul
141c5ff1c3 Merge "Add support for CentOS SUM files" 2023-05-09 09:03:25 +00:00
Zuul
03e88b579e Merge "Revert disabling MD5 checksums" 2023-05-05 08:44:37 +00:00
Dmitry Tantsur
c1c5537ba2 Revert disabling MD5 checksums
This was a significant breaking change that was landed despite explicit
disagreement by some community members (myself included). It has already
resulted in an accidental Ironic CI breakage, has broken Bifrost and has
a potential of breaking Metal3. In case of Metal3, MD5 support is a part
of its public API.

While MD5 is a potential security hazard, I don't see the need to hurry
this change without giving the community time to prepare. This change
reverts the new option md5_enabled to True.

Change-Id: I32b291ea162e8eb22429712c15cb5b225a6daafd
2023-05-04 09:26:10 +02:00
Harald Jensås
e7a048ecbe
Add support for CentOS SUM files
The CentOS Stream SUM files uses format:
  # FILENAME: <size> bytes
  ALGORITHM (FILENAME) = CHECKSUM

Compared to the more common format:
  CHECKSUM  *FILE_A
  CHECKSUM  FILE_B

Use regular expressions to check for filename both
in the middle with parentheses and at the end.
Similarly look for valid checksums at beginning or
end of line. Also look for know checsum patterns in
case file only contain the checksum iteself.

Change-Id: I9e49c1a6c66e51a7b884485f0bcaf7f1802bda33
2023-05-03 21:31:23 +02:00
Dmitry Tantsur
9ed232e77e Add network interface speed to the inventory
This is another fact that Metal3's baremetal-operator is currently
consuming from extra-hardware.

Change-Id: I2ec9d5e9369f5508e7583a4e13c2083f5c8b28ba
2023-05-03 12:20:35 +02:00
Zuul
f37ea85a27 Merge "Disable MD5 image checksums" 2023-05-02 06:41:25 +00:00
Zuul
3cd8c294fb Merge "Deprecate LLDP in inventory in favour of a new collector" 2023-04-27 12:05:11 +00:00
Dmitry Tantsur
3e05a03f7c Deprecate LLDP in inventory in favour of a new collector
Binary LLDP data is bloating inventory causing us to disable its collection
by default. For other similar low-level information, such as PCI devices
or DMI data, we already use inspection collectors instead. Now that the
inventory format is shared with out-of-band inspection, having LLDP
there makes even less sense.

This change adds a new collector ``lldp`` to replace the now-deprecated
inventory field.

Change-Id: I56be06a7d1db28407e1128c198c12bea0809d3a3
2023-04-26 19:33:51 +00:00
Julia Kreger
32df26a22a Disable MD5 image checksums
MD5 image checksums have long been supersceeded by the use of a
``os_hash_algo`` and ``os_hash_value`` field as part of the
properties of an image.

In the process of doing this, we determined that checksum via
URL usage was non-trivial and determined that an appropriate
path was to allow the checksum type to be determined as needed.

Change-Id: I26ba8f8c37d663096f558e83028ff463d31bd4e6
2023-04-24 16:54:42 -07:00
Julia Kreger
76accfb880 Fix UTF-16 result handling for efibootmgr
The tl;dr is that UEFI NVRAM is in encoded
in UTF-16, and when we run the efibootmgr command,
we can get unicode characters back.

Except we previously were forcing everything to be
treated as UTF-8 due to the way oslo.concurrency's
processutils module works.

This could be observed with UTF character 0x00FF
which raises up a nice exception when we try to
decode it.

Anyhow! while fixing handling of this, we discovered
we could get basically the cruft out of the NVRAM,
by getting what was most likey a truncated string
out of our own test VMs. As such, we need to also
permit decoding to be tollerant of failures.
This could be binary data or as simple as flipped
bits which get interpretted invalid characters.
As such, we have introduced such data into one of our
tests involving UEFI record de-duplication.

Closes-Bug: 2015602
Change-Id: I006535bf124379ed65443c7b283bc99ecc95568b
2023-04-17 09:14:24 -07:00
Dmitry Tantsur
0304c73c0e Report system firmware information in the inventory
Change-Id: I5b6ceb9cdcf4baa97a6f0482d1030d14f3f2ecff
2023-03-31 14:28:32 +02:00