773 Commits

Author SHA1 Message Date
Dmitry Tantsur
cfcea55cf6 Automatically configure enabled_***_interfaces
This change makes it easier to configure power and management interfaces
(and thus vendor drivers) by figuring out reasonable defaults.

Story: #2009316
Task: #43717
Change-Id: I8779603e566be5a84daf6f680c0bbe2f191923d9
2021-12-20 15:11:17 +01:00
Steve Baker
3f76724dfb Write initial grub config on startup
This change removes the documentation to copy master_grub_cfg.txt to
/tftpboot/grub/grub.cfg and instead writes it on conductor startup.
This grub config is a simple redirect config requested by grub network
boot. "master" has been renamed to "initial" as a more accurate label
of its function.

New configuration option [pxe]initial_grub_template allows the deployer
to specify a different initial grub template.

Change-Id: I71191dd399a6c49607f91d69b5b1673799a38624
2021-12-10 15:44:50 +13:00
Zuul
3b949b2086 Merge "Use test_with_retry to get the tap device name" 2021-12-08 19:27:53 +00:00
Dmitry Tantsur
b37ee7c911 devstack: provide a default for OS_CLOUD
Not having it breaks the inspector grenade job.

Change-Id: I7ee28a85cb2005dd69e6711b301cd029b8ca40cc
2021-12-08 09:49:26 +01:00
Dmitry Tantsur
f85f649136 Install isolinux on devstack
It is required for virtual media BIOS booting.

Clean up old bindep tags.

Change-Id: I345e5b5287594e62ac7a8abb4de3add242120dfd
2021-12-03 17:50:52 +01:00
Steve Baker
9b4631ae0d Use test_with_retry to get the tap device name
This change replaces the 10 second sleep with a retry that has a
timeout of 20 seconds to discover the name of the tap device.

There are gate failures when there is still not a tap device after the
10 second sleep, so this approach should be faster in the common case,
and the higher timeout should provide more reliability.

Change-Id: I5e59ade9f830182b483b9655aaaf6c93b0bfac44
2021-11-22 13:50:59 +13:00
Julia Kreger
350c2f7a50 CI: Fix devstack plugin with RBAC changes
Changes a neutron call to be project scoped as system
scoped can't create a resource and, and removes the unset
which no longer makes sense now that
I86ffa9cd52454f1c1c72d29b3a0e0caa3e44b829
has merged removing the legacy vars from devstack.

Also renames intenral use setting of OS_CLOUD to IRONIC_OS_CLOUD
as some services were still working with system scope or some sort
of mixed state occuring previously as some of the environment variables
were present still, however they have been removed from devstack.

This change *does* explicitly set an OS_CLOUD variable as well on
the base ironic job. This is because things like grenade for Xena
will expect the variable to be present.

Depends-On: https://review.opendev.org/c/openstack/devstack/+/818449
Change-Id: I912527d7396a9c6d8ee7e90f0c3fd84461d443c1
2021-11-19 08:22:22 -08:00
Zuul
386c15836c Merge "Remove debian packages file for devstack" 2021-10-18 11:58:42 +00:00
Julia Kreger
f205e7d2be Remove debian packages file for devstack
This file duplicages devstack's load of bindep for the ironic
repository. As bindep is the authortative file, removing the
legacy way of installing packages from ironic.

Also revises the bindep file to explicitly remove the devstack
group, as it is all devstack.

Change-Id: Ida7ca230069fc0e4f54bde2fc6fffdc9eb0bdcc2
2021-10-08 10:40:07 -07:00
Julia Kreger
371313214a SRBAC - Prepare for additional services
In order to effectively handle cross-service integrations, we need to
evaluate two separate items which are not standardized in devstack.

Names, and common service references. Unfortunately, only a couple
services presently have support in devstack for these settings, and
cases where it was previously supported has been removed for unknown
reasons, but this seems to be the overall plan.

Sets the stage, so we can be early to the cross-service testing party
of secure rbac.

Change-Id: I8794374c02a24185b6e24a675ad9cb7b3dfd69df
2021-10-08 17:26:16 +00:00
Julia Kreger
044091c146 Retool devstack plugin to use pxe loaders configuration
Begins to peel back some of the override plugin/setting
nature in use in the ironic devstack plugin by trying to
place all of the files and letting the *defaults* take
the service lead while also putting in place the required
configuration for pxe loaders to be used.

Change-Id: I73ca82e0d123fd6efab06dbbdeef40c2d9972887
2021-10-08 17:25:54 +00:00
Zuul
0b36d9afa0 Merge "CI: Change CI ipxe file to snponly" 2021-10-08 11:10:36 +00:00
Zuul
af903094ea Merge "Yoga: Change default boot mode to uefi" 2021-10-07 10:04:24 +00:00
Julia Kreger
2b55c8d388 CI: Change CI ipxe file to snponly
Ironic's configured default is snponly.efi, and realistically
we should be using it for devstack as ipxe.efi lacks snp which
is included in the EFI standard.

Change-Id: I749420b127cc9954bfa02d9e4efaa0980a9242be
2021-10-04 18:59:52 +00:00
Julia Kreger
493b4f0caf Yoga: Change default boot mode to uefi
Change the default boot mode to UEFI, as discussed during the end
of the Wallaby release cycle and previously agreed a very long time
ago by the Ironic community.

Change-Id: I6d735604d56d1687f42d0573a2eed765cbb08aec
2021-10-04 11:57:55 -07:00
Zuul
28a06c1bd4 Merge "Remove legacy rpm install list and use bindep" 2021-10-04 17:08:07 +00:00
Julia Kreger
c1e355011c Remove legacy rpm install list and use bindep
The legacy rpm install list is out of date and includes
packages which no longer exist in newer distribution builds.

This functionality was entirely replaced by bindep, which is
*the* authortative file for things like this. The major difference
is the separate bindep file can't include mysql, or we break
devstack \o/.

Change-Id: Ic86f6efdf75fc2871c03e21b7f9166192b0f212c
2021-09-24 09:44:50 -07:00
Julia Kreger
4775fb3d92 Devstack: don't scan /opt, /etc looking for isolinux
/opt contains a mirror of an insane amount of stuff, and it chews
disk io to scan it looking for isolinux.bin which should be under
/usr on... well... every OS we support.

Also, don't scan /etc. That is just weird.

Change-Id: I52f4c1ba8808fea637df69a631eaa1c674dc8e69
2021-09-22 11:43:46 +00:00
Zuul
0affe4de8d Merge "Use packaged grub efi for network boot" 2021-09-09 17:19:17 +00:00
Steve Baker
fc8601cd02 Use packaged grub efi for network boot
Instead of using the efi written by grub-mknetdir, use the packaged
signed binary. The core.efi generated by grub-mknetdir is not signed
so it does not help with end-to-end secure-boot.

Also, the successful run of
ironic-tempest-ipa-partition-uefi-pxe-grub2[1] demonstrates that grub
continues to boot even when the grub-mknetdir generated
grub/x86_64-efi/*.lst are missing. Avoiding using grub-mknetdir makes
for a much simpler setup of /tftpboot for grub network boot.

[1] https://zuul.opendev.org/t/openstack/build/bab62f6bf032474cb80af3cb5a999117/log/tftpd-journal.txt

Change-Id: Ide0aa416391c20371bbb8d1a18288b262872e313
2021-09-08 13:35:45 +12:00
Zuul
5d79347f97 Merge "Fix upgrade logic to allow for bundled changes" 2021-09-03 13:20:09 +00:00
Vanou Ishii
1c3e20d859 Make curl in DevStack Bypass Proxy
This commit modifies curl option in wait_for_nova_resources to
bypass proxy with --noproxy option. Without this option, if you run
DevStack behind proxy, curl command fails with timeout &
wait_for_nova_resources also fails.
Because curl only accesses Placement service API, this modification
should be fair.

Change-Id: I5524a76594bb784f59be4d4e3970f72d4497891b
2021-08-11 15:19:30 +09:00
Julia Kreger
7b097f016b Fix upgrade logic to allow for bundled changes
The upgrade path logic was built to force a developer pattern to break
things such as new tables and features across multiple patches, and
the status check *can* explicitly fail if we don't explicitly go
hint to it that we've added table. Yes, we have a hard coded list...

Anyway, a better pattern is allow the db sync process to do the
appropriate needful. Run the status check, if it fails, fallback
and update the schema.

Also handles the explicit failure error and tries to return a human
friendly error message for when the table is not present.

In the end this allows us to merge schema changes such as additional
tables with their underlying objects and properly handle things as
long as the schema update works as expected. This allows us to
leverage an operational model of performing upgrades.

Change-Id: Id5f2a8068bc064e1ed1e376524850e4739f79ef2
2021-08-07 22:16:09 +00:00
Dmitry Tantsur
294046befa Use shim-signed on Ubuntu, shim is empty now
Also fix the documentation to use the correct paths and versions.

Change-Id: I7f004d40c1b8c617f9a456216df091e44d69693f
2021-08-03 13:07:22 +02:00
Zuul
c71583fc8a Merge "Scoped RBAC Devstack Plugin support" 2021-07-21 11:27:17 +00:00
Julia Kreger
2cd6468346 Scoped RBAC Devstack Plugin support
Adds support to the ironic devstack plugin to configure
ironic to be used in a scope-enforcing mode in line with
the Secure RBAC effort. This change also defines two new
integration jobs *and* changes one of the existing
integration.

In these cases, we're testing functional crub interactions,
integration with nova, and integration with ironic-inspector.

As other services come online with their plugins and
devstack code being able to set the appropriate scope
enforcement configuration, we will be able to change the
overall operating default for all of ironic's jobs and
exclude the differences.

This effort identified issues in ironic-tempest-plugin,
tempest, devstack, and required plugin support in
ironic-inspector as well, and is ultimately required
to ensure we do not break the Secure RBAC.

Luckilly, it all works.

Change-Id: Ic40e47cb11a6b6e9915efcb12e7912861f25cae7
2021-07-15 21:58:31 +00:00
Zuul
e3b401baf6 Merge "Deprecate [pxe]ip_version parameter" 2021-07-05 15:04:33 +00:00
Zuul
e16513f9fe Merge "Clean up vendor prefixes for iRMC boot" 2021-06-29 09:02:25 +00:00
Julia Kreger
b2a249d186 Deprecate [pxe]ip_version parameter
The parameter has not had any effect in the code since
the dual stack PXE work went into place with Ironic as options
are now generated and transmitted to Neutron for both IPv4 and
IPv6. This option is only used by the internal
``dhcp_options_for_instance`` method in
``ironic.common.pxe_utils`` as a fallback if the calling method
does not specify a version. However a later change resulted in the
default behavior calling the dhcp option generation explicitly
to generate both IPv4 and IPv6 parameters, making the option
entirely redundant.

Third party drivers using the ``dhcp_options_for_instance`` method
should consider generating options for both IPv4 and IPv6 at all
times. See change
If7a296001e204ae0c9a49495731052ab33379628 for examples on how
to do this.

Change-Id: I343783389105f008ce6dafc8d25d93211710771a
2021-06-28 06:59:34 -07:00
Dmitry Tantsur
4f2d1ca94b CI: change ilo_deploy_iso to deploy_iso
Change-Id: Icedc1cd57c64bfc9b0bad535a6eb7c890e843410
2021-06-17 17:55:18 +02:00
Dmitry Tantsur
fc27710fa4 Clean up vendor prefixes for iRMC boot
Generic fields, such as deploy_iso, should not have vendor prefixes.
This patch removes them from the iRMC boot interfaces with deprecation.

Change-Id: Ie24de1893395dca0e2dc4a57a42916f075d29ce6
Story: #2008880
Task: #42431
2021-06-17 17:24:42 +02:00
Julia Kreger
f58cbf1514 Fix ironic-status db index check
When I merged the db status check in for database indexes, I missed
the most improtant line, which is where the object is populated with
the method name to be executed by the upgrade checks framework.

In the rush to try and clean-up after the impact of the Secure RBAC
work, I just didn't manually test the final file I uploaded into
review. I assumed it just worked because the job passed, but didn't
think about the resulting return codes which we *should* experience
on an upgrade from a prior version. Later on, I noticed that because
of the way the status checks are invoked, I also added the code to
do the index check in the wrong order, so I had to restructure things
so the method definition was known by the object on the class which
holds the method names list.

I guess I copied/pasted this over from another file I was testing
in just didn't run the final file. :( Funny enough, the index check
works like a charm now.

Also updates the status check invocation check in the upgrade script
for grenade, *as* warnings *are* permissible and not fatal.

Change-Id: Ifa9da65dc8df5bf9a369d6faeab310386dfd944a
2021-06-10 07:04:15 -07:00
Dmitry Tantsur
2a73f5a84e Clean up kernel_append_params for PXE/iPXE
Currently handling of kernel_append_params is very inconsistent. This
change applies a straightforward process:
1. instance_info[kernel_append_params]
2. driver_info[kernel_append_params]
3. [pxe]kernel_append_params (renamed from pxe_append_params).

Also adds a helper for subsequent fixes in other drivers.

Change-Id: I79bcf4d8ef1f0f55a82e0991dd5bb1685b3f7957
Story: #2008902
Task: #42469
2021-05-17 16:12:30 +02:00
Riccardo Pittau
df368cbd8a Read default cirros version from stackrc
Workaround for grenade jobs to read CIRROS_VERSION variable from
stackrc.
We also give the possibility to specify a custom CIRROS_VERSION

In addition, fix transient iDRAC WS-Man BIOS test.

Depends-On: https://review.opendev.org/c/openstack/ironic/+/786387

Change-Id: Ic7f5dae5e6aa6916f0a7d73f43cc9552349385c5
2021-04-19 13:47:10 -04:00
Vanou Ishii
7552c489e3 Add iRMC Driver Support to DevStack Code
This commit adds logic
  * to determine whether irmc hardware type is enabled
  * (if enabled) to install python package python-scciclient & snmp
into DevStack code to support construction of Ironic environment
with iRMC supported Fujitsu server through DevStack.

Story: 2008722
Task: 42066
Change-Id: Ie50d8e4b43cdbfd8cd46333a75de20015e67829e
2021-03-17 18:48:09 +09:00
Dmitry Tantsur
7abac806a7 devstack: a safeguard for disabled tempurls
Change-Id: Id5fcd4cc1f73b80e8a9e9d2c50e2e4e1667c01cb
2021-02-25 12:09:30 +01:00
Zuul
6e0682377c Merge "Fix broken configdrive_use_object_store" 2021-02-23 18:08:57 +00:00
Dmitry Tantsur
73bdebd127 Fix broken configdrive_use_object_store
When it is set to True, we try to write text data to a binary file,
which is not possible in Python 3. The issue has been "helpfully"
hidden by the fact that we use bytes in unit tests, as well as
by lack of CI coverage.

Change-Id: Ibbf90dcbcb36a5f7cf084a44a221c0c5c003b95a
2021-02-18 10:25:07 +01:00
Zuul
6b9d7fa407 Merge "devstack: support installing ironic-lib from source in DIB IPA" 2021-02-18 04:04:40 +00:00
Zuul
52ff615c98 Merge "Guard conductor from consuming all of the ram" 2021-02-12 18:11:57 +00:00
Dmitry Tantsur
189b5e40cd devstack: support installing ironic-lib from source in DIB IPA
Depends-On: https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/775153
Change-Id: I8734776bf59b5a34327624184c1c2360ccda330a
2021-02-11 14:46:49 +01:00
Vanou Ishii
13e77e2179 Fix Mis-Ordering of Bash Variable Definition in DevStack
In devstack/lib/ironic, IRONIC_DEPLOY_DRIVER is defined at line 341.
However variables which use IRONIC_DEPLOY_DRIVER in default value
(e.g. IRONIC_DEPLOY_RAMDISK, IRONIC_DEPLOY_KERNEL, IRONIC_DEPLOY_ISO
and IRONIC_EFIBOOT) are defined at line 276-282.

This will cause problem at line 295-296:

 if [[ "$IRONIC_BUILD_DEPLOY_RAMDISK" == "False" && \
         ! (-e "$IRONIC_DEPLOY_RAMDISK" && -e "$IRONIC_DEPLOY_KERNEL")

So, this commit moves definition of IRONIC_DEPLOY_DRIVER before
its first use.

Change-Id: I74acb32714ce8830d4697fc796146b894aa7d8c9
2021-02-01 10:17:39 +09:00
Julia Kreger
d9913370de Guard conductor from consuming all of the ram
One of the biggest frustrations larger operators have is when they
trigger a massive number of concurrent deployments. As one would
expect, the memory utilization of the conductor goes up. Except,
even with the default number of worker threads, if we're requested
to convert 80 images at the same time, or to perform the write-out
to the remote node at the same time, we will consume a large amount
of system RAM. Or more specifically, qemu-img will consume a large
amount of memory.

If the amount of memory goes too low, the system can trigger
OOMKiller which will slay processes using ram. Ideally, we do not
want this to happen to our conductor process, much less the work
that is being performed, so we need to add some guard rails to help
keep us from entering into situations where we may compromise the
conductor by taking on too much work.

Adds a guard in the conductor to prevent multiple parallel
deployment operations from running the conductor out of memory.

With the defaults, the conductor will attempt to throttle back
automatically and hold worker threads which will slow down the
amount of work also proceeding through the conductor, as we are
in a memory condition where we should be careful about the work.

The defaults allow this to occur for a total of 15 seconds between
re-check of available RAM, for a total number of six retries.
The minimum default is 1024 (MB), as this is the amount of memory
qemu-img allocates when trying to write images. This quite literally
means no additional qemu-img process can spawn until the default
memory situation has resolved itself.

Change-Id: I69db0169c564c5b22abd0cb1b890f409c13b0ac2
2021-01-29 14:33:57 -08:00
Zuul
6c9e28dd50 Merge "Inject TLS certificate when using virtual media" 2020-12-19 22:14:12 +00:00
Dmitry Tantsur
8b83e9ec62 Revert "devstack: build DIB images with CentOS Stream by default"
This reverts commit 05f2c8b79f0d6b7e9200bbc531ff621d2029da2e.

It is being reverted as the centos stream images
contain extra, un-necessary libraries and packages
installed which swells the ramdisk size up substantially
and is causing failures in CI as the compressed image size
expanded by about 100MB, and uncompressed the stream images
are 1.1GB.

Change-Id: Icc3a18ed12d309fd9a00f02d5e703dfeda50e86b
2020-12-15 14:20:13 +00:00
Dmitry Tantsur
628109f960 Inject TLS certificate when using virtual media
A new option allows embedding a CA certificate in the virtual media
ISO to allow fully secure TLS between ironic and IPA.

Depends-On: https://review.opendev.org/763207
Change-Id: Idaacf44fd829c441d708b11704a97f9cd2b7a74c
2020-12-15 13:41:50 +01:00
Dmitry Tantsur
05f2c8b79f devstack: build DIB images with CentOS Stream by default
Change-Id: I50edd6b2740a26d00be19abc58c3ff770417fb68
2020-12-11 12:02:45 +01:00
Dmitry Tantsur
31f3f9fca1 Document how to build an ESP image for redfish-virtual-media
Also update the devstack plugin to use the same procedure.

Based on https://review.opendev.org/760423.

Change-Id: I8e20ad0fbc7e62e418b24ef56425328ec3a201b0
2020-11-10 19:19:07 +01:00
Zuul
7080f2ce20 Merge "devstack: log all requests to sushy-emulator" 2020-11-02 12:36:31 +00:00
Zuul
09e246294b Merge "CI: increase cleaning timeout and tie it to PXE boot timeout" 2020-10-30 19:19:20 +00:00