Planet Ubuntu

Incus is a manager for virtual machines (VM) and system containers. There is also an Incus support forum.

A virtual machine (VM) is an instance of an operating system that runs on a computer, along with the main operating system. A virtual machine uses hardware virtualization features for the separation from the main operating system. With virtual machines, the full operating system boots up in them.

A system container is an instance of an operating system that also runs on a computer, along with the main operating system. A system container, instead, uses security primitives of the Linux kernel for the separation from the main operating system. You can think of system containers as software virtual machines. System containers reuse the running Linux kernel of the host, therefore you can only have Linux system containers, any Linux distribution.

In this post we see how to create a VM with Incus, install Incus into that VM, and then create a VM through the inner Incus installation. This is also called nested virtualization. Incus works fine with nested virtualization. Any pitfalls arise from the settings of the host (BIOS/UEFI settings, host Linux kernel, etc). We’ll see these together, step by step.

Table of Contents

• Configuring your hardware for virtualization
• Testing your host for virtualization
• Testing your host for nested virtualization
• Launching the outer Incus VM
• Conclusion

Configuring your hardware for virtualization

You would need to enter into the BIOS/UEFI settings and enable the option for VT-x (for Intel CPUs) or AMD-V (for AMD CPUs) virtualization. If you are unsure, you can just follow the instructions in the next step which will complain if you have not enabled the appropriate BIOS/UEFI settings.

As a sidenote there is another setting, Intel VT-d (for Intel CPUs) or AMD-Vi (for AMD CPUs) that allow to move a supported hardware device (like a GPU, if you have more than one) into the VM. Not essential for what we are testing, but keep that in mind if you get too deep into virtualization.

There are also some additional options that are optional, AMD Nested Page tables (NPT) (for AMD) and Rapid Virtualization Indexing (RVI)/Intel Extended Page Tables (EPT) for Intel. These help for performance.

Testing your host for virtualization

The Linux kernel that is available in most Linux distributions supports the KVM hypervisor for virtualization.

Applications use the libvirttoolkit to access the virtualization features.

In order to test if our host supports virtualization, we install cpu-checker and the libvirt-clients packages on the host and then run kvm-ok and virt-host-validate respectively to verify our system. Compared between the two utilities, the latter is better. However, I am including cpu-checkeras it is covered in lots of documentation.

$ sudo apt install -y cpu-checker libvirt-clients ... $ kvm-ok INFO: /dev/kvm exists KVM acceleration can be used $ sudo virt-host-validate QEMU: Checking for hardware virtualization : PASS QEMU: Checking if device /dev/kvm exists : PASS QEMU: Checking if device /dev/kvm is accessible : PASS QEMU: Checking if device /dev/vhost-net exists : PASS QEMU: Checking if device /dev/net/tun exists : PASS QEMU: Checking for cgroup 'cpu' controller support : PASS QEMU: Checking for cgroup 'cpuacct' controller support : PASS QEMU: Checking for cgroup 'cpuset' controller support : PASS QEMU: Checking for cgroup 'memory' controller support : PASS QEMU: Checking for cgroup 'devices' controller support : PASS QEMU: Checking for cgroup 'blkio' controller support : PASS QEMU: Checking for device assignment IOMMU support : PASS QEMU: Checking if IOMMU is enabled by kernel : PASS QEMU: Checking for secure guest support : WARN (Unknown if this platform has Secure Guest support) LXC: Checking for Linux >= 2.6.26 : PASS LXC: Checking for namespace ipc : PASS LXC: Checking for namespace mnt : PASS LXC: Checking for namespace pid : PASS LXC: Checking for namespace uts : PASS LXC: Checking for namespace net : PASS LXC: Checking for namespace user : PASS LXC: Checking for cgroup 'cpu' controller support : PASS LXC: Checking for cgroup 'cpuacct' controller support : PASS LXC: Checking for cgroup 'cpuset' controller support : PASS LXC: Checking for cgroup 'memory' controller support : PASS LXC: Checking for cgroup 'devices' controller support : PASS LXC: Checking for cgroup 'freezer' controller support : FAIL (Enable 'freezer' in kernel Kconfig file or mount/enable cgroup controller in your system) LXC: Checking for cgroup 'blkio' controller support : PASS LXC: Checking if device /sys/fs/fuse/connections exists : PASS $

If you get a failure, try to identify whether the issue is with your computer’s firmware or with the Linux kernel of your host. If in doubt, post below the output.

Why does the output mention both QEMU and LXC? By default, the command shows all libvirt virtualization support, unless you specify something specific. If you wanted only the QEMU output, you would run sudo virt-host-validate qemu. Note that the LXC here is not the Linux Containers LXC. The LXC above is the Libvirt LXC.

Testing your host for nested virtualization

I have not noticed any mention for nested virtualization in the output of virt-host-validate. If you know a tool that shows that information, write it in the comments. In the absence of such a tool, let’s check manually.

If you have an AMD CPU, run the following. If you get 1, then nested virtualization through KVM works.

$ cat /sys/module/kvm_amd/parameters/nested 1 $

If instead you have an Intel CPU, run the following. If you get Y(instead of 1), then nested virtualization through KVM works.

$ cat /sys/module/kvm_intel/parameters/nested Y $

If instead you get an error (such as the following), then something is wrong. Report back your CPU model and motherboard, along with the Linux kernel version and Linux distribution.

cat: /sys/module/kvm_intel/parameters/nested: No such file or directory Launching the outer Incus VM

We launch the outer VM. Get a shell into the outer VM, install Incus and those utilities that show whether KVM virtualization works. Then, we launch an Alpine VM in the outer VM. We get an error regarding Secure Boot (the Alpine Linux kernel is not signed), remove the stuck VM and launch again with Secure Boot disabled. Finally, we get a shell into the inner VM.

$ incus launch images:debian/12 outervm --vm Launching outervm $ incus shell outervm root@outervm:~# # Install Incus according to the documentation. root@outervm:~# sudo apt install -y cpu-checker libvirt-clients ... root@outervm:~# virt-host-validate QEMU: Checking for hardware virtualization : PASS QEMU: Checking if device /dev/kvm exists : PASS QEMU: Checking if device /dev/kvm is accessible : PASS QEMU: Checking if device /dev/vhost-net exists : PASS QEMU: Checking if device /dev/net/tun exists : PASS ... root@outervm:~# incus launch images:alpine/edge innervm --vm Launching innervm Error: Failed instance creation: The image used by this instance is incompatible with secureboot. Please set security.secureboot=false on the instance root@outervm:~# incus delete innervm root@outervm:~# incus launch images:alpine/edge innervm --vm --config security.secureboot=false Launching innervm root@outervm:~# incus list -c ns4t +---------+---------+-----------------------+-----------------+ | NAME | STATE | IPV4 | TYPE | +---------+---------+-----------------------+-----------------+ | innervm | RUNNING | 10.227.169.165 (eth0) | VIRTUAL-MACHINE | +---------+---------+-----------------------+-----------------+ root@outervm:~# uname -a Linux outervm 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux root@outervm:~# incus shell innervm innervm:~# uname -a Linux innervm 6.6.22-1-virt #2-Alpine SMP PREEMPT_DYNAMIC Thu, 14 Mar 2024 02:12:52 +0000 x86_64 Linux innervm:~# Conclusion

We saw how to verify whether our host is able to work with hardware virtualization. This involves checking both the computer firmware settings (BIOS/UEFI) and the host Linux kernel.

Then, we created an outer VM with Incus, got a shell into there, installed Incus, and launched an inner (nested) VM.

I wonder whether we can go further and create a VM inside the inner VM. If you go through these and try to create an inner inner VM, post the error message. It does not feel like it should be possible.

Simos Xenitellisblog.simos.info/

The last Incus release before we go LTS has now been released!

This is quite the feature packed release as this is meant to include just about every features we want in Incus 6.0 LTS except for a few last minute minor additions.

You’ll find new features for just about everyone, from multi-cluster networking with the new network integrations, to enhanced performance on multi-socket servers with the improved NUMA support, to easier authentication with JSON Web Token support, to I/O limits for virtual machines and more USB passthrough options.

The full announcement and changelog can be found here.
And for those who prefer videos, here’s the release overview video:

You can take the latest release of Incus up for a spin through our online demo service at: https://linuxcontainers.org/incus/try-it/

And as always, my company is offering commercial support on Incus, ranging from by-the-hour support contracts to one-off services on things like initial migration from LXD, review of your deployment to squeeze the most out of Incus or even feature sponsorship. You’ll find all details of that here: https://zabbly.com/incus

Donations towards my work on this and other open source projects is also always appreciated, you can find me on Github Sponsors, Patreon and Ko-fi.

Enjoy!

I’ve recently switched back from vscode to Sublime Text, which means that after all the time I spent training my fingers to type “code somefile.txt” instead of “subl somefile.txt” I now need to undo all that conditioning and go back to subl again. So I thought, hey, maybe I should dump a little shell script called code in my bin folder which admonished me in some amusing way, thus Pavlov-ing myself into learning to do it right.

And then I thought, hey, what’d be cool is if I had that Matrix-esque “raining code” effect in the Terminal and then it was superimposed with a box saying “STOP TYPING code AND USE subl INSTEAD”, like the “SYSTEM ERROR” message at the end of the first movie.

And then I thought: someone’s already done this, right? And they have; it is called cmatrix. But I don’t like cmatrix because it doesn’t do the colours right; the text just sorta stops rather than fading away like the movie does, and it feels unreal and too sharp for me. Now, don’t get me wrong, I understand why this is; terminals support a full proper range of colour these days, but writing a program which gets released to actual people and which can deal with the bewildering array of terminal settings out there is a miserable waste of everyone’s time. But I’m not writing this for anyone else; it only has to work in my terminal (in true works on my machine fashion). And this will give me a chance to noodle about with Python terminal libraries such as blessed to make something interesting. Hence, matrix24.py:

It’s a bodge all round, and it still doesn’t look right, and Jess pointed out that making something cool happen when I make a mistake is the opposite of conditioning, but I got to fiddle about with a new library for a bit, so that was fun. Can I do something productive now?

(title from a classic post about the Matrix which still makes me laugh even after all these years, although it is very unfair to Keanu Reeves who is a cool bloke and should be emulated in his approach to life)

Multipass is pretty useful but what a pain this was to figure out, due to Ubuntu’s Node.js package not working with AWS-CDK.

Multipass lets you manage VM in Ubuntu and can take cloud-init scripts as a parameter. I wanted an Ubuntu LTS instance with AWS CDK, which needs Node.js and python3-venv.

#cloud-config packages: - python3-venv - unzip package_update: true package_upgrade: true write_files: - path: "/etc/environment" append: true content: | export PATH=\ /opt/node-v20.11.1-linux-x64/bin:\ /usr/local/sbin:/usr/local/bin:\ /usr/sbin:/usr/bin:/sbin:/bin:\ /usr/games:/usr/local/games:\ /snap/bin runcmd: - wget https://nodejs.org/dist/v20.11.1/node-v20.11.1-linux-x64.tar.xz - tar xvf node-v20.11.1-linux-x64.tar.xz -C /opt - export PATH=/opt/node-v20.11.1-linux-x64/bin:$PATH - npm install -g npm@latest - npm install -g aws-cdk - git config --system user.name "Dougie Richardson" - git config --system user.email "xx@xxxxxxxxx.com" - git config --system init.defaultBranch main - wget https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip - unzip awscli-exe-linux-x86_64.zip - ./aws/install

Save that as cdk.yaml and spin up an new instance:

multipass launch --name cdk --cloud-init cdk.yaml

There’s a couple useful things to note if you’re checking this out:

• Inside the VM there’s a useful log to assist debugging /var/log/cloud-init-output.log.
• While YAML has lots of ways to split text over multiple lines, when you don’t want space use a backslash.

Shell into the new VM with multipass shell cdk, then we can configure programmatic access and bootstrap CDK.

aws sso configure aws sso login --profile profile_name aws sts get-caller-identity --profile profile_name aws configure get region --profile profile_name

The last two commands give the account and region to bootstrap:

cdk bootstrap aws://account_number/region --profile profile_name

A Crowning Achievement

As 24.04 LTS will represent the eighth Long-Term Support release of Ubuntu Studio and its 32nd release. For this release, we wanted to make sure we got some great representation from the community in terms of wallpaper, and while there weren’t as many entries as our previous competition, we were blown out of the way in terms of quality. While not every wallpaper could be included, all of the entries were solid, and narrowing it down to the best of the best was very difficult.

Revealing The Default

Our long-time art lead, Eylul Dogruel, worked diligently on making a quality textured default wallpaper that not only works well for traditional horizontal screens, but for vertical screens as well without losing quality. We have two variations: one with our logo, and one with the mascot that will be rotated-out for the next four releases.

Now to Crown the Winners!

As stated, this was a very difficult decision, but we would like to congratulate the winners of the competition! The full-quality images will be included in Ubuntu Studio 24.04 LTS and are already in our daily builds of Noble Numbat.

Interference by Uday Nakade Glass Wave 1 Light by Alastair Temple Bee 2 by Liber Dovat Brauneck Sunrise by Uday Nakade Banaue by Jean-Daniel Bancal

On Mastodon, the question came up of how Ubuntu would deal with something like the npm install everything situation. I replied:

Ubuntu is curated, so it probably wouldn’t get this far. If it did, then the worst case is that it would get in the way of CI allowing other packages to be removed (again from a curated system, so people are used to removal not being self-service); but the release team would have no hesitation in removing a package like this to fix that, and it certainly wouldn’t cause this amount of angst.

If you did this in a PPA, then I can’t think of any particular negative effects.

OK, if you added lots of build-dependencies (as well as run-time dependencies) then you might be able to take out a builder. But Launchpad builders already run arbitrary user-submitted code by design and are therefore very carefully sandboxed and treated as ephemeral, so this is hardly novel.

There’s a lot to be said for the arrangement of having a curated system for the stuff people actually care about plus an ecosystem of add-on repositories. PPAs cover a wide range of levels of developer activity, from throwaway experiments to quasi-official distribution methods; there are certainly problems that arise from it being difficult to tell the difference between those extremes and from there being no systematic confinement, but for this particular kind of problem they’re very nearly ideal. (Canonical has tried various other approaches to software distribution, and while they address some of the problems, they aren’t obviously better at helping people make reliable social judgements about code they don’t know.)

For a hypothetical package with a huge number of dependencies, to even try to upload it directly to Ubuntu you’d need to be an Ubuntu developer with upload rights (or to go via Debian, where you’d have to clear a similar hurdle). If you have those, then the first upload has to pass manual review by an archive administrator. If your package passes that, then it still has to build and get through proposed-migration CI before it reaches anything that humans typically care about.

On the other hand, if you were inclined to try this sort of experiment, you’d almost certainly try it in a PPA, and that would trouble nobody but yourself.

Launchpad’s new homepage

Launchpad has been around for a while, and its frontpage has remained untouched for a few years now.

If you go into launchpad.net, you’ll notice it looks quite different from what it has looked like for the past 10 years – it has been updated! The goal was to modernize it while trying to keep it looking like Launchpad. The contents have remained the same with only a few text additions, but there were a lot of styling changes.

The most relevant change is that the frontpage now uses Vanilla components (https://vanillaframework.io/docs). This alone, not only made the layout look more modern, but also made it better for a new curious user reaching the page from a mobile device. The accessibility score of the page – calculated with Google’s Lighthouse extension – increased from a 75 to an almost perfect 98!

Given the frontpage is so often the first impression users get when they want to check out Launchpad, we started there. But in the future, we envision the rest of Launchpad looking more modern and having a more intuitive UX.

As a final note, thank you to Peter Makowski for always giving a helping hand with frontend changes in Launchpad.

If you have any feedback for us, don’t forget to reach out in any of our channels. For feature requests you can reach us as feedback@launchpad.net or open a report in https://bugs.launchpad.net/launchpad.

To conclude this post, here is what Launchpad looked like in 2006, yesterday and today.

Launchpad in 2006 Launchpad yesterday Launchpad today

Plasma Pass is a Plasma applet for the Pass password manager

This release includes build fixes for Plasma 6, due to be released later this week.

URL: https://download.kde.org/stable/plasma-pass/
Sha256: 2a726455084d7806fe78bc8aa6222a44f328b6063479f8b7afc3692e18c397ce
Signed by E0A3EB202F8E57528E13E72FD7574483BB57B18D Jonathan Esk-Riddell <jr@jriddell.org>
https://jriddell.org/esk-riddell.gpg

Thanks to all the hard work from our contributors, Lubuntu 22.04.4 LTS has been released. With the codename Jammy Jellyfish, Lubuntu 22.04 is the 22nd release of Lubuntu, the eighth release of Lubuntu with LXQt as the default desktop environment. Support lifespan Lubuntu 22.04 LTS will be supported for 3 years until April 2025. Our […]

We are pleased to announce the release of the next version of our distro, the fourth 22.04 LTS point release. The LTS version is supported for 3 years while the regular releases are supported for 9 months. The new release rolls-up various fixes and optimizations by Ubuntu Budgie team, that have been released since the 22.04.3 release in August: For the adventurous among our community we…

Source

Oxygen Icons is an icon theme for use with any XDG compliant app and desktop.

It is part of KDE Frameworks 6 but is now released independently to save on resources.

This 6.0.0 release requires to be built with extra-cmake-modules from KF 6 which is not yet released, distros may want to wait until next week before building it.

Distros which ship this version can drop the version released as part of KDE Frameworks 5.

sha256: 28ec182875dcc15d9278f45ced11026aa392476f1f454871b9e2c837008e5774

URL: https://download.kde.org/stable/oxygen-icons/

Signed by E0A3EB202F8E57528E13E72FD7574483BB57B18D Jonathan Esk-Riddell <jr@jriddell.org>
https://jriddell.org/esk-riddell.gpg

There’s a YouTube channel called Clickspring, run by an Australian bloke called Chris who is a machinist: a mechanical engineer with a lathe and a mill and all manner of little tools. I am not a machinist — at school I was fairly inept at what we called CDT, for Craft Design and Technology, and what Americans much more prosaically call “shop class”. My dad was, though, or an engineer at least. Although Chris builds clocks and beautiful brass mechanisms, and my dad built aeroplanes. Heavy engineering. All my engineering is software, which actual engineers don’t think is engineering at all, and most of the time I don’t either.

You can romanticise it: claim that software development isn’t craft, it’s art. And there is a measure of truth in this. It’s like writing, which is the other thing I spend a lot of time doing for money; that’s an art, too.

If you’re doing it right, at least.

Most of the writing that’s done, though, isn’t art. And most of the software development isn’t, either. Or most of the engineering. For every one person creating beauty in prose or code or steel, there are fifty just there doing the job with no emotional investment in what they’re doing at all. Honestly, that’s probably a good thing, and not a complaint. While I might like the theoretical idea of a world where everything is hand made by someone who cares, I don’t think that you should have to care in order to get paid. The people who are paying you don’t care, so you shouldn’t have to either.

It’s nice if you can swing it so you get both, though.

The problem is that it’s not possible to instruct someone to give a damn. You can’t regulate the UK government into giving a damn about people who fled a war to come here to find their dream of being a nurse, you can’t regulate Apple bosses into giving a damn about the open web, you can’t regulate CEOs into giving a damn about their employees or governments about their citizens or landlords about their tenants. That’s not what regulation is for; people who give a damn largely don’t need regulation because they want to do the right thing. They might need a little steering into knowing what the right thing is, but that’s not the same.

No, regulation is there as a reluctant compromise: since you can’t make people care, the best you can do is in some rough and ready fashion make them behave in a similar way to the way they would if they did. Of course, this is why the most insidious kind of response is not an attempt to evade responsibility but an attack on the system of regulation itself. Call judges saboteurs or protesters criminals or insurgents patriots. And why the most heinous betrayal is one done in the name of the very thing you’re destroying. Claim to represent the will of the people while hurting those people. Claim to be defending the law while hiding violence and murder behind a badge. Claim privacy as a shield for surveillance or for exclusion. We all sorta thought that the system could protect us, that those with the power could be trusted to use it at least a little responsibly. And the last year has been one more in a succession of years demonstrating just how wrong that is. This and no other is the root from which a tyrant springs; when he first appears he is a protector.

The worst thing about it is that the urge to protect other people is not only real but the best thing about ourselves. When it’s actually real. Look after others, especially those who need it, and look after yourself, because you’re one of the people who needs it.

Chris from Clickspring polishes things to a high shine using tin, which surprised me. I thought bringing out the beauty in something needed a soft cloth but no, it’s done with metal. Some things, like silver, are basically shiny with almost no effort; there’s a reason people have prized silver things since before we could even write down why, and it’s not just because you could find lumps of it lying around the place with no need to build a smelting furnace. Silver looks good, and makes you look good in turn. Tin is useful, and it helps polish other things to a high shine.

Today’s my 48th birthday. A highly composite number. The ways Torah wisdom is acquired. And somewhere between silver and tin. That sounds OK to me.

Linux kernel getting a livepatch whilst running a marathon. Generated with AI.

Livepatch service eliminates the need for unplanned maintenance windows for high and critical severity kernel vulnerabilities by patching the Linux kernel while the system runs. Originally the service launched in 2016 with just a single kernel flavour supported.

Over the years, additional kernels were added: new LTS releases, ESM kernels, Public Cloud kernels, and most recently HWE kernels too.

Recently livepatch support was expanded for FIPS compliant kernels, Public cloud FIPS compliant kernels, and as well IBM Z (mainframe) kernels. Bringing the total of kernel flavours support to over 60 distinct kernel flavours supported in parallel. The table of supported kernels in the documentation lists the supported kernel flavours ABIs, the duration of individual build's support window, supported architectures, and the Ubuntu release. This work was only possible thanks to the collaboration with the Ubuntu Certified Public Cloud team, engineers at IBM for IBM Z (s390x) support, Ubuntu Pro team, Livepatch server & client teams.

It is a great milestone, and I personally enjoy seeing the non-intrusive popup on my Ubuntu Desktop that a kernel livepatch was applied to my running system. I do enable Ubuntu Pro on my personal laptop thanks to the free Ubuntu Pro subscription for individuals.

What's next? The next frontier is supporting ARM64 kernels. The Canonical kernel team has completed the gap analysis to start supporting Livepatch Service for ARM64. Upstream Linux requires development work on the consistency model to fully support livepatch on ARM64 processors. Livepatch code changes are applied on a per-task basis, when the task is deemed safe to switch over. This safety check depends mostly on kernel stacktraces. For these checks, CONFIG_HAVE_RELIABLE_STACKTRACE needs to be available in the upstream ARM64 kernel. (see The Linux Kernel Documentation). There are preliminary patches that enable reliable stacktraces on ARM64, however these turned out to be problematic as there are lots of fix revisions that came after the initial patchset that AWS ships with 5.10. This is a call for help from any interested parties. If you have engineering resources and are interested in bringing Livepatch Service to your ARM64 platforms, please reach out to the Canonical Kernel team on the public Ubuntu Matrix, Discourse, and mailing list. If you want to chat in person, see you at FOSDEM next weekend.

This is a follow up to my previous post about How to test things with openQA without running your own instance, so you might want to read that first.

Now, while hunting for bsc#1219073 which is quite sporadic, and took quite some time to show up often enough so that became noticeable and traceable, once stars aligned and managed to find a way to get a higher failure rate, I wanted to have a way for me and for the developer to test the kernel with the different patches to help with the bisecting and ease the process of finding the culprit and finding a solution for it.

I came with a fairly simple solution, using the --repeat parameter of the openqa-cli tool, and a simple shell script to run it:

```bash $ cat ~/Downloads/trigger-kernel-openqa-mdadm.sh # the kernel repo must be the one without https; tests don't have the kernel CA installed KERNEL="KOTD_REPO=http://download.opensuse.org/repositories/Kernel:/linux-next/standard/" REPEAT="--repeat 100" # using 100 by default JOBS="https://openqa.your.instan.ce/tests/13311283 https://openqa.your.instan.ce/tests/13311263 https://openqa.your.instan.ce/tests/13311276 https://openqa.your.instan.ce/tests/13311278" BUILD="bsc1219073" for JOB in $JOBS; do openqa-clone-job --within-instance $JOB CASEDIR=https://github.com/foursixnine/os-autoinst-distri-opensuse.git#tellmewhy ${REPEAT} \ _GROUP=DEVELOPERS ${KERNEL} BUILD=${BUILD} FORCE_SERIAL_TERMINAL=1\ TEST="${BUILD}_checkmdadm" YAML_SCHEDULE=schedule/qam/QR/15-SP5/textmode/textmode-skip-registration-extra.yaml INSTALLONLY=0 DESKTOP=textmode\ |& tee jobs-launched.list; done;

There are few things to note here:

• the kernel repo must be the one without https; tests don’t have the CA installed by default.
• the --repeat parameter is set to 100 by default, but can be changed to whatever number is desired.
• the JOBS variable contains the list of jobs to clone and run, having all supported architecures is recommended (at least for this case)
• the BUILD variable can be anything, but it’s recommended to use the bug number or something that makes sense.
• the TEST variable is used to set the name of the test as it will show in the test overview page, you can use TEST+=foo if you want to append text instead of overriding it, the --repeat parameter, will append a number incrementally to your test, see os-autoinst/openQA#5331 for more details.
• the YAML_SCHEDULE variable is used to set the yaml schedule to use, there are other ways to modify the schedule, but in this case I want to perform a full installation

Running the script

• Ensure you can run at least the openQA client; if you need API keys, see post linked at the beginning of this post
• replace the kernel repo with your branch in line 5
• run the script $ bash trigger-kernel-openqa-mdadm.sh and you should get the following, times the --repeat if you modified it

1 job has been created: - sle-15-SP5-Full-QR-x86_64-Build134.5-skip_registration+workaround_modules@64bit -> https://openqa.your.instan.ce/tests/13345270

Each URL, will be a job triggered in openQA, depending on the load and amount of jobs, you might need to wait quite a bit (some users can help moving the priority of these jobs so it executes faster)

The review stuff: Looking at the results

• Go to https://openqa.your.instan.ce/tests/overview?distri=sle&build=bsc1219073&version=15-SP5 or from any job from the list above click on Job groups menu at the top, and select Build bsc1219073
• Click on “Filter”
• type the name of the test module to filter in the field Module name, e.g mdadm, and select the desired result of such test module e.g failed (you can also type, and select multiple result types)
• Click Apply
• The overall summary of the build overview page, will provide you with enough information to calculate the pass/fail rate.

A rule of thumb: anything above 5% is bad, but you need to also understand your sample size + the setup you’re using; YMMV.

Ain’t nobody got time to wait

The script will generate a file called: jobs-launched.list, in case you absolutely need to change the priority of the jobs, set it to 45, so it runs higher than default priority, which is 50 cat jobs-launched.list | grep https | sed -E 's/^.*->\s.*tests\///' | xargs -r -I {} bash -c "openqa-cli api --osd -X POST jobs/{}/prio prio=45; sleep 1"

The magic

The actual magic is in the schedule, so right after booting the system and setting it up, before running the mdadm test, I inserted the update_kernel module, which will add the kernel repo specified by KOTD_REPO, and install the kernel from there, reboot the system, and leave the system ready for the actual test, however I had to add very small changes:

--- tests/kernel/update_kernel.pm | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/kernel/update_kernel.pm b/tests/kernel/update_kernel.pm index 1d6312bee0dc..048da593f68f 100644 --- a/tests/kernel/update_kernel.pm +++ b/tests/kernel/update_kernel.pm @@ -398,7 +398,7 @@ sub boot_to_console { sub run { my $self = shift; - if ((is_ipmi && get_var('LTP_BAREMETAL')) || is_transactional) { + if ((is_ipmi && get_var('LTP_BAREMETAL')) || is_transactional || get_var('FORCE_SERIAL_TERMINAL')) { # System is already booted after installation, just switch terminal select_serial_terminal; } else { @@ -476,7 +476,7 @@ sub run { reboot_on_changes; } elsif (!get_var('KGRAFT')) { power_action('reboot', textmode => 1); - $self->wait_boot if get_var('LTP_BAREMETAL'); + $self->wait_boot if (get_var('FORCE_SERIAL_TERMINAL') || get_var('LTP_BAREMETAL')); } }

Likely I’ll make a new pull request to have this in the test distribution, but for now this is good enough to help kernel developers to do some self-service and trigger their own openQA tests, that have many more tests (hopefully in parallel) and faster than if there was a person doing all of this manually.

Special thanks to the QE Kernel team, who do the amazing job of thinking of some scenarios like this, because they save a lot of time.

layout: post title: Testing kernels with sporadic issues until heisenbug shows in openQA categories:

• blog
• linux
• tech
• qualityassurance
• openqa generated on: 2024-01-25 — This is a follow up to my previous post about How to test things with openQA without running your own instance, so you might want to read that first.

Now, while hunting for bsc#1219073 which is quite sporadic, and took quite some time to show up often enough so that became noticeable and traceable, once stars aligned and managed to find a way to get a higher failure rate, I wanted to have a way for me and for the developer to test the kernel with the different patches to help with the bisecting and ease the process of finding the culprit and finding a solution for it.

I came with a fairly simple solution, using the --repeat parameter of the openqa-cli tool, and a simple shell script to run it:

```bash $ cat ~/Downloads/trigger-kernel-openqa-mdadm.sh # the kernel repo must be the one without https; tests don't have the kernel CA installed KERNEL="KOTD_REPO=http://download.opensuse.org/repositories/Kernel:/linux-next/standard/" REPEAT="--repeat 100" # using 100 by default JOBS="https://openqa.your.instan.ce/tests/13311283 https://openqa.your.instan.ce/tests/13311263 https://openqa.your.instan.ce/tests/13311276 https://openqa.your.instan.ce/tests/13311278" BUILD="bsc1219073" for JOB in $JOBS; do openqa-clone-job --within-instance $JOB CASEDIR=https://github.com/foursixnine/os-autoinst-distri-opensuse.git#tellmewhy ${REPEAT} \ _GROUP=DEVELOPERS ${KERNEL} BUILD=${BUILD} FORCE_SERIAL_TERMINAL=1\ TEST="${BUILD}_checkmdadm" YAML_SCHEDULE=schedule/qam/QR/15-SP5/textmode/textmode-skip-registration-extra.yaml INSTALLONLY=0 DESKTOP=textmode\ |& tee jobs-launched.list; done;

There are few things to note here:

• the kernel repo must be the one without https; tests don’t have the CA installed by default.
• the --repeat parameter is set to 100 by default, but can be changed to whatever number is desired.
• the JOBS variable contains the list of jobs to clone and run, having all supported architecures is recommended (at least for this case)
• the BUILD variable can be anything, but it’s recommended to use the bug number or something that makes sense.
• the TEST variable is used to set the name of the test as it will show in the test overview page, you can use TEST+=foo if you want to append text instead of overriding it, the --repeat parameter, will append a number incrementally to your test, see os-autoinst/openQA#5331 for more details.
• the YAML_SCHEDULE variable is used to set the yaml schedule to use, there are other ways to modify the schedule, but in this case I want to perform a full installation

Running the script

• Ensure you can run at least the openQA client; if you need API keys, see post linked at the beginning of this post
• replace the kernel repo with your branch in line 5
• run the script $ bash trigger-kernel-openqa-mdadm.sh and you should get the following, times the --repeat if you modified it

1 job has been created: - sle-15-SP5-Full-QR-x86_64-Build134.5-skip_registration+workaround_modules@64bit -> https://openqa.your.instan.ce/tests/13345270

Each URL, will be a job triggered in openQA, depending on the load and amount of jobs, you might need to wait quite a bit (some users can help moving the priority of these jobs so it executes faster)

The review stuff: Looking at the results

• Go to https://openqa.your.instan.ce/tests/overview?distri=sle&build=bsc1219073&version=15-SP5 or from any job from the list above click on Job groups menu at the top, and select Build bsc1219073
• Click on “Filter”
• type the name of the test module to filter in the field Module name, e.g mdadm, and select the desired result of such test module e.g failed (you can also type, and select multiple result types)
• Click Apply
• The overall summary of the build overview page, will provide you with enough information to calculate the pass/fail rate.

A rule of thumb: anything above 5% is bad, but you need to also understand your sample size + the setup you’re using; YMMV.

Ain’t nobody got time to wait

The script will generate a file called: jobs-launched.list, in case you absolutely need to change the priority of the jobs, set it to 45, so it runs higher than default priority, which is 50 cat jobs-launched.list | grep https | sed -E 's/^.*->\s.*tests\///' | xargs -r -I {} bash -c "openqa-cli api --osd -X POST jobs/{}/prio prio=45; sleep 1"

The magic

The actual magic is in the schedule, so right after booting the system and setting it up, before running the mdadm test, I inserted the update_kernel module, which will add the kernel repo specified by KOTD_REPO, and install the kernel from there, reboot the system, and leave the system ready for the actual test, however I had to add very small changes:

--- tests/kernel/update_kernel.pm | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/kernel/update_kernel.pm b/tests/kernel/update_kernel.pm index 1d6312bee0dc..048da593f68f 100644 --- a/tests/kernel/update_kernel.pm +++ b/tests/kernel/update_kernel.pm @@ -398,7 +398,7 @@ sub boot_to_console { sub run { my $self = shift; - if ((is_ipmi && get_var('LTP_BAREMETAL')) || is_transactional) { + if ((is_ipmi && get_var('LTP_BAREMETAL')) || is_transactional || get_var('FORCE_SERIAL_TERMINAL')) { # System is already booted after installation, just switch terminal select_serial_terminal; } else { @@ -476,7 +476,7 @@ sub run { reboot_on_changes; } elsif (!get_var('KGRAFT')) { power_action('reboot', textmode => 1); - $self->wait_boot if get_var('LTP_BAREMETAL'); + $self->wait_boot if (get_var('FORCE_SERIAL_TERMINAL') || get_var('LTP_BAREMETAL')); } }

Likely I’ll make a new pull request to have this in the test distribution, but for now this is good enough to help kernel developers to do some self-service and trigger their own openQA tests, that have many more tests (hopefully in parallel) and faster than if there was a person doing all of this manually.

Special thanks to the QE Kernel team, who do the amazing job of thinking of some scenarios like this, because they save a lot of time.

Users can now add their Matrix accounts to their profile in Launchpad, as requested by Canonical’s Community team.

We also took the chance to slightly rework the frontend and how we display social accounts in the user profiles. Instead of having different sections in the profile for each social account , all social accounts are now all under a “Social Accounts” section.

Adding a new matrix account to your profile works similarly to how it has always worked for other accounts. Under the “Social Accounts” section in your user profile, you should now see a “No matrix accounts registered” and an edit button that will lead you to the Matrix accounts edit page. To edit, remove or add new ones, you will see an edit button in front of your newly added accounts in your profile.

We also added new API endpoints Person.social_accounts and Person.getSocialAccountsByPlatform() that will list the social accounts for a user. For more information, see our API documentation.

Currently, only Matrix was added as a social platform. But during this process, we made it much easier for Launchpad developers to add new social platforms to Launchpad in the future.