The KernelCI community had a great finish of the year. Since our last update here, we tackled some quite interesting features based on community feedback. See detailed updates in the sections below. 

New KernelCI TSC

KernelCI has announced its new Technical Steering Committee (TSC) composition following the annual election process. The committee, which guides the technical direction of the project and ensures cohesion across the Linux kernel testing ecosystem, now consists of seven members: Ben Copeland (Linaro) serving as TSC Chair, Denys Fedoryshchenko (Collabora) as Infrastructure Committee Lead, and community-elected members Greg KH (Linux Foundation), Gustavo Padovan (Collabora), Mark Brown (Arm), Minas Hambardzumyan (Texas Instruments), and Yogesh Lal (Qualcomm).

This diverse group brings together expertise from major hardware vendors, infrastructure specialists, and kernel maintainers, with the current term running until October 31, 2026. The TSC meets bi-weekly in open sessions to discuss technical decisions, roadmap planning, and ensure KernelCI continues to effectively support the Linux kernel community’s testing needs.

KernelCI meeting public calendar

As you may have noticed, the KernelCI community is growing. New TSC, new working groups, more community members with newcomers arriving frequently. To facilitate access we created a public calendar to announce all our meetings to the community. Through it, it is possible to register yourself to any KernelCI community meeting. Recordings are also available for those who want to watch the meeting at a later time.

Dashboard improvements

This quarter was very focused on the health of the project.

We have introduced monitoring over the API requests and system resources. On top of that, we had a significant increase of backend test coverage, from 40% to nearly 70%, including some benchmark tests. 

There is still plenty to be done, and we intend to continue refining the CI/CD of our project, further automating the delivery of future enhancements.

The main functional achievement we had was reaching the last milestone of KCIDB-ng, placing the ingestion of KCIDB submissions closer to the Django backend.

This allowed the team to detach the KCIDB schema from the dashboard database, which provides more freedom to normalize the data, making the API faster and reducing the storage footprint.

 We are still meeting every two weeks in the Kernel CI Working Group to explore ideas. One experiment we ran this quarter consisted of adding a few simple filters and columns about “Labs” in some of the tables. 

Pull-mode for labs

As part of the work in our Labs Working Group, the team has been working on the Maestro interface to allow labs to pull all the information they need about tests KernelCI can execute. That includes kernel builds, testing rootfs and anything else needed for organizations to run KernelCI tests on their own setup whatever lab technology they are using.

We believe that pull-mode will enable a multitude of labs that were not able to participate in KernelCI before. With pull-mode a lab can be behind a firewall, something quite common in many corporate environments. It is also lab technology agnostic as everything the lab has to do is to listen to Maestro API and then send results to our KCIDB API.

Linaro donating tuxmake, tuxrun, tuxlava

The following repositories are now live under the KernelCl GitHub namespace:

– https://github.com/kernelci/tuxmake

– https://github.com/kernelci/tuxrun

– https://github.com/kernelci/tuxlava 

KernelCI is the Linux kernel’s upstream testing and continuous integration ecosystem. The project has started to use Tuxmake and TuxRun to power its build and test pipelines. Moving these tools under the KernelCI namespace brings them into an active community that will help evolve them alongside the kernel itself.

Linaro engineers will continue as co-maintainers and active contributors. All three projects retain their MIT License, so existing users and contributors will see no disruption.

Blog post announcement: https://www.linaro.org/blog/linaro-transfers-kernel-building-and-testing-tools-to-the-kernelci-project/ 

Core Infrastructure

• More reliable event delivery: Added an optional mode that lets clients reconnect and catch up on missed events instead of losing them.
• Better event discovery: Expanded filtering options so users can find specific events more precisely.
• Faster event queries: Added database optimizations that speed up common event lookups
• Maintenance reliability: Fixed a cleanup issue so old data can be removed as intended
• Platform/tooling updates: Upgraded the database version and refreshed development tooling to improve stability
• Build/CI automation was expanded with a new production workflow and tweaks to container image build automation.
• Tooling/container configs were refreshed to newer toolchains and base images, with many older variants removed and patches updated.
• A large legacy build configuration catalog was removed, alongside updates to runtime and rootfs configuration data.
• A new pull-based lab runtime was added, including callback parsing, log handling, and result mapping; runtime configuration was extended to support it.
• LAVA handling was hardened for retry scenarios, missing logs, and infrastructure failure reporting.
• Storage uploads gained retry logic for network and server-side failures.
• Forecast reporting was upgraded to generate an HTML report in addition to console output.

kci-dev improvements

kci-dev continued evolving as a well-packaged CLI suite that Linux distributions and labs can ship, focused on helping engineers analyze KernelCI results and triage problems quickly from the terminal.

We had several improvements to the project in the past quarter:

• Released kci-dev v0.1.10, adding Arch Linux packaging support and more workflow/polish fixes, including Debian package build workflow updates and multiple fixes in issues/validation reporting and build-node filtering. 
• Released kci-dev v0.1.9, with additional triage and UX refinements on top of the previously reported Q3 feature set: moved the detect workflow under results issues, added a command to fetch new issues for a checkout, and restructured the issues command group for clearer usage.
• Improved validation and reporting ergonomics: enabled list views for boots validation, added runtime fields to boot/test JSON output, and fixed boot-origin filtering and other result-selection edge cases.
• Conference and community outreach (Tokyo, Dec 2025): presented “Getting Started With New KernelCI CLI Tools” at Open Source Summit Japan 2025, and at Linux Plumbers Conference 2025 ran the Kernel Testing & Dependability MC plus gave “KernelCI kci-dev: Closing the developer feedback loop”.

Final Thoughts

We will keep working on making KernelCI easier for the community to benefit from. From greater stability to an improved Web Dashboard and a more complete kci-dev CLI, there’s much more to enhance in KernelCI for everyone. Big thank you to the entire KernelCI community for making this progress possible!Talk to us at kernelci@lists.linux.dev , #kernelci IRC channel at Libera.chat or through our Discord server!

Contributed to this blog post Arisu Tachibana, Denys Fedoryshchenko, Gustavo Padovan, Minas Hambardzumyan and Tales Aparecida.

KernelCI Welcomes Arm and Qualcomm as Premier Members

We are thrilled to announce that two industry leaders, Arm and Qualcomm, have joined KernelCI as Premier members. This marks a significant milestone in our mission to ensure the quality, stability, and long-term maintenance of the Linux kernel through comprehensive testing across the broadest possible range of hardware platforms.

Both companies bring extensive expertise and resources that will significantly strengthen KernelCI’s testing ecosystem. Arm’s deep understanding of processor architecture and their commitment to open-source development aligns perfectly with our goal of standardizing hardware testing across diverse platforms. Arm has been involved with the KernelCI community for many years already, so it is great to see they are taking a step up joining as Premier Members. Meanwhile, Qualcomm’s proven track record in mobile and emerging computing platforms, combined with their existing contributions as a test result submitter to our common results database, demonstrates their ongoing commitment to kernel quality and stability.

“Arm’s commitment to the Linux kernel community is rooted in the belief that open collaboration drives long-term innovation,” said Mark Hambleton, SVP Software at Arm. “Our role in KernelCI reflects the importance we place on scalable, transparent kernel validation across the broad ecosystem of Arm-based solutions, with the aim of improving software quality and accelerating upstream development from cloud to edge.”

“KernelCI is a cornerstone of upstream Linux kernel development, it enables open-source developers to easily validate and test on a large number of different platforms, including Qualcomm’s, ensuring consistency and quality across the entire ecosystem”, said Leendert van Doorn, Qualcomm SVP of Engineering. “It is a capability that is front and center for Qualcomm’s increasing reliance on upstream enablement.”

The addition of Arm and Qualcomm as Premier members comes at an exciting time for KernelCI. Our new infrastructure dramatically improved the possibilities of KernelCI and the ecosystem we are building around it. So KernelCI is well-positioned to leverage the expertise and resources these new Premier members bring. Their involvement will help us expand testing coverage, improve hardware validation processes, and ultimately deliver better Linux kernel quality to the entire open-source community. We look forward to collaborating with both organizations as we continue to grow KernelCI’s impact on upstream kernel development.

The KernelCI community continues to make great progress on multiple fronts of the project. Since our last update here, we have had two in-person events, a few technical achievements and new working groups launched. See detailed updates in the sections below. 

Videos from Automated Testing Summit

KernelCI hosted the Automated Testing Summit(ATS) 2025 in Denver, CO, USA. It is co-located with the Open Source Summit North America. The event was great and we had many interesting discussions.

All the sessions were recorded and videos are available:

KernelCI Workshop in Amsterdam

In the last week of August, we hosted our first KernelCI workshop in a while. We had 12 participants in-person and a few who joined us online. The workshop was great to kick start some important discussions about the new KernelCI Architecture with the community. 

During the workshop, we discussed how to fulfill maintainers’ use case, dealing with KernelCI labs issues, improving data quality and regression identification, KCIDB transition, RISC-V support and more. Check our notes and full video recording.

Transition out of legacy KCIDB

We finally completed our transition out of the legacy KCIDB. This was a very important step for the new KernelCI architecture. KCIDB is our common database for results, so on one end it has an API to receive test results white on the other is it as PostgreSQL database. The submission API has been supplanted by KCIDB-ng, but with no changes to the results submission JSON schema. Submitters are still sending the same files, but to a different API endpoint. KCIDB-ng is a fast Rust-based API to receive and store the JSON results files.

We’re transforming the KCIDB project into a more versatile, cloud-agnostic solution that can be deployed on-premise when needed. This flexibility allowed us to migrate KCIDB to Azure seamlessly. Additionally, we’ve rewritten portions of KCIDB in Rust, which has resolved longstanding performance bottlenecks.

All the other responsibilities were handed over to our Dashboard, so it is responsible for processing and ingesting the data in the JSON files to our PostgreSQL database. The schema of the database remained the same as legacy 

Core Infrastructure

Our new infrastructure has enabled us to significantly reduce complexity by implementing proper DevOps practices. Throughout Q3, we improved our deployment systems.  In daily operations, this translates to production updates that require minimal supervision and minimal downtime – what previously took hours now takes less than one minute. We’ve also developed our own storage solution that gives us greater control over data costs by maintaining a “hot” cache on VMs while storing longer-term data in object storage with lifecycle policies and more economical storage classes.

There was also an effort to enhance monitoring and alerting systems to proactively prevent issues before they affect users. This includes implementing more granular metrics and establishing alerts for critical performance indicators.

These infrastructure improvements have collectively enabled us to scale more effectively, reduce operational costs, and enhance overall reliability. We remain committed to continuing our infrastructure investments to support our growing demands and ensure we can meet the needs of our expanding user base.

New Labs WG

As a great outcome from the KernelCI Workshop, we created the Labs Working Group(WG) to discuss challenges for connecting test labs to KernelCI.

The Labs WG is already a pretty busy space with over 10 people joining the calls in our bi-weekly sync. The current focus includes improving Maestro API for labs to pull test information from KernelCI, adding support for Labgrid and evaluating dashboards for lab metrics.

The team identified several key challenges that prevent labs from connecting to KernelCI. The primary obstacle is that many labs cannot expose their APIs to the public internet due to strict security policies. To address this limitation, we proposed a pull-mode architecture that reverses the traditional workflow – allowing labs to pull test jobs from KernelCI rather than having KernelCI push jobs directly to them. This approach enables labs to maintain their existing security policies while still actively participating in KernelCI testing. We’re currently developing the protocols and implementations necessary to support this pull-mode architecture.

Dashboard WG

Inspired by the Labs WG, we started the KernelCI Dashboard Working Group (first invite), gathering users and the development team from ProFUSION to talk about bugs and features prioritization.

In two meetings, with 8 attendees, we have defined an action plan to improve the performance of the website, which was affecting user experience, stopped the development of a feature that did not spark joy, and defined the next target for the team: Handling hardware data from labs. Which will require more discussions regarding how that data could be used by the users.

kci-dev improvements

kci-dev has evolved from a small command-line tool into a well-packaged suite of tools that Linux distributions can ship, aimed at analyzing KernelCI results and helping engineers triage problems quickly.

We had many improvements to the project in the past quarter:

• Added Debian/RPM packaging and OBS workflows/services automation so labs can build and publish updates automatically, so every client gets the same, reproducible toolchain.
• Added results compare command for commit-to-commit regression detection (with tables + JSON output as well).
• Consolidated results issues group to list/show issues and fetch related builds/tests.
• Improved tree-level views with tree-report. 
• Added code coverage information through new maestro coverage flow (currently only for chromiumos trees) with per-day buckets, a graph view, and report-info helpers, making stability/coverage trends far clearer.
• Improved validation tooling filters (arch filter; better build selection and considering build/job retry during validation). The validation commands allows us to compare if the tests Maestro is running are landing properly in KCIDB.
• Added –history to results summary.
• Fixed results hardware list and added more filters. 

Hardware Information Registry

One of the challenges discussed in various KernelCI forums has been the inconsistency of hardware platform names reported from different testing labs, making it challenging to find/sort/filter results. To address this, a new YAML-based schema has been proposed by Minas Hambardzumyan and is currently in review. The schema organizes information into lists of platforms, processors, and vendors — providing a path to standardization of the reported platform names and adding traceability to product/vendor web pages for more information.

Final Thoughts

We will keep working on making KernelCI easier for the community to benefit from. From greater stability to an improved Web Dashboard and a more complete kci-dev CLI, there’s much more to enhance in KernelCI for everyone. Big thank you to the entire KernelCI community for making this progress possible!Talk to us at kernelci@lists.linux.dev , #kernelci IRC channel at Libera.chat or through our Discord server!

Contributed to this blog post Arisu Tachibana, Denys Fedoryshchenko, Gustavo Padovan, Minas Hambardzumyan and Tales Aparecida.

Free for all

Today is the beginning of the Early Access phase for the new KernelCI API. As explained briefly in the previous blog post, this is to give everyone a chance to create a user account and start using the API for beta-testing purposes. There’s now an Early Access documentation page with a quick guide to get started, so please go take a look there and start taking part.

A work in progress

Although the fundamental principles of the API & Pipeline have now settled a bit, it is still under active development. In particular, we’re expecting to see a fair amount of changes in these areas:

kci command line tool

It’s still very new and only provides some basic features, so now it needs some proper design. For example, new commands should be added and it might become more human-readable with things like kci find node rather than kci node find.

Build and test coverage

This can only grow as right now there’s only KUnit, one x86 build and one QEMU smoke test run for each kernel revision (and only from mainline). Starting to scale this up will help tackle the main bottlenecks and performance issues in the infrastructure before reaching production quality.

Documentation

Yes, now that things are shaping up we should also be taking good care of the overall documentation and general ease of use of the project. This should also encompass things such as moderation rules to ensure continuity of the project.

A two-way process

This system is made for you, all the members of the wider Linux kernel ecosystem. So while there’s a small but growing team of developers still typing away all the code needed to make it happen, we need your feedback to help shape things up in such a way that it actually delivers on its expectations.

Please experiment as much as you like and share your stories, thoughts and questions via the project’s usual communication channels. Also feel free to create issues on GitHub and send pull requests of course.

Happy beta-testing!

Once upon a time, on a Thursday afternoon somewhere in Italy, a KernelCI backend API was created:

commit 08c9b0879ebe81463e124308192670c0e7447e0b
Author: Milo Casagrande <milo@ubuntu.com>
Date:   Thu Feb 20 16:10:41 2014 +0100

    First commit.

As you can see, it was nearly 10 years ago. How much does that represent in the modern software world? Of course, it depends. The Linux kernel is much older, still written in C and still going strong. But in most cases, including this particular one, it means a whole new world. The old API was written in Python 2.7 which stopped being maintained as a language on 1st January 2020. We could have just rewritten it in Python 3, which was the initial thought. But in the meantime, KernelCI was also growing as a project. It wasn’t just about building ARM kernels and doing boot testing on embedded dev boards any more. It had become a Linux Foundation project aiming to test the whole upstream kernel.

What is this new API?

Following this move, an increasing number of people became interested as it got under the spotlight. That is when we started to realise that the architecture needed to fundamentally evolve in order to match the scale of the new mission it had been assigned. The Linux kernel is a vast and complex open-source project with a unique ecosystem. As such, it requires some unique tooling too. We all know that Git was initially created out of a need to manage all the kernel patches. Now KernelCI needs an automated testing tool tailor-made to its unique requirements – and that’s why we’re finally launching the new API & Pipeline.

It comes with lots of improvements and it’s still a work-in-progress. We’ll keep publishing blog posts and update the documentation as things evolve over the next few months. Right now we have a pipeline that can monitor Git repositories for new revisions, take a snapshot of the kernel source tree in a tarball, run KUnit with it as well as an x86 kernel build and smoke test it in QEMU. It can then also send a summary email and detect regressions. That’s basically enough to prove we have a workable system. Nothing too groundbreaking there, you might think. So, what’s all the fuss about?

In a nutshell: a Pub/Sub interface to orchestrate distributed client-side services that can be run anywhere. You could have your own too at home. Also: user accounts so you can keep your own personal test data there, an abstraction for runtime environments so jobs can be run seamlessly in Docker, Kubernetes, a local shell, LAVA, [insert your own system here]… a new kci command line tool to rule them all and a unified Node schema to contain all the test data (revision, build, runtime test, regression…) in a tree. But again, we’ll go through all that later in more detail. It’s all based on requirements gathered from the community over the past few years.

Timeline

The main message in this blog post is the timeline for retiring the old system and getting the new API in production. Here’s the proposal:Early AccessMonday 4th September 2023Production DeploymentMonday 4th December 2023Legacy System DeprecationMonday 4th March 2024Legacy System SunsetMonday 4th November 2024

It only takes four Monday-the-Forth milestones to get through all this. Here’s what they mean:

Early Access

This is when a new API & Pipeline instance becomes available to let the public experiment with it. It can be seen as some form of beta-testing. It will be deployed in the Cloud to evaluate how a real production instance would behave, but it’s only kept online as a best effort. There should be frequent updates as the code evolves, probably at least weekly and at most daily. Only changes that made it through early testing on the staging instance should be deployed so it’s meant to be reasonably stable.

Production Deployment

The plan is to build upon the experience learned from the Early Access deployment to prepare a persistent instance that would eventually become the production one. Data should be carefully kept and backed up, changes in the database schema should go through managed migrations and the API code should be deployed from tagged releases. As soon as this has become reliable enough we might shut down the Early Access instance since it should have become redundant by then.

Legacy System Deprecation

In other words, this is when the new API & Pipeline production instance becomes the official main KernelCI instance. We’ll first be going through a transition phase to ramp up the build and test coverage on the new API while equally reducing the load on the legacy system to avoid overloading the shared infrastructure. Ideally, coverage should have reached 80% on the new API and 20% on the old one by this date.

Legacy System Sunset

After being deprecated, the legacy system will keep running with a bare minimal amount of coverage just to facilitate the transition for users who depended the most on it. It will be definitely shut down and the data will be archived when finally reaching the Sunset milestone.

Stay tuned

These dates have been identified as realistic targets for having the new API rolled out and retiring the old one with a transition in between. We’ll be aiming to have the new API in place by these dates and conversely retire the legacy system no sooner than announced here.

In the meantime, we’ll be posting updates as these milestones get reached or if any alterations need to be managed. We’ll also clarify how to use the API and exactly what features become available alongside the main documentation. Please share with us any feedback you may have, if you need some clarifications or to raise any concerns. The best way to do this is via the mailing list as always. Stay tuned!

Following the UX Analysis RFP, we’ve received a number of questions which seem worth sharing publicly in order to equally benefit all the proposals we receive.

Big Picture

What are your organization’s most important broader goals with this new dashboard?

We’ve identified a requirement to have a web dashboard in a community survey we did a couple of years ago. It’s mostly about providing Linux kernel developers with the information they need to facilitate their daily workflows, and also other types of users for example if they’re basing their products on the upstream kernel and need to monitor its quality.

What are biggest issues or problems you’re having with your current system that prompted this UX Analysis RFP?

We currently have a very old web dashboard with an associated backend that can’t be maintained any more. On top of that, the project has been growing and we’re now redesigning the whole approach to be able to better scale with a new API which doesn’t have any actual web dashboard right now.

What factors made your team decide to release an RFP for this project?

None of the KernelCI LF project members had enough in-house expertise. Also, looking for an independent external supplier appeared as an appropriate choice in this case.

Is there an incumbent bidder on this project?

No, this is the first RFP we do about UX Analysis and web development in general.

How will vendors be evaluated and scored?

We will come up with some criteria as a basic comparison method, then each member of the advisory board will look at all the proposals and we’ll discuss it and eventually hold a vote. We may also inquire further with some vendors if needed.

How many rounds of revisions and how many UX flows are you expecting as part of this project?

This is very hard to predict as we’re still in the early design stages. There should probably be a small number both of revisions and flows (e.g. 2 or 3), maybe later we would be dealing with incremental changes resulting in more revisions as part of the full implementation efforts.

Do you have a preference regarding the vendor’s location?

No, there is no preference over the country where the vendor is located. The KernelCI project’s team is remotely distributed around the world.

Content

Would you need any copywriting or content migration services?

None that we’re currently aware of.

Would you need any original or stock videography or photography?

Not with the UX Analysis phase. We might need some original content for a final website in production.

How much content do you currently have on your website?

Our static websites have tens of pages. Our current dynamic web dashboard has millions of entries, and this is what we’re expecting to see covered by the UX Analysis. Some static content may be part of the interactive web dashboard but it’s not the primary goal.

Is there a CMS that you have a preference for over the other?

No, however we do have an API for storing our data. How this is turned into a UX and web dashboard is up to the vendor to decide as part of the proposal.

What CMS platform do you use currently?

For some static websites we use WordPress and Hugo, but this UX Analysis work is for an interactive web dashboard. We don’t have any particular CMS requirements for it.

Requirements

Would you require hosting, dns or ssl services?

No, the KernelCI project can take care of this.

How much initial research has been done as part of this RFP?

A lot of research has been done in the past few years to try and understand what the public and users need. What’s missing is how this may translate into an actual UX. We now have some ideas about “what” we need but not “how” users can have it.

Are there any factors driving the timeline for the completion of the work?

We’re developing a new API which will be used hand-in-hand with the web dashboard. The timeline isn’t set in stone but having a prototype dashboard or basic demo around the end of September would be great. We’re thinking of having the new API in production in the first half of 2024 so it would be good to see the web dashboard getting finalised around that time too.

Can you give us a high-level overview of the demographics of each persona from the user stories?

• “Someone who cares about the kernel” can be literally anyone, from a student to a high-profile maintainer or developer. The only real criteria is that they need to know about the upstream kernel code quality. There may be a million people in this category.
• “Kernel / subsystem maintainer” are a relatively small set of people in charge of accepting changes into the kernel. They form some sort of pyramid of trust with several maintainers sending their collected changes to a common maintainer etc. Like any kernel contributors, they are located around the world and have various levels of experience. There’s maybe about 100 subsystem maintainers and 1000 maintainers responsible for smaller areas of the code.

Do you have examples of the email reports that are sent with summaries of test results?

• Here’s a particular one about a bisection result which found a commit that caused some test to fail:
  https://lore.kernel.org/all/Y%2Fi1tX6th2I8hY0o@sirena.org.uk/
• Here’s the full archive with all the email reports we’re currently sending:
  https://groups.io/g/kernelci-results/topics
• Here’s also a KCIDB email report, which collects results from other test systems into a single report:
  https://groups.io/g/kernelci-results-staging/message/48385

Are the test results currently stored in a database that the new web dashboard will visualize?

Yes, the new API has an auto-generated documentation with OpenAPI description. This is still a staging instance for experiments, we’re planning to roll out a production-like instance in the coming weeks and start refining the schema. Basically, all the test data is contained in a tree of Node objects. The underlying engine is MongoDB, and we’re looking into using Atlas for this. The API also features a Pub/Sub interface for events that trigger different stages of the testing pipeline on the client side.

The KCIDB database has a different schema, but the web dashboard wouldn’t necessarily need to read data from both sources. That’s something we still need to define, there are several ways to solve this. It’s also something which might depend on the outcome of the UX Analysis. There’s already an interim web dashboard for KCIDB based on Grafana.

KernelCI hackfests span over a few days during which a number of contributors get together to focus on upstream Linux kernel testing. So far, mainly kernel and automated test system developers have been taking part in the hackfests but anyone is welcome to join. Topics mostly include extending test coverage in various ways: enabling new test suites as well as adding test cases to established frameworks such as kselftest and LTP, building additional kernel flavours, bringing up new types of hardware to be tested…

There have been two hackfests to date. The current plan is to hold one every few months. Future hackfests will be announced on the KernelCI mailing list as well as LKML and Twitter. Stay tuned!

Connecting the dots

There is a large ecosystem around the Linux kernel which includes testing in many shapes and forms: kernel developers, test developers, test system developers, OEMs testing fully integrated products… All these teams of people don’t necessarily interact with each other very much outside of their organisations, and kernel developers aren’t necessarily in the habit of writing tests as part of their daily work.

Events such as the KernelCI hackfests give a chance for people from these different areas to work together on solving common issues and keep the ecosystem healthy. It also helps with shifting the upstream Linux kernel development culture towards a more test-driven workflow, to bring mainline Linux closer to the real world where it is actually being used.

Let’s consider a plausible hackfest story. A first participant writes a new test, say in kselftest. A second participant enables the test to run in KernelCI, which fails in some cases and a kernel bug is found. A third participant makes a fix for the bug, which can then be tested directly in KernelCI to confirm it works as expected. This may even happen on the staging instance before the patches for the test and the fix are sent to any mailing list, in which case the fix would get a Tested-by: "kernelci.org bot" <bot@kernelci.org> trailer from the start. This scenario also relies on some hardware previously made available in test labs by other people, putting together efforts from at least four different participants.

Timeline

Here’s a summary of the first two hackfests:

Hackfest #1 – 27th May to 4th June 2021

• Workboard: https://github.com/orgs/kernelci/projects/3
• Participans
  • Several kernel developers from Google Chrome OS
  • Several kernel developers from Collabora
  • A few members of the core KernelCI team
• Achievements
  • New KernelCI instance for Chrome OS on https://chromeos.kernelci.org
  • Added support for building Chrome OS configs on top of mainline
  • Enabled clang-13 builds for Chrome OS and main KernelCI builds
  • New test suite for libcamera enabled on https://linux.kernelci.org
  • Enabled LTP crypto tests with extra kernel config fragment
  • Several patches were also sent to extend kselftest and KUnit coverage in the kernel tree

Hackfest #2 – 6th to 10th September 2021

• Workboard: https://github.com/orgs/kernelci/projects/5
• Participants
  • More members of the TSC took part than first hackfest
  • New contributor: Denis Efremov (floppy disk kernel maintainer)
  • Alice from Civil Infrastructure Platform (CIP) / Cybertrust Japan
  • Chris from Civil Infrastructure Platform (CIP) / Renesas
  • Nikolai from Red Hat’s CKI
  • Several KernelCI developers from Collabora
• Achievements
  • New Docker images and initial steps for running cvehound
  • On-going progress with Chrome OS tests using QEMU and hardware in LAVA
  • Support for CIP kernel configs
  • Initial work to enable CIP Core images (Debian based)
  • Set up redirect from cip.kernelci.org to the web dashboard page with CIP jobs
  • Support for xilinx-zcu102 device
  • Initial work to enable IEC cybersecurity tests
  • Initial work to enable kselftest livepatch
  • Most of the content from https://foundation.kernelci.org replicated on the new kernelci.org static site
  • Started adding support for including kernel firmware files in test jobs
  • Many various bug fixes and improvements in the KernelCI core code

Lessons Learned

What went well

• There were several new contributors. The hackfest is a great way to get people started with KernelCI.
• Hackfest #2 showed more diversity with a wider representation from the ecosystem.
• There were many improvements in various areas (bug fixes, documentation) which is a sign of a healthy project.
• The workflow based on GitHub and the Big Blue Button platform appear to have been easily adopted by the participants.

What needs to be improved

• Having more kernel maintainers involved would help with setting priorities and ojectives for KernelCI in accordance with kernel developers’ needs.
• A hackfest every 3 months may be a bit too often, or maybe some could be shorter or have a more particular theme.
• Changes can take a long time to get merged. The main limitation seems to be the number of people available to do code reviews and drive discussions.
• KernelCI has been focusing on running more tests for over a year (kselftest, LTP, IGT…). Now the core architecture needs to be improved to scale better.

Next hackfest

The actual dates haven’t been confirmed yet, but with the current 3-month frequency the next hackfest should be taking place early December 2021. A proposed theme could be “KernelCI for newbies”, with a selection of tasks well suited for first-time contributors and documentation improved in that area prior to the hackfest. As always, suggestions are always welcome so please do get in touch if you have any.

See you there!

2020 was the first year of the KernelCI project under the Linux Foundation and has been an interesting one.  Maybe slightly less “interesting” than the rest of the world-changing events of 2020, but it’s still been an adventure.  This article aims to give a quick summary of the major milestones of the first year of KernelCI project, and highlight our goals for the next year.

Highlights of the first year

The founding members spent the end of 2019 doing a formal launch and ramping up the project structure and organization. This led to our mission statement and key goals.  Throughout 2020 we gave talks and led discussions at several virtual conferences such as FOSDEM, Open-Source Summit / Embedded Linux Conference (ELC).  Check out our blog for more details about the talks and discussions from these events. 

Community Collaboration

In the middle of 2020, we did a Community survey to get a sense for what the kernel testing and automation community was looking for.  This survey has helped guide where we focus our time and resources.  See our blog for an article covering the full results of the survey.

One highlight of the 2020 conference circuit was Linux Plumbers Conference (LPC). At LPC, we gave talks and held focused discussions with our target audience: kernel developers and maintainers.  The full details are in a blog article covering the event , but this is where we kicked off public discussions of how to unify test results and reporting from various testing and CI efforts in the community.  We’re calling this common datastore for kernel testing results kcidb.  Thanks to the discussions kicked off at LPC, we’re now collecting results from several other projects such as Red Hat CKI, Google syzbot, Arm, Gentoo, and the Fuego project.  Continued collaboration with these projects as well as other new ones will be a focus area for 2021.

Infrastructure

Another area of growth in 2020 was in our IT infrastructure.  As you might expect, we do lots of kernel builds, and that requires lots of compute horsepower.  Our build capacity had been capped by a fixed number of donated build machines.  But now, thanks to the generous donations of founding members Google and Microsoft, we now have scalable cloud compute resources under Google Compute Platform (GCP) and Microsoft Azure which we manage with Kubernetes so that we can dynamically scale as our compute needs grow.

Looking forward

Kicking off 2021

The governing board kicked off our 2nd year with some project organizational matters such as budgeting and electing this year’s executive committee.  We are very happy to welcome Guillaume Tucker (Collabora) as the new board chair and Chris Paterson (CIP/Renesas) as the new treasurer.  We also say a big thank you to outgoing chair Kevin Hilman (BayLibre) and outgoing treasurer Guy Lunardi (Collabora).  

Focus areas

Data

As mentioned above, the collaboration with other testing and CI projects will remain a major focus for 2021.  We want it to be easy for anyone doing kernel testing to be able to submit results to our open, centralized datastore: KCIDB.  The amount of data we’re collecting is growing rapidly, so we’re also looking for help from “big data” experts to help us build the tooling to visualize and learn from all the data.  Please write to us on the mailing list if you have any interest in helping here.

Infrastructure

We’ve been using Jenkins for years to manage our CI pipeline jobs, but as we’ve moved more of our infrastructure into the cloud, we’re looking at ways to migrate our CI infrastructure to a cloud-native framework such as Tekton or Jenkins-X.  We’re in the early stages of exploration here, so anyone with experience here that could help guide us, we’d love to hear from you!

Data Visualization & Analysis

We’re also in the early stages of planning new dashboards for visualization and analysis of our growing data set.  We’re soliciting feedback from the broader community by collecting user stories to better understand what our users want from new dashboards.  In addition to making all the testing data and logs available through advanced visualization tools, we’d also like to enable analytics and deep learning on our growing data set.  Once again, this is something we’d love your help on, so if you’re a big data enthusiast and want to put your skills to use to help the Linux kernel, please let us know.

Get involved!

Did you notice any themes above?  We’re looking for help!  We have some big ideas and plans, but we’re still a very small team and are looking for expertise in a few areas to help guide the future of the project.

Please keep in touch with what we’re up to or to get involved, you can read our blog, follow on twitter @kernelci or join our mailing list.

The Linux Plumbers Conference 2020 was held as a virtual event this year. The online platform provided a really good experience, with talks and live discussions using Big Blue Button for the video and Rocket Chat for text-based discussions. KernelCI was mentioned many times in several micro-conferences, with two talks in Testing & Fuzzing which are now available on YouTube:

• KernelCI: A Growing Ecosystem
• Unifying Test Reporting with KernelCI

The notes below were gathered publicly from a number of attendees, they give a good insight into what was discussed. In short, while there is still a lot to be done, the KernelCI project is healthy and growing well in its role of a central CI system for the upstream Linux kernel.

Real-Time Linux

We’ve been making great progress with running LAVA jobs using the test-definitions repository from Linaro, thanks to Daniel Wagner’s help in particular. This was prompted by the discussions in the real-time micro-conference.

• Initial GitHub PR: https://github.com/kernelci/kernelci-core/pull/475
• Results on staging: v5.4.61-rt37

The next steps from a KernelCI infrastructure point of view is to be able to detect performance regressions, as these are different to binary pass/fail results. KernelCI can already handle measurements, but not yet compare them to detect regressions. Real-time getting merged upstream means it is becoming increasingly important to be able to support this.

There was also an interesting talk about determining the scheduler latency when using RT_PREEMPT and the introduction of a new tool “rtsl” to trace real-time latency.  This might be an interesting area to investigate and potentially run automated tests with:

• Code: https://github.com/bristot/rtsl
• Talk: https://linuxplumbersconf.org/event/7/contributions/645/

Static Analysis

The topic of static analysis and CI systems came up during the Kernel Dependability MC, and in particular, they were looking for a place to do “common reporting” in order to collect results for the various types of static analysis and checkers available.  We pointed them to the KernelCI common reporting talks/BoFs.

Some static analysis can also be done by KernelCI “native” tests using the kernelci.org Cloud infrastructure via Kubernetes, which is currently only used to build kernels. This is probably where KUnit and devicetree validation will be run, but the rest still needs to be defined.

KCIDB

Fuego

Tim Bird, the main developer of Fuego at SONY, started joined the KCIDB BoF  and we had a good discussion. Unfortunately he had not enough time to go through to an actual submission. We got about a quarter way through converting his mock data to KCIDB.

Gentoo Kernel CI

Alice Ferrazzi, maintainer of GKernelCI at Gentoo, had more time available for the KCIDB BoF and we talked through getting the data out of her system. A mockup of her data was made and successfully submitted to the KCIDB playground database setup.

Intel

Tim Orling, Yocto project architect at Intel, has expressed keen interest in KCIDB.  He said he would experiment at home and will push Intel internally to participate.

LLVM/Clang

The recently added support for “LLVM=1” upstream means we can now have better support for making Clang builds. In particular, this means we’re now using all the LLVM binaries and not just clang. It also solved the issue with merge_config.sh and the default CC=gcc in the top-level Makefile.

This was enabled in kernelci.org shortly after LPC.

kselftest

The first kselftest results were produced on staging.kernelci.org during Plumbers as a collective effort.  We have now started enabling them in production, so stay tuned as they should soon start appearing on kernelci.org.

Initial set of results: https://kernelci.org/test/job/next/branch/master/kernel/next-20200923/plan/kselftest/

AutoFDO

AutoFDO will hopefully get merged upstream, once it is it might be useful for CI systems to share profiling data from benchmarking runs in particular.

Building randconfig

The TuxML project carries out some research around Linux kernel builds: determining the build time, what can be optimised, which configurations are not valid… The project could benefit from the kernelci.org Cloud infrastructure to extend its build capacity while also providing more build coverage to kernelci.org. This could be done by detecting kernel configurations that don’t build or lead to problems that can’t be found with the regular defconfigs.

Using tuxmake

The goal of tuxmake is to provide a way to reproduce Linux kernel builds in a controlled environment. This is used primarily by LKFT, but it should be generic enough to cover any use-case related to building kernels. KernelCI uses its kci_build tool to generate kernel configurations and produce kernel builds with some associated meta-data. It could reuse tuxmake to avoid some duplication of effort and only implement the KernelCI-specific aspects.