THE LINUX FOUNDATION PROJECTS
All Posts By

mwaqar

Regzbot Joins KernelCI: Strengthening Linux Kernel Regression Tracking

By Blog

We’re excited to announce that regzbot, the Linux kernel regression tracking bot created by Thorsten Leemhuis, is joining the KernelCI project. This collaboration marks an important step forward in our shared mission to improve the quality, stability, and long-term maintenance of the Linux kernel.

About Regzbot

Regzbot is a regression tracking utility (dashboard, docs) that has become essential for the Linux kernel community. The bot allows developers and users to add regressions reports to the tracking using simple commands, such as #regzbot introduced: v6.1..v6.2-rc1. Regzbot notices those commands on various mailing lists as well as in bugzilla.kernel.org and a few other bug trackers used for kernel development. From then on regzbot watches out for replies to the tracked report as well as references to it on various mailing lists and in git trees. That way it monitors the issue’s progress on various fronts and can automatically mark tracked regression as resolved once a fix lands in the affected Linux series. By connecting and monitoring the various places relevant, regzbot acts as a kind of meta-bugtracking tool tailored for the special needs of the kernel – but is intentionally kept simple to not become yet another bug tracker.

As the Linux kernel’s regression tracker, Thorsten uses regzbot to keep an eye on regressions, which without the bot would not scale well. Regzbot thus allows Thorsten to ensure many regressions that otherwise might have been forgotten are resolved, as he prods reporters and developers when things stall according to regzbot – sometimes by involving Linus Torvalds, for example when developers do not handle regressions like the lead developer wants them to be dealt this is. Thorsten also uses regzbot to semi-automatically generate weekly reports for Linus Torvalds, which he then can use to decide whether to release a new kernel version or extend the development phase. 

Regzbot and Thorsten work with it has earned the community’s trust by automating what was previously exhausting manual work while maintaining a low-overhead approach. 

Why This Collaboration Matters

This partnership addresses critical needs for both projects. For KernelCI, regzbot provides the semi-automatic regression tracking we’ve needed—the ability to learn when a reported regression gets fixed and update our Dashboard accordingly. Regzbot’s strong reputation within the kernel community also creates a trustworthy path for KernelCI to report regressions to maintainers.

For the kernel community, this collaboration means regzbot will receive dedicated resources to expand its capabilities and improve its user experience, benefiting the entire kernel ecosystem.

What’s Next

Our roadmap focuses on three key areas:

Improved Architecture: We’re exploring redesigning regzbot’s scraper functionality as a standalone service that could benefit other kernel community use cases. We’ll be inviting kernel.org to participate in these discussions.

Extended scope: Regzbot will be redesigned to become more autonomous while also allowing subsystem maintainers to better stay on top of regressions in their domain with the bot’s help.

Modern User Experience: We will be creating a new interface for regzbot within the KernelCI Dashboard. This new UX will provide interactive features like filters, search, and grouping while maintaining the functionality the community relies on.

Better Integration: Once KernelCI regressions are tracked by regzbot, we’ll display relevant information directly in the Dashboard, including mail threads, bugzilla entries, git commit references, and regression status.

A Word from Thorsten Leemhuis

Regzbot was an experiment which turned out to be really successful: it has proven to be a useful tool to help ensure new Linux kernel versions work as well as their predecessors. This might sound benign, but it is not, as it is a crucial factor to make people willing to upgrade to new versions – and that is in everybody’s interest, as only those get all the security fixes and new hardening techniques to counteract current and future risks in this ever changing world. This aligns well with the goals of KernelCI, which is why it is a perfect new home for regzbot. The cooperation will furthermore empower regzbot and make it more capable, because as a team we’ll finally be able to polish and enhance regzbot to make it serve the Linux kernel users and developers a lot better than it does now.

Get Involved

We’re committed to transparent, community-driven development. Any changes will be made carefully, with thorough testing and feedback, to preserve the reliability that has made regzbot valuable.

If you’re interested in following this work:

We look forward to working with Thorsten and the entire kernel community to make regression tracking even more effective.

For more information, visit https://kernelci.org and https://linux-regtracking.leemhuis.info/about/.

KernelCI – 2025.Q4 updates

By Blog, Community

The KernelCI community had a great finish of the year. Since our last update here, we tackled some quite interesting features based on community feedback. See detailed updates in the sections below. 

New KernelCI TSC

KernelCI has announced its new Technical Steering Committee (TSC) composition following the annual election process. The committee, which guides the technical direction of the project and ensures cohesion across the Linux kernel testing ecosystem, now consists of seven members: Ben Copeland (Linaro) serving as TSC Chair, Denys Fedoryshchenko (Collabora) as Infrastructure Committee Lead, and community-elected members Greg KH (Linux Foundation), Gustavo Padovan (Collabora), Mark Brown (Arm), Minas Hambardzumyan (Texas Instruments), and Yogesh Lal (Qualcomm).

This diverse group brings together expertise from major hardware vendors, infrastructure specialists, and kernel maintainers, with the current term running until October 31, 2026. The TSC meets bi-weekly in open sessions to discuss technical decisions, roadmap planning, and ensure KernelCI continues to effectively support the Linux kernel community’s testing needs.

KernelCI meeting public calendar

As you may have noticed, the KernelCI community is growing. New TSC, new working groups, more community members with newcomers arriving frequently. To facilitate access we created a public calendar to announce all our meetings to the community. Through it, it is possible to register yourself to any KernelCI community meeting. Recordings are also available for those who want to watch the meeting at a later time.

Dashboard improvements

This quarter was very focused on the health of the project.

We have introduced monitoring over the API requests and system resources. On top of that, we had a significant increase of backend test coverage, from 40% to nearly 70%, including some benchmark tests. 

There is still plenty to be done, and we intend to continue refining the CI/CD of our project, further automating the delivery of future enhancements.

The main functional achievement we had was reaching the last milestone of KCIDB-ng, placing the ingestion of KCIDB submissions closer to the Django backend.

This allowed the team to detach the KCIDB schema from the dashboard database, which provides more freedom to normalize the data, making the API faster and reducing the storage footprint.

 We are still meeting every two weeks in the Kernel CI Working Group to explore ideas. One experiment we ran this quarter consisted of adding a few simple filters and columns about “Labs” in some of the tables. 

Pull-mode for labs

As part of the work in our Labs Working Group, the team has been working on the Maestro interface to allow labs to pull all the information they need about tests KernelCI can execute. That includes kernel builds, testing rootfs and anything else needed for organizations to run KernelCI tests on their own setup whatever lab technology they are using.

We believe that pull-mode will enable a multitude of labs that were not able to participate in KernelCI before. With pull-mode a lab can be behind a firewall, something quite common in many corporate environments. It is also lab technology agnostic as everything the lab has to do is to listen to Maestro API and then send results to our KCIDB API.

Linaro donating tuxmake, tuxrun, tuxlava

The following repositories are now live under the KernelCl GitHub namespace:

https://github.com/kernelci/tuxmake

https://github.com/kernelci/tuxrun

https://github.com/kernelci/tuxlava 

KernelCI is the Linux kernel’s upstream testing and continuous integration ecosystem. The project has started to use Tuxmake and TuxRun to power its build and test pipelines. Moving these tools under the KernelCI namespace brings them into an active community that will help evolve them alongside the kernel itself.

Linaro engineers will continue as co-maintainers and active contributors. All three projects retain their MIT License, so existing users and contributors will see no disruption.

Blog post announcement: https://www.linaro.org/blog/linaro-transfers-kernel-building-and-testing-tools-to-the-kernelci-project/ 

Core Infrastructure

  • More reliable event delivery: Added an optional mode that lets clients reconnect and catch up on missed events instead of losing them.
  • Better event discovery: Expanded filtering options so users can find specific events more precisely.
  • Faster event queries: Added database optimizations that speed up common event lookups
  • Maintenance reliability: Fixed a cleanup issue so old data can be removed as intended
  • Platform/tooling updates: Upgraded the database version and refreshed development tooling to improve stability
  • Build/CI automation was expanded with a new production workflow and tweaks to container image build automation.
  • Tooling/container configs were refreshed to newer toolchains and base images, with many older variants removed and patches updated.
  • A large legacy build configuration catalog was removed, alongside updates to runtime and rootfs configuration data.
  • A new pull-based lab runtime was added, including callback parsing, log handling, and result mapping; runtime configuration was extended to support it.
  • LAVA handling was hardened for retry scenarios, missing logs, and infrastructure failure reporting.
  • Storage uploads gained retry logic for network and server-side failures.
  • Forecast reporting was upgraded to generate an HTML report in addition to console output.

kci-dev improvements

kci-dev continued evolving as a well-packaged CLI suite that Linux distributions and labs can ship, focused on helping engineers analyze KernelCI results and triage problems quickly from the terminal.

We had several improvements to the project in the past quarter:

  • Released kci-dev v0.1.10, adding Arch Linux packaging support and more workflow/polish fixes, including Debian package build workflow updates and multiple fixes in issues/validation reporting and build-node filtering. 
  • Released kci-dev v0.1.9, with additional triage and UX refinements on top of the previously reported Q3 feature set: moved the detect workflow under results issues, added a command to fetch new issues for a checkout, and restructured the issues command group for clearer usage.
  • Improved validation and reporting ergonomics: enabled list views for boots validation, added runtime fields to boot/test JSON output, and fixed boot-origin filtering and other result-selection edge cases.
  • Conference and community outreach (Tokyo, Dec 2025): presented “Getting Started With New KernelCI CLI Tools” at Open Source Summit Japan 2025, and at Linux Plumbers Conference 2025 ran the Kernel Testing & Dependability MC plus gave “KernelCI kci-dev: Closing the developer feedback loop”.

Final Thoughts

We will keep working on making KernelCI easier for the community to benefit from. From greater stability to an improved Web Dashboard and a more complete kci-dev CLI, there’s much more to enhance in KernelCI for everyone. Big thank you to the entire KernelCI community for making this progress possible!Talk to us at kernelci@lists.linux.dev , #kernelci IRC channel at Libera.chat or through our Discord server!

 

Contributed to this blog post Arisu Tachibana, Denys Fedoryshchenko, Gustavo Padovan, Minas Hambardzumyan and Tales Aparecida.

Announcing the new KernelCI Technical Steering Committee (TSC)

By Blog, Community

We are pleased to announce the composition of the new KernelCI Technical Steering Committee (TSC). The TSC plays a vital role in guiding the technical direction of the KernelCI project, ensuring its continued growth and effectiveness in supporting the Linux kernel community.

New TSC Composition

Following our annual election process and in accordance with our project charter, the TSC is now composed of the following members:

  • Ben Copeland, Linaro (TSC-voted member) – TSC Chair
  • Denys Fedoryshchenko, Collabora (Infrastructure Committee Lead – appointed position)
  • Greg KH, Linux Foundation (Community-elected)
  • Gustavo Padovan, Collabora (Community-elected)
  • Mark Brown, Arm (Community-elected)
  • Minas Hambardzumyan, Texas Instruments (TSC-voted member)
  • Yogesh Lal, Qualcomm (Community-elected)

The current term for community-elected and TSC-voted members runs until October 31, 2026. As specified in our charter, the Infrastructure Committee Lead position is appointed and follows different term rules.  Ben Copeland will serve as the TSC Chair.

About the TSC

The Technical Steering Committee is responsible for:

  • Making important technical decisions about the project’s direction
  • Discussing the general roadmap and design principles
  • Ensuring cohesion across the project
  • Participating in votes on critical matters
  • Contributing to the project through code, reviews, documentation, and community engagement

We maintain a policy that no more than two members from the same organization may serve on the TSC simultaneously, ensuring diverse representation across the community.

Looking Ahead

This diverse group brings together expertise from across the kernel testing ecosystem, representing major hardware vendors, infrastructure specialists, and kernel maintainers. Their combined experience will be invaluable as KernelCI continues to evolve its infrastructure and expand testing coverage.

The TSC meets bi-weekly to discuss current topics and make decisions. These meetings are an important part of how we coordinate the technical aspects of the project and ensure we’re meeting the needs of the Linux kernel community. The meetings are open and listed on our calendar.

We thank all TSC members for their service and commitment to improving kernel quality and stability through comprehensive testing.

Get Involved

If you’re interested in contributing to KernelCI or learning more about our work, visit:

To reach the TSC directly, you can contact kernelci-tsc@groups.io.

KernelCI Welcomes Arm and Qualcomm as Premier Members

By Blog, Community

KernelCI Welcomes Arm and Qualcomm as Premier Members

We are thrilled to announce that two industry leaders, Arm and Qualcomm, have joined KernelCI as Premier members. This marks a significant milestone in our mission to ensure the quality, stability, and long-term maintenance of the Linux kernel through comprehensive testing across the broadest possible range of hardware platforms.

Both companies bring extensive expertise and resources that will significantly strengthen KernelCI’s testing ecosystem. Arm’s deep understanding of processor architecture and their commitment to open-source development aligns perfectly with our goal of standardizing hardware testing across diverse platforms. Arm has been involved with the KernelCI community for many years already, so it is great to see they are taking a step up joining as Premier Members. Meanwhile, Qualcomm’s proven track record in mobile and emerging computing platforms, combined with their existing contributions as a test result submitter to our common results database, demonstrates their ongoing commitment to kernel quality and stability.

“Arm’s commitment to the Linux kernel community is rooted in the belief that open collaboration drives long-term innovation,” said Mark Hambleton, SVP Software at Arm. “Our role in KernelCI reflects the importance we place on scalable, transparent kernel validation across the broad ecosystem of Arm-based solutions, with the aim of improving software quality and accelerating upstream development from cloud to edge.”

“KernelCI is a cornerstone of upstream Linux kernel development, it enables open-source developers to easily validate and test on a large number of different platforms, including Qualcomm’s, ensuring consistency and quality across the entire ecosystem”, said Leendert van Doorn, Qualcomm SVP of Engineering. “It is a capability that is front and center for Qualcomm’s increasing reliance on upstream enablement.”

The addition of Arm and Qualcomm as Premier members comes at an exciting time for KernelCI. Our new infrastructure dramatically improved the possibilities of KernelCI and the ecosystem we are building around it. So KernelCI is well-positioned to leverage the expertise and resources these new Premier members bring. Their involvement will help us expand testing coverage, improve hardware validation processes, and ultimately deliver better Linux kernel quality to the entire open-source community. We look forward to collaborating with both organizations as we continue to grow KernelCI’s impact on upstream kernel development.