Skip to main content
search
All Posts By

kernelcifoundation

Feb 20
Love0

KernelCI – February 2025 updates

By Blog, News

For those of you who were at Linux Plumbers Conference last September, you got a great amount of updates from the KernelCI project. However, September is long in the past now. So it is time for a new round of updates.

In summary, in the past few months, we successfully stabilized our new infrastructure, shut down our legacy system. Then, we also developed logspec for parsing test logs, made more progress on the new Web Dashboard, launched kci-dev for kernel developers, and began testing regression notifications via email. Let’s now dive into specific topics for a short update!

Web Dashboard development progress

While we believe the Web Dashboard still has a lot to evolve, it is already able to provide results information for many use cases. The development team, funded by the KernelCI project, is continuously improving the dashboard based on user feedback. We invite everyone to use our new dashboard and share any feedback they may have.

logspec – new log parsing skills

logspec  is a new log parser designed to recognize contextual nuances. It is able to match patterns around build failures, boot issues, NULL pointer dereference, Kernel Panic, Kernel Bug, UBSAN warnings etc. We have integrated it in Maestro so far. There, it sits on the Maestro->KCIDB bridge and creates new issues in KCIDB when it finds any of the  tracked patterns in the code. So, here are examples of a build issue and boot issue. logspec is a new project, but already helping the community find key information quite fast. We hope to evolve it over the next few years.

Announcing kci-dev cmdline

A few months ago, we started kci-dev, our command line tooling for kernel developers and maintainers. kci-dev is still beta but its core features are taking shape:

  • sending specific test requests to Maestro: today we support testing any commit on any tree/branch available in KernelCI’s Maestro.
  • running bisections: we have an experimental tooling that allows us to run bisections for some regressions we found.
  • fetching test results from the dashboard: Today, we have basic support to fetch test results and download logs through the `kci-dev results`  command. 

Example run to fetch summary of results:

We invite developers to try it out and give us feedback on the functionalities they want to see in kci-dev.

Experimenting with results notifications

Over the past month, we’ve been developing  kernelci-notifications that are already capable of watching for new KCIDB issues and generating notifications for build and boot regressions filed in KCIDB. Currently, we are experimenting with issues created by Maestro, but the notification system is being designed for use with any CI system submitting data to KCIDB.  You can see examples of issues reported here. And excerpt follows as well:

KCIDB

KCIDB receives increasing amounts of kernel test data every day. As the project grows, so do its challenges. We are currently looking at improving the performance of KCIDB with long-term needs in mind. A few weeks ago, the KernelCI project published a RFQ for a specialist DBA support for KCIDB.

More hardware support

In the recent weeks, we have seen Penguntronix enable their lab in KernelCI’s Maestro, MediaTek Genio to add their newest hardware through the Collabora lab. As we speak, Qualcomm is working to enable more of their labs in KernelCI too.

New test Infra continues stabilization

We continue improving our core infra. Recently, we added automatic deployment of Maestro through GitHub actions and we also created kernelci-storage, a solution to help the KernelCI ecosystem to store test artifacts.

Final thoughts

We will keep working on making KernelCI easier for the community to benefit from. From greater stability to an improved Web Dashboard and a more complete kci-dev CLI, there’s much more to enhance in KernelCI for everyone. Big thank you to the entire KernelCI community for making this progress possible!
Talk to us at kernelci@lists.linux.dev, #kernelci IRC channel at Libera.chat or through our Discord server!

Sep 13
Love0

Sustaining long-term support for Linux mainline based products

By News

We all know that introducing and supporting hardware in Linux mainline is challenging. At KernelCI, we want to understand how we can help move the ecosystem further upstream. 

The challenges come from many angles:

  • Invest in upstreaming and maintain the hardware support into Linux (and other projects) in the long-run
  • Take IP protection in consideration, making sure that no critical information is shared in the open
  • Time to market pressures for the product release
  • Test mainline based products (continuously), making sure:
    • The hardware remains functional
    • No regressions appear
  • Maintain test infrastructure, so testing can be timely and efficient
  • Interact with the kernel community to improve the state of the art and fix issues
  • And so much more

These challenges come from both technical and business aspects. On the one hand, there are still a lot of difficulties in interacting with the upstream community to implement drivers, review patches, and solve regressions. On the other hand, business stakeholders and decision-makers have a hard time understanding upstream practices, which contributes to insufficient investment to bring enough knowledge and resources for effective participation in upstream. KernelCI aims to help Hardware Vendors address both the technical and business obstacles.

With that in mind, KernelCI wants to start a discussion with all interested Hardware Vendors to share our experiences and pain points. We could then look ahead at how we can collaborate on improving the kernel integration processes and facilitate the maintenance of stable kernels in the long run. Stability and security needs are growing exponentially, with so much of our global infrastructure depending on the hardware and software we build.

If you are in Vienna, Austria on September 18th for Open Source Summit Europe and/or Linux Plumbers Conference, we invite you to join our in-person discussion to happen from 1:30pm to 3pm at LPC Room 1.34.
If you have any questions or comments, please contact us at kernelci-members@groups.io.

Sep 12
Love0

Bring your tree to the new KernelCI

By Blog

KernelCI is entering an exciting new phase, and we’re inviting kernel maintainers to register their trees with us.

We now support over 220 build configurations, run boot tests on 45 different hardware platforms (totaling over 300 machines in all labs), 7 CI systems contributing tests results and have test suites like kselftest, kunit and LTP already enabled, ensuring comprehensive coverage for your kernel trees across diverse platforms.

At the upcoming Linux Plumbers Conference, we’ll be unveiling our new Web Dashboard, which allows you to easily track your test results. Also, our team is ready to support you with custom Grafana dashboards tailored to your needs and optional email notifications. We can also enable your preferred test suites.

We’re committed to delivering real value and working closely with maintainers, developers, hardware vendors and product makers to provide specific solutions. Each subsystem and maintainer has unique needs, and we’re adapting our approach accordingly. We can build custom views and configure personalized notifications to help you focus on the results that matter most, minimizing unnecessary noise.

We encourage all maintainers to register their trees and take advantage of these new features. To get started, submit your request via our GitHub project or contact us directly at kernelci@lists.linux.dev .

During the Linux Plumbers Conference, we’ll have a hallway booth—please stop by to discuss your needs and share your feedback. KernelCI is committed to deliver solutions that matter to you.

Learn more about KernelCI in our documentation.

Sep 11
Love0

KernelCI at Linux Plumbers Conference 2024

By Blog, News

The KernelCI community will have a busy few days at the upcoming Linux Plumbers Conference(LPC) in Vienna, Austria. We will be presenting 2 talks in the Refereed track, 4 Micro-conferences sessions and 1 BoF.

For those interested in an overview of everything happening in KernelCI, come to the “Meeting the New KernelCI” presentation from Don Zickus (Red Hat) and Gustavo Padovan(Collabora) in LPC’s Refereed Track Thursday at 12:00pm CEST.

Following the talk, on Thursday afternoon, starting at 3pm, the KernelCI team will set up a small corner in the “hallway track” to dive into more discussions with the community. Come talk to us and dive further in usecases, demo the dashboard, collect feedback, etc.

Hardware Vendors are key for KernelCI success, so we will be hosting a discussion with Hardware Vendors to understand how we can improve the kernel testing and integration overall and how KernelCI, as an organization can help drive progress. This will happen Wednesday at 1:30pm to 3:00pm (Room 1.34). If you are interested in joining please send an email to kernelci-members@groups.io. Open Source Summit attendees are also welcome in this session.

Apart from that we will also bring several technical challenges for discussions in the micro-conferences, host a talk on GitLab and a Kernel Testing BoF. Check our full agenda:

Wednesday

Thursday

  • 12:00 Meet the New KernelCI in the LPC Refereed Track (Hall L2/L3). By Don Zickus (Red Hat), Gustavo Padovan(Collabora)
  • 15:00 – 18:00 KernelCI Booth in the Hallway Track. By the KernelCI community.
  • 19:00 – 21:00 KernelCI Happy Hour – Location: The View Café-Bar-Restaurant. 25 min walk from the conference venue.

Friday

Times are in the Vienna/CEST Timezone.

As you can see it is a pretty busy agenda. If you want to talk to us please reach out at  kernelci-members@groups.io. We will be happy to see you at LPC!

Jun 17
Love0

Welcoming Texas Instruments to KernelCI

By News

We are pleased to announce that Texas Instruments is joining KernelCI as a Premier member. Texas Instruments(TI) is a lead semiconductor company and a long time contributor to the upstream Linux kernel. To better serve its customers, TI has embraced Open Source, thriving to offer upstream support for the chips they produce.

TI is pushing the boundaries of upstream SoC support and advocating for an industry-wide Upstream-First approach. For the past several years, TI has invested in its testing infrastructure and technologies and applied those improvements to TI’s upstream testing. Now, TI is proud to join KernelCI to bring that knowledge, experience, and capability to the larger community, including other semiconductor companies, in order to collaborate openly and improve upstream SoC quality and support.

As Nishanth Menon, a principal open-source technologist at Texas Instruments, says: “KernelCI is crucial for ensuring the longevity of the Linux ecosystem and vital for providing high-quality software in the upstream Linux community, fostering the confidence required for products to be agnostic of vendor software ecosystems.”

The KernelCI board is glad to have TI joining forces to make the new KernelCI testing capabilities stronger. “We are excited to welcome Texas Instruments to the board and looking forward to leveraging their testing capabilities and expertise around SoCs to contribute to our common goal of ensuring the quality, stability and long-term maintenance of the Linux kernel.” said Don Zickus, Chair of the KernelCI Board.

TI is leading the way in creating an open and upstream embedded testing standard for the Linux kernel. All embedded SoC and product vendors incur significant costs to reinvent and maintain the test ecosystem for standard peripherals. KernelCI provides the groundwork to establish a vendor-agnostic testing framework for user experiences at the product level. With this goal in mind, KernelCI is also the perfect umbrella to align the industry and create a better solution by standardizing the test framework, terminology, interfaces, methodology, tools, and reporting.

There couldn’t be a better time for TI to join KernelCI. The project is going through a major overhaul with our new CI architecture coming into place. It replaces our limited legacy system with a quite capable architecture that can leverage different CI systems, hardware and cloud labs, to funnel all the test data to a common place. So, we can analyze, automatic triage and support the regression reporting process together with the Linux kernel community. Our top priority is to serve kernel maintainers and developers to ensure the long-term stability of the Linux kernel.

Apr 08
Love0

Strategic updates from the KernelCI project

By Blog, Community, News

The KernelCI project continues on its mission to ensure the quality, stability and long-term maintenance of the Linux kernel. That means supporting the community (especially maintainers) to not just run their code in a Continuous Integration (CI) system, but also deliver relevant, high confidence results and reports. In this post, we will give you a strategic overview of the steps we have taken towards our mission over the past few months.

Enabling new infrastructure to run tests

Our legacy system has shown its age and it has been failing to meet the growing testing needs of the community. It is a quite limited, unstable python2 project that focuses on embedded hardware. We continue on the journey to put in place our new infrastructure, so we can finally move away from the legacy KernelCI system.

We bring good news! The new core service for running tests is already in place, but still going through a stabilization phase. So, the team is ramping up the number of tests slowly to deal with issues that arise especially when it comes to the quality of the testing KernelCI provides. We do not want to repeat past mistakes with results that can’t really be trusted, so our focus right now is on quality rather than quantity. The team is iterating quickly on that process to enable open, wide adoption in the coming months.

In our new KernelCI infrastructure, we already have support to run tests in labs (through LAVA only at the moment), Docker containers, Kubernetes and natively. Adding new labs or test environments should be relatively straightforward. Then, as we add more test environments, we are also laying down the foundation for integration with other CI systems across the community so we can share kernel builds and offer test environments.

As we shared before, the new infrastructure exposes an API for users to create accounts, query results and even drive tests themselves. At the moment, we are focusing on enabling our own pipeline there, so we can shut down the legacy system. But anyone is welcome to request a user account and try it out.

Another initiative from the KernelCI community is the GitLab CI support in the mainline kernel. Here, the goal is to offer maintainers a CI environment that they can manage themselves. With time, KernelCI API can be leveraged to provide a backend for builds and test runs for repositories using GitLab CI.

Trusting tests results and reports

On one end, we are stabilizing our new infrastructure to run the tests. On the other end, we are looking into improving the quality of the reports KernelCI sends out, so maintainers and developers can actually trust them. Given the huge amount of data coming out of test systems and bots today, we must invest in improving the delivery of the results, or else, KernelCI will be contributing to increasing the maintainer burnout rather than helping solve it. That means improving the quality and confidence of the data, so maintainers and developers only receive reports packed with relevant information and no noise or false-positives.

At the time of this writing, we have a handful of trees enabled, boot testing and a few tests enabled (including kselftest support) in our new test infrastructure. That setup is enabling the team to triage ALL the results to identify infrastructure failures and test patterns in general (flakiness, config issue, intermittent issues, etc). There is a significant investment to develop better tests together with the community (like the device tree probe kselftest) that is improving the quality of the results compared to what exists in our legacy system.

As part of the effort, we are developing a layer for post-processing the test results in KCIDB – the KernelCI database to collect test results from the entire Linux kernel test ecosystem. The work in this area is at proof-of-concept level, but it is already enabling the team to evaluate the results coming from our new infrastructure. The post-processing layer should be a key part of the feedback loop with the community. The goal is to increase automation in triaging the results, saving precious time from kernel maintainers. Also, because KCIDB collects data from various CI systems, the post-processing of test results can be enabled for more systems than just KernelCI.

On top of that, the KernelCI team is redesigning the Web Dashboard UX to enable rich visualization of all that data for the entire community. A public request for feedback on UX should go out in the coming weeks.

It’s all about engaging the community in testing

Solving CI needs for the Linux kernel community is not just a technical challenge. It is in big part a community engagement challenge too. The KernelCI project has a strong focus on engaging the community in testing processes. With our new infrastructure coming into place, we are ready to give a new spin to our Community Engagement initiative.

For that, we are forming a Community Engagement Working Group (WG). The WG will focus on connecting with maintainers to discuss and implement improvements in test quality for their subsystems and also act as a feedback recipient for improvements in our post-processing of the test results. The Community Engagement WG will be led by Shuah Khan, kernel maintainer & Linux Fellow at The Linux Foundation.

A dedicated announcement of the Community Engagement WG was sent to the KernelCI mailing list. If you are interested in participating, raise your hand!

Where are we going from here?

As you can see, a lot is going on in KernelCI at the moment. The team is iterating quickly on the development of the new infrastructure, so we will be engaging with new maintainers and developers every month from now on, bringing them to the new infra and pushing the system limit further. If you are a maintainer and want to bring your tests to KernelCI please send us an email at kernelci@lists.linux.dev.

That’s all for now! Stay tuned for updates on topics discussed in this article. Likewise, as the new infrastructure stabilizes, expect a significant amount of documentation updates too.

We thank all KernelCI member organizations and developer community who have been investing in the project over the years. It is only because of the continued support from our community that we are making the legacy system a past story!

Apr 02
Love0

Exciting KernelCI TSC updates

By Blog, Community, News

During our last TSC meeting, on March 14th, we had two votes that brought new winds to our community:

Shuah Khan joins the TSC

Shuah, Kernel Maintainer & Linux Fellow at The Linux Foundation, has been elected a new member of the TSC. We believe Shuah’s experience will help the KernelCI project expand its kernel community engagement efforts, taking testing and CI to more parts of the upstream kernel. Welcome, Shuah!

Nikolai Kondrashov elected as the new TSC Chair

Nikolai has been a longstanding contributor under the KernelCI umbrella and the person behind KCIDB. At Red Hat he also contributes to kernel integration technology. His knowledge around the problems KernelCI needs to solve will help the project greatly. His role will be effective on April 1st. All the best to Nikolai in his new role!

While at it, we thank Guillaume for his tenure as TSC Chair of the project!

The detail of the motions which were voted can be found on the TSC documentation.

Dec 04
Love0

UX Research for the new Web Dashboard

By Blog, Community, News

Following the UX Analysis RFP we shared a few months ago, we’re now delighted to announce that the KernelCI Advisory Board has accepted a proposal. First of all, we’re grateful to have received such a series of high-quality proposals. Many thanks once again to all the submitters for your interest in the project.

After carefully considering all the options with the board and some TSC members, the vote came out in favour of ProFUSION. Congratulations! We’re looking forward to the next milestones, in fact the work has already started.

Milestones

The UX Analysis project is broken down into three milestones with four weeks assigned to each, so a total of twelve weeks. Some breaks will also be scheduled during this time so the calendar end date is likely to be in March 2024.

M1: Result of interviews

Some interviews will be carried out with various stakeholders to help refine user stories and start modelling the user experience and some design aspects.

M2: First UX design proposal

A first interactive prototype will then be created and made available for members of the public to evaluate and provide feedback.

M3: Final UX design proposal

Based on the feedback and outcome of the first two milestones, the last part is about creating high-level specifications of the web dashboard design suitable for the implementation phase.

Sep 04
Love0

KernelCI API Early Access

By Blog, Community

Free for all

Today is the beginning of the Early Access phase for the new KernelCI API. As explained briefly in the previous blog post, this is to give everyone a chance to create a user account and start using the API for beta-testing purposes. There’s now an Early Access documentation page with a quick guide to get started, so please go take a look there and start taking part.

A work in progress

Although the fundamental principles of the API & Pipeline have now settled a bit, it is still under active development. In particular, we’re expecting to see a fair amount of changes in these areas:

kci command line tool

It’s still very new and only provides some basic features, so now it needs some proper design. For example, new commands should be added and it might become more human-readable with things like kci find node rather than kci node find.

Build and test coverage

This can only grow as right now there’s only KUnit, one x86 build and one QEMU smoke test run for each kernel revision (and only from mainline). Starting to scale this up will help tackle the main bottlenecks and performance issues in the infrastructure before reaching production quality.

Documentation

Yes, now that things are shaping up we should also be taking good care of the overall documentation and general ease of use of the project. This should also encompass things such as moderation rules to ensure continuity of the project.

A two-way process

This system is made for you, all the members of the wider Linux kernel ecosystem. So while there’s a small but growing team of developers still typing away all the code needed to make it happen, we need your feedback to help shape things up in such a way that it actually delivers on its expectations.

Please experiment as much as you like and share your stories, thoughts and questions via the project’s usual communication channels. Also feel free to create issues on GitHub and send pull requests of course.

Happy beta-testing!

Jul 28
Love0

API Transition Timeline

By Blog, Community

Once upon a time, on a Thursday afternoon somewhere in Italy, a KernelCI backend API was created:

commit 08c9b0879ebe81463e124308192670c0e7447e0b
Author: Milo Casagrande <milo@ubuntu.com>
Date:   Thu Feb 20 16:10:41 2014 +0100

    First commit.

As you can see, it was nearly 10 years ago. How much does that represent in the modern software world? Of course, it depends. The Linux kernel is much older, still written in C and still going strong. But in most cases, including this particular one, it means a whole new world. The old API was written in Python 2.7 which stopped being maintained as a language on 1st January 2020. We could have just rewritten it in Python 3, which was the initial thought. But in the meantime, KernelCI was also growing as a project. It wasn’t just about building ARM kernels and doing boot testing on embedded dev boards any more. It had become a Linux Foundation project aiming to test the whole upstream kernel.

What is this new API?

Following this move, an increasing number of people became interested as it got under the spotlight. That is when we started to realise that the architecture needed to fundamentally evolve in order to match the scale of the new mission it had been assigned. The Linux kernel is a vast and complex open-source project with a unique ecosystem. As such, it requires some unique tooling too. We all know that Git was initially created out of a need to manage all the kernel patches. Now KernelCI needs an automated testing tool tailor-made to its unique requirements – and that’s why we’re finally launching the new API & Pipeline.

It comes with lots of improvements and it’s still a work-in-progress. We’ll keep publishing blog posts and update the documentation as things evolve over the next few months. Right now we have a pipeline that can monitor Git repositories for new revisions, take a snapshot of the kernel source tree in a tarball, run KUnit with it as well as an x86 kernel build and smoke test it in QEMU. It can then also send a summary email and detect regressions. That’s basically enough to prove we have a workable system. Nothing too groundbreaking there, you might think. So, what’s all the fuss about?

In a nutshell: a Pub/Sub interface to orchestrate distributed client-side services that can be run anywhere. You could have your own too at home. Also: user accounts so you can keep your own personal test data there, an abstraction for runtime environments so jobs can be run seamlessly in Docker, Kubernetes, a local shell, LAVA, [insert your own system here]… a new kci command line tool to rule them all and a unified Node schema to contain all the test data (revision, build, runtime test, regression…) in a tree. But again, we’ll go through all that later in more detail. It’s all based on requirements gathered from the community over the past few years.

Timeline

The main message in this blog post is the timeline for retiring the old system and getting the new API in production. Here’s the proposal:Early AccessMonday 4th September 2023Production DeploymentMonday 4th December 2023Legacy System DeprecationMonday 4th March 2024Legacy System SunsetMonday 4th November 2024

It only takes four Monday-the-Forth milestones to get through all this. Here’s what they mean:

Early Access

This is when a new API & Pipeline instance becomes available to let the public experiment with it. It can be seen as some form of beta-testing. It will be deployed in the Cloud to evaluate how a real production instance would behave, but it’s only kept online as a best effort. There should be frequent updates as the code evolves, probably at least weekly and at most daily. Only changes that made it through early testing on the staging instance should be deployed so it’s meant to be reasonably stable.

Production Deployment

The plan is to build upon the experience learned from the Early Access deployment to prepare a persistent instance that would eventually become the production one. Data should be carefully kept and backed up, changes in the database schema should go through managed migrations and the API code should be deployed from tagged releases. As soon as this has become reliable enough we might shut down the Early Access instance since it should have become redundant by then.

Legacy System Deprecation

In other words, this is when the new API & Pipeline production instance becomes the official main KernelCI instance. We’ll first be going through a transition phase to ramp up the build and test coverage on the new API while equally reducing the load on the legacy system to avoid overloading the shared infrastructure. Ideally, coverage should have reached 80% on the new API and 20% on the old one by this date.

Legacy System Sunset

After being deprecated, the legacy system will keep running with a bare minimal amount of coverage just to facilitate the transition for users who depended the most on it. It will be definitely shut down and the data will be archived when finally reaching the Sunset milestone.

Stay tuned

These dates have been identified as realistic targets for having the new API rolled out and retiring the old one with a transition in between. We’ll be aiming to have the new API in place by these dates and conversely retire the legacy system no sooner than announced here.

In the meantime, we’ll be posting updates as these milestones get reached or if any alterations need to be managed. We’ll also clarify how to use the API and exactly what features become available alongside the main documentation. Please share with us any feedback you may have, if you need some clarifications or to raise any concerns. The best way to do this is via the mailing list as always. Stay tuned!

Close Menu