ci: Add dedicated CodeQL workflow for improved analysis

[thompson-tomo]

This adds a dedicated codeql workflow which can analyse any package. In the process it resolves the issue with the current ci failures where in which codeql is tied to a non c++ project.

Only httpd is analyzed as it is the only project which can be natively built without checking out otel-cpp in build. Additional projects are added by simply adding a code-ql config file.

The structure of using artifacts is necessary to optimise performance, without it & purely relying on fetch resulted in a 2hr build for httpd as opposed to 30mins.

[github-advanced-security]

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

• The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
• Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
• You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

[dbarker]

Thanks for contributing to CI. It is needed! Just a request to use native CMake install for otel-cpp and apt packages for the dependencies so conan support is not implied and it is easier maintain.

[dbarker]

Please see below on steps to harden the workflow..

[dbarker]

Thanks for taking this on. Following the discussion above please remove the conanfiles and install the dependencies for the CodeQL test with either apt packages or using CMake's ExternalProject module.

[lalitb]

Thanks for adding the shared CodeQL workflow. Two concerns:

• I agree with @dbarker that CodeQL should not require adding Conan support to each component. Native CMake or system packages are a better fit here.
• The workflow triggers for all exporters/ and instrumentation/ changes, but only analyzes components with a Conan file. This can give a green CodeQL result for code that was not built. It also leaves the webserver module uncovered after ci: Resolving nginx & prometheus build issue + web server consistency #633 removes its existing CodeQL job.

Can we make the supported components explicit and ensure every triggered component is actually analyzed?

[thompson-tomo]

CodeQL should not require adding Conan support to each component. Native CMake or system packages are a better fit here.

Conan was used to only import dependencies hence we didn't add support for conan and they were still built using cmake.

The workflow triggers for all exporters/ and instrumentation/ changes, but only analyzes components with a Conan file. It also leaves the webserver module uncovered after ci #633 removes its existing CodeQL

The biggest difficulty we face is codeql requires everything to be built in the 1 workflow hence why we need a reproducible build process. The problem got even worse when you consider different components needed different versions of the same dependency as otherwise they failed. Hence the conan was added to the components which could be built with the goal being add more once they are buildable. Note prior to this pr there was no codeql at all as the current codeql doesn't work, hence partial coverage is better than none.

[lalitb]

The biggest difficulty we face is codeql requires everything to be built in the 1 workflow hence why we need a reproducible build process. The problem got even worse when you consider different components needed different versions of the same dependency as otherwise they failed. Hence the conan was added to the components which could be built with the goal being add more once they are buildable. Note prior to this pr there was no codeql at all as the current codeql doesn't work, hence partial coverage is better than none.

I agree that partial coverage is better than none. My concern is only that the workflow should clearly represent that partial coverage.

Right now, it triggers for every change under exporters/** and instrumentation/**, but it builds only the components that have a conanfile.txt. This means CodeQL can report green even when the changed component was not analyzed.

CodeQL also does not require every component to use the same build environment. Components with different dependency requirements can use separate jobs or a matrix.

Could we limit the path triggers to the components currently supported, or define those components explicitly in a matrix? We can expand that list as more components become buildable.

[thompson-tomo]

CodeQL also does not require every component to use the same build environment. Components with different dependency requirements can use separate jobs or a matrix.

I don't believe that is the case given that the build must be done between the init & analyze steps and the analyze replaces any previous analyze hence needs to be a single check.

I need to think of a maintainable & reproducible solution on how to perform codeql analysis especially if we can't use conan.

[thompson-tomo]

@dbarker i have significantly rewritten the workflow to now be native cmake. Even after optimising it, the workflow is taking 30mins due to needing to build otel cpp it is however better than the 2 hrs it was at 1 stage.

Please let me know if you are happy with it in this form.

[lalitb]

@thompson-tomo - Could we use the Ubuntu's gRPC and Protobuf packages here - from CI logs, seems building them is taking most of the time ?

[thompson-tomo]

I agree that is taking most of the time. I am already installing libprotobuf-dev & protobuf-compiler ubuntu packages. I will now add libgrpc-dev and see if that makes a difference.

[thompson-tomo]

@lalitb adding grpc has broken the build https://github.com/open-telemetry/opentelemetry-cpp-contrib/actions/runs/36407828626/job/108880680698#step:6:1

[lalitb]

@lalitb adding grpc has broken the build https://github.com/open-telemetry/opentelemetry-cpp-contrib/actions/runs/36407828626/job/108880680698#step:6:1

Error seems to be missing gRPC C++ library. Could you replace libgrpc-dev with libgrpc++-dev and also add protobuf-compiler-grpc for the compiler plugin?

[thompson-tomo]

@lalitb After a couple of attempts, codeql is now running ultra fast due to using system packages and no errors. Let me know if there is anything else I missed.

Additional packages can be added once #653 is merged