Customizing conan settings: Make glibc version part of the Conan package_id on CentOS 9

.github/workflows/full-build.yml

@@ -283,6 +283,16 @@ jobs:
           wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | gpg --dearmor - | tee /etc/apt/trusted.gpg.d/kitware.gpg >/dev/null
           apt-get update
 
+      - name: Add libc setting to Conan (CentOS)
+        # nrel-v2's gcc 14 binaries are built on ubuntu-24.04 with GLIBC 2.39 (AlmaLinux 9 has 2.34). Making the libc version part of the
+        # package_id means they never match, so we build from source (or pull the glibc 2.34 ones once uploaded to nrel-v2).
+        # See https://github.com/NatLabRockies/openstudio-cmake-tools/issues/6
+        if: matrix.platform == 'centos-9-x64'
+        run: |
+          conan config install developer/conan/config
+          conan profile show -pr:a glibc | grep "os.libc"
+          echo "CONAN_EXTRA_ARGS=-pr:a glibc" >> $GITHUB_ENV
+
       - name: Conan install
         env:
           CONAN_BUILD_POLICY: ${{ contains(inputs.conan_build_all, matrix.platform) && '*' || 'missing' }}
@@ -292,7 +302,8 @@ jobs:
             --build="$CONAN_BUILD_POLICY" \
             -c tools.cmake.cmaketoolchain:generator=Ninja \
             -s compiler.cppstd=20 \
-            -s build_type=${{ env.BUILD_TYPE }}
+            -s build_type=${{ env.BUILD_TYPE }} \
+            ${CONAN_EXTRA_ARGS:-}
 
       - name: Locate Ruby
         run: |

CMakeLists.txt

@@ -562,6 +562,10 @@ if(UNIX)
       if (CMAKE_BUILD_TYPE STREQUAL "RelWithDebInfo")
         set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-uninitialized")
       endif()
+      # GCC >= 12 emits false positive -Warray-bounds and -Wstringop-overflow in libstdc++ on `const char* + std::string&&` (GCC bugs 105329, 107852)
+      if (CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL 12)
+        set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-array-bounds -Wno-stringop-overflow")
+      endif()
     endif()
   endif()
 

developer/conan/binary_incompatility_glibc.md

@@ -1,36 +1,47 @@
 # Binary (in)compatibility of Conan package
 
-The GLIBC version that is used to build the conan packages matter, and that is NOT part of the conan `package_id`.
+The GLIBC version that is used to build the conan packages matters, and that is NOT part of the conan `package_id`.
 
-* Ubuntu 20.04 has GLIBC 2.31
-* Ubuntu 18.04 has GLIBC 2.27
-* Centos7 has GLIBC 2.17
+glibc is backward compatible but not forward compatible: a binary built against an older glibc runs on a newer one, but a binary built against a newer glibc may reference symbols that don't exist on an older one.
 
-The conan docker images that are used:
-* gcc7 for eg is based on Artful (17.10), GLIBC 2.26
-* gcc8 is Bionic (18.04),
-* gcc9 is Eoan (19.10), GLIC 2.30
-* gcc10 and gcc11 are Focal (20.04)
+| Distribution           | GLIBC |
+|------------------------|-------|
+| CentOS 7               | 2.17  |
+| Ubuntu 18.04           | 2.27  |
+| Ubuntu 20.04           | 2.31  |
+| CentOS 9 (AlmaLinux 9) | 2.34  |
+| Ubuntu 22.04           | 2.35  |
+| Ubuntu 24.04           | 2.39  |
 
-cf for eg: https://github.com/conan-io/conan-docker-tools/blob/3f59cb379c0e43097a7685c74b44ae7a3676c0f7/legacy/gcc_7/Dockerfile#L1
+So if a remote (conancenter or `nrel-v2`) has binaries built on a newer distro than yours, the `package_id` still matches (same os / arch / compiler / build_type), Conan downloads it, and you get a link error.
 
+For example, on CentOS 9 with gcc 14 (gcc-toolset-14), the boost binary was built against glibc >= 2.38 [^1], where `sscanf` is redirected to the ISO C23 `__isoc23_sscanf`, which doesn't exist in glibc 2.34:
 
-On Centos7, you'll always get issues, every binary package downloaded will be binary incompatible.
+```
+libboost_filesystem.a(operations.o): in function `boost::filesystem::detail::(anonymous namespace)::syscall_initializer::syscall_initializer()':
+operations.cpp:(.text.startup+0x5f): undefined reference to `__isoc23_sscanf'
+```
+
+See https://github.com/NatLabRockies/openstudio-cmake-tools/issues/6
 
-On Ubuntu 18.04, if you use the system compiler which is gcc-7, it's all good because it'll download a binary package that was made with a 17.10 machine, so an older GLIBC.
+[^1]: conan-center-index only provides Linux binaries for gcc 11 and gcc 13, so there is no gcc 14 boost on conancenter. The gcc 14 binaries on `nrel-v2` are ones we build ourselves on `ubuntu-24.04` (glibc 2.39), see [conan-recipes/.github/workflows/conan_v2.yml](https://github.com/jmarrec/conan-recipes/blob/3b0898bc298335efce8396b49d4d69f593c8a490/.github/workflows/conan_v2.yml#L51).
 
-Now on that Ubuntu 18.04 machine you upgrade to gcc-10 (because you want C++20 for example) -> It gets a binary package that was built on 20.04 => **Problem**
-So now you can't link against boost/1.79.0 for eg, because that boost is looking for a GLIBC provided symbol like `fnctl64` or `statx` and that doesn't exist in GLIBC 2.27.
+## Rebuilding all dependencies
 
-For example: `fnctl` was purely redefined to `fnctl64` in GLIBC 2.28.
+The brute-force solution is to rebuild all of the dependencies on the target machine (`conan install . --build="*"`) to **guarantee** binary compatibility. And this is what you should do if developing locally.
 
-## The solution: rebuild all dependencies
+But nothing stops Conan from downloading an incompatible binary again next time (e.g. after a cache eviction in CI), and the rebuilt binaries can't be shared since they have the same `package_id` as the incompatible ones on the remotes.
 
-The only safe solution in that case is to just rebuild all of the dependencies on the target machine to **guarantee** binary compatibility.
+## The solution: make glibc part of the `package_id`
 
-Our `ConanInstall.cmake` allows getting a `-DCONAN_FIRST_TIME_BUILD_ALL:BOOL=ON` to achieve that. That boolean is automatically switched to OFF after the first pass.
+[config/](config/README.md) declares an optional `os.libc` / `os.libc.version` sub-setting (defaulting to `null`, so nothing changes unless you pass it), and a `glibc` profile that sets it from the detected glibc:
 
 ```shell
-/bin/rm -Rf ~/.conan/data
-cmake -DCONAN_FIRST_TIME_BUILD_ALL:BOOL=ON ..
+conan config install developer/conan/config
+conan install . --build=missing -pr:a glibc -s compiler.cppstd=20 -s build_type=Release
+conan upload "*" -r nrel-v2 -c -p "os.libc.version=2.34"
 ```
+
+Since `os.libc` is a sub-setting of `os`, every recipe that declares `os` in its `settings` gets a different `package_id` when you pass the `glibc` profile. No existing conancenter or `nrel-v2` binary matches, so `--build=missing` builds them from source against the local glibc, and they can be uploaded to `nrel-v2` and reused by other machines with the same glibc.
+
+We use this for CentOS 9 (AlmaLinux 9, glibc 2.34) only.

developer/conan/config/.conanignore

@@ -0,0 +1,23 @@
+README.md
+
+.idea/*
+.github/*
+.venv/*
+venv/*
+tests/*
+.gitignore
+LICENSE
+*README.md*
+*.DS_Store*
+.AppleDouble
+.LSOverride
+*.pytest_cache*
+.*cache
+tmp/*
+tmp2/*
+.vscode/*
+.idea/*
+demos/
+
+*.swp
+*.swo

developer/conan/config/README.md

@@ -0,0 +1,33 @@
+# Conan configuration
+
+Shared Conan 2 configuration for OpenStudio, installed into your Conan home with:
+
+```shell
+conan config install developer/conan/config
+```
+
+* `settings_user.yml`: declares an optional `os.libc` / `os.libc.version` sub-setting on Linux
+* `profiles/glibc`: includes your `default` profile, sets `os.libc` / `os.libc.version` from the detected glibc, and sets `CMAKE_POLICY_VERSION_MINIMUM=3.5` for packages built from source
+* `global.conf`: non-interactive mode, parallel downloads, and a source download cache next to the Conan home
+
+## Why: CentOS 9 / AlmaLinux 9
+
+Conan doesn't include the glibc version in the `package_id`, so a binary built against a newer glibc can get downloaded on a machine with an older one, and then fail to link.
+
+This is a problem on CentOS 9 (AlmaLinux 9), which has glibc 2.34: the gcc 14 binaries on `nrel-v2` are built on Ubuntu 24.04 against glibc 2.39 (conancenter has no gcc 14 Linux binaries), and you get errors like `undefined reference to '__isoc23_sscanf'`. (It could be a problem if we were building for an older Ubuntu release too)
+See [binary_incompatility_glibc.md](../binary_incompatility_glibc.md) and https://github.com/NatLabRockies/openstudio-cmake-tools/issues/6
+
+## How it works
+
+Installing this config does **not** change anything by itself: `os.libc` defaults to `null`, so package IDs stay the same, and Ubuntu, macOS and Windows keep using the existing conancenter / `nrel-v2` binaries.
+
+It only kicks in when you pass the `glibc` profile (`-pr:a glibc`), i.e. on CentOS 9. Since `os.libc` is a sub-setting of `os`, every recipe that declares `os` in its `settings` (nearly all of them) then gets a **different `package_id`**. No existing conancenter or `nrel-v2` binary matches, so `--build=missing` builds them from source against the local glibc. Header-only packages are still downloaded, since their `package_id` ignores settings.
+
+## Usage (CentOS 9)
+
+```shell
+conan install . --build=missing -pr:a glibc -s compiler.cppstd=20 -s build_type=Release
+conan upload "*" -r nrel-v2 -c
+```
+
+Once uploaded to `nrel-v2`, subsequent CentOS builds download those binaries instead of rebuilding them. CI (`full-build.yml`) passes `-pr:a glibc` for the CentOS 9 build only.

developer/conan/config/global.conf

@@ -0,0 +1,8 @@
+# Core configuration (type 'conan config list' to list possible values)
+# e.g, for CI systems, to raise if user input would block
+# core:non_interactive = True
+# some tools.xxx config also possible, though generally better in profiles
+# tools.android:ndk_path = my/path/to/android/ndk
+core:non_interactive = True
+core.download:parallel = {{ [os.cpu_count() - 2, 1]|max }}
+core.sources:download_cache={{ os.path.join(os.path.dirname(conan_home_folder), ".conan-download-cache") }}

developer/conan/config/profiles/glibc

@@ -0,0 +1,11 @@
+include(default)
+
+{% set libc, libc_version = detect_api.detect_libc() %}
+
+[settings]
+os.libc={{ libc }}
+os.libc.version={{ libc_version }}
+
+[buildenv]
+# CMake 4 removed compatibility with cmake_minimum_required(VERSION < 3.5), which many older recipes still use
+CMAKE_POLICY_VERSION_MINIMUM=3.5

developer/conan/config/settings_user.yml

@@ -0,0 +1,10 @@
+os:
+    Linux:
+        libc:
+            null:
+            gnu:
+                version: ["2.0", "2.1", "2.2", "2.3", "2.4", "2.5", "2.6", "2.7", "2.8", "2.9",
+                          "2.10", "2.11", "2.12", "2.13", "2.14", "2.15", "2.16", "2.17", "2.18",
+                          "2.19", "2.20", "2.21", "2.22", "2.23", "2.24", "2.25", "2.26", "2.27",
+                          "2.28", "2.29", "2.30", "2.31", "2.32", "2.33", "2.34", "2.35", "2.36",
+                          "2.37", "2.38", "2.39", "2.40", "2.41", "2.42", "2.43", "2.44"]

python/engine/PythonEngine.cpp

@@ -8,6 +8,7 @@
 #include "../../src/utilities/core/Filesystem.hpp"
 #include <fmt/format.h>
 
+#include <algorithm>
 #include <stdexcept>
 #include <string>
 #include <type_traits>

src/epjson/epJSONTranslator.cpp

@@ -21,6 +21,13 @@
 #include <vector>
 #include <string_view>
 
+// GCC >= 14: tells -Wdangling-reference that the returned reference does not bind to any temporary argument
+#if defined(__has_cpp_attribute) && __has_cpp_attribute(gnu::no_dangling)
+#  define OS_NO_DANGLING [[gnu::no_dangling]]
+#else
+#  define OS_NO_DANGLING
+#endif
+
 namespace openstudio::epJSON {
 
 template <typename... KeyType>
@@ -61,7 +68,7 @@ enum class JSONValueType
   NumberOrString
 };
 
-const std::string& toJSONFieldName(std::map<std::string, std::string>& fieldNames, const std::string& fieldNameInput) {
+OS_NO_DANGLING const std::string& toJSONFieldName(std::map<std::string, std::string>& fieldNames, const std::string& fieldNameInput) {
 
   if (const auto& cached_value = fieldNames.find(fieldNameInput); cached_value != fieldNames.end()) {
     return cached_value->second;
@@ -160,8 +167,9 @@ JSONValueType schemaPropertyTypeDecode(const Json::Value& type) {
  * else:
  *  getSchemaObjectProperties > [group_name] > items > properties > [field_name] > [field_property_name]
  */
-const Json::Value& getSchemaObjectFieldProperty(const Json::Value& schema, const std::string& type_description, const std::string& group_name,
-                                                const std::string& field_name, const std::string& field_property_name) {
+OS_NO_DANGLING const Json::Value& getSchemaObjectFieldProperty(const Json::Value& schema, const std::string& type_description,
+                                                               const std::string& group_name, const std::string& field_name,
+                                                               const std::string& field_property_name) {
 
   const auto& objectProperties = getSchemaObjectProperties(schema, type_description);
 
@@ -250,9 +258,9 @@ std::string fixupEnumerationValue(const Json::Value& schema, const std::string&
 }
 
 /* Find name of an extensible group and cache it */
-auto getGroupName(std::map<std::pair<std::string, std::string>, std::pair<std::string, bool>>& group_names,
-                  std::map<std::string, std::string>& /*field_names*/, const Json::Value& schema, const std::string& type_description,
-                  const std::string& group_name) -> const auto& {
+OS_NO_DANGLING auto getGroupName(std::map<std::pair<std::string, std::string>, std::pair<std::string, bool>>& group_names,
+                                 std::map<std::string, std::string>& /*field_names*/, const Json::Value& schema, const std::string& type_description,
+                                 const std::string& group_name) -> const auto& {
 
   const auto key = std::pair{type_description, group_name};
 
@@ -444,12 +452,13 @@ Json::Value toJSON(const openstudio::IdfFile& idf, const openstudio::path& schem
     };
 
     std::size_t cur_group_number = 0;
+    const auto idd_eg = obj.iddObject().extensibleGroup();
 
     for (const auto& g : obj.extensibleGroups()) {
       ++cur_group_number;
       // get first field and try to make a group name out of it
       const auto& [group_name, is_array_group] =
-        openstudio::epJSON::getGroupName(group_names, field_names, schema, type_description, obj.iddObject().extensibleGroup()[0].name());
+        openstudio::epJSON::getGroupName(group_names, field_names, schema, type_description, idd_eg[0].name());
 
       auto& containing_json = [&json_object, &group_name = group_name, is_array_group = is_array_group]() -> auto& {
         if (is_array_group) {
@@ -461,7 +470,7 @@ Json::Value toJSON(const openstudio::IdfFile& idf, const openstudio::path& schem
       }();
 
       for (unsigned int idx = 0; idx < g.numFields(); ++idx) {
-        const auto& iddField = obj.iddObject().extensibleGroup()[idx];
+        const auto& iddField = idd_eg[idx];
 
         const auto fieldName = getFieldName(is_array_group, obj.iddObject(), schema, type_description, cur_group_number, idx,
                                             toJSONFieldName(field_names, iddField.name()));

src/scriptengine/ScriptEngine.hpp

@@ -8,14 +8,17 @@
 
 #include "../utilities/core/ApplicationPathHelpers.hpp"
 #include "../utilities/core/DynamicLibrary.hpp"
+#include <algorithm>
 #include <any>
 #include <functional>
+#include <iterator>
 #include <map>
 #include <memory>
 #include <stdexcept>
 #include <string>
 #include <string_view>
 #include <typeinfo>
+#include <vector>
 
 namespace openstudio {