tsbindgen: a template's specializations are one generic tuple

README.md

@@ -682,7 +682,7 @@ declare function add(a: s32, b: s32): s32;
 declare function dist(p: Reference<Point> | null): f64;
 ```
 
-Structs, enums and `extern "C"` functions are emitted, and polymorphic classes (below). A struct or a class without virtual methods, in any namespace or class, is a tuple when its fields are where a tuple's are: constructors, methods and member initializers don't move them, so `struct Color { float r, g, b, a; Color(); }` is `type Color = [r: f32, g: f32, b: f32, a: f32];`. It is declared under its own name, and `--filter` matches that or the qualified name (`--filter 'nvrhi::*'`); of two of one name in different namespaces, the next is named with its namespaces (`rt_Desc` for `nvrhi::rt::Desc` under `--namespace nvrhi`). A derived struct has its bases' fields first, as C++ lays them out (an empty base has none; MSVC's tail padding after a base is a `_pad` field). A field holding a template's specialization by value declares the specialization, named by its arguments, so a library's types are laid out as that library has them: `std::string` is `basic_string_char` (MSVC's `[_Mypair: ...]` down to its `_Bx` buffer words, size and capacity; libc++'s `[__rep_: [_0: u64, _1: u64, _2: u64]]`), `nvrhi::static_vector<IBindingSet *, 8>` is `static_vector_IBindingSet_8` (its private `std::array`'s elements, then `current_size`), `RefCountPtr<IShader>` is `RefCountPtr_IShader = [ptr_: IShader | null]`. Their member names are the C++ library's own, which differ between MSVC's and libc++; a pointer to a specialization alone stays `Opaque`. One that isn't a tuple is `type Name = Opaque;`, with the reason in a `// skipped:` line: a polymorphic class, a virtual base, a packed or over-aligned layout, or a field tsbindgen can't spell (`long double`, a huge array). Namespace functions (`lib::helper`) and functions with C++ linkage (`cxx_only`) are left out, without a warning. To bind those, expose them through `extern "C"` wrapper functions (with an opaque handle for each class) and run tsbindgen on the header that declares them.
+Structs, enums and `extern "C"` functions are emitted, and polymorphic classes (below). A struct or a class without virtual methods, in any namespace or class, is a tuple when its fields are where a tuple's are: constructors, methods and member initializers don't move them, so `struct Color { float r, g, b, a; Color(); }` is `type Color = [r: f32, g: f32, b: f32, a: f32];`. It is declared under its own name, and `--filter` matches that or the qualified name (`--filter 'nvrhi::*'`); of two of one name in different namespaces, the next is named with its namespaces (`rt_Desc` for `nvrhi::rt::Desc` under `--namespace nvrhi`). A derived struct has its bases' fields first, as C++ lays them out (an empty base has none; MSVC's tail padding after a base is a `_pad` field). A field holding a template's specialization by value is laid out as the library has it, and written once for the template as a generic tuple, its type arguments the parameters and its integer arguments part of the name: `std::vector<Item>` is `vector<Item>`, with `type vector<T> = [_Mypair: [_Myval2: [_Myfirst: Reference<T> | null, ...]]]` under MSVC (the specializations its pointers are in, `_Compressed_pair` and `_Vector_val`, inlined) and `[__begin_: Reference<T> | null, ...]` under libc++; `nvrhi::static_vector<IBindingSet *, 8>` is `static_vector8<IBindingSet | null>` (its private `std::array`'s elements, then `current_size`); `RefCountPtr<IShader>` is `RefCountPtr<IShader>`, with `type RefCountPtr<T> = [ptr_: T | null]`. A specialization whose layout doesn't show its type argument, or that is laid out unlike its template's others (an explicit specialization), is declared on its own, named by its arguments: `std::string` is `basic_string_char` (MSVC's `[_Mypair: ...]` down to its `_Bx` buffer words, size and capacity; libc++'s `[__rep_: [_0: u64, _1: u64, _2: u64]]`). Their member names are the C++ library's own, which differ between MSVC's and libc++; a pointer to a specialization alone stays `Opaque`. One that isn't a tuple is `type Name = Opaque;`, with the reason in a `// skipped:` line: a polymorphic class, a virtual base, a packed or over-aligned layout, or a field tsbindgen can't spell (`long double`, a huge array). Namespace functions (`lib::helper`) and functions with C++ linkage (`cxx_only`) are left out, without a warning. To bind those, expose them through `extern "C"` wrapper functions (with an opaque handle for each class) and run tsbindgen on the header that declares them.
 
 #### C++ classes
 

TSLANG_ISSUES.md

@@ -18,6 +18,7 @@ https://github.com/ASDAlexander77/native_ts_graphics
 | 7 | `Opaque` compared with `Opaque \| null` crashed the compiler | - | **fixed** by #536 (re-checked with the original drawScene shape) |
 | 8 | A ternary of number literals in arithmetic truncated to the other operand's integer type | - | **fixed** by #560 (still wrong on pre-alpha97) |
 | 9-12 | Bool const arrays, `Math.max`, `f32` array literal, `let x = 0` | - | fixed (#537, DefaultLib#31, works, #544 warning) |
+| 13 | A generic function's `T` isn't inferred from an argument of a generic tuple alias (`Vec<T>`) | - | open (pre-alpha100) |
 
 Verified downstream: native_ts_graphics dropped its 71 forwarding constructors (PR #49) and rebuilt
 everything with pre-alpha97 (all objects deleted first): Windows build clean, CTest 61/61, Android
@@ -114,6 +115,31 @@ to TypeScript later; the project now returns pointers to storage that lives as l
 Decide whether this is by design; if so, the C interop docs should say the string must outlive its use,
 otherwise copying on return (or an explicit borrowed annotation) would be safer. Not re-checked.
 
+### 13. `T` not inferred from a generic tuple alias argument
+
+Found on pre-alpha100, writing `stdVectorSize<T>(items: vector<T>)` for native_ts_graphics' NVRHI
+bindings (`vector<T>` a generic tuple alias, as tsbindgen now writes C++ template specializations).
+A call with a `Vec<Item>` argument doesn't infer `T`; giving it explicitly works:
+
+```ts
+type Vec<T> = [first: Ref<T> | null, last: Ref<T> | null];
+type Item = [id: s32, size: s32];
+
+function count<T>(v: Vec<T>): int {
+    return <int>((<index><Opaque>v.last - <index><Opaque>v.first) / sizeof<T>());
+}
+
+function main() {
+    let items: Item[] = [[1, 2], [3, 4]];
+    const v: Vec<Item> = [Ref(items[0]), <Ref<Item>><Opaque>(<index><Opaque>Ref(items[0]) + 2 * sizeof<Item>())];
+    print(count<Item>(v));  // 2
+    print(count(v));        // error: not all types could be inferred
+}
+```
+
+TypeScript infers `T = Item` here (the alias is expanded and matched structurally). native_ts_graphics
+passes the type (`nvrhi.stdVectorSize<nvrhi.BindingSetItem>(...)`) until it is.
+
 ---
 
 ## 8. A union of literals in arithmetic (found live on pre-alpha97)

tslang/include/TsClangImporter/BindingModel.h

@@ -23,6 +23,7 @@ struct TsType
     Kind kind = Kind::Builtin;
     std::string name;            // Builtin: the tslang spelling; Named: the key of a Decl
     std::vector<TsType> pointee; // Reference: exactly one element; Tuple: the field types
+    std::vector<TsType> typeArguments; // Named: a generic's arguments, `vector<Item>`
     std::vector<std::string> fieldNames; // Tuple
     std::string comment;         // shown next to the type: a function pointer's C signature, ...
     bool functionPointer = false;
@@ -44,6 +45,9 @@ struct TsType
 // Spells `type`, asking `nameOf` for the TS name of each named declaration it refers to.
 std::string renderType(const TsType &type, const std::function<std::string(const std::string &)> &nameOf);
 
+// Whether two types are spelled alike (keys compared, not TS names)
+bool sameType(const TsType &a, const TsType &b);
+
 // The declaration keys `type` refers to, in order, each once.
 void collectKeys(const TsType &type, std::vector<std::string> &keys);
 
@@ -73,7 +77,7 @@ enum class DeclKind
 {
     Macro,        // const N = 42;
     OpaqueStruct, // type S = Opaque;
-    Struct,       // type S = [a: T1, b: T2];  (type S = Opaque; when skipped)
+    Struct,       // type S = [a: T1, b: T2];  (type S = Opaque; when skipped); type V<T> = [...] for a generic
     Enum,         // enum E { A = 0 }
     FixedEnum,    // type E = u8; const A = 1;
     Typedef,      // type T = ...;
@@ -102,9 +106,11 @@ struct Decl
     std::string symbol; // Function: the symbol it links as, when an asm label makes it differ from the name
     bool inMainFile = false;
     bool inSystemHeader = false;
+    bool instantiation = false; // a template's specialization, or a class inside one: only declared for its users
     std::string skipReason; // non-empty: left out, with a `// skipped:` line in its place
 
     std::vector<Field> fields;           // Struct, CppClass: fields; Function: parameters
+    std::vector<std::string> typeParameters; // Struct: a generic's, `type vector<T> = [...]`
     TsType type;                         // Function: result; Typedef: aliased type; FixedEnum: underlying
     bool varargs = false;                // Function
     std::vector<Enumerator> enumerators; // Enum, FixedEnum

tslang/include/TsClangImporter/TypeMapper.h

@@ -12,6 +12,7 @@ class Sema;
 }
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/SmallPtrSet.h"
+#include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringSet.h"
 
 #include <utility>
@@ -68,11 +69,14 @@ class TypeMapper
 
     // Whether a struct can have a TS declaration of its own: in C, any; in C++, one in a namespace or
     // a class too (declared under its own name), but not a template, nor a class local to a
-    // function. A template's specialization (std::string, nvrhi::static_vector<T, N>) or a class
-    // inside one is declared once a field holds one by value, whose layout needs it; a pointer to
-    // one alone stays Opaque.
+    // function. A template's specialization (std::string) or a class inside one is declared once a
+    // field holds one by value, whose layout needs it, unless it is a generic's instance
+    // (nvrhi::static_vector<T, N>: genericInstance); a pointer to one alone stays Opaque.
     bool isDeclarable(const clang::RecordDecl *record);
 
+    // a template's specialization, or a class inside one, complete: declarable when a field holds it
+    bool isInstantiated(const clang::RecordDecl *record);
+
     // Why a C++ class can't be a @cpp class ("" if it can): it has to be polymorphic, not a
     // template, on a 64-bit target, with at most one base, itself bindable and not virtual - the
     // layout a @cpp class has (the vtable pointer first, a base object the prefix of a derived one).
@@ -93,6 +97,24 @@ class TypeMapper
         sema = value;
     }
 
+    // A template written once as a generic tuple, `type static_vector8<T> = [...]`: a field holding
+    // one of its specializations is an instance of it, `static_vector8<IBindingSet | null>`.
+    struct Generic
+    {
+        std::string key;           // "generic:nvrhi::static_vector<8>": the template, its non-type arguments
+        std::string name;          // `static_vector8`: the template's name, then its non-type arguments
+        std::string qualifiedName; // `nvrhi::static_vector`
+        std::vector<std::string> parameters;
+        std::vector<Field> fields; // in terms of the parameters
+        const clang::Decl *at;     // the template, where its declaration is from
+    };
+
+    // The generics the specializations mapped so far are instances of, in the order they were met.
+    const std::vector<Generic> &generics() const
+    {
+        return genericList;
+    }
+
     // Every struct and enum a mapped type has named, by key. The AST traversal does not reach a
     // struct first named inside a prototype (`void f(struct S *s);`), so the parser adds these.
     const std::vector<std::pair<std::string, const clang::TagDecl *>> &referencedTags() const
@@ -123,8 +145,20 @@ class TypeMapper
     Slot unionSlot(std::string name, const clang::RecordDecl *unionRecord, uint64_t offset);
     // `[_0: u64, _1: u64, ...]`: `bytes` as words of the largest size up to `align` that fits
     static TsType rawWords(uint64_t bytes, uint64_t align);
-    // a template's specialization, or a class inside one, complete: declarable when a field holds it
-    bool isInstantiated(const clang::RecordDecl *record);
+    // A specialization a field holds as an instance of a generic: its template's type arguments are
+    // its parameters, wherever they are in its layout (a nested specialization holding one inlined:
+    // MSVC's std::vector, through _Compressed_pair and _Vector_val). A skip when it can't be one: a
+    // parameter its layout doesn't show (the raw words of an std::string's characters), an argument
+    // that isn't a type or an integer, or a layout other than its template's other specializations'.
+    TsType genericInstance(const clang::RecordDecl *definition);
+    // what a type argument is in a layout: a @cpp class itself (pointers to it are the class), any
+    // other type as a field of it is
+    TsType argumentType(clang::QualType argument);
+    // `type` with each part that is `arguments[i]` replaced by `parameters[i]` (`used[i]` set), a
+    // specialization holding one inlined
+    TsType abstractType(const TsType &type, const std::vector<TsType> &arguments, const std::vector<std::string> &parameters,
+                        std::vector<bool> &used, int depth);
+    const clang::RecordDecl *recordOfKey(const std::string &key);
 
     TsType mapType(clang::QualType type, Use use);
     std::string referTo(const clang::TagDecl *tag);
@@ -143,6 +177,10 @@ class TypeMapper
     llvm::StringSet<> referencedKeys;
     std::vector<const clang::RecordDecl *> owners; // structs whose fields are being mapped
     llvm::SmallPtrSet<const clang::RecordDecl *, 16> heldByValue; // instantiated records a field holds
+    std::vector<Generic> genericList;
+    llvm::StringMap<size_t> genericIndex;                         // a generic's key -> genericList
+    llvm::DenseMap<const clang::RecordDecl *, TsType> instances;  // a specialization -> its instance, or a skip
+    llvm::SmallPtrSet<const clang::RecordDecl *, 8> generalizing; // specializations being made instances
 };
 
 } // namespace tsbindgen

tslang/lib/TsClangImporter/BindingModel.cpp

@@ -59,8 +59,21 @@ std::string renderType(const TsType &type, const std::function<std::string(const
     {
     case TsType::Kind::Builtin:
         return type.name;
-    case TsType::Kind::Named:
-        return nameOf(type.name);
+    case TsType::Kind::Named: {
+        auto text = nameOf(type.name);
+        if (!type.typeArguments.empty())
+        {
+            std::string arguments;
+            for (auto &argument : type.typeArguments)
+            {
+                arguments += (arguments.empty() ? "" : ", ") + renderType(argument, nameOf);
+            }
+
+            text += "<" + arguments + ">";
+        }
+
+        return text;
+    }
     case TsType::Kind::Reference:
         return "Reference<" + renderType(type.pointee.front(), nameOf) + ">";
     case TsType::Kind::Tuple: {
@@ -77,6 +90,16 @@ std::string renderType(const TsType &type, const std::function<std::string(const
     return type.name;
 }
 
+bool sameType(const TsType &a, const TsType &b)
+{
+    auto same = [](const std::vector<TsType> &x, const std::vector<TsType> &y) {
+        return x.size() == y.size() && std::equal(x.begin(), x.end(), y.begin(), sameType);
+    };
+
+    return a.kind == b.kind && a.name == b.name && a.nullable == b.nullable && a.fieldNames == b.fieldNames &&
+           a.skipReason == b.skipReason && same(a.pointee, b.pointee) && same(a.typeArguments, b.typeArguments);
+}
+
 void collectKeys(const TsType &type, std::vector<std::string> &keys)
 {
     if (type.kind == TsType::Kind::Named && std::find(keys.begin(), keys.end(), type.name) == keys.end())
@@ -88,6 +111,11 @@ void collectKeys(const TsType &type, std::vector<std::string> &keys)
     {
         collectKeys(pointee, keys);
     }
+
+    for (auto &argument : type.typeArguments)
+    {
+        collectKeys(argument, keys);
+    }
 }
 
 std::string nameFromKey(const std::string &key)
@@ -129,7 +157,7 @@ bool isTsBuiltinTypeName(const std::string &name)
         "i8",      "i16",     "i32",     "i64",     "u8",      "u16",    "u32",    "u64",     "s8",
         "s16",     "s32",     "s64",     "f16",     "f32",     "f64",    "f128",   "half",    "float",
         "double",  "boolean", "number",  "string",  "bigint",  "any",    "void",   "never",   "unknown",
-        "object",  "symbol",  "undefined", "null",  "Opaque",  "Reference", "TypeOf", "Array", "String",
+        "object",  "symbol",  "undefined", "null",  "Opaque",  "Ref",    "Reference", "TypeOf", "Array", "String",
         "Number",  "Boolean", "Object",  "Function"};
 
     for (auto *builtin : builtins)

tslang/lib/TsClangImporter/BindingPrinter.cpp

@@ -12,7 +12,8 @@ namespace tsbindgen
 namespace
 {
 
-// The selected declarations: every seed, then whatever a non-skipped selected declaration uses.
+// The selected declarations: every seed, then whatever a non-skipped selected declaration uses. A
+// specialization is no seed: it is declared for what holds it (or inlined in a generic).
 std::vector<bool> selectDecls(const HeaderModel &model, const std::vector<llvm::GlobPattern> &globs)
 {
     llvm::StringMap<size_t> byKey;
@@ -34,9 +35,9 @@ std::vector<bool> selectDecls(const HeaderModel &model, const std::vector<llvm::
     for (size_t index = 0; index < model.decls.size(); ++index)
     {
         auto &decl = model.decls[index];
-        auto seed = globs.empty() ? decl.inMainFile : llvm::any_of(globs, [&](const llvm::GlobPattern &glob) {
+        auto seed = !decl.instantiation && (globs.empty() ? decl.inMainFile : llvm::any_of(globs, [&](const llvm::GlobPattern &glob) {
             return glob.match(decl.name) || (!decl.qualifiedName.empty() && glob.match(decl.qualifiedName));
-        });
+        }));
         if (seed)
         {
             mark(index);
@@ -322,7 +323,13 @@ llvm::Expected<PrintResult> printBindings(const HeaderModel &model, const PrintO
                     fields += (fields.empty() ? "" : ", ") + withComment(field);
                 }
 
-                line(exported + "type " + tsName + " = [" + fields + "];");
+                std::string parameters;
+                for (auto &parameter : decl.typeParameters)
+                {
+                    parameters += (parameters.empty() ? "<" : ", ") + parameter;
+                }
+
+                line(exported + "type " + tsName + (parameters.empty() ? "" : parameters + ">") + " = [" + fields + "];");
                 break;
             }
             case DeclKind::Enum: {

tslang/lib/TsClangImporter/HeaderParser.cpp

@@ -269,6 +269,7 @@ class Collector : public clang::RecursiveASTVisitor<Collector>
         // in a namespace or a class is declared under its own name; --filter matches either.
         pendingRecords.push_back({model.decls.size(), record});
         auto &decl = add(DeclKind::OpaqueStruct, key, TypeMapper::tagName(record), record);
+        decl.instantiation = mapper.isInstantiated(record);
         auto qualified = TypeMapper::qualifiedTagName(record);
         if (qualified != decl.name)
         {
@@ -459,6 +460,18 @@ class Collector : public clang::RecursiveASTVisitor<Collector>
                 }
             }
 
+            // the generic tuples the specializations mapped so far are instances of
+            for (auto &generic : mapper.generics())
+            {
+                if (keys.insert(generic.key).second)
+                {
+                    auto &decl = add(DeclKind::Struct, generic.key, generic.name, generic.at);
+                    decl.qualifiedName = generic.qualifiedName;
+                    decl.typeParameters = generic.parameters;
+                    decl.fields = generic.fields;
+                }
+            }
+
             if (completed == pendingRecords.size())
             {
                 break;