@cppvtable: Itanium ABI calls (x86-64, AArch64): struct returns with @cppnontrivial, narrow arguments extended

docs/cppvtable-classes.md

@@ -60,14 +60,18 @@ these are errors and not warnings.
 | `instanceof`, a cast to or from `any`, a cast to an interface | each asks the object through slot 0, which is a C++ method |
 | `-mm=rc` / `-mm=own` | retains and releases write a count in the word before an object, inside the C++ object |
 | a 32-bit x86 MSVC target | C++ member functions are `__thiscall` there (`this` in ECX) |
-| a method returning a tuple, on a non-MSVC target | the Itanium ABI returns records otherwise (see below) |
+| a method returning a tuple, on a target other than MSVC and Itanium x86-64 / AArch64 | each ABI returns records its own way (see below) |
+| on Itanium, a method returning a trivial tuple other than one or two 8-byte integer or pointer fields | SysV / AAPCS64 pack it into registers by its fields, which a tuple isn't (see below) |
 
 A generic or static class can't be `@cppvtable` either.
 
 ## Writing a binding
 
 - **Slot numbers.** Take them from the compiler, not by counting: MSVC prints a class's vtable
-  with `cl /c /Zs /d1reportSingleClassLayout<Name> file.cpp`. Declare unused slots with
+  with `cl /c /Zs /d1reportSingleClassLayout<Name> file.cpp`; clang (Itanium) with
+  `clang++ -S -emit-llvm -o /dev/null -Xclang -fdump-vtable-layouts file.cpp`, for a class whose
+  vtable the file emits (it defines the first virtual method that isn't inline), where slot N is
+  entry N + 2 (entries 0 and 1 are the offset to top and the RTTI). Declare unused slots with
   placeholder names (`_slot17(): void;`) so the methods you call keep their numbers.
 - **The destructor** is one slot under MSVC (the scalar deleting destructor) and two under the
   Itanium ABI. Don't call it: release objects the way the C++ library does (`Release()`).
@@ -80,6 +84,7 @@ A generic or static class can't be `@cppvtable` either.
 
   ```ts
   type RefCountPtr = [pointer: Opaque | null];          // nvrhi::RefCountPtr<T>
+  @cppnontrivial                                        // it has a destructor
   createBuffer(desc: Ref<BufferDesc>): RefCountPtr;     // const handle = device.createBuffer(Ref(desc))
   ```
 
@@ -90,12 +95,27 @@ A generic or static class can't be `@cppvtable` either.
   instance methods". A method returning a class (`getLayout(): IBindingLayout`, a C++ pointer) or
   a number still returns it in a register.
 
+  The Itanium ABI (g++, clang) decides by the record, and tslang does it on x86-64 and AArch64
+  (Linux, Android):
+
+  - a record with a destructor or a copy constructor - `RefCountPtr<T>` - comes back through a
+    hidden pointer passed *before* `this`, as the call's `sret` argument (in RDI on x86-64, X8 on
+    AArch64). A tuple can't say the C++ type has them, so mark the method `@cppnontrivial`. (A
+    trivial record of more than 16 bytes is returned the same way, so `@cppnontrivial` is right
+    for it too.) MSVC ignores the mark;
+  - a trivial record of one or two 8-byte integer or pointer fields - `nvrhi::Object` - comes
+    back in RAX:RDX / X0:X1, as tslang returns such a tuple: no mark;
+  - any other trivial record of up to 16 bytes is packed into registers by the SysV / AAPCS64
+    classification, which isn't done: such a method is an error, as is any method returning a
+    tuple on another Itanium target (32-bit x86, say).
+
   The record isn't destroyed: a `RefCountPtr` the method returns holds a reference, which the
-  caller gives back with `Release()`. The Itanium ABI passes the pointer before `this`, and only
-  for some records, so on a non-MSVC target such a method is an error. A method stored in a
-  variable and called later takes the hidden pointer as its first argument; call it directly.
-  tslang pre-alpha94 and earlier need the hidden pointer declared as a parameter:
-  `createBuffer(ret: Ref<Opaque | null>, desc: Ref<BufferDesc>): Opaque`.
+  caller gives back with `Release()`. A method stored in a variable and called later takes the
+  hidden pointer as an argument; call it directly. tslang pre-alpha94 and earlier need the hidden
+  pointer declared as a parameter: `createBuffer(ret: Ref<Opaque | null>, desc: Ref<BufferDesc>): Opaque`.
+- **Narrow parameters.** A `bool` is a `boolean`, an `uint8_t` (or an `enum class : uint8_t`) a
+  `u8`, an `int16_t` an `s16`: a call of a `@cppvtable` method extends an argument narrower than
+  32 bits as a C++ caller does (`zeroext`, `signext` for a signed type), which clang's code relies on.
 - **Structs by reference** (`const BufferDesc&`) are a `Ref<T>` to a named tuple with the C++
   layout. That layout depends on the C++ build: `std::string`, for example, is 32 bytes with the
   release CRT and 40 with the debug CRT.

tslang/include/TypeScript/Defines.h

@@ -194,6 +194,16 @@
 #define LINK_NAME "linkname"
 // A class laid over an existing C++ object: its vtable is the C++ one (see ClassInfo::isCppVTable)
 #define CPP_VTABLE "cppvtable"
+// A @cppvtable method's C++ return type has a destructor or a copy constructor (see
+// mlirGenClassCppVTableHiddenReturn)
+#define CPP_NON_TRIVIAL "cppnontrivial"
+// On a call (CallIndirect, then CallInternal or Invoke): the index of the argument that is the hidden
+// pointer the callee returns a record through, which LowerToLLVM marks `sret` for the target to pass
+// it where the C++ ABI does (X8 on AArch64)
+#define SRET_ARG_ATTR_NAME "ts.sret_arg"
+// On a call: it calls a C++ method (of a @cppvtable class), whose arguments narrower than 32 bits
+// LowerToLLVM marks zeroext / signext, as a C++ caller extends them (clang relies on it)
+#define CPP_CALL_ATTR_NAME "ts.cpp_call"
 
 #if __LP64__
 #define TRAMPOLINE_SIZE 48

tslang/include/TypeScript/MLIRLogic/MLIRGenStore.h

@@ -152,9 +152,12 @@ struct MethodInfo
     // @dllname: exported symbol name, empty when the method keeps funcName
     std::string dllName;
     // A @cppvtable method returning a C++ record: the type it is declared with (funcType is the
-    // one its slot is called with, the hidden result pointer after `this`, see
+    // one its slot is called with, the hidden result pointer added, see
     // mlirGenClassCppVTableHiddenReturn); null for any other method.
     mlir_ts::FunctionType declaredFuncType;
+    // @cppnontrivial: the C++ record a @cppvtable method returns has a destructor or a copy
+    // constructor, which decides how the Itanium ABI returns it
+    bool cppNonTrivialReturn = false;
 };
 
 struct GenericMethodInfo

tslang/include/TypeScript/TargetInfo.h

@@ -38,6 +38,12 @@ struct TargetInfo
     // before `this`, and only for some records.
     bool hiddenReturnPointerAfterThis = false;
 
+    // The Itanium ABI on x86-64 and AArch64 returns a record with a destructor or a copy
+    // constructor (marked @cppnontrivial) through a hidden pointer, passed before `this` and marked
+    // `sret` (RDI on x86-64, X8 on AArch64); a record of one or two 8-byte integer or pointer fields
+    // without them comes back in RAX:RDX / X0:X1, as a tuple is returned.
+    bool hiddenReturnPointerBeforeThis = false;
+
     static TargetInfo fromTriple(const llvm::Triple &target, const llvm::Triple &host)
     {
         TargetInfo info;
@@ -62,6 +68,8 @@ struct TargetInfo
             target.getArch() == host.getArch() && target.getOS() == host.getOS();
         info.memberCallsUseThiscall = msvc && x86_32;
         info.hiddenReturnPointerAfterThis = msvc;
+        info.hiddenReturnPointerBeforeThis =
+            !msvc && (target.getArch() == llvm::Triple::x86_64 || target.getArch() == llvm::Triple::aarch64);
 
         return info;
     }

tslang/lib/TypeScript/DeclarationPrinter.cpp

@@ -646,6 +646,13 @@ namespace typescript
                 newline();
             }
 
+            if (method.cppNonTrivialReturn)
+            {
+                os.indent(4);
+                os << "@" CPP_NON_TRIVIAL;
+                newline();
+            }
+
             os.indent(4);
 
             if (method.accessLevel == mlir_ts::AccessLevel::Protected)

tslang/lib/TypeScript/LowerToAffineLoops.cpp

@@ -2220,8 +2220,14 @@ struct CallIndirectOpLowering : public TsPattern<mlir_ts::CallIndirectOp>
                                 : op.getOperands()) llvm::dbgs()
                                << "!! ...call -> invoke operands: " << opit << "\n";);
 
+                auto callAttrs = getCallABIAttrs(op);
                 auto res = rewriter.replaceOpWithNewOp<mlir_ts::InvokeOp>(
                     op, op.getResultTypes(), op.getOperands(), continuationBlock, ValueRange{}, unwind, ValueRange{});
+                for (auto callAttr : callAttrs)
+                {
+                    res->setAttr(callAttr.getName(), callAttr.getValue());
+                }
+
                 return success();
             }
 
@@ -2230,9 +2236,31 @@ struct CallIndirectOpLowering : public TsPattern<mlir_ts::CallIndirectOp>
         }
 
         // just replace
-        rewriter.replaceOpWithNewOp<mlir_ts::CallInternalOp>(op, op.getResultTypes(), op.getOperands());
+        auto callAttrs = getCallABIAttrs(op);
+        auto callInternalOp = rewriter.replaceOpWithNewOp<mlir_ts::CallInternalOp>(op, op.getResultTypes(), op.getOperands());
+        for (auto callAttr : callAttrs)
+        {
+            callInternalOp->setAttr(callAttr.getName(), callAttr.getValue());
+        }
+
         return success();
     }
+
+    // what LowerToLLVM makes the call's argument attributes from (CPP_CALL_ATTR_NAME,
+    // SRET_ARG_ATTR_NAME): the call they are on is replaced, and they go on its replacement
+    static SmallVector<NamedAttribute> getCallABIAttrs(mlir_ts::CallIndirectOp op)
+    {
+        SmallVector<NamedAttribute> callAttrs;
+        for (auto name : {CPP_CALL_ATTR_NAME, SRET_ARG_ATTR_NAME})
+        {
+            if (auto attr = op->getAttr(name))
+            {
+                callAttrs.push_back(NamedAttribute(StringAttr::get(op->getContext(), name), attr));
+            }
+        }
+
+        return callAttrs;
+    }
 };
 
 struct ThrowOpLowering : public TsPattern<mlir_ts::ThrowOp>

tslang/lib/TypeScript/LowerToLLVM.cpp

@@ -2130,6 +2130,65 @@ struct SymbolCallInternalOpLowering : public TsLlvmPattern<mlir_ts::SymbolCallIn
     }
 };
 
+// The argument attributes of a call of a C++ method (a @cppvtable class's), from the ts call's
+// `args` (its operands after the callee):
+// - CPP_CALL_ATTR_NAME: an integer argument narrower than 32 bits, a boolean too, is `zeroext`
+//   (`signext` if signed), as a C++ caller extends it: clang's code relies on it, and tslang's
+//   otherwise leaves the register's upper bits undefined;
+// - SRET_ARG_ATTR_NAME: the hidden pointer the method returns a record through (the Itanium ABI)
+//   is `sret`, so the target passes it where the ABI does: RDI on x86-64, as the first argument
+//   anyway, but X8 on AArch64.
+template <typename LLVMCallOp>
+static void setCallABIArgAttrs(Operation *tsOp, LLVMCallOp llvmOp, mlir::ValueRange args, TypeConverterHelper &tch)
+{
+    auto sretArg = tsOp->getAttrOfType<mlir::IntegerAttr>(SRET_ARG_ATTR_NAME);
+    auto cppCall = tsOp->hasAttr(CPP_CALL_ATTR_NAME);
+    if (!sretArg && !cppCall)
+    {
+        return;
+    }
+
+    auto *context = tsOp->getContext();
+    auto name = [&](StringRef attrName) { return mlir::StringAttr::get(context, attrName); };
+    SmallVector<mlir::Attribute> argAttrs;
+    auto any = false;
+    for (auto [index, arg] : llvm::enumerate(args))
+    {
+        SmallVector<mlir::NamedAttribute> attrs;
+        auto type = arg.getType();
+        if (sretArg && sretArg.getInt() == static_cast<int64_t>(index))
+        {
+            if (auto refType = dyn_cast<mlir_ts::RefType>(type))
+            {
+                attrs.push_back(mlir::NamedAttribute(name(LLVM::LLVMDialect::getStructRetAttrName()),
+                                                     mlir::TypeAttr::get(tch.convertType(refType.getElementType()))));
+            }
+        }
+
+        if (cppCall)
+        {
+            if (isa<mlir_ts::BooleanType>(type))
+            {
+                attrs.push_back(mlir::NamedAttribute(name(LLVM::LLVMDialect::getZExtAttrName()), mlir::UnitAttr::get(context)));
+            }
+            else if (auto intType = dyn_cast<mlir::IntegerType>(type); intType && intType.getWidth() < 32)
+            {
+                attrs.push_back(mlir::NamedAttribute(
+                    name(intType.isSigned() ? LLVM::LLVMDialect::getSExtAttrName() : LLVM::LLVMDialect::getZExtAttrName()),
+                    mlir::UnitAttr::get(context)));
+            }
+        }
+
+        any |= !attrs.empty();
+        argAttrs.push_back(mlir::DictionaryAttr::get(context, attrs));
+    }
+
+    if (any)
+    {
+        llvmOp.setArgAttrsAttr(mlir::ArrayAttr::get(context, argAttrs));
+    }
+}
+
 struct CallInternalOpLowering : public TsLlvmPattern<mlir_ts::CallInternalOp>
 {
     using TsLlvmPattern<mlir_ts::CallInternalOp>::TsLlvmPattern;
@@ -2169,6 +2228,8 @@ struct CallInternalOpLowering : public TsLlvmPattern<mlir_ts::CallInternalOp>
         // generic (TypeRange, ValueRange) builder above does not set it.
         callRes.getProperties().setOperandSegmentSizes({static_cast<int32_t>(transformed.getOperands().size()), 0});
         callRes.setOpBundleSizes({});
+        // the arguments follow the callee pointer, operand #0
+        setCallABIArgAttrs(op, callRes, op.getOperands().drop_front(), tch);
 
         auto returns = callRes.getResults();
         if (returns.size() > 0)
@@ -2326,11 +2387,13 @@ struct InvokeOpLowering : public TsLlvmPattern<mlir_ts::InvokeOp>
         }
         else
         {
-            rewriter.replaceOpWithNewOp<LLVM::InvokeOp>(op, llvmTypes, mlir::FlatSymbolRefAttr(), transformed.getOperands(),
+            auto invokeOp = rewriter.replaceOpWithNewOp<LLVM::InvokeOp>(op, llvmTypes, mlir::FlatSymbolRefAttr(), transformed.getOperands(),
                 op.getNormalDest(), transformed.getNormalDestOperands(), op.getUnwindDest(),
                 transformed.getUnwindDestOperands());
+            // an indirect call: the arguments follow the callee pointer, call operand #0
+            setCallABIArgAttrs(op, invokeOp, op.getCallOperands().drop_front(), tch);
         }
-        
+
         return success();
     }
 };

tslang/lib/TypeScript/MLIRGenAccessCall.cpp

@@ -1752,6 +1752,14 @@ namespace mlirgen
                 auto thisValue = builder.create<mlir_ts::GetThisOp>(location, calledFuncType.getInput(0), funcRefValue);
                 auto unboundFuncRefValue = builder.create<mlir_ts::GetMethodOp>(location, calledFuncType, funcRefValue);
 
+                // a C++ method's narrow integer arguments are extended as C++ callers extend them
+                // (LowerToLLVM)
+                SmallVector<mlir::NamedAttribute> callAttrs;
+                if (isCppVTableMethod(funcRefValue))
+                {
+                    callAttrs.push_back(builder.getNamedAttr(CPP_CALL_ATTR_NAME, builder.getUnitAttr()));
+                }
+
                 // a @cppvtable method returning a record (mlirGenClassCppVTableHiddenReturn): it
                 // writes it to a local, passed after `this`, which is the call's value
                 if (auto resultType = cppVTableHiddenReturnType(funcRefValue))
@@ -1761,8 +1769,31 @@ namespace mlirgen
                                                                          builder.getIndexAttr(0));
                     SmallVector<mlir::Value, 4> slotOperands;
                     slotOperands.push_back(resultSlot);
-                    slotOperands.append(operands.begin(), operands.end());
-                    auto callResult = mlirGenCallFunction(location, calledFuncType, unboundFuncRefValue, thisValue, slotOperands, genContext);
+                    ValueOrLogicalResult callResult(mlir::failure());
+                    if (cppVTableHiddenReturnBeforeThis(funcRefValue))
+                    {
+                        // the Itanium ABI: the slot is called as (Ref<T>, this, args...), the pointer
+                        // `sret`, which the target passes where the ABI does - RDI on x86-64, as the
+                        // first argument would be, but X8 on AArch64
+                        SmallVector<mlir::Type> inputs;
+                        inputs.push_back(calledFuncType.getInput(1));
+                        inputs.push_back(calledFuncType.getInput(0));
+                        inputs.append(calledFuncType.getInputs().begin() + 2, calledFuncType.getInputs().end());
+                        auto swappedFuncType = getFunctionType(inputs, calledFuncType.getResults(), calledFuncType.isVarArg());
+                        auto swappedFuncRefValue = builder.create<mlir_ts::GetMethodOp>(location, swappedFuncType, funcRefValue);
+                        slotOperands.push_back(thisValue);
+                        slotOperands.append(operands.begin(), operands.end());
+                        callAttrs.push_back(builder.getNamedAttr(SRET_ARG_ATTR_NAME, builder.getI32IntegerAttr(0)));
+                        callResult = mlirGenCallFunction(location, swappedFuncType, swappedFuncRefValue, slotOperands, genContext,
+                                                         callAttrs);
+                    }
+                    else
+                    {
+                        slotOperands.append(operands.begin(), operands.end());
+                        callResult = mlirGenCallFunction(location, calledFuncType, unboundFuncRefValue, thisValue, slotOperands, genContext,
+                                                         callAttrs);
+                    }
+
                     if (mlir::failed(callResult))
                     {
                         value = mlir::failure();
@@ -1773,7 +1804,7 @@ namespace mlirgen
                     return;
                 }
 
-                value = mlirGenCallFunction(location, calledFuncType, unboundFuncRefValue, thisValue, operands, genContext);
+                value = mlirGenCallFunction(location, calledFuncType, unboundFuncRefValue, thisValue, operands, genContext, callAttrs);
             })
             .Case<mlir_ts::ExtensionFunctionType>([&](auto calledExtentFuncType) {
                 auto calledFuncType =