test: add regression test for malformed AST import in AsmJsonImporter - #16874
test: add regression test for malformed AST import in AsmJsonImporter#16874cavdarahmet wants to merge 1 commit into
Conversation
|
While working on this, I noticed the generic Before: After (using Happy to open a small separate PR for this if useful, once this one is merged. |
This is intentional and should stay as is. What the handler prints is not a typical error message meant to be shown in the course of normal usage of the compiler. It's more of a crash dump that helps us track down the bug that caused it. If anything, I'd make it more, not less detailed. |
| BOOST_CHECK(!containsError(result, "FatalError", "")); | ||
| } | ||
|
|
||
| BOOST_AUTO_TEST_CASE(import_ast_malformed_inline_assembly) |
There was a problem hiding this comment.
Can these tests be replaced with ASTJSON or astPropertyTests tests? We're trying to reduce the number of Boost-based tests in favor of custom isoltest test cases, which are easier to maintain. I would not add more tests here if we can avoided and even the existing tests should be rewritten with isoltest whenever possible (though this is very low priority).
There was a problem hiding this comment.
Though they do not accommodate error cases currently. We should extend them though so that they can show errors.
| // AsmJsonImporter::createStatement/createExpression: yulAssert(nodeType prefix == "Yul") | ||
| { | ||
| Json result = compileWithMutation( | ||
| baseAstJson, | ||
| [](Json& _ast) | ||
| { | ||
| Json* node = findFirstNodeByType(_ast, "YulIdentifier"); | ||
| BOOST_REQUIRE(node); | ||
| (*node)["nodeType"] = "XulIdentifier"; | ||
| }); | ||
| BOOST_REQUIRE(result["errors"][0]["message"].is_string()); | ||
| BOOST_CHECK( | ||
| result["errors"][0]["message"].get<std::string>().find("Failed to import AST: Invalid nodeType prefix") | ||
| != std::string::npos); | ||
| BOOST_CHECK(result["errors"][0]["type"] == "Exception"); | ||
| } |
There was a problem hiding this comment.
We can't do it like this. This basically amounts to testing a failing assertion, i.e. compiler's undefined behavior resulting from a bug:
solidity/libyul/AsmJsonImporter.cpp
Line 92 in 8a07979
Before this code can be tested, it needs to have proper validations. The fact is that AST import is still an experimental feature and is unfinished. It wrongly uses asserts for things that should be proper validations.
If you want to add a test for this, we need to change that. We need to define a proper exception type to represent these failure conditions (they should not come out as just type: Exception). Then change the exporter to throw that exception (or use solRequire() when the validation is a simple one-liner).
|
Thanks for the review and the explanation. I see the issue now. It makes sense that these conditions should be handled through proper validation rather than testing failing assertions. I'll look into the suggested approaches and rework the implementation accordingly before updating the tests. |
Description
While doing a coverage-gap analysis (clang source-based coverage over the soltest suite,
including SMT/Z3) to find invariant-checking code paths that no existing test reaches, I
found that AsmJsonImporter's validation of the inline-assembly/Yul subtree of an imported
"SolidityAST" input was never exercised by any test.
This PR adds a regression test that feeds a hand-crafted, corrupted AST (via the
"SolidityAST" standard-json import mode) through StandardCompiler and confirms it's
rejected with a clean error instead of crashing. It covers three previously 0-coverage
yulAssert() lines in AsmJsonImporter.cpp (bad nodeType prefix, non-string "src", unmatched
nodeType), plus one additional gap the assert search didn't catch (a missing required
field that surfaces as a raw nlohmann::json type_error instead of a yulAssert).
Verified against both v0.8.36 and current develop.
Checklist
AI Disclosure