Repository navigation
zcbor test cases not working #201
Description
Activity
It's not our code that is giving the errors, but the CDDL parsing library we use. I see you already made an issue there (anweiss/cddl/issues/199).
Could you provide your own CDDLs that are not working? That could be more helpful as they're probably more real-world and less edge cases (that file is called
corner_cases.cddl.)There are definitely things not supported by cddl-codegen though that would be able to parse with
cddl.Some things look a bit weird in that CDDL though:
For the line 64 one if you rewrite it a bit it should be accepted by us. e.g. change
#6.10(boolval: bool)toboolval: #6.10(bool). I can't find any examples using the formatting there in the CDDL RFC, but I also can't find any using tagged members in general so maybe you can do it both ways.For line 34 I'm confused as it uses group choice syntax
//instead of type choice/but it doesn't seem to be declaring a group? At least the examples I've seen before would be inside of parenthesis marking it as a group e.g. how it's done in line 116.I appreciate the quick feedback.
Here's a simpler CDDL based on what I'm trying to do:
CommandIDs = (Zero: 0) / (One: 1) / (Two: 2) / (Three: 3) MyArray = [ a: uint .le 3, b: uint .le 2, c: uint .le 1, ] Info = [ ? MyArray, ? myBStr: bstr .size 10 ] Command = [ cmd: CommandIDs, myInt: uint, ? Info, ] Response = [ cmd: CommandIDs, myInt: uint, rsp: (Ack: 0) / (NAck: 1), ? Info ]And here's the error:
error: parser errors - input:4:15 - 4 - (Zero: 0) / - ^ expected rule identifier followed by an assignment token '=', '/=' or '//=' - 29 - rsp: (Ack: 0) / (NAck: 1), - ^ missing closing delimiter Error: "incremental parsing error"I welcome any feedback. Again, this is working with zcbor but not with, I guess, the CDDL parsing library in use.
What do you want the output generated to look like for
CommandIDs? The way we handle basic groups like that we tend to create a struct with the fields you have there. If you instead remove the fields (they're all single constants anyway) and just make it a type choice on integers we can instead convert it into a simple C-style enum since all the fields of those basic groups are constants.With these changes:
CommandIDs = 0 ; @name Zero / 1 ; @name One / 2 ; @name Two / 3 ; @name Three AckRsp = 0 ; @name Ack / 1 ; @name NAck Response = [ ... rsp: AckRsp, ... ]Those
; @namecomments aren't required, it's just a way to tell the tool exactly what you want things to be called. If you don't add them they'll be calledCommandIds0,CommandIds1, etc.Unfortunately there are two unsupported things even after rewriting those:
-
.le on numbers. We've only implemented it on array/map lengths since that's the only place we've needed it so far in our use cases - but it wouldn't be too hard to add into deserialization code. In the meantime if it's not completely necessary you could remove it and either not have the check or add it in afterwards by hand.
-
We don't support deserialization for array-encoded structs with optional fields. The general case for this can be complicated, if not even totally ambiguous e.g. situations like
[ ? uint, uint, ? (uint, text), ? text]. The situation you have there is actually something we have in one place in one of the Cardano CDDL specs that we use this tool to generate from so I might actually take a look at this today and see what I can do, at least in cases where it's totally non-ambiguous. In the meantime we still generate serialization code, we just don't implement theDeserializetrait.
e.g. after running the code I get after the fact:
Not generating Command::deserialize() - reasons: Array with optional field info: Info Not generating Info::deserialize() - reasons: Array with optional field my_array: MyArray Array with optional field my_b_str: Vec<u8> Not generating Response::deserialize() - reasons: field info: Info couldn't generate serialize Array with optional field info: Info-
The resulting C code for CommandIDs is this:
struct CommandIDs_ { enum { _CommandIDs_Zero = 0, _CommandIDs_One = 1, _CommandIDs_Two = 2, _CommandIDs_Three = 3, } _CommandIDs_choice; };I don't see anything in the RFC about using "; @ label" but I do seem sentences like:
From 2.1.2[A colon] is also the common way of naming elements of an array just for documentation; see Section 3.4.
This less clean solution works and produces basically the same C code (there's an extra underscore '_' for some reason):
Zero = 0 One = 1 Two = 2 Three = 3 CommandIDs = Zero / One / Two / ThreeSo, now I am running with this (removed all "labels")
Zero = 0 One = 1 Two = 2 Three = 3 CommandIDs = Zero / One / Two / Three MyArray = [ uint, uint, uint, ] Info = [ ? MyArray, ? bstr .size 10 ] Command = [ CommandIDs, uint, ? Info, ] Ack = 0 NAck = 1 Response = [ CommandIDs, uint, Ack / NAck, ? Info ]and am getting a crash:
----------------------------------------- - Generating code... ------------------------------------ thread 'main' panicked at 'should not expose Fixed type in member, only needed for serializaiton: Fixed(Uint(0))', src/intermediate.rs:1426:31 note: run with `RUST_BACKTRACE=1` environment variable to display a backtraceAny ideas?
Can you share what you ran with to get your "Not generating..." output?
I think the zcbor solution to an array-encoded struct with option fields is by adding "_present" booleans to the generated struct, since the order has to be maintained. I think Section 3.5.3 of the RFC helps explain things.@nslowell I got the array struct deserialization with optional fields working: #204
Note: regarding the
.leon the integers we still generate without errors, they're just ignored. So the deserialization code looks something like:let a = Ok(raw.unsigned_integer()? as u64) .map_err(|e: DeserializeError| e.annotate("a"))?; let b = Ok(raw.unsigned_integer()? as u64) .map_err(|e: DeserializeError| e.annotate("b"))?; let c = Ok(raw.unsigned_integer()? as u64) .map_err(|e: DeserializeError| e.annotate("c"))?;
without any checks.
So now with the above change to the
CommandIdschoice + Ack one and the PR I linked your code generates code, just without the.lechecks for now. I might take a look later to see if it's a trivial change to add in a quick error check to deserialization code. If it is I can make a PR, if not it's not a high priority for us since we haven't needed to use it for our own generation purposes (outside of ones that just specifying rust primitives likeuint .le 4294967295and such, which we do support as a way to generate the code asu32, etc instead ofu64).I don't see anything in the RFC about using "; @ label" but I do seem sentences like:
It's a non-standard thing. The RFC specifies comments as
;but they don't impact the underlying CBOR. We just use comments as a way to tell cddl-codegen certain things about how we want the generated output code. You could omit them and the underlying CBOR supported by the output generated code wouldn't change, just how the code looks like.Here's the code I used using the default CLI options:
CommandIDs = 0 ; @name Zero / 1 ; @name One / 2 ; @name Two / 3 ; @name Three MyArray = [ a: uint, ; .le 3, b: uint, ; .le 2, c: uint, ; .le 1, ] Info = [ ? MyArray, ? myBStr: bstr .size 10 ] Command = [ cmd: CommandIDs, myInt: uint, ? Info, ] AckRsp = 0 ; @name Ack / 1 ; @name NAck Response = [ cmd: CommandIDs, myInt: uint, rsp: AckRsp, ? Info ]Note: obviously if you run using
masteryou won't get theDeserializetrait generated for some of those types, but if you run on that PR I linked the deserialization should be generated.[A colon] is also the common way of naming elements of an array just for documentation; see Section 3.4.
Yes, we do that in general. It's just that when you make a basic group like
(zero: 0)our generator will then think you want a struct and it will generatepub struct Zero;pub struct One, etc thenpub enum CommandIDs { Zero(Zero), One(One), ... }. If this is a common CDDL pattern for enums we could maybe add in a special case for it. The reason it's generating these structs is we just have the general group code generation which makes more sense when the fields aren't fixed e.g.vec3 = [x: uint, y: uint, z: uint]would generatepub struct Vec3 { pub x: u64, pub y: u64, pub z: u64 }etc.Sorry the delay.
I ran with your exact CDDL and am getting a crash. See below. Any ideas?
> RUST_BACKTRACE=full ~/cddl-codegen/target/release/cddl-codegen --input Test2.cddl --output export int int Switching from scope 'lib' to 'lib' CommandIDs MyArray Info Command AckRsp Response _CDDL_CODEGEN_EXTERN_TYPE_ _CDDL_CODEGEN_RAW_BYTES_TYPE_ ------------------------------------------ - Handling rule: lib:AckRsp ------------------------------------ AckRsp ------------------------------------------ - Handling rule: lib:CommandIDs ------------------------------------ CommandIDs ------------------------------------------ - Handling rule: lib:MyArray ------------------------------------ MyArray ------------------------------------------ - Handling rule: lib:Info ------------------------------------ Info MyArray MyArray ------------------------------------------ - Handling rule: lib:Command ------------------------------------ Command CommandIDs CommandIDs Info Info ------------------------------------------ - Handling rule: lib:Response ------------------------------------ Response CommandIDs CommandIDs AckRsp AckRsp Info Info ------------------------------------------ - Handling rule: lib:_CDDL_CODEGEN_EXTERN_TYPE_ ------------------------------------ _CDDL_CODEGEN_EXTERN_TYPE_ ------------------------------------------ - Handling rule: lib:_CDDL_CODEGEN_RAW_BYTES_TYPE_ ------------------------------------ _CDDL_CODEGEN_RAW_BYTES_TYPE_ ----------------------------------------- - Generating code... ------------------------------------ thread 'main' panicked at 'can't read: static/serialization.rs', src/generation.rs:225:21 stack backtrace: 0: 0x5641b7c903b1 - std::backtrace_rs::backtrace::libunwind::trace::h81b3ec7e24504ef6 at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5 1: 0x5641b7c903b1 - std::backtrace_rs::backtrace::trace_unsynchronized::h06b108a8ac6d8321 at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5 2: 0x5641b7c903b1 - std::sys_common::backtrace::_print_fmt::hddd2a53a0a5c1518 at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/sys_common/backtrace.rs:65:5 3: 0x5641b7c903b1 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h8286d7e4527fc2ef at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/sys_common/backtrace.rs:44:22 4: 0x5641b7cbb0cc - core::fmt::rt::Argument::fmt::h8fd89467fbfd7337 at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/core/src/fmt/rt.rs:138:9 5: 0x5641b7cbb0cc - core::fmt::write::h51943d696b01a399 at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/core/src/fmt/mod.rs:1094:21 6: 0x5641b7c8d68e - std::io::Write::write_fmt::h917c762025be867c at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/io/mod.rs:1714:15 7: 0x5641b7c901c5 - std::sys_common::backtrace::_print::h80196d93ed8f3afc at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/sys_common/backtrace.rs:47:5 8: 0x5641b7c901c5 - std::sys_common::backtrace::print::hfc2f9057cab2ca91 at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/sys_common/backtrace.rs:34:9 9: 0x5641b7c9195a - std::panicking::panic_hook_with_disk_dump::{{closure}}::h044c489edf02dba6 at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/panicking.rs:278:22 10: 0x5641b7c915f3 - std::panicking::panic_hook_with_disk_dump::hc012686c97dee411 at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/panicking.rs:312:9 11: 0x5641b7c91f5b - std::panicking::default_hook::h38859ccab48a30da at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/panicking.rs:239:5 12: 0x5641b7c91f5b - std::panicking::rust_panic_with_hook::hbec509e90f57ee70 at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/panicking.rs:729:13 13: 0x5641b7c91e57 - std::panicking::begin_panic_handler::{{closure}}::h24e3657f27ebdc7b at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/panicking.rs:621:13 14: 0x5641b7c90816 - std::sys_common::backtrace::__rust_end_short_backtrace::h0fb71023354b550a at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/sys_common/backtrace.rs:151:18 15: 0x5641b7c91ba2 - rust_begin_unwind at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/panicking.rs:617:5 16: 0x5641b79be133 - core::panicking::panic_fmt::haa55128da9cd75d4 at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/core/src/panicking.rs:67:14 17: 0x5641b79ece7c - cddl_codegen::generation::GenerationScope::export::he5da078ac6a96e1a 18: 0x5641b7aa257f - cddl_codegen::main::h579e05b56747a76c 19: 0x5641b7aa71d3 - std::sys_common::backtrace::__rust_begin_short_backtrace::hf7258d21fb51bae9 20: 0x5641b7aac421 - std::rt::lang_start::{{closure}}::hd7b4de7351dc80e1 21: 0x5641b7c86ffb - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h2240a7a2e9d7937c at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/core/src/ops/function.rs:284:13 22: 0x5641b7c86ffb - std::panicking::try::do_call::h6d94044a2dac5948 at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/panicking.rs:524:40 23: 0x5641b7c86ffb - std::panicking::try::hc16721a649b6347e at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/panicking.rs:488:19 24: 0x5641b7c86ffb - std::panic::catch_unwind::ha5ec842243b12570 at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/panic.rs:142:14 25: 0x5641b7c86ffb - std::rt::lang_start_internal::{{closure}}::h718bb866ab67d475 at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/rt.rs:148:48 26: 0x5641b7c86ffb - std::panicking::try::do_call::hca57e22a5a3d68fb at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/panicking.rs:524:40 27: 0x5641b7c86ffb - std::panicking::try::hfd3cbded253620e6 at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/panicking.rs:488:19 28: 0x5641b7c86ffb - std::panic::catch_unwind::h1fa2ff1a7cb410d5 at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/panic.rs:142:14 29: 0x5641b7c86ffb - std::rt::lang_start_internal::h1dfc41da9ec083ca at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/rt.rs:148:20 30: 0x5641b7aa2c85 - main 31: 0x7f4218c2a083 - __libc_start_main at /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16 32: 0x5641b79be85e - _start 33: 0x0 - <unknown>cool, thanks. I was able to produce code with your sample you gave me.
So, now I've got this CDDL that's causing a different crash:
CommandIDs = ( Zero: 0 / One: 1 / Two: 2 / Three: 3 / Four: 4 / Five: 5 / Six: 6 / Seven: 7 ) u32 = uint .size 4 Version = [ major: u32, minor: u32, patch: u32, ] Info = [ ? Version, ? myStr1: bstr .size 10, ? myStr2: [*bstr], ? myStr3: [*bstr], ? myStr4: [*bstr], ? myInt: u32, ] Command = [ cmd: CommandIDs, value: u32, ? Info, ] rsp = (Ack: 0 / NAck: 1) Response = [ cmd: CommandIDs, resp: rsp, ? Info, ]backtrace:
----------------------------------------- - Generating code... ------------------------------------ BytesList thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/intermediate.rs:1129:64 stack backtrace: 0: rust_begin_unwind at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/std/src/panicking.rs:617:5 1: core::panicking::panic_fmt at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/core/src/panicking.rs:67:14 2: core::panicking::panic at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/core/src/panicking.rs:117:5 3: core::option::Option<T>::unwrap at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/core/src/option.rs:935:21 4: cddl_codegen::intermediate::RustType::cbor_types at ./src/intermediate.rs:1129:39 5: cddl_codegen::intermediate::EnumVariant::cbor_types at ./src/intermediate.rs:1946:46 6: cddl_codegen::generation::generate_enum at ./src/generation.rs:6140:31 7: cddl_codegen::generation::GenerationScope::generate_type_choices_from_variants at ./src/generation.rs:2947:9 8: cddl_codegen::generation::GenerationScope::generate at ./src/generation.rs:625:25 9: cddl_codegen::main at ./src/main.rs:152:5 10: core::ops::function::FnOnce::call_once at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library/core/src/ops/function.rs:250:5@nslowell If you rewrite those enums as mentioned earlier in the issue it won't panic e.g.:
CommandIDs = 0 ; @name Zero / 1 ; @name One / 2 ; @name Two / 3 ; @name Three / 4 ; @name Four / 5 ; @name Five / 6 ; @name Six / 7 ; @name Seven rsp = 0 ; @name Ack / 1 ; @name NAckUnfortunately even with #204 we can't emit deserialization traits (serialization works though) because the optional array fields are ambiguous.
Look here:
? myStr2: [*bstr], ? myStr3: [*bstr], ? myStr4: [*bstr],Let's say you're deserializing and you get two arrays of bytes. How will the code know if they should be
myStr2andmyStr3? OrmyStr2andmyStr4? OrmyStr3andmyStr4?
I have successfully used https://github.com/NordicSemiconductor/zcbor for generating C code from CDDLs, but my CDDLs and their provided test cases are getting error messages when I try to run them through this tool.
Can you provide some clarification on why the errors are occurring? Is this tool missing support?
Example for https://github.com/NordicSemiconductor/zcbor/blob/main/tests/cases/corner_cases.cddl: