Bakefile tests failing on clean repository

[davidag]

In Debian testing/bullseye, clone the ts repository and run ./Bakefile test-all as suggested in the README:

~/projects/ts$ ./Bakefile test-all
# alpine /bin/bash      69 pass 0 fail 0 skip 1 s
# alpine /bin/zsh       65 pass 4 fail 0 skip 4 s
F alpine /bin/zsh       [./test/suite:213] test_ts_accepts_options_anywhere
F alpine /bin/zsh       [./test/suite:223] test_script_accepts_options_anywhere
F alpine /bin/zsh       [./test/suite:778] test_ts_options
F alpine /bin/zsh       [./test/suite:816] test_undeclared_variable
# centos /bin/bash      69 pass 0 fail 0 skip 2 s
# centos /bin/zsh       64 pass 5 fail 0 skip 5 s
F centos /bin/zsh       [./test/suite:213] test_ts_accepts_options_anywhere
F centos /bin/zsh       [./test/suite:223] test_script_accepts_options_anywhere
F centos /bin/zsh       [./test/suite:778] test_ts_options
F centos /bin/zsh       [./test/suite:816] test_undeclared_variable
F centos /bin/zsh       [./test/suite:829] test_ts_allows_exit_trap
# centos /bin/ksh       68 pass 1 fail 0 skip 1 s
F centos /bin/ksh       [./test/suite:816] test_undeclared_variable
# debian /bin/bash      69 pass 0 fail 0 skip 5 s
# debian /bin/zsh       65 pass 4 fail 0 skip 4 s
F debian /bin/zsh       [./test/suite:213] test_ts_accepts_options_anywhere
F debian /bin/zsh       [./test/suite:223] test_script_accepts_options_anywhere
F debian /bin/zsh       [./test/suite:778] test_ts_options
F debian /bin/zsh       [./test/suite:816] test_undeclared_variable
# debian /bin/ksh       67 pass 2 fail 0 skip 1 s
F debian /bin/ksh       [./test/suite:709] test_ts_uses_TS_DIFF_to_diff_outputs
F debian /bin/ksh       [./test/suite:816] test_undeclared_variable
# fedora /bin/bash      50 pass 0 fail 19 skip 1 s
S fedora /bin/bash      [./test/suite:416] test_script_e_option_executes_test_in_shell
S fedora /bin/bash      [./test/suite:709] test_ts_uses_TS_DIFF_to_diff_outputs
S fedora /bin/bash      [./test/suite:754] test_ts_returns_to_TS_USR_DIR_for_each_test
S fedora /bin/bash      [./test/suite:758] test_ts_exit_fail_in_setup
S fedora /bin/bash      [./test/suite:762] test_ts_exit_fail_in_test
S fedora /bin/bash      [./test/suite:766] test_ts_exit_fail_in_teardown
S fedora /bin/bash      [./test/suite:770] test_ts_exit_fail_after_untrap_EXIT
S fedora /bin/bash      [./test/suite:774] test_ts_fail_examples
S fedora /bin/bash      [./test/suite:778] test_ts_options
S fedora /bin/bash      [./test/suite:796] test_ts_pass_examples
S fedora /bin/bash      [./test/suite:800] test_ts_public_function_conflict
S fedora /bin/bash      [./test/suite:804] test_ts_return_fail_in_setup
S fedora /bin/bash      [./test/suite:808] test_ts_return_fail_in_teardown
S fedora /bin/bash      [./test/suite:812] test_ts_test_detection
S fedora /bin/bash      [./test/suite:816] test_undeclared_variable
S fedora /bin/bash      [./test/suite:853] test_ts_background_job_example
S fedora /bin/bash      [./test/suite:858] test_ts_readme_example
S fedora /bin/bash      [./test/suite:872] test_ts_readme_troubleshoot_fail
S fedora /bin/bash      [./test/suite:876] test_ts_readme_troubleshoot_pass
# fedora /bin/zsh       48 pass 2 fail 19 skip 1 s
F fedora /bin/zsh       [./test/suite:213] test_ts_accepts_options_anywhere
F fedora /bin/zsh       [./test/suite:223] test_script_accepts_options_anywhere
S fedora /bin/zsh       [./test/suite:416] test_script_e_option_executes_test_in_shell
S fedora /bin/zsh       [./test/suite:709] test_ts_uses_TS_DIFF_to_diff_outputs
S fedora /bin/zsh       [./test/suite:754] test_ts_returns_to_TS_USR_DIR_for_each_test
S fedora /bin/zsh       [./test/suite:758] test_ts_exit_fail_in_setup
S fedora /bin/zsh       [./test/suite:762] test_ts_exit_fail_in_test
S fedora /bin/zsh       [./test/suite:766] test_ts_exit_fail_in_teardown
S fedora /bin/zsh       [./test/suite:770] test_ts_exit_fail_after_untrap_EXIT
S fedora /bin/zsh       [./test/suite:774] test_ts_fail_examples
S fedora /bin/zsh       [./test/suite:778] test_ts_options
S fedora /bin/zsh       [./test/suite:778] test_ts_options
S fedora /bin/zsh       [./test/suite:796] test_ts_pass_examples
S fedora /bin/zsh       [./test/suite:800] test_ts_public_function_conflict
S fedora /bin/zsh       [./test/suite:804] test_ts_return_fail_in_setup
S fedora /bin/zsh       [./test/suite:808] test_ts_return_fail_in_teardown
S fedora /bin/zsh       [./test/suite:812] test_ts_test_detection
S fedora /bin/zsh       [./test/suite:816] test_undeclared_variable
S fedora /bin/zsh       [./test/suite:853] test_ts_background_job_example
S fedora /bin/zsh       [./test/suite:858] test_ts_readme_example
S fedora /bin/zsh       [./test/suite:872] test_ts_readme_troubleshoot_fail
S fedora /bin/zsh       [./test/suite:876] test_ts_readme_troubleshoot_pass
# fedora /bin/ksh       50 pass 0 fail 19 skip 1 s
S fedora /bin/ksh       [./test/suite:416] test_script_e_option_executes_test_in_shell
S fedora /bin/ksh       [./test/suite:709] test_ts_uses_TS_DIFF_to_diff_outputs
S fedora /bin/ksh       [./test/suite:754] test_ts_returns_to_TS_USR_DIR_for_each_test
S fedora /bin/ksh       [./test/suite:758] test_ts_exit_fail_in_setup
S fedora /bin/ksh       [./test/suite:762] test_ts_exit_fail_in_test
S fedora /bin/ksh       [./test/suite:766] test_ts_exit_fail_in_teardown
S fedora /bin/ksh       [./test/suite:770] test_ts_exit_fail_after_untrap_EXIT
S fedora /bin/ksh       [./test/suite:774] test_ts_fail_examples
S fedora /bin/ksh       [./test/suite:778] test_ts_options
S fedora /bin/ksh       [./test/suite:796] test_ts_pass_examples
S fedora /bin/ksh       [./test/suite:800] test_ts_public_function_conflict
S fedora /bin/ksh       [./test/suite:804] test_ts_return_fail_in_setup
S fedora /bin/ksh       [./test/suite:808] test_ts_return_fail_in_teardown
S fedora /bin/ksh       [./test/suite:812] test_ts_test_detection
S fedora /bin/ksh       [./test/suite:816] test_undeclared_variable
S fedora /bin/ksh       [./test/suite:853] test_ts_background_job_example
S fedora /bin/ksh       [./test/suite:858] test_ts_readme_example
S fedora /bin/ksh       [./test/suite:872] test_ts_readme_troubleshoot_fail
S fedora /bin/ksh       [./test/suite:876] test_ts_readme_troubleshoot_pass
# opensuse /bin/bash    69 pass 0 fail 0 skip 1 s
# opensuse /bin/zsh     64 pass 5 fail 0 skip 4 s
F opensuse /bin/zsh     [./test/suite:213] test_ts_accepts_options_anywhere
F opensuse /bin/zsh     [./test/suite:223] test_script_accepts_options_anywhere
F opensuse /bin/zsh     [./test/suite:778] test_ts_options
F opensuse /bin/zsh     [./test/suite:816] test_undeclared_variable
F opensuse /bin/zsh     [./test/suite:829] test_ts_allows_exit_trap
# opensuse /bin/ksh     68 pass 1 fail 0 skip 1 s
F opensuse /bin/ksh     [./test/suite:816] test_undeclared_variable
# ubuntu /bin/dash      65 pass 4 fail 0 skip -1 s
F ubuntu /bin/dash      [./test/suite:201] test_ts_runs_cleanly_with_no_tests
F ubuntu /bin/dash      [./test/suite:783] test_include
F ubuntu /bin/dash      [./test/suite:816] test_undeclared_variable
F ubuntu /bin/dash      [./test/suite:862] test_ts_readme_common_tests
# ubuntu /bin/bash      69 pass 0 fail 0 skip 2 s
# ubuntu /bin/zsh       65 pass 4 fail 0 skip 4 s
F ubuntu /bin/zsh       [./test/suite:213] test_ts_accepts_options_anywhere
F ubuntu /bin/zsh       [./test/suite:223] test_script_accepts_options_anywhere
F ubuntu /bin/zsh       [./test/suite:778] test_ts_options
F ubuntu /bin/zsh       [./test/suite:816] test_undeclared_variable
# ubuntu /bin/ksh       68 pass 1 fail 0 skip 1 s
F ubuntu /bin/ksh       [./test/suite:816] test_undeclared_variable

Ubuntu failing tests seem to be the same reported in #22

[davidag]

By the way, is this Bakefile somehow related to the Bakefile project?

[thinkerbot]

Ha - no. I was thinking to mention something about that when I saw Bakefile in the issue title.

Story goes like this... I used to use a Makefile as a command entrypoint for historic reasons. I never really liked it. Earlier this year it dawned on me how to write a shell script that accomplished the same thing and I started calling it my 'bash-makefile' ie 'Bakefile'. The name worked well enough but then I found that Bakefile project.

As of now I've started using Projectfile. It's bland but unlikely to collide with anything. I hadn't gotten to changing this project but just pushed a commit to that effect.

[thinkerbot]

Oh one more thing -- I'm also looking into all the zsh issues. Tracks back to a fascinating weird difference in the way zsh getopts handles OPTIND differently than POSIX says it should.

• https://stackoverflow.com/a/48205341
• https://www.zsh.org/mla/users/2007/msg01191.html

I'm trying to work out a fix.

[thinkerbot]

I made some progress on this. Still a few outstanding failures that I need to get back to, but here is the current state (note I started omitting the skips in this output to better highlight failures):

./Projectfile test-all
# alpine /bin/bash	70 pass 0 fail 0 skip 3 s
# alpine /bin/zsh	70 pass 0 fail 0 skip 5 s
# centos /bin/bash	70 pass 0 fail 0 skip 5 s
# centos /bin/zsh	69 pass 1 fail 0 skip 7 s
F centos /bin/zsh	[./test/suite:830] test_ts_allows_exit_trap
# centos /bin/ksh	70 pass 0 fail 0 skip 3 s
# debian /bin/bash	70 pass 0 fail 0 skip 6 s
# debian /bin/zsh	70 pass 0 fail 0 skip 6 s
# debian /bin/ksh	69 pass 1 fail 0 skip 2 s
F debian /bin/ksh	[./test/suite:695] test_ts_uses_TS_DIFF_to_diff_outputs
# fedora /bin/bash	51 pass 0 fail 19 skip 3 s
# fedora /bin/zsh	51 pass 0 fail 19 skip 2 s
# fedora /bin/ksh	51 pass 0 fail 19 skip 2 s
# opensuse /bin/bash	70 pass 0 fail 0 skip 4 s
# opensuse /bin/zsh	69 pass 1 fail 0 skip 6 s
F opensuse /bin/zsh	[./test/suite:830] test_ts_allows_exit_trap
# opensuse /bin/ksh	70 pass 0 fail 0 skip 2 s
# ubuntu /bin/dash	68 pass 0 fail 2 skip -1 s
# ubuntu /bin/bash	70 pass 0 fail 0 skip 3 s
# ubuntu /bin/zsh	69 pass 1 fail 0 skip 6 s
F ubuntu /bin/zsh	[./test/suite:830] test_ts_allows_exit_trap
# ubuntu /bin/ksh	69 pass 1 fail 0 skip 2 s
F ubuntu /bin/ksh	[./test/suite:695] test_ts_uses_TS_DIFF_to_diff_outputs