Skip to content

Encode the final newline with converted text output - #108

Open
x0Lazarus wants to merge 1 commit into
SigmaHQ:mainfrom
x0Lazarus:fix/encode-output-newline
Open

x0Lazarus wants to merge 1 commit into
SigmaHQ:mainfrom
x0Lazarus:fix/encode-output-newline

Conversation

@x0Lazarus

Copy link
Copy Markdown
Contributor

Converting with --encoding utf-16 currently succeeds but produces a file that cannot be decoded as UTF-16. The CLI encodes the result, then click.echo appends a raw one-byte newline. UTF-32 output has the same problem.

Include the final newline in the text before encoding it, and tell Click not to append another one. This keeps the existing separators and UTF-8 output while producing valid multibyte text, including a single initial BOM where the encoding requires one. The change covers string and JSON results sent to a file or stdout. Binary results and separate-file output are unchanged.

This also touches the dictionary output branch in #102, but leaves its separate -o fix to that PR.

Validation on Windows, Python 3.12:

  • All 134 output tests pass with current pySigma and the minimum supported 1.3.0. On unchanged CLI code, 94 fail and 40 controls pass. Tests compare exact bytes for file/stdout, empty values, Unicode, JSON indentation and UTF-16/32 variants.
  • The broader offline run has 234 passes and one existing skip. Twelve dataset-fetch tests fail with networking disabled on both the original and patched code; nine online plugin test functions (12 cases) were not run.
  • Black 23.3 formats the new tests and changed output calls. The complete convert.py still has pre-existing formatting differences on both revisions; unrelated formatting is left out of this patch.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant