Improve HTML support - #153
Open
nikilase wants to merge 2 commits into
Open
Conversation
- Add <HTML to the magic_data headers for correct HTML file recognition - Limit CSV sniffer sample size to prevent slow detection. Truncate text to 8192 characters and use the first 50 lines when sniffing CSV dialects. -Add HTML XLS detection and performance test
There was a problem hiding this comment.
Pull request overview
This PR addresses #152 by improving HTML recognition (including Excel-generated HTML saved as .xls) and preventing pathological runtimes in CSV detection by limiting the amount of text passed to csv.Sniffer.
Changes:
- Add an uppercase
<HTMLmagic header entry so HTML files starting with<HTMLare detected astext/html. - Limit the CSV sniffer input to the first 50 lines / 8192 characters to avoid extremely slow
csv.Sniffer().sniff(...)calls. - Add a regression test for HTML
.xlsdetection (and a performance-oriented assertion, currently time-based).
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
test/test_common_extensions.py |
Adds a test for HTML-in-.xls detection and a wall-clock performance assertion. |
puremagic/scanners/text_scanner.py |
Caps the CSV sniffer sample size to prevent long runtimes. |
puremagic/magic_data.json |
Adds <HTML header signature to improve HTML detection for uppercase tags. |
Suppressed comments (1)
test/test_common_extensions.py:300
- This test currently (1) has a mismatched docstring/name (
test_html_xls_msg/ Outlook .msg), and (2) asserts a wall-clock threshold (elapsed < 1) which can be flaky across CI environments. Prefer a functional MIME assertion, and add a deterministic regression test thatcsv.Snifferis only given a bounded sample (<=8192 chars / <=50 lines) to prevent pathological runtimes.
def test_html_xls_msg():
"""CFBF scanner correctly identifies Outlook .msg"""
start = time.perf_counter()
mime = puremagic.from_file(os.path.join(OFFICE_DIR, "test_html.xls"), mime=True)
elapsed = time.perf_counter() - start
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add <HTML to the magic_data headers for correct HTML file recognition
Limit CSV sniffer sample size to prevent slow detection. Truncate text to 8192 characters and use the first 50 lines when sniffing CSV dialects.
-Add HTML XLS detection and performance test
Fixes #152