curl: (56) Malformed encoding found in chunked-encoding

[eqyiel]

Some .org files fail to render in any browser when served by elnode. These files export without error, and the rendered html can be validated using http://validator.w3.org/. Only curl gives a clue about malformed encoding. It might just be that elnode is doing such a good job of being asynchronous that it tries to write data before it is available.

Tested on Emacs 24.4.1 with Org-mode 8.2.10 and elnode-0.9.9.8.8, also on Emacs 24.5.1 with Org-mode 8.2.10 and elnode-0.9.9.8.8 (same behaviour for both).

How to reproduce (from emacs -Q):

1. Install elnode somewhere. I temporarily moved my emacs.d and installed from marmalade.
2. Evaluate this snippet. ~/dotemacs/emacs.org is cloned from https://github.com/bzg/dotemacs (arbitrary large and public org file).

(require 'elnode)

(defun org2html (src)
  (let* ((org-inhibit-startup t)
         (visitingp (find-buffer-visiting src))
         (work-buffer (or visitingp (find-file-noselect src)))
         (html (with-current-buffer work-buffer
                 (org-export-as 'html nil t nil
                                '(:html-doctype "html5" :html-html5-fancy t)))))
    html))

(defun my-test-handler (httpcon)
  "Demonstration function"
  (let ((html (org2html "~/dotemacs/emacs.org")))
    (elnode-send-html httpcon html)))

(elnode-start 'my-test-handler :port 9999 :host "localhost")

Try to access in the terminal emulator:

~ $ curl localhost:9999
<!DOCTYPE html>
<html>
<head>
<title>bzg .emacs.el file</title>
--- SNIP ---
<p class="date">Created: 2015-07-13 Mon 11:42</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.4.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
curl: (56) Malformed encoding found in chunked-encoding
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate<~ $

You will see something like the following in *elnode-server-error* buffer:

2015-07-13T11:34:35: elnode-http-start: starting HTTP response on *elnode-webserver-localhost:9999* <127.0.0.1:42417>                                                                                          
2015-07-13T11:34:35: elnode-http-send-string *elnode-webserver-localhost:9999* <127.0.0.1:42417> [[<!DOCTY...]]                                                                                                
2015-07-13T11:34:35: elnode-http-send-string *elnode-webserver-localhost:9999* <127.0.0.1:42417> [[]]   
2015-07-13T11:34:35: elnode--http-end ending socket *elnode-webserver-localhost:9999* <127.0.0.1:42417>

I see the same behaviour using this, and some of my own files that I don't want to share just yet.

Expected behaviour: same for that of smaller org files. Here's an example ~/test.org that I used:

#+title: test.org                                                                                       
#+author: Ruben Maher                                                                                   
#+date: <2015-07-13 Mon 09:25:26>                                                                       

* This is a test to show off =elnode=.

  It's a pretty cool webserver.

Re-evaluate my-test-handler, changing the argument to org2html:

(defun my-test-handler (httpcon)
  "Demonstration function"
  (let ((html (org2html "~/test.org")))
    (elnode-send-html httpcon html)))

Try to access in a terminal emulator:

~ $ curl localhost:9999
<!DOCTYPE html>
<html>
<head>
<title>test.org</title>
--- SNIP ---
<p class="date">Created: 2015-07-13 Mon 11:48</p>
<p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.4.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>~ $

No problem. It renders correctly in a web browser. *elnode-server-error* now shows:

2015-07-13T11:48:59: elnode-http-start: starting HTTP response on *elnode-webserver-localhost:9999* <127.0.0.1:44466>
2015-07-13T11:48:59: elnode-http-send-string *elnode-webserver-localhost:9999* <127.0.0.1:44466> [[<!DOCTY...]]
2015-07-13T11:48:59: elnode-http-send-string *elnode-webserver-localhost:9999* <127.0.0.1:44466> [[]]
2015-07-13T11:48:59: elnode--http-end ending socket *elnode-webserver-localhost:9999* <127.0.0.1:44466>

[eqyiel]

I found the problem. It's a one-line fix so I'll just post it here.

In elnode-http-send-string, (format %x (length str)) doesn't always equal (format %x (string-bytes str). elnode tells the client to expect (string-bytes str), but the chunk size in the header is determined by (length str)!

I guess this is a bug in Emacs C source code. (Edit: this behaviour is actually documented. See here and here, "if the string contains multibyte characters, this is not necessarily
the number of bytes in the string; it is the number of characters.").

Demonstration, using this file

(defun org2html (src)
  (let* ((org-inhibit-startup t)
         (visitingp (find-buffer-visiting src))
         (work-buffer (or visitingp (find-file-noselect src)))
         (html (with-current-buffer work-buffer
                 (set-buffer-file-coding-system 'utf-8)
                 (org-export-as 'html nil t nil
                                '(:html-doctype "html5" :html-html5-fancy t))
                 )))
    html))

(let ((thing (org2html "~/dev/emacs-notes/videos.org")))
  (message (concat "(string-bytes thing) is %x, (length thing) is %x. "
                   "That's %d bytes shorter than it should be!")
           (string-bytes thing) (length thing)
           (- (string-bytes thing) (length thing))))

;; => "(string-bytes thing) is 5dc5, (length thing) is 5dc3. That's 2 bytes
;; shorter than it should be!"

TL;DR: (format "%x\r\n%s\r\n" (length str) (or str ""))) should be (format "%x\r\n%s\r\n" len (or str ""))) (len already calculated in let)

[joelmccracken]

Unicode characters will cause this problem.

On Tue, Jul 14, 2015 at 8:44 PM, Ruben Maher notifications@github.com
wrote:

I found the problem. It's a one-line fix so I'll just post it here.

In elnode-http-send-string
https://github.com/nicferrier/elnode/blob/master/elnode.el#L1783, (format
%x (length str)) doesn't always equal (format %x (string-bytes str).
elnode tells the client to expect (string-bytes str), but the chunk size
in the header is determined by (length str)!

I guess this is a bug in Emacs C source code.

Demonstration, using this file
https://github.com/sachac/emacs-notes/blob/gh-pages/videos.org

(defun org2html (src)
(let* ((org-inhibit-startup t)
(visitingp (find-buffer-visiting src))
(work-buffer (or visitingp (find-file-noselect src)))
(html (with-current-buffer work-buffer
(set-buffer-file-coding-system 'utf-8)
(org-export-as 'html nil t nil
'(:html-doctype "html5" :html-html5-fancy t))
)))
html))

(let ((thing (org2html "~/dev/emacs-notes/videos.org")))
(message (concat "(string-bytes thing) is %x, (length thing) is %x. "
"That's %d bytes shorter than it should be!")
(string-bytes thing) (length thing)
(- (string-bytes thing) (length thing))))

;; => "(string-bytes thing) is 5dc5, (length thing) is 5dc3. That's 2 bytes
;; shorter than it should be!"

TL;DR: (format "%x\r\n%s\r\n" (length str) (or str ""))) should be (format
"%x\r\n%s\r\n" len (or str ""))) (len already calculated in let)

—
Reply to this email directly or view it on GitHub
#98 (comment).

[nicferrier]

Right. Not sure what elnode should do. Automatically convert? (how would we send bytes?) or make it clear that it's not really a string... it's a string of bytes?

[eqyiel]

Unless I've completely misunderstood something (wouldn't be the first time), there's no conversion necessary? The problem is just that size of the chunk reported in the header (%x\r\n) is not always correct when it is retrieved with length (if the string contained unicode characters for instance). The chunk still contains a string and the function still sends a string, but elnode should figure out the chunk size with string-bytes instead of length.

[nicferrier]

I think that's option 2, accept that send-string is sending bytes, not strings.

Either way it needs to change, but it could be either way.

[joelmccracken]

I think it should send the number of bytes.

On Wed, Jul 15, 2015 at 10:29 AM, Nic Ferrier notifications@github.com
wrote:

I think that's option 2, accept that send-string is sending bytes, not
strings.

Either way it needs to change, but it could be either way.

—
Reply to this email directly or view it on GitHub
#98 (comment).

[jiri]

What's the status on this? I just ran into this issue in 2017.

[eqyiel]

@jiri It's been a long time, but IIRC you need to monkeypatch elnode-http-send-string changing (length str) to (string-bytes str) on this line: https://github.com/nicferrier/elnode/blob/master/elnode.el#L1783

Edit: or you can just use the len which is let-bound, see TLDR here: #98 (comment)

[jiri]

Thanks!

I've been looking into making a patch for this, but there already is a pull request. It would be nice if @nicferrier either accepted it or appointed a maintainer of the package. I think elnode is useful enough that it should be maintained :)

That being said, the issue seems to have disappeared as I have not been able to replicate it since I restarted Emacs. If it ever surfaces, I'll bump this issue again :)