http:send-request cookie handling enhancements

[apb2006]

Mail list discussion

Scripting access to web sites often requires sending cookies received in earlier responses. Currently this is hard to do, particularly when redirections are involved.

XProc 3.1 has features in it's p:http-request step that make this easier. Expressed as http:send-request features they could be summarised as:

• The http:send-request step automatically parses incoming Set-Cookie headers from responses and sends matching cookies back in subsequent requests within the same context or XQuery execution.

• You can control or disable automatic cookie tracking by setting a new attribute @suppress-cookies set to true or false on the <http:request> element.

Additionally p:http-request supplies @follow-redirects as an integer rather than boolean allowing greater control.

Implementations of this feature can be seen in:

1. The Kotlin source of XMLCalabash 3 here
2. The earlier cookie-jar extension to XML Calabash 1 in Java here

[ChristianGruen]

A query can now keep the cookies that a server sends (see https://docs.basex.org/13/HTTP_Client_Functions#cookies for details):

• A new request attribute cookies was introduced (false by default; we could possibly change that).
• If set to true, cookies from a response are stored and added to later requests.
• It also works with redirections.
• The stored cookies belong to one query. A new query starts with an empty store; nothing is shared between users.

Your feedback is welcome. I’ll try to provide a new snapshot later this day.

[ChristianGruen]

The snapshot is online.

[apb2006]

Sorry about the delay getting back to you after your super fast implementation. I am starting to look at it now.
Is it true that cookies are always stripped out of the http:send-request response when @cookies='true'?
If this is true - It seems unnecessary and makes debugging harder?

[ChristianGruen]

Hi Andy, nothing is supposed to be removed on our side. All response headers are passed on unchanged; @cookies only attaches a cookie store to the HTTP client:

• If redirects are followed (the default), only the last response is returned. The Set-Cookie of the redirecting response is not part of the result (with or without @cookies).
• On a follow-up request, the cookies are already sent by the client, so servers may not repeat their Set-Cookie headers. Fewer headers arrive, but nothing was dropped in BaseX.

For debugging, you can try follow-redirect='false'.