New pandoc 3.8.3 breaks '<sup>' syntax

[cderv]

Found by looking at new Quarto website after 1.9.17 release

This is a change in Pandoc 3.8.3 we missed. Here is the comparaison between 3.6.3 in Quarto 1.8 and 3.8.3 in new 1.9

text <- "# Welcome to Quarto^[®]{.trademark}^ {.mt-1}"
pandoc::pandoc_convert(text = text, to = "html", version = "3.6.3")
#> <h1 class="mt-1" id="welcome-to-quarto">Welcome to Quarto<sup><span
#> class="trademark">®</span></sup></h1>
pandoc::pandoc_convert(text = text, to = "html", version = "3.8.3")
#> <h1 class="mt-1" id="welcome-to-quarto.trademark">Welcome to Quarto<a
#> href="#fn1" class="footnote-ref" id="fnref1"
#> role="doc-noteref"><sup>1</sup></a>{.trademark}^</h1>
#> <section id="footnotes" class="footnotes footnotes-end-of-document"
#> role="doc-endnotes">
#> <hr />
#> <ol>
#> <li id="fn1"><p>®<a href="#fnref1" class="footnote-back"
#> role="doc-backlink">↩︎</a></p></li>
#> </ol>
#> </section>

So we need document this change and maybe find a way to avoid breakage for users ?

[cderv]

Last working version is 3.8, so change happened in 3.8.1

From release note:

Improve superscript/subscript/inline note parsing (#8652). We do not allow inline notes to be followed by ( or [. Otherwise, we parse inline notes before superscripts. Also, the sub/superscript parsers have been adjusted so that they really exclude unescaped spaces (as they did not before, when the spaces occurred in nested inlines).

to solve

• Markdown footnotes surrounding a word result in superscript jgm/pandoc#8652

commit is jgm/pandoc@cf760c7

Before, superscript had precedence in the syntax for ^ but now it seems they reverted. 🤔
So same syntax but breaking change !! 🤦‍♂️

pandoc::pandoc_convert(text = "^[®]{.trademark}^", to = "native", version = "3.8")
#> [ Para
#>     [ Superscript
#>         [ Span ( "" , [ "trademark" ] , [] ) [ Str "\174" ] ]
#>     ]
#> ]
pandoc::pandoc_convert(text = "^[®]{.trademark}^", to = "native", version = "3.8.1")
#> [ Para
#>     [ Note [ Para [ Str "\174" ] ] , Str "{.trademark}^" ]
#> ]

[cderv]

I think this is bug in pandoc - I opened jgm/pandoc#11409 - we'll see.

However, if we don't want the breaking change to impact our users, we'll need to find a custom reader trick... 🤔

@cscheid I guess new parser behave like old pandoc ?

[cderv]

The trick right now is to use a zero width space to avoid Pandoc parsing to read as Footnote

Done in quarto-dev/quarto-web@0c79ea8 and quarto-dev/quarto-web@aaf1ba1

❯ pandoc -t native -f markdown
text^​[sup]{.class}^
^Z
[ Para
    [ Str "text"
    , Superscript
        [ Str "\8203"
        , Span ( "" , [ "class" ] , [] ) [ Str "sup" ]
        ]
    ]
]

❯ pandoc -t html -f markdown  
text^​[sup]{.class}^
^Z
<p>text<sup>​<span class="class">sup</span></sup></p>

[cderv]

So after discussion, we'll live with the change in Pandoc. Trick will be to not use ^[..] syntax anbd so add a character after ^

Maybe special case will be done for jgm/pandoc#11409

I'll add a not in release note about this breaking change.