Feedback request: proposal to drop Rdiscount and Redcarpet support #179
19 participants
I am using the default markdown engine for my blog so I won't miss them.
Anyway, what is the approximative upgrade date ?
The most obvious problem here is backticks. Kramdown does not support the officially sanctioned Github way of doing things with backticks and this has come up a few times in our own tickets when users select Kramdown and the decision was made to not explicitly work around the lack of flexibility in processors so we never added support for them.
FYI: You can use:
kramdown:
input: GFM
To add support for backticks. Might be a great default (In my humble opinion - I also prefer backticks for code blocks). Cheers.
Also consider that there are private repos also running blogs via GitHub pages, like my own. The percentages are probably fairly similar, but still.
I tried switching over to Kramdown (with GFM as above), but:
- Haven't gotten syntax-highlighting working at all on GitHub Pages itself. I've tried Coderay, Rouge and Pygments (what I used with Redcarpet), and have come up with nothing.
- Locally, Pygments doesn't seem to work with Kramdown, and Coderay doesn't seem to know how to detect language based on whatever I give after the start of the code-fence; the highlighting looks rubbish with YAML, for example.
I've seen some mention of a blessed "Pretty Lights" highlighter that's not been released. I'm a bit in the dark here. What highlighter are we supposed to be using with Kramdown?
I prefer RedCarpet for Markdown syntax. In case it's deprecated is there a way I can configure Kramdown so I don't have to go and change all my posts's syntax? I don't want to wake up one day and see my site does not render as expected.
The numbers as of 2015-11-28 may be slightly different:
- All public sites build using Jekyll 419,532 (100%)
- Public sites explicitly using Redcarpet 23,759 (~5.66%)
- Public sites explicitly using Rdiscount 18,337 (~4.37%)
We are also using kramdown on our sites, so this won't hurt us so much.
But other almost 50K (10%) users will get definitely a bad experience with this stack.
IMO this support shouldn't be simply switched off but the users should rather be notified in advance.
Best,
Haven't gotten syntax-highlighting working at all on GitHub Pages itself... Locally, Pygments doesn't seem to work with Kramdown
I can confirm syntax highlighting and Pygments both work Kramdown (and are actually handled at the Jekyll level, via the highlight tag).
I prefer RedCarpet for Markdown syntax.
@jeremenichelli can you provide some specific examples of where Kramdown doesn't support a syntax that RedCarpet does?
All public sites build using Jekyll 419,532 (100%)
@r-brown to note, a site doesn't need a _config.yml file to be a Jekyll site (the number of Jekyll sites is roughly double that, by my records).
One last call, if there are any specific examples of features or syntaxes that RedCarpet or Rdiscount supports that Kramdown does not (rather than personal experience or preference), it'd go a long way to help inform our decision.
IMO this support shouldn't be simply switched off but the users should rather be notified in advance.
We still need to discuss things internally from a product/support perspective, but right now I'm hoping that when we move to Jekyll 3.0.x we can also announce a six-month sunset for Rdiscount and Redcarpet support (with warnings on push, similar to what we did with Maruku).
I would be sad!
Kramdown does not support the officially sanctioned Github way of doing things
I use Pages to host the site and documentation for atom/electron. This site uses the markdown documentation as it is in the GitHub repository. The problem is that GitHub.com renders markdown different than Pages. Redcarpet is the most similar, though it's still does not have feature parity.
If Pages switches to Kramdown I'm guessing none of the code blocks in the Electron documentation will render correctly on the site? I think the ultimate (and kind of obvious) goal should be to have markdown on Pages and GitHub.com be the same.
The problem is that GitHub.com renders markdown different than Pages.
@jlord as @geraldb suggested above, have you tested putting Kramdown into GitHub-flavored Markdown mode?
kramdown:
input: GFMEdit: Kramdown-GFM docs
I can confirm syntax highlighting and Pygments both work Kramdown (and are actually handled at the Jekyll level, via the highlight tag).
From a "imma write some Markdown now" perspective, the highlight tag never made much sense to me, since you have to leave Markdown to use it. (Although the fenced code syntax we chose itself was a misstep too, in hindsight, but I digress). +1 for definitely using the GFM setting as a default in Jekyll; the interface should just be Markdown, and Markdown's flavor differences are confusing enough. GitHub should at least put a consistent foot forward.
Let's flip the question around a little bit, if we can.
From a product/support perspective, what (if any) additional burden does supporting Redcarpet/Rdiscount cause Pages? I understand that it's not for free, but what harm are we causing by not removing it?
I would think there would be less burden on their build servers by allowing the C equivs.
FYI: I've put together a simple test page for kramdown using the gfm mode (see testgfm.html and the source testgfm.md using these settings:
markdown: kramdown
kramdown:
input: GFM
hard_wrap: false
It works as expected (if you turn off the hard_wrap) :-)
have you tested putting Kramdown into GitHub-flavored Markdown mode?
@benbalter Hmm, I feel like if that had worked I wouldn't be in the situation I am now with the site and wouldn't have had so much back and forth with you over the past few months re: markdown/Pages. But it has been months now and memory is fuzzy so I will give it a go again. If there is true parity between GitHub.com and this then
@geraldb mind filing a ticket on Jekyll with those settings so I can add them to the Kramdown rewrite as default options? I'll forget by the time I get about to working on it again but I think those would be sane defaults for Jekyll
@envygeeks Sorry I'm not at GitHub issue expert (do not know how to reference the ticket). Find it at jekyll/issues/4202. Cheers. PS: Thanks for your great work. Keep it up.
I've tried this configuration locally:
highlighter: pygments
markdown: kramdown
kramdown:
input: GFM
hard_wrap: falseAnd the result is that it works good except that inside the pre code tags I don't have the tokenization I had previously identifying comments for example so I can style them.
Yeah Kramdown doesn't support Pygments, it only supports Rouge and Coderay because both of those are pure Ruby and it's a pure Ruby Markdown processor.
If there's no native alternative for highlighting I'm against removing all Markdown engines except for Kramdown since you're removing a feature from Jekyll.
@jeremenichelli well... technically Pygments isn't a feature of Jekyll anymore, in 3.0 we switched to Rouge ourselves as the default processor, Pygments is an optional now.
@envygeeks well, but it's available, that's what I meant. My point is that I'm using Jekyll locally and I'm able to use highlighting, GitHub Pages at least would need to offer an alternative if Kramdown will be the only Markdown engine present.
Disclaimer: I'm pretty new to Jekyll (and web development in general), but right now the biggest inconsistencies for me while I write the content for my new blog on GH are codeblocks (eg, backticks, including a filename in addition to using GFM, Pygments Vs. Rouge).
My 2¢: Most of this has already been mentioned
- Settle on a single markdown converter (I've only used Redcarpet, but I'm willing to use Kramdown if that's the only option) and a single highlighter.
- Set GFM as default for Kramdown (if this is the final choice). This is GH pages, after all. I'd love the ability to include a filename in addition to syntax after the triple backticks, but I know this is a nice-to-have and I don't think it's part of standard GFM any way.
- For syntax highlighting, whatever produces HTML that uses the same classes as pygments. I think this is fair since the library of CSS options for different languages seems far more extensive with pygments, and my attempts to use rouge required I add highlight.js, which I'm trying to avoid if possible.
-
Concerns:
- Pagination? I'm not currently using the previous plugin, but this seems like it might be a bigger breaking issue for a lot of sites, no?
- Pretty Lights? I've heard talk of this as well. Will the syntax be identical to that of Rouge? Pygments? It'd be a bummer to have all new sites move to Rouge if they'll eventually need to be replaced by a native GH syntax highlighter.
If there's no native alternative for highlighting I'm against removing all Markdown engines except for Kramdown since you're removing a feature from Jekyll.
@envygeeks @jeremenichelli If I'm understanding correctly, Kramdown, in GFM mode, with Rouge should support the desired behavior (syntax highlighting via backticks).
Set GFM as default for Kramdown... +1 for definitely using the GFM setting as a default in Jekyll.. From a "imma write some Markdown now" perspective, the highlight tag never made much sense to me, since you have to leave Markdown to use it.
As always have to agree with @holman.
GFM for Kramdown would be a nice, sane default, but I'd rather this be done in Jekyll-core, than specific to Pages, where a user using the default configuration locally would have a different experience than the default experience when generated via GitHub Pages (we currently don't have a way to set defaults that differ from Jekyll's defaults locally).
If pygments should be a supported syntax highlighter in kramdown, it can easily be implemented since syntax highlighting is more or less plug-and-play. To start have a look at one of the current implementations https://github.com/gettalong/kramdown/blob/master/lib/kramdown/converter/syntax_highlighter/rouge.rb
Regarding GFM: Since there is no spec for GFM, the implementation is currently just guess work.
I prefer build on local, and push static files to GitHub Pages, that I will not be affected by the upgrade.
@benbalter Rouge uses different class names AFAIW which would mean everybody who currently uses Pygments would need to port their CSS, however... given it's Kramdown, you could probably just build a custom syntax highlighter for it, and I'd be happy to build it and ship a PR to do it if you like... out of all the things that could go wrong, Pygments vs. Rouge is probably the least relevant since it's the one that can be worked around the easiest.
Yeap, I think that the most critical thing here is not silently break other people styles because of this hiighlighting changes, if class names sets can be configured/maintain, then I'm ok with this.
Rouge uses different class names AFAIW which would mean everybody who currently uses Pygments would need to port their CSS
From the Rouge Readme:
The HTML output from Rouge is fully compatible with stylesheets designed for pygments.
@jeremenichelli disable pygments in your _config.yml, do what @geraldb did and see how your site looks....
The HTML output from Rouge is fully compatible with stylesheets designed for pygments.
Well. I guess I have no more to worry about. Everything I was concerned about has been addressed.
FYI: There's an (old) article titled "Syntax Highlighting in Jekyll With Rouge" for details on getting Jekyll with Rouge and Backticks (GFM) work / setup that might be of interest.
Also if I may state the obvious @gettalong (see posting above) is the author of kramdown - the best source to ask ;-) (As always - thanks for the great library. Keep it up.)
Update Found a better (newer) article using Jekyll 3.0 titled "Upgrading to Jekyll 3.0" by Jonas Kersulis. See the section titled "Pygments to Rouge" and "Redcarpet to Kramdown".
@envygeeks using last version of Jekyll with this config:
markdown: kramdown
kramdown:
input: GFM
syntax_highlighter: rougeAnd I get no highlighting, just plain text inside the code tags. Maybe I have to install Rouge locally, but then that would be this is still not built inside Jekyll, right?
@jeremenichelli If you changed the _config have you tried to restart the server? For more tips, see How to install Rogue in Jekyll. Cheers.
@geraldb yes, I restarted the server. Though I did not install Rouge since I thought it was already supported in Jekyll. If that's not the case then my suggestion would be to propose Rouge to be present in Jekyll as Kramdown is, and then GitHub Pages can shut down other makrdown engines and leave Kramdown only, at least that's my humble opinion around this.
@jeremenichelli If you want to test kramdown with rouge, you can do the following (from a POSIX-Shell):
$ echo -e "~~~ ruby\nclass New\nend\n~~~" | kramdown --syntax-highlighter rougeOutput with rouge:
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="nc">New</span>
<span class="k">end</span>
</code></pre>
</div>If this works, then kramdown with rouge works. So if the output with Jekyll is not correct, then there is probably a problem with Jekyll.
Also note that if the rouge library is not installed, kramdown will fall back to not doing any highlighting. It doesn't currently emit a warning in such a case but this can be changed if needed.
Fallback output when rouge is not available:
<pre><code class="language-ruby">class New
end
</code></pre>
@jeremenichelli Also note that you need kramdown >= 1.5.0 for rouge support.
@jeremenichelli Yes, you need to have rouge installed locally if you build locally.
As for Github Pages, if the information on https://pages.github.com/versions/ is correct, kramdown is at version 1.5.0 but they don't have rouge installed. So it seems that Github Pages currently doesn't support rouge?
Well, this is already a bad sign, I imagine this can cause some fragmentation between Jekyll and GitHub Pages and that's not good in my opinion. I prefer features to be aligned across them.
it seems that Github Pages currently doesn't support rouge?
That would potentially change. We would go the pure-Ruby route of kramdown + Rouge.
@parkr if that happens, then, no fragmentation across GitHubPages gem and highlighting engines. GitHub staff can potentially let everyone know in advance and that's a thumbs up to me.
Sounds like we have a path forward. Thanks everyone for the feedback. The additional context is extremely helpful. Stay tuned for more information about GitHub Pages upgrading to Jekyll 3.0.x.!
@benbalter @gettalong I'd slow down on that "path forward", today we tried to convert our largest site to Kramdown and ran into a problem, GFM doesn't actually enable fenced codeblocks the way they work on Github, ```sh does not behave like ~~~sh actually we get no highlighting at all with the former.
BTW this is on Jekyll 3.1.alpha through: https://github.com/jekyll/jekyll/blob/master/lib/jekyll/configuration.rb#L71-L72
I don't know if this is intended but it's certainly how it's behaving for us on Kramdown 1.9, for us it's not a big deal, we don't process with Github so we worked around it by pre-parsing our source before Jekyll get's to it, but it might affect others
@envygeeks I get the following results when running on the command line (just kramdown):
Input:
```sh
bash test
```
Output with kramdown -i GFM:
<div class="highlighter-coderay"><div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>bash test
</pre></div>
</div>
</div>Output with kramdown -i GFM --syntax-highlighter rouge:
<div class="highlighter-rouge"><pre class="highlight"><code>bash <span class="nb">test</span>
</code></pre>
</div>Seems about right to me.
@gettalong Given this snippet (taken from https://github.com/jekyll/docker):
## Running
***If you do not provide a command then it will default to `jekyll s`.***
### Native Docker
```sh
# Switch to 80:80 or 4000:80 if you wish to use only Nginx with `jekyll build`
docker run --rm --label=jekyll --volume=$(pwd):/srv/jekyll \
-it -p 127.0.0.1:4000:4000 jekyll/jekyll
```
### Docker-Machine
~~~sh
# Switch to 80:80 or 4000:80 if you wish to use only Nginx with `jekyll build`
docker run --rm --label=jekyll --volume=$(pwd):/srv/jekyll \
-it -p $(docker-machine ip `docker-machine active`):4000:4000 \
jekyll/jekyll
~~~
Inside of "index.md" with a _config.yml that states markdown: kramdown with the default options (of which I linked to earlier) in 3.1.alpha on our builder server I get the following results:
<p><code class="highlighter-rouge">sh
# Switch to 80:80 or 4000:80 if you wish to use only Nginx with `jekyll build`
docker run --rm --label=jekyll --volume=$(pwd):/srv/jekyll \
-it -p 127.0.0.1:4000:4000 jekyll/jekyll
</code></p><div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># Switch to 80:80 or 4000:80 if you wish to use only Nginx with `jekyll build`</span>
docker run --rm --label<span class="o">=</span>jekyll --volume<span class="o">=</span><span class="k">$(</span><span class="nb">pwd</span><span class="k">)</span>:/srv/jekyll <span class="se">\</span>
-it -p <span class="k">$(</span>docker-machine ip <span class="sb">`</span>docker-machine active<span class="sb">`</span><span class="k">)</span>:4000:4000 <span class="se">\</span>
jekyll/jekyll
</code></pre>
</div>Only the second one (the ~~~) is properly highlighted.
On the command line I get the following output using kramdown -i GFM --syntax-highlighter rouge --no-hard-wrap:
<h2 id="running">Running</h2>
<p><strong><em>If you do not provide a command then it will default to <code class="highlighter-rouge">jekyll s</code>.</em></strong></p>
<h3 id="native-docker">Native Docker</h3>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># Switch to 80:80 or 4000:80 if you wish to use only Nginx with `jekyll build`</span>
docker run --rm --label<span class="o">=</span>jekyll --volume<span class="o">=</span><span class="k">$(</span><span class="nb">pwd</span><span class="k">)</span>:/srv/jekyll <span class="se">\</span>
-it -p 127.0.0.1:4000:4000 jekyll/jekyll
</code></pre>
</div>
<h3 id="docker-machine">Docker-Machine</h3>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># Switch to 80:80 or 4000:80 if you wish to use only Nginx with `jekyll build`</span>
docker run --rm --label<span class="o">=</span>jekyll --volume<span class="o">=</span><span class="k">$(</span><span class="nb">pwd</span><span class="k">)</span>:/srv/jekyll <span class="se">\</span>
-it -p <span class="k">$(</span>docker-machine ip <span class="sb">`</span>docker-machine active<span class="sb">`</span><span class="k">)</span>:4000:4000 <span class="se">\</span>
jekyll/jekyll
</code></pre>
</div>So kramdown seems to do the correct thing in both cases. Maybe you can find out what is actually passed to kramdown as input?
@gettalong What is the Ruby code that the CLI produces? Jekyll uses the kramdown Ruby API, and it may be missing a configuration option or two that the kramdown CLI sets.
@gettalong @parkr I get the same broken result when running the command shown above with the source file, this is on Ubuntu 14.04 x64 Ruby 2.2.3p137, if you need more information:
ruby -r rbconfig -e 'puts RbConfig::CONFIG["configure_args"]'
'--disable-install-doc' '--enable-shared' '--prefix=/usr/local'
ii libffi-dev:amd64 3.1~rc1+r3.0.13-12ubuntu0.1
ii libreadline6-dev:amd64 6.3-4ubuntu2
ii libssl-dev:amd64 1.0.1f-1ubuntu2.16
ii libxml2-dev:amd64 2.9.1+dfsg1-3ubuntu4.5
ii libxslt1-dev:amd64 1.1.28-2build1
ii libyaml-dev:amd64 0.1.4-3ubuntu3.1
The result also happens on Alpine Linux 3.3 x64
@gettalong @parkr after I finish working on this Kramdown+Rouge segfault problem that keeps happening I'll jump into Pry and hit up our Kramdown processor and see exactly what content is passed, we did recently update it but... if it was a problem there I wouldn't get the same results when hitting Kramdown without Jekyll.
@parkr See https://github.com/gettalong/kramdown/blob/master/bin/kramdown#L59, nothing really special.
@envygeeks I'm using kramdown 1.9.0 and ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-linux] on Ubuntu 15.04.
@envygeeks Maybe you could try the following in an irb console:
require 'kramdown/document'
Kramdown::Document.new(File.read('/tmp/README.md'), input: :GFM, hard_wrap: false).to_html;
$".grep(/kramdown/).sort
The last line outputs all loaded kramdown source files. May this shows that some files are loaded from a wrong directory or something.
@parkr @gettalong I have feeling this is related to the bug in Rouge I'm currently working out, because it started working suddenly and then stopped working again, and stopped working again and then started again. At this point I'm going to assume they are related.
For those interested, I posted some additional context behind our decision in this blog post.
As we're preparing to upgrade to Jekyll 3.0.x, we're looking for ways to make things even easier for new users to get started. One point of unnecessary cognitive burden is choosing a markdown interpreter.
Kramdown? Rdiscount? Redcarpet? What's the difference? How do I choose? Why can't you just pick the best for me? When I fire up WordPress or Microsoft Word or my phone, I don't choose the rendering engine. I trust the publisher to make that choice for me. Using Jekyll should be no different.
There used to be a lot of difference between the various Markdown rendering engines. Some supported some features, others implemented different features in different ways. Today, they've largely converged on a standard feature set, with Kramdown implementing most, if not all features (maybe not options), across all renderers.
My question
Other than speed, is there anything Rdiscount or Redcarpet does that Kramdown doesn't? Would you miss them terribly, or would your site fail to render if GitHub Pages dropped support for any markdown renderer other than Kramdown?
If there are any specific examples of features or syntaxes that RedCarpet or Rdiscount supports that Kramdown does not (rather than personal experience or preference), it'd go a long way to help inform our decision.
Rdiscount and Redcarpet breakdown:
As a baseline:
For open source projects I normally like to optimize for the 80% use case (not edge cases or power users), but here this means that 95%+ of users are using Kramdown (and 90%+ don't care enough to specify). For these numbers, you can assume 1M+ public sites.
To note, this proposed change would only affect GitHub Pages. Users would be free to use any rendering engine they'd like when running Jekyll locally or on their own server.