Learning Crystal by Implementing a Static Site Generator
What?
A while back (10 YEARS???? WTH.) I wrote a static site generator. I mean, I wrote one that is large and somewhat popular, called Nikola but I also wrote a tiny one called Nicoletta
Why? Because it's a nice little project and it shows the very basics of how to do a whole project.
All it does is:
- Find markdown files
- Build them
- Use templates to generate HTML files
- Put those in an output folder
And that's it, that's a SSG.
So, if I wanted a "toy" project to practice new (to me) programming languages, why not rewrite that?
And why not write about how it goes while I do it?
Hence this.
So, what's Crystal?
It's (they say) "A language for humans and computers". In short: a compiled, statically typed language with a ruby flavoured syntax.
And why? Again, why not?
Getting started
I installed it using curl and that got me version 1.8.2 which is the latest at the time of writing this.
You can get your project started by running a command:
nicoletta/crystal ✦ > crystal init app nicoletta . create /home/ralsina/zig/nicoletta/crystal/.gitignore create /home/ralsina/zig/nicoletta/crystal/.editorconfig create /home/ralsina/zig/nicoletta/crystal/LICENSE create /home/ralsina/zig/nicoletta/crystal/README.md create /home/ralsina/zig/nicoletta/crystal/shard.yml create /home/ralsina/zig/nicoletta/crystal/src/nicoletta.cr create /home/ralsina/zig/nicoletta/crystal/spec/spec_helper.cr create /home/ralsina/zig/nicoletta/crystal/spec/nicoletta_spec.cr Initialized empty Git repository in /home/ralsina/zig/nicoletta/crystal/.git/
Some maybe interesting bits:
- It inits a git repo, with a gitignore in it
- Sets you up with a MIT license
- Creates a reasonable README with nice placeholders
- We get a
shard.yml
with metadata - Source code in
src/
-
spec/
seems to be for tests?
Mind you, I still have zero idea about the language :-)
This apparently compiles into a do-nothing program, which is ok. Surprisied to see starship seems to support crystal in the prompt!
crystal on main [?] is 📦 v0.1.0 via 🔮 v1.8.2 > crystal build src/nicoletta.cr crystal on main [?] is 📦 v0.1.0 via 🔮 v1.8.2 > ls -l total 1748 -rw-rw-r-- 1 ralsina ralsina 2085 may 31 18:15 journal.md -rw-r--r-- 1 ralsina ralsina 1098 may 31 18:08 LICENSE -rwxrwxr-x 1 ralsina ralsina 1762896 may 31 18:15 nicoletta* -rw-r--r-- 1 ralsina ralsina 604 may 31 18:08 README.md -rw-r--r-- 1 ralsina ralsina 167 may 31 18:08 shard.yml drwxrwxr-x 2 ralsina ralsina 4096 may 31 18:08 spec/ drwxrwxr-x 2 ralsina ralsina 4096 may 31 18:08 src/
Perhaps a bit surprising that the do-nothing binary is 1.7MB tho (1.2MB stripped) but it's just 380KB in "release mode" which is nice.
Learning a Bit of Crystal
At this point I will stop and learn some syntax:
- How to declare a variable / a literal / a constant
- How to do an if / loop
- How to define / call a function
Because you know, one has to know at least that much 😁
There seems to be a decent set of tutorials at this level. let's see how it looks.
Good thing: this is valid Crystal:
module Nicoletta VERSION = "0.1.0" 😀 = "Hello world" puts 😀 end
Also nice that variables can change type.
Having the docs say integers are int32
and anything else is "for special use cases" is not great. int32
is small.
Also not a huge fan of separate unsigned types.
I hate the "spaceship operator" <==>
which "compares its operands and returns a value that is either zero (both operands are equal), a positive value (the first operand is bigger), or a negative value (the second operand is bigger)" ... hate it.
Numbers have named methods, which is nice. However it randomly shows some weird syntax that has not been seen before. One of these is not like the others:
p! -5.abs, # absolute value 4.3.round, # round to nearest integer 5.even?, # odd/even check 10.gcd(16) # greatest common divisor
Or maybe the ?
is just part of the method name? Who knows! Not me!
Nice string interpolation thingie.
name = "Crystal" puts "Hello #{name}"
Why would anyone add an underscore
method to strings? That's just weird.
Slices are reasonable, whatever[x..y]
uses negative indexes for "from the right".
We have truthy values, 0 is truthy, only nil, false and null pointers are falsy. Ok.
I strongly dislike using unless
as a keyword instead of if
with a negated condition. I consider that to be keyword proliferation and cutesy.
Methods support overloading. Ok.
Ok, I know just enough Crystal to be slightly dangerous. Those feel like good tutorials. Short, to the point, give you enough rope to ... make something with rope, or whatever.
Learning a Bit More Crystal
So: errors? Classes? Blocks? How?
Classes are pretty straightforward ... apparently they are a bit frowned upon for performance reasons because they are heap allocated, but whatevs.
Inheritance with method overloading is not my cup of tea but 🤷
Exceptions are pretty simple but begin / rescue / else / ensure / end
? Eek.
Also, I find that variables have nil
type in the ensure
block confusing.
Requiring files is not going to be a problem.
Blocks are interesting but I am not going to try to use them yet.
Dinner Break
I will grab dinner, and then try to implement Nicoletta, somehow. I'll probably fail 😅
Implementing Nicoletta
The code for nicoletta is not long so this should be a piece of cake.
No need to have a main
in Crystal. Things just are executed.
First, I need a way to read the configuration. It looks like this:
TITLE: "Nicoletta Test Blog"
That is technically YAML so surely there is a crystal thing to read it. In fact, it's in the standard library! This fragment works:
require "yaml" VERSION = "0.1.0" tpl_data = File.open("conf") do |file| YAML.parse(file) end p! tpl_data
And when executed does this, which is correct:
crystal on main [!?] is 📦 v0.1.0 via 🔮 v1.8.2 > crystal run src/nicoletta.cr tpl_data # => {"TITLE" => "Nicoletta Test Blog"}
Looks like what I want to store this sort of data is a Hash
Next step: read templates and put them in a hash indexed by path.
Templates are files in templates/
which look like this:
<h2><a href="${link}">${title}</a></h2> date: ${date} <hr> ${text}
Of course the syntax will probably have to change, but for now I don't care.
To find all files in templates
I can apparently use Dir.glob
And I swear I wrote this almost in the first attempt:
# Load templates templates = {} of String => String Dir.glob("templates/*.tmpl").each do |path| templates[path] = File.read(path) end
Next is iterating over all files in posts/
(which are meant to be markdown with YAML metadata on top) and do things with them.
Iterating them is the same as before (hey, this is nice)
Dir.glob("posts/*.md").each do |path| # Stuff end
But I will need a Post
class and so on, so...
Here is a Post
class that is initialized by a path, parses metadata and keeps the text.
class Post def initialize(path) contents = File.read(path) metadata, @text = contents.split("\n\n", 2) @metadata = YAML.parse(metadata) end @metadata : YAML::Any @text : String end
Next step is to give that class a method to parse the markdown and convert it to HTML.
I am not implementing that so I googled for a Crystal markdown implementation and found markd which is sadly abandoned but looks ok.
Using it is surprisingly painless thanks to Crystal's shards dependency manager. First, I added it to shard.yml
:
dependencies: markd: github: icyleaf/markd
Ran shards install
:
crystal on main [!+?] is 📦 v0.1.0 via 🔮 v1.8.2 > shards install Resolving dependencies Fetching https://github.com/icyleaf/markd.git Installing markd (0.5.0) Writing shard.lock
Then added a require "markd"
, slapped this code in the Post
class and that's it:
def html Markd.to_html(@text) end
Here is the code to parse all the posts and hold them in an array:
posts = [] of Post Dir.glob("posts/*.md").each do |path| posts << Post.new(path) end
Now I need a Crystal implementation of some template language, something like handlebars, I don't need much!
The standard library has a template language called ECR which is pretty nice but it's compile-time and I need this to be done in runtime. So googled and found ... Kilt
I will use the crustache variant, which implements the Mustache standard.
Again, added the dependency to shard.yml
and ran shards install
:
dependencies: markd: github: icyleaf/markd crustache: github: MakeNowJust/crustache
After some refactoring of template code, the template loader now looks like this:
class Template @text : String @compiled : Crustache::Syntax::Template def initialize(path) @text = File.read(path) @compiled = Crustache.parse(@text) end end # Load templates templates = {} of String => Template Dir.glob("templates/*.tmpl").each do |path| templates[path] = Template.new(path) end
I changed the templates from whatever they were before to mustache:
<h2><a href="{{link}}">{{title}}</a></h2> date: {{date}} <hr> {{text}}
I can now implement Post.render
... except that top-level variables like templates
are not accessible from inside classes and that messes up my code, so it needs refactoring. So.
This sure as hell is not idiomatic Crystal, but bear with me, I am a beginner here.
This scans for all posts, then prints them rendered with the post.tmpl
template:
class Post @metadata = {} of YAML::Any => YAML::Any @text : String @link : String @html : String def initialize(path) contents = File.read(path) metadata, @text = contents.split("\n\n", 2) @metadata = YAML.parse(metadata).as_h @link = path.split("/")[-1][0..-4] + ".html" @html = Markd.to_html(@text) end def render(template) Crustache.render template.@compiled, @metadata.merge({"link" => @link, "text" => @html}) end end posts = [] of Post Dir.glob("posts/*.md").each do |path| posts << Post.new(path) p! p.render templates["templates/post.tmpl"] end
Believe it or not, this is almost done.
Now I need to make it output that (passed through another template) into the right path in a output/
folder.
This almost works:
Dir.glob("posts/*.md").each do |path| post = Post.new(path) rendered_post = post.render templates["templates/post.tmpl"] rendered_page = Crustache.render(templates["templates/page.tmpl"].@compiled, tpl_data.merge({ "content" => rendered_post, })) File.open("output/#{post.@link}", "w") do |io| io.puts rendered_page end end
For some reason all my HTML is escaped, I think that's the template engine trying to be safe 😤
Turns out I had to use TRIPLE handlebars to print unescaped HTML, so after a small fix in the templates...
So, success! It has been fun, and I quite like the language!
I published it at my git server but here's the full source code, all 60 lines of it:
# Nicoletta, a minimal static site generator. require "yaml" require "markd" require "crustache" VERSION = "0.1.0" # Load config file tpl_data = File.open("conf") do |file| YAML.parse(file).as_h end class Template @text : String @compiled : Crustache::Syntax::Template def initialize(path) @text = File.read(path) @compiled = Crustache.parse(@text) end end # Load templates templates = {} of String => Template Dir.glob("templates/*.tmpl").each do |path| templates[path] = Template.new(path) end class Post @metadata = {} of YAML::Any => YAML::Any @text : String @link : String @html : String def initialize(path) contents = File.read(path) metadata, @text = contents.split("\n\n", 2) @metadata = YAML.parse(metadata).as_h @link = path.split("/")[-1][0..-4] + ".html" @html = Markd.to_html(@text) end def render(template) Crustache.render template.@compiled, @metadata.merge({"link" => @link, "text" => @html}) end end Dir.glob("posts/*.md").each do |path| post = Post.new(path) rendered_post = post.render templates["templates/post.tmpl"] rendered_page = Crustache.render(templates["templates/page.tmpl"].@compiled, tpl_data.merge({ "content" => rendered_post, })) File.open("output/#{post.@link}", "w") do |io| io.puts rendered_page end end