fasterthanlime

Nov 03, 2014 29 min #s-expressions

S-exps in your browser

I've been interested in reactive JavaScript for a while. At memoways, we strive to build snappy user interfaces for clients who like to interact with their data with as little latency as possible.

In the past two years, I learned front-end development on-the-fly, as the needs of the clients required it. Two years ago, I was still using jQuery. Then, I discovered space-pen thanks to my colleague Nicolas. It was nice to have proper 'view' objects, and use jQuery's event system to have messages propagate throughout a hierarchy.

But state management was still an issue. Anything a tiny bit complex broke way too easily. Surely with a better code organization, I might've been able to hold on to space-pen longer (after all, it's used by Atom), but no matter how much effort you put into it, space-pen isn't really suited to rapid prototyping, something you need to do a lot of when you touch domains like dataviz.

Then I discovered Ractive.js, by the wonderful team at The Guardian. It seemed to me like a version of React without the baggage. Two-way binding is magical when it works: just store dynamic properties within the data object, use the getter/setter methods (or built-into-JS accessors if you're feeling adventurous and independent from all IE compatibility needs) and everything updates smoothly.

However, Ractive proved very hard to debug. Its custom sort-of-HTML parser (as is the tradition with almost all reactive frameworks nowadays) didn't give much context as to where errors were, when I last tried it. I had to quickly hack support for "printing surrounding code when a syntax error occurs" into my copy, just so my app wouldn't collapse.

Error reporting is often overlooked by new language implementors. "You should be writing correct code in the first place!", right? The truth is: good error reporting & debugging support is essential - when refactoring, tests are only useful to a degree. After that, you're left to the whims of whatever compiler, processor, macro expander you're using.

I had been hearing good stuff about Om for a while. I think it's one of the recent Prismatic blog posts that sold me on it. That was before I learned that they're pretty much the only high-profile company using ClojureScript in production!

But at this point, I was fed up with suddenly-breaking Ractive code, and anything new was good to distract me of my daily dev problems. Turns out, Om is probably awesome, but it's too verbose for me.

Here's how the start of Om's tutorial looks like:

(om/root
  (fn [app owner]
    (reify om/IRender
      (render [_]
        (dom/h1 nil (:text app)))))
  app-state
  {:target (. js/document (getElementById "app"))})

Now, when you're discovering a new paradigm and a new language, that's a bit steep.

Let's walk through things step by step. First off, it's S-expressions all the way down. That means instead of writing:

log("A completely unremarkable number: ", 40 + 2)

You'd write:

(log "A completely unremarkable number: " (+ 40 2))

Yes, even + looks like a function call. It's handy, too, because you can pass it any number of arguments, unlike the infix version:

(log "Look, ma" (+ 1 2 3 4 5 6))

Same goes for =, and lots of other primitives.

So, now we can see that we're calling something called om/root. The slash is Clojure's way of saying it likes to keep things nice and tidy, arranged into namespaces. In this case, what they don't tell you is that on the top of this file is probably something like:

(ns myapp.core
  (:require [om.core :as om]))

And so, whenever we type om/something, we refer to something within the om.core namespace.

Note that ClojureScript allows you to refer symbols so that you can use them without any qualification, for example we could've done the following:

(ns myapp.core
  (:require [om.core :as om :refer [root]]))

(root arg1 arg2 arg3)

You see what I'm getting at.

Then we have array syntax:

(def heroes ["Samus" "Peach" "Sarah Kerrigan"])

Like arguments to a function, arrays are space-separated. Maps are about the same, except you need to have an even number of elements and the odd ones are the keys.

Usually we'll use symbols instead of strings as map keys:

(def langs {:clojure "JVM" :haxe "Neko" :vala "GLib"})

As for functions definitions, they look like a call as well:

(fn [a b c]
  (log "In a function somewhere!")
  (+ a b c))

Now with our newly-acquired notions, let's go back to our original code:

(om/root
  (fn [app owner]
    (reify om/IRender
      (render [_]
        (dom/h1 nil (:text app)))))
  app-state
  {:target (. js/document (getElementById "app"))})

We have a call to om/root, and the first argument is a function:

(fn [app owner]
  (reify om/IRender
    (render [_]
      (dom/h1 nil (:text app)))))

The function returns a type - really, an interface implementation if you ask me:

(reify om/IRender
  (render [_]
    (dom/h1 nil (:text app))))

So, we implement the om/IRender interface, and it has only one method:

(render [_]
  (dom/h1 nil (:text app)))

It takes one argument, which we bind to _ to mark that we really don't care about it (in this case it's the local state of the component, but let's not get into this...)

And then we call the h1 dom construction function, with no special options (nil is Clojure for null), and what the value for the key :text stored in app.

In the interest of clarity, (:text app) is roughly equivalent to (get app :text), or app['text'] in JavaScript.

The second argument to om/root is simply app-state, an atom (we'll get back to that).

And the third argument is a map, with a single key, :target, whose value is:

(. js/document (getElementById "app"))

That's ClojureScript/JavaScript interoperability - equivalent to the following JavaScript:

document.getElementById('app')

Now that we've learned a few basics to be able to read ClojureScript code, what does this actually do? Well, anything implementing the om/IRender interface is an Om component, and calling om/root mounts it to a DOM element somewhere. Somewhere in our HTML, we probably have that:

<div id="app"></div>

And in there we'll have an h1 tag containing whatever is in (:text app).

So then, how do we put stuff in the app state? As we mentioned earlier, app-state is an atom. We could define it like that:

(def app-state (atom {:text "Initial text"}))

And then, if we want to change the text, we can do:

(swap! app-state assoc :text "New text!")

Which is really just a shorthand for:

(swap! app-state #(assoc % :text "New text!"))

Itself a shorthand for:

(swap! app-state (fn [state] (assoc state :text "New text!")))

What does swap! do? Well, an atom is basically a reference to a value. The value itself is immutable, as all Clojure data structures, but we can change the reference to something else.

So, wherever in Ractive you'd do state.set(key, value), changing the state itself, in ClojureScript you'd rather swap the reference to a slightly changed version of the state.

However, the whole state isn't copied - the parts that didn't change are immutable as well, so they're just referenced from the new data structure.

Let's take a more complex state as an example.

(let [state (atom {:a [1 2 3]
                   :b {:first-name "John"
                       :last-name "Doe"}})]
  (log "Old state: " (pr-str @state))
  (swap! state assoc :a [4 5 6])
  (log "New state: " (pr-str @state)))

In this example, not only are the old and new (:b @state) value-equal but they're also reference-equal - they're the same data structures, at the same place in memory.

Note: @some-atom is the atom dereference operation. It evalutes to the value the atom references, rather than the atom itself. And pr-str is basically a toString.

So, whenever the state atom is changed, comparing the old state and the new state is very fast - still in our "John Doe" example above, no need to compare whether (:first-name (:b @state)) has changed, for example - we can see that (:b @state) still has the same address, so all the substructure has to be the same - because of immutability.

Even when the app state gets complicated, small updates to part of the state tree are still fast, because it's inexpensive to find out which parts of the tree are changed. Hence, only the minimal amount of re-rendering is done.

It might not seem like much, but writing all those reify forms and having to require and call methods for every dom element get old really quick. Call me impatient, but I knew there was something simpler lurking around the corner.

And there was! Reagent provides a much simpler interface to React. So, in the same ballpark as Om but without all the running around in circles.

In reagent, components are simply functions, and HTML tags are specified as keywords, with a Hiccup-like syntax.

For example, I love FontAwesome icons. Here's a simple component that displays the icon of your choice:

(defn fa-icon [icon]
  [:i {:class (str "fa fa-" icon)}])

Using it is as simple as any HTML tag:

(defn profile [user]
  [:div.user
    [:h2 (:name @user)]
    [:p (:bio @user)]
    [:span.button
      {:on-click #(follow user)}
      "Follow " (:name @user)
      [fa-icon "star"]]])

That would produce HTML like:

<div class="user">
  <h2>The Jester</h2>
  <p>Some obscure hacker at some point in the internet's history.</p>
  <span class="button">
    Follow The Jester <i class="fa fa-star"></i>
  </span>
</div>

I was pretty happy with Reagent, and wrote the UI for the memoways issue tracking system in it. It works very well, and I can prototype quickly, although errors are still kind of painful.

But now I'm seriously looking at ClojureScript alternatives.

I was okay with having to learn a completely new build tool, leiningen. In fact, it's really nice to have mostly one tool to interact with for the whole language.

I was okay with ugly JavaScript interop, leading to code like this:

(.getElementById js/document (.- value (.- target e)))

I was okay with syntax errors yielding 50-lines-long stacktraces in my console.

I was even okay with the REPL taking around 4s to start (on a 2.2Ghz Core i7 with a recent SSD).

But I'm not okay with the completely erratic behavior of lein-cljsbuild and the Google Closure compiler.

Let me get you up-to-speed on the whole ClojureScript/Closure shenanigans.

Google Closure compiler has two main components:

• An SDK covering a lot of ground
• An optimizing/minifying JavaScript compiler

ClojureScript relies on the former to reimplement most of the Clojure standard library in JavaScript (while also using its module loading facilities), and on the latter to keep the compiler's output to reasonable sizes.

And by "reasonable size" I mean 330k minified, 80k minified+gzipped, with just a few libraries. Turns out cljs.core (imported by default) is huge. Like, over nine thousand lines in a single file huge.

In dev, it's not that bad. Sure, you have to go through a few hoops, like figuring out the right options to put in your project.clj, including some goog/base.js file on top of your yourproject.js file and having an inline JavaScript call to goog.require.

With all that figured out, it's really not that bad. Again, in dev. Recompilations with lein cljsbuild auto dev are fast, around 200ms, it loads all dependencies correctly with XHR calls (about 20 separate .js files to do anything useful), everything's fine.

In production, it's another story entirely. First off, you want to set the Closure compiler to perform advanced optimizations. Basically it'll attempt to:

• Perform dead code elimination (remove the parts you don't need)
• Minify by renaming symbols (variables, functions) to shorter things like Gl or aQ instead of reverse or forEach
• And of course, removing all the whitespace it can get away with.

So first off, it's slow. It's really really slow. If you have 9000+ lines of ClojureScript as input, you can imagine how much JavaScript it spits out. The Closure compiler has several megabytes of JavaScript to chew through, and it's not uncommon for it to take around 30 seconds to process completely.

Second, you have to be extra careful about foreign libraries. Because:

• Chances are, they don't use the google module system at all (goog.require, goog.provide, etc.)
• They define symbols that can not be renamed by the closure compiler in your code.

But there's a thing for that! Just add them to an externs list as an option to the compiler and you're good to go. Except, again, it's slow. Very slow. And it won't hesitate to emit thousands of warnings on sources over which you have absolutely no control.

And if it wasn't enough, once it's done taking dozens of seconds compiling all that stuff - sometimes it just emits erroneous code. I suspect that, when scanning a directory for sources files, the tree it constructs is wrong, because if you run it in watch mode (so that it recompiles automatically on file changes) and touch just the right file (the one you'd pass to the compiler if it did allow that), then it takes another 8 to 10 seconds and generates correct code.

Oh, and if you want to modularize your codebase, say you want to output one .js file per "view type" of your application, say one for the issue tracker UI, one for the client-side slideshow visualizer, etc. - not only do you have to maintain a contrived directory infrastructure and specify tons of redundant dependencies in your project.clj - you also get the 30s to get the wrong code then 10s to get the right one but only if you launch it in watch mode for every separate target file you want to have.

In that light, it appears that ClojureScript is - currently - quite painful to use in production. Sure, the tooling may improve in the coming months, but in a fast-moving world, I don't really have the time to wait.

Honestly, the ecosystem feels just wrong - even though I love reading Clojure code (the language itself is a pleasure) - and it's full of nice people, fetching .js files as a .pom + .jar from a Maven repository... it feels wrong, and so dissociated from the rest of the JavaScript world.

When looking for alternatives, I knew that I wanted something:

• Lighter than ClojureScript
• With a similar syntax (S-exps are <3)
• With immutable data structures.

Ki is just that. It uses mori, which brings ClojureScript's data structures, and is, besides that, mostly a set of sweet.js macros to turn S-expression back into good ol' JavaScript.

I won't detail the whole language, it has a reference page on the website. The main differences from ClojureScript are missing features (at the time of this writing: destructuring assignment is the one I miss the most), and snake_case instead of kebab-case.

Of course, a bunch of macros (as sophisticated as they might get) will never do as much as a full compiler will - but in this case, it's kind of the point. A compiled ki program is usually pretty small, the only constant cost being mori, which is rather small (by cljs standards).

Ki presents itself as a JavaScript script, the command-line tool provided with the distribution is ran with Node.js by default. After having played around with the language a bit, I was seduced. Moving on to the next problem: how the hell do I integrate that to my Rails application?

With ClojureScript, I had given up - even though Rails asset pipeline support for ClojureScript does exist, the documentation made it clear that it had to spin up a JVM each time any asset was modified. So I was just using the lein cljsbuild auto command by hand.

In Ki's case, though... I had a feeling we might do better. A lot better. We could, obviously, hook into the Rails asset pipeline and, when, a .ki.js file needs to be compiled, write it to a temporary file, then call the ki command-line tool on it, ask it to write to another temporary file, then read that file and return it to Rails.

But that's no fun at all! There's a reason tools like the Coffee compiler, sass compiler, ki compiler, have "watch" modes - they take longer to initialize than they do to recompile something. And we only need to pay that cost once.

So, let's go with a much harder - but more rewarding - alternative. Keeping our own instance of V8 around and running the ki compiler through that. A handy way to interact with v8 from Ruby is therubyracer. Ideally we'd use something runtime-agnostic like execjs - but we'll have enough trouble making our stuff work in a single JavaScript engine like that.

therubyracer is very nice to play with. The first thing we want to do is create a context.

context = V8::Context.new

Then we can eval some JavaScript expressions in it:

context.eval("2 + 2") # => 4

You can interact with JS expression from Ruby, for example, JS arrays come back as V8::C::Array instances, which you can coerce to real ruby arrays, or just iterate with each.

context.eval("[1, 2, 3]").to_a # => [1, 2, 3]

Similarly, you can iterate over JavaScript objects (ie. dictionaries/maps) with each if you want.

context.eval("{a: 'apple', b: 'bananas'}").each do |key, val|
  puts "#{key} => #{val}"
end

# prints:
# a => apple
# b => bananas

The context retains globals across eval calls, so we can store whatever we want in this (the equivalent of window when there's no browser around.)

this.$modules = {}

As always, though, globals are evil, if only because they might collide with other globals, so we'll use that sparingly.

We can load JavaScript files via eval using a little trick to escape the string properly:

require 'multi_json'

source = File.read("foobar.js")
context.eval %Q{
  // preamble
  #{MultiJson.dump(source)}
  // epilogue
}

This approch has an advantage - you can fence the code with a preamble and epilogue of your choice. You can make your code execute within a scope where you override some variables, like define or exports for example.

If you don't need to do that, you can just tell V8 to load the file directly:

context.load("foobar.js")

In this case, you don't control the scope the file is evaluted in (it's always the global scope) - but you do retain file / line number information within stack traces, which is invaluable.

So, now that we know how to evaluate arbitrary code within V8, and retrieve the results, and how to load any JS file in it, we're done right? We just load ki.js, call ki.compile and be on our merry way. Right? Riiiiiight?

Wrong. You see, in the JS world (the real, vibrant, "god save us lest we forget a var" JS world, not the compiles-to-JS world), they've been busy figuring out the best way to have modular code that can be loaded in different environments (say, Node.js, or a browser module loader).

The idea is for libraries to avoid clobbering the global scope with stuff like _, or $, or jQuery, or React, or whatever variable they've chosen.

That's why it's not uncommon to see JS libraries nowadays structured like this:

(function(root, factory) {
  if (typeof exports === 'object') {
    // Node.js-style require
    factory(exports, require('dep1'), require('dep2'));
  } else if (typeof define === 'function' && define.amd) {
    // AMD-style define
    define(['exports', 'dep1', 'dep2'], factory);
  } else {
    // Fuck-it style "let's pollute the global scope"
    // if you haven't included 'dep1.js' and 'dep2.js' before, this will break.
    var exports = {};
    factory(exports, Dep1, Dep2);
    root.mylib = exports;
  }
}(this, function (exports, dep1, dep2) {
  // we can access dep1 & dep2 here. mostly.
  exports.something = something;
  exports.somethingElse = somethingElse;
}));

In the case of ki.js, the main file for the ki compiler, the fuck-it style isn't included at all. Which means we can't get away with the naive gross approach of just loading everything in the right order, crossing our fingers, and sacrificing a virgin veal to the gods of globals.

The Node.js style is kind of painful to handle for us from the Ruby side. Whenever require is called, something must be done - but we load files not by XHR like some JS module loaders do, but by reading files from the filesystem ourselves (or calling context.load, or...). Point is - we need some way to get back to the Ruby side from that JS call.

I know that it's possible. In particular, commonjs.rb does it - but it doesn't solve problem #2 with this approach, which is that the ki compiler and its dependencies assume that, if it's loaded with the Node.js style, it can access Node.js modules such as "fs" (the async, libuv-based thingy to interact with the filesystem. so cute.) And since we're running in V8, not in node.js, we have zero access to that kind of module. We'll do the I/O ourselves, thank you very much.

Which leaves us with the AMD approach - the cleanest, in my view. I like the idea. Each module calls define with a list of dependencies, and a factory function that should be called with these dependencies.

It leaves us plenty of time to inspect those 'module specifications' from the Ruby side, load the dependencies and then pass them back to the factory function.

Now, at this stage of my journey I've lost a lot of time trying to do something sane. I thought - well it's easy: we'll start by loading up ki.js, then see what its dependencies are, then load them up, see what their dependencies are and so forth until we have a complete dependency graph. Then we know in which order to call the factory functions.

It looked a little bit like this:

ki         => exports sweet
sweet      => exports underscore parser expander syntax text!./stxcase.js escodegen escope
underscore =>
parser     => exports expander
expander   => exports underscore parser syntax scopedEval patterns escodegen
syntax     => exports underscore parser expander
escodegen  => exports
escope     => exports estraverse
scopedEval => exports
patterns   => exports underscore parser expander syntax
estraverse => exports

Or, in graphviz form:

From this graph, though, it's immediately apparent that we have a serious problem. Circular dependencies are lurking everywhere! parser depends on expander, but expander depends on parser as well. And many others.

So, my beautiful plan of loading everything in order has quickly fallen apart. Instead, we can do the next best thing: load them in the order they are required, and when something's not properly loaded yet, reserve a empty-object space in our modules list that we'll pass to the dependants. I suspect that's how most AMD loaders work, because this approach actually worked.

In the case of ki-rails, I've decided to store modules in a global object, so that I can easily access them later. At the time of this writing, the code isn't entirely cleaned up, so don't hit me with a large trout - just be patient.

There's a few things I haven't mentioned yet, but need to be taken care of:

• First off, exports is a special requirement. It's not a js file we load, it's the object in which all the exported symbols of a library will be stored. That must be accounted for.
• Second, my dependency graph only shows sanitized module names. In the real world, it's not uncommon to see modules require ./module or module.js instead of just module
• Thirdly, AMD loaders allow two different define signatures - one that goes: name, dependencies, factory, and an anonymous one. Most modules I've encountered use the latter, but named modules exist (within the sweet.js codebase) and must be handled correctly.
• Lastly, dependencies starting with text! are not JavaScript files and must be passed as a string without any evaluation, to the module asking for it.

With all that, we can finally call our ki compiler from V8, a naive version of which looks something like:

# sweet.js macros for ki (actually implement the language)
ki_core = File.read("ki.sjs")

# our ki.js code that we want to compile.
source = File.read("somefile.ki.js")

ret = context.eval %Q{
  var ki = __modules.ki;
  var options = {};
  options.ki_core = #{MultiJson.dump(ki_core)};
  var source = #{MultiJson.dump(source)};
  ki.compile(source, options);
}
ret[:code] # => our compiled JavaScript code!

However, this approach is pretty naive:

• It parses the ki core macros every time
• It has no source map support

Thankfully, Sweet.js has source map support built-in. And the ki compiler interface allows its usage, like so:

options = {
  source: source,
  ki_core: ki_core,
  sourceMap: true,
  filemap: "path/to/yourfile.js.ki",
  mapfile: "path/to/yourfile.map"
}
ret = ki.compile(source, options);
ret.sourceMap // => the sourceMap content
ret.code // => the compiled JavaScript code, including the sourceMappingURL comment.

To hook that into ki-rails, I took inspiration from Mark Bates' coffee-rails-source-maps. Basically, the original ki file along with its source file are written into public/assets/source_maps and the correct URL is specified in sourceMappingURL, in the compiled JavaScript file.

Apart from erroring in production because the server was very protective about where the web app could and could not write, it worked very nicely!

I said one of the problems with our approach, was that the ki macros were being parsed everytime. This is actually pretty slow. At the time of this writing, they only amount to about 900 lines of rules, but the sweet.js parser is quite an involved piece of work, and as such, I was seeing end-to-end compilation times of about a second or two.

We can avoid that by precompiling ki_core and keeping it around in the global context.

context.eval %{Q
  this.__macros = sweet.loadModule(#{MultiJson.dump(ki_core)});
}

And then, later, passing it as the modules option to the ki compiler instead of passing the ki_core option.

This has a negative side-effect, however. See, the ki compiler has a piece of code like this:

if (!options.modules && options.ki_core) {
  var module = joinModule(src,options.ki_core,options.rules);
  options.modules = sweet.loadModule(module);
}

ie. when passed the source of the ki macros instead of pre-compiled modules, it calls joinModule on the input (ie. src) and the ki macros, then asks sweet to load that. What does joinModule do exactly?

var joinModule = function(src, ki_core, additionalRules) {
  rules = additionalRules || [];
  rules.push(parseMacros(src));
  return ki_core.replace('/*__macros__*/',rules.join('\n'));
}

How interesting! A parseMacros method. That's where I saw that my little trick didn't quite work. Sure, it made compilations faster, but it broke ki macro support.

To understand why, we have to learn a bit more about how ki macros and how ki is implemented. Ki is implemented, as previously mentioned, via a set of sweet.js macros. They look like that.

macro _arr {
  rule { () } => {
  }

  rule { ($arg) } => {
    _sexpr $arg
  }

  rule { ($arg $args ...) } => {
    _sexpr $arg, _arr ($args ...)
  }
}

The largest macro, by far, is _sexpr. It contains rules like:

rule { (fn [$args ...] $sexprs ...) } => {
  function ($args(,)...) {
    _return_sexprs ($sexprs ...)
  }
}

That's the anonymous function definition syntax! Sweet.js is definitely readable. Here's an example of ki macro in action:

ki macro (thunk $body ...)
         (fn [] $body ...)

ki (thunk (alert "Whoops!"));

The parseMacros call we've seen earlier in the ki compiler looks for the "ki macro" string with a regular expression, then converts the definitions to sweet.js rules, like so:

var rules = macros.map(function(macro) { 
  return 'rule { ' + macro[0] + ' } => { _sexpr ' + macro[1] + ' }' 
});

That leaves us with this mysterious line in joinModule:

return ki_core.replace('/*__macros__*/',rules.join('\n'));

Apparently it's replacing a comment within the ki core macros' source with the user's macro definitions. Why is it doing that? Let's see where it gets inserted:

macro _sexpr {

  rule { () } => { 
  }
  
  /*__macros__*/

  /* ... rest of the _sexpr macro */

}

Of course! It's smack dab in the middle of the _sexpr macro, so that we may use ki user-defined macros anywhere in an S-expression.

However, it's bad news for our Asset pipeline integration dreams. Does this mean that if we want to support user-defined ki macros, we'll have to recompile the whole thing every time, and thus have a minimum of 1-2s compilation time?

Not so fast! First off, ki has no concept of import or require or include. It allows one to define namespaces, but there's no built-in support for importing other files, unlike, for example, sass. (Not a language that compiles to JS, but it is well-supported by rails and has its own @import statement).

So, we have to fall back to the next best thing: Sprockets directives. Sprockets is the little processing meta-engine that could, powering the whole Rails asset pipeline, from preprocessing to transforming to postprocessing to fingerprinting, manifest-generating, caching, and serving-with-the-right-headers.

As anyone who has written JavaScript or CoffeeScript in Rails, Sprockets has directives, the most used of which are require, require_tree and require_self:

//= require preamble
//= require_self
//= require_tree ./components

console.log("This will be between preamble and all things in ./components");

Now, since we want to write modular ki code, we want to be able to require stuff.

In the case of functions, the ki compiler doesn't have to know about it. They're just regular JavaScript functions, it's not like any compilation-time checking is done.

In the case of macros, however, the ki compiler has to know about every macro definition in the dependencies of our file, so that it may apply the right transformations.

In other words, we need to do the following (in pseudo code):

when compiling a ki file within Rails:
  - figure out all the files it requires
  - read them all
  - concatenate them in a single string
  - extract only the macro definition parts
  - load them with sweet.js
  - pass them to the ki compiler.

Now, the Sprockets API is not that hard to figure out, although I wish better documentation existed on the subject. Turns out, from within a tilt template called by Sprockets, like this one:

class KiTemplate < Tilt::Template
  def prepare
    # must be defined
  end

  def evaluate(scope, locals, &block)
    # here's the magic
    compile(data)
  end
end

..we can access the set of dependencies with a simple scope._dependency_assets, and we can even coerce it to an array with a .to_a call. Giving us something like:

["/Users/amos/Dev/memoways/memowaysv2/app/assets/javascripts/experiments.js.ki", "/Users/amos/Dev/memoways/memowaysv2/app/assets/javascripts/experiments/console-log.js.ki", "/Users/amos/Dev/memoways/memowaysv2/app/assets/javascripts/experiments/macro-usage.js.ki", "/Users/amos/Dev/memoways/memowaysv2/app/assets/javascripts/experiments/macro.js.ki", "/Users/amos/Dev/memoways/memowaysv2/app/assets/javascripts/experiments/take-range.js.ki"]

Reading all those files and concatenating them is done in two lines of Ruby. Extracting the macro part can be done by calling the JS method ki.parseMacros via our V8 context and all that's left is to call sweet.loadModule and pass the result to the ki compiler.

Of course, that only solves the problem of "if A requires B, A should be able to use macros defined in B". It doesn't solve the problem of "I want compilations to be fast because brain switches are expensives".

However, we can easily solve that with a simple cache. I've opted for a very naive solution for the time being: I simply have a global object __macrocache in the V8 environment that stores compiled macros, indexed by the SHA-1 hash of their sources.

Often, I'll change the ki code without touching any macro definition. In that case, the SHA-1 of the macro sources stays the same, and no reloading of the sweet.js module happens. In that case, recompilation time is not even noticeable!

Comment on /r/fasterthanlime

Here's another article just for you:

Getting in and out of trouble with Rust futures

Jul 07, 2020

30 min #rust · #async

I started experimenting with asynchronous Rust code back when futures 0.1 was all we had - before async/await. I was a Rust baby then (I'm at least a toddler now), so I quickly drowned in a sea of .and_then, .map_err and Either<A, B>.

But that's all in the past! I guess!

Now everything is fine, and things go smoothly. For the most part. But even with , there are still some cases where the compiler diagnostics are, just, .

Read more