Fast File Tree Navigation, Powered by WebAssembly

We recently shipped a web-based file tree navigator so that you can see the files from a task's output. Seeing the file tree is helpful for configuring output filters. We currently do not display the contents of the files, but we will add that in the future.

To make browsing the file tree as fast as possible, we implemented it in WebAssembly.

#Background on RWX's Architecture

Tasks run on RWX with content-based caching. This means that before running a task, a cache key is generated based on the exact contents of all of the files on the filesystem.

We generate a manifest of the entire filesystem using a proprietary binary format. Before running tasks, we merge all of the manifests for the dependencies together, apply filters, and then produce a cache key.

The algorithm is performance-intensive, and we want it to run as fast as possible, so we wrote it in Go.

#Using WebAssembly

When building the web-based file tree browser, we wanted to reuse the code we had already written for parsing our proprietary binary format. We also wanted it to be as fast as possible.

Fortunately, Go makes it easy to compile to WebAssembly.

go build -o build/manifests.wasm ./wasm

The Go code:

1
package main
2

3
import (
4
	"bytes"
5
	"encoding/json"
6
	"os"
7
	"syscall/js"
8

9
	"github.com/rwx-cloud/manifests/parsing"
10
)
11

12
func main() {
13
	rwxManifestsJs := js.Global().Get("Object").New()
14
	rwxManifestsJs.Set("parseManifest", js.FuncOf(parseManifest))
15
	rwxManifestsJs.Set("quit", js.FuncOf(quit))
16

17
	js.Global().Set("RwxManifests", rwxManifestsJs)
18
	js.Global().Get("document").Call("dispatchEvent", js.Global().Get("CustomEvent").New("rwx-manifests:ready"))
19

20
	select {}
21
}

We use this approach to ship the entire manifest to the browser, parse it client-side, and provide fast tree navigation.

#Demo