Golang from Python

Articles by Owen Raccuglia

OpenSimplex Noise in Go

One of my Go projects uses procedurally-generated terrain — which means I want smooth random noise. Perlin noise is a common choice, and there are existing Go implementations, but Perlin noise suffers from axis-aligned artifacts. Ken Perlin's successor to Perlin noise (Simplex) fixes those problems, but adds a worse one: it's patented. Thankfully, there's an alternative: OpenSimplex noise by Kurt Spencer. I couldn't find a Go version, so I've ported OpenSimplex noise to Go, and released it under the same Unlicense as the OpenSimplex algorithm.

Besides getting the code to compile and look basically right, my goal was to get identical output to the reference implementation — ideally, regardless of platform. My first attempt at testing was to render and compare greyscale PNGs of the noise. That caught the first few glaring bugs, but it failed at the larger goal of correctness: I was crushing 64-bit floats into a tiny color channel.

I needed to preserve all 64-bits of the inputs and output. I also wanted less uniform test samples and a wider range of input, in case there were subtle bugs masked by the regular interval and all-positive numbers. For the second pass, I generated 2048 samples per dimension, randomly choosing coordinates in [-1000, 1000] from the reference implementation.

Now that my test failures were in terms of float64s, I found my final bug: everything was off by around 0.00000000001. The problem was with my very first choice when porting — high-precision Go constants.

The OpenSimplex Java code starts with a few lines like this:

double STRETCH_CONSTANT_2D = -0.211324865405187;    //(1/Math.sqrt(2+1)-1)/2;
double SQUISH_CONSTANT_2D = 0.366025403784439;      //(Math.sqrt(2+1)-1)/2;
double STRETCH_CONSTANT_3D = -1.0 / 6;              //(1/Math.sqrt(3+1)-1)/3;
double SQUISH_CONSTANT_3D = 1.0 / 3;                //(Math.sqrt(3+1)-1)/3;
double STRETCH_CONSTANT_4D = -0.138196601125011;    //(1/Math.sqrt(4+1)-1)/4;
double SQUISH_CONSTANT_4D = 0.309016994374947;      //(Math.sqrt(4+1)-1)/4;

I jumped at the chance to reevaluate the functions and pile on as much precision as I could. Go distinguishes itself from most languages with high-precision compile-time math, but this port needs consistency over precision. Restoring the original constants did the unthinkable: I compared floating point numbers with == and passed tests.

I have two long-term goals for this code: testing on other architectures (to look for floating-point math surprises), and benchmarking. It can take a lot of samples of noise to generate a procedural scene, so speed is important; It's also an opportunity to get more experience with Go's built-in benchmarking tools. Even without those improvements, I think this code is still useful enough to publish; OpenSimplex is great for anyone who wants higher quality than Perlin noise, but isn't willing to deal with patents.

One Project Down

It's time for a retrospective post: After seven months of off-and-on porting, I've finished my first "complete" web project in Go. I'm happy with how a few things worked out, but this post focuses on the other stuff. I've already written up my thoughts on templates, and I'm pretty happy with how that approach worked.

Middleware and Request Context

I used negroni. I liked it, but I think I may have overused it, too. The next project will start with built-in middleware: panic recovery, request/response logging and serving static files. Before I add any of my own, I'll write it as a http.Handler or http.HandlerFunc first.

I tried using gorilla's context library for storing per-request info. It was fine, but I'm going to try out Google's net/context next.

Exceptions

I missed having stack traces. Some of that pain could have been solved if I caught more bugs with small unit tests, but something always slips through the cracks. Sometimes you just need a line number. I'm going to try out go-errors/errors from Bugsnag — they wrote an introduction to go-errors here. I'm interested to see how much of a hassle it will be to change my uses of err == Expected to errors.Is(err, Expected).

Silly Scope Mistakes

You may have seen this already: the := operator interacts with variable shadowing in a way that surprised me.

func find(toFind valueType, items []valueType) (bool, error) {
    found := false
    for _, item := range items {
        found, err := compareDangerously(toFind, item)
        if err != nil {
            return false, err
        }

        if found {
            break
        }
    }

    return found, nil
}

This function can never return true, because the name found refers to two different bools. := prefers to declare variables; any names on the left-hand side that were declared at a different scope will be shadowed, not assigned to. When I use Go on a multi-person team, I'll add a lint step to the tests that treats implicitly shadowed variables as an error.

Auto Reloading

I want my server to reload whenever I make a code change. After a little searching, I found some code from rsc that works, with a little effort. Here's how I changed my server to use it:

listen := fmt.Sprintf("0.0.0.0:%d", *addr)
if flag.Arg(0) == "LISTEN_STDIN" {
    log.Printf("Running dev server over stdin")
    l, err := net.FileListener(os.Stdin)
    if err != nil {
        log.Fatal(err)
    }   
    log.Fatal(http.Serve(l, nil))
} else {
    log.Printf("Listening on %s\n", listen)
    log.Fatal(http.ListenAndServe(listen, nil))
}

It looks like devweb may have been built with Windows in mind -- it runs go build -o prox.exe [1st argument], and then runs prox.exe LISTEN_STDIN. The above code handles LISTEN_STDIN; the last step is to make sure your PATH is set up right. I'm running it like this: PATH=$PATH:. devweb ..

What Next?

Unlike the first project, this one is going to be closed source, because it's a reimplementation of an internal tool we use at Demiurge Studios to manage our localization database. It's a decent fit for Go — in particular, I'm looking forward to improving our asynchronous exports, which should be relatively easy to manage with a single-process server and goroutines.

Testing and the Database

Writing Go database code took several tries before I settled on something that worked well for tests and HTTP handlers. The root decision was that all database functions should require a *sql.DB. Everything else falls into place — concise tests, separation between the application and test databases, and flexible HTTP request handlers.

The application *sql.DB is created in main(), and associated with each HTTP request context with some request middleware, which I'll talk about more in another post. I spent some time with a global *sql.DB, but it made testing HTTP handlers uncomfortable. Instead, some middleware associates the database connection with the *http.Request, and I retrieve it when needed:

func someHandler(w http.ResponseWriter, r *http.Request) {
    user, err := db.GetUserByGuid(GetDb(r), guid)
    // ...do other stuff
}

Tests work in the exact opposite way, by declaring var tdb *sql.DB globally in a _test.go file, so it isn't compiled in to my regular builds. Instead of GetDb(r), I pass tdb.

Cleaning Up After

Or, "Life Without tear_down." My usual test (if it exercises the database) runs some set-up, calls functions, and cleans up any data created in the process. Go cleans up resources by deferring a cleanup function; because cleaning up database data is such a common task, my first cleanup helper was a tearDown function for deleting test database rows. Using that, a test for a function that inserts a new row into the user table would look something like this:

func TestNewUser(t *testing.T) {
    username := randStr()
    defer tearDown("user", "Username = $1", username)

    user, err := CreateUser(username)
    // ...assertions follow
}

I make a point of deferring the tearDown before the actual insert. It lets me handle panics that happen between inserting data and returning from a function — which will be useful later. Here's how I implemented tearDown:

func tearDown(table, where string, params ...interface{}) {
    _, err := tdb.Exec(
        fmt.Sprintf("DELETE FROM \"%s\" WHERE %s", table, where),
        params...)
    if err != nil {
        // Usually I'd return the error, but using panic means less code in
        // my tests, which is more important.
        panic(fmt.Sprintf("Problem tearing down %s data: %v", table, err))
    }
}

More Rows, Fewer Lines

This works for tests that insert a row or two, but as you start using more supporting data in your tests, it makes sense to bundle up common object creation helpers in their own functions:

func TestSomethingElse(t *testing.T) {
    user, cleanup := newTestUser()
    defer cleanup()
    // ...test follows
}

You may have noticed, after all the fuss I made about queueing the cleanup before inserting the rows, there's no way to do that when you're returning a cleanup function. That means newTestUser has to handle cleaning up if there are any problems. Just remember, "fewer lines" refers to your tests, not your helpers:

func newTestUser() (User, func()) {
    username := randStr()
    cleanup := func() {
        tearDown("user", "Username = $1", username)
    }

    doCleanup := true
    defer func() {
        if doCleanup {
            cleanup()
        }
    }()

    user, err := CreateUser(username)
    if err != nil {
        panic(err)
    }

    doCleanup = false
    return user, cleanup
}

Writing test functions like this really starts to click when your helpers start calling each other — if any part of the test function fails, they all collapse back down, leaving a clean database.

Go HTML Templates: Functions and Flow

Here's a Jinja2 template I ported to Go:

{% for user in users %}
<input type="checkbox"
       name="users"
       value="{{ user.guid }}"
       id="user-{{ user.guid }}"
       {% if user.guid in selected_users %} checked{% endif %}>
<label for="user-{{ user.guid }}">{{ user.name }}</label>
{% endfor %}

It renders a checkbox list of users, some of whom are pre-selected. I'm going to focus on the test {% if user.guid in selected_users %}. In Python, I passed a list of guids, and was done. There's no generic Go function for "slice contains," so I tried some silly things - first, I tried to register a template function.

Actually registering the function was easy: call Template.Funcs before parsing your templates, with a mapping of strings to your new functions. I gave up on this approach because the most generic contains function I could write would only work on []string containers. Since I couldn't make something globally useful, I tried passing a contains function to the template itself:

var ctx struct{
    Users []db.User
    Selected func(string)bool
}
ctx.Users, err = listUsers() // error handling left to the imagination
ctx.Selected = func(guid string) {
    for _, u := range selectedUsers {
        if u == guid {
            return true
        }
    }

    return false
}

templates.ExecuteTemplate(w, "form.html", ctx)

This had more promise — calling the function looked like this: {{ if index $.Selected .Guid }}. It worked, but ultimately, it was less clear than a map[string]bool, and more code. The test {{ if index .Map $key }} treats missing keys as false, so you only need to fill in the true keys.

In the Go code above, you can see that I'm rendering templates by applying them to anonymous structs. It's concise if you only call the template once; for templates I have to reuse (mostly forms), unexported structs work well.

Go HTML Templates: Applying Data

The last post on structuring Go templates left out the main feature of templating: varying output. Let's look at my header.html again:

<html>
<head><title>{{ . }}</title></head>
<body>

{{ . }} means "put the value of . here." So, if you apply that template to the string "Index", it outputs <title>Index</title>. The same goes for structs; if you apply a template to a struct{Title string}, {{ .Title }} prints the title field.

There's one catch with using ., and the solution was deep enough in the docs that I missed it completely. Here's something that might look right, but isn't:

type Folder struct {
    Name string
    Files []string
}

f := Folder{"/root", []string{".bash_history", ".bashrc"}}

// It's not okay to ignore errors... unless you're a blog post.
tmpl, _ := template.New("t").Parse(
    `Contents of {{ .Name }}:
    {{ range _, $f := .Files }}
        {{ .Name }}/{{ $f }}
    {{ end }}
    `)
tmpl.Execute(os.Stdout, f)

This doesn't work, because . changes in context of a range, to refer to the current list element. The fix is easy, but it's buried a few pages down in the docs: $ always holds the original value of .. Here's the correct template (or run it on play.golang.org):

`Contents of {{ .Name }}:
{{ range .Files }}
    {{ $.Name }}/{{ . }}
{{ end }}`

here's another context in which . changes — when you invoke another template. Earlier, I applied index.html to a Go string with {{ template "header.html" "Index" }}. This works the same way as applying templates to values in Go code. For a small project, passing a string works great. For a larger project, I expect to pass a larger struct into my header and footer to include CSS and JavaScript files that are needed on each page, rather than a static list in the header.

In the next post, I'll look at functions and control flow. Go templates have a stricter separation between code and formatting, and it's the biggest departure from working with Jinja2.

Go HTML Templates: Not Jinja2

My first impression of Go's template system was pretty rough — and to understand why, let's talk about Jinja2. To make a page in Python, I inherit most of the structure from a base template:

{# base.html #}
<html>
<head><title>{% block title %}OVERRIDE THIS{% endblock %}</title></head>
<body>

{% block content %}{% endblock %}
</body></html>

If I wanted to write an index.html, I'd start from base.html and override the sections marked with {% block %} and {% endblock %} tags:

{# index.html #}
{% extends "base.html" %}

{% block title %}Index{% endblock %}

{% block content %}
This is the index page!
{% endblock %}

When you render a template that extends another, it takes the base content and fills in any blocks overridden in the children. So, rendering index.html would look like:

<html>
<head><title>Index</title></head>
<body>

This is the index page!
</body></html>

Go's template system is not like that.

It took me back to my time with PHP — which is the last thing I ever expected to say about Go. To render a page with common scaffolding in html/template, you break base.html up into a header and footer, and include them at the top and bottom:

header.html:

<html>
<head><title>{{ . }}</title></head>
<body>

footer.html:

</body></html>

index.html:

{{ template "header.html" "Index" }}

This is the index page!

{{ template "footer.html" }}

This felt like a step back: Jinja2's blocks took time to learn, but they let you build HTML content in a way that made total sense — write elements, and slot them into place. Go's template system is less abstract — to add a common header, you put a {{ template "header.html" }} exactly where you want it to show up.

While I miss Jinja2's abstractions, I've also started to appreciate the directness of Go's templates. In the next post, I talk a bit about how Go applies templates to values — it's also very different from Jinja2, and (like many things in the Go standard library) as you get into the details, Go's choices start making more sense.

Writing Go Web Services

Google Go (“golang” to your search engine) is a promising language for web services, and I expect to use it professionally within the next five years. But, unlike other languages and frameworks with a longer history, there aren’t many people or resources to help me get productive. So, I’m going to learn the old fashioned way: by taking an app I wrote and rewriting it in Go, and documenting the ups and downs.

If you’re looking for the “right” way to do things, don’t take what I say too seriously; I’ll try to be thoughtful, but I expect to change my mind several times over the next few months. My goal here is to make a lot of beginner mistakes now, so I can be somewhat competent in the future. Perhaps the distant future.

Besides my main goal (make mistakes; learn; repeat), I have a few things in mind for this project. First, I’m using some code I already worked on — a Django-based web game with the working title of Asphodel. Second, I’m going to try and use the standard library as much as I can — I’m partly inspired by this post, by the author of the Martini middleware framework, on why standard interfaces are important. As a counterpoint, I’m also going to try some libraries — though compatibility with the standard library will influence my choices. I’m trying to learn to be productive, and using open source libraries is a big part of that. And finally, I’m going to balance pragmatism with my quest for knowledge: ultimately, I don’t mind writing some code that seems rushed if it gets the job done while I focus on something else. At worst, I rewrite it later; at best, I actually like the “quick” code I wrote.

I’m going to start this off with a post about HTML templates; coming from Jinja or Django templates, it took some getting used to.