defer Panic

Compile Time Code Weaving in Go

One of the things in Go that frustrates me is packaging and it’s not because I have 20 dependencies that I’m including in each project. I don’t have a problem with that cause we don’t do that at DeferPanic.

What we do do is routinely jump into third party codebases we’ve never seen with tens of thousands of lines of code and we are told it “has problems”.

How do we find those problems so fast? More importantly, how do we fix these problems so fast? Sure, we instrument these codebases with our APM agent that everyone knows but we also apply other behavior to the codebases to get at problems fast.

The Problem

Debugging

What type of behavior do we apply?

Logging

We might wish to know how data flows throughout a callgraph. (eg: Does this password ever hit this function?)

Performance Analysis

We know a particular function is ‘heavy’ but how many times is it called in the past 5 minutes? How long does it take each time? Counting the calls and collecting the latency of the call gives us clues.

Data Validation

A codebase might be lacking in various forms of coverage and subtle bugs like converting “$10” to the integer ‘10’ via strconv.Atoi could easily be recognized by applying strict input validation to various functions.

Data/Behavior Mutation

Like the above, perhaps you know a particularly bad place but you are having trouble replicating it’s behavior - mutating input at runtime is one way of exposing this. Doing this in Go is a bit of a pain.

Security

Misconfigured authorization can lead to all sorts of interesting problems and expose APIs that you thought were locked down. Gradually changing authorization can hunt down these problems but it’s time-consuming and error-prone to do it manually.

This is but a short and incomplete list and most readers will probably recognize some of these common debugging/troubleshooting solutions.

Unfortunately, most of the time these solutions are applied manually.

First you have to code it in. You’ll probably miss quite a lot of spots.

Then you have to hunt down your bug. Still haven’t found it? Oh wait, we screwed up the code we put in. Let’s fix that.

Days later you have valiantly slayed the dragon by finding and fixing the performance problem or error and found/fixed 5 other problems, because problems are generally Knuthian in nature - they are trees and you don’t know if you are root node or not.

Now we have to rip out the code we put in there. Then we notice the tests aren’t passing. Oh yeh, we modified that one method with a different receiver. Crap. Now we have to fix that. Crap, now we have to re-write these tests.

This is the problem with manually applying debugging behavior like this.

It’s time consuming and difficult to work with.

Extending 3rd Party Packages

My other gripe with Go also deals with packages. Let’s say you want to time the latency of database calls. Your adapter might provide a simple logging functionality for this. However, maybe you want to pipe that to statsd or something instead? Now what?

Well, you could fork the package and apply the behavior or you could create another package that wraps the behavior then modify every single place in the code you need to modify.

There have been other tools written to deal with one-time changes like go fix && go fmt but they are more for simple one time changes - not applying complex changes on demand.

There really aren’t that many other good answers to this question. These methods are time consuming to implement, limited in scope and chances are if you are shoehorning your code in, someone else is as well - why isn’t the code shared?

Solution

I was talking to a guy from London a few months ago and he asked me if I had ever heard of AspectJ. I had and I had also heard lots of trash talk from JVM developers who had used it.

It wasn’t the fact that AOP itself is considered ‘bad’ - it is that it had gotten abused in many different ways because it’s so powerful. You could say that about lots of things though. Used appropriately, it’s actually a highly useful and effective way of thinking about your projects.

Go is considered a very pragmatic language and opinionated to the tilt to boot. I believe it’s that trait that really attracted me to it. I code to “Get Shit Done”™. When I was younger I might have explored the more creative side of software but now I code first and foremost to get the job done and I like to work with powertools and earth movers. I don’t have time to sit around and bang rocks together.

I have a responsibility to not waste time on something that can and should be automated.

I consider code weaving one of those powertools.

Just like any developer I have a few directory trees littered with quick scripts and common tools that I use to do things fast. Like other developers I have just as large of a collection of quick ‘notes’ (mostly code snippets) that I use often enough to make a point of saving them for future use.

Around a month ago I spent a few nights hacking together a generic common interface/tool to make this code more standard/programmatic. It has the ability to intelligently apply arbitrary code to a large codebase programatically at compile-time with no code residue left in the source tree.

The result was goweave.

Goweave allows the end developer to place one or more .weave files in their project root and instantly apply code to anywhere in their codebase they see fit.

No coding whatsoever - just pure automagic sauce.

Logging Strconv

Let’s look at a quick example.

package main

import (
  "fmt"
  "strconv"
)

func blah() {
  fmt.Println(strconv.Itoa(2))
}

func main() {

  fmt.Println(strconv.Itoa(44))

  blah()
}

Fairly simple - this prints out two integers - 44 && 2.

Now let’s weave in some before call advice.

aspect {
  pointcut: call(strconv.Itoa(int i))
  advice: {
    before: {
      fmt.Println("strconv occuring")
    }
  }
}

Then let’s run it.

➜  strconvs  goweave
2015/07/12 12:51:58 goweave v0.1
2015/07/12 12:51:58 main.weave
➜  strconvs  ./strconvs
strconv occuring
44
strconv occuring
2

What happened? Goweave took a dump of the AST and it pre-pended code to where we specified allowing us to quickly and cleanly apply our behavior in our project without modifying one single line of code in our original project.

This is how we find and solve problems fast at DeferPanic. We have a general rule that if you find yourself doing something more than a few times it needs to be automated.

Now, we’re making a general purpose open source tool so that everyone can do this and not spend hours on something that should take seconds.

Input Validation/Run Time Method Stubbing

Let’s look at another example.

package main

import (
  "fmt"
)

func blah(stuff string) bool {
  return true
}

func main() {
  fmt.Println("hi")

  fmt.Println(blah("stuff"))
  fmt.Println(blah("otherstuff"))

}

In this contrived example we return true automatically whenever this method is called. When testing/debugging sometimes we wish to stub out whole blocks of code in many places. Sometimes this is not so easy to do as you can’t just comment it out without the compiler complaining. Even if you do that you are sure to miss a few places when you un-comment it back in.

Lastly, maybe there is a case where you want it to be nop’d out 90% of the time but there is 10% of the time you wish it to actually execute.

If we apply this before advice:

aspect {
  pointcut: execute(blah(stuff string))
  advice: {
    before: {
      if stuff == "stuff" {
        return false
      }
    }
  }
}

We see when weaving this advice in that our first call to blah returns false but the second call returns true.

➜  t  ./t
hi
false
true

You could imagine that you have several long running functions that you would like to stub out in development. This allows you to do that without contaminating your source control and preventing those changes from finding their way into production.

Our time to Aha! has been dramatically reduced.

Database Query Latency Timing

One more example just to show the versatilty of the tool.

Let’s suppose we have a database and we know there are a few queries that are being slow. We either don’t have logging with our adapter or we don’t want to put that code into production.

Let’s simulate a two second long database query:

package main

import (
  "database/sql"
  _ "github.com/lib/pq"
  "log"
)

func main() {
  db, err := sql.Open("postgres", "dbname=dptest sslmode=disable")

  var id int
  var sleep string

  err = db.QueryRow("select 1 as num, pg_sleep(2)").Scan(&id, &sleep)
  if err != nil {
    log.Println("oh no!")
  }
}

On every call of QueryRow we’d like to log the latency of each query.

aspect {
  imports (
    "time"
  )
  pointcut: call(QueryRow(s string))
  advice: {
    before: {
      startTime := time.Now()
    }
  }
}

aspect {
  imports (
    "fmt"
    "time"
  )
  pointcut: call(QueryRow(s string))
  advice: {
    after: {
      endTime := time.Now()
      weavet := int(((endTime.Sub(startTime)).Nanoseconds() / 1000000))
      fmt.Printf("query took %d ms\n", weavet)
    }
  }
}

When we run this we get:

➜  t  ./t
query took 2006 ms

Which is exactly what we wanted - no code - no production issues - no wasted time hunting down anything.

Feeling the power here yet?

What is Aspect Oriented Programming?

So by now you might know how the tool is working but when looking at the half json, half go (open to discussion on ways forward btw) you are probably asking yourself - what the hell is a pointcut? What is advice?

How do I use this thing?

I didn’t have a ton of time to spend for my talk at GopherCon this past week and there was quite a bit of interest so felt it was necessary to expand a little bit. So let’s briefly go over some of these definitions now.

Join Point

A join point is put simply - a place in your code where you can apply ‘behavior’. For example a db.QueryRow call is a join point. If you compared join points and point cuts to regex the regex itself is a pointcut while the join point is the places in your code that it refers to. So every place in your code that you call db.QueryRow would be a join point.

Point Cut

If the joinpoint references the literal places in your code the point cut is a way of telling the pre-processor where to apply said behavior. You can think of it as a cross between a regex && an xpath expression only for your program’s abstract syntax tree.

Pointcuts will refer to a set of joinpoints to apply behavior to.

We currently support three pointcut primitives right now:

If we had the following code:

func blah() {
  a()
  b()
  c()
}

func main() {
  blah()
}

Call

Calls are whenever you call a method/function. You can apply behavior {before, after, around} a function is called. The code’s scope is outside the function not within it.

  call(blah())

Would place the code before/after the call in main().

Execute

Executes are basically the same as calls except the advice is put inside the function. So you have local variable access and local scope inside the function.

  execute(blah())

Would place the code within blah() itself.

Within

Within also happens within a function but instead of happening at the beginning or the end of a function it happens on every method call within that method.

Lastly,

  within(blah())

would place code before/after a(), b(), and c() inside blah().

Advice

The advice is simply code you wish to apply. There are primarily three types of advice: {before, after, around}.

Before advice applies code before each joinpoint. After advice applies after and around wraps the method.

Aspect

The aspect is the combination of the pointcut && advice together that applies behavior to your code.

There may be multiple aspects in a .weave file but generally if they are unrelated you’d probably expect them to be in different .weave files.

A Word to the Wise

I wrote the majority of this code in around 2 nights and there is a lot of refactoring for the enterprising amongst us to do. Most of the design decisions were not decisions at all - merely me trying to do the simplest thing that could work. If you are looking for an open source project to hack on by all means hack it up!

There is currently lots of in-efficient code, missing documentation, and large swaths of code cleanup to do - we could use YOUR help! There is honestly a lot left on the table because the problem domain is rather large and some of the individual problems are non-trivial to solve.

For more current information please visit the goweave github . The project is 100% open source and we happily accept pull requests.

The Loom

One of the rationales behind this tool is that you don’t want to have to be ripping code in/out of your app everytime you need to do something. Likewise, there is no need to re-invent the wheel each and every time for what can be common usecases.

The Loom provides a central place for examples of using goweave and for sharing common aspects. Got something to contribute? Want to send us your snippets directory as code weaves? Open a pull request!

Vote on HN
Looking to monitor your go applications?