It's easy to solve problems in go, but there's a big difference from works-on-my-machine to doesn't-break-at-2am-in-production. I'll walk you through the build tools, dependency management and deployment techniques to get your work into production; then we'll discuss how to do the service configuration, coordination and management tools to gracefully handle changes in production; and finally the metrics, logging and monitoring tools to tackle performance tuning, capacity planning and tracking down those bugs that only happen under production load.
This talk was presented at the GolangVan Kickoff meeting on 2014-02-05 17:00 Pacific.
This repo contains the source of the talk, written in present format. Please feel free to submit pull requests for typos, errors or improvements.
Slides: http://go-talks.appspot.com/github.com/josephholsten/prod-ready-go-svcs/prod-ready-svcs.slide Video: https://docs.google.com/file/d/0B4reqmVqk84qQnBSQXlfelc3QTA/preview
Some of the conversations during and after the talk made me go look up lots of relevent information. I hope it's useful to you.
Write your go packages to just work with go get and go build by
following the guidelines in Effective
Go. More detail is available
in the Go Language Specification.
In depth discussion of ARG_MAX, maximum length of arguments for a new
process,
and the googler's patch to linux which adds variable length argument
support.
If you want the strict syntax advantages of JSON with the readability and commentability of YAML, TOML config parsing is available in the github.com/bbangert/toml package. (Thanks to whoever in the audience asked!)
Lua support is available in the github.com/stevedonovan/luar package.
Blue Green Deployments are a common deployment pattern that requires orchestration or coordination.
Etcd is a highly consistant key-value store, with a client avalable in the github.com/coreos/go-etcd/etcd package.
Serf provides fast membership discovery, event messaging, and (in 0.4) key-value tags.
Oh, and remember how I said I once found a useful go library on
launchpad? That's Canonical's zookeeper client
gozk, available in the
launchpad.net/gozk package. (Thanks Neno
Lakinski!)
GNU libc explains why it's not possible to handle SIGKILL in its Termination
Signals
documentation. “In fact, if SIGKILL fails to terminate a process, that
by itself constitutes an operating system bug which you should report.”
The full explanation of why double-forking/daemonization isn't possible in go is available at Issue 227: runtime: support for daemonize
A battle tested and well maintained tool for doing the whole set of well-behaved daemon behaviour is available in daemonize. (Thanks Vincent Janelle!)
For Python fans, supervisord is an excellent process manager, much like god or bluepill. Of course, I like ye olde monit.
If you'd like more about my favorite monolithic beast of an init replacement: systemd, here's the most recent entry in the systemd for system administrators series: Socket Activated Internet Services and OS Containers. It contains references to all previous entries.
Also, since this talk, both Debian and Upstart have decided to switch to systemd. (Thanks Dan Ballard!)
I was wrong in the talk, the log/syslog package is totally amenable to the standard
log package API! Using the log package with a log/syslog output is as simple as
package main
import (
"log"
"log/syslog"
)
func main() {
logger, err := syslog.NewLogger(syslog.LOG_LOCAL0, 0)
if err != nil {
log.Fatal("Could not open connection to syslog")
}
logger.Print("Hello syslog!")
logger.Fatal("Goodbye syslog!")
}
The nginx bug for why they do not support syslog (in the open source
version) is #95 Integrate syslog patch into
modules.
Russ Cox's patch to allow pprof to work on Mac OS X is described in Hacking the OS X Kernel for Fun and Profiles (Thanks to Kamil Kisiel!)
- Document opscode's chef-server runit setup
To Github for hosting this code. To GoDoc.
And especially to everyone at Golang Vancouver who pointed me at all this stuff!