The Reliability of Go

As part of the Canonical Cloud Sprint taking place in San Francisco last week I attended Dave Cheney's talk at the GoSF meetup on the porting and extension of juju. Juju is an open-source cloud management and service orchestration tool that if you haven't heard of yet, you soon will have.

After the talk an audience member asked if Go was reliable. Having used Go in production for coming up to three years now, without incident, this came as a bit of a surprise to me. Prior to moving to Canonical I worked for one of the UK's largest market makers. A market maker is basically a wholesaler for institutional share traders and stock brokers. During my time there I replaced several key systems components with Go.

System monitoring.
The services within the system were monitored by a python script, pinging each node, discovering services, connecting the networking dots, checking health etc. Due to the complex nature of the system this script could take up to three minutes to scan nodes and process the results. The script would often stall whilst processing the vast amounts of data produced. After porting the script to Go the runtime was reduced to under one second, and we never saw a single stall when processing.

Data store.
A legacy relational database was replaced with a Go based key/value store to remove bottlenecks at market open. This service is now the key piece of architecture in the system, processing all inbound and outbound quotes/orders to and from the London Stock Exchange, the Multi-lateral Trading Facilities, and key exchanges across Europe. This service processes instructions at an average of 7 microseconds (actually, 6 under Go1.1), and never once failed, even at peaks, processing tens of thousands of instructions per second. Go is currently providing key infrastructure components within the finance industry.

As I left my old position I was in the process of swapping the messaging middleware and the third-party price feeds with services written in Go.

Go's adoption is gathering pace thanks to the terse syntax, straightforward powerful standard library, excellent tooling and concurrency primitives.

Go shows real maturity beyond its relatively young age due to the experience of the core development team and the consideration that is shown when introducing language constructs and extending the standard library.

I changed positions so that I could work with Go full-time. Ask anyone that knows me and they'll tell you that I'm not a betting man; you better believe Go is reliable.

Waiting for Golang channels to drain

Golang's channels easily map onto the producer consumer pattern. Lets assume that everything that is produced needs to be consumed, even if the process receives a SIGTERM.

The following example shows how we can register channels, monitor for a kill signal, and then wait for everything to be consumed.



package main

import (
"log"
"os"
"os/signal"
"reflect"
"syscall"
"time"
)

var (
BufferSize = 512
MaxIter    = 10
monitored  []interface{}
c          = make(chan int, BufferSize)
stopping   bool
)

func RegisterChannel(i interface{}) {
monitored = append(monitored, i)
}

func MonitorSigTerm() chan bool {
s := make(chan os.Signal, 1)
b := make(chan bool)
signal.Notify(s, syscall.SIGTERM)

go func(c chan os.Signal, b chan bool) {
_ = <-c
log.Println("Cleaning up")
// tell the caller
b <- true
for _, i := range monitored {
ch := reflect.ValueOf(i)
if ch.Kind() != reflect.Chan {
continue
}
prev := 0
iteration := 0
for {
if ch.Len() == 0 {
break
}

if prev == ch.Len() {
iteration++
// enough?
if iteration >= MaxIter {
log.Println("Dropping")
break
}
} else {
iteration = 0
}

prev = ch.Len()
log.Printf("Draining:%v\n", prev)
// other goroutines are working, let them
time.Sleep(1e9)
}
}
os.Exit(1)
}(s, b)
return b
}

func main() {
RegisterChannel(c)
stop := MonitorSigTerm()

go func() {
i := 0
for {
if stopping {
break
}
i++
c <- i
time.Sleep(1e9)
}
}()

go func() {
for {
i := <-c
log.Printf("rx:%v\n", i)
// slower read
time.Sleep(2e9)
}
}()

stopping = <-stop

// wait for cleanup to finish
select {}

}

Golang: Overflowing JSON

Go's json.Unmarshal function works perfectly by taking a JSON blob and attempting to create a known structure from the data.

type S struct {
    A int
    B string
} 

var s S

json.Unmarshal([]byte(`{"A": 42, "B": "b","C": "c"}`),&s) 

I recently had to construct a structure in this way, but also needed to store data that overflowed the structure.

I came up with this little function to do just that.

func UnmarshalJSON(src []byte, dst interface{}) (remainder []byte, err error) {
    var m map[string]interface{}
    o := make(map[string]interface{})
    _ = json.Unmarshal(src, &m)

    // put anything that doesnt match dst into a map
    rv := reflect.ValueOf(dst).Elem()
    for k, v := range m {
        if rv.FieldByName(k).IsValid() == false {
            o[k] = v
        }
    }

    // marshal the map to JSON
    remainder, _ = json.Marshal(o)

    // now fill the dst
    err = json.Unmarshal(src, dst)

    return
} 

Now if you set src bytes to the remainder bytes in the call you can use this function to 'consume' structures from JSON blobs.

src, _ = UnmarshalJSON(src,&myStruct)

GOBing Down Secure Websockets

Golang's gob package allows you to do something interesting when designing message protocols. 

Consider these structures:

Now I can construct a message and assign whichever message body is required. 



The gob package handles the passing of the populated and mismatched fields during the encode / decode.

Below is a noddy example of passing such message structures through a secure websocket. The code not only shows the gob package in action but also highlights how trivial golang makes the coding of a secure websocket for both the client and server.

Server:

Client:

Backwards Thinking

I'm a golang retard. I can't help it; I keep thinking backwards. I keep thinking in old paradigms.

I had to serialise a structure into JSON and wanted to suppress nil values.

type Data struct {

    Batch []map[string]interface{}

}

Now, I tried using the omitempty tag and that didn't work as the library only operates on structure fields and not the elements of the array.

So....... I sat there...... and I thought......... right I need a method on this structure that will serialise it into JSON and omit the individual array elements that are not yet populated.

Then I sat there and thought............that's a 'lot' of typing........there must be a better way...........

I sat.....

I sat a bit more......

I got up for a drink........my mind wondered..........to....... a video I had seen a few months back.........

Ken: “The thing I'm most pleased about is slices”.

BOOM!

batch.Batch = batch.Batch[0:pos]
b, e := json.Marshal(batch)

This may have been immediately obvious to many people; but it wasn't to me.

Part of my brain is still stuck in the old world of OOP. Oh, I must have a method to work on an objects data. I can't copy arrays; it's too expensive. I must type pages and pages of code to do 'real work'.

I think it's time for me to re-read effective go.

Perhaps we all should. We're probably missing more than we realise.

Message Hub

At the minute I'm having to push a lot of JSON around. To aid in this I've written msgHub. This utility sets up a TCP server allowing the pub/sub of generic JSON data. I've found it to be extremely useful; you might too.

msgHub -tcpPort=[port] -tcpDelimiter=[ascii code]

e.g. start on 12345 using line feed as delimiter

msgHub -tcpPort=12345 -tcpDelimiter=10

JSON message format:

{"event": "sub", "type": "messageType", "key": "messageKey"}

{"event": "unsub", "type": "messageType", "key": "messageKey"}

{"event": "pub", "type": "messageType", "key": "messageKey", ........ any json}

jsonSpew

Recently I've been throwing a lot of JSON down websockets for display using processing.js. To help in this I wrote a little Go script.  The script serves files over http, as well as relaying any JSON recevied down a websocket. The script can monitor data files and serves tcp to receive JSON updates. All the files are here. It was a bit of a stream of consciousness, so the structure could be better, and half the code should really be put in seperate packages, but hey.

go run jsonSpew.go -httpPort=8081 -fileList="/test.html:/processing.js:/jquery.min.js" -watchList="test.json" -tcpPort=12345

2012/08/27 12:02:39.482810 jsonSpew 1.0.a

2012/08/27 12:02:39.482989 Registered /test.html

2012/08/27 12:02:39.483003 Registered /processing.js

2012/08/27 12:02:39.483014 Registered /jquery.min.js

2012/08/27 12:02:39.483037 Watching:test.json

2012/08/27 12:02:39.484011 Serving tcp on [127.0.1.1] 12345

2012/08/27 12:02:39.484050 Serving http on [127.0.1.1]:8081

So now we have a process sitting on 8081 serving up test.html,processing.js, and jquery.min.js. The process is also serving up tcp connections on 12345, and watching test.json.

Now, hitting 127.0.1.1:8081/test.html will give us a a view onto the JSON data available. Test.html simply iterates over the JSON and display the values using processing.js's draw loop.

Now we can send some updates.

netcat 127.0.1.1 12345 < test2.json

netcat 127.0.1.1 12345 < test3.json

touch test.json

The flexibility of Go's standard library meant that I could knock this useful little utility up in an afternoon.