Go - Protocol Buffer, Protoc tool & generated structs

Protocol Buffers with Go

So, today I'm having a look at protocol buffers with generated output in Go. Let's start at the top with:

Q. What is a protocol buffer?
Well, google says that it's a flexible, efficient, automated mechanism for serializing structured data, like XML but smaller, faster and simpler.

Q. That's great, thanks. Now what does it mean?
Well, seemingly we can define once how we'd like our data to be structured. Then we just simply use the specially generated source code to read and write the data a variety of streams in a variety of languages. From this we can summarize that they allow us to define the data contract between systems or services. Whereas data in formats like XML or JSON are still text-based protocol buffers are binary.

Q. What are the benefits of this?

Stronger interfaces are a good thing.
Smaller size than text based alternatives are another advantage.
typically they are speed optimised.
type and order are in place so ambiguity is minimised.
data access classes generated are considered to be easier to work with.

A tight interface with our data is absolutely essential for designing microservices well.

Protocol buffers in Go can be used on different transport layers such as HTTP/2 or AMQP (Advanced Message Queuing Protocol) where the binary data being transferred can only be understood between the client and the server.

Protocol Buffer language

I know this is a bit wordy with no code yet and it can feel a bit TL;DR but stick with me. A protocol buffer is a file with a minimalist language syntax we use to define our data structures. We compile this file and it generates a new file for a given programming language - in our case this means generating .go files. The language gives us types that we will use to create interfaces. This is called a protobuf which is one of the terms of the technology. Let's see an example of this for version 3.



syntax 'proto3';

# note that we have type, name, index position in the file for each entry
message SomeInterface {
    int id = 1;
    int code = 2;
    string name =3;
    string nickname = 4;
}

This file when compiled will give us Go style structures.

Key types:

Scalar values.
Enumerations and repeated values.
nested fields.

Enumerations

A numerical ordering of a given set of elements, defaults from 0 to n. This is fairly standard and should remind you of an iota in Go. Where it differs is by using option allow_alias = true we can have the same numeric value assigned to more than one option. essentially allowing duplicates.



syntax = 'proto3';

message Move{
    enum Moves{
        ROCK = 0
        PAPER = 1
        SCISSORS = 2
    }
}

Repeated

Repeated values are much like an array in JSON. We use the [values, values, values] syntax.

Nested fields

We can see below how to define nested types. Below we have a person who can have multiple contact numbers.



syntax = 'proto3';

message MyPerson {
    string name = 1;
    string nickname = 2;
    repeated ContactType contacts = 3;
}

message ContactType {
    string desc =1;
    string number =2;s
}

Protoc and Protocol Buffer compilation

Install the protoc tool for your system.
save your protobuf files with a .proto extension.
compile to target a particular language, for us that is the mighty Go!
Import structs from generated file and add the required data

Here we go, the generating part....

So now you want to generate the Go files, this can be real headache. I must have dropped a few pounds in the swear jar here but if you're trying to do this (or you're me several months from now) remember to put the option go_package = "github.com/USERNAME/MODULE_NAME"; in the .proto file.

By now this will look like:



syntax = 'proto3';

option go_package = "github.com/USERNAME/MODULE_NAME";

package protofiles;

message MyPerson {
    string name = 1;
    string nickname = 2;
    repeated ContactType contacts = 3;
}

message ContactType {
    string desc =1;
    string number =2;s
}

to do the generation of our files we can run the following in the CLI.



protoc --go_out=. --go_opt=Mpaths=source_relative  FILENAME.proto

Afterwards, you'll have generated a new folder set in the protofiles directory and in the project name subfolder you'll find a person.pb.go. This autogenerated code should look similar to below.



// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
//  protoc-gen-go v1.27.1
//  protoc        v3.17.3
// source: mock.proto

package protocol_buffers

import (
    protoreflect "google.golang.org/protobuf/reflect/protoreflect"
    protoimpl "google.golang.org/protobuf/runtime/protoimpl"
    reflect "reflect"
    sync "sync"
)

const (
    // Verify that this generated code is sufficiently up-to-date.
    _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
    // Verify that runtime/protoimpl is sufficiently up-to-date.
    _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
)

type MyPerson struct {
    state         protoimpl.MessageState
    sizeCache     protoimpl.SizeCache
    unknownFields protoimpl.UnknownFields

    Name     string         `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
    Nickname string         `protobuf:"bytes,2,opt,name=nickname,proto3" json:"nickname,omitempty"`
    Contacts []*ContactType `protobuf:"bytes,3,rep,name=contacts,proto3" json:"contacts,omitempty"`
}

func (x *MyPerson) Reset() {
    *x = MyPerson{}
    if protoimpl.UnsafeEnabled {
        mi := &file_mock_proto_msgTypes[0]
        ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
        ms.StoreMessageInfo(mi)
    }
}

func (x *MyPerson) String() string {
    return protoimpl.X.MessageStringOf(x)
}

func (*MyPerson) ProtoMessage() {}

func (x *MyPerson) ProtoReflect() protoreflect.Message {
    mi := &file_mock_proto_msgTypes[0]
    if protoimpl.UnsafeEnabled && x != nil {
        ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
        if ms.LoadMessageInfo() == nil {
            ms.StoreMessageInfo(mi)
        }
        return ms
    }
    return mi.MessageOf(x)
}

// Deprecated: Use MyPerson.ProtoReflect.Descriptor instead.
func (*MyPerson) Descriptor() ([]byte, []int) {
    return file_mock_proto_rawDescGZIP(), []int{0}
}

func (x *MyPerson) GetName() string {
    if x != nil {
        return x.Name
    }
    return ""
}

func (x *MyPerson) GetNickname() string {
    if x != nil {
        return x.Nickname
    }
    return ""
}

func (x *MyPerson) GetContacts() []*ContactType {
    if x != nil {
        return x.Contacts
    }
    return nil
}

type ContactType struct {
    state         protoimpl.MessageState
    sizeCache     protoimpl.SizeCache
    unknownFields protoimpl.UnknownFields

    Desc   string `protobuf:"bytes,1,opt,name=desc,proto3" json:"desc,omitempty"`
    Number string `protobuf:"bytes,2,opt,name=number,proto3" json:"number,omitempty"`
}

func (x *ContactType) Reset() {
    *x = ContactType{}
    if protoimpl.UnsafeEnabled {
        mi := &file_mock_proto_msgTypes[1]
        ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
        ms.StoreMessageInfo(mi)
    }
}

func (x *ContactType) String() string {
    return protoimpl.X.MessageStringOf(x)
}

func (*ContactType) ProtoMessage() {}

func (x *ContactType) ProtoReflect() protoreflect.Message {
    mi := &file_mock_proto_msgTypes[1]
    if protoimpl.UnsafeEnabled && x != nil {
        ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
        if ms.LoadMessageInfo() == nil {
            ms.StoreMessageInfo(mi)
        }
        return ms
    }
    return mi.MessageOf(x)
}

// Deprecated: Use ContactType.ProtoReflect.Descriptor instead.
func (*ContactType) Descriptor() ([]byte, []int) {
    return file_mock_proto_rawDescGZIP(), []int{1}
}

func (x *ContactType) GetDesc() string {
    if x != nil {
        return x.Desc
    }
    return ""
}

func (x *ContactType) GetNumber() string {
    if x != nil {
        return x.Number
    }
    return ""
}

var File_mock_proto protoreflect.FileDescriptor

var file_mock_proto_rawDesc = []byte{
    0x0a, 0x0a, 0x6d, 0x6f, 0x63, 0x6b, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x0a, 0x70, 0x72,
    0x6f, 0x74, 0x6f, 0x66, 0x69, 0x6c, 0x65, 0x73, 0x22, 0x6f, 0x0a, 0x08, 0x4d, 0x79, 0x50, 0x65,
    0x72, 0x73, 0x6f, 0x6e, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01,
    0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x1a, 0x0a, 0x08, 0x6e, 0x69, 0x63, 0x6b,
    0x6e, 0x61, 0x6d, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x6e, 0x69, 0x63, 0x6b,
    0x6e, 0x61, 0x6d, 0x65, 0x12, 0x33, 0x0a, 0x08, 0x63, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74, 0x73,
    0x18, 0x03, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x17, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x66, 0x69,
    0x6c, 0x65, 0x73, 0x2e, 0x43, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74, 0x54, 0x79, 0x70, 0x65, 0x52,
    0x08, 0x63, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74, 0x73, 0x22, 0x39, 0x0a, 0x0b, 0x43, 0x6f, 0x6e,
    0x74, 0x61, 0x63, 0x74, 0x54, 0x79, 0x70, 0x65, 0x12, 0x12, 0x0a, 0x04, 0x64, 0x65, 0x73, 0x63,
    0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x64, 0x65, 0x73, 0x63, 0x12, 0x16, 0x0a, 0x06,
    0x6e, 0x75, 0x6d, 0x62, 0x65, 0x72, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x06, 0x6e, 0x75,
    0x6d, 0x62, 0x65, 0x72, 0x42, 0x23, 0x5a, 0x21, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63,
    0x6f, 0x6d, 0x2f, 0x65, 0x6c, 0x65, 0x73, 0x71, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f,
    0x6c, 0x2d, 0x62, 0x75, 0x66, 0x66, 0x65, 0x72, 0x73, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f,
    0x33,
}

var (
    file_mock_proto_rawDescOnce sync.Once
    file_mock_proto_rawDescData = file_mock_proto_rawDesc
)

func file_mock_proto_rawDescGZIP() []byte {
    file_mock_proto_rawDescOnce.Do(func() {
        file_mock_proto_rawDescData = protoimpl.X.CompressGZIP(file_mock_proto_rawDescData)
    })
    return file_mock_proto_rawDescData
}

var file_mock_proto_msgTypes = make([]protoimpl.MessageInfo, 2)
var file_mock_proto_goTypes = []interface{}{
    (*MyPerson)(nil),    // 0: protofiles.MyPerson
    (*ContactType)(nil), // 1: protofiles.ContactType
}
var file_mock_proto_depIdxs = []int32{
    1, // 0: protofiles.MyPerson.contacts:type_name -> protofiles.ContactType
    1, // [1:1] is the sub-list for method output_type
    1, // [1:1] is the sub-list for method input_type
    1, // [1:1] is the sub-list for extension type_name
    1, // [1:1] is the sub-list for extension extendee
    0, // [0:1] is the sub-list for field type_name
}

func init() { file_mock_proto_init() }
func file_mock_proto_init() {
    if File_mock_proto != nil {
        return
    }
    if !protoimpl.UnsafeEnabled {
        file_mock_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} {
            switch v := v.(*MyPerson); i {
            case 0:
                return &v.state
            case 1:
                return &v.sizeCache
            case 2:
                return &v.unknownFields
            default:
                return nil
            }
        }
        file_mock_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} {
            switch v := v.(*ContactType); i {
            case 0:
                return &v.state
            case 1:
                return &v.sizeCache
            case 2:
                return &v.unknownFields
            default:
                return nil
            }
        }
    }
    type x struct{}
    out := protoimpl.TypeBuilder{
        File: protoimpl.DescBuilder{
            GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
            RawDescriptor: file_mock_proto_rawDesc,
            NumEnums:      0,
            NumMessages:   2,
            NumExtensions: 0,
            NumServices:   0,
        },
        GoTypes:           file_mock_proto_goTypes,
        DependencyIndexes: file_mock_proto_depIdxs,
        MessageInfos:      file_mock_proto_msgTypes,
    }.Build()
    File_mock_proto = out.File
    file_mock_proto_rawDesc = nil
    file_mock_proto_goTypes = nil
    file_mock_proto_depIdxs = nil
}

I'll be using these in a further examples where we get to write some Go and use this, but we've seen the phaff and pain required to setup and generate your files. I hope if you're dabbling in this area at all that is somewhat useful to you.

Best wishes.

Blog