Background

I've made a few attempts to learn go, including some online courses. These fizzled out after some time. What I've generally found is that to start learning a new programming language, you need a specific goal and time to focus on that goal. If you have an ambitious goal, you won't have enough time. Things like "finish this course" is not really a good enough goal for me.

Program goal

I finally settled on a goal, a radio net checkin manager. In amateur radio, we often have what we call "directed nets", which is something like a conference call, but one person is designation as "net control". They start the by making an annoucement on the air, ask for check-ins, and they direct the conversation. A typical small net requests check-ins, takes
5 or so comments, asks for more check-ins, and continues taking comments in 2 or more rounds, before "closing" the net.

After the net, they send out a net report stating when the net started, finished, and who checked in. They may also inclue a roster of who checked in, as well as any noteworthy
news from the net. These simple nets are usually simple to track and write a quick email summarizing the net activity.

A longer, more complicated net may change net control operators several times, and regular operators may check in and out of the net multiple times.

Specific Goals

I want a web app to

Allow members to schedule nets, list upcoming and past nets.
Record each action (net open, checkin, comments, change net control) for the net
Able to view all activity for a single net, or all activity across multiple nets
Have a simple "after action" net report form to create and record net activity all at once
1. The net start and stop
2. Net control operator(s)
3. Net checkins (early, regular)
4. Net comments
5. Upon submitting the form, a planned net will be recorded, with individual activites populated, timestamps calculated based on net start/stop fields.
Have an interface to edit net activity after the fact. The simple net after action would go here.
Have an interface to record net activity in real time, timestamps defaulting to "now". This would probably be about the same form as the net edit page.

I'm also circumventing a lot of my normal process. Like I am an API-first type of backend developer. I typically write the API, generate the swagger documentation, and work on the frontend last. Either another team is doing the frontend, or I might make one as way of demonstrating the API. In this case, I am writing everything for the frontend, pretty much all HTML forms. Things like auth, testing, and api docs are in the "nice but not needed for MVP" (I can always implement basic http auth if needed).

A future goal of this project is to setup a local call sign lookup database, with the ability to lookup from a service like hammcall.dev.

Timing

I have about 5 days while on vacation, Monday through Friday. Saturday I drive home. Yes, I'm spending part of my vacation time writing code, but I really take pleasure in producing things. This is a bit of a gamble, because if I fail to make most of this work, I might get frustrated.

Process

The consensus of the go community is to eschew heavy frameworks and learn by writing from scratch. In fact, as someone from Python that moved from Django to Flask to FastApi, I can agree with that. However, I have 5 days, so I dived through a bunch of options and settled on Buffalo. It has a router, model, and controller that seems familiar to me, and some django-like generators. I don't know if it's the best framework, too much framework, if it will get in my way, etc. There were some other frameworks I wanted to check out such as Beego and Gorilla. Beego seems even more like django, and Gorilla calls itself a toolkit (you pick and choose). Martini reminded me a lot of Flask. I ended up going with Buffalo because it claimed hot code reload, a frontend pipeline, uses gorilla under the hood for routing, and packs an ORM with it.

Database will be sqlite3 to start, since if you're not needing HA, sqlite is pretty performant and low friction.

Getting Started

The getting started documents worked, I was able to do the install, generate my project, add a few temporary custom pages. There is a main app.go for your router, and each route is mapped to a Handler. The handler has a Context which includes all the variables it should need, and you can access that context within a plush template.

Database

I did end up recreating the project a time or two to before setting the db-type to sqlite database instead of Postgres
default.

buffalo new --db-type sqlite3 gonetninja

I made a data/ directory for my database files, added those to .gitignore, edited the database.yml file to correct the path, and then ran buffalo pop create to create the
empty database files. With sqlite, I don't really think you need to do that last step, since most sqlite drives create the file automatically.

models

I went to do database models. I found this is where the documentation first let me down. If you soda generate mode {modelname}, it creates a relatively empty model and some migration files with a base set of columns (ID, CreatedAt, UpdatedAt). If I edit that model and run try to regenerate migration files, those files come out empty.

In tools like django, sqlalchemy, and alembic, I expect to make my model, then make my migrations, which will create a file with the changes necessary to get the database up to
speed.

buffalo pop generate model foo -d  # creates model, migration up, migration down
# edit model to add Name column
buffalo pop generate fizz foo2  # creates empty migration files

I also tried making the table first, and pop/soda does not inspect the database to make the models or fixtures.

sqlite> create table otherfoo (id uuid NOT NULL, created_at timestamp NOT NULL, updated_at timestamp NOT NULL, name character varying(255));

Followed by

buffalo pop generate model otherfoo

So for now, I ended up creating the for Nets and I created the tables by hand. I don't have migration files for this table, I'll just have to come back and figure out how that is supposed to work later.

My modified models/net.go:

type Net struct {
    ID           uuid.UUID `json:"id" db:"id"`
    CreatedAt    time.Time `json:"created_at" db:"created_at"`
    UpdatedAt    time.Time `json:"updated_at" db:"updated_at"`
    Name         string    `json:"name" db:"name"`
    PlannedStart time.Time `json:"planned_start" db:"planned_start"`
    PlannedEnd   time.Time `json:"planned_end" db:"planned_end"`
}

My sql by hand:

sqlite> CREATE TABLE nets (id uuid NOT NULL, created_at timestamp NOT NULL, updated_at timestamp NOT NULL, name character varying(255), planned_start timestamp, planned_end timestamp);
sqlite> insert into nets values (uuid(), strftime('%Y-%m-%d %H-%M-%S','now'), strftime('%Y-%m-%d %H-%M-%S','now'), "test one", strftime('%Y-%m-%d %H-%M-%S','now'), strftime('%Y-%m-%d %H-%M-%S','now'));
sqlite> insert into nets values (uuid(), strftime('%Y-%m-%d %H-%M-%S','now'), strftime('%Y-%m-%d %H-%M-%S','now'), "test two", strftime('%Y-%m-%d %H-%M-%S','now'), strftime('%Y-%m-%d %H-%M-%S','now'));
sqlite> insert into nets values (uuid(), strftime('%Y-%m-%d %H-%M-%S','now'), strftime('%Y-%m-%d %H-%M-%S','now'), "test three", strftime('%Y-%m-%d %H-%M-%S','now'), strftime('%Y-%m-%d %H-%M-%S','now'));
sqlite> insert into nets values (uuid(), strftime('%Y-%m-%d %H-%M-%S','now'), strftime('%Y-%m-%d %H-%M-%S','now'), "test the fourth", strftime('%Y-%m-%d %H-%M-%S','now'), strftime('%Y-%m-%d %H-%M-%S','now')); 
sqlite> .dump nets
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE nets (id uuid NOT NULL, created_at timestamp NOT NULL, updated_at timestamp NOT NULL, name character varying(255), planned_start timestamp, planned_end timestamp);
INSERT INTO nets VALUES('c31911fc-5d3c-4b18-b4b1-1e081aa6effd','2022-06-20 23-47-29','2022-06-20 23-47-29','test one','2022-06-20 23-47-29','2022-06-20 23-47-29');
INSERT INTO nets VALUES('ccad3aad-c9ea-4891-a604-8d02e0968ce8','2022-06-20 23-47-47','2022-06-20 23-47-47','test two','2022-06-20 23-47-47','2022-06-20 23-47-47');
INSERT INTO nets VALUES('66b685a9-ea20-4a14-b766-4d23f362be4b','2022-06-20 23-48-17','2022-06-20 23-48-17','test three','2022-06-20 23-48-17','2022-06-20 23-48-17');
INSERT INTO nets VALUES('2bc9c10a-4056-45bc-bbb6-6a482bbb30a9','2022-06-21 13-20-20','2022-06-21 13-20-20','test the fourth','2022-06-21 13-20-20','2022-06-21 13-20-20');
COMMIT;
sqlite>

I am pretty sure at this point in time, I have an issue with my timestamp and timezones. To be solved later ^tm.

Displaying Data

Now that I have my Nets, time to read it to a page. The querying page gave some basic idea, but not enough to get me anywhere. This is probably because of a lack of familiarity with go. I remembered from other learning that methods will return an err, and if you find that set, you throw an exception. But their query example for all rows is basically this:

// To retrieve records from the database in a specific order, you can use the Order method
users := []User{}
err := models.DB.Order("id desc").All(&users)

I could see it emitting SQL statements in debug logs, and I even learned to set the "users" (net in my case) to the context. But ultimately found I needed to setup a connection,
wrap that err inside an if to get a proper raise, then set the context.

func NetListHandler(c buffalo.Context) error {
    tx := c.Value("tx").(*pop.Connection)
    nets := models.Nets{}
    if err := tx.Order("name").All(&nets); err != nil {
        return errors.WithStack(err)
    }
    c.Set("nets", nets)
    return c.Render(http.StatusOK, r.HTML("home/netlist.plush.html"))
}

The "models.Nets" is predefined. A number of other examples had you do something like nets := []models.Net{}, but that already existed inside the generated models/nets.go.

The plush template will know of a "nets" variable, which you can iterate over.

        <%= for (net) in nets { %>
        <tr>
          <td class="left">
            <a href="/nets/<%= net.ID %>"><%= net.Name %></a>
          </td>
          <td>
            <%= net.PlannedStart %>
          </td>
          <td>
            <%= net.PlannedEnd %>
          </td>
        </tr>
        <% } %>

Some examples I saw had <%= variablename => which gave me all sorts of interesting errors, but the <%= variablename %>

Paths

I hard-coded /nets/{netid} in the url, but the bufallo/gorilla router lets me name these paths. There are auto-generated ones, but I like naming them explicitly.

        app.GET("/nets", NetListHandler).Name("netlistPath")
        //app.GET("/nets/{id}", func(c buffalo.Context) error {
        //  return c.Render(200, r.String(c.Param("id")))
        //}).Name("netViewPath")
        app.GET("/nets/{id}", NetHandler).Name("netViewPath")

Then back in my template, I can put the net.ID as a parameter.

   <a href="<%= netViewPath({id: net.ID}) %>"><%= net.Name %></a>

Single entry

To view a single net, it's almost identical, but I get the id from the router.

app.GET("/nets/{id}", NetHandler).Name("netViewPath")

The router add Params and Param to the context, so c.Param("id") gets me the {id} part of the path.

Additionally, I learned that models.DB replaces that tx pop.Connection!

func NetHandler(c buffalo.Context) error {
   //tx := c.Value("tx").(*pop.Connection)
   net := models.Net{}
   //if err := tx.Find(&net, c.Param("id")); err != nil {
   //   return errors.WithStack(err)
   //}
   if err := models.DB.Find(&net, c.Param("id")); err != nil {
     return errors.WithStack(err)
   }

   c.Set("net", net)
   return c.Render(http.StatusOK, r.HTML("home/netview.plush.html"))
}

Hot Reload Sadness

Running in Development says

The dev command will watch your .go and .html files and the asset folder by default. It will rebuild and restart your binary for you automatically, so you don’t have to worry about such things.

In my experience, only html file edits were live reloaded. I don't believe it actually rebuilt the binary, but just sees the new template. I didn't see any relevant issues, except for #602 from 2017 related to docker and NFS mounts. But this could be a new issue with MacOS Montery (on M1). If it keeps bugging me, I'll dig deeper and maybe submit a request. It's going to be related to inotify/fsnotify.

https://github.com/gobuffalo/buffalo/issues/510
https://github.com/fsnotify/fsnotify/issues/152

(Timer note: started tracking 6/21 @ 7:03PM. Probably have 3 hours in?)

go: upgraded github.com/fsnotify/fsnotify v1.5.1 => v1.5.4
go: upgraded golang.org/x/sys v0.0.0-20211205182925-97ca703d548d => v0.0.0-20220412211240-33da011f77ad

Create a Net

After creating the form with a name, started time and stopped time, making CreateNetHandler was really straightforward.

The models.DB.ValidateAndCreate(net) kind of exists by default. My database however, didn't originally make "name" a required field.


func CreateNetHandler(c buffalo.Context) error {
    net := &models.Net{}
    if err := c.Bind(net); err != nil {
        return err
    }

    newId, err := uuid.NewV1()
    if err != nil {
        return err
    }
    net.ID = newId

    // Validate the data from the html form
    verrs, err := models.DB.ValidateAndCreate(net)
    if err != nil {
        return errors.WithStack(err)
    }
    if verrs.HasAny() {
        c.Set("net", net)
        // Make the errors available inside the html template
        c.Set("errors", verrs)
        return c.Render(422, r.HTML("home/netnew.plush.html"))
    }
    c.Flash().Add("success", "Net was created successfully")
    return c.Redirect(302, "/nets/%s", net.ID)
}

What I did discover is that even after updating nets table to enforce not null on every column, I can still submit forms with an empty name. It appears that we're getting an empty
string, which isn't actually a null. This I think I do want to fix now instead of later.

My first attempt was to block this on the Validate function. If we had lots of data with blank names, I would do it at ValidateCreate. But in this stage of development, I plan to purge any bad data in the database.

// Validate gets run every time you call a "pop.Validate*" (pop.ValidateAndSave, pop.ValidateAndCreate, pop.ValidateAndUpdate) method.
// This method is not required and may be deleted.
func (n *Net) Validate(tx *pop.Connection) (*validate.Errors, error) {
    if n.Name == "" {
        return validate.NewErrors(), errors.New("Name can not be blank")
    }
    return validate.NewErrors(), nil
}

The problem with this validation is that it throws a 500 with stack trace. I'd rather get a smaller error that is flashed at the form. Fortunately, this is documented in the buffalo validation usage. Unfortunately, no clear examples on using that in concert with validate.NewErrors().

func (n *Net) Validate(tx *pop.Connection) (*validate.Errors, error) {
    verrs := validate.NewErrors()
    if n.Name == "" {
        verrs.Add("name", "Name must not be blank!")
    }
    //verrs.Add(&validators.StringIsPresent{Field: n.Name, Name: "Name", Message: "Name can not be blank"})
    return verrs, nil
}

There is a different syntax of &validators.StringIsPresent but the usage is different and the example given is a different use case than the prior examples. My best hint at how to implement comes from issue 2177, which demonstrates using the validate.NewErrors().Add / verrs.Add approach.

Day 2 Wrap-Up

I started writing the app Monday evening, and spent most time on Tuesday. It wasn't till a couple hours in that I started making these notes and setup a timer. I'll say I've got 2-3
hours unaccounted for, and according to my timer now, I am at 1 hour, 37 minutes.

An festering issue is the timestamps. I do want to store date time in UTC, but display it in the user's local timezone. The dates from the web form are being submitted using local time, and stored in the database as UTC (+0000). In other (Python) projects, I was able to use javascript to get the browser's timezone, and I could use datetime tz methods in the backend to save as UTC. I'm sure there's a similiar approach here, so hopefully I'll find time to implement correct time zoning here.

Another future validation is to not allow a net to end before it starts.

Project is published to github ytjohn/gonetninja