TheLazy.Dev
Portfolio GitHub

Blog

go |ghost |hugo

Attempting to Learn Go - Ghost to Hugo 2

shindakun

Steve Layton

Posted on September 23, 2022

Attempting to Learn Go - Ghost to Hugo 2

Intro

Welcome back! We are continuing on our journey to make a prototype program that converts an exported Ghost database to Markdown. With the end goal being that we can get shindakun.net up and running with Hugo. Last time, we took it pretty easy and focused mostly on reading the file into memory and converting the JSON to a Go struct. From there we printed out the first post.

Post Data

As a recap here is what one of the post fields contain.

{
    "id": "60710b90705967038fe662d6",
    "uuid": "71ba3d71-ac18-4f33-82f7-1962baa83a07",
    "title": "db test",
    "slug": "db-test",
    "mobiledoc": "{\"version\":\"0.3.1\",\"markups\":[],\"atoms\":[],\"cards\":[[\"markdown\",{\"cardName\":\"card-markdown\",\"markdown\":\"\\n<strike>This is a db test</strike>.\\n\"}]],\"sections\":[[10,0]],\"ghostVersion\":\"3.0\"}",
    "html": "<!--kg-card-begin: markdown--><p><strike>This is a db test</strike>.</p>\n<!--kg-card-end: markdown-->",
    "comment_id": "2",
    "plaintext": "This is a db test.",
    "feature_image": null,
    "featured": 0,
    "type": "post",
    "status": "published",
    "locale": null,
    "visibility": "public",
    "email_recipient_filter": "none",
    "author_id": "60710b8d705967038fe66214",
    "created_at": "2004-08-09T19:11:20.000Z",
    "updated_at": "2004-08-09T19:11:20.000Z",
    "published_at": "2004-08-09T19:11:20.000Z",
    "custom_excerpt": null,
    "codeinjection_head": null,
    "codeinjection_foot": null,
    "custom_template": null,
    "canonical_url": null
},
Enter fullscreen mode Exit fullscreen mode

The majority of the fields we need to craft a post with appropriate frontmatter exist within this the object we’re getting back. We can see the title, slug, published date, etc. The section that contains the Markdown is in a format known as Mobiledoc.

Mobiledoc

According to the Ghost documentation Mobiledoc is

…a standardised JSON-based document storage format, which forms the heart of publishing with Ghost.

When extracted from the JSON object and cleaned up we’ll have another bit of JSON we can work with.

{
    "version": "0.3.1",
    "markups": [],
    "atoms": [],
    "cards": [
        [
            "markdown",
            {
                "cardName": "card-markdown",
                "markdown": "\n<strike>This is a db test</strike>.\n"
            }
        ]
    ],
    "sections": [
        [
            10,
            0
        ]
    ],
    "ghostVersion": "3.0"
}
Enter fullscreen mode Exit fullscreen mode

Sounds Easy

First we’ll use our favorite site, the JSON to Go converter to convert the JSON object to a struct we can work with.

type Mobiledoc struct {
    Version string `json:"version"`
    Markups []interface{} `json:"markups"`
    Atoms []interface{} `json:"atoms"`
    Cards [][]interface{} `json:"cards"`
    Sections [][]int `json:"sections"`
    GhostVersion string `json:"ghostVersion"`
}
Enter fullscreen mode Exit fullscreen mode

Our code is pretty long now so I’m going to leave out the other struct GhostDatabase struct, it’ll be in the complete code listing below though. We’re still going to be dumping code to the screen since we’re still working on our decoding logic.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "os"
    "strconv"
    "time"
)

type GhostDatabase struct {...}

type Mobiledoc struct {
    Version string `json:"version"`
    Markups []interface{} `json:"markups"`
    Atoms []interface{} `json:"atoms"`
    Cards [][]interface{} `json:"cards"`
    Sections [][]int `json:"sections"`
    GhostVersion string `json:"ghostVersion"`
}

func main() {
    fmt.Println("ghost2hugo")

    file, err := os.Open("shindakun-dot-net.ghost.2022-03-18-22-02-58.json")
    if err != nil {
        fmt.Println(err)
    }

    defer file.Close()

    b, err := io.ReadAll(file)
    if err != nil {
        fmt.Println(err)
    }

    var db GhostDatabase

    err = json.Unmarshal(b, &db)
    if err != nil {
        fmt.Println(err)
    }
Enter fullscreen mode Exit fullscreen mode

Let’s continue to focus on the first post for now since once we have that working it should just be a matter of looping through the “database”. This is where it gets a little tricky. We’re working with a couple of nested arrays so to work our way down to the appropriate section we use db.Db[0].Data.Posts[0].Mobiledoc. This will give use the escaped version of our JSON object.

"{\"version\":\"0.3.1\",\"markups\":[],\"atoms\":[],\"cards\":[[\"markdown\",{\"cardName\":\"card-markdown\",\"markdown\":\"\\n<strike>This is a db test</strike>.\\n\"}]],\"sections\":[[10,0]],\"ghostVersion\":\"3.0\"}"
Enter fullscreen mode Exit fullscreen mode

Let’s Go On A Trip

I knew that there would be a way to unescape the string that we get, checking the Go documentation led me to strconv.Unquote.

Unquote interprets s as a single-quoted, double-quoted, or backquoted Go string literal, returning the string value that s quotes. (If s is single-quoted, it would be a Go character literal; Unquote returns the corresponding one-character string.)

Exactly what we need! Except that when I tried to unquote the string I kept receiving an invalid syntax error. This had me confused for a little bit. After puzzling over it I realized if I prepended and appended a back tick to the string it seems to be treated as a raw string literal. This leads to code that looks like the following.

    c := "`" + db.Db[0].Data.Posts[0].Mobiledoc + "`"

    un, err := strconv.Unquote(c)
    if err != nil {
        fmt.Println(err)
    }
    fmt.Printf("%v\n", un)
Enter fullscreen mode Exit fullscreen mode

Finally! We have the JSON!

{"version":"0.3.1","markups":[],"atoms":[],"cards":[["markdown",{"cardName":"card-markdown","markdown":"\n<strike>This is a db test</strike>.\n"}]],"sections":[[10,0]],"ghostVersion":"3.0"}
Enter fullscreen mode Exit fullscreen mode

Now, we can unmarshal that into the Mobiledoc struct we set up earlier!

    var md Mobiledoc

    err = json.Unmarshal([]byte(un), &md)
    if err != nil {
        fmt.Println(err)
    }

    fmt.Printf("%#v", md)
Enter fullscreen mode Exit fullscreen mode

Checking our dumped result we can see that we’ve got the expected data.

main.Mobiledoc{Version:"0.3.1", Markups:[]interface {}{}, Atoms:[]interface {}{}, Cards:[][]interface {}{[]interface {}{"markdown", map[string]interface {}{"cardName":"card-markdown", "markdown":"\n<strike>This is a db test</strike>.\n"}}}, Sections:[][]int{[]int{10, 0}}, GhostVersion:"3.0"}
Enter fullscreen mode Exit fullscreen mode

Right now we’re concerned with the section called Cards which if we naively want to access we can use the following.

    card := md.Cards[0][1]
    fmt.Printf("\n\ncard: %#v\n", card)
Enter fullscreen mode Exit fullscreen mode 
card: map[string]interface {}{"cardName":"card-markdown", "markdown":"\n<strike>This is a db test</strike>.\n"}
Enter fullscreen mode Exit fullscreen mode

Let’s convert that to something a bit easier to access.

    bbb := card.(map[string]interface{})
    fmt.Println(bbb["markdown"])
Enter fullscreen mode Exit fullscreen mode 
<strike>This is a db test</strike>.
Enter fullscreen mode Exit fullscreen mode

Next Time

So far so good, we’ve extracted the Markdown from the first post as expected. Next time around we’ll be writing a small loop to start trying to extract all the posts. I have a feeling that’s where the fun will begin.

Enjoy this post?
How about buying me a coffee?

GitHub logo shindakun / atlg

Source repo for the "Attempting to Learn Go" posts I've been putting up over on dev.to



View on GitHub


Code Listing 

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "os"
    "strconv"
    "time"
)

type GhostDatabase struct {
    Db []struct {
        Meta struct {
            ExportedOn int64 `json:"exported_on"`
            Version string `json:"version"`
        } `json:"meta"`
        Data struct {
            Posts []struct {
                ID string `json:"id"`
                UUID string `json:"uuid"`
                Title string `json:"title"`
                Slug string `json:"slug"`
                Mobiledoc string `json:"mobiledoc"`
                HTML string `json:"html"`
                CommentID string `json:"comment_id"`
                Plaintext string `json:"plaintext"`
                FeatureImage interface{} `json:"feature_image"`
                Featured int `json:"featured"`
                Type string `json:"type"`
                Status string `json:"status"`
                Locale interface{} `json:"locale"`
                Visibility string `json:"visibility"`
                EmailRecipientFilter string `json:"email_recipient_filter"`
                AuthorID string `json:"author_id"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
                PublishedAt time.Time `json:"published_at"`
                CustomExcerpt interface{} `json:"custom_excerpt"`
                CodeinjectionHead interface{} `json:"codeinjection_head"`
                CodeinjectionFoot interface{} `json:"codeinjection_foot"`
                CustomTemplate interface{} `json:"custom_template"`
                CanonicalURL interface{} `json:"canonical_url"`
            } `json:"posts"`
            PostsAuthors []struct {
                ID string `json:"id"`
                PostID string `json:"post_id"`
                AuthorID string `json:"author_id"`
                SortOrder int `json:"sort_order"`
            } `json:"posts_authors"`
            PostsMeta []interface{} `json:"posts_meta"`
            PostsTags []struct {
                ID string `json:"id"`
                PostID string `json:"post_id"`
                TagID string `json:"tag_id"`
                SortOrder int `json:"sort_order"`
            } `json:"posts_tags"`
            Roles []struct {
                ID string `json:"id"`
                Name string `json:"name"`
                Description string `json:"description"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"roles"`
            RolesUsers []struct {
                ID string `json:"id"`
                RoleID string `json:"role_id"`
                UserID string `json:"user_id"`
            } `json:"roles_users"`
            Settings []struct {
                ID string `json:"id"`
                Group string `json:"group"`
                Key string `json:"key"`
                Value string `json:"value"`
                Type string `json:"type"`
                Flags interface{} `json:"flags"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"settings"`
            Tags []struct {
                ID string `json:"id"`
                Name string `json:"name"`
                Slug string `json:"slug"`
                Description interface{} `json:"description"`
                FeatureImage interface{} `json:"feature_image"`
                ParentID interface{} `json:"parent_id"`
                Visibility string `json:"visibility"`
                OgImage interface{} `json:"og_image"`
                OgTitle interface{} `json:"og_title"`
                OgDescription interface{} `json:"og_description"`
                TwitterImage interface{} `json:"twitter_image"`
                TwitterTitle interface{} `json:"twitter_title"`
                TwitterDescription interface{} `json:"twitter_description"`
                MetaTitle interface{} `json:"meta_title"`
                MetaDescription interface{} `json:"meta_description"`
                CodeinjectionHead interface{} `json:"codeinjection_head"`
                CodeinjectionFoot interface{} `json:"codeinjection_foot"`
                CanonicalURL interface{} `json:"canonical_url"`
                AccentColor interface{} `json:"accent_color"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"tags"`
            Users []struct {
                ID string `json:"id"`
                Name string `json:"name"`
                Slug string `json:"slug"`
                Password string `json:"password"`
                Email string `json:"email"`
                ProfileImage string `json:"profile_image"`
                CoverImage interface{} `json:"cover_image"`
                Bio interface{} `json:"bio"`
                Website interface{} `json:"website"`
                Location interface{} `json:"location"`
                Facebook interface{} `json:"facebook"`
                Twitter interface{} `json:"twitter"`
                Accessibility string `json:"accessibility"`
                Status string `json:"status"`
                Locale interface{} `json:"locale"`
                Visibility string `json:"visibility"`
                MetaTitle interface{} `json:"meta_title"`
                MetaDescription interface{} `json:"meta_description"`
                Tour interface{} `json:"tour"`
                LastSeen time.Time `json:"last_seen"`
                CreatedAt time.Time `json:"created_at"`
                UpdatedAt time.Time `json:"updated_at"`
            } `json:"users"`
        } `json:"data"`
    } `json:"db"`
}

type Mobiledoc struct {
    Version string `json:"version"`
    Markups []interface{} `json:"markups"`
    Atoms []interface{} `json:"atoms"`
    Cards [][]interface{} `json:"cards"`
    Sections [][]int `json:"sections"`
    GhostVersion string `json:"ghostVersion"`
}

func main() {
    fmt.Println("ghost2hugo")

    file, err := os.Open("shindakun-dot-net.ghost.2022-03-18-22-02-58.json")
    if err != nil {
        fmt.Println(err)
    }

    defer file.Close()

    b, err := io.ReadAll(file)
    if err != nil {
        fmt.Println(err)
    }

    var db GhostDatabase

    err = json.Unmarshal(b, &db)
    if err != nil {
        fmt.Println(err)
    }

    c := "`" + db.Db[0].Data.Posts[0].Mobiledoc + "`"

    un, err := strconv.Unquote(c)
    if err != nil {
        fmt.Println(err)
    }
    fmt.Printf("%v", un)

    var md Mobiledoc

    err = json.Unmarshal([]byte(un), &md)
    if err != nil {
        fmt.Println(err)
    }

    fmt.Printf("%#v", md)

    card := md.Cards[0][1]
    fmt.Printf("\n\ncard: %#v\n", card)

    bbb := card.(map[string]interface{})
    fmt.Println(bbb["markdown"])
}
Enter fullscreen mode Exit fullscreen mode
💖 💪 🙅 🚩
shindakun
Steve Layton

Posted on September 23, 2022

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

Attempting to Learn Go - Ghost to Hugo 2
go Attempting to Learn Go - Ghost to Hugo 2

September 23, 2022