Intro

We’re back once again to continue our work on the ghost2hugo prototype. So far we can open the JSON file, load the data into memory, and print out the Markdown for the first post. The next step is to make sure we can read and process every post included in the backup.

Looping

We’re going to replace all the code that prints a single article with a loop that prints out each article. For a refresher, here is the code that prints the first article.

	c := "`" + db.Db[0].Data.Posts[0].Mobiledoc + "`"

	un, err := strconv.Unquote(c)
	if err != nil {
		fmt.Println(err)
	}
	fmt.Printf("%v", un)

	var md Mobiledoc

	err = json.Unmarshal([]byte(un), &md)
	if err != nil {
		fmt.Println(err)
	}
	
	fmt.Printf("%#v", md)
	
	card := md.Cards[0][1]
	fmt.Printf("\n\ncard: %#v\n", card)
	
	bbb := card.(map[string]interface{})
	fmt.Println(bbb["markdown"])

Most of this will live inside of our new loop.

	for i := 0; i < len(db.Db[0].Data.Posts); i++ {
		fmt.Println(db.Db[0].Data.Posts[i].Title)
		fmt.Println(db.Db[0].Data.Posts[i].Slug)
		fmt.Println(db.Db[0].Data.Posts[i].Status)
		fmt.Println(db.Db[0].Data.Posts[i].CreatedAt)
		fmt.Println(db.Db[0].Data.Posts[i].UpdatedAt)

		cc := "`" + db.Db[0].Data.Posts[i].Mobiledoc + "`"

		ucn, err := strconv.Unquote(cc)
		if err != nil {
			fmt.Println(err, ucn)
		}
		fmt.Printf("\n\n\n%v\n\n\n", ucn)

		var md Mobiledoc

		err = json.Unmarshal([]byte(ucn), &md)
		if err != nil {
			fmt.Println(err)
		}

		card := md.Cards[0][1]
		bbb := card.(map[string]interface{})

		fmt.Println(bbb["markdown"])
		}
	}

And when we run the code this time we see a bunch of articles fly by! Until, that is, we hit an error! The invalid syntax error indicates this is a problem with strconv.Unquote().

invalid syntax 

unexpected end of JSON input
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
main.main()
        /Users/steve/Code/ghost2hugo/main.go:214 +0x769
exit status 2

Roadblocks

Well that’s no good, we can’t just have our code crash out like that! Let’s take a closer look at the problem and see if we can work our way through it. OK, now for the master of all debugging techniques! We add a fmt.Println to print out the chunk of text.

		fmt.Println(cc)

Running the code again we crash out as expected but this time we’re presented with the problem text:

`{"version":"0.3.1","markups":[],"atoms":[],"cards":[["markdown",{"cardName":"card-markdown","markdown":"\n* Incresed title to 64 characters. Should be more then enough.  \n * ics and other code work over at angstridden dot net tonight.\n\nALTER TABLE `posts` CHANGE `posttitle` `posttitle` VARCHAR( 64 ) NOT NULL\n\n\n"}]],"sections":[[10,0]],"ghostVersion":"3.0"}`

See the problem? Yeah, it looks like there are back ticks in the Markdown that are causing the string convert to not work as expected. To get around this we’re going to do a simple strings.ReplaceAll before the conversion to change the back tick to something else. We can then reverse that process to get the normal text. We’ll replace the back tick with “%'”.

		c := strings.ReplaceAll(db.Db[0].Data.Posts[i].Mobiledoc, "`", "%'")
		cc := "`" + c + "`"

After we “unquote” we can run another strings.ReplaceAll to convert back.

		ucn = strings.ReplaceAll(ucn, "%'", "`")

Running again we run into another problem!

---
Archives
archives-post
published
2007-07-01 09:46:18 +0000 UTC
2007-08-12 20:50:02 +0000 UTC
`{"version":"0.3.1","markups":[],"atoms":[],"cards":[],"sections":[[1,"p",[[0,[],0,""]]]],"ghostVersion":"3.0"}`
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
main.main()
        /Users/steve/Code/ghost2hugo/main.go:214 +0x7a6
exit status 2

Argh!

More Roadblocks

Our naive implementation from the previous post has returned to bite us! We were assuming that cards exist.

		card := md.Cards[0][1]
		bbb := card.(map[string]interface{})

Looking at the returned JSON confirms this.

{"version":"0.3.1","markups":[],"atoms":[],"cards":[],"sections":[[1,"p",[[0,[],0,""]]]],"ghostVersion":"3.0"}

Now, how do we check to see if the cards field is empty?! I noodled on it a little bit and figured the best way would be to use reflect.ValueOf().Len(). This will check the length of md.Cards to ensure we actually have some data.

		if reflect.ValueOf(md.Cards).Len() > 0 {
			card := md.Cards[0][1]
			bbb := card.(map[string]interface{})

			fmt.Println(bbb["markdown"])
		}

On rerunning we get what looks to be all the posts. Except the final draft which looks odd! It turns out this one has no Markdown card. Is this the only one? I certainly hope so since it looks like it may be a pain to convert to Markdown.

Questing
questing
draft
2019-08-23 21:26:44 +0000 UTC
2021-04-12 04:53:26 +0000 UTC
`{"version":"0.3.1","atoms":[],"cards":[["paywall",{}]],"markups":[["a",["href","https://shindakun.dev"]],["a",["href","https://dev.to/shindakun"]]],"sections":[[1,"p",[[0,[],0,"It's been quite sometime since I've posted anything here. I've done a bit of posting over on "],[0,[0],1,"shindakun.dev"],[0,[],0," and over on "],[0,[1],1,"DEV"],[0,[],0," which has been kind of nice. But, those don't really cover gaming at all. But, I updated the site recently and everyone once in a while I use the site to try something out for work and seeing an old post over and over was no good."]]],[1,"p",[[0,[],0,"I only seem to have a small sliver of time for gaming now. With everything else I want/need to get done there is only so much time. Heed this warning - don't get older! Just kidding, it's not so bad."]]],[1,"p",[[0,[],0,"My 5 year old hasn't been introduced into gaming much outside of some basic games she can play on her tablet. I have discovered that she enjoys watching me play the digital version of Warhammer Quest. Which is good since she's going to be getting a crash course in some more difficult board games sooner or later. I have a copy of the Gloomhaven digital board game and the boxed game (and expansion) but haven't actually event opened it yet. Maybe I should fix that this weekend..."]]],[10,0],[1,"p",[]]],"ghostVersion":"3.0"}`
json: cannot unmarshal string into Go struct field Mobiledoc.sections of type int
<nil>

Looks like we also have an issue unmarshaling, on that last line there. This is easy enough to fix for now. We just need to update sections in our Mobiledoc struct to use [][]interface{} and not [][]int.

type Mobiledoc struct {
	Version      string          `json:"version"`
	Markups      []interface{}   `json:"markups"`
	Atoms        []interface{}   `json:"atoms"`
	Cards        [][]interface{} `json:"cards"`
	Sections     [][]interface{} `json:"sections"`
	GhostVersion string          `json:"ghostVersion"`
}

Next Time

We’ve made some good progress over the last few posts. On the first look it seems we’re extracting what we need. I think we’re ready to do a little refactoring, remove our current debug print statements, and get ready for the next part of our converter.


Enjoy this post?
How about buying me a coffee?

Code Listing

package main

import (
	"encoding/json"
	"fmt"
	"io"
	"os"
	"reflect"
	"strconv"
	"strings"
	"time"
)

type GhostDatabase struct {
	Db []struct {
		Meta struct {
			ExportedOn int64  `json:"exported_on"`
			Version    string `json:"version"`
		} `json:"meta"`
		Data struct {
			Posts []struct {
				ID                   string      `json:"id"`
				UUID                 string      `json:"uuid"`
				Title                string      `json:"title"`
				Slug                 string      `json:"slug"`
				Mobiledoc            string      `json:"mobiledoc"`
				HTML                 string      `json:"html"`
				CommentID            string      `json:"comment_id"`
				Plaintext            string      `json:"plaintext"`
				FeatureImage         interface{} `json:"feature_image"`
				Featured             int         `json:"featured"`
				Type                 string      `json:"type"`
				Status               string      `json:"status"`
				Locale               interface{} `json:"locale"`
				Visibility           string      `json:"visibility"`
				EmailRecipientFilter string      `json:"email_recipient_filter"`
				AuthorID             string      `json:"author_id"`
				CreatedAt            time.Time   `json:"created_at"`
				UpdatedAt            time.Time   `json:"updated_at"`
				PublishedAt          time.Time   `json:"published_at"`
				CustomExcerpt        interface{} `json:"custom_excerpt"`
				CodeinjectionHead    interface{} `json:"codeinjection_head"`
				CodeinjectionFoot    interface{} `json:"codeinjection_foot"`
				CustomTemplate       interface{} `json:"custom_template"`
				CanonicalURL         interface{} `json:"canonical_url"`
			} `json:"posts"`
			PostsAuthors []struct {
				ID        string `json:"id"`
				PostID    string `json:"post_id"`
				AuthorID  string `json:"author_id"`
				SortOrder int    `json:"sort_order"`
			} `json:"posts_authors"`
			PostsMeta []interface{} `json:"posts_meta"`
			PostsTags []struct {
				ID        string `json:"id"`
				PostID    string `json:"post_id"`
				TagID     string `json:"tag_id"`
				SortOrder int    `json:"sort_order"`
			} `json:"posts_tags"`
			Roles []struct {
				ID          string    `json:"id"`
				Name        string    `json:"name"`
				Description string    `json:"description"`
				CreatedAt   time.Time `json:"created_at"`
				UpdatedAt   time.Time `json:"updated_at"`
			} `json:"roles"`
			RolesUsers []struct {
				ID     string `json:"id"`
				RoleID string `json:"role_id"`
				UserID string `json:"user_id"`
			} `json:"roles_users"`
			Settings []struct {
				ID        string      `json:"id"`
				Group     string      `json:"group"`
				Key       string      `json:"key"`
				Value     string      `json:"value"`
				Type      string      `json:"type"`
				Flags     interface{} `json:"flags"`
				CreatedAt time.Time   `json:"created_at"`
				UpdatedAt time.Time   `json:"updated_at"`
			} `json:"settings"`
			Tags []struct {
				ID                 string      `json:"id"`
				Name               string      `json:"name"`
				Slug               string      `json:"slug"`
				Description        interface{} `json:"description"`
				FeatureImage       interface{} `json:"feature_image"`
				ParentID           interface{} `json:"parent_id"`
				Visibility         string      `json:"visibility"`
				OgImage            interface{} `json:"og_image"`
				OgTitle            interface{} `json:"og_title"`
				OgDescription      interface{} `json:"og_description"`
				TwitterImage       interface{} `json:"twitter_image"`
				TwitterTitle       interface{} `json:"twitter_title"`
				TwitterDescription interface{} `json:"twitter_description"`
				MetaTitle          interface{} `json:"meta_title"`
				MetaDescription    interface{} `json:"meta_description"`
				CodeinjectionHead  interface{} `json:"codeinjection_head"`
				CodeinjectionFoot  interface{} `json:"codeinjection_foot"`
				CanonicalURL       interface{} `json:"canonical_url"`
				AccentColor        interface{} `json:"accent_color"`
				CreatedAt          time.Time   `json:"created_at"`
				UpdatedAt          time.Time   `json:"updated_at"`
			} `json:"tags"`
			Users []struct {
				ID              string      `json:"id"`
				Name            string      `json:"name"`
				Slug            string      `json:"slug"`
				Password        string      `json:"password"`
				Email           string      `json:"email"`
				ProfileImage    string      `json:"profile_image"`
				CoverImage      interface{} `json:"cover_image"`
				Bio             interface{} `json:"bio"`
				Website         interface{} `json:"website"`
				Location        interface{} `json:"location"`
				Facebook        interface{} `json:"facebook"`
				Twitter         interface{} `json:"twitter"`
				Accessibility   string      `json:"accessibility"`
				Status          string      `json:"status"`
				Locale          interface{} `json:"locale"`
				Visibility      string      `json:"visibility"`
				MetaTitle       interface{} `json:"meta_title"`
				MetaDescription interface{} `json:"meta_description"`
				Tour            interface{} `json:"tour"`
				LastSeen        time.Time   `json:"last_seen"`
				CreatedAt       time.Time   `json:"created_at"`
				UpdatedAt       time.Time   `json:"updated_at"`
			} `json:"users"`
		} `json:"data"`
	} `json:"db"`
}

type Mobiledoc struct {
	Version      string          `json:"version"`
	Markups      []interface{}   `json:"markups"`
	Atoms        []interface{}   `json:"atoms"`
	Cards        [][]interface{} `json:"cards"`
	Sections     [][]interface{} `json:"sections"`
	GhostVersion string          `json:"ghostVersion"`
}

func main() {
	fmt.Println("ghost2hugo")

	file, err := os.Open("shindakun-dot-net.ghost.2022-03-18-22-02-58.json")
	if err != nil {
		fmt.Println(err)
	}

	defer file.Close()
	
	b, err := io.ReadAll(file)
	if err != nil {
		fmt.Println(err)
	}

	var db GhostDatabase

	err = json.Unmarshal(b, &db)
	if err != nil {
		fmt.Println(err)
	}

	for i := 0; i < len(db.Db[0].Data.Posts); i++ {
		fmt.Println(db.Db[0].Data.Posts[i].Title)
		fmt.Println(db.Db[0].Data.Posts[i].Slug)
		fmt.Println(db.Db[0].Data.Posts[i].Status)
		fmt.Println(db.Db[0].Data.Posts[i].CreatedAt)
		fmt.Println(db.Db[0].Data.Posts[i].UpdatedAt)

		c := strings.ReplaceAll(db.Db[0].Data.Posts[i].Mobiledoc, "`", "%'")
		cc := "`" + c + "`"

		fmt.Println(cc)

		ucn, err := strconv.Unquote(cc)
		if err != nil {
			fmt.Println(err, ucn)
		}

		ucn = strings.ReplaceAll(ucn, "%'", "`")

		var md Mobiledoc

		err = json.Unmarshal([]byte(ucn), &md)
		if err != nil {
			fmt.Println(err)
		}

		if reflect.ValueOf(md.Cards).Len() > 0 {
			card := md.Cards[0][1]
			bbb := card.(map[string]interface{})

			fmt.Println(bbb["markdown"])
		}

		fmt.Println("---")
	}
}