Intro
We’re back once again to continue our work on the ghost2hugo prototype. So far we can open the JSON file, load the data into memory, and print out the Markdown for the first post. The next step is to make sure we can read and process every post included in the backup.
Looping
We’re going to replace all the code that prints a single article with a loop that prints out each article. For a refresher, here is the code that prints the first article.
c := "`" + db.Db[0].Data.Posts[0].Mobiledoc + "`"
un, err := strconv.Unquote(c)
if err != nil {
fmt.Println(err)
}
fmt.Printf("%v", un)
var md Mobiledoc
err = json.Unmarshal([]byte(un), &md)
if err != nil {
fmt.Println(err)
}
fmt.Printf("%#v", md)
card := md.Cards[0][1]
fmt.Printf("\n\ncard: %#v\n", card)
bbb := card.(map[string]interface{})
fmt.Println(bbb["markdown"])
Most of this will live inside of our new loop.
for i := 0; i < len(db.Db[0].Data.Posts); i++ {
fmt.Println(db.Db[0].Data.Posts[i].Title)
fmt.Println(db.Db[0].Data.Posts[i].Slug)
fmt.Println(db.Db[0].Data.Posts[i].Status)
fmt.Println(db.Db[0].Data.Posts[i].CreatedAt)
fmt.Println(db.Db[0].Data.Posts[i].UpdatedAt)
cc := "`" + db.Db[0].Data.Posts[i].Mobiledoc + "`"
ucn, err := strconv.Unquote(cc)
if err != nil {
fmt.Println(err, ucn)
}
fmt.Printf("\n\n\n%v\n\n\n", ucn)
var md Mobiledoc
err = json.Unmarshal([]byte(ucn), &md)
if err != nil {
fmt.Println(err)
}
card := md.Cards[0][1]
bbb := card.(map[string]interface{})
fmt.Println(bbb["markdown"])
}
}
And when we run the code this time we see a bunch of articles fly by! Until, that is, we hit an error! The invalid syntax error indicates this is a problem with strconv.Unquote().
invalid syntax
unexpected end of JSON input
panic: runtime error: index out of range [0] with length 0
goroutine 1 [running]:
main.main()
/Users/steve/Code/ghost2hugo/main.go:214 +0x769
exit status 2
Roadblocks
Well that’s no good, we can’t just have our code crash out like that! Let’s take a closer look at the problem and see if we can work our way through it. OK, now for the master of all debugging techniques! We add a fmt.Println to print out the chunk of text.
fmt.Println(cc)
Running the code again we crash out as expected but this time we’re presented with the problem text:
`{"version":"0.3.1","markups":[],"atoms":[],"cards":[["markdown",{"cardName":"card-markdown","markdown":"\n* Incresed title to 64 characters. Should be more then enough. \n * ics and other code work over at angstridden dot net tonight.\n\nALTER TABLE `posts` CHANGE `posttitle` `posttitle` VARCHAR( 64 ) NOT NULL\n\n\n"}]],"sections":[[10,0]],"ghostVersion":"3.0"}`
See the problem? Yeah, it looks like there are back ticks in the Markdown that are causing the string convert to not work as expected. To get around this we’re going to do a simple strings.ReplaceAll before the conversion to change the back tick to something else. We can then reverse that process to get the normal text. We’ll replace the back tick with “%'”.
c := strings.ReplaceAll(db.Db[0].Data.Posts[i].Mobiledoc, "`", "%'")
cc := "`" + c + "`"
After we “unquote” we can run another strings.ReplaceAll to convert back.
ucn = strings.ReplaceAll(ucn, "%'", "`")
Running again we run into another problem!
---
Archives
archives-post
published
2007-07-01 09:46:18 +0000 UTC
2007-08-12 20:50:02 +0000 UTC
`{"version":"0.3.1","markups":[],"atoms":[],"cards":[],"sections":[[1,"p",[[0,[],0,""]]]],"ghostVersion":"3.0"}`
panic: runtime error: index out of range [0] with length 0
goroutine 1 [running]:
main.main()
/Users/steve/Code/ghost2hugo/main.go:214 +0x7a6
exit status 2
Argh!
More Roadblocks
Our naive implementation from the previous post has returned to bite us! We were assuming that cards exist.
card := md.Cards[0][1]
bbb := card.(map[string]interface{})
Looking at the returned JSON confirms this.
{"version":"0.3.1","markups":[],"atoms":[],"cards":[],"sections":[[1,"p",[[0,[],0,""]]]],"ghostVersion":"3.0"}
Now, how do we check to see if the cards field is empty?! I noodled on it a little bit and figured the best way would be to use reflect.ValueOf().Len(). This will check the length of md.Cards to ensure we actually have some data.
if reflect.ValueOf(md.Cards).Len() > 0 {
card := md.Cards[0][1]
bbb := card.(map[string]interface{})
fmt.Println(bbb["markdown"])
}
On rerunning we get what looks to be all the posts. Except the final draft which looks odd! It turns out this one has no Markdown card. Is this the only one? I certainly hope so since it looks like it may be a pain to convert to Markdown.
Questing
questing
draft
2019-08-23 21:26:44 +0000 UTC
2021-04-12 04:53:26 +0000 UTC
`{"version":"0.3.1","atoms":[],"cards":[["paywall",{}]],"markups":[["a",["href","https://shindakun.dev"]],["a",["href","https://dev.to/shindakun"]]],"sections":[[1,"p",[[0,[],0,"It's been quite sometime since I've posted anything here. I've done a bit of posting over on "],[0,[0],1,"shindakun.dev"],[0,[],0," and over on "],[0,[1],1,"DEV"],[0,[],0," which has been kind of nice. But, those don't really cover gaming at all. But, I updated the site recently and everyone once in a while I use the site to try something out for work and seeing an old post over and over was no good."]]],[1,"p",[[0,[],0,"I only seem to have a small sliver of time for gaming now. With everything else I want/need to get done there is only so much time. Heed this warning - don't get older! Just kidding, it's not so bad."]]],[1,"p",[[0,[],0,"My 5 year old hasn't been introduced into gaming much outside of some basic games she can play on her tablet. I have discovered that she enjoys watching me play the digital version of Warhammer Quest. Which is good since she's going to be getting a crash course in some more difficult board games sooner or later. I have a copy of the Gloomhaven digital board game and the boxed game (and expansion) but haven't actually event opened it yet. Maybe I should fix that this weekend..."]]],[10,0],[1,"p",[]]],"ghostVersion":"3.0"}`
json: cannot unmarshal string into Go struct field Mobiledoc.sections of type int
<nil>
Looks like we also have an issue unmarshaling, on that last line there. This is easy enough to fix for now. We just need to update sections in our Mobiledoc struct to use [][]interface{} and not [][]int.
type Mobiledoc struct {
Version string `json:"version"`
Markups []interface{} `json:"markups"`
Atoms []interface{} `json:"atoms"`
Cards [][]interface{} `json:"cards"`
Sections [][]interface{} `json:"sections"`
GhostVersion string `json:"ghostVersion"`
}
Next Time
We’ve made some good progress over the last few posts. On the first look it seems we’re extracting what we need. I think we’re ready to do a little refactoring, remove our current debug print statements, and get ready for the next part of our converter.
| Enjoy this post? |
|---|
| How about buying me a coffee? |
Code Listing
package main
import (
"encoding/json"
"fmt"
"io"
"os"
"reflect"
"strconv"
"strings"
"time"
)
type GhostDatabase struct {
Db []struct {
Meta struct {
ExportedOn int64 `json:"exported_on"`
Version string `json:"version"`
} `json:"meta"`
Data struct {
Posts []struct {
ID string `json:"id"`
UUID string `json:"uuid"`
Title string `json:"title"`
Slug string `json:"slug"`
Mobiledoc string `json:"mobiledoc"`
HTML string `json:"html"`
CommentID string `json:"comment_id"`
Plaintext string `json:"plaintext"`
FeatureImage interface{} `json:"feature_image"`
Featured int `json:"featured"`
Type string `json:"type"`
Status string `json:"status"`
Locale interface{} `json:"locale"`
Visibility string `json:"visibility"`
EmailRecipientFilter string `json:"email_recipient_filter"`
AuthorID string `json:"author_id"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
PublishedAt time.Time `json:"published_at"`
CustomExcerpt interface{} `json:"custom_excerpt"`
CodeinjectionHead interface{} `json:"codeinjection_head"`
CodeinjectionFoot interface{} `json:"codeinjection_foot"`
CustomTemplate interface{} `json:"custom_template"`
CanonicalURL interface{} `json:"canonical_url"`
} `json:"posts"`
PostsAuthors []struct {
ID string `json:"id"`
PostID string `json:"post_id"`
AuthorID string `json:"author_id"`
SortOrder int `json:"sort_order"`
} `json:"posts_authors"`
PostsMeta []interface{} `json:"posts_meta"`
PostsTags []struct {
ID string `json:"id"`
PostID string `json:"post_id"`
TagID string `json:"tag_id"`
SortOrder int `json:"sort_order"`
} `json:"posts_tags"`
Roles []struct {
ID string `json:"id"`
Name string `json:"name"`
Description string `json:"description"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
} `json:"roles"`
RolesUsers []struct {
ID string `json:"id"`
RoleID string `json:"role_id"`
UserID string `json:"user_id"`
} `json:"roles_users"`
Settings []struct {
ID string `json:"id"`
Group string `json:"group"`
Key string `json:"key"`
Value string `json:"value"`
Type string `json:"type"`
Flags interface{} `json:"flags"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
} `json:"settings"`
Tags []struct {
ID string `json:"id"`
Name string `json:"name"`
Slug string `json:"slug"`
Description interface{} `json:"description"`
FeatureImage interface{} `json:"feature_image"`
ParentID interface{} `json:"parent_id"`
Visibility string `json:"visibility"`
OgImage interface{} `json:"og_image"`
OgTitle interface{} `json:"og_title"`
OgDescription interface{} `json:"og_description"`
TwitterImage interface{} `json:"twitter_image"`
TwitterTitle interface{} `json:"twitter_title"`
TwitterDescription interface{} `json:"twitter_description"`
MetaTitle interface{} `json:"meta_title"`
MetaDescription interface{} `json:"meta_description"`
CodeinjectionHead interface{} `json:"codeinjection_head"`
CodeinjectionFoot interface{} `json:"codeinjection_foot"`
CanonicalURL interface{} `json:"canonical_url"`
AccentColor interface{} `json:"accent_color"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
} `json:"tags"`
Users []struct {
ID string `json:"id"`
Name string `json:"name"`
Slug string `json:"slug"`
Password string `json:"password"`
Email string `json:"email"`
ProfileImage string `json:"profile_image"`
CoverImage interface{} `json:"cover_image"`
Bio interface{} `json:"bio"`
Website interface{} `json:"website"`
Location interface{} `json:"location"`
Facebook interface{} `json:"facebook"`
Twitter interface{} `json:"twitter"`
Accessibility string `json:"accessibility"`
Status string `json:"status"`
Locale interface{} `json:"locale"`
Visibility string `json:"visibility"`
MetaTitle interface{} `json:"meta_title"`
MetaDescription interface{} `json:"meta_description"`
Tour interface{} `json:"tour"`
LastSeen time.Time `json:"last_seen"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
} `json:"users"`
} `json:"data"`
} `json:"db"`
}
type Mobiledoc struct {
Version string `json:"version"`
Markups []interface{} `json:"markups"`
Atoms []interface{} `json:"atoms"`
Cards [][]interface{} `json:"cards"`
Sections [][]interface{} `json:"sections"`
GhostVersion string `json:"ghostVersion"`
}
func main() {
fmt.Println("ghost2hugo")
file, err := os.Open("shindakun-dot-net.ghost.2022-03-18-22-02-58.json")
if err != nil {
fmt.Println(err)
}
defer file.Close()
b, err := io.ReadAll(file)
if err != nil {
fmt.Println(err)
}
var db GhostDatabase
err = json.Unmarshal(b, &db)
if err != nil {
fmt.Println(err)
}
for i := 0; i < len(db.Db[0].Data.Posts); i++ {
fmt.Println(db.Db[0].Data.Posts[i].Title)
fmt.Println(db.Db[0].Data.Posts[i].Slug)
fmt.Println(db.Db[0].Data.Posts[i].Status)
fmt.Println(db.Db[0].Data.Posts[i].CreatedAt)
fmt.Println(db.Db[0].Data.Posts[i].UpdatedAt)
c := strings.ReplaceAll(db.Db[0].Data.Posts[i].Mobiledoc, "`", "%'")
cc := "`" + c + "`"
fmt.Println(cc)
ucn, err := strconv.Unquote(cc)
if err != nil {
fmt.Println(err, ucn)
}
ucn = strings.ReplaceAll(ucn, "%'", "`")
var md Mobiledoc
err = json.Unmarshal([]byte(ucn), &md)
if err != nil {
fmt.Println(err)
}
if reflect.ValueOf(md.Cards).Len() > 0 {
card := md.Cards[0][1]
bbb := card.(map[string]interface{})
fmt.Println(bbb["markdown"])
}
fmt.Println("---")
}
}
