shindakun dot dev

obsidian-markdown-export-plugin

Fri, 28 Jun 2024 09:00:40 -0700

Intro

Dusting off my blog has lead me to look again at my posting workflow. Since I want it to be as low friction as possible. Typically I write my posts in Obsidian and then copy the Markdown file to my posts directory. That directory is checked into GitHub. This left a lot to be desired, images were typically left out of the loop and were uploaded separately to my server.

A Better Way

Looking to smooth out the process led me to a post from a little while back from Cassidy Williams.

The repository for this blog is separate from my Obsidian vault, so normally I have to do some copying and pasting across folders, which is fine but really slows me down.

Amen to that! But there is a solution the cleverly named obsidian-markdown-export-plugin which after a feature request supports out of the vault exports with attachments no less! Thanks Cassidy for that request. And thanks to bingryan for the plugin!

An Even Better Way

Problem is it’s not quite what I needed for my Hugo blog. I need it to export using the Markdown filename as part of the path and to include the images attachment directory within that.

So if my content was in

/posts

I would want the output to be

/posts/Title of Markdown File/
/posts/Title of Markdown File/attachments

At first I put in a feature request of my own. But after a few moments I got the itch to see if I could just hack something together. So I cloned the repo to take a look at the code.

A bit a sniffing around and I found the couple of places where the code creates the directories and exports the image files.

A quick path addition later…

		var dir = file.name.replace(".md", "")
		// try create attachment directory
		await tryCreateFolder(
			this,
			path.join(this.settings.output, dir, this.settings.attachment),
		);

And we had the right folder structure in place!

One more quick tweak and we had images saving in the correct spot too!

					var dir = filename.replace(".md", "")
					const targetPath = path
						.join(
							plugin.settings.output,
							dir,
							plugin.settings.attachment,
							imageLinkMd5.concat(imageExt),
						)
						.replace(/\\/g, "/");

Alright! That’s the end of our journey right?! Well, not quite. In this case I ended up with the output being

/posts/Title of Markdown File/Title of Markdown File.md
/posts/Title of Markdown File/attachments/randomimage.png

This seemed to cause Hugo to render extra directories which would make my site look like this

http://shindakun.dev/title-of-markdown-file/title-of-markdown-file/index.html

And that just can’t stand! So instead of saving as the filename I had it save directly to index.md.

				case "markdown": {
					const targetFile = path.join(outDir, "index.md");// file.name);
					await tryCreate(plugin, targetFile, content);
					break;
				}

This tells Hugo to not render any further and instead we end up with the path of

http://shindakun.dev/title-of-markdown-file/index.html

Just what the doctor ordered!

Wrapping Up

Now I just need to decide if I’m going to brush off my TypeScript knowledge and clean up the patch I did to actually use a setting and give people the option to have output like this. What do you say? Should I?

Enjoy this post?
How about buying me a coffee?

go-feedi

Wed, 26 Jun 2024 20:03:43 -0700

Intro

For months now I’ve had a tab open in my browser on my main computer - Reclaiming the Web with a Personal Reader. I’ve read the article a couple of times and really like the idea of using a personal and extendable feed reader.

Finally around a week ago I cloned the repo from GitHub to check it out. After a couple of tweaks to the Makefile to make it work with my laptops configuration I was up and running. Someone else noticed the thing with the port being in use so they submitted a pull request on the repo so I didn’t have to.

Overall, I really liked it! I did discover a small typo which kept feeds from syncing when you clicked the sync button. Though to be fair I shouldn’t have needed to click sync, I think it should have imported the initial feeds when I ran make feed-load. But that’s just a little thing.

Python isn’t my strongest language so while I can follow along with the code OK I don’t feel like I can extend it as I see fit…

Desire to Extend

This led me to wanting to create my own version of the “personal feed reader”. Figuring this would be a good project for me to continue practicing my Go.

First, I made a list of packages that would get me to most of the basic functionality of the feed reader, leaving off Mastodon for now.

https://github.com/spf13/cobra
https://git.deanishe.net/deanishe/go-favicon
https://github.com/go-shiori/go-readability
https://github.com/go-co-op/gocron
https://github.com/mmcdole/gofeed
https://github.com/PuerkitoBio/goquery

I’m going to stick with using htmx and Bulma CSS as it’ll be a nice excuse to learn a bit about them.

To start my hacking away on the project I began with using the Cobra command line tools to stand up a basic app. From here I added skeleton commands, version, feed-load, serve.

version - doesn’t do much of anything at the moment.
feed-load - reads a local CSV file and prints out the contents.
serve - starts the HTTP server on 3000 and is slowly being extended to do other tasks.

What other tasks, you say?

While right now running serve will start up the web server, begin the scheduler and schedule a dummy job that just says hello. It will also load the feed CSV and do some parsing of the included feeds and then use readability on the feed links.

Databasing

My next step is to start working on the database. I need to do some research on properly doing migrations. The Go program/library migrate looks like it may be useful in that regard. At first we’ll likely be modeling tables after what’s already presented to us as using the base feedi as a design building block.

0_create_table.down.sql

DROP TABLE IF EXISTS feeds;

0_create_table.up.sql

CREATE TABLE feeds (
    id                  INTEGER   NOT NULL,
    user_id             INTEGER   NOT NULL,
    url                 VARCHAR,
    type                VARCHAR   NOT NULL,
    name                VARCHAR,
    icon_url            VARCHAR,
    created             TIMESTAMP NOT NULL,
    updated             TIMESTAMP NOT NULL,
    last_fetch          TIMESTAMP,
    raw_data            VARCHAR,
    folder              VARCHAR,
    etag                VARCHAR,
    modified_header     VARCHAR,
    filters             VARCHAR,
    mastodon_account_id INTEGER,
    PRIMARY KEY (
        id
    ),
    UNIQUE (
        user_id,
        name
    )
);

With the up/down files in ./migrations I should be able to add a command migrate and use the library to do the heavy lifting. I think, we’ll have to give it a whirl. Maybe I’ll write a small test project…

And my small test project was a success it seems. If you are interested here is the code.

package main

import (
	"database/sql"
	"fmt"
	"log"

	"github.com/golang-migrate/migrate/v4"
	"github.com/golang-migrate/migrate/v4/database/sqlite3"
	"github.com/golang-migrate/migrate/v4/source/file"
	_ "github.com/mattn/go-sqlite3"
)

func main() {
	// Connect to database
	db, err := sql.Open("sqlite3", "feeds.db")
	if err != nil {
		log.Panic(err)
	}
	// defer close
	defer db.Close()

	dbDriver, err := sqlite3.WithInstance(db, &sqlite3.Config{})
	if err != nil {
		fmt.Printf("instance error: %v \n", err)
	}

	fileSource, err := (&file.File{}).Open("file://migrations")
	if err != nil {
		fmt.Printf("opening file error: %v \n", err)
	}

	m, err := migrate.NewWithInstance("file", fileSource, "feeds.db", dbDriver)
	if err != nil {
		fmt.Printf("migrate error: %v \n", err)
	}

	if err = m.Up(); err != nil {
		fmt.Printf("migrate up error: %v \n", err)
	}

	fmt.Println("Migrate up done with success")

}

Enjoy this post?
How about buying me a coffee?

Getting Reacquainted With Hugo Again

Tue, 25 Jun 2024 20:55:20 -0700

Testing testing testing

It’s been a bit since I’ve looked at my Hugo blog here on shindakun.dev.

It happens - life gets in the way of blogging.

The nice thing about having everything in Git and GitHub is that for the most part all I need to do is make sure Hugo is up to date and works on the server. This post is a part of that process.

Hugo

First I built Hugo

➜  shindakun.dev git:(master) ✗ CGO_ENABLED=1 go install -tags extended github.com/gohugoio/hugo@latest

Easy enough!

➜  shindakun.dev git:(master) ✗ hugo version
hugo v0.127.0+extended linux/amd64 BuildDate=unknown

Hmm… unknown build date? I mean I just built it so that should be todays date. Funny though, I have to tell you I don’t feel like chasing that down at the moment.

shindakun.dev

Next lets make sure shindakun.dev is in a good spot, a quick pull should take care of that.

➜  shindakun.dev git:(master) ✗ git pull --recurse-submodules
warning: redirecting to https://github.com/shindakun/shindakun.dev/
Fetching submodule content/posts
Fetching submodule themes/solar-theme-hugo
Fetching submodule themes/solara-hugo-test
Already up to date.

Everything appears to be in order! Yay.

Now we’ll just finish up this quick post and close the loop.

Enjoy this post?
How about buying me a coffee?

Attempting to Learn Go - Ghost to Hugo 4

Sat, 01 Oct 2022 13:54:29 -0700

Intro

Here we are once again! If you missed the last couple of posts I suggest reading them first to get a feeling for how our little prototype has evolved. This time around we’re going to be doing some refactoring to clean up a bit. The majority of what we’ll be doing is adding methods to our “database” struct.

The Method

What is a method? Well, you can think of it as a function that is attached to a struct. Or as A Tour of Go puts it

A method is a function with a special receiver argument.

In our code we have a struct GhostDatabase. We can add methods with the following signature where the receiver is our struct.

func (gd *GhostDatabase) getLength() int

We could then call the method with the following

var db GhostDatabase

...load JSON code here...

length := db.getLength()

OK so what’s our complete method look like? In the case of most of the methods we’ll be putting in place is pretty easy to reason about.

func (gd *GhostDatabase) getLength() int {
	return len(gd.Db[0].Data.Posts)
}

Many Methods

Now lets quickly knock out some useful methods. For the most part they do what they say on the tin, so I won’t go into detail on each one.

func (gd *GhostDatabase) getMobiledoc(i int) string {
	return gd.Db[0].Data.Posts[i].Mobiledoc
}

func (gd *GhostDatabase) getPostId(i int) string {
	return gd.Db[0].Data.Posts[i].ID
}

func (gd *GhostDatabase) getPostTitle(i int) string {
	return gd.Db[0].Data.Posts[i].Title
}

func (gd *GhostDatabase) getPostSlug(i int) string {
	return gd.Db[0].Data.Posts[i].Slug
}

func (gd *GhostDatabase) getPostStatus(i int) string {
	return gd.Db[0].Data.Posts[i].Status
}

func (gd *GhostDatabase) getPostCreatedAt(i int) time.Time {
	return gd.Db[0].Data.Posts[i].CreatedAt
}

func (gd *GhostDatabase) getPostUpdatedAt(i int) time.Time {
	return gd.Db[0].Data.Posts[i].UpdatedAt
}

func (gd *GhostDatabase) getPostPublishedAt(i int) time.Time {
	return gd.Db[0].Data.Posts[i].PublishedAt
}

func (gd *GhostDatabase) getPostFeatureImage(i int) string {
	return gd.Db[0].Data.Posts[i].FeatureImage
}

Next Time

That’s all of our methods for now. With them out of the way I believe we have all the parts need to actually start writing our Markdown files!

Enjoy this post?
How about buying me a coffee?

Code Listing

package main

import (
	"encoding/json"
	"fmt"
	"io"
	"os"
	"reflect"
	"strconv"
	"strings"
	"time"
)

type GhostDatabase struct {
	Db []struct {
		Meta struct {
			ExportedOn int64  `json:"exported_on"`
			Version    string `json:"version"`
		} `json:"meta"`
		Data struct {
			Posts []struct {
				ID                   string      `json:"id"`
				UUID                 string      `json:"uuid"`
				Title                string      `json:"title"`
				Slug                 string      `json:"slug"`
				Mobiledoc            string      `json:"mobiledoc"`
				HTML                 string      `json:"html"`
				CommentID            string      `json:"comment_id"`
				Plaintext            string      `json:"plaintext"`
				FeatureImage         string      `json:"feature_image"`
				Featured             int         `json:"featured"`
				Type                 string      `json:"type"`
				Status               string      `json:"status"`
				Locale               interface{} `json:"locale"`
				Visibility           string      `json:"visibility"`
				EmailRecipientFilter string      `json:"email_recipient_filter"`
				AuthorID             string      `json:"author_id"`
				CreatedAt            time.Time   `json:"created_at"`
				UpdatedAt            time.Time   `json:"updated_at"`
				PublishedAt          time.Time   `json:"published_at"`
				CustomExcerpt        interface{} `json:"custom_excerpt"`
				CodeinjectionHead    interface{} `json:"codeinjection_head"`
				CodeinjectionFoot    interface{} `json:"codeinjection_foot"`
				CustomTemplate       interface{} `json:"custom_template"`
				CanonicalURL         interface{} `json:"canonical_url"`
			} `json:"posts"`
			PostsAuthors []struct {
				ID        string `json:"id"`
				PostID    string `json:"post_id"`
				AuthorID  string `json:"author_id"`
				SortOrder int    `json:"sort_order"`
			} `json:"posts_authors"`
			PostsMeta []interface{} `json:"posts_meta"`
			PostsTags []struct {
				ID        string `json:"id"`
				PostID    string `json:"post_id"`
				TagID     string `json:"tag_id"`
				SortOrder int    `json:"sort_order"`
			} `json:"posts_tags"`
			Roles []struct {
				ID          string    `json:"id"`
				Name        string    `json:"name"`
				Description string    `json:"description"`
				CreatedAt   time.Time `json:"created_at"`
				UpdatedAt   time.Time `json:"updated_at"`
			} `json:"roles"`
			RolesUsers []struct {
				ID     string `json:"id"`
				RoleID string `json:"role_id"`
				UserID string `json:"user_id"`
			} `json:"roles_users"`
			Settings []struct {
				ID        string      `json:"id"`
				Group     string      `json:"group"`
				Key       string      `json:"key"`
				Value     string      `json:"value"`
				Type      string      `json:"type"`
				Flags     interface{} `json:"flags"`
				CreatedAt time.Time   `json:"created_at"`
				UpdatedAt time.Time   `json:"updated_at"`
			} `json:"settings"`
			Tags []struct {
				ID                 string      `json:"id"`
				Name               string      `json:"name"`
				Slug               string      `json:"slug"`
				Description        interface{} `json:"description"`
				FeatureImage       interface{} `json:"feature_image"`
				ParentID           interface{} `json:"parent_id"`
				Visibility         string      `json:"visibility"`
				OgImage            interface{} `json:"og_image"`
				OgTitle            interface{} `json:"og_title"`
				OgDescription      interface{} `json:"og_description"`
				TwitterImage       interface{} `json:"twitter_image"`
				TwitterTitle       interface{} `json:"twitter_title"`
				TwitterDescription interface{} `json:"twitter_description"`
				MetaTitle          interface{} `json:"meta_title"`
				MetaDescription    interface{} `json:"meta_description"`
				CodeinjectionHead  interface{} `json:"codeinjection_head"`
				CodeinjectionFoot  interface{} `json:"codeinjection_foot"`
				CanonicalURL       interface{} `json:"canonical_url"`
				AccentColor        interface{} `json:"accent_color"`
				CreatedAt          time.Time   `json:"created_at"`
				UpdatedAt          time.Time   `json:"updated_at"`
			} `json:"tags"`
			Users []struct {
				ID              string      `json:"id"`
				Name            string      `json:"name"`
				Slug            string      `json:"slug"`
				Password        string      `json:"password"`
				Email           string      `json:"email"`
				ProfileImage    string      `json:"profile_image"`
				CoverImage      interface{} `json:"cover_image"`
				Bio             interface{} `json:"bio"`
				Website         interface{} `json:"website"`
				Location        interface{} `json:"location"`
				Facebook        interface{} `json:"facebook"`
				Twitter         interface{} `json:"twitter"`
				Accessibility   string      `json:"accessibility"`
				Status          string      `json:"status"`
				Locale          interface{} `json:"locale"`
				Visibility      string      `json:"visibility"`
				MetaTitle       interface{} `json:"meta_title"`
				MetaDescription interface{} `json:"meta_description"`
				Tour            interface{} `json:"tour"`
				LastSeen        time.Time   `json:"last_seen"`
				CreatedAt       time.Time   `json:"created_at"`
				UpdatedAt       time.Time   `json:"updated_at"`
			} `json:"users"`
		} `json:"data"`
	} `json:"db"`
}

type Mobiledoc struct {
	Version      string          `json:"version"`
	Markups      []interface{}   `json:"markups"`
	Atoms        []interface{}   `json:"atoms"`
	Cards        [][]interface{} `json:"cards"`
	Sections     [][]interface{} `json:"sections"`
	GhostVersion string          `json:"ghostVersion"`
}

func (gd *GhostDatabase) getLength() int {
	return len(gd.Db[0].Data.Posts)
}

func (gd *GhostDatabase) getMobiledoc(i int) string {
	return gd.Db[0].Data.Posts[i].Mobiledoc
}

func (gd *GhostDatabase) getPostId(i int) string {
	return gd.Db[0].Data.Posts[i].ID
}

func (gd *GhostDatabase) getPostTitle(i int) string {
	return gd.Db[0].Data.Posts[i].Title
}

func (gd *GhostDatabase) getPostSlug(i int) string {
	return gd.Db[0].Data.Posts[i].Slug
}

func (gd *GhostDatabase) getPostStatus(i int) string {
	return gd.Db[0].Data.Posts[i].Status
}

func (gd *GhostDatabase) getPostCreatedAt(i int) time.Time {
	return gd.Db[0].Data.Posts[i].CreatedAt
}

func (gd *GhostDatabase) getPostUpdatedAt(i int) time.Time {
	return gd.Db[0].Data.Posts[i].UpdatedAt
}

func (gd *GhostDatabase) getPostPublishedAt(i int) time.Time {
	return gd.Db[0].Data.Posts[i].PublishedAt
}

func (gd *GhostDatabase) getPostFeatureImage(i int) string {
	return gd.Db[0].Data.Posts[i].FeatureImage
}

func (gd *GhostDatabase) getTags(pid string) []string {
	var r []string
	for i := 0; i < len(gd.Db[0].Data.PostsTags); i++ {
		if gd.Db[0].Data.PostsTags[i].PostID == pid {
			tagId := gd.Db[0].Data.PostsTags[i].TagID
			for j := 0; j < len(gd.Db[0].Data.Tags); j++ {
				if gd.Db[0].Data.Tags[j].ID == tagId {
					r = append(r, gd.Db[0].Data.Tags[j].Name)
				}
			}
		}
	}
	return r
}

func main() {
	fmt.Println("ghost2hugo")

	file, err := os.Open("shindakun-dot-net.ghost.2022-03-18-22-02-58.json")
	if err != nil {
		fmt.Println(err)
	}

	defer file.Close()

	b, err := io.ReadAll(file)
	if err != nil {
		fmt.Println(err)
	}

	var db GhostDatabase

	err = json.Unmarshal(b, &db)
	if err != nil {
		fmt.Println(err)
	}

	for i := 0; i < db.getLength(); i++ {
		fmt.Println(db.getPostTitle(i))
		fmt.Println(db.getPostSlug(i))
		fmt.Println(db.getPostStatus(i))
		fmt.Println(db.getPostCreatedAt(i))
		fmt.Println(db.getPostUpdatedAt(i))
		fmt.Println(db.getPostPublishedAt(i))
		fmt.Println(db.getPostFeatureImage(i))

		id := db.getPostId(i)
		tags := db.getTags(id)
		fmt.Println(tags)

		c := strings.ReplaceAll(db.getMobiledoc(i), "`", "%'")
		cc := "`" + c + "`"

		ucn, err := strconv.Unquote(cc)
		if err != nil {
			fmt.Println(err, ucn)
		}

		ucn = strings.ReplaceAll(ucn, "%'", "`")

		var md Mobiledoc

		err = json.Unmarshal([]byte(ucn), &md)
		if err != nil {
			fmt.Println(err)
		}

		if reflect.ValueOf(md.Cards).Len() > 0 {
			card := md.Cards[0][1]
			bbb := card.(map[string]interface{})

			fmt.Println(bbb["markdown"])
		}

		fmt.Println("---")
	}
}

Attempting to Learn Go - Ghost to Hugo 3

Tue, 27 Sep 2022 17:00:05 -0700

Intro

We’re back once again to continue our work on the ghost2hugo prototype. So far we can open the JSON file, load the data into memory, and print out the Markdown for the first post. The next step is to make sure we can read and process every post included in the backup.

Looping

We’re going to replace all the code that prints a single article with a loop that prints out each article. For a refresher, here is the code that prints the first article.

	c := "`" + db.Db[0].Data.Posts[0].Mobiledoc + "`"

	un, err := strconv.Unquote(c)
	if err != nil {
		fmt.Println(err)
	}
	fmt.Printf("%v", un)

	var md Mobiledoc

	err = json.Unmarshal([]byte(un), &md)
	if err != nil {
		fmt.Println(err)
	}
	
	fmt.Printf("%#v", md)
	
	card := md.Cards[0][1]
	fmt.Printf("\n\ncard: %#v\n", card)
	
	bbb := card.(map[string]interface{})
	fmt.Println(bbb["markdown"])

Most of this will live inside of our new loop.

	for i := 0; i < len(db.Db[0].Data.Posts); i++ {
		fmt.Println(db.Db[0].Data.Posts[i].Title)
		fmt.Println(db.Db[0].Data.Posts[i].Slug)
		fmt.Println(db.Db[0].Data.Posts[i].Status)
		fmt.Println(db.Db[0].Data.Posts[i].CreatedAt)
		fmt.Println(db.Db[0].Data.Posts[i].UpdatedAt)

		cc := "`" + db.Db[0].Data.Posts[i].Mobiledoc + "`"

		ucn, err := strconv.Unquote(cc)
		if err != nil {
			fmt.Println(err, ucn)
		}
		fmt.Printf("\n\n\n%v\n\n\n", ucn)

		var md Mobiledoc

		err = json.Unmarshal([]byte(ucn), &md)
		if err != nil {
			fmt.Println(err)
		}

		card := md.Cards[0][1]
		bbb := card.(map[string]interface{})

		fmt.Println(bbb["markdown"])
		}
	}

And when we run the code this time we see a bunch of articles fly by! Until, that is, we hit an error! The invalid syntax error indicates this is a problem with strconv.Unquote().

invalid syntax 

unexpected end of JSON input
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
main.main()
        /Users/steve/Code/ghost2hugo/main.go:214 +0x769
exit status 2

Roadblocks

Well that’s no good, we can’t just have our code crash out like that! Let’s take a closer look at the problem and see if we can work our way through it. OK, now for the master of all debugging techniques! We add a fmt.Println to print out the chunk of text.

		fmt.Println(cc)

Running the code again we crash out as expected but this time we’re presented with the problem text:

`{"version":"0.3.1","markups":[],"atoms":[],"cards":[["markdown",{"cardName":"card-markdown","markdown":"\n* Incresed title to 64 characters. Should be more then enough.  \n * ics and other code work over at angstridden dot net tonight.\n\nALTER TABLE `posts` CHANGE `posttitle` `posttitle` VARCHAR( 64 ) NOT NULL\n\n\n"}]],"sections":[[10,0]],"ghostVersion":"3.0"}`

See the problem? Yeah, it looks like there are back ticks in the Markdown that are causing the string convert to not work as expected. To get around this we’re going to do a simple strings.ReplaceAll before the conversion to change the back tick to something else. We can then reverse that process to get the normal text. We’ll replace the back tick with “%'”.

		c := strings.ReplaceAll(db.Db[0].Data.Posts[i].Mobiledoc, "`", "%'")
		cc := "`" + c + "`"

After we “unquote” we can run another strings.ReplaceAll to convert back.

		ucn = strings.ReplaceAll(ucn, "%'", "`")

Running again we run into another problem!

---
Archives
archives-post
published
2007-07-01 09:46:18 +0000 UTC
2007-08-12 20:50:02 +0000 UTC
`{"version":"0.3.1","markups":[],"atoms":[],"cards":[],"sections":[[1,"p",[[0,[],0,""]]]],"ghostVersion":"3.0"}`
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
main.main()
        /Users/steve/Code/ghost2hugo/main.go:214 +0x7a6
exit status 2

Argh!

More Roadblocks

Our naive implementation from the previous post has returned to bite us! We were assuming that cards exist.

		card := md.Cards[0][1]
		bbb := card.(map[string]interface{})

Looking at the returned JSON confirms this.

{"version":"0.3.1","markups":[],"atoms":[],"cards":[],"sections":[[1,"p",[[0,[],0,""]]]],"ghostVersion":"3.0"}

Now, how do we check to see if the cards field is empty?! I noodled on it a little bit and figured the best way would be to use reflect.ValueOf().Len(). This will check the length of md.Cards to ensure we actually have some data.

		if reflect.ValueOf(md.Cards).Len() > 0 {
			card := md.Cards[0][1]
			bbb := card.(map[string]interface{})

			fmt.Println(bbb["markdown"])
		}

On rerunning we get what looks to be all the posts. Except the final draft which looks odd! It turns out this one has no Markdown card. Is this the only one? I certainly hope so since it looks like it may be a pain to convert to Markdown.

Questing
questing
draft
2019-08-23 21:26:44 +0000 UTC
2021-04-12 04:53:26 +0000 UTC
`{"version":"0.3.1","atoms":[],"cards":[["paywall",{}]],"markups":[["a",["href","https://shindakun.dev"]],["a",["href","https://dev.to/shindakun"]]],"sections":[[1,"p",[[0,[],0,"It's been quite sometime since I've posted anything here. I've done a bit of posting over on "],[0,[0],1,"shindakun.dev"],[0,[],0," and over on "],[0,[1],1,"DEV"],[0,[],0," which has been kind of nice. But, those don't really cover gaming at all. But, I updated the site recently and everyone once in a while I use the site to try something out for work and seeing an old post over and over was no good."]]],[1,"p",[[0,[],0,"I only seem to have a small sliver of time for gaming now. With everything else I want/need to get done there is only so much time. Heed this warning - don't get older! Just kidding, it's not so bad."]]],[1,"p",[[0,[],0,"My 5 year old hasn't been introduced into gaming much outside of some basic games she can play on her tablet. I have discovered that she enjoys watching me play the digital version of Warhammer Quest. Which is good since she's going to be getting a crash course in some more difficult board games sooner or later. I have a copy of the Gloomhaven digital board game and the boxed game (and expansion) but haven't actually event opened it yet. Maybe I should fix that this weekend..."]]],[10,0],[1,"p",[]]],"ghostVersion":"3.0"}`
json: cannot unmarshal string into Go struct field Mobiledoc.sections of type int
<nil>

Looks like we also have an issue unmarshaling, on that last line there. This is easy enough to fix for now. We just need to update sections in our Mobiledoc struct to use [][]interface{} and not [][]int.

type Mobiledoc struct {
	Version      string          `json:"version"`
	Markups      []interface{}   `json:"markups"`
	Atoms        []interface{}   `json:"atoms"`
	Cards        [][]interface{} `json:"cards"`
	Sections     [][]interface{} `json:"sections"`
	GhostVersion string          `json:"ghostVersion"`
}

Next Time

We’ve made some good progress over the last few posts. On the first look it seems we’re extracting what we need. I think we’re ready to do a little refactoring, remove our current debug print statements, and get ready for the next part of our converter.

Enjoy this post?
How about buying me a coffee?

Code Listing

package main

import (
	"encoding/json"
	"fmt"
	"io"
	"os"
	"reflect"
	"strconv"
	"strings"
	"time"
)

type GhostDatabase struct {
	Db []struct {
		Meta struct {
			ExportedOn int64  `json:"exported_on"`
			Version    string `json:"version"`
		} `json:"meta"`
		Data struct {
			Posts []struct {
				ID                   string      `json:"id"`
				UUID                 string      `json:"uuid"`
				Title                string      `json:"title"`
				Slug                 string      `json:"slug"`
				Mobiledoc            string      `json:"mobiledoc"`
				HTML                 string      `json:"html"`
				CommentID            string      `json:"comment_id"`
				Plaintext            string      `json:"plaintext"`
				FeatureImage         interface{} `json:"feature_image"`
				Featured             int         `json:"featured"`
				Type                 string      `json:"type"`
				Status               string      `json:"status"`
				Locale               interface{} `json:"locale"`
				Visibility           string      `json:"visibility"`
				EmailRecipientFilter string      `json:"email_recipient_filter"`
				AuthorID             string      `json:"author_id"`
				CreatedAt            time.Time   `json:"created_at"`
				UpdatedAt            time.Time   `json:"updated_at"`
				PublishedAt          time.Time   `json:"published_at"`
				CustomExcerpt        interface{} `json:"custom_excerpt"`
				CodeinjectionHead    interface{} `json:"codeinjection_head"`
				CodeinjectionFoot    interface{} `json:"codeinjection_foot"`
				CustomTemplate       interface{} `json:"custom_template"`
				CanonicalURL         interface{} `json:"canonical_url"`
			} `json:"posts"`
			PostsAuthors []struct {
				ID        string `json:"id"`
				PostID    string `json:"post_id"`
				AuthorID  string `json:"author_id"`
				SortOrder int    `json:"sort_order"`
			} `json:"posts_authors"`
			PostsMeta []interface{} `json:"posts_meta"`
			PostsTags []struct {
				ID        string `json:"id"`
				PostID    string `json:"post_id"`
				TagID     string `json:"tag_id"`
				SortOrder int    `json:"sort_order"`
			} `json:"posts_tags"`
			Roles []struct {
				ID          string    `json:"id"`
				Name        string    `json:"name"`
				Description string    `json:"description"`
				CreatedAt   time.Time `json:"created_at"`
				UpdatedAt   time.Time `json:"updated_at"`
			} `json:"roles"`
			RolesUsers []struct {
				ID     string `json:"id"`
				RoleID string `json:"role_id"`
				UserID string `json:"user_id"`
			} `json:"roles_users"`
			Settings []struct {
				ID        string      `json:"id"`
				Group     string      `json:"group"`
				Key       string      `json:"key"`
				Value     string      `json:"value"`
				Type      string      `json:"type"`
				Flags     interface{} `json:"flags"`
				CreatedAt time.Time   `json:"created_at"`
				UpdatedAt time.Time   `json:"updated_at"`
			} `json:"settings"`
			Tags []struct {
				ID                 string      `json:"id"`
				Name               string      `json:"name"`
				Slug               string      `json:"slug"`
				Description        interface{} `json:"description"`
				FeatureImage       interface{} `json:"feature_image"`
				ParentID           interface{} `json:"parent_id"`
				Visibility         string      `json:"visibility"`
				OgImage            interface{} `json:"og_image"`
				OgTitle            interface{} `json:"og_title"`
				OgDescription      interface{} `json:"og_description"`
				TwitterImage       interface{} `json:"twitter_image"`
				TwitterTitle       interface{} `json:"twitter_title"`
				TwitterDescription interface{} `json:"twitter_description"`
				MetaTitle          interface{} `json:"meta_title"`
				MetaDescription    interface{} `json:"meta_description"`
				CodeinjectionHead  interface{} `json:"codeinjection_head"`
				CodeinjectionFoot  interface{} `json:"codeinjection_foot"`
				CanonicalURL       interface{} `json:"canonical_url"`
				AccentColor        interface{} `json:"accent_color"`
				CreatedAt          time.Time   `json:"created_at"`
				UpdatedAt          time.Time   `json:"updated_at"`
			} `json:"tags"`
			Users []struct {
				ID              string      `json:"id"`
				Name            string      `json:"name"`
				Slug            string      `json:"slug"`
				Password        string      `json:"password"`
				Email           string      `json:"email"`
				ProfileImage    string      `json:"profile_image"`
				CoverImage      interface{} `json:"cover_image"`
				Bio             interface{} `json:"bio"`
				Website         interface{} `json:"website"`
				Location        interface{} `json:"location"`
				Facebook        interface{} `json:"facebook"`
				Twitter         interface{} `json:"twitter"`
				Accessibility   string      `json:"accessibility"`
				Status          string      `json:"status"`
				Locale          interface{} `json:"locale"`
				Visibility      string      `json:"visibility"`
				MetaTitle       interface{} `json:"meta_title"`
				MetaDescription interface{} `json:"meta_description"`
				Tour            interface{} `json:"tour"`
				LastSeen        time.Time   `json:"last_seen"`
				CreatedAt       time.Time   `json:"created_at"`
				UpdatedAt       time.Time   `json:"updated_at"`
			} `json:"users"`
		} `json:"data"`
	} `json:"db"`
}

type Mobiledoc struct {
	Version      string          `json:"version"`
	Markups      []interface{}   `json:"markups"`
	Atoms        []interface{}   `json:"atoms"`
	Cards        [][]interface{} `json:"cards"`
	Sections     [][]interface{} `json:"sections"`
	GhostVersion string          `json:"ghostVersion"`
}

func main() {
	fmt.Println("ghost2hugo")

	file, err := os.Open("shindakun-dot-net.ghost.2022-03-18-22-02-58.json")
	if err != nil {
		fmt.Println(err)
	}

	defer file.Close()
	
	b, err := io.ReadAll(file)
	if err != nil {
		fmt.Println(err)
	}

	var db GhostDatabase

	err = json.Unmarshal(b, &db)
	if err != nil {
		fmt.Println(err)
	}

	for i := 0; i < len(db.Db[0].Data.Posts); i++ {
		fmt.Println(db.Db[0].Data.Posts[i].Title)
		fmt.Println(db.Db[0].Data.Posts[i].Slug)
		fmt.Println(db.Db[0].Data.Posts[i].Status)
		fmt.Println(db.Db[0].Data.Posts[i].CreatedAt)
		fmt.Println(db.Db[0].Data.Posts[i].UpdatedAt)

		c := strings.ReplaceAll(db.Db[0].Data.Posts[i].Mobiledoc, "`", "%'")
		cc := "`" + c + "`"

		fmt.Println(cc)

		ucn, err := strconv.Unquote(cc)
		if err != nil {
			fmt.Println(err, ucn)
		}

		ucn = strings.ReplaceAll(ucn, "%'", "`")

		var md Mobiledoc

		err = json.Unmarshal([]byte(ucn), &md)
		if err != nil {
			fmt.Println(err)
		}

		if reflect.ValueOf(md.Cards).Len() > 0 {
			card := md.Cards[0][1]
			bbb := card.(map[string]interface{})

			fmt.Println(bbb["markdown"])
		}

		fmt.Println("---")
	}
}

Attempting to Learn Go - Ghost to Hugo 2

Thu, 22 Sep 2022 19:27:29 -0700

Intro

Welcome back! We are continuing on our journey to make a prototype program that converts an exported Ghost database to Markdown. With the end goal being that we can get shindakun.net up and running with Hugo. Last time, we took it pretty easy and focused mostly on reading the file into memory and converting the JSON to a Go struct. From there we printed out the first post.

Post Data

As a recap here is what one of the post fields contain.

{
	"id": "60710b90705967038fe662d6",
	"uuid": "71ba3d71-ac18-4f33-82f7-1962baa83a07",
	"title": "db test",
	"slug": "db-test",
	"mobiledoc": "{\"version\":\"0.3.1\",\"markups\":[],\"atoms\":[],\"cards\":[[\"markdown\",{\"cardName\":\"card-markdown\",\"markdown\":\"\\n<strike>This is a db test</strike>.\\n\"}]],\"sections\":[[10,0]],\"ghostVersion\":\"3.0\"}",
	"html": "<!--kg-card-begin: markdown--><p><strike>This is a db test</strike>.</p>\n<!--kg-card-end: markdown-->",
	"comment_id": "2",
	"plaintext": "This is a db test.",
	"feature_image": null,
	"featured": 0,
	"type": "post",
	"status": "published",
	"locale": null,
	"visibility": "public",
	"email_recipient_filter": "none",
	"author_id": "60710b8d705967038fe66214",
	"created_at": "2004-08-09T19:11:20.000Z",
	"updated_at": "2004-08-09T19:11:20.000Z",
	"published_at": "2004-08-09T19:11:20.000Z",
	"custom_excerpt": null,
	"codeinjection_head": null,
	"codeinjection_foot": null,
	"custom_template": null,
	"canonical_url": null
},

The majority of the fields we need to craft a post with appropriate frontmatter exist within this the object we’re getting back. We can see the title, slug, published date, etc. The section that contains the Markdown is in a format known as Mobiledoc.

Mobiledoc

According to the Ghost documentation Mobiledoc is

…a standardised JSON-based document storage format, which forms the heart of publishing with Ghost.

When extracted from the JSON object and cleaned up we’ll have another bit of JSON we can work with.

{
    "version": "0.3.1",
    "markups": [],
    "atoms": [],
    "cards": [
        [
            "markdown",
            {
                "cardName": "card-markdown",
                "markdown": "\n<strike>This is a db test</strike>.\n"
            }
        ]
    ],
    "sections": [
        [
            10,
            0
        ]
    ],
    "ghostVersion": "3.0"
}

Sounds Easy

First we’ll use our favorite site, the JSON to Go converter to convert the JSON object to a struct we can work with.

type Mobiledoc struct {
	Version      string          `json:"version"`
	Markups      []interface{}   `json:"markups"`
	Atoms        []interface{}   `json:"atoms"`
	Cards        [][]interface{} `json:"cards"`
	Sections     [][]int         `json:"sections"`
	GhostVersion string          `json:"ghostVersion"`
}

Our code is pretty long now so I’m going to leave out the other struct GhostDatabase struct, it’ll be in the complete code listing below though. We’re still going to be dumping code to the screen since we’re still working on our decoding logic.

package main

import (
	"encoding/json"
	"fmt"
	"io"
	"os"
	"strconv"
	"time"
)

type GhostDatabase struct {...}

type Mobiledoc struct {
	Version      string          `json:"version"`
	Markups      []interface{}   `json:"markups"`
	Atoms        []interface{}   `json:"atoms"`
	Cards        [][]interface{} `json:"cards"`
	Sections     [][]int         `json:"sections"`
	GhostVersion string          `json:"ghostVersion"`
}

func main() {
	fmt.Println("ghost2hugo")

	file, err := os.Open("shindakun-dot-net.ghost.2022-03-18-22-02-58.json")
	if err != nil {
		fmt.Println(err)
	}

	defer file.Close()

	b, err := io.ReadAll(file)
	if err != nil {
		fmt.Println(err)
	}

	var db GhostDatabase

	err = json.Unmarshal(b, &db)
	if err != nil {
		fmt.Println(err)
	}

Let’s continue to focus on the first post for now since once we have that working it should just be a matter of looping through the “database”. This is where it gets a little tricky. We’re working with a couple of nested arrays so to work our way down to the appropriate section we use db.Db[0].Data.Posts[0].Mobiledoc. This will give use the escaped version of our JSON object.

"{\"version\":\"0.3.1\",\"markups\":[],\"atoms\":[],\"cards\":[[\"markdown\",{\"cardName\":\"card-markdown\",\"markdown\":\"\\n<strike>This is a db test</strike>.\\n\"}]],\"sections\":[[10,0]],\"ghostVersion\":\"3.0\"}"

Let’s Go On A Trip

I knew that there would be a way to unescape the string that we get, checking the Go documentation led me to strconv.Unquote.

Unquote interprets s as a single-quoted, double-quoted, or backquoted Go string literal, returning the string value that s quotes. (If s is single-quoted, it would be a Go character literal; Unquote returns the corresponding one-character string.)

Exactly what we need! Except that when I tried to unquote the string I kept receiving an invalid syntax error. This had me confused for a little bit. After puzzling over it I realized if I prepended and appended a back tick to the string it seems to be treated as a raw string literal. This leads to code that looks like the following.

	c := "`" + db.Db[0].Data.Posts[0].Mobiledoc + "`"

	un, err := strconv.Unquote(c)
	if err != nil {
		fmt.Println(err)
	}
	fmt.Printf("%v\n", un)

Finally! We have the JSON!

{"version":"0.3.1","markups":[],"atoms":[],"cards":[["markdown",{"cardName":"card-markdown","markdown":"\n<strike>This is a db test</strike>.\n"}]],"sections":[[10,0]],"ghostVersion":"3.0"}

Now, we can unmarshal that into the Mobiledoc struct we set up earlier!

	var md Mobiledoc

	err = json.Unmarshal([]byte(un), &md)
	if err != nil {
		fmt.Println(err)
	}
	
	fmt.Printf("%#v", md)

Checking our dumped result we can see that we’ve got the expected data.

main.Mobiledoc{Version:"0.3.1", Markups:[]interface {}{}, Atoms:[]interface {}{}, Cards:[][]interface {}{[]interface {}{"markdown", map[string]interface {}{"cardName":"card-markdown", "markdown":"\n<strike>This is a db test</strike>.\n"}}}, Sections:[][]int{[]int{10, 0}}, GhostVersion:"3.0"}

Right now we’re concerned with the section called Cards which if we naively want to access we can use the following.

	card := md.Cards[0][1]
	fmt.Printf("\n\ncard: %#v\n", card)

card: map[string]interface {}{"cardName":"card-markdown", "markdown":"\n<strike>This is a db test</strike>.\n"}

Let’s convert that to something a bit easier to access.

	bbb := card.(map[string]interface{})
	fmt.Println(bbb["markdown"])

<strike>This is a db test</strike>.

Next Time

So far so good, we’ve extracted the Markdown from the first post as expected. Next time around we’ll be writing a small loop to start trying to extract all the posts. I have a feeling that’s where the fun will begin.

Enjoy this post?
How about buying me a coffee?

Code Listing

package main

import (
	"encoding/json"
	"fmt"
	"io"
	"os"
	"strconv"
	"time"
)

type GhostDatabase struct {
	Db []struct {
		Meta struct {
			ExportedOn int64  `json:"exported_on"`
			Version    string `json:"version"`
		} `json:"meta"`
		Data struct {
			Posts []struct {
				ID                   string      `json:"id"`
				UUID                 string      `json:"uuid"`
				Title                string      `json:"title"`
				Slug                 string      `json:"slug"`
				Mobiledoc            string      `json:"mobiledoc"`
				HTML                 string      `json:"html"`
				CommentID            string      `json:"comment_id"`
				Plaintext            string      `json:"plaintext"`
				FeatureImage         interface{} `json:"feature_image"`
				Featured             int         `json:"featured"`
				Type                 string      `json:"type"`
				Status               string      `json:"status"`
				Locale               interface{} `json:"locale"`
				Visibility           string      `json:"visibility"`
				EmailRecipientFilter string      `json:"email_recipient_filter"`
				AuthorID             string      `json:"author_id"`
				CreatedAt            time.Time   `json:"created_at"`
				UpdatedAt            time.Time   `json:"updated_at"`
				PublishedAt          time.Time   `json:"published_at"`
				CustomExcerpt        interface{} `json:"custom_excerpt"`
				CodeinjectionHead    interface{} `json:"codeinjection_head"`
				CodeinjectionFoot    interface{} `json:"codeinjection_foot"`
				CustomTemplate       interface{} `json:"custom_template"`
				CanonicalURL         interface{} `json:"canonical_url"`
			} `json:"posts"`
			PostsAuthors []struct {
				ID        string `json:"id"`
				PostID    string `json:"post_id"`
				AuthorID  string `json:"author_id"`
				SortOrder int    `json:"sort_order"`
			} `json:"posts_authors"`
			PostsMeta []interface{} `json:"posts_meta"`
			PostsTags []struct {
				ID        string `json:"id"`
				PostID    string `json:"post_id"`
				TagID     string `json:"tag_id"`
				SortOrder int    `json:"sort_order"`
			} `json:"posts_tags"`
			Roles []struct {
				ID          string    `json:"id"`
				Name        string    `json:"name"`
				Description string    `json:"description"`
				CreatedAt   time.Time `json:"created_at"`
				UpdatedAt   time.Time `json:"updated_at"`
			} `json:"roles"`
			RolesUsers []struct {
				ID     string `json:"id"`
				RoleID string `json:"role_id"`
				UserID string `json:"user_id"`
			} `json:"roles_users"`
			Settings []struct {
				ID        string      `json:"id"`
				Group     string      `json:"group"`
				Key       string      `json:"key"`
				Value     string      `json:"value"`
				Type      string      `json:"type"`
				Flags     interface{} `json:"flags"`
				CreatedAt time.Time   `json:"created_at"`
				UpdatedAt time.Time   `json:"updated_at"`
			} `json:"settings"`
			Tags []struct {
				ID                 string      `json:"id"`
				Name               string      `json:"name"`
				Slug               string      `json:"slug"`
				Description        interface{} `json:"description"`
				FeatureImage       interface{} `json:"feature_image"`
				ParentID           interface{} `json:"parent_id"`
				Visibility         string      `json:"visibility"`
				OgImage            interface{} `json:"og_image"`
				OgTitle            interface{} `json:"og_title"`
				OgDescription      interface{} `json:"og_description"`
				TwitterImage       interface{} `json:"twitter_image"`
				TwitterTitle       interface{} `json:"twitter_title"`
				TwitterDescription interface{} `json:"twitter_description"`
				MetaTitle          interface{} `json:"meta_title"`
				MetaDescription    interface{} `json:"meta_description"`
				CodeinjectionHead  interface{} `json:"codeinjection_head"`
				CodeinjectionFoot  interface{} `json:"codeinjection_foot"`
				CanonicalURL       interface{} `json:"canonical_url"`
				AccentColor        interface{} `json:"accent_color"`
				CreatedAt          time.Time   `json:"created_at"`
				UpdatedAt          time.Time   `json:"updated_at"`
			} `json:"tags"`
			Users []struct {
				ID              string      `json:"id"`
				Name            string      `json:"name"`
				Slug            string      `json:"slug"`
				Password        string      `json:"password"`
				Email           string      `json:"email"`
				ProfileImage    string      `json:"profile_image"`
				CoverImage      interface{} `json:"cover_image"`
				Bio             interface{} `json:"bio"`
				Website         interface{} `json:"website"`
				Location        interface{} `json:"location"`
				Facebook        interface{} `json:"facebook"`
				Twitter         interface{} `json:"twitter"`
				Accessibility   string      `json:"accessibility"`
				Status          string      `json:"status"`
				Locale          interface{} `json:"locale"`
				Visibility      string      `json:"visibility"`
				MetaTitle       interface{} `json:"meta_title"`
				MetaDescription interface{} `json:"meta_description"`
				Tour            interface{} `json:"tour"`
				LastSeen        time.Time   `json:"last_seen"`
				CreatedAt       time.Time   `json:"created_at"`
				UpdatedAt       time.Time   `json:"updated_at"`
			} `json:"users"`
		} `json:"data"`
	} `json:"db"`
}

type Mobiledoc struct {
	Version      string          `json:"version"`
	Markups      []interface{}   `json:"markups"`
	Atoms        []interface{}   `json:"atoms"`
	Cards        [][]interface{} `json:"cards"`
	Sections     [][]int         `json:"sections"`
	GhostVersion string          `json:"ghostVersion"`
}

func main() {
	fmt.Println("ghost2hugo")

	file, err := os.Open("shindakun-dot-net.ghost.2022-03-18-22-02-58.json")
	if err != nil {
		fmt.Println(err)
	}

	defer file.Close()
	
	b, err := io.ReadAll(file)
	if err != nil {
		fmt.Println(err)
	}

	var db GhostDatabase

	err = json.Unmarshal(b, &db)
	if err != nil {
		fmt.Println(err)
	}

	c := "`" + db.Db[0].Data.Posts[0].Mobiledoc + "`"

	un, err := strconv.Unquote(c)
	if err != nil {
		fmt.Println(err)
	}
	fmt.Printf("%v", un)

	var md Mobiledoc

	err = json.Unmarshal([]byte(un), &md)
	if err != nil {
		fmt.Println(err)
	}
	
	fmt.Printf("%#v", md)
	
	card := md.Cards[0][1]
	fmt.Printf("\n\ncard: %#v\n", card)
	
	bbb := card.(map[string]interface{})
	fmt.Println(bbb["markdown"])
}

Attempting to Learn Go - Ghost to Hugo 1

Sun, 18 Sep 2022 20:54:29 -0700

Intro

You probably don’t know this but shindakun.net has been effectively offline now for months. Currently, visiting it will only show a white page and a pop-up, no content is accessible. This is a side effect to my upgrading to a new server and moving away from Ghost as my platform of choice.

I love(d) Ghost but it just got too large to effectively run multiple sites on a small instance on Digital Ocean. I want to get the site back up and running but this time using Hugo. This seems like a great opportunity to make a new series of ATLG posts… plus it’s been a while since I’ve written any Go.

Post Data

Before shutting down the original server I exported a copy of the Ghost database as JSON. I should be able to use this to build out the needed Markdown files to make up the content of the site. Here’s the first post.

{
	"id": "60710b90705967038fe662d6",
	"uuid": "71ba3d71-ac18-4f33-82f7-1962baa83a07",
	"title": "db test",
	"slug": "db-test",
	"mobiledoc": "{\"version\":\"0.3.1\",\"markups\":[],\"atoms\":[],\"cards\":[[\"markdown\",{\"cardName\":\"card-markdown\",\"markdown\":\"\\n<strike>This is a db test</strike>.\\n\"}]],\"sections\":[[10,0]],\"ghostVersion\":\"3.0\"}",
	"html": "<!--kg-card-begin: markdown--><p><strike>This is a db test</strike>.</p>\n<!--kg-card-end: markdown-->",
	"comment_id": "2",
	"plaintext": "This is a db test.",
	"feature_image": null,
	"featured": 0,
	"type": "post",
	"status": "published",
	"locale": null,
	"visibility": "public",
	"email_recipient_filter": "none",
	"author_id": "60710b8d705967038fe66214",
	"created_at": "2004-08-09T19:11:20.000Z",
	"updated_at": "2004-08-09T19:11:20.000Z",
	"published_at": "2004-08-09T19:11:20.000Z",
	"custom_excerpt": null,
	"codeinjection_head": null,
	"codeinjection_foot": null,
	"custom_template": null,
	"canonical_url": null
},

Yes, I have some posts dating back to 2004! Back then I wrote my own custom blog software in PHP. It was a neat time, too bad none of the code survived. It almost makes me want to try and do it again only in Go… but I’ll resist that urge.

Quick and Dirty

Alright let’s write a program which will read our JSON and spit out a single post. First things first we need a struct that maps to our “database”. We get this by simply slapping the JSON into the awesome JSON to Go converter.

type GhostDatabase struct {
	Db []struct {
		Meta struct {
			ExportedOn int64  `json:"exported_on"`
			Version    string `json:"version"`
		} `json:"meta"`
		Data struct {
			Posts []struct {
				ID                   string      `json:"id"`
				UUID                 string      `json:"uuid"`
				Title                string      `json:"title"`
				Slug                 string      `json:"slug"`
				Mobiledoc            string      `json:"mobiledoc"`
				HTML                 string      `json:"html"`
				CommentID            string      `json:"comment_id"`
				Plaintext            string      `json:"plaintext"`
				FeatureImage         interface{} `json:"feature_image"`
				Featured             int         `json:"featured"`
				Type                 string      `json:"type"`
				Status               string      `json:"status"`
				Locale               interface{} `json:"locale"`
				Visibility           string      `json:"visibility"`
				EmailRecipientFilter string      `json:"email_recipient_filter"`
				AuthorID             string      `json:"author_id"`
				CreatedAt            time.Time   `json:"created_at"`
				UpdatedAt            time.Time   `json:"updated_at"`
				PublishedAt          time.Time   `json:"published_at"`
				CustomExcerpt        interface{} `json:"custom_excerpt"`
				CodeinjectionHead    interface{} `json:"codeinjection_head"`
				CodeinjectionFoot    interface{} `json:"codeinjection_foot"`
				CustomTemplate       interface{} `json:"custom_template"`
				CanonicalURL         interface{} `json:"canonical_url"`
			} `json:"posts"`
			PostsAuthors []struct {
				ID        string `json:"id"`
				PostID    string `json:"post_id"`
				AuthorID  string `json:"author_id"`
				SortOrder int    `json:"sort_order"`
			} `json:"posts_authors"`
			PostsMeta []interface{} `json:"posts_meta"`
			PostsTags []struct {
				ID        string `json:"id"`
				PostID    string `json:"post_id"`
				TagID     string `json:"tag_id"`
				SortOrder int    `json:"sort_order"`
			} `json:"posts_tags"`
			Roles []struct {
				ID          string    `json:"id"`
				Name        string    `json:"name"`
				Description string    `json:"description"`
				CreatedAt   time.Time `json:"created_at"`
				UpdatedAt   time.Time `json:"updated_at"`
			} `json:"roles"`
			RolesUsers []struct {
				ID     string `json:"id"`
				RoleID string `json:"role_id"`
				UserID string `json:"user_id"`
			} `json:"roles_users"`
			Settings []struct {
				ID        string      `json:"id"`
				Group     string      `json:"group"`
				Key       string      `json:"key"`
				Value     string      `json:"value"`
				Type      string      `json:"type"`
				Flags     interface{} `json:"flags"`
				CreatedAt time.Time   `json:"created_at"`
				UpdatedAt time.Time   `json:"updated_at"`
			} `json:"settings"`
			Tags []struct {
				ID                 string      `json:"id"`
				Name               string      `json:"name"`
				Slug               string      `json:"slug"`
				Description        interface{} `json:"description"`
				FeatureImage       interface{} `json:"feature_image"`
				ParentID           interface{} `json:"parent_id"`
				Visibility         string      `json:"visibility"`
				OgImage            interface{} `json:"og_image"`
				OgTitle            interface{} `json:"og_title"`
				OgDescription      interface{} `json:"og_description"`
				TwitterImage       interface{} `json:"twitter_image"`
				TwitterTitle       interface{} `json:"twitter_title"`
				TwitterDescription interface{} `json:"twitter_description"`
				MetaTitle          interface{} `json:"meta_title"`
				MetaDescription    interface{} `json:"meta_description"`
				CodeinjectionHead  interface{} `json:"codeinjection_head"`
				CodeinjectionFoot  interface{} `json:"codeinjection_foot"`
				CanonicalURL       interface{} `json:"canonical_url"`
				AccentColor        interface{} `json:"accent_color"`
				CreatedAt          time.Time   `json:"created_at"`
				UpdatedAt          time.Time   `json:"updated_at"`
			} `json:"tags"`
			Users []struct {
				ID              string      `json:"id"`
				Name            string      `json:"name"`
				Slug            string      `json:"slug"`
				Password        string      `json:"password"`
				Email           string      `json:"email"`
				ProfileImage    string      `json:"profile_image"`
				CoverImage      interface{} `json:"cover_image"`
				Bio             interface{} `json:"bio"`
				Website         interface{} `json:"website"`
				Location        interface{} `json:"location"`
				Facebook        interface{} `json:"facebook"`
				Twitter         interface{} `json:"twitter"`
				Accessibility   string      `json:"accessibility"`
				Status          string      `json:"status"`
				Locale          interface{} `json:"locale"`
				Visibility      string      `json:"visibility"`
				MetaTitle       interface{} `json:"meta_title"`
				MetaDescription interface{} `json:"meta_description"`
				Tour            interface{} `json:"tour"`
				LastSeen        time.Time   `json:"last_seen"`
				CreatedAt       time.Time   `json:"created_at"`
				UpdatedAt       time.Time   `json:"updated_at"`
			} `json:"users"`
		} `json:"data"`
	} `json:"db"`
}

Perfect. Note that not all of the struct is strictly required but I’ll leave it in for now. Who knows I might write something to pull the tags for posts too.

Reading the File

To start with we’re going to open our file. Note that this is indeed a quick and dirty version, maybe we’ll add some proper error handling later.

	file, err := os.Open("shindakun-dot-net.ghost.2022-03-18-22-02-58.json")
	if err != nil {
		fmt.Println(err)
	}

Once opened we need to read the file into memory as a []byte.

	b, err := io.ReadAll(file)
	if err != nil {
		fmt.Println(err)
	}

Now we’ll declare our database variable, db. Using json.Unmarshal() we convert the JSON into the struct we will be using.

	var db GhostDatabase

	err = json.Unmarshal(b, &db)
	if err != nil {
		fmt.Println(err)
	}

Finally, we print out the text of the first post.

	fmt.Printf("%#v", db.Db[0].Data.Posts[0].HTML)

This leaves us with the output

<!--kg-card-begin: markdown--><p><strike>This is a db test</strike>.</p>\n<!--kg-card-end: markdown-->

Next Time

This part of our program is relatively easy. We don’t want the HTML though, what I’d rather do is pull the Markdown out of the mobiledoc value.

"mobiledoc": "{\"version\":\"0.3.1\",\"markups\":[],\"atoms\":[],\"cards\":[[\"markdown\",{\"cardName\":\"card-markdown\",\"markdown\":\"\\n<strike>This is a db test</strike>.\\n\"}]],\"sections\":[[10,0]],\"ghostVersion\":\"3.0\"}",

Which, as luck would have it, is just an embedded JSON object. It looks like we’ll need to do some cleaning up of the object removing escaped quotes. Once cleaned up it should look similar to

{
    "version": "0.3.1",
    "markups": [],
    "atoms": [],
    "cards": [
        [
            "markdown",
            {
                "cardName": "card-markdown",
                "markdown": "\n<strike>This is a db test</strike>.\n"
            }
        ]
    ],
    "sections": [
        [
            10,
            0
        ]
    ],
    "ghostVersion": "3.0"
}

I haven’t looked to closely at the database but I’m hoping when I wrote my posts in Ghost they all were in a single card-markdown. Oh well, we’ll cross that bridge eventually.

Until next time!

Enjoy this post?
How about buying me a coffee?

Code Listing

package main

import (
	"encoding/json"
	"fmt"
	"io"
	"os"
	"time"
)

type GhostDatabase struct {
	Db []struct {
		Meta struct {
			ExportedOn int64  `json:"exported_on"`
			Version    string `json:"version"`
		} `json:"meta"`
		Data struct {
			Posts []struct {
				ID                   string      `json:"id"`
				UUID                 string      `json:"uuid"`
				Title                string      `json:"title"`
				Slug                 string      `json:"slug"`
				Mobiledoc            string      `json:"mobiledoc"`
				HTML                 string      `json:"html"`
				CommentID            string      `json:"comment_id"`
				Plaintext            string      `json:"plaintext"`
				FeatureImage         interface{} `json:"feature_image"`
				Featured             int         `json:"featured"`
				Type                 string      `json:"type"`
				Status               string      `json:"status"`
				Locale               interface{} `json:"locale"`
				Visibility           string      `json:"visibility"`
				EmailRecipientFilter string      `json:"email_recipient_filter"`
				AuthorID             string      `json:"author_id"`
				CreatedAt            time.Time   `json:"created_at"`
				UpdatedAt            time.Time   `json:"updated_at"`
				PublishedAt          time.Time   `json:"published_at"`
				CustomExcerpt        interface{} `json:"custom_excerpt"`
				CodeinjectionHead    interface{} `json:"codeinjection_head"`
				CodeinjectionFoot    interface{} `json:"codeinjection_foot"`
				CustomTemplate       interface{} `json:"custom_template"`
				CanonicalURL         interface{} `json:"canonical_url"`
			} `json:"posts"`
			PostsAuthors []struct {
				ID        string `json:"id"`
				PostID    string `json:"post_id"`
				AuthorID  string `json:"author_id"`
				SortOrder int    `json:"sort_order"`
			} `json:"posts_authors"`
			PostsMeta []interface{} `json:"posts_meta"`
			PostsTags []struct {
				ID        string `json:"id"`
				PostID    string `json:"post_id"`
				TagID     string `json:"tag_id"`
				SortOrder int    `json:"sort_order"`
			} `json:"posts_tags"`
			Roles []struct {
				ID          string    `json:"id"`
				Name        string    `json:"name"`
				Description string    `json:"description"`
				CreatedAt   time.Time `json:"created_at"`
				UpdatedAt   time.Time `json:"updated_at"`
			} `json:"roles"`
			RolesUsers []struct {
				ID     string `json:"id"`
				RoleID string `json:"role_id"`
				UserID string `json:"user_id"`
			} `json:"roles_users"`
			Settings []struct {
				ID        string      `json:"id"`
				Group     string      `json:"group"`
				Key       string      `json:"key"`
				Value     string      `json:"value"`
				Type      string      `json:"type"`
				Flags     interface{} `json:"flags"`
				CreatedAt time.Time   `json:"created_at"`
				UpdatedAt time.Time   `json:"updated_at"`
			} `json:"settings"`
			Tags []struct {
				ID                 string      `json:"id"`
				Name               string      `json:"name"`
				Slug               string      `json:"slug"`
				Description        interface{} `json:"description"`
				FeatureImage       interface{} `json:"feature_image"`
				ParentID           interface{} `json:"parent_id"`
				Visibility         string      `json:"visibility"`
				OgImage            interface{} `json:"og_image"`
				OgTitle            interface{} `json:"og_title"`
				OgDescription      interface{} `json:"og_description"`
				TwitterImage       interface{} `json:"twitter_image"`
				TwitterTitle       interface{} `json:"twitter_title"`
				TwitterDescription interface{} `json:"twitter_description"`
				MetaTitle          interface{} `json:"meta_title"`
				MetaDescription    interface{} `json:"meta_description"`
				CodeinjectionHead  interface{} `json:"codeinjection_head"`
				CodeinjectionFoot  interface{} `json:"codeinjection_foot"`
				CanonicalURL       interface{} `json:"canonical_url"`
				AccentColor        interface{} `json:"accent_color"`
				CreatedAt          time.Time   `json:"created_at"`
				UpdatedAt          time.Time   `json:"updated_at"`
			} `json:"tags"`
			Users []struct {
				ID              string      `json:"id"`
				Name            string      `json:"name"`
				Slug            string      `json:"slug"`
				Password        string      `json:"password"`
				Email           string      `json:"email"`
				ProfileImage    string      `json:"profile_image"`
				CoverImage      interface{} `json:"cover_image"`
				Bio             interface{} `json:"bio"`
				Website         interface{} `json:"website"`
				Location        interface{} `json:"location"`
				Facebook        interface{} `json:"facebook"`
				Twitter         interface{} `json:"twitter"`
				Accessibility   string      `json:"accessibility"`
				Status          string      `json:"status"`
				Locale          interface{} `json:"locale"`
				Visibility      string      `json:"visibility"`
				MetaTitle       interface{} `json:"meta_title"`
				MetaDescription interface{} `json:"meta_description"`
				Tour            interface{} `json:"tour"`
				LastSeen        time.Time   `json:"last_seen"`
				CreatedAt       time.Time   `json:"created_at"`
				UpdatedAt       time.Time   `json:"updated_at"`
			} `json:"users"`
		} `json:"data"`
	} `json:"db"`
}

func main() {
	fmt.Println("ghost2hugo")

	file, err := os.Open("shindakun-dot-net.ghost.2022-03-18-22-02-58.json")
	if err != nil {
		fmt.Println(err)
	}
	
	b, err := io.ReadAll(file)
	if err != nil {
		fmt.Println(err)
	}

	var db GhostDatabase

	err = json.Unmarshal(b, &db)
	if err != nil {
		fmt.Println(err)
	}

	fmt.Printf("%#v", db.Db[0].Data.Posts[0].HTML)
}

WIP Hugo Theme

Thu, 15 Sep 2022 20:06:00 -0700

Intro

I’ve been putting in some time “Indiewebifying” my Hugo theme, which started as a fork of the Solar theme by Bake. It’s slowly becoming more and more of my own. I’m not much of a front end developer so I’ve keep most of the basic CSS, removing only small bits and adding things where needed to fit my needs. This little post is just to remind me of what I’ve been doing, if you would like to know more just ask.

An IndieWeb Post

The first portion of getting the theme updated for the IndieWeb involved adding to the HTML markup to include h-entry. This adds a nice structure to the posts that is machine readable. This is really just adding specific classes to your HTML elements. So your post template might look something like the following:

<article class="post h-entry">
  <h1 class="post-title p-name" itemprop="name headline">
    <a class="u-url" href="{{ .Permalink }}">{{ .Title }}</a> {{ if .Draft
    }}(Draft){{ end }}
  </h1>

  {{ if .Params.cover_image }}
  <div class="image">
    <img src="{{ .Params.cover_image }}" />
  </div>
  {{ end }}
  <div class="post-content e-content" itemprop="articleBody">
    {{ .Content | safeHTML }}
  </div>

  <p class="meta">
    Posted on
    <span class="postdate"
      ><time class="dt-published"
        >{{ .Site.Params.DateForm | default "Jan 2, 2006" | .Date.Format
        }}</time
      ></span
    >
    {{ if .Params.author }} {{ with index .Site.Data.authors .Params.author }}
    by
    <span class="p-author h-card"
      ><a href="{{ .link }}">{{ .name }}</a>
      <img class="u-photo avatar" style="display: none;" src="{{ .photo }}" />
    </span>
    {{ end }} {{ end }}
  </p>
</article>

Alright maybe it’s not adding too much to the page. If I were to add full Schema.org JSON-LD as well we’d worry more about the page size.

Syndication

One other aspect of the IndieWeb is the idea that you are posting your content in multiple locations. This syndication is include in the post with a relatively straight forward partial.

{{ if isset $.Params "syndication" }}
    {{ $synLen := len $.Params.syndication }}
    {{ if gt $synLen 0 }}
        <p>Syndicated to:</p>
        <div class="syndication">
        {{ range $k, $v := $.Params.syndication }}<a class="post__syn btn u-syndication syn-link" rel="syndication" href="{{ . }}">{{ . }}</a></br>
    {{ end }}
        </div>
    {{ end }}
{{ end }}

This will loop through any syndicated links found in the posts frontmatter and write the link to the page under the current post.

Webmentions

I’ve already written a bit about adding webmentions to sites and this advice works for Hugo. I didn’t happen to note who wrote the JavaScript that takes care of fetching mentions sadly. I did track down a similar script in a gist from Nooshu so maybe that’s that.

Post Types

One of the departures from Solar is that I’m working on having different post types, either microblog style notes or full on blog posts. I need to break out the code a little bit though as right now the code is just rendered slightly differently depending on post type. I want to take each and move them into individual partials. I’ve just updated the code to implement a proper h-feed. An example of how the feed might look can be found on Monocle

{{ define "main" }}
<main class="h-feed">
  {{ range .Paginator.Pages }} {{ if eq .Type "posts" }}
  <article class="post h-entry">
    {{ if .Params.cover_image }}
    <a href="{{ .Permalink }}"><img src="{{ .Params.cover_image }}" /></a>
    {{ end }}
    <h1 class="p-name">
      <a class="u-url" href="{{ .Permalink }}">{{ .Title }}</a> {{ if .Draft
      }}(Draft){{ end }}
    </h1>

    <div class="post-content">
      <p class="e-content">
        {{ .Summary | plainify | safeHTML }} {{ if .Truncated }} … {{ end }}
      </p>
    </div>

    <p class="meta">
      Posted on
      <span class="postdate"
        >{{ .Site.Params.DateForm | default "Jan 2, 2006 " | .Date.Format
        }}</span
      >
    </p>
  </article>
  {{ end }} {{ if eq .Type "notes" }}
  <article class="note h-entry">
    {{ if .Params.cover_image }}
    <a href="{{ .Permalink }}"><img src="{{ .Params.cover_image }}" /></a>
    {{ end }}

    <div class="note-content">
      <p class="p-content">
        {{ .Summary | plainify | safeHTML }} {{ if .Truncated }} … {{ end }}
      </p>
    </div>

    <p class="meta">
      Posted on
      <a href="{{ .Permalink }}"
        ><span class="postdate dt-published"
          >{{ .Site.Params.DateForm | default "Jan 2, 2006 " | .Date.Format
          }}</span
        ></a
      >
    </p>
  </article>
  {{ end }} {{ end }} {{ partial "pagination.html" . }}
</main>
{{ end }}

Meanwhile, the core single.html partial is currently used for both post types, I will want to change that too so I can style the posts slightly differently.

CSS

As I mentioned I have been staying away from the CSS with the exception of removing styling and setting some defaults that fit what I’d like more. I need to go through the SCSS and make sure that each of the class names follow a similar pattern as I’ve borrowed some CSS from various sources with wildly different name styles. For instance I have post-content and post__tags. I’ll probably stick with kebab-case.

The Future

I have a bunch more work to do on the theme to tidy it up then I might release it publicly. I suppose the world could use another IndieWeb compatible theme.

Enjoy this post?
How about buying me a coffee?

Adding gcc to git bash

Sat, 03 Sep 2022 12:43:41 -0700

So today I was working on my “main” computer which didn’t have much set up on it. Basically only git bash. Problem is I wanted to work on my Hugo site and theme.

To do that I needed to install the version of Hugo with extended support for SCSS. This requires gcc. The quickest solution I found was to just install msys2. Once installed and added to the path it was just a matter of changing to the Hugo directory and running go install --tags extended.

$ cd /d/Code/hugo/
$ go install --tags extended
$ hugo version
hugo v0.103.0-DEV-06c3ac67411e2faaf1545427d4df4e326fffd3c0+extended windows/amd64 BuildDate=2022-09-03T09:38:03Z

Enjoy this post?
How about buying me a coffee?

Getting Reacquainted With Hugo

Mon, 23 May 2022 18:55:20 -0700

Testing testing testing

It’s been a bit since I’ve looked at my Hugo blog here on shindakun.dev.

Hugo

First things first, I need to get all my repositories in line. When I originally set everything up I had it separated in a couple different repos. The main repo held everything needed to build the base Hugo. I had two submodules for the themes that power the site. Finally, I had a repo just for blog posts. Why I didn’t include the blog posts repo as a submodule originally is lost to time though. We’re going to add them in as a submodule much like our themes.

git submodule add https://github.com/shindakun/posts

Now we can just run

git clone --recursive  https://github.com/shindakun/shindakun.dev.git

And we’ll get the entire site. Since the site running behind Nginx we can then just build it and server from the public directory.

Nginx

The currently deployed version of the site is just a copy of the public directory and the configuration looks like

server {
    server_name shindakun.dev;
    root /var/www/shindakun.dev;
    index index.html;

    location / {
        try_files $uri $uri/ =404;
    }

    listen [::]:443 http2 ssl; # managed by Certbot
    listen 443 ssl http2; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/shindakun.dev/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/shindakun.dev/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

server {
    if ($host = shindakun.dev) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen 80;
    listen [::]:80;

    server_name shindakun.dev;
    return 404; # managed by Certbot
}

We’ll update root /var/www/shindakun.dev; to be root /var/www/shindakun.dev/public; once everything is sorted on the server itself.

But why?

Why do any of this? Well, keeping a blog is something I’ve always enjoyed - though sometimes I’m not very good at it. Mostly because I link to tinker. While I don’t fully run my server I manage most aspects of the Digital Ocean droplet I use. I used to use Ghost, but it has slowly gotten to the state where I could only run a single blog on a droplet. Whereas if I use Hugo I can run countless websites with nothing but Nginx.

Enjoy this post?
How about buying me a coffee?

Polywork Poster in Go

Sun, 09 May 2021 17:54:00 -0700

Intro

A couple of days ago I finally got access to Polywork. Which, for those who don’t know, is like a new-aged Linkedin. It allows you to create a timeline of your work and accomplishments. The idea is that people will be able to contact you through the site to offer you work or opportunities.Take a look at meta.shindakun.dev for a simple example timeline.

One of the things that struck me was that there is no public API at the moment. Which bummed me out as I didn’t want to start entering blog posts by hand. So, I decided to poke around a bit and see what the flow is for creating a new post. With the objected being crafting some Go code to create posts.

This would dovetail nicely into our data connector series.

Getting Started

First, we need to look to see how the site lets us put up a new post. This can be done via the network tab in Chromes Dev Tools.

![[newpost.gif]]

We can see that clicking on the “Post” button brings up a form that we can use to enter our details. Looking at the details of the edit network call shows the entire Turbo Frame that makes up. new in this case seems to just redirect to edit.

![[Pasted image 20210509185610.png]]

Perfect! This will allow us to grab the details we need from the form. We need both the update ID and the authenticity token. Though at present it seems the authenticity token isn’t actually required when posting a new update. 🤷‍♂️ I won’t show the complete posting flow via the website since it includes cookies but you get the idea.

![[Pasted image 20210509185946.png]]

Our Plan

So here is what we are going to do:

make a request to https://www.polywork.com/shindakun/highlights/new/
- extract the update ID and authenticity token
create a multipart form with information we want to post
POST to https://www.polywork.com/shindakun/highlights/{{update_id}}

Enjoy this post?
How about buying me a coffee?

Getting Polywork to Go

As always you can find the full code listing at the bottom of this post. Also, this code is pretty sloppy and a bit brittle since it is just a proof of concept.

package main

import (
	"bytes"
	"fmt"
	"io"
	"log"
	"mime/multipart"
	"net/http"
	"regexp"
	"strings"

	"github.com/andybalholm/cascadia"
	"golang.org/x/net/html"
)

const cookie = "_polywork_session=...snip..."
const username = "shindakun"

We start up by setting our cookie value. This is necessary to be able to POST to the website. I simply grabbed this value from the network tab in Chrome. We could start with a “magic” link login but this seemed easier. We’ll also set our username.

func main() {
	req, err := http.NewRequest(http.MethodGet, "https://www.polywork.com/"+username+"/highlights/new/", nil)
	req.Header.Add("Cookie", cookie)
	req.Header.Add("Turbo-Frame", "navbar-post-highlight")
	if err != nil {
		log.Fatal(err)
	}

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		log.Fatal(err)
	}
	defer res.Body.Close()

If you’ve worked with Go before this probably looks very familiar. What we are doing is using http.NewRequest to request to /{{username}}/highlights/new. Note how we are setting the cookie directly in the header and setting a header for the Turbo-Frame.

Calling http.DefaultClient.Do(req) makes the request to the remote server and if successful will return that data in res.

	doc, err := html.Parse(res.Body)
	if err != nil {
		log.Fatal(err)
	}

	// inputs := cascadia.MustCompile("input").MatchAll(doc)

	buttons := cascadia.MustCompile("button").MatchAll(doc)

	re := regexp.MustCompile(`\/shindakun\/highlights\/(.*)\/save_draft`)
	r := re.FindStringSubmatch(buttons[1].Attr[5].Val)

Here we’re going to get a bit tricky. The first thing we want to do is use the parse our incoming form as HTML. This then allows us to use the cascadia library and CSS selectors to grab all the buttons. We don’t really need all of them but for this first version what the heck.

Using the regular expression \/shindakun\/highlights\/(.*)\/save_draft we extract the ID for the post we are working with.

This is the main section of the code that is very brittle since I’m not doing anything to validate we are looking at the right button key and instead just hard coding the values that I saw during testing.

	// _method: patch
	// authenticity_token: ...snip...
	// draft_highlight[end_date]: 01-01-2021
	// draft_highlight[content]: <div>Testing</div>
	// draft_highlight[activity_ids][]: 76199
	// commit: Post

During testing, I made a copy of the form data POSTed from Chrome for reference. We’ll use this as our base form data. As you’ll see below not every field is required. I think I can leave out the _method field but have not tried as of yet. I mentioned before but you should also note that I’m not using the authenticity_token token as it seems it is not required.

	body := &bytes.Buffer{}
	writer := multipart.NewWriter(body)

	fw, _ := writer.CreateFormField("_method")
	_, err = io.Copy(fw, strings.NewReader("patch"))
	if err != nil {
		log.Fatal(err)
	}

	// fw, _ = writer.CreateFormField("authenticity_token")
	// _, err = io.Copy(fw, strings.NewReader(inputs[1].Attr[2].Val))
	// if err != nil {
	// 	log.Fatal(err)
	// }
	
	fw, _ = writer.CreateFormField("draft_highlight[content]")
	_, err = io.Copy(fw, strings.NewReader("<div>Posting via Golang</div> :) <a href=\"https://shindakun.dev\">shindakun</a>"))
	if err != nil {
		log.Fatal(err)
	}
	
	fw, _ = writer.CreateFormField("commit")
	_, err = io.Copy(fw, strings.NewReader("Post"))
	if err != nil {
		log.Fatal(err)
	}
	
	writer.Close()

It’s a bit awkward to create a form in Go but hey, it works. We make a form field with writer.CreateFormField and then io.Copy our data into that field. Right now our payload is hardcoded. We could easily adapt my data connector post to automatically add my Ghost blog posts to Polywork.

	post, err := http.NewRequest("POST", "https://www.polywork.com/shindakun/highlights/"+r[1], bytes.NewReader(body.Bytes()))
	if err != nil {
		log.Fatal(err)
	}
	post.Header.Set("Content-Type", writer.FormDataContentType())
	post.Header.Add("Cookie", cookie)
	rsp, _ := http.DefaultClient.Do(post)
	if rsp.StatusCode != http.StatusOK {
		log.Printf("Request failed with response code: %d", rsp.StatusCode)
	}
}

And now we do the actual form submit. And that’s all there is too it.

What’s Next?

Next, I think I am going to adapt this code to use with the data connector we are working on. I’ll probably try to make it pretty generic so I can use it as its own module.

Code Listing

package main

import (
	"bytes"
	"fmt"
	"io"
	"log"
	"mime/multipart"
	"net/http"
	"regexp"
	"strings"

	"github.com/andybalholm/cascadia"
	"golang.org/x/net/html"
)

const cookie = "_polywork_session=...snip..."
const username = "shindakun"

func main() {
	req, err := http.NewRequest(http.MethodGet, "https://www.polywork.com/"+username+"/highlights/new/", nil)
	req.Header.Add("Cookie", cookie)
	req.Header.Add("Turbo-Frame", "navbar-post-highlight")
	if err != nil {
		log.Fatal(err)
	}

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		log.Fatal(err)
	}
	defer res.Body.Close()

	doc, err := html.Parse(res.Body)
	if err != nil {
		log.Fatal(err)
	}

	// inputs := cascadia.MustCompile("input").MatchAll(doc)

	buttons := cascadia.MustCompile("button").MatchAll(doc)

	re := regexp.MustCompile(`\/shindakun\/highlights\/(.*)\/save_draft`)
	r := re.FindStringSubmatch(buttons[1].Attr[5].Val)

	// _method: patch
	// authenticity_token: ...snip...
	// draft_highlight[end_date]: 01-01-2021
	// draft_highlight[content]: <div>Testing</div>
	// draft_highlight[activity_ids][]: 76199
	// commit: Post

	body := &bytes.Buffer{}
	writer := multipart.NewWriter(body)
	fw, _ := writer.CreateFormField("_method")
	_, err = io.Copy(fw, strings.NewReader("patch"))
	if err != nil {
		log.Fatal(err)
	}
	// fw, _ = writer.CreateFormField("authenticity_token")
	// _, err = io.Copy(fw, strings.NewReader(inputs[1].Attr[2].Val))
	// if err != nil {
	// 	log.Fatal(err)
	// }
	fw, _ = writer.CreateFormField("draft_highlight[content]")
	_, err = io.Copy(fw, strings.NewReader("<div>Posting via Golang</div> :) <a href=\"https://shindakun.dev\">shindakun</a>"))
	if err != nil {
		log.Fatal(err)
	}
	fw, _ = writer.CreateFormField("commit")
	_, err = io.Copy(fw, strings.NewReader("Post"))
	if err != nil {
		log.Fatal(err)
	}
	writer.Close()

	post, err := http.NewRequest("POST", "https://www.polywork.com/shindakun/highlights/"+r[1], bytes.NewReader(body.Bytes()))
	if err != nil {
		log.Fatal(err)
	}
	post.Header.Set("Content-Type", writer.FormDataContentType())
	post.Header.Add("Cookie", cookie)
	rsp, _ := http.DefaultClient.Do(post)
	if rsp.StatusCode != http.StatusOK {
		log.Printf("Request failed with response code: %d", rsp.StatusCode)
	}
}

2020 Roguelikedev Tutorial Week 2

Mon, 29 Jun 2020 22:40:26 -0700

RoguelikeDev Does The Complete Roguelike Tutorial

Week 2 - June 23rd - 29th 2020

Introduction

That’s right - we are continuing our work on the Roguelike Tutorial! This week we’ll try to get through parts 2 and 3 (and almost make it). I did have one small update after the last post that I briefly mentioned on Twitter so, I figured I should mention it here.

And with that out of the way, let’s continue on!

Part 2 - The Generic Entity, the Render Functions, and the Map

Part 2 is jammed packed! I’m not going to strictly follow everything but for the most part will keep the basic structure the same.

The Generic Entity

I’ve decided to go with a “constructor” function that will return a new “entity” table. Note that I’m calling it obj in the code but it’s just a table of key/value pairs. Maybe ent is a better as a name. 🤔

function new_entity(typ, name, sp, x, y, spd)
  local obj = {
    typ=typ,
    name=name,
    sp=sp,
    x=x,
    y=y,
    spd=spd,
    move=function(self, dx, dy)
      self.x += dx
      self.y += dy
    end
    attack=function(self, tx, ty)
      --
    end
  }
  return obj
end

Let’s take a closer look at the basic setup. I expect this to grow over time as I add things like damage. We may be limited on what we can squeeze in though.

  local obj = {
    typ = typ,
    name = name,
    sp = sp,
    x = x,
    y = y,
    spd = spd,

typ is the “type” of entity. Right now I only am planing on two entity types, “pc” (player character) and “monst” (monster). This could be extended to include things like “npc”, “pet”, “animal” - maybe
name the entities “name”, it may be possible to name your player character. For monsters it would typically say what kind of monster, for instance ant
sp holds the number of the sprite on the sprite sheet that should be used for this particular entity
x, y the current coordinates of the entity
spd how fast (far) the entity can move in one turn

    move = function(self, dx, dy)
      self.x += dx
      self.y += dy
    end,
    attack = function(self, tx, ty)
      --
    end
  }

The generic entity will have some standard methods which may or may not call out to other functions within the cart. I’m not sure exactly what methods they will end up with, maybe just a few basics.

As a quick and dirty example, this would allow us to do something like the following.

  ents={
    new_entity("monst", "ant", ant, xpos-16, ypos-16, 8),
    new_entity("monst", "ant", ant, xpos-32, ypos+16, 8)
  }

Now we have two “ants” which we can then access using all().

  for e in all(ents) do
    spr(e.sp, e.x, e.y)
  end

Now, we have two a ant’s on the screen. This will probably evolve over the weeks but that should be good to get started. That brings us to the next step of Part 2.

The Render Functions

The tutorial render functions are each pretty straight forward. I think we can use a very similar setup in Lua.

def render_all(con, entities, screen_width, screen_height):
    # Draw all entities in the list
    for entity in entities:
        draw_entity(con, entity)

    libtcod.console_blit(con, 0, 0, screen_width, screen_height, 0, 0, 0)


def clear_all(con, entities):
    for entity in entities:
        clear_entity(con, entity)


def draw_entity(con, entity):
    libtcod.console_set_default_foreground(con, entity.color)
    libtcod.console_put_char(con, entity.x, entity.y, entity.char, libtcod.BKGND_NONE)


def clear_entity(con, entity):
    # erase the character that represents this object
    libtcod.console_put_char(con, entity.x, entity.y, ' ', libtcod.BKGND_NONE)

We aren’t too concerned about clearing at the moment so let’s just begin with drawing. Just like the Python example, we’ll make render_all() which will call draw_entity().

function render_all(ents)
  for ent in all(ents) do
    draw_entity(ent)
  end
end

function draw_entity(ent)
  rectfill(ent.x, ent.y, ent.x+7, ent.y+7, 0)
  spr(ent.sp, ent.x, ent.y)
end

So our drawing snippet from above becomes:

  render_all(ents)

We pass in all of our entities, each one is forwarded on for drawing, and then it appears on the screen.

So far so good! You may have noticed that we’re using rectfill() in draw_entity(). This is because when the map is drawn, the entire thing is drawn. We need to ensure we remove those elements when they are below an entity.

The Map

The tutorial version now goes off into creating a class for the game map. My Pico-8 version currently just uses _draw() to fill in our floor “tile” and our wall “tiles”. It’s pretty brute force for now. We’ll improve it while we work on our random generation a bit later on - likely in the next section or two.

function _draw()
  cls(0)
  for iy=0,127,8 do
    for ix=0,127,8 do
    if ix == xpos and iy == ypos then
      --
    else
      spr(flor, ix, iy)
    end
  end
  end

  -- fill in left and right "walls"
  for iy=0,127,8 do
    rectfill(0, iy, 7, iy+8, 0)
    spr(wall, 0, iy)
    rectfill(120, iy, 128, iy+8, 0)
    spr(wall, 120, iy)
  end

  -- fill in top and bottom "walls"
  for ix=0,127,8 do
    rectfill(ix, 0, ix+8, 7, 0)
    spr(wall, ix, 0)
    rectfill(ix, 120, ix+8, 128, 0)
    spr(wall, ix, 120)
  end
  spr(plyr, xpos, ypos)
  print('('..xpos..', '..ypos..')', 8, 8, 5)
  render_all(ents)

This renders a screen similar to the following table:

0	1	2	3	4	5	6	7	8
1	#	#	#	#	#	#	#	#
2	#							#
3	#							#
4	#							#
5	#							#
6	#							#
7	#							#
8	#	#	#	#	#	#	#	#

Part 3 - Generating a dungeon

Of course I’m a bit behind schedule so I haven’t quite finished my implementation of the dungeon carving routine. I’ve also have yet to put in collision. create_room() is a pretty straight forward function. We simply pass in the screen X,Y we want to start at and then the width and height of the room. The Python implementation takes a “room” which we’re skipping over for now.

   def create_room(self, room):
       # go through the tiles in the rectangle and make them passable
       for x in range(room.x1 + 1, room.x2):
          for y in range(room.y1 + 1, room.y2):
              self.tiles[x][y].blocked = false
              self.tiles[x][y].block_sight = false

For now we are simply going to return our table of X,Y coordinates and the sprite.

function create_room(x,y,w,h)
  local obj={}
  for iy=y,y+h+1,1 do
    for ix=x,x+w+1,1 do
      add(obj,{
       x=ix,
       y=iy,
       sp=flor
     })
    end
  end
  return obj
end

Let’s see if we can get the tunnel functions in real quick. Again, we just have a basic implementation that returns a table of the coordinates we need to plot our “tunnels”.

function create_h_tunnel(x1,x2,y)
  local obj={}
  for x=min(x1,x2),max(x1,x2)+1,1 do
    add(obj,{
       x=x,
       y=y,
       sp=flor
     })
  end
  return obj
end

function create_v_tunnel(y1,y2,x)
  local obj={}
  for y=min(y1,y2),max(y1,y2)+1,1 do
    add(obj,{
       x=x,
       y=y,
       sp=flor
     })
  end
  return obj
end

Looks nice, right?

Yeah, it’s missing the walls!

Wrapping Up, For Now

I’ll have to try and wrap the last bit of the weeks code and update the GitHub project over the next day or so. I’ll also need to change around how we are writing the “map” so we can take into account impassable terrain and get ready for field of view. Then we’ll try and place some more monsters!

Enjoy this post?
How about buying me a coffee?

2020 Roguelikedev Tutorial Week 1

Sun, 21 Jun 2020 23:20:29 -0700

RoguelikeDev Does The Complete Roguelike Tutorial

Week 1 - June 16th - 22nd 2020

Introduction

Welcome back! It’s been a bit since my last post. The quarantine was not the boon I thought it would be for side projects. I haven’t really been motivated to do much of anything.

I thought I might take part in the Portland Indie Game Squads Summer Slow Jam but, that didn’t really work out so far. Instead I’m going to try to take part in the ==RoguelikeDev Does The Complete Roguelike Tutorial==… and I’m already behind schedule.

Beyond that though I’ve decided to only loosely follow the tutorial. The first departure will be that I’m going to try to do it in PICO-8. Which hopefully won’t go that poorly. Since I already have PICO-8 installed I don’t really need to do much in the way of setup.

!arial10x10.png

Using PICO-8 means I won’t be using libtcod but instead will be hacking away using lua. I’ve only recently started messing with it so hopefully it goes OK. Instead of the Arial font you see above I’m going to try to make use of the PICO-8 font.

This may not work out though since the font is an odd size. I could always recreate the glyphs I need in the sprite editor and have a nice 8x8 sprite. That may work nicely, I’ll have to look into that - for now though lets get on with it.

Part 0 - Setting Up

http://rogueliketutorials.com/tutorials/tcod/part-0/

The tutorial starts out with getting python running and then writing ==HELLO WORLD== to verify its working. If you happen to have PICO-8 and want to follow along, boot it up and type in:

print("hello world")

Hitting enter should cause the text “hello world” to be echoed on the next line before returning to the prompt. There we go!

!PICO-8_2.gif

Part 1 - Drawing the ‘@’ symbol and moving it around

http://rogueliketutorials.com/tutorials/tcod/part-1/

We are going to take the shortest route to completing our second goal for the week. Which might be a bit silly since we’ll likely switch over to sprite basing graphics, maybe. I haven’t quite decided. Alright lets see what we’ve got!

Upon booting a “cartridge” the system will run _init(). This can be used fro initializing starting variables or “constraints”. Here we are going to set our base position to the center of the screen. We’ll use this when rendering our @.

function _init()
  xpos = 64
  ypos = 64
end

The PICO-8 “console” has what amounts to a built in game loop. Which we can make use of by using _draw() and _update(). This allows us to draw to the screen and perform “updates” every 30 (or 60) frames. As seen below we use _update() to check for button presses and current position. Depending on what button was pressed and where we currently are on the screen we can then move our @ about.

function _draw()
  cls(3)
  print("@", xpos, ypos)
end

function _update()
  if (btnp(0) and xpos > 0) xpos -= 4
  if (btnp(1) and xpos < 127-5) xpos += 4
  if (btnp(2) and ypos > 0+4) ypos -= 6
  if (btnp(3) and ypos < 127-9) ypos += 6
end

If you happen to pick up PICO-8 the above code can be copied into the editor. After that simply type run from the command line and enjoy moving about.

Wrapping Up

I’ll be working over the next couple days to figure out a good release mechanism so I push builds to itch.io and GitHub. I suppose I could also release on the PICO-8 BBS once I have a “build”.

I am planning on also doing a read-through on the next two parts of the tutorial so I can decide if I need to switch to using sprites or if I should stick with “text”. If I can stay on track we should also see some more blog posts!

Additional Resources

The PICO-8 Manual /r/roguelikedev

Enjoy this post?
How about buying me a coffee?

SDL2 Test 03 - Part 01

Tue, 31 Mar 2020 23:33:03 -0700

Background

So here we are at the end of month and what a crazy month it’s been. You may not be too surprised to hear that I’ve been trying to start this post for the last two weeks. It’s been quite crazy around here though so I never actually got around to doing it.

However, now that we are down to the wire I wanted to make sure at least I got a post up that covers a bit of what I wanted to have up today. We’ll have to wait a couple days until the demo programs up on itch.io.

This post is likely to be even shorter than our February post!

Cellular Automata

This time around I’ll be posting up some information on my knock-off rendition of using cellular automata to dig out caves.

I know this is suppose to be an SDL2 post but we aren’t going to make that leap until the second part. A very small scale version might look like the text below.

Hopefully this renders OK. 🤞

111111111111111111111111111111111111111111111111111111111111
111111111111111111111111111111111111111111000111111111100111
111111111111111111001110011111111111111000000011111110000011
111111111111111110000000001111111111110000000011111100000001
111111111111111100000000000111111111110000000011111100000001
111111111111110000000000000011111111110000000011111100000001
111111111111100000000000000001100001100000000001111100000001
111111111111100000000000000000000000000000000001111110000011
111111111111100000000000000000000000000000000001111111000011
111111111111000000000000000000000000000000000011111111100011
111111111110000000000000000000000000000000000011111111111111
111111111100000000000000000000000000000000000001110011111111
111111111000000000000000000000000000000000000000000001111111
111000000000000000000000000000000000000000000000000001111111
110000000000000000000000000000000000000000000000000000111111
110000000000000000000000000000000000000000000000000000000111
110000000000000000000000000000000000000000000000000000000011
110000000000000000000000000000000000000000000000000000000011
110000000000000000000000000000000000000000000000000000000011
110000000000000000000011000000000000000000000000000000000011
110000000000000000000111100000000000000000000000000000000011
111000000000000000000111100000000000000000000000000000000011
111000000000000000000111100000000000000000001110000000000011
111000000000000000000111000000000000000000011111000000000011
111000000000000000001111000000000000000000011111100000000011
111000000000000000001111100000000000000000011111110000000011
111000000000000000000111110000000000000000011111110000000011
111100000000000000000111111000000000000000111111110000000111
111110000001100110001111111111000111111111111111111001111111
111111111111111111111111111111111111111111111111111111111111

Cellular automata is broken up into several phases. First thing we are going to do is take a grid - which in our case above is I believe 30x60.

  for x := range l.Grid {
    for y := range l.Grid[x] {
      if rand.Intn(max-min)+min < RandFill || x == 0 || x == len(l.Grid)-1 || y == 0 || y == len(l.Grid[x])-1 {
        l.Grid[x][y] = 1
      } else {
        l.Grid[x][y] = 0
      }
    }
  }

Alright, so, here’s our first stop we’re going to take our slice Grid and randomly fill in 0 or 1. We loop all the way through (and may need to flip the x and y now that I’m looking at it) filling in each tile randomly. During this we are also checking to make sure that if we are on an edge that we fill in a 1 to wrap our “level” in walls so we don’t need to worry about wrapping around for now.

Maybe at some point we’ll extend this out so it is larger and have a wrapping world. But that’s beyond the scope of what I’d like to have accomplished by the end of the year.

Give It A Try

Instead of giving it a try now, we’re going to need to hang on a couple of days. This months project isn’t very exciting as it is more just setting the stage for our next steps. But I’ll be putting up the example executables within the next week or so. I am aiming to have a much more in depth post up around the 10th to cover the past months work as well as bridge what is coming next….

Anyway until next time!

Enjoy this post?
How about buying me a coffee?

SDL2 Test 02

Wed, 26 Feb 2020 18:39:03 -0800

Background

Welcome back! Or welcome for the first time! If you do happen to be stopping by for the first time you may want to read the first two posts in this little series.

Of course if you want to stay here you are most welcome. So, what are we doing?

Creating prototypes that can form together into something resembling a game.

Lets Get Shaking

Looking back on our last post I noted at the end that it’d be nice to add a screen shake effect. Something say like this one:

As it turns out it is quite easy to do.

Camera

We are going to use the camera position and alter it to achieve a shaking effect. The camera itself is an sdl.Rect() that is the size of our output resolution. It doesn’t need to be an sdl.Rect(), but this removes the need to declare a struct.

We calculate the coordinate points of the rectangle relative to the player. Centering it around the “player character”. Unless it is near the edge of the “map”, then the edge is set to the background edge. A chunk of the background is then copied to the renderer using those coordinates. The player “sprite” is then copied over next. Since we are using the hardware render so we call renderer.Present() to “flip” the back buffer to the screen.

Let’s see what our current camera looks like.

  camera := sdl.Rect{X: 0, Y: 0, W: wWidth, H: wHeight}

  ...snip...

  camera.X = int32((d.posX + d.width/2) - wWidth/2)
  camera.Y = int32((d.posY + d.height/2) - wHeight/2)

  if camera.X < 0 {
    camera.X = 0
  }
  if camera.Y < 0 {
    camera.Y = 0
  }
  if camera.X > bgWidth-camera.W {
    camera.X = bgWidth - camera.W
  }
  if camera.Y > bgHeight-camera.H {
    camera.Y = bgHeight - camera.H
  }

  renderer.Copy(bg, &sdl.Rect{
    X: camera.X,
    Y: camera.Y,
    W: wWidth,
    H: wHeight,
  },
    &sdl.Rect{
      X: 0,
      Y: 0,
      W: wWidth,
      H: wHeight,
    })

Not very exciting is it.

Implementation

To keep this post short I’m not going to repeat all the code. I need to figure out how I’m going to put it up on GitHub. Do I want to use a single repo for all the code or split it up into sections? I should get that taken care of.

We start outside our main game loop and do a bit of setup, with some initial values. Then after our camera bounding code, we check if shaking is true. If it is we use the very inelegant shake * rand.Float64() * maxShake to get our camera offset. This might lead to the following values:

camerax 184, cameray 228
targetx 3.421793651571817, targety 1.0282214265561018

Adding 184+3 (we cast to int32 which drops the decimals) gives us 187 for the X. Likewise, we add 228+1 and get 229. This gives a jitter to the image since the value we’re adding will change every frame. We don’t have any negative offsets when done this way though. This isn’t the perfect screen shake but at this stage it will do. We can look into smoothing it a bit down the road.

  shaking := true
  targetx := 0.0
  targety := 0.0
  shake := 0.8
  maxShake := 5

  ...snip...

  if camera.Y > bgHeight-camera.H {
    camera.Y = bgHeight - camera.H
  }

  if shaking {
    targetx = shake * rand.Float64() * maxShake
    targety = shake * rand.Float64() * maxShake
  }

  renderer.Copy(bg, &sdl.Rect{
    X: int32(targetx) + camera.X,
    Y: int32(targety) + camera.Y,
    W: wWidth,
    H: wHeight,
  },

  ...snip...

The render.Copy() call now uses the original camera value and the shake offset. We don’t want to actually mutate the camera values since we need them once the shaking stops. It’s not visible here but the current code allows us to enable and disable shaking by pressing the space bar. In that bit of code I’m making sure to reset targetx and targety to 0.0.

Wrapping Up

A short month calls for a short post. We didn’t really set out to do to much this month but we accomplished a few things anyway. Besides this post, I finished my makefile! For now at least. I can build for Windows, macOS, and Linux and release to itch.io pretty with no trouble. In theory I can also compile to Android (and maybe iOS).

I may switch gears a little and try to accomplish generating a “map” which will eventually be used as our base for a level. This is going to be cellular automata based. I also may try to get an implementation diamond-square algorithm completed. At the very least I’m hoping to present something on screen for the cellular automata. Where we go after that I’m not quite sure yet.

Anyway until next time!

Give It A Try

I’ve added the prototype over on itch.io. It can be download for Windows, macOS, and Linux. Please give them a try and let me know if you have any issues.

Enjoy this post?
How about buying me a coffee?

SDL2 Test 01

Fri, 31 Jan 2020 22:27:02 -0800

Background

Welcome back! This is the first release of a few bits of SDL2/Go prototype code. The plan is to combine those bits together into a whole at some point. This method may work out well. We’ve had a good start on our build and release tooling. As part of that, we also have a very simple example of displaying an image on screen.

Looking Back

The Windows build script worked as expected! You may need to install the SDL2 binaries but after that, you should be good to go. We will have to look into distributing the DLLs along with the package. The previous SDL logo test Windows and macOS versions are over on itch.io.

Next Steps

This time around we’re going to put together a small 240x160 window. It will act window (or camera) into a larger 1440x960 background image. We even have a small 8x16 “thing” that we can move around the space. This will serve us well down the road and is usable in all types of games.

Camera

The basic idea is we have a large background but we only need to render a small part of that to the screen. First we need our camera, so we use sdl.Rect{} and create a rectangle the same size as our display window. Note that this is an example, not the full implementation. At some point most of this code should be on GitHub.

  camera := sdl.Rect{X: 0, Y: 0, W: wWidth, H: wHeight}

We want to constrain the camera in such a way that the “thing” we are moving around will be near the center of the window. We do this by taking the “things” current position (plus half its width/height) and dividing the X and Y by half the windows width and height.

  camera.X = int32((d.posX + d.width/2) - wWidth/2)
  camera.Y = int32((d.posY + d.height/2) - wHeight/2)

We then constrain the camera to keep it from rendering “off-screen”. This allows the “thing” to continue moving toward the edges and have the camera unbound so it doesn’t follow.

  if camera.X < 0 {
    camera.X = 0
  }
  if camera.Y < 0 {
    camera.Y = 0
  }
  if camera.X > bgWidth-camera.W {
    camera.X = bgWidth - camera.W
  }
  if camera.Y > bgHeight-camera.H {
    camera.Y = bgHeight - camera.H
  }

Now we are free to take a chunk of our background texture, which matches the cameras coordinates. renderer.Copy() allows us to copy that chunk to the destination is our current renderer.

  renderer.Copy(bg, &sdl.Rect{
    X: camera.X,
    Y: camera.Y,
    W: wWidth,
    H: wHeight,
  },
    &sdl.Rect{
      X: 0,
      Y: 0,
      W: wWidth,
      H: wHeight,
    })

We then do the same thing with our “thing”. We draw it to the screen on top of the background as centered as need be. And that’s all there is to it!

Give It A Try

Like last time I have release the demo version over on itch.io. Both Windows and macOS versions are up for download. Please give them a try and let me know if they run at all. The movement code is a bit off when the “thing” gets near the edges. I will look into updating that as I have time.

Next Up

I’m not quite sure what I’m going to work on next. I do want to do some internal tooling to make the initial project setup easier. But that won’t count toward next months release. Maybe I’ll work on a little bit of procedural generation. It’d be nice to generate the background or map rather then drawing a shoddy picture in Aseprite. Or we could work on adding camera shake - can’t call yourself indie if you don’t have that massive shake!

Enjoy this post?
How about buying me a coffee?

Game Dev 2020

Wed, 29 Jan 2020 19:09:12 -0800

Background

Having grown up with the rise of entertainment software I have a pretty long history with gaming. As part of that, I’ve flirted off and on with making games. This takes the form of some general messing around. I’ll do that until something else catches my scattered attention.

I’ve been thinking since the beginning of the year that I want to try to cobble something together. I’d be a good reason for me to blog and gives me a nice excuse to continue messing around with Go and SDL2. Why Go? Because it’s portable, I can compile to Windows, macOS, and Linux with ease.

The Plan

I’m planning on trying to release something at the end of every month. It will not be a complete game. Only little bits and pieces that we can put together into a whole at some point.

Each piece will get its own page over at itch.io.

As part of this, I want to first tackle a release process. I’ve decided to do using a makefile. Each project will have the same base layout so I’ll write a small helper script (in Go) to help with that.

I imagine I’ll end up with something very like the following. It’s pretty close to what I already have. macOS and Windows will need to separate build but the Linux build can happen in Docker. That won’t come until the second (or third) release. If I do add it, new builds of older releases get updated.

The layout will look a bit like the following.

sdl2-test00
│   main.go
|   makefile
│   README.md
|
└───assets
│   │   background.png
│
└───buildfiles
|   │   README.md
|
└───macos_buildfiles
|   │   Info.plist
|
└───windows_buildfiles
    |   SDL2.dll

makefile

Below you can see an example of my current makefile. It’s not quite done but it’s getting close enough. I need to do some tweaking to pull the main BINARY_NAME from the current directory. To keep this post short I’ve trimmed out a bit and only left the macOS related.

GOCMD=go
GOBUILD=$(GOCMD) build
GOCLEAN=$(GOCMD) clean
GOTEST=$(GOCMD) test
GOGET=$(GOCMD) get
ITCH_ACCOUNT=$(HANDLE)
ITCH_GAME_URI=$(notdir $(shell pwd))
BINARY_VER ?= $(shell git tag)
BINARY_NAME=sdl2-test00
BINARY_DARWIN=$(BINARY_NAME)
RELEASE_DARWIN=release-darwin

all: clean deps build

build:
  @echo 💻 Building
  @$(GOBUILD) -o $(BINARY_NAME) -v

clean:
  @echo 🧹 Cleaning up
  @$(GOCLEAN)
  @echo ⚠️ Removing binaries
  @rm -f $(BINARY_NAME)
  @rm -f $(BINARY_LINUX)
  @rm -f $(BINARY_WIN)
# Danger ;)
  @echo ⚠️ Removing release directories
  @rm -rf ./$(RELEASE_WIN)
  @rm -rf ./$(RELEASE_LINUX)
  @rm -rf ./$(RELEASE_DARWIN)
  @echo 🧹 Done cleaning

release-darwin:
  @echo 💻 Creating $(RELEASE_DARWIN)/$(BINARY_DARWIN)-$(BINARY_VER).app folders
  @mkdir -p $(RELEASE_DARWIN)/$(BINARY_DARWIN)-$(BINARY_VER).app/Contents/MacOS
  @mkdir -p $(RELEASE_DARWIN)/$(BINARY_DARWIN)-$(BINARY_VER).app/Contents/PlugIns
  @mkdir -p $(RELEASE_DARWIN)/$(BINARY_DARWIN)-$(BINARY_VER).app/Contents/Resources

  @echo 💻 Building $(RELEASE_DARWIN)/$(BINARY_DARWIN)-$(BINARY_VER)
  @GOOS=darwin GOARCH=amd64 $(GOBUILD) -ldflags="-s -w" -o $(RELEASE_DARWIN)/$(BINARY_DARWIN)-$(BINARY_VER).app/Contents/MacOS/$(BINARY_DARWIN)-$(BINARY_VER) -v

  @echo 💻 Copying assets
  @cp -r ./assets $(RELEASE_DARWIN)/$(BINARY_DARWIN)-$(BINARY_VER).app/Contents/MacOS/
  @cp ./macos_buildfiles/Info.plist $(RELEASE_DARWIN)/$(BINARY_DARWIN)-$(BINARY_VER).app/
  @# cp -r ./macos_buildfiles/icons.icns $(RELEASE_DARWIN)/$(BINARY_DARWIN)-$(BINARY_VER).app/Contents/Resources/$(BINARY_DARWIN)-$(BINARY_VER)

  @echo 💻 Uploading to itch.io
  @butler push $(RELEASE_DARWIN) $(ITCH_ACCOUNT)/$(ITCH_GAME_URI):mac

The idea is to make building and releasing easier up front. To that end, I’ve worked on building out putting together a .app package which can then be immediately released to itch.io. I did my test release last night. It can be found over on itch.io.

Enjoy this post?
How about buying me a coffee?

Adding a Book Cover Shortcode for Hugo

Wed, 01 Jan 2020 03:05:29 -0800

Background

There has been some chatter on the micro.blog timeline about adding some better tools for book covers when sharing book suggestions, etc. @michaelfransen started off asking about a gallery of images similar to the way the Photos page works. As the conversation went on @jsonbecker brought up the idea of ISBN to cover image inside of a Hugo data page.

There are a few services that have APIs that should get the image, Goodreads (Amazon) and Google Books are the biggest. Since Hugo can grab JSON when building pages it is possible to just use a short-code to build out what you’d need for a naive implementation.

Drawbacks

While looking into this I noticed that both the Goodreads and Google APIs have their problems. On the Google side of things not every result is going to have larger image sizes:

"imageLinks": {
"smallThumbnail": "http://books.google.com/books/content?id=DRlhnQEACAAJ&printsec=frontcover&img=1&zoom=5&source=gbs_api",
"thumbnail": "http://books.google.com/books/content?id=DRlhnQEACAAJ&printsec=frontcover&img=1&zoom=1&source=gbs_api"
},

You can change the “zoom” but that doesn’t result in an image sadly. Other than that the Google API works pretty much as we’d need. Search for an ISBN and get an object back with results! Oh but wait… there are a bunch of ISBNs that return no cover image.

Not to worry, Goodreads can help us out there. After all, they have the Amazon library of images along with crowd sourced covers. Except in order to look up books on Goodreads you need the ID - and the API call to look up the ID by ISBN doesn’t work. And, from what I’ve read, isn’t very reliable when it does. Also, it responds with XML so, that won’t help.

The final draw back would be that the images aren’t “download” they would be linked from the Goodreads or Google hosting where they are stored. That may not seem like a drawback but at some point they will stop allowing the images to be linked from certain referring domains.

Overcoming Obstacles

Since Google can’t be relied on we going to move them to backup status. We’ll use Goodreads as the primary source and fallback as needed.

That still presents us with the problem of the Goodreads API not working. Well, it turns out the workaround for that is pretty easy. I noticed that if you searched for an ISBN in on the main page you would get a single response with what I was looking for.

Taking a look in Chrome console I was pleased to discover that the search GET request returns a nice JSON object.

[
  {
    "imageUrl": "https://i.gr-assets.com/images/S/compressed.photo.goodreads.com/books/1388494987i/38379._SY75_.jpg",
    "bookId": "38379",
    "workId": "1921329",
    "bookUrl": "/book/show/38379.Ring",
    "from_search": true,
    "qid": "Xoh78SXtTz",
    "rank": 1,
    "title": "Ring (Ring, #1)",
    "bookTitleBare": "Ring",
    "numPages": 282,
    "avgRating": "3.82",
    "ratingsCount": 13682,
    "author": {
      "id": 14465,
      "name": "Kōji Suzuki",
      "isGoodreadsAuthor": false,
      "profileUrl": "https://www.goodreads.com/author/show/14465.K_ji_Suzuki",
      "worksListUrl": "https://www.goodreads.com/author/list/14465.K_ji_Suzuki"
    },
    "kcrPreviewUrl": null,
    "description": {
      "html": "A mysterious videotape warns that the viewer will die in one week unless a certain, unspecified act is performed. Exactly one week after watching the tape, four teenagers die…",
      "truncated": true,
      "fullContentUrl": "https://www.goodreads.com/book/show/38379.Ring"
    }
  }
]

What’s more, we have an image URL right there in the response!

Almost! If only this image was the full cover. It turns out that it is, sort of. If we drop the ._SY75_ we’ll see the “full size image”.

SY75 seems to be some sort of scaling, maybe. If you play around with that you can get back different size images up to what is likely the maximum size. Now we are getting somewhere and we have enough to create our basic short-code!

But first, let’s talk about the third drawback - the eventual loss of images. The way the short-code would work is to build out an <img> link, Hugo can’t download the images. This means the image data remains on a remote server. I can envision one way to work on getting out from under that which would not require attempting to mirror all the images. A simple proxy which looked locally for downloaded images before passing the request upstream would work. This would be “easy” to setup I think - but it’s beyond what we are trying to do right now.

Shortcode

I am still new to working with Hugo templating so there may be better methods to accomplish this but it works. In my theme I have a file at layouts/shortcodes/book.html. Here we’ll take a closer look at what is in it, the complete code can be found a bit further down. I think I may need to make a short-code repo on GitHub as well. I’ll add a link here if/when I do.

First, we’ll declare a variable and assign the URL to an image that says “image not available”. We’ll use this as our image if nothing comes back from the API.

{{ $fin := "https://books.google.com/books/content?id=DRlhnQEACAAJ&printsec=frontcover&img=1&zoom=2&source=gbs_api" }}

Next we will set our API URLs. We’ll also put together our query parameters. This sets us up to use getJSON.

{{ $grUrlPre := "https://www.goodreads.com/book/auto_complete" }}
{{ $googUrlPre := "https://www.googleapis.com/books/v1/volumes" }}

{{ $grIsbn := printf "%s%s" "?format=json&q=" .Params.isbn }}
{{ $googIsbn := printf "%s%s" "?q=isbn:" .Params.isbn }}

We make two calls out to the APIs. I thought about checking one and then the other only if needed. In the end though I think it keeps the snippet much easier to read. After all what’s one more GET request? Anyway, we store the resulting object in a variable.

{{ $grJ := getJSON $grUrlPre $grIsbn }}
{{ $googJ := getJSON $googUrlPre $googIsbn }}

Since we are more concerned with Goodreads we’ll check that first to make sure that the length of the object is larger then 0. If it is then we next assign the first object within the response to a variable. Within that object we’ll pull out the imageUrl value and using regular expression select and replace the “scaling” tag, ._SY75_. with just a .. That’s the end so just write out or final HTML.

{{ if gt (len $grJ) 0 }}
  {{ $i := index $grJ 0 }}
  {{ $fin := replaceRE "._.*_." "." $i.imageUrl }}
  <img src="{{ $fin }}" class="book_cover"/>

But, what if?! Well, we can carry on over to the Google API response if needed. We start of the same, checking that the response has data. If it does have data again we assume that the first portion of the object is what we want. In this example, because we can’t be sure if a larger image exists I am using volumeInfo.imageLinks.thumbnail.

{{ else if gt ($googJ.totalItems) 0 }}
  {{ $i := index $googJ.items 0 }}
  <img src="{{ $i.volumeInfo.imageLinks.thumbnail }}" class="book_cover"/>

And if all that went off the rails? The we default to displaying our image not available image.

{{ else}}
  <img src="{{ $fin }} class="book_cover""/>
{{ end }}

layouts/shortcodes/book.html

{{ $fin := "https://books.google.com/books/content?id=DRlhnQEACAAJ&printsec=frontcover&img=1&zoom=2&source=gbs_api" }}

{{ $grUrlPre := "https://www.goodreads.com/book/auto_complete" }}
{{ $googUrlPre := "https://www.googleapis.com/books/v1/volumes" }}

{{ $grIsbn := printf "%s%s" "?format=json&q=" .Params.isbn }}
{{ $googIsbn := printf "%s%s" "?q=isbn:" .Params.isbn }}

{{ $grJ := getJSON $grUrlPre $grIsbn }}
{{ $googJ := getJSON $googUrlPre $googIsbn }}

  {{ if gt (len $grJ) 0 }}
    {{ $i := index $grJ 0 }}
    {{ $fin := replaceRE "._.*_." "." $i.imageUrl }}
    <img src="{{ $fin }}" class="book_cover"/>
  {{ else if gt ($googJ.totalItems) 0 }}
    {{ $i := index $googJ.items 0 }}
    <img src="{{ $i.volumeInfo.imageLinks.thumbnail }}" class="book_cover"/>
  {{ else}}
    <img src="{{ $fin }}" class="book_cover"/>
  {{ end }}

Usage

That’s great but how does it work? Next time you are writing a post add:

{{&lt; book isbn="9781493415465" &gt;}}

Note, that Hugo was having a heck of a time not actually using this as a short-code. You may need to covert the great-than and less-than signs back to symbols

If all goes well you should see your cover image. Note that we have done no styling so you could end up with an image that’s to larger (or even too small).

Over the next little while I’ll see if we can get our covers to display as a gallery, like the photos do. Until next time…

Enjoy this post?
How about buying me a coffee?

Webmentions are Great (Mostly)

Thu, 26 Dec 2019 22:54:29 -0800

Oh No

Why are there two author icons?! We’ll get back to that in a moment.

Webmetion Bliss

After setting up and documenting how I deployed webmentions on micro.blog it got picked up by a few people on the site! That was really nice - especially since it worked for the most part. I attempted to make it as clear as possible so others could follow in my footsteps and I succeeded - a few people got set up and were running with mentions. There is a great example over on Miraz Jordan’s site, which you can see a snippet of below.

The Troubles

All is not perfect though. So, I thought I’d touch on some of that here. First, if you were to look at the interactions on Miraz’s site you would see some “likes” down near the bottom that rendered with just an icon and a data. Those posts came from Mathew Packer testing out site to site replies.

For some reason those are coming out of webmention.io with what looks to be incorrectly parsed values. Note the '' where you would expect a comment or, more likely, a title to appear.

sentence: "https://www.mathewpacker.com/2019/12/2523/ posted '' linking to https://miraz.me/2019/12/21/thanks-to-shindakun.html"

Though in this case it doesn’t matter since we are not rendering the sentence (or sentence_html). I believe the JSON issue might be why the original implementation for webmention.js didn’t show “shares” or “likes” with text and just showed an author picture and name. I don’t use Wordpress so I am not sure just how far I may troubleshoot that - seems like the “Post Kinds” plugin might not be formatting the post 100% correctly.

Problems at Home

That brings us back to the header image. A few days after the post went up I back ported my changes to my local Hugo theme and pushed them live. Within a day or so I had a few interactions.Chris Aldrich posted a reply to a post by Jan-Lukas Else which linked back to the implementation post. However, it was showing two author photos.

At first I figured that the JavaScript parsing the webmention data was incorrect. However, nothing jumped out as an issue. I then used a microformat2 parser to see if I could pin down the cause from the original mention.

Behold! It turns out that within the reply post Jetpack Lazy Image loading is causing the reply-to author block to end up with an array of images. This seems to bubble back up through the content that webmentions.io is sending to my page.

In this case I could render the sentence_html and not the content but then I’d lose out some nice conversation context.

I noticed one other thing that I’ll need to keep an eye out for and see if I can work around. Most replies aren’t going to have a title so you may end up with something like the following, where what we are replying to is blank.

I have some off looking replies on the webmention post mostly due to bad CSS. And the fact that I appear to have messed up the CSS I was using for content with no author image isn’t great. It makes it difficult to see if the issue is something I’ve done.

I should get those fixed up… and then I need to take a much closer look at the webmention.js and it’s associated CSS.

Ack! Seems rendering the entire content load of a “share” means an entire article might get shared. Will have to look at that closer as well, maybe tomorrow.

Enjoy this post?
How about buying me a coffee?

Adding Webmentions To micro.blog

Fri, 13 Dec 2019 23:16:31 -0800

Last update 12/21/19

removed reply count buttons HTML and JS

adding a note about switching to native micro.blog webmentions in the future

Past updates

added <ul> for shares back in

Background

As part of getting back into using micro.blog I’ve been working to customize my hosted site. One of the things I wanted to do was add webmentions rather then comments like Disqus. Webmentions are coming to MB at some point and most themes already contain references to them:

<link rel="webmention" href="https://micro.blog/webmention" />

I didn’t want to wait though and we don’t need to thanks to Webmention.io! I’m going to try and document exactly what I did through this post. Parts of it will be specific to the theme I’m using, Hello in this case but, hopefully will have enough detail that anyone can follow.

To be clear, if you use this method you may lose some mentions sent to the external endpoint if you switch from webmentions.io back to the native micro.blog webmention endpoint once it’s live

This isn’t new territory but I thought it’d be nice to record exactly what I did to get this working on micro.blog. In fact, I hacked most of this in place based on code released by Keith Grant. Thanks Keith!

Pre-requisites

To get started you really only need a few things:

a micro.blog hosted blog or a static website you want to add webmentions to
a custom theme
basic web development skills (some HTML/JavaScript)

As I mentioned in the lightbox post, @manton has a great YouTube video which covers the basics of getting up and running with a theme. I will be referring to the web based editor but you don’t really need to use that if you want to make changes locally and then push to Git.

Webmention

First thing to do is to visit Webmention.io. Here we’re going to paste in our Micro.blog URL.

After signing in we’ll be brought to the dash board. We are currently only concerned about the to <link>’s in the setup section.

HeadSpace

Step one out of the way we’ll need to edit our theme header to include these links. In this example I’m using the Hyde theme. From here on this will vary from theme to theme but hopefully not too much.

We need to, at this point make sure we’ve read up on custom themes. I can wait, take your time. Go head and switch over to a custom theme if you aren’t already using one.

Alight lets continue.

To get to the micro.blog theme editor goto your “Posts” page, then click on “Design”. Scroll down and click on “Edit Custom Themes”. Now choose the theme you want to edit.

You should now see something similar to the following.

We’ll first edit layouts/partials/head.html. Again this may be slightly different for the theme you are using. Our first change is to pull out the micro.blog webmentions placeholder. So find:

<link rel="webmention" href="https://micro.blog/webmention" />

Found it! Now paste our webmentions.io links in it’s place. For me that would be:

<link rel="webmention" href="https://webmention.io/shindakun-test.micro.blog/webmention" />
<link rel="pingback" href="https://webmention.io/shindakun-test.micro.blog/xmlrpc" />

Click “Update Template” button to save changes. One down.

HTMLing

Alright, now we should get some stuff into the template in order to actually show the mentions. This will be kind of barebones and may not be very pretty, I’ll leave the CSSing to the reader - that comes later though. For now we need to edit our single post HTML. For Hyde that is layouts/post/single.html.

If you look at the screenshot above you’ll see about where we are going to add our HTML. For now we’ll put it after the <div> that is just under {{ .Content }}. And below you can find that HTML - I decided it was easiest to just include it here, even though its quite a big block.

<!-- First we have out reply and like template HTML -->
<template id="reply-template">
  <li class="h-entry">
    <div class="comment-note">
      <div class="comment-note__avatar">
        <a class="js-author" href="">
          <img class="comment-avatar u-photo js-avatar" src=""/>
        </a>
      </div>
      <div class="comment-note__body">
        <div class="p-author">
          <a class="comment-author u-author js-author-name" href=""></a>
          <a class="comment-timestamp u-url js-date js-source" href=""></a>
        </div>
        <div class="comment-body js-content e-entry"></div>
      </div>
    </div>
  </li>
</template>
<template id="like-template">
  <li class="h-entry">
    <div class="reply h-card p-author">
      <a class="reply__bar u-url js-source" href="">
        <div class="reply__author p-name js-sentence js-author-name" href=""></div>
        <div class="reply__date js-date" href=""></div>
      </a>
      <a class="reply__avatar u-author js-author" href="">
        <img class="u-photo js-avatar" src=""/>
      </a>
    </div>
  </li>
</template>

<!-- The webmentions.js file we look at in a bit will fill out our lists -->
<ul class="replies" id="replies"></ul>
<ul class="likes" id="likes"></ul>
<ul class="shares" id="shares"></ul>

<div id="comments"></div>

<!-- I'm including this here to make sure it's only used on the single post page. We'll get into this a bit later. -->
<script type="text/javascript">
  document.addEventListener('DOMContentLoaded', function () {
    fetchWebmentions({{ .Permalink }}, {{ .Aliases }});
  });
</script>

Once more, click on “Update Template” to save our changes. Saving brings us back to the template file list - which works out quite nicely.

JavaScripty

On the file list screen click the “New Template” button. This should really say “New File” since we’re not creating a whole new template but… We should now be looking at an empty text-box. This time we need to note where we want this file to be, in my case to stick with the theme convention I’ll use static/js/webmentions.js. This path may change slightly depending on how the theme or you want it to be - for instance in the Hello theme it might be static/assets/webmentions.js.

Into the actual code text-box we’ll need to paste the following JavaScript. There’s quite a bit. I’ve tested this on two micro.blog sites and it seems to work OK, your milage may vary though. I’m going to put a few comments in just to try and describe a bit of what’s happening in places. I’m not going to go to crazy though so a little JavaScript to general programming knowledge is probably helpful here.

You’ll need to supply you’re own anonymous avatar image for any mentions that don’t have an image.

// You will need to supply an image, for MB hosted blogs I
// recommend uploading  to the image gallery and then pasting
// the link in here
const ANON_AVATAR = '/images/anon-avatar.png';

// fetchWebmentions retireves the actual messages for the URL
// passed in. First we check to see if we're on a page that
// has a comment `div` and then if no URL is passed in we'll
// put one together from the current page and fetch webmentions.
function fetchWebmentions(url, aliases) {
  if (!document.getElementById('comments')) {
    return;
  }
  if (!url) {
    url = document.location.origin + document.location.pathname;
  }
  const targets = getUrlPermutations(url, aliases);

  var script = document.createElement('script');
  var src =
    'https://webmention.io/api/mentions?perPage=500&jsonp=parseWebmentions';
  targets.forEach(function(targetUrl) {
    src += `&target[]=${encodeURIComponent(targetUrl)}`;
  });
  src += `&_=${Math.random()}`;
  script.src = src;
  script.async = true;
  document.getElementsByTagName('head')[0].appendChild(script);
}

// getUrlPermutations builds up a list of potential URLs to
// check for mentions on. You will need to update the URLs
// to point to your own MB instance. The `localhost:1313` one
// can probably be removed unless your going to be testing
// the theme locally with Hugo.
function getUrlPermutations(url, aliases) {
  const urls = [];
  url = url.replace('http://localhost:1313', 'https://shindakun-test.micro.blog/');
  urls.push(url);
  urls.push(url.replace('https://', 'http://'));
  if (url.substr(-1) === '/') {
    var noslash = url.substr(0, url.length - 1);
    urls.push(noslash);
    urls.push(noslash.replace('https://', 'http://'));
  }
  if (aliases) {
    aliases.forEach(function(alias) {
      urls.push(`https://shindakun-test.micro.blog/${alias}`);
    });
  }
  return urls;
}

function parseWebmentions(data) {
  var links = data.links.sort(wmSort);
  var likes = [];
  var reposts = [];
  var replies = [];
  links.map(function(l) {
    if (!l.activity || !l.activity.type) {
      console.warning('unknown link type', l);
      return;
    }
    if (!l.verified) {
      return;
    }
    switch (l.activity.type) {
      case 'like':
        likes.push(l);
        break;
      case 'repost':
      case 'link':
        reposts.push(l);
        break;
      default:
        replies.push(l);
        break;
    }
  });
  renderLikes(likes);
  renderReposts(reposts);
  renderReplies(replies);
  showInteractions();
}
window.parseWebmentions = parseWebmentions;

function wmSort(a, b) {
  const dateA = getWmDate(a);
  const dateB = getWmDate(b);
  if (dateA < dateB) {
    return -1;
  } else if (dateB < dateA) {
    return 1;
  }
  return 0;
}

function getWmDate(webmention) {
  if (webmention.data.published) {
    return new Date(webmention.data.published);
  }
  return new Date(webmention.verified_date);
}

var months = [
  'Jan',
  'Feb',
  'Mar',
  'Apr',
  'May',
  'Jun',
  'Jul',
  'Aug',
  'Sep',
  'Oct',
  'Nov',
  'Dec',
];

function renderLikes(likes) {
  var t = document.getElementById('like-template').content;
  var list = document.getElementById('likes');
  likes.map(function(l) {
    console.log(l)
    fillTemplate(t, {
      photo: l.data.author.photo || ANON_AVATAR,
      name: l.data.author.name,
      authorUrl: l.data.author.url,
      url: l.data.url,
      sentence: l.activity.sentence,
      date: new Date(l.data.published || l.verified_date),
    });
    var clone = document.importNode(t, true);
    list.appendChild(clone);
  });
}

function getHostName(url) {
  var a = document.createElement('a');
  a.href = url;
  return (a.hostname || '').replace('www.', '');
}

function renderReposts(reposts) {
  var t = document.getElementById('like-template').content;
  var list = document.getElementById('shares');
  reposts.map(function(l) {
    let data;
    if (l.data.author) {
      data = {
        photo: l.data.author.photo || ANON_AVATAR,
        name: l.data.author.name,
        authorUrl: l.data.author.url,
        url: l.data.url,
        date: new Date(l.data.published || l.verified_date),
      };
    } else {
      data = {
        photo: ANON_AVATAR,
        name: getHostName(l.data.url) || 'inbound link',
        authorUrl: l.data.url,
        url: l.data.url,
        date: new Date(l.data.published || l.verified_date),
      };
    }
    fillTemplate(t, data);
    var clone = document.importNode(t, true);
    list.appendChild(clone);
  });
}

function renderReplies(replies) {
  var t = document.getElementById('reply-template').content;
  var list = document.getElementById('replies');
  replies.map(function(l) {
    let data;
    if (l.data.author) {
      data = {
        photo: l.data.author.photo || ANON_AVATAR,
        name: l.data.author.name,
        authorUrl: l.data.author.url,
        url: l.data.url,
        date: new Date(l.data.published || l.verified_date),
        content: l.data.content,
      };
    } else {
      data = {
        photo: ANON_AVATAR,
        name: getHostName(l.data.url) || 'inbound link',
        authorUrl: l.data.url,
        url: l.data.url,
        date: new Date(l.data.published || l.verified_date),
        content: l.data.content,
      };
    }

    fillTemplate(t, data);
    var clone = document.importNode(t, true);
    list.appendChild(clone);
  });
}

// fillTemplate marries a signle webmention to the
// template. The completed template is used in the
// render functions to actually display on the page.
function fillTemplate(template, vals) {
  template.querySelector('.js-avatar').src = vals.photo;
  template.querySelector('.js-author').href = vals.authorUrl;
  template.querySelector('.js-author-name').innerHTML = vals.name;
  template.querySelector('.js-author-name').href = vals.authorUrl;
  template.querySelector('.js-source').href = vals.url;
  if (vals.sentence) {
    template.querySelector('.js-sentence').innerText = vals.sentence;
  }
  const date = template.querySelector('.js-date');
  if (date) {
    date.innerHTML = formatDate(vals.date);
  }
  if (vals.content) {
    template.querySelector('.js-content').innerHTML = vals.content;
  }
}

// formatDate formats the date.
function formatDate(date) {
  if (!date) {
    return '';
  }
  return `${date.getDate()} ${months[date.getMonth()]} ${date.getFullYear()}`;
}

// This is leftover from the original implmentation,
// commented out since I don't use it in my version.
// Maybe I should just remove it for clarity sake.
// ¯\_(ツ)_/¯
function showInteractions() {
 // document.getElementById('comments-loader').classList.add('is-hidden');
 // document.getElementById('comments').classList.remove('is-hidden');
}

I recommend reading it through even if you don’t understand all of it …

Now save the file which brings us back to the file list.

Back to the Head

JavaScript in place we now need to go back into the header file layouts/partials/head.html. Scroll all the way down and just before the </head> past in:

<script src="{{ "js/webmentions.js" | relURL }}"></script>

And save the file.

Mission Accomplished

We did it! Technically we’re all done at this point. If we were to look at a post on our side we should see something like the following from my test site

Neat!

Alright but how can we tell if it’s really working? Webmention Rocks will help us make sure everything is actually working as it should. The site has a number of tests to ensure your site is set up and ready to go. But for the purposes of this post we are mostly interested in the receiver tests. First one is first I suppose - visit https://webmention.rocks/receive/1 and enter the URL of your site to sign-in.

Perfect! Now lets enter a URL from one of our MB posts.

If all went well you should see something similar to the following image.

Awesome! Now go over to the same page on MB and refresh…

Yes the image Webmention Rocks uses is pretty big by default but, who cares we have one webmention! Alright, I care… For best results you’ll want to use some CSS to style the replies. I have left that out since that’s a but outside what we are trying to do and I don’t really enjoy mucking about with CSS.

Enjoy this post?
How about buying me a coffee?

Adding an Image Lightbox To micro.blog

Fri, 06 Dec 2019 23:15:31 -0800

Background

A day or so ago someone posted in one of the micro.blog slack channels asking how to add a lightbox. I had been meaning to update my setup here on the blog and add a lightbox since my current images weren’t linked. Mostly though, I wanted to get rid of jQuery. So with that in mind we’re going to go through the first steps of adding a lightbox. We’ll first add it so images in individual post are linked. This also lays some of the ground work for us to extend the “photos” page to include a lightbox.

The first step was to find a “modern” lightbox solution. As I mentioned the majority of older solutions used jQuery and I don’t want another dependancy. jQuery is great and all but, I’m trying to watch my overall page weight. After some searching around I settled on GLightbox.

Hugo Themes

micro.blog, like this site, uses Hugo for hosted blogs. This setup allows to have control over the theme so we can easily pick a custom theme and extend it with our gallery script. One thing to take note of - I am using the Hello and while this should work on any Hugo there will be differences if you are using another theme.

GLightbox JS and CSS

GLightbox is very easy to install which makes it perfect for this. We’ll need to edit our theme so I recommend first going through the steps in the help.micro.blog section on custom themes.

CSS

Assuming you are using the Hello theme we now need to edit layouts/partials/head.html and include our CSS and the actually Glightbox JavaScript. This is relatively easy though, just follow the instructions on the Micro.blog help (Manton has a great YouTube video which covers the basics).

Once in the editor, we’ll want to add the following two lines:

<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/glightbox/dist/css/glightbox.min.css">
<script src="https://cdn.jsdelivr.net/gh/mcstudios/glightbox/dist/js/glightbox.min.js"></script>

JavaScript

So far so good! I have a feeling the next part is going to be different with every theme, so it might not be as easy. When adding an image to a post through the micro.blog website it is simply included as an <img> tag. The last thing I want to do is have to go in and add all the tags to images as I post them.

With that in mind I crafted a passible bit of JavaScript which will pull out any images that appear within post content and automatically link them so they will be shown in the lightbox.

Neat.

To add the JavaScript we’ll again need to edit part of the theme. In this case we’ll want to open static/assets/main.js for editing. Once open we can added the following JavaScript to the top. Feel free to take out the comments, I’ve just added them for this post.


// First we'll grab all the images inside a post paragraph
let ims = document.querySelectorAll('.post-content p img')

// If ims === 0 lets not do anything
if (ims.length > 0) {

  // Now we'll loop through all of the images that were picked up
  // Note that I'm doing no real error handling, you might want to clean this up
  for (let i = 0; i < ims.length; i++) {

    // For each image we get the parent element, in this case it will be <p>
    let parentElm = ims[i].parentElement;

    // Using the paraent element and the image object we replace the innerHTML
    // with our image with the class "glightbox" and a link to the image
    parentElm.innerHTML = '<a href="' + ims[i].src + '"><img src="' + ims[i].src + '" class="glightbox"></a>'
  }
}

// Finally we call GLightbox and if all went well...
const lightbox = GLightbox({});

Caveats

“This code works on my computer!” Hopefully, if everything went smoothly then the image above should be clickable and should appear in the lightbox. The JavaScript image selection code is likely extremely brittle and probably won’t work for many themes. There should be enough information here for others to follow. For best results you will probably want to keep to only one image per post (though it should work with more).

Update

As Jason Becker points out since we are using Hugo we can take advantage of custom shortcodes. Also as @bix noted, the docs for Glightbox indicate that the class is added to the <a> anchor tag and not the image itself. It seems to work OK either way in my testing but if you are having an issue that could be why.

Enjoy this post?
How about buying me a coffee?

Attempting to Learn Go - Issuer 04 - Addendum

Sat, 13 Jul 2019 10:52:32 -0700

Whoops

Our last Issuer post left out one small bit of code. Let’s fix that and wrap this up. Also, let’s cover how to test the webhook before moving to Google Cloud Functions.

Missed `if`

As I published Issuer to Google Cloud Functions I meant to update the code. The idea is to only create new issues when the originating payload action was opened. The final revision of the post left that out though so here we go. Let’s look at our updated HandleWebhook().

func HandleWebhook(res http.ResponseWriter, req *http.Request) {
  var Payload Payload
  defer req.Body.Close()

  p, err := github.ValidatePayload(req, []byte(Secret))
  if err != nil {
    http.Error(res, "bad request: "+err.Error(), 400)
    log.Printf("bad request: %v", err.Error())
    return
  }

  err = json.Unmarshal(p, &Payload)
  if err != nil {
    http.Error(res, "bad request: "+err.Error(), 400)
    log.Printf("bad request: %v", err.Error())
    return
  }

Here we are - we check Payload.Action to ensure it’s opened and if so then we create out new issue.

  if Payload.Action == "opened" {
    err = createNewIssue(&Payload)
    if err != nil {
      log.Printf("bad request: %v", err.Error())
      return
    }
  }
}

Testing Webhooks

Before pushing to GCF it was necessary to test the webhooks to ensure I was receiving what I needed. This is actually pretty easy thanks to ngrok. ngrok is a command line tool which allows us to access our internal program from an external network. No need to worry about setting up your firewall or tunneling.

`ngrok`

To do this we’ll need to make sure we have ngrok installed. On a Mac, which is where I tend to do most of my development lately, it’s a simple matter of installing via brew.

brew install ngrok

If you happen to be using Windows, you can download from the ngrok website, https://ngrok.com/download. Follow the instructions to install on the website. Once installed you should be able to open a command prompt or terminal window and run ngrok.

➜  ~ ngrok
NAME:
   ngrok - tunnel local ports to public URLs and inspect traffic

DESCRIPTION:
    ngrok exposes local networked services behinds NATs and firewalls to the
    public internet over a secure tunnel. Share local websites, build/test
    webhook consumers and self-host personal services.
    Detailed help for each command is available with 'ngrok help <command>'.
    Open http://localhost:4040 for ngrok's web interface to inspect traffic.
...

Now What

Now what? That’s easy! Let’s run…

ngrok http 3000

Assuming you are running the code we wrote last time, that runs on port 3000. Now open a new terminal window and lets run our code.

go run main.go

Now we’re up and running, lets enable some webhooks on GitHub and do some actual testing.

GitHub Webhooks

With both ngrok and our code running we go to GitHub! Pick a repo, or create a test one. Under the settings tab, chose “Webhooks” on the left. Click “Add webhook” on the upper right. Enter our ngrok URL, ex: https://9ba7d0f6.ngrok.io and add /webhook. In “Content type” make sure you select application/json. For this project I chose “Let me select individual events” and then selected only “Issues”. Then click “Add webhook” at the bottom.

GitHub will send off a test post, we should see something right away.

And there we have it! It’s mostly blank because it’s just a test but it’s enough to show that we’re working as expected.

You can find the code for this and most of the other Attempting to Learn Go posts in the repo on GitHub.

{% github shindakun/atlg %}

Enjoy this post?
How about buying me a coffee?

Attempting to Learn Go - Issuer 03 - Cloud Function Go!

Sat, 22 Jun 2019 16:39:49 -0700

Functioning In The Cloud

Welcome back! We are just done with our GitHub TODO issue creator thing. This time around we’re going to go through the steps to deploy it as a Google Cloud Function. To do this we’ll need to alter the code. But before we dive into the code itself we’ll need to get setup to use Go modules.

Go Mod

First, we’ll need to get our mod file ready. You may or may not already have modules enabled. If you do you should be able to leave off the GO111MODULE=on part of the command.

GO111MODULE=on go mod issuer

Now, we need to make sure we’ve got all the modules we need.

➜  issuer GO111MODULE=on go mod tidy
go: finding github.com/google/go-github/v25/github latest
go: finding golang.org/x/oauth2 latest
go: downloading google.golang.org/appengine v1.4.0
go: extracting google.golang.org/appengine v1.4.0
go: downloading github.com/golang/protobuf v1.2.0
go: extracting github.com/golang/protobuf v1.2.0
go: downloading golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6
go: extracting golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6

Looking good so far! Next we want to vendor our dependancies. This copies the modules we downloaded in the previous step into our code directory.

GO111MODULE=on go mod vendor

We should now see the vendor directory.

➜  issuer ls
go.mod  go.sum  main.go vendor

Alterations

Now let’s make a few changes to our code to get it ready for deploying as a Cloud Function. We’ll be starting off the same as last time. You may notice that I’ve removed the ability to query /status, it is not longer needed.

package issuer

import (
  "context"
  "encoding/json"
  "fmt"
  "log"
  "net/http"
  "os"
  "strconv"

  "github.com/google/go-github/v25/github"
  "github.com/shindakun/envy"
  "golang.org/x/oauth2"
)

Note that I’ve pulled out our constants and replaced them will variables. This allows us to make the code a bit more generic and so others can use it. We’ll cover how we are setting up these variables a bit later on. Note, some people may not like the use of global variables here. Since our function is so small and we have a good idea what’s happening in each step it’s OK to use them for now.

Of course if you’d rather not have globals feel free to remove them. You can hard code as constants or load them inside our main function call.

// RepoOwner is the owner of the repo we want to open an issue in
var RepoOwner string

// IssueRepo is the repo we want to open this new issue in
var IssueRepo string

// ProjectColumn is the TODO column number of the project we want to add the issue to
var ProjectColumn int64

// Token is the GitHub Personal Access Token
var Token string

// Secret is used to validate payloads
var Secret string

// Payload of GitHub webhook
type Payload struct {
  Action string `json:"action"`
  Issue  struct {
    URL           string `json:"url"`
    RepositoryURL string `json:"repository_url"`
    Number        int    `json:"number"`
    Title         string `json:"title"`
    Body          string `json:"body"`
  } `json:"issue"`
  Repository struct {
    Name string `json:"name"`
  } `json:"repository"`
}

I’ve also changed handleWebhook() to HandleWebhook(). Exporting the function is what allows us to call it as the cloud function.

func HandleWebhook(res http.ResponseWriter, req *http.Request) {
  var Payload Payload
  defer req.Body.Close()

  p, err := github.ValidatePayload(req, []byte(Secret))
  if err != nil {
    http.Error(res, "bad request: "+err.Error(), 400)
    log.Printf("bad request: %v", err.Error())
    return
  }

As @kunde21 pointed out last week we are better off using json.Unmarshal() here. This also allows us to remove the imports for bytes and io/ioutil.

  err = json.Unmarshal(p, &Payload)
  if err != nil {
    http.Error(res, "bad request: "+err.Error(), 400)
    log.Printf("bad request: %v", err.Error())
    return
  }

  err = createNewIssue(&Payload)
  if err != nil {
    log.Printf("bad request: %v", err.Error())
    return
  }
}

func createNewIssue(p *Payload) error {
  log.Printf("Creating New Issue.\n")
  log.Printf("  Name: %#v\n", p.Repository.Name)
  log.Printf("  Title: %#v\n", p.Issue.Title)
  log.Printf("  Body: %#v\n", p.Issue.Body)
  log.Printf("  URL: %#v\n", p.Issue.URL)

  ctx := context.Background()
  ts := oauth2.StaticTokenSource(
    &oauth2.Token{AccessToken: Token},
  )
  tc := oauth2.NewClient(ctx, ts)

  client := github.NewClient(tc)

  title := fmt.Sprintf("[%s] %s", p.Repository.Name, p.Issue.Title)
  body := fmt.Sprintf("%s\n%s/%s#%d", p.Issue.Body, RepoOwner, p.Repository.Name, p.Issue.Number)

  issue := &github.IssueRequest{
    Title: &title,
    Body:  &body,
  }

  ish, _, err := client.Issues.Create(ctx, RepoOwner, IssueRepo, issue)
  if err != nil {
    log.Printf("error: %v", err)
    return err
  }

  id := *ish.ID
  card := &github.ProjectCardOptions{
    ContentID:   id,
    ContentType: "Issue",
  }

  _, _, err = client.Projects.CreateProjectCard(ctx, ProjectColumn, card)
  if err != nil {
    log.Printf("error: %v", err)
    return err
  }

  return nil
}

Now we get to one of the biggest changes. Since the cloud function is going to call HandleWebhook(), we no longer need our main(). But that presents an issue, we have some environment variables we want to use. We could load them in the HandleWebhook() call it’s more appropriate to make use of Go’s init().

init() runs before main (or our handler in this case), which allows us to load our variables as normal.

func init() {
  log.Println("Issuer")
  var err error
  Token, err = envy.Get("GITHUBTOKEN")
  if err != nil || Token == "" {
    log.Printf("error: %v", err)
    os.Exit(1)
  }

  Secret, err = envy.Get("SECRET")
  if err != nil || Secret == "" {
    log.Printf("error: %v", err)
    os.Exit(1)
  }

  RepoOwner, err = envy.Get("REPOOWNER")
  if err != nil || RepoOwner == "" {
    log.Printf("error: %v", err)
    os.Exit(1)
  }

  IssueRepo, err = envy.Get("ISSUEREPO")
  if err != nil || IssueRepo == "" {
    log.Printf("error: %v", err)
    os.Exit(1)
  }

ProjectColumn requires a bit more setting up since I never extended envy to return ints. So we need to convert to an int64 before we can use the column numeric ID to create our TODO card on the kanban board.

  n, err := envy.Get("PROJECTCOLUMN")
  if err != nil || n == "" {
    log.Printf("error: %v", err)
    os.Exit(1)
  }
    ProjectColumn, err = strconv.ParseInt(n, 10, 64)
      if err != nil || ProjectColumn == 0 {
    log.Printf("error: %v", err)
    os.Exit(1)
  }
}

Go To The Cloud

I am going to assume that you have the Google Cloud command line tools installed and a project set up. If you do not Google has some very good tutorials. Checkout https://cloud.google.com/functions/docs/quickstart for more

Note: You wouldn’t want to have the secret and the token deployed like this in production. You would instead want to use Cloud KMS or Runtime Configurator. I’m living dangerously.

➜  issuer gcloud functions deploy issuer --entry-point HandleWebhook --runtime go111 --trigger-http --memory=128MB --set-env-vars SECRET=secret,GITHUBTOKEN=token,REPOOWNER=shindakun,ISSUEREPO=to,PROJECTCOLUMN=5647145

Deploying function (may take a while - up to 2 minutes)...done.
availableMemoryMb: 128
entryPoint: HandleWebhook
environmentVariables:
  GITHUBTOKEN: token
  ISSUEREPO: to
  PROJECTCOLUMN: '5647145'
  REPOOWNER: shindakun
  SECRET: secret
httpsTrigger:
  url: https://us-east1-golang-slackbot.cloudfunctions.net/issuer
labels:
  deployment-tool: cli-gcloud
name: projects/golang-slackbot/locations/us-east1/functions/issuer
runtime: go111
serviceAccountEmail: golang-slackbot@appspot.gserviceaccount.com
sourceUploadUrl: https://storage.googleapis.com/gcf-upload-us-central1-eeb5af0e-fd09-4fe7-30851592ebba/bcc11c6f-55fc-4d73-864a-6a89813206a6.zip?GoogleAccessId=service-84452958925@gcf-gserviceaccount.com&Expires=1561244032&Signature=QX%2BKy5j6YTA6%D%3D
status: ACTIVE
timeout: 60s
updateTime: '2019-06-22T22:24:37Z'
versionId: '1'

Next time

And there we have it! We’re now live and update any source repos to use our trigger URL. That makes this stage of our issuer complete. New issues will appear in our target repo and on the TODO section of the kanban board!

What should we tackle next time? I’ll have to take a look at the kanban board and see if any ideas jump out at me. Until then feel free to let me know if you spot something to refactor.

You can find the code for this and most of the other Attempting to Learn Go posts in the repo on GitHub.

{% github shindakun/atlg %}

Enjoy this post?
How about buying me a coffee?

Attempting to Learn Go - Issuer 02

Sat, 15 Jun 2019 12:12:44 -0700

GitHub Issuer

Welcome back! Though if you haven’t read the first part, you may want to. We’re expanding on the code that we write last time. Adding in the ability to actually create new issues in our TODO repository and add them to the kanban board. Yes the most over-engineered TODO “system” is going to get an upgrade. With that out of the way lets get right into it.

Lets Go

Our imports have expanded as we’re pulling in a bunch of bits from the standard library and a few external packages. The go-github package is going to do a quite a bit of heavy lifting for us. oauth2 is coming along for the ride so we can use a GitHub personal access token to authorize our requests.

package main

import (
  "bytes"
  "context"
  "encoding/json"
  "fmt"
  "io/ioutil"
  "log"
  "net/http"
  "os"

  "github.com/google/go-github/v25/github"
  "github.com/shindakun/envy"
  "golang.org/x/oauth2"
)

Currently, we’re setting a few constants. We may bring theses up out of the code and make them environment variables in the “production” version. For local testing though it’s probably fine. The token however, is already set as an environment variable, which should keep me from accidentally committing it to GitHub. It’s good practice to keep tokens out of the code whenever possible.

const (

  // RepoOwner is the owner of the repo we want to open an issue in
  RepoOwner = "shindakun"

  // IssueRepo is the repo we want to open this new issue in.
  IssueRepo = "to"

  // ProjectColumn is the TODO column number of the project we want to add the issue to
  ProjectColumn = 5647145
)

// Token is the GitHub Personal Access Token
var Token string

// Secret is used to validate webhook payloads
var Secret string

Our Payload is pretty much set, we don’t need anything else from the responses for now. Our status handler will remain the same as well.

// Payload of GitHub webhook
type Payload struct {
  Action string `json:"action"`
  Issue  struct {
    URL           string `json:"url"`
    RepositoryURL string `json:"repository_url"`
    Number        int    `json:"number"`
    Title         string `json:"title"`
    Body          string `json:"body"`
  } `json:"issue"`
  Repository struct {
    Name string `json:"name"`
  } `json:"repository"`
}

func status(res http.ResponseWriter, req *http.Request) {
  fmt.Fprintf(res, "Hello!")
}

The webhook handler starts off the same. But quickly deviates.

func handleWebhook(res http.ResponseWriter, req *http.Request) {
  var Payload Payload
  defer req.Body.Close()

We take our incoming request and pass it and our Secret into github.ValidatePayload(). The X-Hub-Signature on the incoming request comes with a signature compare against our calculated signature. If it matches we’re good to go.

The HMAC hex digest of the response body. This header will be sent if the webhook is configured with a secret. The HMAC hex digest is generated using the sha1 hash function and the secret as the HMAC key.

This protects us from some one accidentally finding our endpoint and submitting requests. Sure the chances are low but why take chances. If the request doesn’t pass validation we simply return and carry on.

  p, err := github.ValidatePayload(req, []byte(Secret))
  if err != nil {
    http.Error(res, "bad request: "+err.Error(), 400)
    log.Printf("bad request: %v", err.Error())
    return
  }

github.ValidatePayload() returns a []byte of the payload which we need to wrap in a “ReadCloser” which we can then pass to jsonNewDecoder() so we can parse the JSON object as our final Payload. Again, if anything goes wrong we’ll log the error and return. If all goes well, we pass our Payload to createNewIssue().

  decoder := json.NewDecoder(ioutil.NopCloser(bytes.NewBuffer(p)))
  err = decoder.Decode(&Payload)
  if err != nil {
    http.Error(res, "bad request: "+err.Error(), 400)
    log.Printf("bad request: %v", err.Error())
    return
  }

  err = createNewIssue(&Payload)
  if err != nil {
    log.Printf("bad request: %v", err.Error())
    return
  }
}

createNewIssue() first starts by logging out the details of our payload. This is just for testing purposes and will be removed I think.

func createNewIssue(p *Payload) error {
  log.Printf("Creating New Issue.\n")
  log.Printf("  Name: %#v\n", p.Repository.Name)
  log.Printf("  Title: %#v\n", p.Issue.Title)
  log.Printf("  Body: %#v\n", p.Issue.Body)
  log.Printf("  URL: %#v\n", p.Issue.URL)

First things first, we’ll get our oauth2 and GitHub client ready to go. This is as recommended by the go-github repo.

  ctx := context.Background()
  ts := oauth2.StaticTokenSource(
    &oauth2.Token{AccessToken: Token},
  )
  tc := oauth2.NewClient(ctx, ts)

  client := github.NewClient(tc)

Now it’s time to build our new issue. I wanted the title to reflect which repo it was coming from.

[From repo] Remember to write a post

The body of the repo holds whatever was originally entered and a link back to the source repo. We then pack the title and body into github.IssueRequest and create the new issue!

  title := fmt.Sprintf("[%s] %s", p.Repository.Name, p.Issue.Title)
  body := fmt.Sprintf("%s\n%s/%s#%d", p.Issue.Body, RepoOwner, p.Repository.Name, p.Issue.Number)

  issue := &github.IssueRequest{
    Title: &title,
    Body:  &body,
  }

  ish, _, err := client.Issues.Create(ctx, RepoOwner, IssueRepo, issue)
  if err != nil {
    log.Printf("error: %v", err)
    return err
  }

We are not quite done though. I want to make sure the new issue is added to the TODO kanban board. So we take the details from the new issue, extract the issue ID number and setup a new “card” with github.ProjectCardOptions.

  id := *ish.ID
  card := &github.ProjectCardOptions{
    ContentID:   id,
    ContentType: "Issue",
  }

We aren’t to concerned with the details return from this call so we just check for an error and return if need be.

  _, _, err = client.Projects.CreateProjectCard(ctx, ProjectColumn, card)
  if err != nil {
    log.Printf("error: %v", err)
    return err
  }

  return nil
}

And that brings us to our updated main(). We’ve added a bit of code to grab our environment variables and if not set we’ll bail out with an error.

func main() {
  log.Println("Issuer")
  var err error

  Token, err = envy.Get("GITHUBTOKEN")
  if err != nil || Token == "" {
    log.Printf("error: %v", err)
    os.Exit(1)
  }

  Secret, err = envy.Get("SECRET")
  if err != nil || Secret == "" {
    log.Printf("error: %v", err)
    os.Exit(1)
  }

  http.HandleFunc("/", status)
  http.HandleFunc("/webhook", handleWebhook)
  http.ListenAndServe(":3000", nil)
}

Running

Alright lets run it and make a new issue in our test “from” repo.

SECRET=TESTSECRET GITHUBTOKEN=1234567890 go run main.go
2019/06/15 11:23:32 Issuer
2019/06/15 11:24:42 Creating New Issue.
2019/06/15 11:24:42   Name: "from"
2019/06/15 11:24:42   Title: "asdfasdf"
2019/06/15 11:24:42   Body: "asdfasdfasdfasdfasdf"
2019/06/15 11:24:42   URL: "https://api.github.com/repos/shindakun/from/issues/13"

Perfect! Now all we need to do is throw it on a box and point our GitHub repos webhook settings at the proper URL.

Next time

That went pretty smooth! Next time I think we’ll convert this into something we can deploy on Google Cloud Functions! Which will make it much easier to deploy.

Questions and comments are welcome!

You can find the code for this and most of the other Attempting to Learn Go posts in the repo on GitHub.

Enjoy this post?
How about buying me a coffee?

Attempting to Learn Go - Issuer 01

Sat, 08 Jun 2019 10:49:37 -0700

GitHub Issuer

If you didn’t happen to read the last post I’ll give you a brief overview. I’m trying out a new system to tackle my project load and my TODO list.

The problem is that the current setup is very manual. Each “thing”, or project, I want to do gets its own repository with its own automated kanban board. I open an issue in the repo, assign it to the project board and “bam”, new task. But, I then need to add that issue to my master “TODO” repo by hand.

We can’t have that! It makes the system more work than it should be. That means we won’t use it.

But, we have a solution! GitHub repositories can have webhooks enabled. So, on every new issue, a POST request with the issue details goes to our webhook. That’s where Issuer is going to come in. It will live on a server waiting for incoming POST events. Once one comes it will use the GitHub API to create an issue in the TODO repo.

Lets Go

OK! Let’s jump right in and see what we are dealing with. If you read the second or third articles in the Uploader series then parts of the base code will look similar. We’re using a very standard approach to a Go web server.

package main

import (
  "encoding/json"
  "fmt"
  "log"
  "net/http"
)

The GitHub issue payload is “massive” compared to what we actually need from it. So our Payload struct is only going to include exactly what we need to create a new issue.

type Payload struct {
  Action string `json:"action"`
  Issue  struct {
    URL           string `json:"url"`
    RepositoryURL string `json:"repository_url"`
    Number        int    `json:"number"`
    Title         string `json:"title"`
    Body          string `json:"body"`
  } `json:"issue"`
  Repository struct {
    Name     string `json:"name"`
  } `json:"repository"`
}

Hello! This isn’t needed but I like to have it as a small health check. In a more robust application you may want to return some sort of metrics from the server. Then if I wanted I could point Uptime Robot at / to alert if its down.

func status(res http.ResponseWriter, req *http.Request) {
  fmt.Fprintf(res, "Hello!")
}

I’ll cover how to enable webhooks in a repo and how we can test a bit later on. For now, let’s take a closer look at our webhook handler function. We begin by setting up our Payload variable and defer closing the request body. We then use json.NewDecoder() to get ready and decode our JSON payload. If decode.Decoder() returns an error we’ll first return a 400 to the originating server. We then log out the error to the console and then return from the function.

func handleWebhook(res http.ResponseWriter, req *http.Request) {
  var Payload Payload
  defer req.Body.Close()

  decoder := json.NewDecoder(req.Body)
  err := decoder.Decode(&Payload)
  if err != nil {
    http.Error(res, "bad request: "+err.Error(), 400)
    log.Printf("bad request: %v", err.Error())
    return
  }

Assuming all has gone well we print the webhook details in to the console.

  log.Printf("%#v\n", Payload.Repository.Name)
  log.Printf("%#v\n", Payload.Issue.URL)
  log.Printf("%#v\n", Payload.Issue.Title)
  log.Printf("%#v\n", Payload.Issue.Body)
  log.Printf("%#v\n", Payload.Issue.Number)
  log.Printf("%#v\n", Payload.Issue.RepositoryURL)
}

For this basic example, our main consists of two handlers and http.ListenAndServe(). We may want to extend this at some point to allow the use of alternative HTTP clients. To keep the code simple for now we’ll leave it.

func main() {
  log.Println("Issuer")

  http.HandleFunc("/", status)
  http.HandleFunc("/webhook", handleWebhook)
  http.ListenAndServe(":3000", nil)
}

Next time

When next we meet we’ll be adding the ability to create an issue based on the payload received. If that goes well - we’ll even add the code to add issues to the TODO project. 🎉

You can find the code for this and most of the other Attempting to Learn Go posts in the repo on GitHub.

{% github shindakun/atlg %}

Enjoy this post?
How about buying me a coffee?

Working On A New Flow

Tue, 04 Jun 2019 15:08:36 -0700

New Flow

I’ve realized over the last couple of weeks that I have way more projects that I want to work on then time in the day. I just don’t have as much free time as I used to and can’t seem to burn the midnight oil anymore.

I’ve tried a few methods to try and keep my projects well defined and under a manageable scope. Each has had its drawbacks and never quite worked. My suspicion is that they have been too tailored too whoever thought of it.

A few days back @ben posted about limiting procrastination which brought me back to thinking about this.

My attention will always be focused on what I shouldn’t be doing at any time - it’s taken me at least 10 minutes to get back to this sentence. With some practice and a system that works for me maybe I can tackle that and squeeze in getting some more projects done.

Repo Madness

After some brainstorming, I decided I would try a different approach. I already keep most of my post ideas in a repo. Why can’t I keep all my todo’s in repo as well? This approach allows me to continue using tools I already use, no adding new services is a plus.

I’ve just started planning this out but here is what I’ve got so far. It’s a bit manual at this point while I figure it out but bits should be automated as it comes into focus.

Each repo has a project with an automated kanban board. Issues from each repo need to bubble up to the TODO repo which I’ll use as my master todo list. From there I can shuffle them around to any other columns I might need as needed. At the start I think we’ll stick to just “To Do”, “In Progress”, and “Done”.

Example repos:

TODO Repo
- TODO Project (automated kanban board)
Posts Repo
- Posts Project (automated kanban board)
Hugo Theme Repo
- Theme Project (automated kanban board)
Reading Repo
- Reading Project (automated kanban board)

OK, let’s say we want to write this post. We first make an issue in the Posts repo. For now, we’ll also need to manually make a post in the TODO repo. In that issue, we’ll refer back to the posts repo shindakun/posts#1. This will link back so we can open the TODO kanban board and quickly move around to the other repos.

We can then do the same in our Hugo repo. Or if I have a new book I want to read, Procedural Generation in Game Design. I can throw an issue into the reading repo.

I can create issues from my phone if I need to, or jot them down in a notebook to be entered later. I’d need to remember to sync any issues to the TODO repo at first. Which leads us to…

The Flaw

There is one major flaw - it’s too manual. But that shouldn’t stop us! It’s just another opportunity to write some code. This way we can enable webhooks on each of our repos, these will ping a small service which will, in turn, open a new issue in the TODO repo. This can be done with cloud functions or on a server. I’ll probably do a server version but, that can be easily converted to a cloud function I believe. This should be a nice refresher for some of the API stuff we worked on in the past.

Once this is setup I expect that the system could actually work - at least for me.

Example JSON Payload:

{
  "title":"read PCG in game design",
  "body":"it's on the kindle should probably read it\n\nshindakun/reading#1",
}

It will need to be added to the TODO kanban board as well. That should just be a matter of using details from the newly created issue. I’ll worry about that once we get the first part working.

I’ve been trying out the manual setup and it kind of works. For non-code tasks, I can close an issue then click through to the source repo and close that. That can probably be automated as well. For code related TODOs I can simply use the closes shindakun/todo#1 shorthand to close the TODO issue and closes #2 shorthand to close the “local” issue.

Next time

Just us next time as we embark on the most “over-engineered” TODO system I could come up with. We’ll be writing the first part of the code portion of this project - a service to accept webhooks from GitHub.

Enjoy this post?
How about buying me a coffee?

Hello World!

Sat, 01 Jun 2019 00:00:00 +0000

👋

Just getting ready to launch my Hugo powered blog.

package main

import "fmt"

func main() {
     fmt.Println("Hello World!")
}

Notes

I suppose I should make some notes on what I’m doing. This likely won’t be complete but should capture all the major steps.

First, I dug through the Hugo site looking for a good starter theme. I eventually decided on Solar a Hugo riff on a Ghost theme.

I’m planning on using them as a starting point and will be expanding it out in an off-shoot theme. I don’t think it will be too crazy, just changes to page headers and footers maybe.

I hadn’t updated my local Hugo install in a while so I grabbed it via go get.

go get github.com/gohugoio/hugo

Next, I made sure all the modules were installed. I’m on my Windows machine at the moment which doesn’t have Go modules enabled by default as of yet so we turn that on.

GO111MODULE=on go mod tidy

Now to create the base site and grab the theme.

hugo site new shindakun.dev
cd shindakun.dev
git submodule add https://github.com/bake/solar-theme-hugo

I then created my sub-theme “Solara”, which is in a private repo for now and added it to my site.

git submodule add https://github.com/shindakun/solara-hugo

We need to add both themes to the config so we edit config.toml and add the line:

theme = ["solara-hugo-test", "solar-theme-hugo"]

This will load my modifications first and then fall back to the basic theme. I imagine that at some point mine will be customized enough that I can remove the base.

Quick Updates

I wanted to make a few quick changes but didn’t want to think to hard so I borrowed some ideas for a few things from another Hugo theme Binario. This is mostly so I can get a handle on the best set up for a theme and make some tweaks pretty quickly. I borrowed Binario’s 404 page and its basic 404 CSS. This also allows me to test including a custom CSS file just for Solara. It works exactly as expected so, that’s nice.

Post Content

I’m thinking I’ll have content/posts as a ~~submodule~~ separate repo so I can keep that repo separate. I have been toying with a new method of working on posts and that may fit in nicely. It also keeps much of the raw content on its own so it can basically be dropped in anywhere.

Install

Clone shindakun.dev repo
Install submodules, Solar and Solara
Clone posts repo

Build

The site will be living on a Digital Ocean server and will be deployed from there.

hugo -d /path/to/site

No fancy deploy process for now. I’ll probably over engineer some sort of CI thing. Or maybe a commit webhook that will fetch and build the latest site.

Enjoy this post?
How about buying me a coffee?

Better Static Glitch Sites

Tue, 23 May 2017 20:33:44 -0700

Here’s a small tip that comes out of the Glitch support forums. By default Glitch will start a static web server on your ./app/ directory once it does not find a package.json file in the root of the app folder. This is not always the desired result - for instance what if the site generation is kicked off using grunt (or one of the countless other build tools) and you can’t remove package.json? In a case like this, the start script in your package.json may be something related to that build or attempting to start a static server of it’s own. Take the following snippet:

"scripts": {
  "build": "grunt build",
  "start": "grunt connect:server:keepalive",
  "test": "grunt test",
  "postinstall": "make build"
}

Grunt is going try and start it’s own static server but, on Glitch we’re likely not able to reach that from the internet at large.

What can we do instead? Well, as luck would have it Glitch uses local-web-server as it’s static web server. This allows us leave the package.json in place - so the default server won’t start but, by updating the start command to be

"start": "ws --port 3000 --directory './' --forbid '/.env' --forbid '/.data' --log-format combined",

we’ll start the local-web-server on our own and leaving it accessible to the public. Make sure you do not leave off the forbid switches if you use .env or .data, it would not do to have those leak.

Finally, if your static site needs to start in a different directory just update the directory switch, for example --directory './www' and you should be good to go.

Enjoy this post?
How about buying me a coffee?