Golang Data Connector Part 2

Last time we started our small data collector. The idea is we want to move data from a Ghost blog over to something else. This time we're going to extend our starting point to actually make use of the data that comes back from the API call.

Golang Data Connector Part 2

Intro

Last time we started our small data collector. The idea is we want to move data from a Ghost blog over to something else. This time we're going to extend our starting point to actually make use of the data that comes back from the API call. That means we need to take a closer look at the incoming data... and with that in mind let's get started.

Adding Struct-ure

Currently, we are receiving our data and converting it from []byte to string so we can print it to the screen. That's cool and all but not very useful. Instead, we need to see if we can parse the data and make it usable for the next stage of this project.

To do this we'll need to convert the JSON into a struct. OK, so how do we do that? There are a couple of different ways but the easiest I've found is to use the site JSON to Go. This site converts a JSON object into what is a matching Golang struct.

We can simply paste the raw output from the call to https://demo.ghost.io/ghost/api/v4/content/posts/?key=22444f78447824223cefc48062 into the form. It will spit out the exact struct we need:

type AutoGenerated struct {
	Posts []struct {
		ID                   string      `json:"id"`
		UUID                 string      `json:"uuid"`
		Title                string      `json:"title"`
		Slug                 string      `json:"slug"`
		HTML                 string      `json:"html"`
		CommentID            string      `json:"comment_id"`
		FeatureImage         string      `json:"feature_image"`
		Featured             bool        `json:"featured"`
		Visibility           string      `json:"visibility"`
		CreatedAt            time.Time   `json:"created_at"`
		UpdatedAt            time.Time   `json:"updated_at"`
		PublishedAt          time.Time   `json:"published_at"`
		CustomExcerpt        string      `json:"custom_excerpt"`
		CodeinjectionHead    interface{} `json:"codeinjection_head"`
		CodeinjectionFoot    interface{} `json:"codeinjection_foot"`
		CustomTemplate       interface{} `json:"custom_template"`
		CanonicalURL         interface{} `json:"canonical_url"`
		EmailRecipientFilter string      `json:"email_recipient_filter"`
		URL                  string      `json:"url"`
		Excerpt              string      `json:"excerpt"`
		ReadingTime          int         `json:"reading_time"`
		Access               bool        `json:"access"`
		OgImage              interface{} `json:"og_image"`
		OgTitle              interface{} `json:"og_title"`
		OgDescription        interface{} `json:"og_description"`
		TwitterImage         interface{} `json:"twitter_image"`
		TwitterTitle         interface{} `json:"twitter_title"`
		TwitterDescription   interface{} `json:"twitter_description"`
		MetaTitle            interface{} `json:"meta_title"`
		MetaDescription      interface{} `json:"meta_description"`
		EmailSubject         interface{} `json:"email_subject"`
		Frontmatter          interface{} `json:"frontmatter"`
		Plaintext            string      `json:"plaintext,omitempty"`
	} `json:"posts"`
	Meta struct {
		Pagination struct {
			Page  int         `json:"page"`
			Limit int         `json:"limit"`
			Pages int         `json:"pages"`
			Total int         `json:"total"`
			Next  interface{} `json:"next"`
			Prev  interface{} `json:"prev"`
		} `json:"pagination"`
	} `json:"meta"`
}

Let's start by updating AutoGenerated to say Ghost. From there we can add the entire struct to the top of our previous code - right above main(). That is if we want to just added. In our imagined receiving system the entire struct isn't necessary so I've left out a bit. I've also updated several of the interface{} entries to be strings since that's what we'll actually be getting if the value is set.

package main

import (
	"encoding/json"
	"fmt"
	"io"
	"log"
	"net/http"
	"time"
)

type Ghost struct {
	Posts []struct {
		ID                   string      `json:"id"`
		UUID                 string      `json:"uuid"`
		Title                string      `json:"title"`
		Slug                 string      `json:"slug"`
		HTML                 string      `json:"html"`
		FeatureImage         string      `json:"feature_image"`
		Featured             bool        `json:"featured"`
		Visibility           string      `json:"visibility"`
		CreatedAt            time.Time   `json:"created_at"`
		UpdatedAt            time.Time   `json:"updated_at"`
		PublishedAt          time.Time   `json:"published_at"`
		CustomExcerpt        string      `json:"custom_excerpt"`
		URL                  string      `json:"url"`
		Excerpt              string      `json:"excerpt"`
		ReadingTime          int         `json:"reading_time"`
		Access               bool        `json:"access"`
		OgImage              string      `json:"og_image"`
		OgTitle              string      `json:"og_title"`
		OgDescription        string      `json:"og_description"`
		TwitterImage         string      `json:"twitter_image"`
		TwitterTitle         string      `json:"twitter_title"`
		TwitterDescription   string      `json:"twitter_description"`
		MetaTitle            string      `json:"meta_title"`
		MetaDescription      string      `json:"meta_description"`
		Plaintext            string      `json:"plaintext,omitempty"`
	} `json:"posts"`
	Meta struct {
		Pagination struct {
			Page  int         `json:"page"`
			Limit int         `json:"limit"`
			Pages int         `json:"pages"`
			Total int         `json:"total"`
			Next  interface{} `json:"next"`
			Prev  interface{} `json:"prev"`
		} `json:"pagination"`
	} `json:"meta"`
}

func main() {
    
    ...snip...

Enjoy this post?
How about buying me a coffee?

More Printing

Alright, we're slowly getting somewhere! Now, we are going to replace our fmt.Println() with code to actually use our struct. We'll read the body in using the same manner as last time.

	body, err := io.ReadAll(resp.Body)
	if err != nil {
		log.Fatal(err)
	}

But this time instead of printing we'll create a variable p of the type Ghost.  Using p we then json.Unmarshal our received body.

	var p Ghost
	err = json.Unmarshal(body, &p)
	if err != nil {
		log.Fatal(err)
	}

From here we now have access to range over the entirety of p.Posts and once again we print them out to the screen.

	for i := range p.Posts {
		fmt.Printf("%v", p.Posts[i])
	}

Next Time

That's all for this quick post, you can see the full code listing below. Next time we'll be taking one more step forward and actually use the http package to send our posts to another service.

Code Listing

package main

import (
	"encoding/json"
	"fmt"
	"io"
	"log"
	"net/http"
	"time"
)

type Ghost struct {
	Posts []struct {
		ID                   string      `json:"id"`
		UUID                 string      `json:"uuid"`
		Title                string      `json:"title"`
		Slug                 string      `json:"slug"`
		HTML                 string      `json:"html"`
		FeatureImage         string      `json:"feature_image"`
		Featured             bool        `json:"featured"`
		Visibility           string      `json:"visibility"`
		CreatedAt            time.Time   `json:"created_at"`
		UpdatedAt            time.Time   `json:"updated_at"`
		PublishedAt          time.Time   `json:"published_at"`
		CustomExcerpt        string      `json:"custom_excerpt"`
		URL                  string      `json:"url"`
		Excerpt              string      `json:"excerpt"`
		ReadingTime          int         `json:"reading_time"`
		Access               bool        `json:"access"`
		OgImage              string      `json:"og_image"`
		OgTitle              string      `json:"og_title"`
		OgDescription        string      `json:"og_description"`
		TwitterImage         string      `json:"twitter_image"`
		TwitterTitle         string      `json:"twitter_title"`
		TwitterDescription   string      `json:"twitter_description"`
		MetaTitle            string      `json:"meta_title"`
		MetaDescription      string      `json:"meta_description"`
		Plaintext            string      `json:"plaintext,omitempty"`
	} `json:"posts"`
	Meta struct {
		Pagination struct {
			Page  int         `json:"page"`
			Limit int         `json:"limit"`
			Pages int         `json:"pages"`
			Total int         `json:"total"`
			Next  interface{} `json:"next"`
			Prev  interface{} `json:"prev"`
		} `json:"pagination"`
	} `json:"meta"`
}

func main() {
	req, err := http.NewRequest(http.MethodGet, "https://demo.ghost.io/ghost/api/v4/content/posts/?key=22444f78447824223cefc48062", nil)
	if err != nil {
		log.Fatal(err)
	}

	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		log.Fatal(err)
	}

	defer resp.Body.Close()

	body, err := io.ReadAll(resp.Body)
	if err != nil {
		log.Fatal(err)
	}

	var p Ghost
	err = json.Unmarshal(body, &p)
	if err != nil {
		log.Fatal(err)
	}

	for i := range p.Posts {
		fmt.Printf("%v", p.Posts[i])
	}
}