Tit Petric's Blog, page 2

Go: Instrumenting the HTTP service with Elastic APM

Elastic APM is an application performance monitoring product Elastic, the makers of Elasticsearch, and most notably the ELK stack (Elasticsearch, Logstash, Kibana). APM is another plug and play product of theirs, that hooks up to your existing ELK installation and provides endpoints for application agents to receive performance metrics data.

Elastic APM provides a Go agent, which we will use to instrument various parts of our code. The instrumentation will help us not only with performance...

Improving our database handling

As we start to develop real services, we found out that our database handling needs some improvements. We are going to do a series of improvements of the current state by adding some new features, and restructuring some existing ones.

Data source name - DSN

We aren’t doing any kind of data source handling so far, but we need to inject some options into the DSN, like the following:

parseTime=true - needed for decoding date/datetime values into a time.Time, instead of []byte, loc=Local -...

Issuing requests against our microservice

In order to test our microservice, we require a database, running migrations, and our service itself. We’re going to build a docker-compose.yml file which takes care of all of this.

The docker-compose file

In order to use the docker-compose.yml file, first you need to run make && make docker to build the required docker images.

I started with a simple docker-compose.yml file, like this:

version: '3.4' services: stats: image: titpetric/service-stats restart: always environment:...

Go: implementing a microservice

We should move towards writing actual code for our service. We are writing a simple statistics collection service for tracking page views. We are going to create a database schema for collecting page views and improve things along the way.

The database schema

To refresh, we will be implementing our Push() RPC call:

service StatsService { rpc Push(PushRequest) returns (PushResponse); } message PushRequest { string property = 1; uint32 section = 2; uint32 id = 3; } message PushResponse {}

...

Docker: Building images with security in mind

When it comes to running our microservices in production, we need to build docker images. Security is unfortunately an afterthought, so let’s try to figure out what we can do to increase security so it’s better than most of the stuff out there.

Configuring Makefile targets

We will start by adding Makefile rules to enable us to build and push docker images.

# docker image build IMAGE_PREFIX := titpetric/service- docker: $(shell ls -d cmd/* | sed -e 's/cmd\//docker./') @echo OK....

Go: Dependency injection with Wire

Our microservices ultimately need at least a database connection in order to query and store data in a MySQL database. We need to pass this object, possibly from the main() function, to our service implementation.

The Go way

I have learned through trial and error, that “Dependency Injection” is a term which manages to evoke a strong emotional response in gophers. As I’ve said before, all that the term represents in Go is “a way to pass stuff into our handlers”,...

Go: Generating database schema documentation

Since we are testing migrations and generating Go struct types from the database schema, we also have all the available information to generate documentation snippets for each schema. We will generate markdown formatted tables with the final database schema, after all the migrations have been applied.

Updating the Go generators

We need to adjust the generator code, since each table will need to be rendered in an individual markdown file. We need to add a parameter to db-schema-cli, which we...

Go: Database first struct generation

As we set up our protobuf structures and data migrations, we will need to interface between them. Unfortunately, protobuf structs don’t have good support for adding go tags, so we can’t easily pin db tags to it, nor should we really. There’s always going to be some mismatching with what a RPC request/response is, and what’s actually stored in the database.

SQL databases have pretty good support over their information_schema interface. We can list the database tables in...

Go: Testing database migrations with Drone CI

Since we can now list our database migrations, the next logical step is testing them out on a real database. For that we still need to implement our Run function. Based on what we have in Print(), we only need to update the migrate() function.

execQuery := func(idx int, query string, useLog bool) error { if useLog { log.Println() log.Println("-- Statement index:", idx) log.Println(query) log.Println() } if _, err := db.Exec(query); err != nil && err != sql.ErrNoRows { return...

Go: Scaffolding database migrations

As we prepared the database migration files and embedded them into the db package, we are now left to implement the details required to process these migrations. Let’s start with extending the FS type, to actually provide functionality for reading them.

// ReadFile returns decoded file contents from FS func (fs FS) ReadFile(filename string) ([]byte, error) { if val, ok := fs[filename]; ok { return base64.StdEncoding.DecodeString(val) } return nil, os.ErrNotExist }

Reading a file from...