gRPC With Go Crash Course – Service Schemas

This article is part of the series gRPC With Go Crash Course

In this article, you will create a gRPC/Protobuf schema for a small micro service system for an online casino. 

The scope of the article is to: 

• Create the service schemas
• Correctly generate the protobuf Go sources
• Implement the gRPC servers/clients for the appropriate microservices
• Integrate the microservices with each other

The Starting Point

To get started, run this command to clone the necessary exercise materials in a convenient folder:

git clone --branch v1-initial-scaffold https://github.com/preslavmihaylov/go-grpc-crash-course

NOTE: If you are using GOPATH mode, you should clone the repo exactly in <your-gopath>/src/github.com/preslavmihaylov/go-grpc-crash-course.

Next up, install the libraries you’ll need (if you haven’t already) by executing this at the root of the repo:

go mod download

This line will install the required dependencies in your $GOPATH.

Your Goal

In this article, we’ll be creating the IDLs we’ll use to generate the interfaces to be implemented by our server and client:

First, take a look around the scaffold and familiarize yourself with what you’re building.

You have three main folders – casino, client, payment_statements

• casino – this contains the source code for a casino microservice. This will contain the bulk of the backend logic.
• client – this is a client application, which connects to the casino service and allows the user to use the application via a CLI interface.
• payment_statements – this contains the source code for a payment statements microservice. It will be solely responsible for generating a user’s payment statement given a list of payments. The casino service will connect to this one and create user payment statements upon request.

All schemas for our services are in the idl directory. This is where you’ll spend most of your time throughout this part of the course.

Finally, you have make_protos.sh, which is a simple bash script which compiles all your protobuf schemas together.

This is a utility script, enabling you to more easily re-compile proto schemas.

If you want to attempt the exercise on your own, read on. Otherwise skip to the Full Walkthrough section.

What you have to do is: 

Step 1. Complete the IDL definitions

Start in the idl directory and create the appropriate schemas (requirements are in the following section). 

Afterwards, generate the protobuf sources by running the make_protos.sh script.
It will output the sources in the gen directory.

The script shouldn’t output any warnings or errors

Step 2. Implement the grpc clients and servers

Now, go to client/commands.go and casino/main.go
add the appropriate imports (you simply have to remove the surrounding comments).

• Next, implement the casinopb.CasinoServer interface in casino/main.go and pb.PaymentStatementsServer in payment_statements/main.go.
  
  These are interfaces implementing the corresponding gRPC server routines (the ones you define in idl/…).
  
  Leave the overriden interface methods with a panic(“not implemented”) for now.
  
• Implement the following functions:
  • setupPaymentStatementsServer func in payment_statements/main.go
    • The server should listen on localhost:10001
  • setupCasinoServer func in casino/main.go
    • The server should listen on localhost:10000
  • setupPaymentStatementsClient func in casino/main.go
    • use grpc.WithInsecure() and grpc.WithBlock() options when creating the connection to specify that you’re not using TLS and you’re making blocking calls
  • setupClient func in client/commands.go
    • use grpc.WithInsecure() option when creating the connection to avoid creating communication credentials

All these functions deal with initializing either a gRPC server or client and should look pretty identical to one another.

Make sure all three applications can be run in this order:

go run payment_statements/*.go
go run client/*.go
go run casino/*.go

In the following section you’ll find specific requirements to understand what you have to do in terms of proto definitions.

And in the section after that, you’ll find a complete walkthrough of the exercise. Use it in case you get stuck!

However, even if you succeed in creating the proto schemas on your own, consult the walkthrough to calibrate the types you decided to create with the types intended for the exercise.

Otherwise, not all instructions in the following exercises will work due to mismatches between your schemas & the exercise ones. 

Finally, if you want to skip this exercise and move on to the next ones, then checkout the branch mentioned at the end of the exercise. You can also use it to compare your solution to mine.

Good luck!

Oh, and one more thing.
For help, refer to the proto3 Language Guide and the gRPC Go Tutorial.

Service Schema Requirements

You are building an online casino application. Here is a breakdown of what the service schemas should contain.

Hint:
When importing types from other files, make sure to prefix the type with the package name. 

E.g. instead of writing User, write types.User if it’s part of common/types.proto

idl/common/types.proto

This file doesn’t have any service definitions, only three types:

• User – message with a single field string id
• Payment – message with fields User user and int32 amount
• PaymentStatement – message with a single field string data

idl/casino/casino.proto

This proto file should contain a single service, named Casino and several defined types.

The types are as follows: 

• Tokens – message with a single field int32 count
• WithdrawRequest – message with fields User user and int32 tokensCnt
• ActionType – enum with values BUY and SELL
• Action – message with fields: 
  • User user 
  • ActionType type 
  • int32 stocksCount
• GambleType – enum with values STOCK_INFO and ACTION_RESULT
• StockInfo – message with fields string name and int32 price
• ActionResult – message with field string msg 
• GambleInfo – message with fields:
  • GambleType type
  • StockInfo info
  • ActionResult result

The service should contain the following routines:

• BuyTokens – accepts a Payment and returns Tokens
• Withdraw – accepts a WithdrawRequest and returns a Payment
• GetTokenBalance – accepts a User and returns Tokens
• GetPayments – accepts a User and returns a stream Payment
• GetPaymentStatement – accepts a User and returns a PaymentStatement
• Gamble – accepts a stream Action and returns a stream GambleInfo

The types User, Payment and PaymentStatement are defined in idl/common/types.proto.

idl/payment_statements/payment_statements.proto

This file has a single service PaymentStatements.

The service contains a single routine CreateStatement which accepts a stream Payment and returns a PaymentStatement.

Both these types are defined in idl/common/types.proto.

Full Walkthrough

Remember! Before going through this walkthrough, make sure you’ve attempted to complete the exercise on your own. 

Refer to it if you get stuck or are interested in how I did it.

No cheating! 🙈🙈🙈

Additionally, before proceeding with compiling the proto schemas, compare your schema definitions with mine to make sure there are no mismatches.

Now that we’ve gotten that out of the way, let’s get started!

Step 1. Create the common types

The proto schema in idl/common/types.proto doesn’t contain any service definition but it exports some types reused across the other services.

Step 2. Create the casino proto schema

Now let’s write the casino schema definition. This is the hardest one as it contains the most routines.

Based on the given requirements, this is the Casino service we’ll define:

Next, come the types.

We’ll start with WithdrawRequest and Tokens:

Then, we have the Action and ActionType enum:

Finally, we have the GambleInfo with all its nested types:

Step 3. Create the payment_statements proto schema

This service is pretty simple, it contains only a single service definition with no types of its own. It also imports the common types.

This is the entire proto file contents:

Step 4. Compile the proto schemas

There is nothing too special to do from your side here, simply run:

./make_protos.sh

What’s interesting is to inspect what the script contains & what’s its output.

Here is the script’s contents:

The first few lines are not interesting. The last three are.

We’re using the protoc command (protoc == protobuf compiler) to generate golang sources from our input .proto schemas.

The -I flag tells the compiler to “look for imports in this folder”.
Without it, import “common/types.proto” will throw an error.

Next is the input file to compile and the go_out and go-grpc_out options, which tells the protoc compiler where to output Golang sources. In our case, it’s the gen directory.

There’s also an additional require_unimplemented_servers=false option, which turns off a feature which allows one to preserve forward compatibility in schema changes. This is done to simplify the task at hand, but you shouldn’t use this flag in production. More info.

The final two options specify that the output files should be relative to the current source tree. In example, the output will be in folder gen,
instead of github.com/preslavmihaylov/go-grpc-crash-course/gen.

After running this script, you’ll have some source files available in gen/

Check them out & explore a bit!

These are the compiled protobuf schemas. 

The important parts from them are the public interfaces, which can be used in your application to implement a gRPC server/client.

For example, in gen/casino/casino_grpc.pb.go, you can find the CasinoServer interface, which we’ll implement in casino/main.go. 

This interface corresponds to the service we defined in idl/casino/casino.proto:

The other interesting routine is RegisterCasinoServer:

This function will register a casino server implementation (which we’ll write!) and will take care of all the machinery for clients to communicate with your server.

From your application’s perspective, it will seem like some code is invoking normal Go functions inside your codebase. 

All you have to do is implement them. You need not care about how the data is encoded, processed, etc. 

Isn’t that cool? 😎

The rest of the file is not as important, but you’ll have to poke around it throughout the next exercises in order to see what data types you have at your disposal.

Step 5. Configure the payment_statements server

What we’ll do in this step is to setup a gRPC server, implementing the PaymentStatementsServer interface. 

Go to payment_statements/main.go and implement setupPaymentStatementsServer:

What we’re doing here is we first open a TCP socket on the target address we want the service to be running on (in our case – localhost:10001).

Next, we initialize a new gRPC server and register an instance implementing the payment_statements.PaymentStatementsServer interface.

This is the service we defined in our proto schemas.

Finally, we return the new server & tcp socket to be used in the main func.

Next, implement the payment_statements.PaymentStatementsServer interface for our server struct:

Test it out by running it and make sure there are no compilation errors.

go run payment_statements/*.go

Step 6. Configure the casino server

Now, we’ll setup a gRPC server, implementing the CasinoServer interface. 

We’ll also setup a PaymentStatements client in order to be able to communicate to the payment_statements service, which we already configured.

First, uncomment the protobuf imports in casino/main.go:

Next, implement the setupCasinoServer function:

Everything we’re doing here is pretty similar to the configuration code we wrote for payment_statements.

Afterwards, implement the setupPaymentStatementsClient function:

The grpc options we specify are WithInsecure, meaning that we won’t be using TLS for communication and WithBlock meaning that the calls to the payment statements service will be a blocking call (rather than an async one).

Side note:
you can use TLS communication for your gRPC services if you want. 

However, you can instead rely on encryption on an upper layer in your services stack – e.g. you can utilise TLS encryption when you use Consul for service discovery.

With the next line, we dial the real service on a target address:port and finally, initialize a new gRPC client we can use in our codebase.

If you do these steps so far, you shouldn’t have any compilation errors other than the one saying that your casinoServer instance doesn’t implement the casinopb.CasinoServer interface. 

This is expected and it’s what we’re going to do next. Simply define all the interface methods and leave them unimplemented for now:

That finishes the casino server initial setup. Try running it:

go run casino/*.go

If it did run successfully with no compilation errors, you’re good so far!

Step 7. Configure the client

Now, let’s do the same client setup code we did for the payment_statements.PaymentStatementsClient in step 5.

Go to client/commands.go and implement the setupClient func:

Test it out by running it. There shouldn’t be any compilation errors.

Step 8. Final integration test

Now that you’ve setup all your servers & clients, it’s time to test if they correctly integrate with each other.

Run these commands in separate terminal tabs:

go run payment_statements/*.go
go run casino/*.go
go run client/*.go

If everything went well, you should see this message from the casino server:

And this one from the client:

If you got these, then well done! 

Finale

Congratulations. 👏👏👏

You’ve successfully completed this part of the course. 

You should now have a good understanding of how to create Protobuf schemas, compile them & boot up the gRPC servers/clients in your code.

But that’s there’s so much more we have to cover.

How do you actually implement the server routines?
How do you interact with them from your client application?

What’s a client-side/server-side/bidirectional stream?

Don’t worry, we’ll explore that in the following articles, so stay tuned. 🙌🙌🙌
If you want to view the final solution for this exercise, checkout the v2-initial-integration branch.

Go to Part 2 – Basic RPC Routines