Ivan Font

Pac-Man Application Using Kubernetes Federation

Ivan Font — Sat, 11 Mar 2017 01:33:34 GMT

Pac-Man Microservices Architecture Game Update

Previously I discussed the next phase of the Pac-Man + NGINX + PHP + MongoDB Kubernetes reference application was to deploy the application using a Kubernetes Federation. This was accomplished and you can read about the details here. But in general, federation made it really easy to manage multiple Kubernetes clusters through one pane of glass by deploying the Pac-Man Kubernetes resources onto the clusters that are part of the federation. It automatically added the necessary DNS entries for the services deployed to provide a load balanced application across all of the clusters. This turned the Pac-Man microservices architecture game into a fully scalable high-availability game.

New Features

MongoDB Replica Set

Part of moving to a federation required a data replication strategy so your application's data could scale horizontally as well. This required setting up a MongoDB Replica Set. Unfortunately, MongoDB requires many operations to only be performed on the primary. For example, MongoDB allows reading from many instances, but only the primary can receive writes. In addition, adding replica set members can only be done on the primary. Unfortunately this, combined with the fact that federation at the time of this writing does not yet support things like StatefulSets, involved a bit of manual intervention to bootstrap the MongoDB replica set than I would like.

Pac-Man Zone

Once the application was deployed in the federation, I modified it to add a Zone: field that would retrieve the zone of the Pac-Man game instance you were connecting to via the load balanced DNS. That is, the federated DNS was updated to resolve to the set of IP addresses of the Pac-Man game service in each of the federated zones to provide load balancing. Multiple refreshes of the game would continually update the zone you were connecting to and display that for you above your score. Once you were done playing and saved your high score, the zone information was also saved along with it.

Learnings

At first, I was creating the Kubernetes federation manually but this became cumbersome and time consuming so I moved to use kubefed which really helped speed up the process of creating a federated cluster. Steps for doing that are captured here.

In addition, I learned about some limitations with MongoDB and not having StatefulSets that may pose problems as the application evolves.

Next Steps

I'd like to see about performing coordinated migrations of the application as well as handling failover. What would make those tests more interesting is having a larger volume of transactions happening in the game, such as updating more data points on a regular interval defined in the game, while performing those tests. In order to increase complexity in the game I've contemplated migrating the backend to something more adept at handling that task. In any event, doing failover tests may bump up against limitations of MongoDB and the lack of Statefulsets. Time will tell.

GitHub Link

You can read all about the details of how to set up the Pac-Man microservices architecture game here.

bash-it: an alternative to oh-my-zsh

Ivan Font — Tue, 28 Feb 2017 21:56:50 GMT

Trying out zsh

I have noticed that many people seem to be switching shells to zsh. In fact, I even tried it myself to see what the hype was all about. :)

So for a while there I was actually feeling just as good using zsh as bash. That was until I started discovering some idiosyncrasies with zsh that were different from bash. It appeared that zsh was not quite as backwards compatible with bash. Even though it seems to support the same features, it does so with similar but different syntax. This made it quite painful because every system I work on has bash installed and used by default, but not necessarily zsh. Also, many scripts out there in repositories are all coded to use bash. Now of course you can specify the shebang to make sure the scripts use bash, but why would I want to use a different shell than the scripts or systems I'm working with use themselves? I would then have to learn and maintain scripts for two different shells depending on what I'm working on.

oh-my-zsh

I'll also add that the most promising feature of switching to zsh was the awesome oh-my-zsh GitHub project that gives you a highly customizable framework. This seemed to clearly be lacking in bash and so it was this that made it quite compelling for me at first. In addition, I agree that zsh does have some nice features that work right out of the gate, but it felt like it was more to do with oh-my-zsh than zsh itself. Therefore, nothing was really compelling enough that you couldn't do with bash, albiet with some customizations.

Moving back to bash

bash-it

This is where I stumbled upon bash-it. It literally promotes itself as being a shameless ripoff of oh-my-zsh. Finally, this seemed to be the piece that was missing from bash that one was able to get with zsh quite easily. I immediately started using it and enjoyed bashing again.

bash-it has all of the benefits of a framework with support for plugins, aliases, completions, custom scripts and functions, themes, updates, search, etc. Here is a quick summary of some help options:

bash-it show aliases        # shows installed and available aliases
bash-it show completions    # shows installed and available completions
bash-it show plugins        # shows installed and available plugins
bash-it help aliases        # shows help for installed aliases
bash-it help completions    # shows help for installed completions
bash-it help plugins        # shows help for installed plugins

I even stumbled upon a few issues and feature requests myself, which I proceeded to contribute back as time allowed. I have most recently created and contributed my own font theme. It supports the following prompt features:

time
Python virtual environment
user
host
path
git repo
git branch
git dirty/not dirty
return code status of last command

Here is a screenshot of it in action.

Check it out and let me know how you like it!

Pac-Man NGINX PHP MongoDB Application

Ivan Font — Tue, 07 Feb 2017 00:35:14 GMT

Intro

I've started working on a Kubernetes reference application for demonstration purposes. This reference application will continue to evolve to also work with cloud federation capabilities in Kubernetes. I will provide updates as the application evolves.

Pac-Man Game Architecture Brief Overview

Pac-Man

The Pac-Man game is a slightly modified version of the open source Pac-Man game written in HTML5 with Javascript. You can get the modified Pac-Man game source code here. The modifications are particularly around the backend PHP API. The original game used SQLite3 as the storage backend for reading and writing high scores. This change replaces SQLite3 with MongoDB as the storage mechanism.

NGINX + PHP FPM

NGINX is used as the web server to host the Pac-Man game application. It is configured with PHP FPM support for the backend PHP API.

PHP

PHP FPM is used for the PHP API to receive read and write requests from clients and perform database operations. The PHP MongoDB driver extension contains the minimal API for core driver functionality. In addition, the PHP library for MongoDB provides the higher level APIs. Both are needed.

MongoDB

MongoDB is used as the backend database to store the Pac-Man game's high score user data.

Conclusion

For a detailed overview of the Kubernetes reference application and how to set it up yourself, see the README in the repository.

Have fun!

Deploying Containerized Ceph on a Kubernetes Cluster Using Google Compute Engine

Ivan Font — Fri, 03 Feb 2017 19:38:03 GMT

Quick Summary

Recently we've been hard at work trying to set up dynamic provisioning of Ceph Rados Block Device storage on a Kubernetes cluster. This required:

Kubernetes version >= 1.5
The ability to install Ceph and RBD packages on all host nodes i.e. master and slaves

Unfortunately we could not use Google Container Engine (GKE) for this because at first, it did not support version 1.5 since it was too new. Google quickly resolved that and has been really good at keeping up-to-date with the latest Kubernetes releases. However, the subsequent problem is that we needed to install Ceph and RBD packages on all host nodes. Even though Google does provide a CONTAINER_VM image that is able to install packages, GKE does not provide access to the master to install the necessary packages. This seems like a limitation with GKE. So it was decided to move towards a manual deployment of a Kubernetes cluster. For the manual deployment we used Google Compute Engine and followed the nice example Kelsey Hightower provides:

Kubernetes the Hard Way

At first, I was hitting a DNS problem until I realized that there was a firewall rule missing for the POD CIDR that was being used. See issue 88 and the pull request that resolves it for more details. It was also nice to script the deployment so we created some scripts for that as well but they have not yet been merged. In the meantime you can get them here:

https://github.com/font/kubernetes-the-hard-way/tree/scripts

In the end it was good to understand how the different components of Kubernetes fit together and be able to debug it a bit.

After Kubernetes was deployed we deployed Ceph and went about testing CephFS and RBD volume mounts, static provisioning of CephFS and RBD persistent volume claims, as well as dynamic provisioning of RBD persistent volume claims and they all worked great!

Documentation

If you're interested you can get the detailed documentation here:

https://github.com/ceph/ceph-docker/tree/master/examples/kubernetes/gce

Video Demonstration

In the documentation you'll also find a link to this video demonstration that walks through those steps:

Enjoy!

Creating custom Kubernetes cluster using Google Compute Engine (GCE)

Ivan Font — Fri, 16 Dec 2016 01:11:31 GMT

Overview

If you're looking to get started with Kubernetes, there are 3 ways as of this writing:

Prebuilt Binary Release
Building from source
- Note that if you're cloning Kubernetes from source, you will need to build a release in order to generate the necessary binaries in an archive for deployment. Building can take some time - so take a break.
Download Kubernetes and automatically set up a default cluster

We'll be discussing how to download any version of Kubernetes and automatically set up a default cluster as that's the quickest, most configurable option. Particularly, we'll be focusing on downloading Kubernetes and creating a customized deployment on Google Compute Engine (GCE). However, there are also numerous others e.g. gke, aws, azure, vagrant, etc.

So, if you're looking to create a Kubernetes cluster on Google Compute Engine (GCE), then an easy way to get started is by following the getting-started-guide for GCE which we'll walk through briefly here.

Prerequisites

Make sure to pay attention to the prereqs discussed here as they discuss making sure that you have your Google Cloud Platform setup and configured properly for an automatic deployment. Particularly, you'll want to make sure you can access the Compute Engine Instance Group Manager API, make sure that gcloud is set to use the Google Cloud Platform project you want, and that gcloud has the proper credentials. A good test for this is to run some gcloud commands to test that you can create an instance and SSH into it:

$ gcloud compute instances create example-instance --image-family ubuntu-1604-lts --image-project ubuntu-os-cloud
Created [https://www.googleapis.com/compute/v1/projects/gce-kube-cluster/zones/us-west1-a/instances/example-instance].
NAME              ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
example-instance  us-west1-a  n1-standard-1               10.138.0.2   104.196.227.36  RUNNING
gcloud compute ssh example-instance
Updating project ssh metadata...|Updated [https://www.googleapis.com/compute/v1/projects/gce-kube-cluster].
Updating project ssh metadata...done.
Warning: Permanently added 'compute.2723409803693978954' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-53-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.


To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

test@example-instance:~$

Starting a cluster with default configuration

Now we're ready for an automatic download and deployment of the Kubernetes cluster in our GCE project. Kubernetes allows automatic download and deployment of the cluster onto one of many providers using the KUBERNETES_PROVIDER variable:

$ export KUBERNETES_PROVIDER=gce

If this variable is not specified, it uses gce, Google's Compute Engine, by default.

If you're ready to just accept the default options for instantiating the cluster using the latest version of Kubernetes you can just run:

$ curl -sS https://get.k8s.io | bash

$ wget -q -O - https://get.k8s.io | bash

Starting a cluster with custom configuration

However, sometimes you'd like to download a different version of Kubernetes and/or modify some of the available GCE configuration options depending on your needs. In order to do this, we want to allow the automated download of a specific version of the Kubernetes binaries, but allow us to modify parameters before the automated deployment. This is achieved by downloading the script and saving it to a file:

$ curl -sS https://get.k8s.io -o kube.sh

$ wget -q https://get.k8s.io -O kube.sh

Then setting:

$ export KUBERNETES_RELEASE=v1.5.1
$ export KUBERNETES_SKIP_CREATE_CLUSTER=true

Feel free to set any version of Kubernetes that you'd like. Then run

$ bash kube.sh

After some interactive prompts, which can be avoided by setting KUBERNETES_SKIP_CONFIRM=true, Kubernetes is downloaded and we can start to modify some parameters in the cluster/gce/config-default.sh file.

$ cd kubernetes
$ grep -E '(\w)+=' cluster/gce/config-default.sh
KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../..
GCLOUD=gcloud
ZONE=${KUBE_GCE_ZONE:-us-central1-b}
REGION=${ZONE%-*}
RELEASE_REGION_FALLBACK=${RELEASE_REGION_FALLBACK:-false}
REGIONAL_KUBE_ADDONS=${REGIONAL_KUBE_ADDONS:-true}
NODE_SIZE=${NODE_SIZE:-n1-standard-2}
NUM_NODES=${NUM_NODES:-3}
MASTER_SIZE=${MASTER_SIZE:-n1-standard-$(get-master-size)}
MASTER_DISK_TYPE=pd-ssd
MASTER_DISK_SIZE=${MASTER_DISK_SIZE:-20GB}
NODE_DISK_TYPE=${NODE_DISK_TYPE:-pd-standard}
NODE_DISK_SIZE=${NODE_DISK_SIZE:-100GB}
REGISTER_MASTER_KUBELET=${REGISTER_MASTER:-true}
PREEMPTIBLE_NODE=${PREEMPTIBLE_NODE:-false}
PREEMPTIBLE_MASTER=${PREEMPTIBLE_MASTER:-false}
KUBE_DELETE_NODES=${KUBE_DELETE_NODES:-true}
KUBE_DELETE_NETWORK=${KUBE_DELETE_NETWORK:-false}
MASTER_OS_DISTRIBUTION=${KUBE_MASTER_OS_DISTRIBUTION:-${KUBE_OS_DISTRIBUTION:-gci}}
NODE_OS_DISTRIBUTION=${KUBE_NODE_OS_DISTRIBUTION:-${KUBE_OS_DISTRIBUTION:-debian}}
CVM_VERSION=container-vm-v20161208
GCI_VERSION="gci-dev-56-8977-0-0"
MASTER_IMAGE=${KUBE_GCE_MASTER_IMAGE:-}
MASTER_IMAGE_PROJECT=${KUBE_GCE_MASTER_PROJECT:-google-containers}
NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-${CVM_VERSION}}
NODE_IMAGE_PROJECT=${KUBE_GCE_NODE_PROJECT:-google-containers}
CONTAINER_RUNTIME=${KUBE_CONTAINER_RUNTIME:-docker}
RKT_VERSION=${KUBE_RKT_VERSION:-1.14.0}
RKT_STAGE1_IMAGE=${KUBE_RKT_STAGE1_IMAGE:-coreos.com/rkt/stage1-coreos}
NETWORK=${KUBE_GCE_NETWORK:-default}
INSTANCE_PREFIX="${KUBE_GCE_INSTANCE_PREFIX:-kubernetes}"
CLUSTER_NAME="${CLUSTER_NAME:-${INSTANCE_PREFIX}}"
MASTER_NAME="${INSTANCE_PREFIX}-master"
INITIAL_ETCD_CLUSTER="${MASTER_NAME}"
ETCD_QUORUM_READ="${ENABLE_ETCD_QUORUM_READ:-false}"
MASTER_TAG="${INSTANCE_PREFIX}-master"
NODE_TAG="${INSTANCE_PREFIX}-minion"
MASTER_IP_RANGE="${MASTER_IP_RANGE:-10.246.0.0/24}"
CLUSTER_IP_RANGE="${CLUSTER_IP_RANGE:-10.244.0.0/14}"
    NODE_SCOPES="${NODE_SCOPES:-compute-rw,monitoring,logging-write,storage-ro,https://www.googleapis.com/auth/ndev.clouddns.readwrite}"
    NODE_SCOPES="${NODE_SCOPES:-compute-rw,monitoring,logging-write,storage-ro}"
EXTRA_DOCKER_OPTS="${EXTRA_DOCKER_OPTS:-}"
SERVICE_CLUSTER_IP_RANGE="${SERVICE_CLUSTER_IP_RANGE:-10.0.0.0/16}"  # formerly PORTAL_NET
ALLOCATE_NODE_CIDRS=true
ENABLE_DOCKER_REGISTRY_CACHE=true
ENABLE_L7_LOADBALANCING="${KUBE_ENABLE_L7_LOADBALANCING:-glbc}"
ENABLE_CLUSTER_MONITORING="${KUBE_ENABLE_CLUSTER_MONITORING:-influxdb}"
ENABLE_NODE_LOGGING="${KUBE_ENABLE_NODE_LOGGING:-true}"
LOGGING_DESTINATION="${KUBE_LOGGING_DESTINATION:-gcp}" # options: elasticsearch, gcp
ENABLE_CLUSTER_LOGGING="${KUBE_ENABLE_CLUSTER_LOGGING:-true}"
ELASTICSEARCH_LOGGING_REPLICAS=1
  EXTRA_DOCKER_OPTS="${EXTRA_DOCKER_OPTS} --insecure-registry 10.0.0.0/8"
RUNTIME_CONFIG="${KUBE_RUNTIME_CONFIG:-}"
FEATURE_GATES="${KUBE_FEATURE_GATES:-}"
ENABLE_CLUSTER_DNS="${KUBE_ENABLE_CLUSTER_DNS:-true}"
DNS_SERVER_IP="${KUBE_DNS_SERVER_IP:-10.0.0.10}"
DNS_DOMAIN="${KUBE_DNS_DOMAIN:-cluster.local}"
ENABLE_DNS_HORIZONTAL_AUTOSCALER="${KUBE_ENABLE_DNS_HORIZONTAL_AUTOSCALER:-true}"
ENABLE_CLUSTER_REGISTRY="${KUBE_ENABLE_CLUSTER_REGISTRY:-false}"
CLUSTER_REGISTRY_DISK="${CLUSTER_REGISTRY_PD:-${INSTANCE_PREFIX}-kube-system-kube-registry}"
CLUSTER_REGISTRY_DISK_SIZE="${CLUSTER_REGISTRY_DISK_SIZE:-200GB}"
CLUSTER_REGISTRY_DISK_TYPE_GCE="${CLUSTER_REGISTRY_DISK_TYPE_GCE:-pd-standard}"
ENABLE_CLUSTER_UI="${KUBE_ENABLE_CLUSTER_UI:-true}"
ENABLE_NODE_PROBLEM_DETECTOR="${KUBE_ENABLE_NODE_PROBLEM_DETECTOR:-true}"
ENABLE_CLUSTER_AUTOSCALER="${KUBE_ENABLE_CLUSTER_AUTOSCALER:-false}"
  AUTOSCALER_MIN_NODES="${KUBE_AUTOSCALER_MIN_NODES:-}"
  AUTOSCALER_MAX_NODES="${KUBE_AUTOSCALER_MAX_NODES:-}"
  AUTOSCALER_ENABLE_SCALE_DOWN="${KUBE_AUTOSCALER_ENABLE_SCALE_DOWN:-true}"
ENABLE_RESCHEDULER="${KUBE_ENABLE_RESCHEDULER:-true}"
ADMISSION_CONTROL=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota
KUBE_UP_AUTOMATIC_CLEANUP=${KUBE_UP_AUTOMATIC_CLEANUP:-false}
STORAGE_BACKEND=${STORAGE_BACKEND:-}
NETWORK_PROVIDER="${NETWORK_PROVIDER:-kubenet}" # none, opencontrail, kubenet
OPENCONTRAIL_TAG="${OPENCONTRAIL_TAG:-R2.20}"
OPENCONTRAIL_KUBERNETES_TAG="${OPENCONTRAIL_KUBERNETES_TAG:-master}"
OPENCONTRAIL_PUBLIC_SUBNET="${OPENCONTRAIL_PUBLIC_SUBNET:-10.1.0.0/16}"
NETWORK_POLICY_PROVIDER="${NETWORK_POLICY_PROVIDER:-none}" # calico
HAIRPIN_MODE="${HAIRPIN_MODE:-promiscuous-bridge}" # promiscuous-bridge, hairpin-veth, none
E2E_STORAGE_TEST_ENVIRONMENT=${KUBE_E2E_STORAGE_TEST_ENVIRONMENT:-false}
EVICTION_HARD="${EVICTION_HARD:-memory.available<250Mi,nodefs.available<10%,nodefs.inodesFree<5%}"
SCHEDULING_ALGORITHM_PROVIDER="${SCHEDULING_ALGORITHM_PROVIDER:-}"
ENABLE_DEFAULT_STORAGE_CLASS="${ENABLE_DEFAULT_STORAGE_CLASS:-true}"
SOFTLOCKUP_PANIC="${SOFTLOCKUP_PANIC:-false}" # true, false

You can see there are many options. You can either edit the variables directly in the file or export the environment variable that is used to set the default. Some of the more interesting options include:

ZONE, REGION - for you zone and region snobs. Export the KUBE_GCE_ZONE environment variable to edit these settings.
NODE_SIZE - the default of n1-standard-2 is generally good for my use.
NUM_NODES - the default of 3 nodes plus a master is probably okay for many tests. But for anything with higher demands such as a Ceph cluster, you may want to increase this.
MASTER_OS_DISTRIBUTION and NODE_OS_DISTRIBUTION - This should match the name of the subdirectory within the cluster/gce directory. As of this writing only coreos, debian, gci, and trusty are available. In order to edit these settings you'll want to export KUBE_MASTER_OS_DISTRIBUTION and KUBE_NODE_OS_DISTRIBUTION to one of the available subdirectory options.
- NOTE: it appears that the trusty distribution is not currently working due to this issue.
MASTER_IMAGE, MASTER_IMAGE_PROJECT and NODE_IMAGE, NODE_IMAGE_PROJECT- Default for the master is the gci image while nodes use container-vm, defined by the GCI_VERSION and CVM_VERSION variables respectively. To get a list of available images to use execute gcloud compute images list. To edit these settings export the KUBE_GCE_MASTER_IMAGE, KUBE_GCE_MASTER_PROJECT, KUBE_GCE_NODE_IMAGE, and KUBE_GCE_NODE_PROJECT to match the image name and project that corresponds to the image you want to use.

Edit to your hearts content, then follow it with:

$ ./cluster/kube-up.sh

If things go south or you want to correct some things, ctrl-c the execution and run:

$ ./cluster/kube-down.sh

Make your changes and re-launch.

Conclusion

This is a pretty easy and quick way to get a custom Kubernetes cluster downloaded and deployed onto Google Compute Engine (GCE) without much hassle. This is great if you find that you want a particular version of Kubernetes with the ability to set some GCE configuration options.

Hello World

Ivan Font — Sat, 27 Aug 2016 00:40:01 GMT

Hello world! This is my first post.