Experiment 2

Preparation

Install application platform

Install platform components:

• Monitoring (Promehteus)
• Logging (Elasticsearch, Filebeat, Kibana)
• Log alerting (Elastalert)
• Service mesh (Istio)
• Chaos testing (Chaos Mesh)
• Root Cause Analysis

Configure OpenRCA

Delete default Prometheus alert rules:

$ kubectl -n monitoring delete prometheusrules $(kubectl -n monitoring get prometheusrules |awk '{print $1}')

Apply custom Prometheus alert rules:

kubectl -n rca apply -f $HOME/Workspace/orca/orca/helm/examples/integrations/prometheus/rules.yaml

Deploy Hipster test app

Create namespace:

$ kubectl create namespace hipster
$ kubectl label namespace hipster istio-injection=enabled

Apply Kubernetes manifests:

$ kubectl -n hipster apply -f $HOME/Workspace/orca/orca-testapps/hipster/kubernetes-manifests.yaml

Apply Istio manifests:

$ kubectl -n hipster apply -f $HOME/Workspace/orca/orca-testapps/hipster/istio-manifests.yaml

Scale Cart Service down to offload database during startup:

$ kubectl -n hipster scale deploy cartservice-v1 --replicas=0

Setup application database

Deploy Percona MongoDB Operator using Helm (cluster-wide mode is not supported):

$ helm install --namespace hipster --create-namespace psmdb-operator percona/psmdb-operator --version 1.9.0 --set nodeSelector.role=exp-subject

Wait until operator is ready:

$ kubectl -n hipster get pods |grep operator

Deploy MongoDB cluster:

$ kubectl -n hipster apply -f $HOME/Workspace/orca/orca-testapps/hipster/mongodb-manifests.yaml

Wait until cluster is ready:

$ watch kubectl -n hipster get pods
$ watch 'kubectl -n hipster describe psmdb mdb |grep State'

Create DB user:

$ kubectl -n hipster exec -it $(kubectl -n hipster get pods |grep mongos |head -n1 |awk '{print $1}') -- mongo -u userAdmin -p userAdmin123456 --authenticationDatabase admin

> use hipster
> db.createUser(
  {
    user: "hipster",
    pwd: "hipster",
    roles: [
      {"role": "readWrite", "db": "hipster"}
    ]
  }
)

Enable sharding:

$ kubectl -n hipster exec -it $(kubectl -n hipster get pods |grep mongos |head -n1 |awk '{print $1}') -- mongo -u clusterAdmin -p clusterAdmin123456 --authenticationDatabase admin

> sh.enableSharding("hipster")

Create collection:

kubectl -n hipster exec -it $(kubectl -n hipster get pods |grep mongos |head -n1 |awk '{print $1}') -- mongo -u hipster -p hipster --authenticationDatabase hipster

> use hipster
> db.createCollection("carts")

Shard the collection:

$ kubectl -n hipster exec -it $(kubectl -n hipster get pods |grep mongos |head -n1 |awk '{print $1}') -- mongo -u clusterAdmin -p clusterAdmin123456 --authenticationDatabase admin

> sh.shardCollection("hipster.carts", {"_id": "hashed"})

Set MongoDB address in app environment:

$ kubectl -n hipster set env deployment/cartservice-v1 "MONGODB_URL=mongodb://hipster:hipster@mdb-mongos.hipster:27017/hipster"

Scale Cart Service back to the original replicas number:

$ kubectl -n hipster scale deploy cartservice-v1 --replicas=1

Cleanup

Delete Hipster test app:

$ kubectl -n hipster delete -f $HOME/Workspace/orca/orca-testapps/hipster/kubernetes-manifests.yaml
$ kubectl -n hipster delete -f $HOME/Workspace/orca/orca-testapps/hipster/istio-manifests.yaml

Delete MongoDB cluster:

$ kubectl -n hipster delete -f $HOME/Workspace/orca/orca-testapps/hipster/mongodb-manifests.yaml

Wait until MongoDB pods are terminated:

$ watch kubectl -n hipster get pod

Delete MongoDB Operator:

$ helm -n hipster delete psmdb-operator

Delete test app namespace:

$ kubectl delete ns hipster