Harmony Controller micro-service State

State of the all micro services can be monitored using Operator Console that is shipped with the product. Please refer to the end use documentation of Harmony Controller for more details. Operator console data can also be accessed using RESTful APIs and integrated into any third party enterprise monitoring systems.


Harmony Controller Infrastructure Software (Kubernetes)

Monitoring of infrastructure software consists of monitoring kubernetes basic services and kubernetes objects like clusters, PODs, nodes and deployment.

Monitoring Cluster Status

One can monitor the Kubernetes Cluster status by running following command:

$ kubectl get cs 

Expected Output:
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok                   
controller-manager   Healthy   ok                   
etcd-0               Healthy   {“health”: “true”}

Things to monitor in the output:

  1. All three components are listed
  2. All are in Healthy state

Monitoring System PODS Status

The status of the system Pods created by Kubernetes can be checked using following command:

$ kubectl get po -o wide -n kube-system

Sample Output:
NAME                                   READY     STATUS    RESTARTS   AGE       IP            NODE
dummy-526455539-mz82t                  1/1       Running   0          7h    k-new-master
etcd-k-new-master                      1/1       Running   0          7h    k-new-master
etcd-k-new-slave1                      1/1       Running   0          7h   k-new-slave1
etcd-k-new-slave2                      1/1       Running   0          7h

Things to monitor in the output:

  1. All Pods in “Running” state
  2. All containers of the Pods are in “READY” state

Monitoring Node Status

The status of the nodes in the Kubernetes cluster can be monitored using the following command. This will give you list of nodes in the Kubernetes cluster and their status:

$ kubectl get nodesSample output:
NAME           STATUS         AGE
k-new-master   Ready,master   12h
k-new-slave1   Ready          12h
k-new-slave2   Ready          12h

Things to monitor in the output

  1. The nodes are added as per the requirement
  2. All nodes are in Ready state.

Available Pods Vs Desired Pods

For all the deployments in the Kubernetes one can monitor if the desired number of Pods are available in the deployment of components of Harmony Controller. If the desired number is not available it indicates issue with the component deployment and possibly a non-working setup. You can use following command to do the same:

$ kubectl get deploymentsSample Output:
barista            1         1         1            1           12h
edge               1         1         1            1           12h
message-bus        1         1         1            1           12h
metrics-master     3         3         3            3           12h
operatorconsole    1         1         1            1           12h
persistent-proxy   1         1         1            1           12h
persistent-store   1         1         1            1           12hregistry-server    1         1         1            1           12h
reporting-engine   1         1         1            1           12h
rtp-manager        1         1         1            1           12h
ui                 1         1         1            1           12h

Things to monitor in the output:

  1. The “DESIRED” count for every deployment is as “AVAILABLE”.