Setup vmoperator with only vmagent and vmservicescrape

The vmoperator streamlines the deployment and management of vmagent on Kubernetes, optimizing for ease of use while retaining native configuration options inherent to Kubernetes environments. It achieves this by introducing various custom resource definitions (CRDs) into the Kubernetes ecosystem and send metrics semalessly to Last9.

Custom Resource Definitions (CRDs)

vmagent
vmnodescrapes
vmservicescrapes
vmprobes
vmpodscrapes
vmstaticscrapes

These CRDs empower users to effortlessly create and manage vmagent instances along with scrape configurations like VMServiceScrape, VMPodScrape, etc. These configurations closely resemble the Prometheus Operator’s ServiceMonitor and PodMonitor. This eliminates the need for manual setup of vmagent deployment, image configuration, and other intricate details.

Prerequisites

Make sure you have the following prerequisites installed:

kubectl - Kubernetes command-line tool

Clone this repository to your local machine:

git clone https://github.com/last9/vmagent-operator-levitate.git
cd vmagent-operator-levitate

Install Custom Resource Definitions (CRDs)

Navigate to the crds/ directory:
Terminal window
```
cd ./crds/
```
Install the CRDs using kubectl:
Terminal window
```
$ kubectl apply -f ./crd.yaml
```

Verify that the CRDs are successfully installed:

$ kubectl get crd --sort-by=.metadata.creationTimestamp
# Output
NAME                                                         CREATED AT
vmagents.operator.victoriametrics.com                        2023-12-26T12:02:48Z
vmnodescrapes.operator.victoriametrics.com                   2023-12-26T12:02:49Z
vmservicescrapes.operator.victoriametrics.com                2023-12-26T12:02:50Z
vmprobes.operator.victoriametrics.com                        2023-12-26T12:02:50Z
vmpodscrapes.operator.victoriametrics.com                    2023-12-26T12:02:50Z
vmstaticscrapes.operator.victoriametrics.com                 2023-12-26T12:02:51Z

You should see the names of the installed CRDs in the output.

Install vmoperator

Navigate to the operator/ directory:
Terminal window
```
cd ./operator/
```
Install the operator and rbac using kubectl:
Terminal window
```
$ kubectl apply -f ./manager.yaml -f rbac.yaml
```
This creates a separate last9-monitoring namespace to not clash with your existing namespaces.

Verify the status of the operator:

$ kubectl get pods -n monitoring-system
# Output
NAME                           READY   STATUS    RESTARTS   AGE
vm-operator-667dfbff55-cbvkf   1/1     Running   0          101s          2023-12-26T12:02:51Z

You should see the vmoperator pod in running status.

Install vmagent

Navigate to the vmagent/ directory:
Terminal window
```
cd ./vmagent/
```

You will need to obtain your Last9 cluster’s Remote Write URL and its credentials. Here is a quick way to create your cluster and obtain your credentials. Run the below command to list all the placeholder values in this file vmagent.yaml

$ cat ./vmagent.yaml | grep -n "Todo"
# Output
11:  levitate_cluster_username: "<levitate_cluster_username>" # Todo: append levitate cluster username
12:  levitate_cluster_password: "<levitate_cluster_password>" # Todo: append levitate cluster password
31:    via_cluster: <via_cluster> # Todo: add a relevant cluster name. e.g: k8s cluster name
33:    - url: <levitate_remote_write_url> # Todo: append levitate remote write URL
55:            storage: 20Gi # Todo: Default is 20Gi. Scale up after you have provisioned more if you need more
58:# Todo: Below configs need to be enabled depending upon your affinity towards nodegroups.
59:# Todo: Ensure that the below selector terms and tolerations are exactly same as the metadata of the nodegroups itself.

Proceed to installation once you have replaced the placeholder values with actual values
Install vmagent in the last9-monitoring namespace using kubectl:
Terminal window
```
$ kubectl apply -f ./vmagent.yaml -n last9-monitoring
```

Verify the status of the vmagent and ensure that it’s running:

$ kubectl get pods -n last9-monitoring -l "last9_monitoring_agent=vmagent"
# Output
NAME                            READY   STATUS    RESTARTS   AGE
vmagent-demo-6785f7d7b9-zpbv6   2/2     Running   0          72s

Install VMServiceScrape (i.e ServiceMonitor and PodMonitor)

Navigate to the vmservicescrape/ directory:

```bash
cd ./vmservicescrape/
```

Caveats

VMServiceScrape works similar to Prometheus Operator’s ServiceMonitor and PodMonitor where you can define scrape selectors to do service and pod discovery.

In this file vmservicescrape.yaml you can override default scrape selectors to suit your requirements. By default this assumes the default labels that are applied to the exporter as part of their installations.

Below is the generic command to find you the labels of your K8s Services.

```bash
$ kubectl get services -n <namespace> <service-name> -o jsonpath='{.metadata.labels}'
```

Once, you have inspected the labels for the services that you chose to scrape, you can then proceed to modify this file vmservicescrape.yaml and match the scrape selector labels with the labels of your services.

Another caveat to note here is to ensure that the namespaces are also declared correctly for the scrape selectors to correctly perform service discovery.

This file also includes scrape configs for Common Exporters such as Kafka, Redis, Node, RabbitMQ, Prometheus Pushgateway etc.

Run this command to list all the Todo comments which will guide you to customize this file vmservicescrape.yaml as required.

```bash
$ cat ./vmservicescrape.yaml | grep -n "Todo"
# Output
48:    matchNames: [ "kube-system" ] # Todo: append namespaces here
63:    matchNames: [ "kube-system" ] # Todo: append namespaces here
86:# Todo: Uncomment this if you have custom application enabled svcs running and you want to scrape them
97:#    matchNames: [ ] # Todo: Append more namespaces here
105:#      app: "" # Todo: Append app name label here
107:# Todo: Uncomment this if you have node exporters svcs running and you want to scrape them
118:#    matchNames: [ ] # Todo: Append more namespaces here
129:# Todo: Uncomment this if you have rabbitMQ exporters svcs running and you want to scrape them
140:#    matchNames: [  ] # Todo: Append more namespaces here
151:# Todo: Uncomment this if you have kafka exporters svcs running and you want to scrape them
162:#    matchNames: [  ] # Todo: Append more namespaces here
173:# Todo: Uncomment this if you have redis exporters svcs running and you want to scrape them
184:#    matchNames: [ ] # Todo: Append more namespaces here
195:# Todo: Uncomment this if you have pushgateway svcs running and you want to scrape them
206:#    matchNames: [ ] # Todo: Append more namespaces here
```

Install VMServiceScrape using kubectl:

```bash
$ kubectl apply -f ./vmservicescrape.yaml -n last9-monitoring
# Output
vmservicescrape.operator.victoriametrics.com/last9-vmservicescrape-vmagent-01 created
vmservicescrape.operator.victoriametrics.com/last9-servicescrape-k8s-01 created
vmservicescrape.operator.victoriametrics.com/last9-servicescrape-metrics-server-01 created
```

Troubleshooting

Please get in touch with us on Discord or Email if you have any questions.