Entangle CRDs

Inter-connecting Kubernetes clusters without the need of exposing any service to the public via E2E P2P encrypted networks.

Kairos has two Kubernetes Native extensions ( entangle and entangle-proxy ) that allows to interconnect services between different clusters via P2P with a shared secret.

The clusters won’t need to do any specific setting in order to establish a connection, as it uses libp2p to establish a connection between the nodes.

Entangle can be used to connect services running on different clusters or can be used with entangle-proxy to control another cluster remotely via P2P.

Prerequisites

To entangle two or more clusters you need one or more Kubernetes cluster; entangle depends on cert-manager:

kubectl apply -f https://github.com/jetstack/cert-manager/releases/latest/download/cert-manager.yaml
kubectl wait --for=condition=Available deployment --timeout=2m -n cert-manager --all
  • entangle needs to run on all the clusters that you wish to interconnect. It provides capabilities to interconnect services between clusters
  • entangle-proxy only on the cluster that you wish to use as control cluster

Install the CRD and entangle

First, add the kairos helm repository:

helm repo add kairos https://kairos-io.github.io/helm-charts
helm repo update

Install the CRDs with:

helm install kairos-crd kairos/kairos-crds

Install entangle:

helm install kairos-entangle kairos/entangle
## To use a different image:
## helm install kairos-entangle kairos/entangle --set image.serviceTag=v0.18.0 --set image.tag=latest

Install entangle-proxy

Now install entangle-proxy only on the cluster which is used to control, and which dispatches manifests to downstream clusters.

helm install kairos-entangle-proxy kairos/entangle-proxy

Controlling a remote cluster

control

To control a remote cluster, you need a cluster where to issue and apply manifest from (the control cluster, where entangle-proxy is installed) and a cluster running entangle which proxies kubectl with a ServiceAccount/Role associated with it.

They both need to agree on a secret, which is the network_token to be able to communicate, otherwise it won’t work. There is no other configuration needed in order for the two cluster to talk to each other.

Generating a network token

Generating a network token is described in the p2p section

Managed cluster

The cluster which is the target of our manifests, as specified needs to run a deployment which entangles kubectl:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  namespace: default
type: Opaque
stringData:
  network_token: YOUR_NETWORK_TOKEN_GOES_HERE
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: entangle
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: entangle
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - create
  - delete
  - get
  - list
  - update
  - watch

- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
---
apiVersion: v1
kind: List
items:
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: entangle
    subjects:
    - kind: ServiceAccount
      name: entangle
      namespace: default
    roleRef:
      kind: ClusterRole
      name: entangle
      apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: agent-proxy
  name: agent-proxy
  namespace: default
spec:
  selector:
    matchLabels:
      app: agent-proxy
  replicas: 1
  template:
    metadata:
      labels:
        app: agent-proxy
        entanglement.kairos.io/name: "mysecret"
        entanglement.kairos.io/service: "foo"
        entanglement.kairos.io/target_port: "8001"
        entanglement.kairos.io/direction: "entangle"
    spec:
      serviceAccountName: entangle
      containers:
        - name: proxy
          image: "quay.io/kairos/kubectl"
          imagePullPolicy: Always
          command: ["/usr/bin/kubectl"]
          args:
            - "proxy"

Note: replace YOUR_NETWORK_TOKEN_GOES_HERE with the token generated with the kairos-cli.

Control

To control, from the cluster that has entangle-proxy installed we can apply:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  namespace: default
type: Opaque
stringData:
  network_token: YOUR_NETWORK_TOKEN_GOES_HERE
---
apiVersion: entangle-proxy.kairos.io/v1alpha1
kind: Manifests
metadata:
  name: hello
  namespace: default
  labels:
   entanglement.kairos.io/name: "mysecret"
   entanglement.kairos.io/service: "foo"
   entanglement.kairos.io/target_port: "9090"
spec:
   serviceUUID: "foo"
   secretRef: "mysecret"
   manifests:
   - |
      apiVersion: v1
      kind: Pod
      metadata:
        name: test
        namespace: default
      spec:
            containers:
            - name: hello
              image: busybox:1.28
              command: ['sh', '-c', 'echo "Hello, ssaa!" && sleep 3600']
            restartPolicy: OnFailure      

Note: replace YOUR_NETWORK_TOKEN_GOES_HERE with the token generated with the kairos-cli and used in the step above.

Expose services

The entangle CRD can be used to interconnect services of clusters, or create tunnels to cluster services.

  • Can inject a sidecar container to access a remote services exposed
  • Can create a deployment which exposes a remote service from another cluster

Deployment

entangle can be used to tunnel a connection or a service available from one cluster to another.

entangle-A In the image above, we can see how entangle can create a tunnel for a service running on Cluster A and mirror it to to Cluster B.

It can also expose services that are reachable from the host Network: entangle-B

Consider the following example that tunnels a cluster 192.168.1.1:80 to another one using an Entanglement:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  namespace: default
type: Opaque
stringData:
  network_token: _YOUR_SECRET_GOES_HERE_
---
apiVersion: entangle.kairos.io/v1alpha1
kind: Entanglement
metadata:
  name: test2
  namespace: default
spec:
   serviceUUID: "foo2"
   secretRef: "mysecret"
   host: "192.168.1.1"
   port: "80"
   hostNetwork: true
---
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  namespace: default
type: Opaque
stringData:
  network_token: _YOUR_SECRET_GOES_HERE_
---
apiVersion: entangle.kairos.io/v1alpha1
kind: Entanglement
metadata:
  name: test3
  namespace: default
spec:
   serviceUUID: "foo2"
   secretRef: "mysecret"
   host: "127.0.0.1"
   port: "8080"
   inbound: true
   serviceSpec:
    ports:
    - port: 8080
      protocol: TCP
    type: ClusterIP

Sidecar injection

The controller can inject a container which exposes a connection (in both directions):

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  namespace: default
type: Opaque
stringData:
  network_token: _YOUR_SECRET_GOES_HERE_
---
apiVersion: v1
kind: Pod
metadata:
  name: hello
  namespace: default
  labels:
   # Here we use the labels to refer to the service on the network, and the secret which contains our network_token
   entanglement.kairos.io/name: "mysecret"
   entanglement.kairos.io/service: "foo"
   entanglement.kairos.io/target_port: "9090"
spec:
      containers:
      - name: hello
        image: busybox:1.28
        command: ['sh', '-c', 'echo "Hello, Kubernetes!" && sleep 3600']
      restartPolicy: OnFailure

Or we can combine them together:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  namespace: default
type: Opaque
stringData:
  network_token: _YOUR_SECRET_GOES_HERE_
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: entangle-proxy
  name: entangle-proxy
  namespace: default
spec:
  selector:
    matchLabels:
      app: entangle-proxy
  replicas: 1
  template:
    metadata:
      labels:
        app: entangle-proxy
        entanglement.kairos.io/name: "mysecret"
        entanglement.kairos.io/service: "foo"
        entanglement.kairos.io/target_port: "8001"
        entanglement.kairos.io/direction: "entangle"
      name: entangle-proxy
    spec:
      containers:
        - name: proxy
          image: "quay.io/mudler/k8s-resource-scheduler:latest"
          imagePullPolicy: Always
          command: ["/usr/bin/kubectl"]
          args:
            - "proxy"
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  namespace: default
type: Opaque
stringData:
  network_token: _YOUR_SECRET_GOES_HERE_
---
apiVersion: entangle.kairos.io/v1alpha1
kind: Entanglement
metadata:
  name: test
  namespace: default
spec:
   serviceUUID: "foo"
   secretRef: "mysecret"
   host: "127.0.0.1"
   port: "8080"
   inbound: true
   serviceSpec:
    ports:
    - port: 8080
      protocol: TCP
    type: ClusterIP

Last modified February 23, 2024: Reduce sizes and remove warnings (0e183ae)