Secure Credential Management on GCP with HashiCorp Vault and Jenkins Pipelines

This blog is collaborative work of Samirsaha and Ashutosh Singh

When it comes to managing secrets and credentials, HashiCorp Vault is one of the most popular and reliable tools in the market. With its flexible architecture, it can be used for managing any type of secret, whether it’s API keys, passwords, certificates, or anything else. In this blog, we’ll explore how to set up HashiCorp Vault on GCP using Jenkins pod deployed on GKE, and pipelines for vault renew, set credentials, delete credentials, and vault installation with parameters to add cred, deleted cred, use Cloud Source Repo as git provider, and have every command and script used.

Using HashiCorp Vault from the UI or CLI can be quite complex, and it requires a certain level of expertise to use effectively. Additionally, granting access to Vault to everyone in an organization can create security risks, as users may accidentally or intentionally expose sensitive data. Using Jenkins to manage HashiCorp Vault provides a centralized and secure solution that allows administrators to grant access to Vault on a per-user or per-team basis. By utilizing Jenkins pipelines, administrators can automate and simplify common Vault operations such as unsealing, setting credentials, and deleting credentials. This helps to ensure that Vault is being used securely and effectively without exposing sensitive data to unnecessary risks.

Prerequisites:

Before we dive into the steps, make sure that you have the following prerequisites:

• A GCP account with billing enabled
• A project on GCP
• Basic knowledge of Kubernetes, GKE, and Jenkins
• A GitHub or Cloud Source Repo account
• HashiCorp Vault CLI installed on your local machine

Step 1: Create a GKE cluster

1. Open the Cloud Console:

gcloud init

2. Go to the GKE page:

gcloud services enable container.googleapis.com

3. Create a GKE cluster by running the following command:

gcloud container clusters create [CLUSTER_NAME] \
    --num-nodes=[NUM_NODES] \
    --machine-type=[MACHINE_TYPE] \
    --zone=[ZONE] \
    --scopes=[SCOPES] \
    --enable-autoscaling \
    --max-nodes=[MAX_NODES] \
    --min-nodes=[MIN_NODES] \
    --enable-stackdriver-kubernetes \
    --enable-network-policy \
    --enable-ip-alias \
    --enable-autorepair \
    --enable-autoupgrade

Replace the variables with appropriate values:

• [CLUSTER_NAME]: The name of the cluster you want to create.
• [NUM_NODES]: The number of nodes to create in the cluster.
• [MACHINE_TYPE]: The type of machine to use for each node.
• [ZONE]: The zone in which to create the cluster.
• [SCOPES]: Comma-separated list of GCP scopes to be granted to the cluster nodes.
• [MAX_NODES]: The maximum number of nodes in the cluster.
• [MIN_NODES]: The minimum number of nodes in the cluster.

Step 2: Deploy Jenkins pod on GKE

1. Open the Cloud Console.

2. Go to the Kubernetes Engine page.

gcloud container clusters get-credentials [CLUSTER-NAME] --zone [ZONE] --project [PROJECT-ID]

3. Create a Kubernetes secret for Jenkins credentials.

kubectl create secret generic jenkins --from-file=jenkins-admin-password=./password.txt --namespace jenkins

4. Create a Kubernetes ConfigMap to store the Jenkins configuration file.

kubectl create configmap jenkins --from-file=jenkins.yaml=./jenkins.yaml --namespace jenkins

5. Create a Kubernetes Deployment for Jenkins.

kubectl apply -f jenkins-deployment.yaml --namespace jenkins

6. Create a Kubernetes Service for Jenkins.

kubectl apply -f jenkins-service.yaml --namespace jenkins

7. Verify that the Jenkins pod is running.

kubectl get pods --namespace jenkins

Step 3: Install HashiCorp Vault on GKE

1. Create a Kubernetes YAML file for deploying HashiCorp Vault on GKE:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault
  labels:
    app: vault
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vault
  template:
    metadata:
      labels:
        app: vault
    spec:
      containers:
        - name: vault
          image: vault
          imagePullPolicy: Always
          args:
            - "server"
            - "-dev-listen-address=0.0.0.0:8200"
          ports:
            - containerPort: 8200
              name: http
          volumeMounts:
            - name: vault-data
              mountPath: /vault/data
      volumes:
        - name: vault-data
          persistentVolumeClaim:
            claimName: vault-claim

2. Use the kubectl apply command to deploy the YAML file:

kubectl apply -f [PATH-TO-YAML-FILE]

Replace [PATH-TO-YAML-FILE] with the actual path to the YAML file you created in step 1.

3. Verify that HashiCorp Vault is deployed successfully:

kubectl get pods

This will give you a list of all the pods running on your GKE cluster. Look for the vault pod and make sure that it has a status of Running.

Step 4: Set up Jenkins pipelines for HashiCorp Vault

1. Create a Jenkins pipeline and script for Vault installation

First, we need to write a Jenkins pipeline and script to install HashiCorp Vault on the GKE cluster. Here is an example pipeline script:

pipeline {
    agent {
        kubernetes {
            yaml """
            apiVersion: v1
            kind: Pod
            metadata:
              name: vault-installation
            spec:
              containers:
              - name: vault
                image: hashicorp/vault:1.6.3
                command: [ "/bin/sh", "-c" ]
                args:
                  - "vault server -dev"
                ports:
                - containerPort: 8200
            """
        }
    }
    stages {
        stage(‘Install Vault’) {
            steps {
                sh "curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/"
                sh 'sleep 30s'
                sh 'kubectl port-forward pods/vault-installation 8200:8200 &'
                sh 'sleep 10s'
                sh 'export VAULT_ADDR=http://127.0.0.1:8200'
                sh 'vault status'
            }
        }
    }
}

This script uses a Kubernetes YAML file to create a pod with a single container running the HashiCorp Vault image. We also set the server.dev.enabled=true flag to enable the development mode, which generates a new root token and unseals the Vault automatically. The server.image.repository and server.image.tag flags specify the image to use.

After creating the pod, we use the kubectl port-forward command to forward port 8200 on the pod to a local port on the Jenkins pod. We then set the VAULT_ADDR environment variable to the local URL and verify that Vault is running using the vault status command.

Note that we’ve added some sleep commands to wait for the pod to start up and for the port forwarding to be established before we attempt to interact with Vault. These values may need to be adjusted depending on your specific setup.

2. Create a Jenkins pipeline and script for Vault unseal

Once Vault is installed, we need to unseal it before we can use it. Here is an example pipeline script:

pipeline {
    agent any
    stages {
        stage(‘Unseal Vault’) {
            steps {
                script {
                    sh "curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/"
                    def vaultToken = sh (
                        script: "kubectl -n vault get secret vault-unseal-keys -o jsonpath='{.data.vault-root}' | base64 --decode | vault login -method=token -field=token",
                        returnStdout: true
                    ).trim()
                    sh "echo $vaultToken | vault operator unseal -address=http://vault.vault.svc:8200 -"
                    sh "echo $vaultToken | vault operator unseal -address=http://vault.vault.svc:8200 -"
                    sh "echo $vaultToken | vault operator unseal -address=http://vault.vault.svc:8200 -"
                }
            }
        }
    }
}

Note:-The Vault token is typically stored in the system environment variable VAULT_TOKEN. This allows other processes or scripts to access Vault without requiring the token to be passed directly. The token can also be stored in a file or provided directly as a command line parameter when executing Vault commands.

In this script, we first retrieve the Vault token using kubectl and the vault login command. We then use this token to unseal Vault using the vault operator unseal command.

Note that we’re unsealing Vault three times in this script, as we need to enter three unseal keys to fully unseal Vault.

3. Create a Jenkins pipeline and script for setting credentials with parameters to add credentials

Next, we need to create a pipeline and script to set credentials in Vault. Here is an example pipeline script:

pipeline {
    agent any
    parameters {
        string(name: ‘username’, defaultValue: ‘’, description: ‘The username for the credential’)
        string(name: ‘password’, defaultValue: ‘’, description: ‘The password for the credential’)
    }
    stages {
        stage(‘Set Credential’) {
            steps {
                script {
                    sh "curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/"
                    def username = params.username
                    def password = params.password
                    sh "echo '{\"username\":\"$username\", \"password\":\"$password\"}' | vault kv put secret/my-credential -"
                }
            }
        }
    }
}

In this script, we are using a parameterized pipeline to accept the username and password for the credential. We then use the vault kv put command to set the credential in Vault.

4. Create a Jenkins pipeline and script for deleting credentials with parameters as version of credential:

pipeline {
    agent any
    parameters {
        string(name: ‘VERSION’, description: ‘Version of the credential to delete’)
    }
    stages {
        stage(‘Delete Credential’) {
            steps {
                script {
                    sh "curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/"
                    sh "vault login -method=token token=$VAULT_TOKEN"
                    sh "vault kv metadata delete -address=http://vault.vault.svc:8200 secret/data/creds/${params.VERSION}"
                    sh "vault kv delete -address=http://vault.vault.svc:8200 secret/data/creds/${params.VERSION}"
                }
            }
        }
    }
}

In this script, we are using a parameterized pipeline to accept the version of the credential to delete. We then use the vault kv metadata delete command to delete the metadata associated with the credential, and then the vault kv delete command to delete the actual credential data. We’re also specifying the Vault address in each command using the -address flag to ensure that we’re interacting with the correct Vault instance. Finally, we’re logging in to Vault using the $VAULT_TOKEN environment variable that we set earlier in the Jenkinsfile.

In conclusion, HashiCorp Vault is a powerful tool for managing secrets and sensitive data, and it can be easily deployed on GKE using Jenkins pipelines. By following the steps outlined in this blog, you can install, unseal, set credentials, and delete credentials using a secure and automated process. With the help of Jenkins pipelines, you can streamline your development workflows and ensure that your secrets are always protected. By using the power of GKE and Jenkins, you can easily deploy and manage your HashiCorp Vault instances, and keep your applications secure and reliable.