Migrating Jenkins from EC2 to Kubernetes: A Comprehensive Guide

In today’s cloud-native world, containerization has become the go-to approach for deploying applications. This blog post details our journey of migrating Jenkins from a traditional EC2 setup to Kubernetes, offering improved scalability, reliability, and easier maintenance.

All the code referenced in this blog post is available in my GitHub repository: jenkins-eks-migration

Why Migrate to EKS?

Before diving into the technical details, let’s understand why migrating Jenkins to EKS makes sense:

High Availability: EKS provides built-in redundancy and automatic pod scheduling
Better Resource Utilization: Kubernetes efficiently manages compute resources
Infrastructure as Code: Everything is defined in YAML, making it version-controllable
Simplified Scaling: Easy to scale Jenkins based on workload demands
Reduced Maintenance: AWS manages the Kubernetes control plane

Prerequisites

Before starting the migration, ensure you have:

A running EKS cluster
kubectl CLI configured with cluster access
Backup of your existing Jenkins data
Basic understanding of Kubernetes concepts

Migration Process

1. Setting Up the Kubernetes Infrastructure

First, create a dedicated namespace for Jenkins:

kubectl create namespace jenkins

2. Configuring Persistent Storage

Jenkins requires persistent storage to maintain its data. We use an EBS-backed PersistentVolumeClaim:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pvc
  namespace: jenkins
spec:
  storageClassName: ebs-csi
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi

Apply the PVC configuration:

kubectl apply -f pvc.yaml

3. Creating the Jenkins Deployment

Here’s the complete deployment configuration that defines our Jenkins pod:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: jenkins
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      serviceAccountName: jenkins-sa
      initContainers:
        - name: rw-permissions
          image: busybox
          command: ["sh", "-c", "chown -R 1000:1000 /var/jenkins_home"]
          volumeMounts:
            - name: jenkins-home
              mountPath: /var/jenkins_home
      containers:
        - name: jenkins
          image: jenkins/jenkins:2.479.3-lts
          ports:
            - name: httpport
              containerPort: 8080
            - name: jnlpport
              containerPort: 50000
          livenessProbe:
            httpGet:
              path: /login 
              port: 8080
            initialDelaySeconds: 90
            periodSeconds: 30
            successThreshold: 1
          readinessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 10
            failureThreshold: 3 
          volumeMounts:
            - name: jenkins-home
              mountPath: /var/jenkins_home
          resources:
            requests:
              cpu: "2" 
              memory: "2G" 
            limits:
              cpu: "4" 
              memory: "4G"   
      volumes:
        - name: jenkins-home
          persistentVolumeClaim:
            claimName: jenkins-pvc
      nodeSelector:
        app: jenkins

Apply the deployment:

kubectl apply -f deployment.yaml

4. Setting Up Network Access

We configure two essential networking components:

ClusterIP Service for internal communication:

apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: jenkins
spec:
  selector:
    app: jenkins
  ports:
    - protocol: TCP
      name: httpport
      port: 8080
      targetPort: 8080
    - name: jnlpport
      port: 50000
      targetPort: 50000
  type: ClusterIP

Apply the service:

kubectl apply -f svc.yaml

ALB Ingress for external access:

The Ingress resource is crucial for exposing Jenkins to external users. Here’s why we need it:

Provides a secure HTTPS endpoint using AWS Certificate Manager
Enables external access to Jenkins through Application Load Balancer
Allows custom domain routing with host rules
Handles SSL/TLS termination
Provides better monitoring and access logging capabilities

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jenkins
  namespace: jenkins
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/group.name: pa-dev
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:<acount_id>:certificate/<certificate_id>
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
spec:
  ingressClassName: alb
  rules:
    - host: jenkins.test.com
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: jenkins
              port:
                number: 8080

Apply the ingress:

kubectl apply -f ingress.yaml

5. Creating Service Account

To allow Jenkins to interact with AWS resources:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-sa
  namespace: jenkins
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<acount_id>:role/jenkins

Apply the service account:

kubectl apply -f service-account.yaml

6. Data Migration Strategy

The migration of Jenkins data requires careful handling. Here’s our approach:

Selective Backup: Exclude these directories to prevent conflicts:
- config.xml
- users/
- plugins/
Data Transfer:

kubectl cp /home/ec2-user/jenkins-bkp/. <pod_name>:/var/jenkins_home/ \
  --exclude=config.xml \
  --exclude=users \
  --exclude=plugins -n jenkins

7. Post-Migration Configuration

After migrating the data:

Update Jenkins URL in system configuration
Install required plugins through the Jenkins UI
Configure authentication and security settings
Verify Jenkins slave connectivity

Best Practices and Lessons Learned

Resource Management:
- Set appropriate resource requests and limits
- Monitor resource usage patterns
- Use node selectors for specific workload placement
Security Considerations:
- Use ServiceAccount with minimal permissions
- Enable HTTPS through ALB
- Implement network policies
Backup Strategy:
- Regular backup of Jenkins home directory
- Version control for configuration files
- Document restoration procedures
Monitoring and Maintenance:
- Set up proper liveness and readiness probes
- Implement monitoring for both Jenkins and Kubernetes metrics
- Regular updates for Jenkins and plugins

Common Challenges and Solutions

Permission Issues:
- Solution: Use init container to set correct permissions
initContainers: - name: rw-permissions image: busybox command: ["sh", "-c", "chown -R 1000:1000 /var/jenkins_home"] volumeMounts: - name: jenkins-home mountPath: /var/jenkins_home
Plugin Compatibility:
- Solution: Fresh plugin installation rather than migration
- Test plugin compatibility in a staging environment
Job Migration:
- Solution: Selective migration of job configurations
- Update build agent configurations for Kubernetes

Conclusion

Migrating Jenkins to EKS represents a significant improvement in our CI/CD infrastructure. While the process requires careful planning and execution, the benefits of improved scalability, reliability, and maintainability make it worthwhile.

The containerized Jenkins deployment provides us with a more robust and scalable CI/CD platform, better aligned with modern cloud-native practices. Regular monitoring and maintenance ensure optimal performance and security of the setup.

For the complete code and configuration files, visit our GitHub repository: jenkins-eks-migration

Remember to adjust the configurations based on your specific needs and always test the migration process in a staging environment first.

Write a comment

From EC2 to Kubernetes: A Practical Guide to Jenkins Migration