Microservices with Kafka and EKS for Shipping Carriers

Building Java Spring Cloud Microservices with Kafka for Shipping Carriers and Deploying on EKS. In today’s digital landscape, microservices architecture has become the de facto standard for building scalable and resilient applications. Java Spring Cloud, combined with Apache Kafka, offers a powerful toolkit for developing microservices that require asynchronous communication, such as those used in shipping carrier systems. When it comes to deploying these microservices at scale, Kubernetes, specifically Amazon Elastic Kubernetes Service (EKS), provides an excellent platform. In this article, we’ll explore how to build Java Spring Cloud microservices using Kafka for shipping carrier systems and how to deploy them on EKS.

Why Use Microservices for Shipping Carriers?

Shipping carriers often deal with complex workflows, including order processing, shipment tracking, and real-time updates from multiple carriers. Microservices architecture is well-suited for these scenarios because it allows developers to break down the application into smaller, manageable services that can be developed, deployed, and scaled independently.

The Role of Kafka in Microservices

Apache Kafka is a distributed streaming platform that excels in handling real-time data feeds. In a microservices architecture, Kafka serves as the backbone for asynchronous communication between services. It ensures that messages (events) are reliably delivered to various microservices, even in the face of failures, making it ideal for shipping systems where real-time updates and resilience are critical.

Key Components of the Architecture

Java Spring Cloud Microservices:

Order Service: Handles order creation and updates, sending shipment requests to the appropriate carrier service.
Carrier Service: Manages interactions with different shipping carriers (e.g., UPS, FedEx), including rate calculations and tracking updates.
Tracking Service: Receives tracking updates from carriers and provides real-time tracking information to customers.

Apache Kafka:

Topics: Kafka topics are used to decouple microservices. For instance, an “order-placed” topic might be used by the Order Service to notify the Carrier Service of new orders.
Producers/Consumers: Each microservice acts as a Kafka producer or consumer, publishing events to or consuming events from topics.

Amazon EKS (Elastic Kubernetes Service):

Kubernetes: Provides orchestration and management of containerized microservices.
EKS: A managed Kubernetes service by AWS, simplifying the deployment, management, and scaling of Kubernetes clusters.

Step-by-Step Guide: Building and Deploying the System

1. Setting Up the Spring Cloud Microservices

Order Service: Start by creating a Spring Boot application for the Order Service. Use Spring Cloud Stream to integrate with Kafka.

  @EnableBinding(OrderProcessor.class)
  @SpringBootApplication
  public class OrderServiceApplication {
      public static void main(String[] args) {
          SpringApplication.run(OrderServiceApplication.class, args);
      }
  }

  @RestController
  public class OrderController {
      private final OrderProcessor orderProcessor;

      public OrderController(OrderProcessor orderProcessor) {
          this.orderProcessor = orderProcessor;
      }

      @PostMapping("/order")
      public ResponseEntity<String> createOrder(@RequestBody Order order) {
          orderProcessor.output().send(MessageBuilder.withPayload(order).build());
          return ResponseEntity.ok("Order placed successfully");
      }
  }

  public interface OrderProcessor {
      String OUTPUT = "order-placed";
      @Output(OUTPUT)
      MessageChannel output();
  }

Carrier Service: The Carrier Service consumes messages from the “order-placed” Kafka topic and interacts with external shipping carriers.

  @EnableBinding(OrderProcessor.class)
  @SpringBootApplication
  public class CarrierServiceApplication {
      public static void main(String[] args) {
          SpringApplication.run(CarrierServiceApplication.class, args);
      }
  }

  @StreamListener(OrderProcessor.INPUT)
  public void handleOrder(Order order) {
      // Logic to communicate with shipping carriers
      log.info("Processing order: " + order.getId());
  }

Tracking Service: The Tracking Service updates tracking information by consuming messages from another Kafka topic (e.g., “shipment-updates”).

  @EnableBinding(TrackingProcessor.class)
  @SpringBootApplication
  public class TrackingServiceApplication {
      public static void main(String[] args) {
          SpringApplication.run(TrackingServiceApplication.class, args);
      }
  }

  @StreamListener(TrackingProcessor.INPUT)
  public void handleTrackingUpdate(TrackingInfo trackingInfo) {
      // Logic to update tracking information
      log.info("Received tracking update for order: " + trackingInfo.getOrderId());
  }

2. Configuring Apache Kafka

Deploy Kafka using a managed service like AWS MSK (Managed Streaming for Apache Kafka) or set up Kafka in your Kubernetes cluster. Ensure the necessary topics are created:

kafka-topics.sh --create --topic order-placed --bootstrap-server <kafka-broker>
kafka-topics.sh --create --topic shipment-updates --bootstrap-server <kafka-broker>

3. Deploying Microservices on EKS

Create an EKS Cluster: Use AWS Management Console or the AWS CLI to create an EKS cluster.

  eksctl create cluster --name shipping-cluster --region us-west-2 --nodegroup-name shipping-nodes --node-type t3.medium

Deploy Kafka (if not using MSK): Use Helm to deploy Kafka to your EKS cluster.

  helm repo add bitnami https://charts.bitnami.com/bitnami
  helm install kafka bitnami/kafka

Deploy Spring Boot Microservices: Containerize each microservice using Docker and push the images to Amazon ECR (Elastic Container Registry).

  docker build -t order-service .
  docker tag order-service:latest <aws_account_id>.dkr.ecr.us-west-2.amazonaws.com/order-service:latest
  docker push <aws_account_id>.dkr.ecr.us-west-2.amazonaws.com/order-service:latest

Create Kubernetes deployment manifests for each microservice and apply them to your EKS cluster.

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: order-service
  spec:
    replicas: 3
    selector:
      matchLabels:
        app: order-service
    template:
      metadata:
        labels:
          app: order-service
      spec:
        containers:
        - name: order-service
          image: <aws_account_id>.dkr.ecr.us-west-2.amazonaws.com/order-service:latest
          ports:
          - containerPort: 8080

Apply the manifest:

  kubectl apply -f order-service-deployment.yaml

Set Up Service Discovery and Load Balancing: Use Spring Cloud Kubernetes for service discovery. Expose each service using Kubernetes services, and if necessary, create an ingress to manage traffic.

  apiVersion: v1
  kind: Service
  metadata:
    name: order-service
  spec:
    selector:
      app: order-service
    ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
    type: LoadBalancer

Apply the manifest:

  kubectl apply -f order-service-service.yaml

4. Monitoring and Scaling

Monitoring: Use Amazon CloudWatch and Prometheus/Grafana for monitoring the health and performance of your microservices.
Auto-scaling: Implement Horizontal Pod Autoscaling (HPA) to scale your microservices based on metrics like CPU and memory usage.

  kubectl autoscale deployment order-service --cpu-percent=50 --min=1 --max=10

Conclusion

Building Java Spring Cloud microservices with Kafka for shipping carrier systems offers a scalable and resilient architecture for handling complex workflows. Deploying these microservices on EKS further enhances scalability and manageability, enabling your system to handle large volumes of shipping data with ease. By leveraging the combination of Spring Cloud, Kafka, and EKS, you can create a robust, real-time shipping platform that meets the demands of modern eCommerce and logistics.