Technology stack of BigData

Read more...

Docker versus Kubernetes - comparison

1. Introduction
   
   1. Containers have become popular thanks to their focus on consistency across platforms from development to production.
      
   2. The rise in interest to containers has in turn brought in higher demands for their deployment and management.
      
   3. The need for better control attracted a number of software options as solutions for container orchestration, which allows for abstraction of individual containers to services with a number of instances or replicas.
      
   4. Two of the major players developing container orchestration are Docker and Kubernetes.
      
2. Kubernetes
   
   1. Kubernetes is an open-source platform for container deployment automation, scaling, and operations across clusters of hosts. The production ready orchestrator draws on Google’s
      extensive experience of years of working with Linux containers.
      
   2. Kubernetes aims to provide the components and tools to relieve the burden of running applications in public and private clouds by grouping containers into logical units. Their strengths lie in flexible growth, environment agnostic portability, and easy scaling.
      
3. Docker Swarm
   
   1. Swarm is the native clustering for Docker. Originally Docker Swarm did not provide much in the sense of container automation, but with the update to Docker Engine 1.12, container orchestration is now built into its core with first party support.
      
   2. Docker Swarm is designed around four core principles:
      
      1. Simple yet powerful with a “just works” user experience,
         
      2. Resilient zero single-point-of-failure architecture,
         
      3. Secure by default with automatically generated certificates, and
         
      4. Backwards compatibility with existing components.
         
   3. The promise of backwards compatibility is especially important to the existing users. Any tools or containers that work with Docker run equally well in Docker Swarm.
      
4. Comparisons
   Although both orchestrators provide much of the same functionality to one another, there are fundamental differences in between how the two operate. Below are listed some of the most notable points on where these rivals diverge.
   
   
   
   
   
   
   
5. Summary
   
   1. Throughout the comparison, it is possible to note how Kubernetes and Docker Swarm fundamentally differ.
      
   2. Swarm focuses on ease of use with integration with Docker core components while Kubernetes remains open and modular.
      
   3. The same difference can be noticed while installing and configuring each of the orchestrators.
      
   4. Docker provides a simple solution that is fast to get started with while Kubernetes aims to support higher demands with higher complexity.
      
   5. For much of the same reasons, Docker has been popular among developers who prefer simplicity and fast deployments.
      
   6. At the same time, Kubernetes is used in production environments by many high profile internet companies running popular services
      

Read more...

Kafka Multiple Topic

Producer and multiple Topics

Download a recent stable version of Apache Kafka
1. Untar the package
   
2. Enter into Kafka directory
   
3. Start Zookeeper Server
   
4. bin/zookeeper-server-start.sh config/zookeeper.properties
   
5. In a different terminal start Kafka Server
   
6. bin/kafka-server-start.sh config/server.properties
   
7. Create a topic test (if not exists)
   
8. bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic example2
   
9. Create a topic test1 (if not exists)
   
10. bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic example3
    
11. Start consumer on topic test
    
12. bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic example2 --from-beginning
    
13. Start consumer on topic test1
    
14. bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic example3 --from-beginning
    
15. Run mvn clean compile exec:exec
    
16. You should see the message in the consumer terminal.
    

Here are the project sources
1. Project POM.xml
   

   
   
     4.0.0
    
     ../pom.xml
     1.0.0-SNAPSHOT
     com.example.kafka
     kafka-examples
    
     kafka-producer-multiple-topics
     kafka-producer-multiple-topics
   
    
     
      
       org.codehaus.mojo
       exec-maven-plugin
       
        
         
          exec
         
        
       
       
        java
        
         -classpath
         
         com.example.kafka.ProducerMultipleTopic
        
       
      
     
    
   
   
   

   
   
2. Producer Java File
   

   package com.example.kafka;
   
   import java.util.Properties;
   
   import org.apache.kafka.clients.producer.KafkaProducer;
   import org.apache.kafka.clients.producer.ProducerRecord;
   import org.apache.kafka.common.serialization.StringSerializer;
   
   public class ProducerMultipleTopic {
   
       public static void main(String[] args) {
   
           Properties props = new Properties();
           props.put("bootstrap.servers", "localhost:9092");
           props.put("key.serializer", StringSerializer.class.getName());
           props.put("value.serializer", StringSerializer.class.getName());
           
           KafkaProducer prod = new KafkaProducer(props);
   
           ProducerRecord data1 = new ProducerRecord("example2", "example2");
           ProducerRecord data2 = new ProducerRecord("example3", "example3");
   
           prod.send(data1);
           prod.send(data2);
   
           prod.close();
       }
   }
   

3. Logger file
   

   # Root logger option
   log4j.rootLogger=INFO, stdout
   
   # Direct log messages to stdout
   log4j.appender.stdout=org.apache.log4j.ConsoleAppender
   log4j.appender.stdout.Target=System.out
   log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
   log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
   
   

   
   
   
4. This is the PARENT POM
   

   
   
    4.0.0
    com.example.kafka
    kafka-examples
    1.0.0-SNAPSHOT
    pom
   
    
     
      org.apache.kafka
      kafka_2.10
      0.9.0.1
     
     
      org.apache.kafka
      kafka-clients
      0.9.0.1
     
     
      log4j
      log4j
      1.2.17
     
    
   
    
     UTF-8
    
   
    
     
      
       
        org.apache.maven.plugins
        maven-compiler-plugin
        3.2
        
         1.7
         1.7
        
       
       
        org.codehaus.mojo
        exec-maven-plugin
        1.3.2
       
      
     
    
    
      kafka-producer-multiple-topics
     
   
   
   

   
   

Read more...