Sunday, May 21, 2017

Continuous Delivery with Docker on Mesos and Marathon

Full functional development and continuous integration environments


  • Here is the full docker-compose.yml file that includes all parts of the system.
  • In addition to Jenkins and Docker registry we have Mesos master, single Mesos slave, Mesosphere Marathon and Zookeeper for internal Mesos communication.
docker-compose.yml
===============================================
version: '3'
services:
  zookeeper:
    image: garland/zookeeper
    hostname: "zookeeper"
    container_name: zookeeper
    ports:
      - "2181:2181"
      - "2888:2888"
      - "3888:3888"
  mesos-master:
    image: garland/mesosphere-docker-mesos-master
    hostname: "mesos-master"
    container_name: mesos-master
    environment:
      - MESOS_QUORUM=1
      - MESOS_ZK=zk://192.168.99.100:2181/mesos
      - MESOS_REGISTRY=in_memory
      - MESOS_LOG_DIR=C:/var/log/mesos
      - MESOS_WORK_DIR=C:/var/lib/mesos
    links:
      - zookeeper:zk
    ports:
      - "5050:5050"
  mesos-slave:
    image: garland/mesosphere-docker-mesos-master:latest
    hostname: "mesos-slave-1"
    container_name: mesos-slave-1
    environment:
      - MESOS_HOSTNAME=192.168.99.100
      - MESOS_MASTER=zk://192.168.99.100:2181/mesos
      - MESOS_LOG_DIR=C:/var/log/mesos
      - MESOS_LOGGING_LEVEL=INFO
    entrypoint: mesos-slave
    links:
      - zookeeper:zk
      - mesos-master:master
    ports:
      - "5151:5151"
  marathon:
    image: garland/mesosphere-docker-marathon
    hostname: "marathon"
    container_name: marathon
    environment:
      - MARATHON_MASTER=zk://192.168.99.100:2181/mesos
      - MARATHON_ZK=zk://192.168.99.100:2181/marathon
    command: --master zk://zk:2181/mesos --zk zk://zk:2181/marathon
    links:
      - zookeeper:zk
      - mesos-master:master
    ports:
      - "8080:8080"
###Environment 
===============================================


It is important to note that the Jenkins container now includes a link to Marathon. This is required to be able to post the requests from the Jenkins container to the Marathon container.

Now we can restart the system and see it all up and running:
docker-compose up


A quick introduction to Apache Mesos

Apache Mesos is a centralised fault-tolerant cluster manager. It’s designed for distributed computing environments to provide resource isolation and management across a cluster of slave nodes.
In some ways, Mesos provides the opposite to virtualization:
  • Virtualization splits a single physical resource into multiple virtual resources
  • Mesos joins multiple physical resources into a single virtual resource
It schedules CPU and memory resources across the cluster in much the same way the Linux Kernel schedules local resources.

A Mesos cluster is made up of four major components:
  • ZooKeeper
  • Mesos masters
  • Mesos slaves
  • Frameworks
ZooKeeper
Apache ZooKeeper is a centralized configuration manager, used by distributed applications such as Mesos to coordinate activity across a cluster. Mesos uses ZooKeeper to elect a leading master and for slaves to join the cluster.

Mesos masters
A Mesos master is a Mesos instance in control of the cluster.A cluster will typically have multiple Mesos masters to provide fault-tolerance, with one instance elected the leading master.
Mesos slaves
A Mesos slave is a Mesos instance which offers resources to the cluster.
They are the ‘worker’ instances - tasks are allocated to the slaves by the Mesos master.

Frameworks:
  • On its own, Mesos only provides the basic “kernel” layer of your cluster. It lets other applications request resources in the cluster to perform tasks, but does nothing itself.
  • Frameworks bridge the gap between the Mesos layer and your applications. They are higher level abstractions which simplify the process of launching tasks on the cluster.
  1. Chronos
    Chronos is a cron-like fault-tolerant scheduler for a Mesos cluster. You can use it to schedule jobs, receive failure and completion notifications, and trigger other dependent jobs.
  2. Marathon
    Marathon is the equivalent of the Linux upstart or init daemons, designed for long-running applications. You can use it to start, stop and scale applications across the cluster.

Others :

There are a few other frameworks:
The quick start guide

Using Vagrant

Vagrant and the vagrant-mesos Vagrant file can help you quickly build:
  • a standalone Mesos instance
  • a multi-machine Mesos cluster of ZooKeepers, masters and slaves

Unfortunately, the network configuration is a bit difficult to work with - it uses a private network between the VMs, and SSH tunnelling to provide access to the cluster.
Using Mesosphere and Amazon Web Services

Mesosphere provide Elastic Mesosphere, which can quickly launch a Mesos cluster using Amazon EC2.

Vagrant files to build individual components of a Mesos cluster. It’s a work in progress, but it can already build a working Mesos cluster without the networking issues. It uses bridged networking, with dynamically assigned IPs, so all instances can be accessed directly through your local network.

You’ll need the following GitHub repositories:
ian-kent/vagrant-zookeeper
ian-kent/vagrant-mesos-master
ian-kent/vagrant-mesos-slave

At the moment, a cluster is limited to one ZooKeeper, but can support multiple Mesos masters and slaves.

Each of the instances is also built with Serf to provide decentralized service discovery. You can use serf members from any instance to list all other instances.

To help test deployments, there’s also a MongoDB build with Serf installed:
ian-kent/vagrant-mongodb

Like the ZooKeeper instances, the MongoDB instance joins the same Serf cluster but isn’t part of the Mesos cluster.

Once your cluster is running

You’ll need to install a framework.
Mesosphere lets you choose to install Marathon on Amazon EC2, so that could be a good place to start.
With Marathon or Aurora, you can even run other frameworks in the Mesos cluster for scalability and fault-tolerance.

Friday, May 19, 2017

(Creational) Abstract Factory Pattern

Abstract Factory Pattern: 
  • Abstract Factory pattern adds another layer of abstraction for Factory pattern.
  • Abstract Factory is a super-factory which creates other factories. We can call it  "Factory of factories".
Interesting points:
  1. Abstract Factory, Builder, and Prototype can use Singleton in their implementation. Abstract Factory Pattern is often used along with Factory Method but also can be implemented using Prototype pattern to increase the performance and simplifying the code.
  2. Abstract Factory can be used as an alternative to Façade pattern to hide platform specific classes
  3. Abstract Factory class declares only an interface for creating the products. The actual creation is the task of the ConcreteProduct classes, where a good approach is applying the Factory Method design pattern for each product of the family.

Difference between Abstract Factory & Factory Method pattern:
  1. Factory Method pattern exposes a method to the client for creating the object whereas in case of Abstract Factory they expose a family of related objects which may consist of these Factory methods.
  2. Designs start out using Factory Method (less complicated, more customizable, subclasses proliferate) and evolve toward Abstract Factory, Prototype, or Builder (more flexible, more complex) as the designer discovers where more flexibility is needed.
  3. Factory Method pattern hides the construction of single object where as abstract factory method hides the construction of a family of related objects. Abstract factories are usually implemented using (a set of) factory methods.
Example: click here












Monday, May 8, 2017

Java Collections Framework


  • A collection in java is framework that provides architecture to store and manipulate the group of Object.
  • All the operation that you perform on a data such as searching, sorting, insertion .manipulation, deletion etc.
  • Java collection provide many interface(Set,List,Queue,Deque,SortedSet,SortedMap) and
    class(ArrayList,Vector,LinkedList, PriorityQueue, HashSet, HashMap, LinkedHashSet, TreeSet,TreeMap).







 

The Set Interface

HashSet, LinkedHashSet, and TreeSet Classes

  • HashSet class is the default implementation used in most cases. 
  • LinkedHashSet is like a combination of HashSet and List in that it does not allow duplicate entries as with Sets, but traverses its elements in the order they were inserted, like a List would do. 
  • TreeSet will constantly keep all its elements in some sorted order.

SortedSet and Navigable Set Interfaces

As the name implies, SortedSet is a Set with the property that it is always sorted. The NavigableSet interface, added with Java 6, allows us to navigate through the sorted list, providing methods for retrieving the next element greater or smaller than a given element of the Set.

The List Interface

ArrayList and LinkedList Classes

  • ArrayList is the default implementation for List, it does allow duplicate elements and iteration in the order of insertion. As it is based on arrays, it is very fast to iterate and read from, but very slow to add or remove an element at random positions, as it has to rebuild the underlying array structure. 
  • In contrast, LinkedList makes it easy to add or remove elements at any position in the list while being slower to read from at random positions.

The Queue Interface

Thing to mention about LinkedList is that while it implements List, it actually also implements Queue. It does so based on the fact that its actual implementation as a doubly-linked list makes it quite easy to also implement the Queue interface.

PriorityQueue class

Besides LinkedList, another common Queue implementation is PriorityQueue. It is an implementation that keeps its elements ordered automatically. It has functionality similar to TreeSet, except that it allows duplicate entries.

The Map Interface

No relation to the Collection interface. A Collection operates on one entity, while a Map operates on two: a unique key, e.g. a vehicle identification number, and an object related to the key, e.g. a car. To retrieve an object from a Map, you would normally use its key.

Hashtable, Hashmap, and LinkedHashMap Classes

The Hashtable class was the first Collection in Java 1 that was based on the hash-table data structure. Unfortunately, like Vector, the class is deprecated because of its suboptimal performance. We can forget about it and use the other Map implementations instead. HashMap is the default implementation that you will find yourself using in most cases.
A Map usually doesn’t make any guarantee as to how it internally stores elements. An exception to this rule, however, is LinkedHashMap, which allows us to iterate the map in the order of insertion.
map-interface-1
Figure 2 The Map Hierarchy

SortedMap

Let’s look at the interfaces that extend Map. As the name implies, SortedMap extends Map and defines the contract of a constantly sorted map. NavigableMap takes it even further, adding methods to navigate sorted maps. This allows us to get all entries smaller or bigger than some entry, for example. There are actually many similarities between the Map and Set hierarchies. The reason is that Set implementations are actually internally backed by Map implementations.

Map Inteface

Now let’s look at the map interface. This interface has no relation to the Collection interface. A Collection operates on one entity, while a map operates on two entities – A unique key, for example a Vehicle Identification Number, and an object that is related to this key, for example a car object. With the help of the key you can retrieve the object it relates to. The interface map is the root of a lot of interfaces and classes, which we will look at now. The class Hashtable was the first collection in Java JDK1 that was based on the data structure hashtable, so the Java creators called it Hashtable. Unfortunately, this makes it hard to differentiate between the two. Like Vector, the class is deprecated because of its suboptimal performanceso let’s remove and forget about it. Instead, use one of the other classes that implement the map interface. HashMap is the default implementation that you should use in the majority of cases. A Map usually does not make any guarantees on how it internally stores its elements. An exception to this rule is LinkedHashMap, which allows to iterate the map in the order of insertion. Last but not least, TreeMap is a constantly sorted map.
sortedmap


Synchronized Wrappers

The synchronization wrappers add automatic synchronization (thread-safety) to an arbitrary collection. Each of the six core collection interfaces — Collection, Set, List, Map, SortedSet, and SortedMap — has one static factory method.
public static  Collection synchronizedCollection(Collection c);
public static  Set synchronizedSet(Set s);
public static  List synchronizedList(List list);
public static <K,V> Map<K,V> synchronizedMap(Map<K,V> m);
public static  SortedSet synchronizedSortedSet(SortedSet s);
public static <K,V> SortedMap<K,V> synchronizedSortedMap(SortedMap<K,V> m);
Each of these methods returns a synchronized (thread-safe) Collection backed up by the specified collection.

Unmodifiable wrappers

Unmodifiable wrappers take away the ability to modify the collection by intercepting all the operations that would modify the collection and throwing an UnsupportedOperationException. It’s main usage are;
  • To make a collection immutable once it has been built. In this case, it’s good practice not to maintain a reference to the backing collection. This absolutely guarantees immutability.
  • To allow certain clients read-only access to your data structures. You keep a reference to the backing collection but hand out a reference to the wrapper. In this way, clients can look but not modify, while you maintain full access.
These methods are;
public static  Collection unmodifiableCollection(Collection extends T> c);
public static  Set unmodifiableSet(Set extends T> s);
public static  List unmodifiableList(List extends T> list);
public static <K,V> Map<K, V> unmodifiableMap(Map extends K, ? extends V> m);
public static  SortedSet unmodifiableSortedSet(SortedSet extends T> s);
public static <K,V> SortedMap<K, V> unmodifiableSortedMap(SortedMap<K, ? extends V> m);

Thread Safe Collections

Java 1.5 Concurrent package (java.util.concurrent) contains thread-safe collection classes that allow collections to be modified while iterating. By design iterator is fail-fast and throws ConcurrentModificationException. Some of these classes are CopyOnWriteArrayList, ConcurrentHashMap, CopyOnWriteArraySet.
Read these posts to learn about them in more detail.

Collections API Algorithms

Java Collections Framework provides algorithm implementations that are commonly used such as sorting and searching. Collections class contain these method implementations. Most of these algorithms work on List but some of them are applicable for all kinds of collections.

Sorting

The sort algorithm reorders a List so that its elements are in ascending order according to an ordering relationship. Two forms of the operation are provided. The simple form takes a List and sorts it according to its elements’ natural ordering. The second form of sort takes a Comparator in addition to a List and sorts the elements with the Comparator.

Shuffling

The shuffle algorithm destroys any trace of order that may have been present in a List. That is, this algorithm reorders the List based on input from a source of randomness such that all possible permutations occur with equal likelihood, assuming a fair source of randomness. This algorithm is useful in implementing games of chance.

Searching

The binarySearch algorithm searches for a specified element in a sorted List. This algorithm has two forms. The first takes a List and an element to search for (the “search key”). This form assumes that the List is sorted in ascending order according to the natural ordering of its elements. The second form takes a Comparator in addition to the List and the search key, and assumes that the List is sorted into ascending order according to the specified Comparator. The sort algorithm can be used to sort the List prior to calling binarySearch.

Composition

The frequency and disjoint algorithms test some aspect of the composition of one or more Collections.
  • frequency: counts the number of times the specified element occurs in the specified collection
  • disjoint: determines whether two Collections are disjoint; that is, whether they contain no elements in common

Min and Max values

The min and the max algorithms return, respectively, the minimum and maximum element contained in a specified Collection. Both of these operations come in two forms. The simple form takes only a Collection and returns the minimum (or maximum) element according to the elements’ natural ordering.
The second form takes a Comparator in addition to the Collection and returns the minimum (or maximum) element according to the specified Comparator.


JDK design pattern

First, the design pattern is what (1) the problem of repeated solutions (2) to enhance the flexibility of the software (3) to adapt to the changing software

(1) learn from the excellent code design, help to improve the code design capabilities (2) JDK design reflects the majority of the design pattern is to learn the design pattern of a better way (3 ) Can be more in-depth understanding of JDK

Third, the relationship between class inheritance, commission, dependency, aggregation, combination


Fourth, the introduction of the way (1) Role: to summarize a design pattern of the basic points (2) JDK embodied: a design pattern in the JDK is how to reflect the (3) class diagram: a design pattern in the JDK corresponding to the class Figure

Fifth, the classic design patterns in the implementation of the JDK
1.Singleton (singleton)
Role: Guaranteed class has only one instance; provide a global access point
JDK embodies:
(1) Runtime
(2) NumberFormat
Class Diagram:


2.Factory (static factory)
effect:
(1) instead of the constructor to create the object (2) method name than the constructor function is clear
JDK embodies:
(1) Integer.valueOf
(2) Class.forName
Class Diagram:


3.Factory Method (factory method)
Role: Subclass determines which class is instantiated
JDK embodied: Collection.iterator method class diagram:


4.Abstract Factory (abstract factory)
Role: Create a class of objects
JDK embodies:
(1) java.sql package (2) UIManager (swing appearance)
Class Diagram:


5.Builder (constructor)
effect:
(1) the construction logic mentioned in a separate class (2) separation of the class structure logic and performance
JDK embodied: DocumentBuilder (org.w3c.dom)
Class Diagram:


6.Prototype (prototype)
effect:
(1) copy object (2) shallow copy, deep copy
JDK embodied: Object.clone; Cloneable
Class Diagram:

7.Adapter (adapter)
Function: Make incompatible interface compatible
JDK embodies:
(1) java.io.InputStreamReader (InputStream)
(2) java.io.OutputStreamWriter (OutputStream)
Class Diagram:


8.Bridge (bridge)
The role: the abstract part and its realization part of the separation, so that they can be independent changes
JDK embodies: Handler and Formatter in java.util.logging
Class Diagram:

9.Composite (combination)
Role: Consistent treatment of composite objects and independent objects
JDK embodies:
(1) org.w3c.dom
(2) javax.swing.JComponent # add (Component)
Class Diagram:


10.Decorator (decorator)
Role: add new features to the class; prevent class inheritance from the explosive growth
JDK embodies:
(1) java.io package (2) java.util.Collections # synchronizedList (List)
Class Diagram:

11.Façade (appearance)
effect:
(1) encapsulates a set of interactive classes, consistently providing an interface (2) encapsulating subsystems, simplifying subsystem calls
JDK embodied: java.util.logging package class diagram:

12.Flyweight
Role: shared object, saving memory
JDK embodies:
(1) Integer.valueOf (int i); Character.valueOf (char c)
(2) String constant pool diagram:

14.Proxy (proxy)
effect:
(1) transparent call by the proxy object, without having to know the details of the complex implementation (2) to increase the function of the proxy class
JDK embodied: dynamic agent; RMI
Class Diagram:


15.Iterator (iterator)
Role: Separate the set of iterations and the collection itself
JDK embodied: Iterator, Enumeration interface class diagram:

16.Observer (observer)
Action: Notify the object state change
JDK embodies:
(1) java.util.Observer, Observable
(2) Listener in Swing
Class Diagram:

17.Mediator (coordinator)
Role: Used to coordinate the operation of multiple classes
JDK embodies: Swing's ButtonGroup
Class Diagram:

18.Template method (template method)
Role: the definition of the structure of the algorithm, sub-class only to achieve different parts
JDK embodied: ThreadPoolExecutor.Worker
Class Diagram:

19.Strategy (strategy)
Role: provide different algorithms
JDK in the expression: ThreadPoolExecutor in the four types of rejection strategy class:

20.Chain of Responsibility
Action: The request is handled by the object on the chain, but the client does not know which objects the request will be processed
JDK embodies:
(1) java.util.logging.Logger delegates log to parent logger
(2) ClassLoader delegate model class diagram:

21.Command (command)
effect:
(1) package operation, so that the interface is consistent (2) the caller and receiver in space and time on the decoupling
JDK embodied: Runnable; Callable; ThreadPoolExecutor
Class Diagram:

22. Null Object (empty object)
Function: do not need to empty each time, treat the null value, as to treat an object of the same interface
JDK embodied: Collections.EMPTY_LIST
Class Diagram:

23.State (state)
Role: the main object and its state separation, the state object is responsible for the state of the main object conversion, so that the main object class function
JDK reflected: no class map found:

24.Visitor (Visitor)
Role: heterogeneous class to add polymerization operations; collecting aggregated data
JDK in the embodiment: no class map:

25.Interpreter (interpreter)
Function: Use a set of classes to represent a rule
JDK embodied: java.util.regex.Pattern
Class diagram: four operations

26.Memento (memo)
Role: to maintain the object state, if necessary, can be restored
JDK reflected: no class map found:

Sixth, reference
1. Design Pattern (GoF)
2. Software Architecture Design Patterns in Java
3. JDK 5 Documentation
4. http://stackoverflow.com/questions/1673841/examples-of-gof-design-patterns
5. http://java.csdn.Net/a/20101129/282644.html

Thursday, May 4, 2017

8 Benefits of Microservices

8 benefits of microservices
Microservices are small, independent services that work together. In other words, these services are small, highly decoupled and focus on doing a small task at a time.
  • Follow the Single Responsibility Principle
  • Resilient/Flexible – failure in one service does not impact other services.
  • High scalability – demanding services can be deployed in multiple servers to enhance performance and keep away from other services so that they don’t impact other services
  • Easy to enhance – less dependency and easy to change and test
  • Low impact on other services – being an independent service, this has less chance to impact other services
  • Easy to understand –  since they represent the small piece of functionality
  • Ease of deployment – microservice support agile development
  • Freedom to choose technology – allows to choose technology, best suited for a particular functionality
  • Process-centric components – Each microservice typically runs in a separate process as distinct from a thread within a larger process. This ensures the component is neither too small nor too large.  
  • Data-driven interfaces –  with a few inputs and outputs. In practice, most productive Microservices typically have only a few inputs and outputs (often less than 4 each). Communication between Microservices is document-centric an important feature of Microservices architecture.
  • Focused functionality :  A Microservice is typically organized around a single, focused capability: e.g. access/update a Database, Cache input data, update a Bank account, Notify patients, etc.
  • Independent interfaces: Each Microservice typically has a GUI Associated with itself, for end-user interaction and configuration.  Each Microservice is essentially a ‘product’ in its own right in that it has defined inputs and outputs and defined, non-trivial fun
  • Easier Governance : since each Microservice runs in its own process and shares no dependencies with others, it can be monitored, metered and managed individually.
  • Decentralized Data Management : Each Microservice can store data as it pleases : a given Microservice can use SQL, a second MongoDB, a third a Mainframe database, etc.
  • Automated Deployment and Infrastructure Automation : Implementation independence allows assemblies of Microservices can be ‘moved’ from one deployment environment to another (for example, Development —> QA —> Stating —> Production), based just on profile settings.
  • Orchestration and Choreography: Synchronous vs. Asynchronous :  If the Microservices are orchestrated using a classic synchronous mechanism (blocking calls, each waiting for downstream calls to return), potential performance problems can occur as the call-chain increases in size. A more efficient mechanism is to use an asynchronous protocol, such as JMS or any other enterprise-messaging protocol/tool (IBM MQ, MSMQ, etc.) to choreograph the Microservices. This approach ensures that there are no bottlenecks in the final application-system since most of the communication is via non-blocking asynchronous calls, with blocking, synchronous calls limited to things like user-interactions. A simple rule of thumb is to avoid as many blocking calls as one can.
  •  

    Disadvantages to using a microservices architecture :
    • Services form information barriers.
    • The architecture introduces additional complexity and new problems to deal with, such as network latency, message formats, load balancing, and fault tolerance; ignoring one of these belongs to the "fallacies of distributed computing."
    • Testing and deployment are more complicated.
    • The complexity of a monolithic application is only shifted into the network, but it persists.
    • Too-fine-grained microservices have been criticized as an anti-pattern.
    • Overcoming the Domino Effect (Cascade Failure)
      When working with microservices, it’s possible for a microservice you don’t even know about to cause a cascade failure. When microservice A fails and there is no available failover, microservice B could also fail, which might bring microservice C to a standstill as well.
      The point is to be proactive in assuming that failures will occur in order to avoid potential cascade failures.