tag:blogger.com,1999:blog-61878790327857915982024-02-07T11:43:02.208+05:30* Techie(S)pArK *Learn... Unlearn... Relearn...Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.comBlogger2125tag:blogger.com,1999:blog-6187879032785791598.post-81199593110948656002015-01-03T17:00:00.000+05:302015-01-03T21:29:43.094+05:30Apache Mesos : Writing your own distributed framework<span style="font-family: Trebuchet MS, sans-serif;">In the previous <a href="http://femgeekz.blogspot.in/2014/12/introduction-and-getting-started-with.html">post</a>, we saw what mesos is, how is it useful and getting started with it. In this post, we shall see how to write your own framework on mesos.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">(In mesos, a framework is any application running on it.) </span><br />
<span style="font-family: Trebuchet MS, sans-serif;">This post explains about a framework called "mesos-pinspider" which fetches the user profile information and user board information of a pinterest page of a user.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<br />
<h3>
<span style="font-family: Trebuchet MS, sans-serif;">Mesos Framework</span></h3>
<span style="font-family: Trebuchet MS, sans-serif;">In general, a Mesos framework has three basic components. </span><br />
<span style="font-family: Trebuchet MS, sans-serif;">- <b>Driver</b> which submits the tasks to the framework</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">- <b>Scheduler</b> which registers with the master to be offered resources, takes the tasks and runs them on executor</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">- <b>Executor</b> process that is launched on slave nodes to run the frameworkâs tasks</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<br />
<h3>
<span style="font-family: Trebuchet MS, sans-serif;">Pinspider Framework Example</span></h3>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">You may check the code here on <a href="https://github.com/SwathiMystery/mesos-pinspider">github</a>. Let's break it down to PinDriver, PinScheduler and Pin</span><span style="font-family: 'Trebuchet MS', sans-serif;">UserProfileExecutor</span><span style="font-family: Trebuchet MS, sans-serif;">.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<br />
<h3>
<span style="font-family: Trebuchet MS, sans-serif;">Driver</span></h3>
<span style="font-family: Trebuchet MS, sans-serif;">The driver component of the framework is PinDriver. </span><br />
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Create Executor Info</span></li>
</ul>
</ul>
<span style="font-family: Trebuchet MS, sans-serif;">Describe the information about the executor using the Builder pattern and mesos use Google Protocol Buffers for the data interchange. Here, we need to set the executorID, command which is basically a shell command, executed via:</span><span style="font-family: Courier New, Courier, monospace;"> '/bin/sh -c value'</span><span style="font-family: Trebuchet MS, sans-serif;">. Any URIs specified are fetched before executing the command. The name is set by </span><span style="font-family: Courier New, Courier, monospace;">setName()</span><span style="font-family: Trebuchet MS, sans-serif;">. The source is set by </span><span style="font-family: Courier New, Courier, monospace;">setSource()</span><span style="font-family: Trebuchet MS, sans-serif;">, an identifier style string used by frameworks to track the source of an executor. This is useful when it's possible for different executor ids to be related semantically.</span><br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">Protos.ExecutorInfo userProfileExecutorInfo = Protos.ExecutorInfo.newBuilder().setExecutorId(Protos.ExecutorID.newBuilder().setValue("PinUserProfileExecutor")).setCommand(commandInfoUserProfile).setName("PinUserProfileExecutor Java").setSource("java").build();</span></blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Create Framework Info</span></li>
</ul>
</ul>
<span style="font-family: Trebuchet MS, sans-serif;">Describe the framework information. The user field is used to determine the Unix user that an executor/task should be launched as. If the user field is set to an empty string Mesos will auto-magically set it to the current user. The amount of time that the master will wait for the scheduler to fail-over before removing the framework is specified by </span><span style="font-family: Courier New, Courier, monospace;">setFailoverTimeout()</span><span style="font-family: Trebuchet MS, sans-serif;">. The name of the framework is set by </span><span style="font-family: Courier New, Courier, monospace;">setName()</span><br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">Protos.FrameworkInfo.Builder frameworkBuilder = Protos.FrameworkInfo.newBuilder().setFailoverTimeout(120000)</span><span style="font-family: 'Courier New', Courier, monospace;">.setUser("").setName("Pinspider Framework"); </span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Instantiate Scheduler</span></li>
</ul>
</ul>
<span style="font-family: 'Trebuchet MS', sans-serif;">You need to instantiate the Scheduler with the number of tasks that needs to be submitted for the executor to run.</span><br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">Scheduler scheduler = args.length == 1 ?</span><span style="font-family: 'Courier New', Courier, monospace;">new PinScheduler(userProfileExecutorInfo,userBoardExecutorInfo) : </span><span style="font-family: 'Courier New', Courier, monospace;">new PinScheduler(userProfileExecutorInfo, userBoardExecutorInfo, Integer.parseInt(args[1]), args[2]);</span></blockquote>
</blockquote>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Note: Please note that two ExecutorInfo are used ie. one for fetching user profile information and the other one for user board information for demonstration. This explanation involves only one executorinfo - userProfileExecutorInfo</span><br />
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Starting the mesos scheduler driver.</span></li>
</ul>
</ul>
<span style="font-family: 'Trebuchet MS', sans-serif;">MesosSchedulerDriver is an implementation of SchedulerDriver which is an abstract interface to connect scheduler to mesos. This is done by managing the life-cycle of the scheduler ( start, stop and wait for tasks to finish) and also to interact with Mesos (launch tasks, kill tasks etc). </span><br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">MesosSchedulerDriver schedulerDriver = new MesosSchedulerDriver(scheduler,frameworkBuilder.build(), args[0]);</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">int status = schedulerDriver.run() == Protos.Status.DRIVER_STOPPED ? 0 : 1;</span><span style="font-family: 'Courier New', Courier, monospace;">schedulerDriver.stop();</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">System.exit(status);</span></blockquote>
</blockquote>
<h3>
<span style="font-family: Trebuchet MS, sans-serif;">Executor Implementation</span></h3>
<span style="font-family: 'Trebuchet MS', sans-serif;">The </span><span style="font-family: 'Trebuchet MS', sans-serif;">Executor</span><span style="font-family: 'Trebuchet MS', sans-serif;"> component of the framework is PinUserProfileExecutor.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Executor is a callback interface which is implemented by frameworks' executors. In our implementation, let us concentrate on launchTask()</span><br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">@Override public void launchTask(final ExecutorDriver executorDriver, final Protos.TaskInfo taskInfo) { </span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: 'Courier New', Courier, monospace;">}</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> Set the task status by setting the ID and the state with a builder pattern.</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">Protos.TaskStatus taskStatus = Protos.TaskStatus.newBuilder().setTaskId(taskInfo.getTaskId())</span><span style="font-family: 'Courier New', Courier, monospace;">.setState(Protos.TaskState.TASK_RUNNING).build();</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> Send the status update to the framework scheduler retrying as necessary until an acknowledgement has been received or the executor is terminated, in which case, a TASK_LOST status update will be sent.</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">executorDriver.sendStatusUpdate(taskStatus);</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Get the data from the tasks and run your logic.</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">try {</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>message = ("userprofile :" + getUserProfileInfo(url)).getBytes();</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>} catch (IOException e) {</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>LOGGER.error("Error parsing the Pinterest URL :" + e.getMessage());</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> Send the framework the message.</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">executorDriver.sendFrameworkMessage(message);</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> Mark the state of the task as finished and send the status update to the framework scheduler.</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">taskStatus = Protos.TaskStatus.newBuilder().setTaskId(taskInfo.getTaskId())</span><span style="font-family: Courier New, Courier, monospace;"> .setState(Protos.TaskState.TASK_FINISHED).build();</span><span style="font-family: Courier New, Courier, monospace;">executorDriver.sendStatusUpdate(taskStatus);</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> main() method to create an instance of MesosExecutorDriver and run </span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">mesosExecutorDriver.run() == Protos.Status.DRIVER_STOPPED ? 0 : 1</span></blockquote>
</blockquote>
<h3>
<span style="font-family: 'Trebuchet MS', sans-serif;">Scheduler Implementation</span></h3>
<span style="font-family: 'Trebuchet MS', sans-serif;">The Scheduler</span><span style="font-family: 'Trebuchet MS', sans-serif;"> component of the framework is Pin</span><span style="font-family: 'Trebuchet MS', sans-serif;">Scheduler</span><span style="font-family: 'Trebuchet MS', sans-serif;">.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Scheduler is a callback interface to be implemented by frameworks' schedulers. In our implemenation, let us concentrate on resourceOffers(), statusUpdate() and frameworkMessage()</span><br />
<ul>
<li><span style="font-family: Trebuchet MS, sans-serif;">Constructor : construct with the executor information and the number of launch tasks.</span></li>
</ul>
<br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span class="Apple-tab-span" style="font-family: 'Trebuchet MS', sans-serif; white-space: pre;"> </span><span style="font-family: Courier New, Courier, monospace;">public PinScheduler(Protos.ExecutorInfo pinUserProfileExecutor , Protos.ExecutorInfo pinUserBoardExecutor ) {<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>this(pinUserProfileExecutor,pinUserBoardExecutor, 5, "http://www.pinterest.com/techcrunch");<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span></blockquote>
</blockquote>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">public PinScheduler(Protos.ExecutorInfo pinUserProfileExecutor,Protos.ExecutorInfo pinUserBoardExecutor, int totalTasks, String url) {</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>this.pinUserProfileExecutor = pinUserProfileExecutor;</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>this.pinUserBoardExecutor = pinUserBoardExecutor;</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>this.totalTasks = totalTasks;</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>this.crawlQueue = Collections.synchronizedList(new ArrayList<String>());</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>this.crawlQueue.add(url);</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span></blockquote>
</blockquote>
<ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> Resource Offers</span></li>
<ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> A resource offer can be resources like CPU, memory etc. From the offers list, get the scalar value of the resources. We need to give our requirements of resources for the tasks while setting the task info.</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">for (Protos.Offer offer : list) {</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>List<Protos.TaskInfo> taskInfoList = new ArrayList<Protos.TaskInfo>();</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>double offerCpus = 0;</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>double offerMem = 0;</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>for (Protos.Resource resource : offer.getResourcesList()) {</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>if (resource.getName().equals("cpus")) {</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>offerCpus += resource.getScalar().getValue();</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>} else if (resource.getName().equals("mem")) {</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>offerMem += resource.getScalar().getValue();</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>LOGGER.info("Received Offer : " + offer.getId().getValue() + " with cpus = " + offerCpus + " and mem ="</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>+ offerMem);</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Create task ID.</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">Protos.TaskID taskID = Protos.TaskID.newBuilder().setValue(Integer.toString(launchedTasks++)).build();</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Create task info by setting task ID, adding resources, setting data and setting executor.</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">Protos.TaskInfo pinUserProfileTaskInfo = Protos.TaskInfo.newBuilder().setName("task " + taskID.getValue())</span><span style="font-family: Courier New, Courier, monospace;">.setTaskId(taskID).setSlaveId(offer.getSlaveId())</span><span style="font-family: Courier New, Courier, monospace;">.addResources(Protos.Resource.newBuilder().setName("cpus")</span><span style="font-family: Courier New, Courier, monospace;">.setType(</span><span style="font-family: Courier New, Courier, monospace;">Protos.Value.Type.SCALAR)</span><span style="font-family: Courier New, Courier, monospace;">.setScalar(Protos.Value.Scalar</span><span style="font-family: Courier New, Courier, monospace;">.newBuilder().setValue(</span><span style="font-family: Courier New, Courier, monospace;">CPUS_PER_TASK)))</span><span style="font-family: Courier New, Courier, monospace;">.addResources(Protos.Resource.newBuilder().setName("mem")</span><span style="font-family: Courier New, Courier, monospace;">.setType(</span><span style="font-family: Courier New, Courier, monospace;">Protos.Value.Type.SCALAR)</span><span style="font-family: Courier New, Courier, monospace;">.setScalar(Protos.Value.Scalar</span><span style="font-family: Courier New, Courier, monospace;">.newBuilder()</span><span style="font-family: Courier New, Courier, monospace;">.setValue(MEM_PER_TASK)))</span><span style="font-family: Courier New, Courier, monospace;">.setData(ByteString.copyFromUtf8(crawlQueue.get(0)))</span><span style="font-family: Courier New, Courier, monospace;">.setExecutor(Protos.ExecutorInfo.newBuilder(</span><span style="font-family: Courier New, Courier, monospace;">pinUserProfileExecutor))</span><span style="font-family: 'Courier New', Courier, monospace;">.build();</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> Launch the tasks through the SchedulerDriver.</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">...<span class="Apple-tab-span" style="white-space: pre;"> </span></span> <span style="font-family: Courier New, Courier, monospace;">taskInfoList.add(pinUserProfileTaskInfo);</span><span style="font-family: Courier New, Courier, monospace;">taskInfoList.add(pinUserBoardTaskInfo);</span><span style="font-family: Courier New, Courier, monospace;">}</span><span style="font-family: Courier New, Courier, monospace;">schedulerDriver.launchTasks(offer.getId(), taskInfoList);</span></blockquote>
</blockquote>
<ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> Status update</span></li>
</ul>
<span style="font-family: 'Trebuchet MS', sans-serif;">This is invoked when the status of a task has changed ie., a slave is lost and so the task is lost, a task finishes and an executor sends a status update saying so.</span><br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">@Override public void statusUpdate(SchedulerDriver schedulerDriver, Protos.TaskStatus taskStatus) {</span><span style="font-family: Courier New, Courier, monospace;">...</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Stop the SchedulerDriver if tasks are finished</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span class="Apple-tab-span" style="font-family: 'Trebuchet MS', sans-serif; white-space: pre;"> </span><span style="font-family: Courier New, Courier, monospace;">if (taskStatus.getState() == Protos.TaskState.TASK_FINISHED) {</span><span class="Apple-tab-span" style="font-family: 'Courier New', Courier, monospace; white-space: pre;"> </span><span style="font-family: 'Courier New', Courier, monospace;">finishedTasks++;</span><span class="Apple-tab-span" style="font-family: 'Courier New', Courier, monospace; white-space: pre;"> </span><span style="font-family: 'Courier New', Courier, monospace;">LOGGER.info("Finished tasks : " + finishedTasks);</span><span class="Apple-tab-span" style="font-family: 'Courier New', Courier, monospace; white-space: pre;"> </span><span style="font-family: 'Courier New', Courier, monospace;">if (finishedTasks == totalTasks) {</span><span class="Apple-tab-span" style="font-family: 'Courier New', Courier, monospace; white-space: pre;"> </span><span style="font-family: 'Courier New', Courier, monospace;">schedulerDriver.stop();</span><span class="Apple-tab-span" style="font-family: 'Courier New', Courier, monospace; white-space: pre;"> </span><span style="font-family: 'Courier New', Courier, monospace;">}</span><span class="Apple-tab-span" style="font-family: 'Courier New', Courier, monospace; white-space: pre;"> </span><span style="font-family: 'Courier New', Courier, monospace;">}</span></blockquote>
</blockquote>
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> Abort the SchedulerDriver if the tasks are killed, lost or failed</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span class="Apple-tab-span" style="font-family: 'Trebuchet MS', sans-serif; white-space: pre;"> </span><span style="font-family: Courier New, Courier, monospace;">if (taskStatus.getState() == Protos.TaskState.TASK_FAILED<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>|| taskStatus.getState() == Protos.TaskState.TASK_KILLED<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>|| taskStatus.getState() == Protos.TaskState.TASK_LOST) {<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>LOGGER.error("Aborting because the task " + taskStatus.getTaskId().getValue() + " is in unexpected state : "<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>+ taskStatus.getState().getValueDescriptor().getName() + "with reason : " + taskStatus.getReason()<br /><span class="Apple-tab-span" style="white-space: pre;"> </span> .getValueDescriptor()<br /><span class="Apple-tab-span" style="white-space: pre;"> </span> .getName()<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>+ " from source : " + taskStatus.getSource().getValueDescriptor().getName() + " with message : "<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>+ taskStatus.getMessage());<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>schedulerDriver.abort();<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span></blockquote>
</blockquote>
<ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;"> Framework Message</span></li>
</ul>
<span style="font-family: 'Trebuchet MS', sans-serif;">This is invoked when an executor sends a message.</span><br />
<ul><ul>
<li><span style="font-family: 'Trebuchet MS', sans-serif;">Handle your message</span></li>
</ul>
</ul>
<blockquote class="tr_bq">
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>@Override public void frameworkMessage(SchedulerDriver schedulerDriver, Protos.ExecutorID executorID,</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Protos.SlaveID slaveID, byte[] bytes) {</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>String data = new String(bytes);</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>System.out.println(data);</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>LOGGER.info("User Profile Information : " + data);</span><span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span></blockquote>
</blockquote>
<span style="font-family: Trebuchet MS, sans-serif;">Complete code is available <a href="https://github.com/SwathiMystery/mesos-pinspider">here</a> with the instructions to run and sample output.</span><br />
Happy Learning! :)Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0Los Angeles, CA, USA34.0522342 -118.243684933.2099567 -119.5345784 34.8945117 -116.95279140000001tag:blogger.com,1999:blog-6187879032785791598.post-46985957125851608882014-12-20T17:38:00.002+05:302014-12-20T17:38:44.049+05:30Introduction and getting started with Apache Mesos<h3>
<span style="font-family: Trebuchet MS, sans-serif;">Introduction to Apache Mesos</span></h3>
<span style="font-family: Trebuchet MS, sans-serif;">In this era of distributed computing, where we spin up clusters for Hadoop, Storm, Jenkins, Cassandra, etc separately, we are not making effective use of the resources. There would be long pauses in the cluster after entering a burst of information, thus making it very in-efficient. </span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Now, what if all these frameworks shared the same set of machines and resources, then small slices of time spent waiting for some resources could be granted to other frameworks. This is the concept of Time Sharing. </span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Apache Mesos is a datacenter operating system and it shares the same philosophy of time-sharing. Mesos is called a datacenter because it hosts different frameworks under a single roof. It is called an operating system because shares many concepts of Linux.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">1. <b>Isolation</b> : Linux creates isolation through processes where, each of these processes has its own file descriptors and its own address space. This is achieved by Linux Containers (wiki:<a href="https://en.wikipedia.org/wiki/LXC">LXC</a>) in Mesos</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">2. <b>Process Scheduler</b> : The processes have accesses to the system resources by balancing the work loads across multiple computing resources, thereby optimizing resources, maximize the throughput, minimizing the response time and avoiding the overhead by any one resource. There are various scheduling algorithms to execute more than one process at a time (wiki:<a href="https://en.wikipedia.org/wiki/Computer_multitasking">Multitasking</a>) and also transmit multiple data streams simultaneously across a single physical channel(wiki:<a href="https://en.wikipedia.org/wiki/Multiplexing">Multiplexing</a>). Mesos uses such scheduling algorithms.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">3. <b>Common Infrastructure</b> : Linux has a set of calls irrespective of filesystems, drivers etc. Similarly, Mesos has a common set of calls which helps in the execution of tasks.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">4.<b> Package Manager</b> : Linux has apt-get, aptitude, synaptic, yum etc that helps in the automation of the process of installing, upgrading, configuring, and removing software. Similarly, Mesos has a recent support for Docker(wiki:<a href="https://en.wikipedia.org/wiki/Docker_(software)">Docker</a>)</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Traditionally, distributed systems has 2 components in a non-peer to peer systems.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">1. <b>Coordinator</b> : Generate tasks, send the tasks to worker and receive results from Worker.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">2.<b> Worker</b> : Execute the tasks and send the status and results back to Worker.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">With Mesos, there are three levels <b>Coordinator</b>,<b> Mesos master </b>and<b> Mesos slaves</b> where coordinator negotiates with mesos master and then master decides on partitioning the cluster to distribute the tasks. Thus, we can schedule jobs across the machines, thereby running hadoop, cassandra, spark etc.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">All the distributed systems that run on Mesos are called applications or frameworks and the coordinator is called as scheduler in Mesos vocabulary.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<h3>
<span style="font-family: Trebuchet MS, sans-serif;"><b>How does Mesos work?</b></span></h3>
<span style="font-family: Trebuchet MS, sans-serif;">In summary, Mesos works on a request/offer based model. Whenever, you want to run a job, you send a request. These requests are simplified subset of specification like number of GPUs, RAM etc, at that point of time. Mesos, checks for the request specification and it will reply back with the resource offers of what resources are available on a set of machines. This is non-blocking and has two level of scheduling : Offering and Scheduling.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><b>Mesos master</b>: Control the resource allocation to the schedulers</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><b>Scheduler</b>: Uses the resource offers to decide which tasks to run and which one to run next.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">More information on Mesos architecture is here : <a href="http://mesos.apache.org/documentation/latest/mesos-architecture/">Mesos Architecture</a></span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<h3>
<span style="font-family: Trebuchet MS, sans-serif;"><b>Getting started with mesos</b></span></h3>
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">1. Download the tarball from the </span><span style="font-family: Courier New, Courier, monospace;"><a href="http://www.apache.org/dyn/mirrors/mirrors.cgi/mesos/0.21.0/">Mirror</a></span><span style="font-family: Trebuchet MS, sans-serif;"> Apache Mesos v0.21.0 and untar it.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">tar -zvxf mesos-0.21.0.tar.gz </span><br />
<span style="font-family: Courier New, Courier, monospace;">cd mesos-0.21.0/</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">2. Install the dependencies</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">sudo apt-get update</span><br />
<span style="font-family: Courier New, Courier, monospace;">sudo apt-get install build-essential openjdk-6-jdk python-dev python-boto libcurl4-nss-dev libsasl2-dev maven libapr1-dev libsvn-dev</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">3. Building Mesos</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Please make sure it has appropriate permissions while building.</span><br />
<span style="font-family: 'Trebuchet MS', sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">mkdir build</span><br />
<span style="font-family: Courier New, Courier, monospace;">cd build</span><br />
<span style="font-family: Courier New, Courier, monospace;">../configure</span><br />
<span style="font-family: Courier New, Courier, monospace;">make</span><br />
<span style="font-family: Courier New, Courier, monospace;">make check </span><br />
<span style="font-family: Courier New, Courier, monospace;">make install</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Start Mesos Master</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Start Mesos Slave</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">./bin/mesos-slave.sh --master=127.0.0.1:5050</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Web UI</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">http://127.0.0.1:5050</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Trebuchet MS, sans-serif;">Running a test framework in Java</span><br />
<span style="font-family: Trebuchet MS, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">./src/examples/java/test-framework 127.0.0.1:5050</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<div class="separator" style="clear: both; text-align: center;">
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirTguizq_SefhnnfyGoFn0IZaWC_F2b8CzFVyimVTtRYPy0DgfNFp1zAYfe3H3hUZYQpY-OjhWMAOcey3O7qcNqFotaUqsPL21HX0RYk-1rxvh4uaDgUFs1pgcZwcdLrYaVkGmBAPWGy9N/s1600/mesos_home.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirTguizq_SefhnnfyGoFn0IZaWC_F2b8CzFVyimVTtRYPy0DgfNFp1zAYfe3H3hUZYQpY-OjhWMAOcey3O7qcNqFotaUqsPL21HX0RYk-1rxvh4uaDgUFs1pgcZwcdLrYaVkGmBAPWGy9N/s1600/mesos_home.png" height="350" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Mesos Home</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuKZ9hqj5is9eHGHq-9bZ9QCkn43eRsos1JooGuSyQsBlFEgZc6EzR2e-vdpEhyphenhyphenZAC374WNxFePRXFAgClXN7JBewD13FKyv79W9DXW7j35mNq26g_mjy3TOyBEpVWEOhSVeN9rH8hJGy5/s1600/mesos_frameworks.png" imageanchor="1" style="margin-left: auto; margin-right: auto; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuKZ9hqj5is9eHGHq-9bZ9QCkn43eRsos1JooGuSyQsBlFEgZc6EzR2e-vdpEhyphenhyphenZAC374WNxFePRXFAgClXN7JBewD13FKyv79W9DXW7j35mNq26g_mjy3TOyBEpVWEOhSVeN9rH8hJGy5/s1600/mesos_frameworks.png" height="306" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Mesos Frameworks</td></tr>
</tbody></table>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirTguizq_SefhnnfyGoFn0IZaWC_F2b8CzFVyimVTtRYPy0DgfNFp1zAYfe3H3hUZYQpY-OjhWMAOcey3O7qcNqFotaUqsPL21HX0RYk-1rxvh4uaDgUFs1pgcZwcdLrYaVkGmBAPWGy9N/s1600/mesos_home.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><br /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirTguizq_SefhnnfyGoFn0IZaWC_F2b8CzFVyimVTtRYPy0DgfNFp1zAYfe3H3hUZYQpY-OjhWMAOcey3O7qcNqFotaUqsPL21HX0RYk-1rxvh4uaDgUFs1pgcZwcdLrYaVkGmBAPWGy9N/s1600/mesos_home.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKGzMr0GyMQMeVdGpAd-jDmQoRXMMjHO0K0Os2_PPmIA6hU0UIQOZY-9VE4S5Y45HkcVa0ndwl0p8on_TfnuaoWx7WvGNP4QvufKU2uWR9AScYPp7w0g4UEx84cmtbINkqrpZUOhvzR-FW/s1600/mesos_task_execution.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKGzMr0GyMQMeVdGpAd-jDmQoRXMMjHO0K0Os2_PPmIA6hU0UIQOZY-9VE4S5Y45HkcVa0ndwl0p8on_TfnuaoWx7WvGNP4QvufKU2uWR9AScYPp7w0g4UEx84cmtbINkqrpZUOhvzR-FW/s1600/mesos_task_execution.png" height="310" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Mesos executor tasks</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgc1vjJ1b33-JvkZiHDEh3kIBK254r5Gb_kPruzqxOyYHr5yItuFdh2bieC5CHSKU25RQR75KESWBwq1qyT6bwlSQvmOKrv1_hCKlFvVnt9qWdTbCzEGI34QAZ09HrSgpHzKYJlC5avH5ep/s1600/mesos_slaves.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgc1vjJ1b33-JvkZiHDEh3kIBK254r5Gb_kPruzqxOyYHr5yItuFdh2bieC5CHSKU25RQR75KESWBwq1qyT6bwlSQvmOKrv1_hCKlFvVnt9qWdTbCzEGI34QAZ09HrSgpHzKYJlC5avH5ep/s1600/mesos_slaves.png" height="336" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Mesos Slave Nodes</td></tr>
</tbody></table>
<br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<div class="separator" style="clear: both; text-align: center;">
</div>
<span style="font-family: 'Trebuchet MS', sans-serif;">In the next post, let us see how to build our own distributed framework on Apache Mesos.</span><br />
<span style="font-family: Trebuchet MS, sans-serif;">Happy Learning! :)</span><br />
<br />Anonymoushttp://www.blogger.com/profile/05847487127434744704noreply@blogger.com0Los Angeles, CA, USA34.0522342 -118.243684933.2099567 -119.5345784 34.8945117 -116.95279140000001