login
Deliverables

D6.2 (WP6): Scalability Case Studies: Scalable Sim-Diasca for the Blue Gene

Submission date: March 27, 2015

Purpose: To study how to apply RELEASE technologies to scale substantial distributed Erlang systems to large scale architectures, i.e. architectures with hundreds of hosts and up to 10,000 cores.

Results: The main results of this deliverable are as follows.

  • We have devised and implemented City-example, a full, relevant simulation for the main case study of this deliverable, which is the scalability of a demanding Erlang application, a simulation engine named Sim-Diasca.
  • We have investigated the reliable scalability of two distributed Erlang and SD Erlang benchmarks, Orbit and ACO, on up to 256 hosts (6144 cores).
  • We have investigated the scalability and performance of the Sim-Diasca City instance using both conventional tools, and the new BenchErl and Percept2 RELEASE tools.
  • We have performed additional scalability investigations using RELEASE technologies, which included demonstrating the deployment of Sim-Diasca with the WombatOAM load management tool; the port of the Erlang runtime onto the Blue Gene/Q supercomputer; and an outline design of how SD Erlang could be applied to Sim-Diasca.

Conclusion: We have shown the challenges of significant scalability studies: even simply being able to run non-trivial applications in larger settings is dificult, not to mention that the benchmarking tools, and our ability to synthesize the resulting data, must scale as well. We established some common architectural changes that distributed Erlang applications may adopt in order to alleviate some typical scalability bottlenecks.


D3.5 (WP3): SD Erlang Performance Portability Principles

Submission date: March 22, 2015

Purpose: Design and validate performance portability principles for SD Erlang.

Results: The main results of this deliverable are as follows.

  • We have designed and implemented a library which enables Erlang nodes to examine attributes of other nodes in a distributed system, facilitating the choice of a node which has suitable properties to support the spawning of a particular process.
  • We have tested this library with a distributed Erlang application, showing that the use of attributes can lead to improved performance.
  • We have designed and implemented a library which uses an abstract model of communication times between Erlang nodes in order to deploy distributed Erlang applications in such a way as to minimise overheads due to communication delays.
  • A careful examination of communication times in several multi-node systems on different scales suggests that our abstract model provides a good description of real communication latencies.
  • Earlier deliverables in D3.5 described SD Erlang, which extends Erlang with a mechanism for partitioning sets of Erlang nodes into s_groups in order to improve communication efficiency. Our libraries are not closely coupled with the new features introduced in SD Erlang, but admit easy interoperability with s_groups. The work described here is perhaps best viewed as a complement to SD Erlang rather than an extension of it.

Conclusion: We have designed and implemented libraries which provide methods for implementing distributed Erlang applications in such a way as to obtain efficient performance without requiring detailed information about system structure to be coded into the application. We have carried out experiments which suggest that our libraries do indeed enable programmers to achieve efficient performance in a portable way.


D6.7 (WP6): Scalability and Reliability for a Popular Actor Framework

Submission date: March 21, 2015

Purpose: To investigate the feasibility and limitations of adding SD Erlang scalability / reliability constructs, and design patterns, to a popular Actor framework for a dominant language.

Results: The main results of this deliverable are as follows.

  • We have implemented the Orbit benchmark in Cloud Haskell and showed that it has similar scalability constraints as distributed Erlang.
  • With a view to discovering the wider applicability of our results, we explore the application of SD Erlang techniques to Scala and Akka. Their explicit placement of actors and management of connections means that the s_group construct cannot be directly applied.
  • We outline some approaches applying SD Erlang features, i.e. s_groups and semi-explicit pro- cess placement, directly in Cloud Haskell, indirectly in Scala and Akka.

Conclusion: We analyse the architecture of distributed applications in Cloud Haskell, Scala, and Akka. We conclude that applying SD Erlang approaches to reliable scalability could be directly applied to Cloud Haskell, but only indirectly applied to Scala or Akka, i.e. by introducing a library that will collect and update information about attributes of connected nodes.


D2.4 (WP2): Robust Scalable Erlang Virtual Machine

Submission date: March 17, 2015

Purpose: To describe additions and improvements made to key components of the Erlang Virtual Machine that improve its robustness, scalability and responsiveness on big multicore machines.

Results: The main results presented in this deliverable are:

  • Two new additions to the Erlang Virtual Machine (VM), namely a scheduler utilization balancing mechanism and the ability to interrupt long running garbage collecting BIFs, that improve the responsiveness of the Erlang/OTP system.
  • Changes to the memory carrier migration mechanism, a new super carrier memory allocation scheme, and the new scheme for time management in the VM.
  • A detailed description of the implementation of the Erlang Term Storage (ETS) and a study of the scalability and performance improvements to ETS across Erlang/OTP releases.
  • New designs for ETS' implementation that improve its performance and scalability further.
  • Scalable tracing support for profiling and monitoring SD Erlang applications.
  • Scalability measurements of the VM improvements in Erlang/OTP releases during the project.

Conclusion: Additions and changes made during the RELEASE project have allowed many key components of the Erlang runtime system to become more efficient and scalable and have eliminated bottlenecks that previously hindered the scalability and responsiveness of its VM. Further improvements are possible by incorporating into the Erlang/OTP distribution components that currently exist in development branches of the system or in prototypes.


D5.4 (WP5): Interactive SD Erlang Debugging

Submission date: February 28, 2015

Purpose: This deliverable provides functionality to support debugging and monitoring systems written in SD Erlang, complementing the tools already provided in Erlang/OTP and in earlier RELEASE deliverables.

Results: The main results of this deliverable are as follows.

  • We have developed a new tool, SD-Mon, designed specifically to monitor SD-Erlang systems (Tasks 5.6, 5.7).
  • We have made a thorough revision of the Devo tool, so that it now provides a more scalable visualisation of SD-Erlang systems (Task 5.7).
  • We have provided patches for the Erlang/OTP distribution to deliver greater tracing scalability.

Conclusion: We have augmented Erlang with tools – Percept2, Devo and SD-Mon – designed to support the development, monitoring and debugging the ‘Scalable Distributed’ aspects of SD Erlang systems.


D4.5 (WP4): Scalable Infrastructure Performance Evaluation report

Submission date: February 16, 2015

Purpose: To evaluate the performance of the scalable infrastructure (called WombatOAM) developed as part of the RELEASE project.

Results: In the course of this deliverable we:

  • described how we improved the infrastructure to be more scalable, and
  • presented the performance evaluation of the scalable infrastructure.

Conclusion: In this document we have presented an adequate evaluation of the scalable infrastructure focusing on the performance achieved, on the intrusiveness of WombatOAM and on the ability to scale out. We have also given an insight into the final efforts aiding in improving the scalability and robustness capabilities of the infrastructure.


D6.6 (WP6): Capability-matching Heterogeneous Deployment of a Load Testing Tool

Submission date: October 21, 2014

Purpose: To validate the ability of the prototype being developed in the Scalable Virtualisation Infrastructure work package (WP4) to deploy a real-world distributed Erlang application using capability-matching deployment.

Results: The main result presented in this deliverable is the deployment of Megaload -- a scalable, high performance load-testing tool written in Erlang -- on several deployment domains backed by Amazon EC2 and the HP Public Cloud, using WombatOAM Orchestration's capability-matching deployment feature, which was developed in RELEASE deliverable D4.4.

Conclusion: Capability-matching deployment means that when deploying Erlang nodes into a heterogeneous cloud, the user will specify deployment demands, and our prototype called WombatOAM will select the most suitable deployment domain for the nodes based on how well the demands provided by the user match the capabilities of the different domains. In this deliverable, we demonstrated that using capability-matching deployment and letting WombatOAM choose the best domain for the nodes is just as seamless as explicitly specifying the domains to WombatOAM.


D7.5 (WP7): RELEASE Open Day – 11 June 2014

Submission date: August 22, 2014

Purpose: A summary of the activities undertaken as part of the dissemination and exploitation activity during the RELEASE final project workshop, held in conjunction with the Erlang User Conference from 9 to 11 June 2014 in Stockholm, Sweden.

Results: Combining talks and tutorials specific to the RELEASE project with other invited speakers relevant to our areas of research has allowed us to attract a substantial audience that represents a mix of universities and industry, maximising the exposure of the results we have achieved to date and helping us generate interest and cooperative interaction in the future. Several other STREP FP7 projects were also represented at the workshop, contributing their own results and experiences.

Conclusion: With 350 attendees at the main event and nearly 70 additional attendees at the tutorial day, the strategy of conducting the RELEASE final project workshop together with the most important event in the Erlang calendar proved vital to maximising exposure to a diverse and receptive demographic of industry professionals and interested parties. Feedback from the tutorials show a highly positive level of interest in the use of the tools and language extensions, achieving our aims of securing new adopters, while also allowing us to gain useful feedback and consider future activities and improvements. The event confirmed the validity and significance of the RELEASE project's vision and the impact it will continue to have in revolutionising the future of multicore computing.


D3.4 (WP3): Scalable Reliable OTP Library Release

Submission date: September 23, 2014

Purpose: Design, implement and validate a set of reusable reliable massively parallel programming patterns as a scalable, performance portable OTP library.

Results: The main results of this deliverable are as follows

  • We have discussed scalability recipes of distributed Erlang applications using four benchmarks.
  • We have provided scalability and reliability principles, application methodology, and introduced generic patterns.
  • We have briefly outlined performance portability under development for deliverable D3.5.

Conclusion: We have identified generic patterns, introduced methodology for scalable reliable SD Erlang applications, and both designed and implemented them in a number of SD Erlang benchmarks.


D5.3 (WP5): Systematic Testing and Debugging Tools

Submission date: June 30, 2014

Purpose: To describe techniques and tools developed within RELEASE for the systematic testing and debugging of Erlang applications.

Results: The main results presented in this deliverable are:

  • an overview of the refactoring assistance provided within Wrangler to guide the application of refactorings to particular systems based on monitoring data and its analysis.
  • the architecture and the implementation technology of Concuerror, a systematic testing tool for concurrent Erlang programs.

Both tools are publicly available and have been presented to the Erlang programming community on various occasions during the lifetime of the project.

Conclusion: The deliverable has provided tool support for the development of Erlang systems with a focus on SD Erlang, and Erlang concurrency.


D7.4 (WP7): RELEASE project Workshop report

Submission date: January 02, 2014

Purpose: This deliverable describes the workshop we had on the 6th of December, 2013.

Results: Work was summarized during the presentations and tutorials on the workshop which helps to exploit the result of the project.

Conclusion: As a result of the project language extension and tools were developed, which are suitable for the purpose of RELEASE project.


D7.3 (WP7): Exploitation Plan for RELEASE

Submission date: October 14, 2013

Purpose: Summarize the exploitation activities to be done by the partners during the project

Results:

  • We have conducted a market analysis by providing an overview how different programming languages support writing scalable, distributed software.
  • We have summarized the advantages of the Erlang programming language
  • We have identified the exploitable results of the RELEASE project
  • We have created a SWOT analysis to investigate market findings at the current stage of the project

Conclusion: There is a market need to strengthen the language support to write disturbed software.


D3.3 (WP3): Scalable SD Erlang Reliability Model

Submission date: October 17, 2013

Purpose: Implement and validate a scalable reliability model for SD Erlang.

Results: The main results of this deliverable are as follows:

  • We have implemented a node recovery mechanisms for SD Erlang s_groups.
  • We have minimised the probability s_groups with the same name occurring, and ensured correct node behaviour in case duplicate names occur.
  • We have provided an s_group operational semantics together with a QuickCheck correctness check of the s_group module functions.

Conclusion: We have implemented a node recovery mechanism, and both speci ed and validated correctness of s_group SD Erlang functions.


D5.2 (WP5): Online SD Erlang Profiling and Refactoring Tools

Submission date: October 14, 2013

Purpose: To design and deliver an online visualisation tool to support the interactive exploration of the data from a system during execution; to extend the Wrangler refactoring tool to support the refactoring from Erlang/OTP to SD Erlang.

Results: The main results of this deliverable, reported here and in three published papers, are:

  • A prototype online visualisation tool to support the interactive exploration of runtime profiling data.
  • Various improvements to the DTrace/SystemTap backend for online and online profiling, as well as the design and implementation of DTrace/SystemTap probes for SD Erlang.
  • Various improvements to the online profiling tool Percept2.
  • Extension of the Wrangler refactoring tool for refactoring from Erlang/OTP to SD Erlang.

Conclusion: We have delivered a first set of tools to support the online profiling for SD Erlang and refactoring from Erlang/OTP to SD Erlang.


D2.3 (WP2): Prototype Scalable Runtime System Architecture

Submission date: September 30, 2013

Purpose: To describe additions and improvements made to key components of the Erlang Virtual Machine that improve its performance, scalability and responsiveness on big multicore machines.

Results: The main results presented in this deliverable are:

  • Efficient functionality that allows the Erlang runtime system to determine thread progress.
  • Infrastructure for memory allocation and deallocation by multiple scheduler threads that requires less locking and support for memory carrier migration.
  • Better organizations of process and port tables.
  • More scalable ways to manage processes and handle port signals.
  • Non-blocking mechanisms for code loading and trace setting.
  • An algorithm that preserves term sharing in copying and message passing and its low-level implementation on the Erlang VM.

Conclusion: The set of changes described in this deliverable have allowed many of the key components of the Erlang runtime system to become more ecient and scalable and have eliminated many of the bottlenecks that hindered the scalability and responsiveness of its VM.


D6.4 (WP6): Homogeneous Deployment of a Load Testing Tool

Submission date: May 31, 2013

Purpose: To validate that the prototype developed in WP4 is capable of deploying a real-world Erlang application on a homogeneous virtual infrastructure.

Results: The main results of the deliverable are as follows:

  • The development of the WP4 prototype has been aligned with the requirements of a real-world distributed Erlang application.
  • It has been validated that the developed prototype is capable of deploying a real-world distributed Erlang application on a homogeneous virtual infrastructure.

Conclusion: The developed prototype, WombatOAM, has been used to successfully deploy a real-world Erlang application, Megaload, on a homogenous virtual infrastructure, Amazon EC2. Upcoming deliverables will build upon the results presented in this deliverable and will focus on demonstrating the ability of the developed prototype not only to deploy applications on a heterogeneous virtual infrastructure, but also to carry out a deployment driven by matching deployment requirements with existing cluster capabilities.


D6.3 (WP6): Distributed Erlang Component Ontology

Submission date: September 30, 2013

Purpose: To define a common ontology and a set of concepts for systems in the Erlang domain. This is to remove ambiguities when referring to an Erlang system, to its components, to the relations between its components and to the operations which can be performed on them.

Results: The main results of the deliverable are as follows:

  • the first ontology for Erlang systems.
  • a description of how the ontology relates to Wombat and SD-Erlang as well as what the relationship between Wombat and SD-Erlang is.

Conclusion: With this initial DECO (Distributed Erlang Component Ontology) in place we have a foundation for talking with less ambiguity about Erlang systems consisting of heterogeneous Erlang releases. We have also described the relationship between Wombat, SD-Erlang and DECO. DECO has been quite useful in understanding where the differences and similarities between Wombat and SD-Erlang are, and also how to integrate SD-Erlang into Wombat. Application outside ESL is still pending and will be started in the near future.


D6.1 (WP6): Reliable Sim-Diasca Simulation

Submission date: July 22, 2013

Purpose: The overall objective is to prepare the Sim-Diasca simulation engine for the use of the larger-scale computing infrastructures targeted by RELEASE; such massive number of cores requires that various reliability mechanisms be added. This includes notably the “k-crash resilience” feature described here.

Results: Now, since the 2.2.0 version of Sim-Diasca, the user is able to specify a value k corresponding to the maximum number of computing hosts whose failure in the course of simulation may be overcome.

Conclusion: Even if some restrictions still apply (some of which being scheduled for removal in subsequent Sim-Diasca versions), at the application-level the engine is ready to take advantage of massive resources with an increased reliability. If additionally taking into account the Erlang-level improvements brought by the other work-packages of RELEASE, we believe the whole solution will be soon fully able to tackle the future demanding larger-scale simulation cases we can currently foresee.


D2.2 (WP2): Prototype Scalable Erlang VM Release

Submission date: April 8, 2013

Purpose: To describe the implementation of key components of the prototype scalable Erlang Virtual Machine release: the implementation of its scalable Erlang Term Storage, its efficient tracing support for profiling and monitoring, and the preliminary port of Erlang VM on a Blue Gene/Q.

Results: The main results of this deliverable are

  • A scalable implementation of the Erlang Term Storage (ETS) mechanism that is already included in Erlang/OTP R16B (released in late February 2013) and the description of alternative designs, which currently exist as prototypes, that will increase the scalability of ETS further.
  • An efficient tracing support based on DTrace/SystemTap that Erlang/OTP offers since release R15B01 (April 2012) for profiling and monitoring Erlang applications.
  • A preliminary port of the Erlang VM on the Blue Gene/Q and the description of its status.

Conclusion: The two key components of the scalable Erlang VM release that we describe in this document, scalable ETS and efficient tracing support, are already part of Erlang/OTP and used by the Erlang community. The Blue Gene/Q port of Erlang/OTP is complete and fully functional on the front end nodes, but significant work still remains for the Erlang VM to execute efficiently on the compute nodes of the machine.


D3.2 (WP3): Scalable SD Erlang Computation Model

Submission date:August 8, 2013

Purpose: To implement and validate a scalable computation model and a high-level process placement control for Scalable Distributed (SD) Erlang.

Results: The main results of this deliverable are as follows

  • To improve the scalability of Distributed Erlang we have split the global namespace and reduced the number of connections following the design presented in deliverable D3.1. That is we group Erlang nodes into s_groups where each s_group has its own namespace.
  • We have implemented sixteen functions to support node grouping and name registration.
  • To support semi-explicit placement we have added node attribute parameter and implemented choose nodes/1 function. The function returns a list of nodes that satisfy given restrictions.

Conclusion: We have implemented and started the validation of a scalable computation model and a high-level process placement control for SD Erlang.


D4.3 (WP4): Heterogeneous super-cluster Infrastructure

Submission date: January 9, 2013 / July 2, 2014

Purpose: To build and describe a framework that enables automatic deployment and monitoring of an Erlang/OTP application into a given number of Erlang nodes running in a heterogeneous cloud environment.

Results: The main results of this deliverable are

  • We revisited the CCL architecture presented in deliverable D4.2, moving from a command-line tool to a fully-blended virtualization infrastructure.
  • We moved from a homogenous deployment environment to a heterogeneous one.
  • We presented the RELEASE project at two major conferences (HiPEAC and Codemotion), collaborated with a team from a parallel FP7 project (Prowess) and started a thesis project about the implementation of a CCL plugin.

Conclusion: In this deliverable we have introduced a revised architecture for the developed prototype such that we are now capable of deploying Distributed Erlang applications on different Cloud infrastructures. The introduced architectural changes pave the way for adding control and monitoring functionalities which will enable users to deploy Distributed Erlang applications in a heterogeneous infrastructure, based on capability pro file matching.


D5.1 (WP5): Offline SD Erlang Profiling Tool

Submission date:September 30, 2012

Purpose: To design and deliver an offline profi ling tool reporting monitoring and visualization results in static reports and interactively.

Results: The main results of this deliverable are

  • Percept2, a version of the Percept tool enhanced with functionality to support multicore and distributed systems, and also refactored to support scalability.
  • Extension of the built-in Erlang tracing mechanism to support scalability.
  • Sampling-based profi ling of Erlang systems.
  • Extension of the DTrace/SystemTap system for Erlang including new probes, and reports.

Conclusion: We have delivered a fi rst set of tools to support pro filing for SD Erlang.


D7.2 (WP7): Dissemination and Collaboration Plan

Submission date:September 30, 2012

Purpose: To identify and outline the main methods used to disseminate results from the project to the members of the public by means of academic and commercial activities. Also to identify other Framework Programme projects with similar research areas and how RELEASE will work with them.

Results:

  • We have developed, and are pursuing, a multi-strand dissemination policy.
  • We have established a process for evolving the dissemination plan during the period of the project.
  • We are engaging the scientific community with publications, talks, interactions with related projects etc.
  • We are engaging the commercial sector with presentations at industry forums.
  • We are contributing to Open Source software.

Conclusion: We have developed, and are pursuing, a multi-strand and evolving dissemination plan. We are engaging the public via a website, press releases, posters etc. We are engaging the scientific community with publications, talks, interactions with related projects etc. We are engaging the commercial sector with presentations at commercial conferences and industry forums. We are contributing software, for example improving the Erlang/OTP framework with Virtual Machine upgrades, improved tools and a benchmark suite. We will continue to disseminate results through the described channels in the next year of the project and will update this document appropriately at the end of the second year.


D4.2 (WP4): Homogeneous Cluster Infrastructure

Submission date:July 16, 2012

Purpose: To build and describe a framework that enables automatic deployment of an Erlang/OTP application into a given number of Erlang nodes running in a homogeneous cloud environment.

Results: The main results of this deliverable are

  • We developed a Command Line Interface (CLI) which can be used to automatically deploy a given Distributed Erlang Application into a set of N machines running in the Amazon EC2 cloud environment.
  • We have run EC2 clusters of Erlang nodes within a secure environment, by using the Amazon Virtual Private Cloud (VPC) functionalities.

Conclusion: The approach we propose to automate deployment of an Erlang OTP application into a homogeneous cloud environment worked smoothly. We now face two major challenges, which will be tackled as part of the next deliverables: moving from a homogeneous environment to a heterogeneous one and adding cluster control and monitoring functionalities to the framework.


D3.1 (WP3): Scalable Reliable SD Erlang Design

Submission date:June 30, 2012

Purpose: To design Scalable Distributed (SD) Erlang, an extension to Distributed Erlang to allow scalability while preserving reliability.

Results: The main results of this deliverable are

  • We have analysed possible failures and hardware architecture trends.
  • We have analysed exemplar requirements for SD Erlang.
  • We have developed designs for minimising connectivity (scalable groups), implicit placement, and considered security implication.

Conclusion: We have designed Scalable Distributed Erlang.


D2.1 (WP2): Scalability Analysis

Submission date:June 30, 2012

Purpose: To study the scalability characteristics of a representative set of existing applications of Erlang running on the current virtual machine (VM) of Erlang/OTP.

Results: The main results of this deliverable are

  • The design and implemention of bencherl, a publicly available extensible benchmark suite suitable for studying the scalability of the Erlang/OTP system and its VM.
  • A study of the scalability characteristics of the current set of benchmarks in this suite across a number of implementation platforms and dimensions.

Conclusion: The scalability benchmark suite will help us prioritize changes and extensions in the architectural design of the runtime system and the VM of Erlang/OTP. It will also allow the Erlang community to have a standard place where applications can be contributed in order to study them and provide inspiration for changes in the VM (WP2), the language (WP3), and its tools (WP5).


D7.1 (WP7): Website and Initial Press Engagement

Submission date:December 31, 2011

Purpose: To establish website, domain, and make initial press engagement

Results: The main results of this deliverable are

  • We have purchased a domain, and established an outward facing website
  • We have established integrated Redmine and git repositories linked to the website to facilitate interaction between partners
  • We have issued several press releases and made announcements at international conferences

Conclusion: We have made a good start publicising the project to both general and technical audiences.