An approach is to use a peer to peer file transfer protocol, like BitTorrent,  to more efficiently distribute content to this large number of clients. Efficiency in our context is about reducing the time required to propagate the files, and limiting the peak network bandwidth usage.

The following are examples of the utilization of BitTorrent applied to datacenter automation:

• Twitter recently released an open source version of the bittorrent based system they use for release management, called Murder, more information is available in that blog entry.
• SystemImager supports a BitTorrent based transport option. A benchmark published on the SystemImager site shows that almost 1200 servers could be provisioned in 15mn.
• The Rocks Avalanche installer uses BitTorrent to distribute packages to cluster nodes. The Avalanche Installer allows almost identical install time for 1 node compared to 128 (12min vs. 15min) while implementing several throttling techniques.
• The CERN VM Kiosk is backed by BitTorrent and SCP (as seen in this presentation). As a side note, the CERN is also developing an image distribution based on SCP,  SCP Tsunami, borrowing some of the BitTorrent properties.

All of that is very well, but the deployment of BitTorrent in datacenters need to be done carefully. The following points have to be considered:

• BitTorrent protocol relies on a tracker service to maintain the list of peers. Clients will have to have access to this service at the start of the download. This tracker service has the potential of being a single point of failure.
• Datacenter topology has to be taken into consideration in order to optimize the bandwidth usage going across the core network layer and across datacenters.
• Initialization of the transfer requires the creation of a Seed. The location and number of initial seeds is critical to ensured the best efficiency.

This first post will focus on the first problem, the availability of the tracker. Other posts will address the two remaining problems.

Tracker Availability

There are basically two strategies to address the dependency of BitTorrent clients to the tracker. The first one, is to simply use the trackerless mode, relying on distributed hash tables (i.e. Kademlia), hence the name BitTorrent DHT. The second one is to use multiple trackers, either by simply using multi-tracker torrents or by implementing different distribution techniques .

As far as we know, murder, SystemImager or Rocks do not use BitTorrent DHT. This is however something that should be explored in this specific use case as the distance calculation in DHT could be modified to be topology aware as discussed in this paper or this presentation (distance calculation in key space is not  a representation of network or geographical distance). This would help solving the other challenge mentioned in this post.

Assuming that a tracker will be used, if just for priming the swarm, we need to explore the distribution options. We can consider two flavors: the first one creates a partition of the peer space, the second one creates a virtually centralized tracker, or HA tracker.

Tracker distribution

Our use case for BitTorrent is a bit different than the most notorious one, namely distributing legal or illegal files to internet population at large. In our case, the partitioning of the swarm is an interesting property as it could be used to contain traffic within a network domain, one of the other problems we have to address. Let’s explore how this would work.

A naive placement for the BitTorrent tracker can be described as follow:

The tracker is connected at the distribution layer level, like other infrastructure components would be. In this scenario, the clients will be configured with a single tracker. Clients in Site 1 and 2 will contact this single tracker, creating a single swarm.

This setup is problematic in multiple ways. In addition to the fact that this tracker is now a single point of failure, the Site 1 clients may potentially try to get files from peers not only in Site 1, but also in Site 2, creating traffic at the core network layer, and also cross datacenters. This is not unlike the inter ISP traffic generated by BitTorrent.

A better setup would be, at the minimum to deploy another tracker in Site 2, and have different torrent configurations for both sites, with the primary tracker being the one in the datacenter where the torrents are published.

This would mean for example a configuration like this :

Site 1 torrents : d['announce-list'] = [ [tracker1-s1], [tracker1-s2] ]

Site 2 torrents : d['announce-list'] = [ [tracker1-s2], [tracker1-s1] ]

With this setup, clients would first try the primary in their datacenter, and if it does not respond, try the one in the other datacenter.

Since it’s unlikely that the two trackers will be kept in sync, this would create split swarms, each sub swarm with the clients from a given datacenter.

If there were a large number of nodes within each distribution network, it could be envisioned to have one tracker per distribution “bubble”. The resulting topology would be like the following :

HA Tracker

Even though we have seen that distribution of trackers may be desirable, it remains that each distributed instance should be as available as possible. It does not seem that there is a standard defining tracker clustering or synchronization, but some tracker will implement one, like opentracker, which uses UDP multicast between members of the cluster. Then, in order to balance the load between members of the clusters, the multi-tracker configuration should be used, but with the multiple trackers specified in the same tier. If each tracker in the above topology was deployed in a cluster, we would have this kind of configuration in the torrent files, assuming a cluster of 2 nodes:

Site 1 / distribution 1 torrents : d['announce-list'] = [ [tracker1-s1.ds1, tracker2-s1.ds1], [tracker1-s2.ds1, tracker2.s2.ds1] ]

Clients will shuffle the list, try trackers one after the other while keeping track of success and failure to keep the same order for subsequent requests. This behavior is specific to BitTornado, and actual behavior of the selected client will have to be verified to avoid unexpected cross datacenter traffic or unbalance in the usage of the members of the cluster.

Conclusion

Because of the characteristics of each of the techniques mentioned in this section, it will be likely that a combination of partitioning, clustering and even DHT will have to be used. While not applied to the same problem space, this paper, is reaching a similar conclusion. Also, from the same paper, it is interesting to note that tracker availability is a real issue and should treated with care in the deployment of BitTorrent  for mission critical use cases.

The paper concludes that the only foolproof solution is to limit sharing with potentially hostile tenants:

A user might insist on using physical machines populated only with their own VMs and, in exchange, bear the opportunity costs of leaving some of these machines under-utilized. For an optimal assignment policy, this additional overhead should never need to exceed the cost of a single physical machine, so large users — consuming the cycles of many servers — would incur only minor penalties as a fraction of their total cost.
Regardless, we believe such an option is the only foolproof solution to this problem and thus is likely to be demanded by customers with strong privacy requirements.

I have one issue with this recommendation: the colocation of many VMs from the same tenant on fewer physical hosts is increasing the risk of having single points of failure. Assuming 8 small instances per physical machine (based on the document estimates), and given the default limit of 20 active VMs per account, most accounts will need less than 3 physical servers, limiting the spread across the availability zones. At that point the tradeoff will be between availability, security and cost.

The puppet community has split

There was a split in the puppet community and a new project saw life as a result: Chef. Chef is describing itself as :

Chef is a systems integration framework, built to bring the benefits of configuration management to your entire infrastructure. With Chef, you can:

• Manage your servers by writing code, not by running commands. (via Cookbooks)
• Integrate tightly with your applications, databases, LDAP directories, and more. (via Libraries)
• Easily configure applications that require knowledge about your entire infrastructure (“What systems are running my application?” “What is the current master database server?”)

More details about the Chef differentiators can be found here.

In a future post, I’ll explore in more details the challenges around configuration automation, and the procedural approach.

Reductive Labs received funding

Reductive Labs, the company responsible for Puppet, has received  $2 Million in funding. Puppet has been gaining traction against cfengine, but it will be interesting to see how Reductive Labs uses its funding, and how the new Chef solution is impacting this progression.

Cloud Computing brought configuration automation in the spotlight

One of the cornerstones of Cloud Computing is the automation of the infrastructure configuration. Either because you want to build a highly automated infrastructure supporting cloud users, or you are putting your application in the cloud. In both cases, infrastructure and applications configuration has to be captured, maintained and automatically provisioned. This will enable rapid scale out, fail over, or in general deployment and redeployment of the managed components.

A highly scalable datacenter automation tool merging discover, update, provisioning, monitoring, and reporting technologies from Sun Connection and N1SM into one tool.

However, by looking at the Oracle World demo, it seems that the UI is radically different from the N1 System Manager (gone the embedded CLI ?).

Also, by looking at the supported platform, it seems that Windows platform is not supported anymore :

Sun N1 System Manager :

From a centralized management console, customers can provision Solaris, Linux, and Windows with a simple drag-and-drop, and monitors the health of systems in an efficient manner.

Sun Ops Center :

The comprehensive, highly scalable Linux and Solaris life cycle management tool.

The good news is that Sun Ops Center will be delivered as open source too :

Building on Sun’s commitment to open standards and customer choice, Sun will continue to innovate the Sun xVM platform and collaborate with open source communities. The first of Sun’s contributions will be the Common Agent Container (CAC) code to the OpenxVM.org community under GPLv3. The CAC is the heart of the management infrastructure for many of Sun’s products, including the Sun xVM Ops Center. In addition, Sun plans to make the entire code base used by Sun xVM Ops Center available to the OpenxVM.org community in the first quarter of 2008.

It’s not clear however if this means the end of life for the N1 System Manager, since right now, the Ops Center does not provide a complete replacement.

Also, in this announce, Sun presents Ops Center. This is a re-branded N1 System Manager. It seems that the focus will be on the management of the hardware, up to the operating system, combined with Sun Connection.

Mark Hamilton told The Register:

Sun has also announced a new management software package, Ops Center, that will work as a command and control console for physical and virtual gear- that’s to say the hypervisor and Solaris Containers. Sun said the software also includes discovery and inventory, application provisioning, software lifecycles automation, hardware and software monitoring and compliance reporting. Sun brazenly says “it does everything except unpack boxes and rack and cable systems.”

Ops Center is going to be released in December. It will be interesting to see how this can be leveraged by Configuration Automation tools like what BMC announced recently.

• Discovery and mapping of the complete IT environment, including its relationships to business processes and transactions
• Full-scale Configuration Automation capability across network devices, physical and virtual servers, and client systems
• Change Management to support configuration policy adherence and process automation
• Orchestration and automation of activities supporting IT processes, reducing errors and expenses in task execution across operational processes and toolsets
• Automation of Continuous Compliance, including audit and enforcement of standard configurations across heterogeneous IT environments

I’ve been working on the second item in the list for the past year, and now that more of this is announced, I’ll provide more information about what this does, and how runbook automation, coupled with configuration automation can solve many of the challenges that IT operators face today.

Emprisa Networks, Inc., the leading provider of smart network configuration, compliance and change management solutions, offers solutions for extending IT service management initiatives to the network infrastructure. Emprisa’s E-NetAware solution provides immediate return on investment in several key areas – improves service availability with less change errors; automates configuration management for improved operational efficiencies, enhances network security through policy-based configuration management and greatly improves service responsiveness. E-NetAware provides support for routers, switches, firewalls, load balancers, wireless access points and other network devices from over 25 vendors.

I’ll provide more details on this technology in a future blog.

The authors are proposing an interesting taxonomy to evaluate the tools:

1. Abstraction Level: The language used by a configuration management solution can be classified at different levels of abstraction, ranging from high-level end-to-end requirements, to low-level bit-configurations.

2. Specification Language: In this section, we discuss four specification language criteria dealing with usability, domain coverage of a configuration management solution, the grouping mechanism and specifications at multiple abstraction levels.

3. Consistency: In this section, we discuss three criteria that ensure consistency in a computer infrastructure: modeling dependencies, conflict management and workflow management.

4. Distributed Management: Distributed management deals with federated management and distributes translation of configuration specifications.

The conclusion summarize the capabilities of the tools in a nice table.

RealOps AutoPilot products automate the business of operations allowing IT organizations to deliver higher levels of service at lower costs to business users. Built on the industry leading RealOps AMP (Automation Management Platform), AutoPilot solutions for Problem, Change and Service Management automate the repetitive, manual tasks IT staffs perform every day.

You will get more details on that blog soon.