Monday, February 3, 2014

Remote Dependencies, Convenience, Risk and Other Considerations for Operating Distributed Systems

One deeply held principle by experienced distributed system operators that I have worked with is that you should have no external dependencies to your software other than the ties to minimum requirements of the OS such as common system libraries, utilities, and the kernel of the base OS. This approach should enable recreating a distributed system deployment without any dependencies on the outside world. When something goes wrong, you should have control over your own destiny. Reliance on any external dependency that is managed or hosted by someone else introduces risk that something outside your system can affect your ability to restore and recreate the system any time you need to.
To use a simple metaphor, imagine your system is represented by Jenga blocks and it falls over as Jenga towers inevitably do. However, instead of being able to rebuild your tower you find out that a mandatory required component at the base of your tower is missing or unavailable now no matter what you cannot rebuild the tower exactly how it was before. Your new tower is going to behave differently in unexpected ways and you might topple over because you do not understand all the behaviors when using different building blocks combined in a different way.
Some of the original designers of the software deployment project for Cloud Foundry named BOSH (Mark LucovskyVadim SpiwakDerek Collison) embraced this principle and tried to create a prescriptive framework that encouraged this approach. They had experience managing large scale distributed systems at Google (the web services APIs). Kent Skaar also did similar for SaaS provider Zendesk. Given a software release that references specific versions of multiple software packages (known as a BOSH release), an instantiation of that release (a BOSH deployment) can be reconstructed at any time with the deployment configuration (a BOSH deployment manifest), the base OS images (the BOSH stemcells) and the software release (the BOSH packages and job templates for applying configuration). at any point in time, properly implemented BOSH releases of large scale distributed systems can be recreated without external dependencies. That means this holds true even when the internet is unavailable.
BOSH does give you the framework hooks to break out of this prescriptive principle and use external dependencies or at least external dependency formats if you choose to for convenience or other reasons. Dr Nic Williams recently implemented tooling to use apt packages instead of compiling from source. another example: some of the Pivotal big data software intentionally targets CentOS/RHEL only and therefore only ships rpm packages rather than compiling Hadoop. A guiding principle is that you should be mindful of the tradeoffs you are making of convenience vs risk and tying your release to only one OS distributor.
Examples of the tradeoffs:
  • relying on an externally hosted package manager like apt-get could affect the availability or correctness of that dependency when you need it most
  • relying on debian packages could prevent someone from using your release unmodified with a CentOS image
A recent real-world example demonstrated the risk of an external dependency changing unexpectedly. The coreos/etcdproject that Cloud Foundry is using for storing stateful configuration data for the new Cloud Foundry Health Manager codebase had one of the dependencies (goraft/raft) force push to master of their git repository that overwrote some git history required by git to work properly. This situation has limited the flexibility of some users to make code modifications on several previous releases of Cloud Foundry without some tedious intervention.
A common reaction when learning about Cloud Foundry BOSH is to question the prescriptive guidance to compile from source when commonly used distributed package management systems exist in the Linux distributions. My recommendation is to understand the tradeoffs involved and make the best choice for your situation. You should explicitly call out external dependencies if you have them in your system. When you tower inevitably falls over, know how to rebuild it.


Esteem Host said...

Web hosting India offers you a domain name and more space. The virtual market is the best way to directly interact with the target audience. Because of the large number of businesses provides web hosting services in India and abroad.

Steve Hawks said...

The attitude towards cloud computing is influenced by multiple factors acting as drivers and barriers for its adoption. This research paper aims at the reconceptualization and operationalization of widely accepted influencing factors on technology adoption in relation to cloud computing.

cloud computing training chennai

Arthur Mac said...

Understanding the latest concepts is possible only through contents like this. Thanks for sharing this page in here. It will be useful for my future projects as well. Keep blogging articles like this.

Hadoop Training Chennai

Mathew Stephen said...

The content published here was worth able to read and share. The aspect in which you have written the content is amazing. I have bookmarked this page for future use. Thanks for sharing this in here. Keep blogging content like this.

Software testing training in chennai | Software testing training | Automation testing courses in chennai

Jhon Abraham said...

To keep ourselves up to date with the current trend is not an easy task in IT. But we can, through quality and worth able content like this. Thanks for sharing this web page. Please write more articles like this in future.

Hadoop Course in Chennai
| Hadoop training institutes in chennai | Manual testing training in Chennai

akhil sharma said...

very good post ....i enjoyed it while reading
best valentine day poem
valentine day whatsapp status
valentine day message for facebook

Rahul Singh said...

happy valentine day
valentine day wishes for boyfriend
valentine day poems for girlfriend
valentines day message for wife
valentine day message for girlfriend

Vikyjohn said...

Thanks for the post.Hadoop is an open-source software for storing data and running applications on a hardware. It provides storage for any kind of data, enormous processing power and the ability to handle tasks. Hadoop changes the enterprise store, process, and analyze data.

Hadoop training in Chennai

Chandrika Madapudi said...

Excellent post!!! In this competitive market, customer relationship management plays a significant role in determining a business success. That too, cloud based CRM product offer more flexibility to business owners to main strong relationship with the consumers. Cloud Computing Training in Chennai

Mansoor said...

Really informative blog , also visit us for : nebosh courses in chennai | nebosh in chennai | nebosh course in chennai

saki said...

thanks for sharing your technical knowledge of article for user purpose. its more informative keep sharing.
Informatica Training in Chennai

Thanu Abi said...

This post is amazing.It is very useful for me.Important topic to explained very nice.thanks a lot for this information.

cloud computing training in chennai

Akshay Saini said...

Server Wala Data Centers is a Dedicated Server provider located in Jaipur(Rajasthan). We provide the same excelling Bare Metal Dedicated Server, Managed Dedicated Server. We offer real live 24/7/365 support via phones, email, and chat and we are constantly striving to be better. We offer tons of extra features that other hosts do not provide, plus the ability to host unlimited domains on most of our packages! We offer a 30 day money back guarantee on most plans, so try us and if you’re not satisfied you can cancel for a full refund.

Keerthi said...

Now it is known to me that articles is nothing but inspiring is everything to do something great. This is a great article for the people who want to come in freelancing.

Informatica Training in Chennai

John Snow said...

Can anyone recommend me a dedicated server hosting company, with cheap dedicated server hosting ?

Luzy Anvi said...

i enjoyed it while reading safety courses in chennai|Nebosh courses in chennai|IOSH courses in chennai|safety courses in chennai|Diploma in safety courses training institute in chennai

Fazill .A said...

Nice One :)

fire and safety courses in chennai

Priya said...

The cloud computing offers many benefits for your business that can reduce the cost of managing and maintaining the IT systems.
cloud computing training in chennai | cloud computing courses in chennai

Nasreen Basu said...

interesting piece of information, I had come to know about your web-page from my friend pramod, jaipur,i have read atleast eight posts of yours by now, and let me tell you, your blog gives the best and the most interesting information. This is just the kind of information that i had been looking for, i'm already your rss reader now and i would regularly watch out for the new posts, once again hats off to you! Thanks a million once again, Regards Guidewire training in hyderabad

Durga Spplimited said...

safety course in chennai
safety courses in chennai
construction safety courses in chennai
Industrial safety courses in chennai
offshore safety courses in chennai
fire and safety courses in chennai
fire and safety course in chennai
nebosh courses in chennai
safety engineering courses in chennai
Industrial safety engineering courses in chennai
fire and safety engineering courses in chennai
Iosh course in chennai

Paul Miller said...

Informatica is a (ETL) tool Extraction, Transformation and Loading. It is used for building enterprises with data warehouses.Learn Informatica course in Chennai and it has a massive demand in the market.To know more visit our website.
Informatica institutes in Chennai | Best Informatica Training Center in Chennai

Andrew Son said...

wow!!This is very interesting and i have all your blogs.Keep on blogging about cloud.
cloud training in Chennai | cloud computing training in chennai

Arthur Mac said...

Thank you for taking time to provide us some of the useful and exclusive information with us.
Informatica Training in Chennai | Informatica course in Chennai

Rajapriya R said...

nice and really helpful article to everyone... thanks for sharing

selenium training in chennai | selenium training institute in chennai | Android training in chennai | android training institute in chennai

Saradha Devi said...

great one

Best Selenium Training in Chennai | Android Training in Chennai | Java Training in chennai | Webdesigning Training in Chennai

Geetha Devi said...

Thank you for sharing
Best Selenium Training in Chennai |Selenium Training in Chennai |
Android Training in ChennaiBest Manual Testing Training in chennai

Rajapriya R said...

informative blog

selenium training in chennai | selenium training institute in chennai | Android training in chennai | android training institute in chennai

Geetha Devi said...

Nice post Best Selenium Training in Chennai |Selenium Training in Chennai |
Android Training in ChennaiBest Manual Testing Training in chennai

Rajapriya R said...

informative blog

java training in chennai | java training institute in chennai | java j2ee training in chennai | java j2ee training institute in chennai

Priya R said...

Excellent post!!!. The strategy you have posted on this technology helped me to get into the next level and had lot of information in it.
salesforce training in chennai | salesforce training institute in chennai

Haxie 100 said...

Happy New Year 2017 Wishes
Happy New Year 2017 Informations
Happy New Year 2017 Quotes Check These Outt

Ranjit Ram said...

Want to know about Veterans Day,which is a day to celebrate the great officers of the US army,just checkout my blog Happy Veterans Day and stay updated.

Amirtha rao said...

Nowadays, most of the businesses rely on cloud based CRM tool to power their business process. They want to access the business from anywhere and anytime. In such scenarios, salesforce CRM will ensure massive advantage to the business owners.Cloud Computing Training in Chennai