What does it take to run a business cloud?

There are consistent truths that cannot be ignored. Speed of light is one. There are others below.

  1. Cloud means distributed systems connected by networks. A vast majority of cloud implementations are really rebranded platform services with their infrastructure in a fluffy icon. So I’m going to replace cloud with platform for the rest of this post.
  2. Platform services exist to support your front end. Period. End of story. That means platform in most cases, is the L in your P&L ( Profit and Loss.) The L must be smaller than the P. I’ll write a follow up post on economics for those that disagree with me.
  3. What make money leads the direction of the services required. So a solid, well documented, consistent feedback loop with the platform customers is a sign of a healthy organization. There are some cases of planning on what your front end services need before they ask, like CI/CD or adding new features to existing products, but a majority of what platform does is respond the front end services.
  4. All your services need to be deployable by code rather than by hand. There are many options to accomplish this with no silver bullets yet. Your organization will have its own flavor. Those skills and knowledge must be automated for maximum rate of success. Even those few organizations that rely solely on public infrastructure still have many processes around managing all your platform services.
  5. Your engineering culture must serve your DevOps needs. Meaning do everything within reason to keep your engineering staff happy, healthy, and productive. Listen to their needs. Snacks, activities, and nice chairs are a start. Root access to their laptops, IT kiosks staffed with friendly, available staff, and a culture of mentoring new skills is much, much better.
  6. Your compute physical infrastructure must be network wise close to your data. Latency and throughput can be mitigated, but not ignored.
  7. Public infrastructure does not solve all platform problems. It only makes someone else responsible for them. Do you trust another company exclusively with your future? I didn’t think so. So even if you truly believe public infrastructure all the way, you still need an alternative option. The complexity just gets moved around.
  8. So for those of us that still need physical infrastructure, it is always going to be your largest and longest term cost. Retain the best and the brightest people for where, when, and how to build your physical infrastructure. For example, what are the tax implications of building in Singapore versus Hong Kong, data privacy laws in Switzerland, or what’s the electricity availability In San Francisco? The right people can help you to avoid making long term, expensive mistakes.
  9. The time and people required for acquiring, installing, configuring, and testing hardware and software is a constant. Public infrastructure or containers doesn’t make these things disappear, it just moves the problem to another system. Constants must be dealt with a robust capacity planning team. These are the people that are in the middle of your customer feedback loop and know your operations and physical infrastructure developers very well.
  10. Operations that doesn’t thoroughly understand their workloads will suffer major outages. It’s not if, but when. The best way to understand your workloads is by exhaustive research and development. Everything breaks. Figure out how your hardware and software breaks so you can mitigate the pending failures. Software pipelines, Performance Engineering, Quality Assurance, and a culture of R&D as a practice for all of your engineers will get you most of the way there.

OpenStack Seattle 2016 Keynote

I was fortunate enough to be asked to give the opening keynote to the OpenStack Seattle Days 2016 conference. These types of events are mini-conferences put on in partnership with the local OpenStack user group. The user group gets tickets at a discount that is subsidized by the vendor sponsorships. We generally follow the user group rules of focusing on community works, not selling a product with our talks. This is my second year attending this event in Seattle put on by Sriram Subramanian. Excellent food, entertaining talks, and a great venue. I will definitely come back again. Read more at http://www.openstackseattle.com/, http://www.meetup.com/OpenStack-Seattle/events/232134301/.

OpenStack Silicon Valley 2016 keynote

I was asked to give the one of the keynote addresses to the OpenStack Silicon Valley 2016 conference put on by Mirantis. These types of events are mini-conferences put on in partnership with the local OpenStack user group. The user group gets tickets at a discount that is subsidized by the vendor sponsorships. We generally follow the user group rules of focusing on community works, not selling a product with our talks. This was my third year attending this mini-conference and the first year presenting.

See https://www.openstacksv.com/ and https://www.meetup.com/openstack/ for more details.

Take Your Open Source Project to the Foundation Level

I have put together below a high level “go to market” plan for an open source project becoming an standalone organization. It roughly covers similar ground to what OpenStack went through, so if you are familiar with OpenStack history, this plan will sound almost identical. The difference is I have mixed in some of what the Linux Foundation has been successful with over the last few years. With this plan, there is the assumption that you have decided to create a open source foundation for good reasons. See my other posts on open source projects if you want to read up on options other than creating a foundation like joining an existing foundation or not going the formal route at all.

Executive Overview

Our project is important to our company. It is so important, we have decided to create a standalone organization to support the project. We represents in this document, our company, that is currently supporting the project. Our objective with a standalone organization is to create a public open source community to support the development of the project. By placing the project code base into the public space, we expect that the project will have greater innovation and support over the long run, due to the greater participation of many engineers. Since this project is an important part of our production operations, it is important to strongly support the project, as we would expect any vendor to do. We also expect the side effects of going public with the project of getting better engineering recruits and good engineering publicity. Having a strong reputation for engineering excellence is important to the company.

Creating an standalone organization is the same process as you go through setting up a new company. The major difference is that this organization will be a non-profit. We want to focus on building the community around the project, so we want put off the distracting activities like filing taxes and creating a government recognized entity for as long as possible. We will need to handle money for salaries and accounts receivables to create the project community, so we need to offload that responsibility to our company for time being. Laser focus on building the standalone project community is critical for success.

We need a clear strategy to achieve our objective up front. Therefore we need define milestones within a larger timeline. Our objective of creating a standalone organization to support the project can be defined as hundreds of monthly contributions to the project and tens of production implementations. The strategy in more detailed implementation steps can be found below.

No plan would be complete without stating the risks of not implementing the plan:

  • If we continue to with the status quo of internal only development, we risk the project features will stagnate over the next few years. The lack of alternative ideas and single implementation will cause the project functionality to degrade.
  • If we place the project in an existing foundation, the risk is that the project founders will have less goverance control over the direction of the project. The project will need to conform to the style and tools of the organization it joins.
  • If we simply publish the project codebase to github, the risk is stagnant contributions and implementations. Most projects have a 6 – 12 month honeymoon period when potential contributions can exceed community participation without losing people’s interest. After the honeymoon is over, interest will drop quickly. For our company, it will also be difficult to justify supporting both a public code base and the private, internal code base. Once we push the project to a different repository, it is a fork and must be treated as such. Head count is always difficult to justify and will be increasing difficult to properly support two branches, when only one branch is actively being developed against. The end result will an orphaned public repository with our company’s name on it. Orphaned projects hurt a company’s reputation. It is better to not open source a project to begin with, unless it will be properly supported.

Plan

  1. Establish a small group of project leaders from within the project. Find an executive to support the project objective of creating the foundation. Empower the project leaders with the authority to make timely decisions.
  2. Establish the timeline and budget for the foundation effort. I suggest implementation will take 2 to 4  years to execute depending on budget.
    1. Summit Manager: We will hold a design summit every 3 – 12 months that gathers every contributor. Each large event needs to be organized and planned 6-24 months in advance depending on the size and complexity.
      We will not be able to organize events effectively without this head count.
    2. Community Manager: We need to pull the developer and operator community together, so people contribute and show up to the design summit. This role will manage the user groups as well. The user groups are a very important detail to get right early. It creates the swell of unplanned contributors that any development community needs.
      We will not have community growth velocity without this head count.
    3. Operations Manager: We need one person to coordinate financial and logistics day to day. This person works with the outsourced accounting. This person coordinates with the summit manager and business development roles to make sure the money coming in can cover the costs for the next 6-12 months.
      We will not be able to keep the books organized without this head count.
    4. Infrastructure Manager: We need one person to run the Continuous Integration (CI) and other collaboration tools. This person will also double as the developer advocate. This is a critical role for keeping the development community trust and moving forward.
      We will not be able to keep the community supporting tools working without this head count.
    5. Business Development Manager: We need one person that brings companies and organizations to the community. This person will bring companies to the user group meetings and design summits with their contributors. We need businesses to be involved and engaged, so that the contributors are committed as well. This person manages the sponsorship commitments, levels, and sponsor ROI.
      We will not have many corporate contributors without this head count.First we hire the core community support team over the first 3-6 months costing ~$2M per year
    6. Executive Director: Every team needs a leader. The ED leads the administration team. Expect that the community will look to the ED to keep the community moving forward. You do want the ED to defer technical leadership to the community.
    7. Marketing Director: This can be a role the Summit Manager covers in the early stages of the community. Once interest in the community picks up, this will become a full time, paid role.
  3. Hold public developer summit events at least every 6 months. These events will be similar to LinuxCon and OpenStack summits. Starting out the developer events can be under 300 people and cost $100K to $2M annually. For 300+ people, expect the annual cost to be ~$2-5M. The cost varies depending on how much food and entertainment is committed. Event costs and sponsorships requires the Summit Manager, Operations Manager, and Business Development Manager to very coordinated.
  4. Develop 3 – 6 partners to collaborate operations and a few features. Find at least one early contribution partner that is looking for the tooling for their business strategy. Start from there.
  5. Establish base operations with that first contribution partners. The next 2-5 contribution partners will find you through user group events and word of mouth.
  6. Create a collaboration tiger team that will go out and find partners. The tiger team can be made of people associated with the project, rather than directly involved.
  7. Establish a 6-8 person governing body to start. Our company gets 3 seats and the rest 3 -5 seats. Single seats per company. The people should represent a mix of governance and technical leadership. No talk of elections yet. That will come after this organization turns into an entity.
  8. Create a brand around the purpose of the project, our focus on meritocracy, transparency, collaboration, and our company’s purpose in creating the public organization for the code base. Work with our corporate marketing on this.
  9. The Project Manager (volunteer from the community) will need to create a high level roadmap with release cycle, 3-6 months. The Infrastructure Manager and Community Manager need to help with setting the public release schedule. As the public project matures between now and the first release date, the Infrastructure Manager will need to take over the public release schedule. As the community evolves over the first year or so, the Release Manager role will need to be filled by the entity. The Release Manager and the Project Manager then could be merged and become a paid position.
  10. Build public support with incremental successes.
    Organize the development community into slightly smaller focused governance and technical groups of 6-12+ people. The Community Manager will organize this with the assistance of the Infrastructure Manager.
  11. Public CI is critical very early on. Infrastructure hosted and managed by our company or an early contribution partner to start. Put jenkins as job manager and gatekeeper for github. Gerrit for collaboration. OpenStack CI infrastructure is a good model to follow since it is published and well documented.
  12. Establish committer roles early on. Allow any community member to contribute code to any project. Allow any community member to comment on a code contributions. Only allow code merge to trunk by a “core reviewer”. The core reviewer should be a project contributor that has a proven track record of consistent contributions and meeting participation.
  13. Contribution licensing is a lawyer distraction. Follow Linux and OpenStack lead here to avoid endless meetings with lawyers.
  14. Public freenode IRC and mailman mailing lists for all discussions. Slack doesn’t scale well and isn’t used broadly used in the public space.
  15. Public hosted jira instance for project release management. Open source projects can get free unlimited licenses.
  16. Public meetings always. Communities made up of frienemies will be very suspicious of each other. The sales and marketing staff can accidentally cause rumors. Have very public decision making.
  17. Have very few rules, with just enough structure to function. Focus on innovation.
  18. Once on the path to success then discuss long term options like creating an entity, but not before. Creating an entity is very distracting. Stay in this mode until 2-4 long term partners can share the management load. Communicate the going to entity strategy very early, so it can be tabled until it is time to work on it.
  19. Only very small changes once started. Big changes shock the system and lead to less participation.
  20. Simple mission statement. The roadmap should have accomplishing the mission statement within a 2-3 year period.

Do you know where your OpenStack vote is?

Every January the OpenStack Foundation community votes on Individual Board Directors. This role is critical part of keeping OpenStack, well OpenStack. Two thirds of the OpenStack Board of Directors are paid seats by companies. Individual directors are roles held by people. When I left Yahoo and VMware, the board seats I held, were retained by those companies. It’s an important detail that most people don’t know. Having this non-company specific role was an early consideration we made when the Foundation was put together. It puts the community right in the driver seat of making decisions. 

So make your vote count. Read up on the OpenStack candidates. See whose ideas for the future align with your own. It’s another way you as an OpenStack Foundation member can contribute.