How to use Public Projects to Build Products

Lanai Rock with glass

Project to Product

I want to focus on one specific aspect of project to product development from the Open Source First article. Coordinating upstream (publicly licensed) and downstream (private non-licensed) development work. In any organization that is developing and supporting software, there are some number of engineers that work on public projects. It is a common problem that the Development Managers and the Scrum Masters do a poor job of tracking that public work. Let’s assume that the Development Manager is aware and supports the public development work.

Support the Developer

What typically happens is that upper management is not aware of what the upstream work purpose is and its importance to supporting the overall development strategy. What likely started as support or even direction from the Development Manager to work on the public project evolves into a fog of light understanding.

Many months later, when performance reviews and head count updates come around, typically the upstream developer gets lets left out, as the public work is not part of what upper management understands. This leads to either dropping the upstream work, alienating one of your likely best developers, and/or the upstream work continues, but with zero Development Manager insight. In some cases all three things happen, in which case, you now have a increasingly separated, forked development effort, with a increasing annoyed developer, that is going to be putting in more and more time into the public projects. Not a good workplace situation for the developer or the manager.

Treat Public Projects Like an Company

It takes more work, but there is a proven solution. Treat the public projects like another engineering organization. If you are going to work with another engineering team, you expect to have a clear understanding of responsibilities and timelines. You need to have the exact same expections when working with a public project.

Right at the beginning, when the developer has plans to work upstream, set out responsibilities and timelines. This is just as much to protect the developer as is it is the company. Define what the public work deliverable will be and when it will be delivered. Make sure your Product Managers understand how the public work contributes to delivering the product. Make sure the Scrum Master works with the upstream developer just like the rest of the downstream developers. The upstream developer is still part of the team. As the upstream work starts to take shape, get the code into a testable branch on your software pipeline. Make sure you have quality unit and build tests to verify the upstream work, so it can be more easily merged into your code base trunk when the time comes.

Track all your development work and report on progress. Highlight the upstream and downstream work. Update leadership on the upstream progress. Keep up to date with the public projects release schedule and strategy. Publish both your private release schedule along side the public projects release schedule.

Make It Happen

This is just the broad strokes. You will need to take this and put in specifics for your implementation. Organization structure and tools vary greatly, but hold to the basic tenants outlined above.

Transforming with an Open Garden

Chateau du Versailles, walk towards the Summer Palace

I was invited to speak at one of the TODO meetings, 17 Jan 2017, at the Yahoo Sunnyvale offices. My talk was centered around Open Source First as a strategy using the Open Garden Development and Project to Product concepts.

What is the TODO group

The primary purpose of the TODO group is to bring together companies who run open source programs. Open source leaders, as a community, do not talk amongst ourselves very often. We deal with lawyers, security, and technology products, so being shy comes as no surprise. Henceforth to combat the lack of communication, the TODO group was formed to help improve the shared knowledge. Generously, the Linux Foundation supports the TODO group as one of the Linux Foundation Collaborative Projects.

Open Garden as part of Open Source First

My talk highlighted the accomplishments of Walmart as a company and OpenStack as an open source software development community. I want the success of OpenStack as software development community to work at Walmart. Therefore, I have created an strategy for success called Open Source First to work on making this happen. To begin with, we have started with the Open Garden Development and Project to Product parts of the plan. Put simply, we are improving collaboration through better communication and creating a better workflow from feature development to product implementation. For your review, find the talk slides are below for more details. For more information on the Linux TODO group visit http://todogroup.org/.

#UPDATE 08 Feb 2017: I rewrote the first paragraph with better verbiage and added a second paragraph with details on the content of the talk

Public Product Management Transforms Private at Walmart

For those of you that know me,

know that I am more a of engineer than a product guy. Even so, I was hired on to transform the Walmart Platform Products using my engineering and open source experience as the Walmart Platform Product Director. It’s been very interesting so far. Not without some bumps and bruises along the way. I am happy to say that we are in our 5th month running an internal Platform Product Guild, largely based on the success we the OpenStack community have had with the OpenStack Product working group. We have development teams publishing Product Roadmaps that we regularly review across product teams and with customers. We are directing customer and engineering feature requests into our Product Roadmaps and product development cycles. We still a lot of quality to improve on, but we have made great progress so far. I am looking forward into 2017 to include more open source behaviors in the product development processes at Walmart. Stay tuned for more details as I can share them.

Open Source First

This is a manifesto that any private organization can use to frame their collaboration transformation. Take a read. Let me know what you think.

I presented a talk at the Linux TODO group ( TODO Open Source Presentation 17 January 2017) using this post as my material. For those of you that are not familiar with the TODO group, they support open source leadership at commercial companies. It is important to lean on each other as legal, security, and other shared knowledge is so important for the open source community to move forward. This is especially true, as we need to represent both the commercial and public community best interests.

Open source first means that we look to open source before we consider vendor based products to meet our needs. To use open source technology correctly, you need to do more than just consume, you need to participate in order that the open source technology survives long term. To participate in open source requires your engineer’s time be split between working for your company and the open source project. We expect to bring the open source contribution intent and collaboration internal to our private company. We need to define, build, and maintain a culture of contribution, collaboration, and merit based work.

Open Garden Development

Our private company strives will be a leader in technology through its contributions to the technology community. This will require more than just the use of open source code. To be a leader requires participation. To be a leader, it will require various types of participation with groups (communities) outside of the company. Each of the communities will be organized around a specific Research and Development (R&D) project. Participation in each of these communities is much like working for a company. Substantial results require substantial participation.

Code More, Live Better

We must be generous with computing resources, stingy with space, and encourage the messy, creative stew that results from this. Allowing people access to the tools of their business will transform them. We must have spontaneous interactions. We must build the online and physical spaces that encourage creativity through collaboration. Collaboration doesn’t happen without access to each other in real time.

Innovation through Meritocracy

We must create a meritocracy. The quality of ideas have to overcome the group structure and tenure of those in it. Promotion by merit encourages everyone to better people and employees. While we are being the best badass we can be, hardy debates between passionate people will happen. Our culture should encourage the obligation to dissent. Strong opinions and ideas lead to a passionate work ethic. The ideas and opinions can and should come from all. It shouldn’t make difference who you are, rather what you do. As meritocracy takes hold, we need to invest in teams that are going to do the right thing without permission.

Project to Product

As our private company embraces open source contribution, we must also create clearer separation between working upstream on an R&D project and implementing the resulting product in production. A project is R&D where failing fast and developing features is the status quo. A product is what you put into production, has SLAs, and is using the results of the R&D project. The separation requires at least separate repositories for projects and products. Normal separation is different communities working on the projects and products. Each of the communities require substantial contribution and participation. In order to keep these activities separate, there needs to be a workflow of customer feature and bug fix requests from project to product. Below, we highlight the major steps in creating, supporting, and expanding open source at our private company.

School for the Technically Gifted

The seniors must mentor the inexperienced. As new skills are learned, you pass it on the next person. As you train the next person, you move on to new challenges. Never expect to stay in one position for very long. Get skills, become awesome, pass it, move on.

Find the best people for your family

We love our work. Love it so much, we want to work with our friends. We are part of a community that is larger than our company. Recruitment of the best people to work with us, should always be on our mind. We find awesome jobs for the people around us. Even if that isn’t with the company we are at. Thinking this way makes hiring great people a way of life. As hiring becomes common, then reviewing and helping new hires becomes easy.

#UPDATE: 06 Feb 2017, I added to the opening paragraph, the URL for the TODO meeting where I referenced this post content.

#UPDATE: 08 Feb 2017, I added a some details of what the TODO group is about to the second paragraph.

What does it take to run a business cloud?

There are consistent truths that cannot be ignored. Speed of light is one. There are others below.

  1. Cloud means distributed systems connected by networks. A vast majority of cloud implementations are really rebranded platform services with their infrastructure in a fluffy icon. So I’m going to replace cloud with platform for the rest of this post.
  2. Platform services exist to support your front end. Period. End of story. That means platform in most cases, is the L in your P&L ( Profit and Loss.) The L must be smaller than the P. I’ll write a follow up post on economics for those that disagree with me.
  3. What make money leads the direction of the services required. So a solid, well documented, consistent feedback loop with the platform customers is a sign of a healthy organization. There are some cases of planning on what your front end services need before they ask, like CI/CD or adding new features to existing products, but a majority of what platform does is respond the front end services.
  4. All your services need to be deployable by code rather than by hand. There are many options to accomplish this with no silver bullets yet. Your organization will have its own flavor. Those skills and knowledge must be automated for maximum rate of success. Even those few organizations that rely solely on public infrastructure still have many processes around managing all your platform services.
  5. Your engineering culture must serve your DevOps needs. Meaning do everything within reason to keep your engineering staff happy, healthy, and productive. Listen to their needs. Snacks, activities, and nice chairs are a start. Root access to their laptops, IT kiosks staffed with friendly, available staff, and a culture of mentoring new skills is much, much better.
  6. Your compute physical infrastructure must be network wise close to your data. Latency and throughput can be mitigated, but not ignored.
  7. Public infrastructure does not solve all platform problems. It only makes someone else responsible for them. Do you trust another company exclusively with your future? I didn’t think so. So even if you truly believe public infrastructure all the way, you still need an alternative option. The complexity just gets moved around.
  8. So for those of us that still need physical infrastructure, it is always going to be your largest and longest term cost. Retain the best and the brightest people for where, when, and how to build your physical infrastructure. For example, what are the tax implications of building in Singapore versus Hong Kong, data privacy laws in Switzerland, or what’s the electricity availability In San Francisco? The right people can help you to avoid making long term, expensive mistakes.
  9. The time and people required for acquiring, installing, configuring, and testing hardware and software is a constant. Public infrastructure or containers doesn’t make these things disappear, it just moves the problem to another system. Constants must be dealt with a robust capacity planning team. These are the people that are in the middle of your customer feedback loop and know your operations and physical infrastructure developers very well.
  10. Operations that doesn’t thoroughly understand their workloads will suffer major outages. It’s not if, but when. The best way to understand your workloads is by exhaustive research and development. Everything breaks. Figure out how your hardware and software breaks so you can mitigate the pending failures. Software pipelines, Performance Engineering, Quality Assurance, and a culture of R&D as a practice for all of your engineers will get you most of the way there.