Open Core Summit Event Report

Today I participated in the inaugural Open Core Summit.

When I first heard about this event from the organizers, what I heard was that they were trying to put together a small boutique event. Clearly, they had struck a gold mine of untapped demand, because next thing I’ve heard is that the event is two days and the attendees are approaching 1000. Hats off to them.

Because of the family emergency of a sort, I was only there for several hours and therefore I missed most of the talks. Of the ones that I saw, the quality was all over the map — some were really just thinly vailed company pitches, and some “fire side chat” sessions were lame. Acoustic of the room was not great, and noise from the hallway track was overwhelming.

But despite all those noticeable shortcomings, I really liked the overall event. Perhaps because this was an inaugural event, speakers and audiences were quite diverse. It was as if nobody knew who else will be there. I’ve seen VCs, CEOs of open-source companies, well-known engineers in the open-source communities, entrepreneurs, vendors, open-source foundation people, and more. As a result, from speakers I’ve heard a lot of perspectives that are new to me, even if I might disagree with them. In contrast, more established conferences tend to self select people, which results in an echo chamber. Once again, diversity wins! It also helped that every talk was 20 mins, as you get to hear even more perspectives.

I later learned that this event apparently created a little Twitter storm. Clearly some “open source” people have developed aversion and distastes for what they thought this event is supposed to represent. I could be missing some important nuances, but I found this somewhat bizarre. As far as I’m concerned, a part of what made “open source” so much more impactful than “free software” is that it created a bigger umbrella that brought more people underneath. But here, the nay sayers seem to be making the same mistakes that fanatic free software advocates have made. I fundamentally believe that open source is a better development model, so more people who can invest in it the better, and clearly commercial open source companies have invested a lot.

I just wish the organizers had a guts to invite those people to the event and let them share their perspectives. I would have loved to hear them. I think that would have added even more diversity to an already diverse event, and in doing so it would have really highlighted what makes open source shine.

More pictures from the event are here.

Trail of Coincidence

A guest was coming to our office over lunch, so food was ordered for everyone. Except it turned out that due to a miscommunication, mine wasn’t there. No problem, I said. Happy accident. I’ll go out for lunch later. Maybe somebody up there is trying to tell me that I should eat out today.

I was just going to a nearby restaurant, but a random lunch conversation led to somebody mentioning Tofu House, my favorite Korean restaurant. Great, I thought. Somebody up there must be telling me that that’s where I should go.

On my way back from Tofu House, there was a road work going on at an intersection where I was normally going to make a right turn. I decided that that’s a sign that I should keep going straight. I can just get back to the office in a different way.

It was a hot day, and this road was going to go right in front of my favorite smoothie shop. Surely that must also be a sign? So I made a stop and got myself a smoothie.

At this point, I was starting to get excited. It started to feel like a series of serendipitous events was happening to me. Is this going somewhere? Wherever that is, I’m feeling lucky today.

I resumed my drive back to the office, and because I was driving a road I don’t normally take, in this one complicated intersection I ended up in a wrong lane that forced me to a wrong road. A series of no U turn and no left turn prevented me from taking an obvious way back. The detour is getting big.

See, something IS happening! This cannot be a coincidence! Maybe I’ll encounter a future soul mate? Love of my life (OK, maybe not, I’m already married.) Save somebody’s life? Find a million dollar?

I looked around. Nothing. I kept driving and looking around. Still nothing. I eventually came back to the office parking lot. The trail of serendipity is gone, abruptly.

I still can’t shake that off from my head as I write this. I was convinced that this was going to take me somewhere. What clue did I miss?

Suddenly, I’m not feeling lucky anymore.

Camping in Livermore

In the Saturday Japanese school my daughter goes to, there are 4-5 families that mine has been particularly close to. With some families relocating and what not, faces have slowly changed over time, but some of us go all the way back to when my daughter was a baby.

Over the memorial day weekend, the group went to an overnight camping together. It’s one of those camps where we all show up with loads of gears, food, and drinks in our cars. Adults would indulge in non-stop eating and conversation around a camp fire, while kids disappear into the nature and enjoy themselves.

The next morning I woke up and my daughter wants me to see a project she and her younger friends built together in a creek nearby. Cool, I guess that’s what they were working on when I didn’t see them. I’ve followed them upstream to a make-shift dam made by rocks. It’s only partially completed — perhaps a few meters from a bank. Satisfied with my wow-ing and ohh-ing, they resumed their project, picking stones and dropping them into the creek, taking turns, self-organized. I aimed my camera in the hope of taking some pictures of their joyful expressions.

That’s when I understood. It just came to me like that.

See, my daughter is right in the middle of teenage years, where they struggle and worry to figure out what’s their passion. My frame of reference, namely how I grew up, is completely useless here because I just run into computers in the 7th grade, I got hooked, and the rest is history. Like countless other parents, naturally this has been at the top of our mind; Exposing her to various extra curricular activities, from astronomy to performing arts; Cutting back to make time & space that she can use to explore and find her passion freely. Somewhere in the back of my mind I knew I was worried too much, but how can you not, when the stake is so high? I didn’t want her to just assemble a LEGO set I bought her. I wanted her to assemble something on her own out of the LEGO bin. That’s so important to me that’s the meaning of her name I gave her.

Well, guess what, while I was having fun with beer in my hand, fiddling with a camp fire with my friends, she explored the creek a lot further out than I thought, took care of younger kids, somehow latched on to the idea of building a dam, and put that into action. I didn’t suggest any of that to her. She did it all on her own.

Seeing her creating something passionately, not just consuming what’s given to her, was such a tremendous relief. I mean, look at her happy face and you will see. She is perfectly capable of finding something she wants to do. I don’t know what that’s going to be in the real life, but I just felt so confident that she will be fine. As parents, I and my wife were watering and fertilizing this little plant every day hoping that some day it will grow into a beautiful flower, whatever color that might be, always worrying that it has too few leaves, or too many leaves. But maybe parenting is more like where we think this plant will bloom into a flower, then discover that the soil turns into a clay and the clay turns into a beautiful flower vase. One giant sequence of beautiful unintended consequences.

When I look at it like that, my heart is filled with deep, deep appreciations to so many.

First and the foremost, my wife. Throughout this post, I said “we” like I played the equal part, but let’s be honest, I was mostly doing just talking and thinking. It is my wife who has been doing almost all the doing, while I was flying around the world pursuing my own passion. Unintended consequence doesn’t mean you can skip parenting. It is her 15 years of constant watering and feeding that created enough mass that allows this little plant to turn into something. And only I get to watch and appreciate that never-ending devotion. I just hope a similar lightning strikes her and relieves her from the fear and the pressure.

Then there are people of the group. They got me hooked into camping in the first place, which led to this moment, and they keep on giving. Activities to my daughter that her parents wouldn’t have thought of themselves is one but that’s not the only thing. They provide her compliments, push her further, teach her fishing, and otherwise interact with her in their own unique ways. I don’t have any real extended family, certainly not on this side of the Pacific ocean, but I just realized these people are practically my big extended family. And they don’t know what it means to me. What a wonderful bunch of folks they are.

I’m surrounded by amazing people. And it’s wonderful to be aware of that. This is a day that I’m going to remember.

Art of asking questions

A part of my work is a journalistic work. I go out, I grok how & what software people are building, and that knowledge informs what we do next. This is primarily done by asking questions and listening to the answers. So I think a lot about how to ask questions.

One of the first things I noticed is that if you are trying to find out something, it’s never a good idea to ask a direct question, such as, “does X matter to you?” Because for almost any meaningful X, say cost, culture, people, process, … the answer will be “yes,” but that’s useless knowledge. What you really want to ask is how people trade off two good things X and Y. The world is full of good things we don’t get around doing it.

Another problem with these piecemeal speed dating questions is that every time you come up with a new question, you have to go back to them and ask. My knowing what color you like and what food you like doesn’t really help me infer what car you want to drive.

Instead, I strive to understand the picture they are seeing. What do they perceive as the reality around them? What are they trying to change in there and what do they see as constraints? Why? Tell them somebody else’s story in a similar situation and ask them how those are different? Ask them to draw diagrams, or draw a diagram yourself and ask them to correct your misunderstanding.

When I’m successful in doing that, I can put myself in their shoes. I can think like them, and I can infer what they do for why. I can retrace their decisions. I pitch some of those back to them, and see if I got that right. If there are discrepancies, that’s a sign that I grossed something over. Once I build such a mental model, it’s very powerful. I can answer the speed dating questions myself based on it.

This takes time. I usually ask for 2 hours to conduct these interviews. I then take several more hours to process the raw notes into a readable material. It also only makes sense in a small circle setting, ideally 1:1. IMHO a group discussion is useless if you are trying to really understand anything. Some people dominates the air time and others disengage. People use the same word to mean different things. They latch on to details that are irrelevant. I’m sure you know what I mean.

Yet I see a lot of people asking speed dating questions, in a group setting. And that’s been puzzling to me. When you put 10 people in a room and ask questions like “is testing important?” what kind of knowledge can you possibly gain out of it? What am I missing?

KubeCon event report (2 of 2)

(This is the continuation from the previous post.)

New Era of Open Source

In the 2010s, open-source was driven mainly by startups as a strategy to build a viable business, which led to companies like Red Hat, CloudBees, Elastic, & Mongo.

It was also painfully clear in KubeCon that the forces that shaped the landscape of open-source has changed completely. Today, open-source is primarily driven by unicorn end user companies who needed to solve problems that vendors didn’t even knew existed. That’s where innovations are. And they don’t need to make any money from those projects unlike startups. Substantive projects highlighted in KubeCon were from companies like Uber, Lyft, and Spotify. Sessions by engineers from those companies were packed. Everyone wants to be like them, no wonder their projects are popular.

The other kind of open-source projects are from vendors. They are either Hail Mary gamble moves out of desperation (e.g., IBM’s razee), or torpedos designed to drag down the market leaders through indirection or commoditization.

I still saw plenty of startups who are trying to repeat the single vendor open-source model of the 2000s, but the going will be really tough for them. First, they have to find problems that aren’t already solved by unicorn end users. Then they have to cross the sea where 3 cloud titans are shooting torpedos to each other and random Hail Mary are shot into from the sidelines. Really the only sensible move is SaaS.KubeCon escalator

3 Cloud Titans

Speaking of 3 cloud titans, Microsoft won my respect once again with their SMI announcement. Along with KEDA, together they clearly show the Microsoft OSS strategy of making sure innovations in the Kubernetes ecosystem get contained behind the common portable interface/spec. Then Microsoft can “embrace and extend” or build their proprietary implementation nicely integrated with the rest of Azure. This is the exact same play IBM, BEA, Oracle, etc. have done with JavaEE. Use JCP to create portable APIs, let Sun implement the free tier, and focus on building the for-pay value-add tier. The move makes sense, and it works because end users share the same interest.

I didn’t see Amazon, but I assume they are making sure that EKS is good and otherwise doing the bare minimum in the Kubernetes ecosystem. That way, they can focus on building up their own services, like Microsoft.

What made me wondering is Google’s plan. Maybe it’s just that they didn’t get the air time they deserve in KubeCon, but it felt like they are going down the route Sun did. They seem to be busy building awesome commons that benefits everyone, which wins tremendous developer love, but no $$$. I’m sure GKE is the best managed Kubernetes service right now, but it is a thin value layer that requires constant upkeep.

It’s a tragedy that it takes a truly selfless act of engineering brilliance to create a great developer platform, like Sun and Google, yet the inventing company itself never financially gains from the innovation. But then maybe that’s why they gain so much respect and fame. I truly love those companies.KubeCon mosaic

End Users Need Industry Standard

It was also very clear that app developers are overwhelmed with choices and chaos of the ecosystem. It was proclaimed, as if that’s what people want, that “Kubernetes is a platform to create platforms.” Well, most people there are writing apps, not creating a platform. The last thing they want is to pick & choose their own cloud native platform out of 100+ CNCF ecosystem software just so that they can write apps. I wonder who solves this problem for them. That gotta be the biggest opportunity.

I noticed that quite a few consultancy shops were there posing as Kubernetes experts, claiming to solve that problem on behalf of the customers, but I question how much they know themselves.

Jenkins X is clearly in the right spot, with this regard. Its code and governance are growing at a breakneck speed thanks to the Jenkins X community and the CDF, but the market is growing faster.

My money is on Microsoft, Red Hat, and end user companies to come up with JavaEE/WS-I/WHATWG-esque “standard” body that defines a set of the common APIs, like SMI and KEDA. This will include the space Jenkins X is in, so it will have a role to play, too. That way, end users get to avoid lock-in, cloud providers get to provide implementations specific to their cloud and compete with value-add. It will be like the LAMP stack of the cloud native.Kubernetes 5th anniversary

Conclusions

KubeCon is worth a visit. History is repeating itself. Open-source is thriving, though the rules have changed.

KubeCon event report (1 of 2)

KubeCon / CloudNativeCon in Barcelona was amazing. I’ve been hearing great things from my colleagues who have been to past KubeCon, but hearing and seeing are two entirely different experiences.

Excitement

The energy was clearly in the air, reflecting the amount of interest, excitement, innovation, and investment in the cloud native space.

The number of projects that were represented there in some shape or form were staggering. Endless list of “meet the maintainer of the XYZ project,” “introduction to the XYZ project,” or “deep dive into the XYZ project.”

So many people were there, so many talks were offered in parallel. Everyone was trying to absorb what’s going on, and there is simply no way anyone can keep up. Even at lunch tables, conversations were “what interesting things did you see?”

As if that wasn’t enough, at least two significant new projects were launched and announced during keynotes. There must have been 100+ vendors in the exhibit space. Dozen+ co-located events. And numerous private small meetings.

Nobody knows exactly what’s happening, but everybody knows something significant is happening. It’s basically a giant week-long festival for geeks.

KubeCon show floor

Chaos

Just as clear as the energy was how incredibly chaotic the whole scene is. How can you expect otherwise? CNCF has no single command-n-control structure where the entire effort gets planned and coordinated. It’s a group of entities with conflicting interest, acting and reacting to each other.

This chaos was perhaps the most visible in keynotes. Take the Day 2 keynote as an example. First, the host from VMware does his opening and declares that the theme of the keynote is “Kubernetes is a platform for creating platforms.”

Then an SRE from Spotify comes on the stage and proceeds to tell a great story of how they can recover from a major Ops mistake without any visible disruption to customers, except Kubernetes didn’t play any positive role in that story, and the story delivery was so clumsy I couldn’t help but wonder how magical the experience would have been if the speaker did a bit more practice.

The next one on the stage was a VP from Oracle Cloud, and he proceeds to present a strange mix of the ecosystem praise, followed by a marketing video of an Oracle customer (CERN) praising how WebLogic is awesome on Kubernetes. Wait, CERN guys had their own keynote yesterday and there was no mention of WebLogic. And WebLogic on Kubernetes, well, let’s just say that story felt weak. This short Oracle keynote was followed by another end user company, who explained their quite ordinary run-of-the-mill cloud transformation journey but in a beautiful presentation by an eloquent speech. Basically, the complete opposite of the Spotify keynote.

KubeCon keynote

There’s no story arc that connects those 3 segments. None of them have anything to do with the supposed theme of the keynote “Kubernetes is a platform for creating platforms.” There was probably no coordination and not even a concerted effort to push presenters to practice. Just about the only upside is that so many competing vendors working together nicely enough sends a very powerful message.

Compare that with, say, Microsoft Build keynote, and the contrast cannot be starker. That also had quite a few segments run by other people, but they were designed to tell a consistent story to drive a point. Or compare that with Dockercon, where even ordinary session speakers are asked to do dry-run in front of the engineers of the event team and to iterate based on the feedback.

What Creates Ecosystem?

But as I was walking out of the keynote somewhat frustrated and trying to make sense of what I just saw, it dawned on me that I was looking at it completely wrongly.

Sure, Microsoft Build has shown off a well produced keynote and the breadth and depth of what Microsoft does. But that’s not where the future is getting created; It’s created in this CNCF ecosystem. In fact, Mighty Microsoft and even Amazon are forced to play along with this. As RedMonk says, developers are the kingmakers.

And if I look at it as a developer, it’s obvious why KubeCon is the carnival and why Microsoft Build isn’t. KubeCon sells the aspirational lifestyle of devs and ops. We all want to work in a blameless continuous learning culture, and to survive a data center loss without any customer impact like Netflix or Spotify. And those customer stories tell you such lifestyles are not your imagination. You could be them! Except you can’t really. It’s like Nike and “just do it.”

Then KubeCon shows you the vast boom town under construction. The city hall at the center is beautiful and rock solid, but the church over there is only half built and not roofed yet. Where the market is supposed to go up is currently just a sprawl of tents. Three police stations are being built by 4 groups, but nobody is building a fire station. The Kube city has no mayor and every volunteer group is run totally differently. If you are a developer, this is legitimately amazing. There’s so many ways to participate in this grand construction project. Everyone is screaming for more hands. Anyone can instantly spot 10 things that need to be worked on. The whole scene is screaming “opportunities.”

OTOH, the Microsoft city is run by the party, like China. Smart people are in charge, and they have a great plan, but there’s no election. They have a beautiful story of how the Microsoft city lets its residents focus on building your apps that will make a real world impact. The city is still a long way from what the scale model makes you believe, but it’s already quite livable and convenient. They even let you participate in some construction activities here and there. But for a developer, this place feels suffocating. You are a consumer in this city.

KubeCon Party

I think there’s something important to learn here for product managers and marketers who are trying to attract developers. These people tend to value a beautiful vision, and coherent messages, and incorrectly believe that developers would enjoy them too. Yet I think what attracts developers is the opposite. Every end user companies have a certain number of engineers who feel a lot more passionate about how they build, as opposed to what they build. Those people come to the Kube city.

As Peter Drucker said, “culture eats strategy for breakfast.” The Kube City eats the Microsoft city for breakfast.

(Continues to the next post)

Microsoft Build event report

This is my event report from Microsoft Build last week. More pictures of the event is here. The last time I’ve been to a flagship Microsoft event is more than 10 years ago, where we showed Sun hardware that was certified for Windows Server and I was handing out Sun T-shirts. Before I joined Sun in 2001, I was a developer fully immersed in the Microsoft ecosystem, so I always had a soft spot in my heart for Microsoft.

Obviously a lot has changed, but there are many moments that reminded me that what made Microsoft great back then are still alive and kicking.

As you can see, I loved the show, and I came back fully impressed with what Microsoft is up to.

SNY02879

Depth & Breadth

What struck me on the first day is the depth and the breadth of technology that Microsoft has in its arsenal. This is in your face in so many ways, from the size of the exhibit space to the list of sessions so long it takes forever to scroll.

Microsoft covers everything from …:

  • Established “boring” products everyone need & use today. Compute & storage on cloud, developer tools, Windows, Office, Xbox, …
  • Nextgen developer infrastructure that look solidly production ready, like managed Kubernetes service, serverless, Cosmos DB, monitoring & alerting, plethora of data services covering ingestion to storage to analysis, business intelligence, machine learning, …
  • More research-y stuff like vision/language AI, holo-lens, industrial automation AI, …

And then on some of the key parts of that stack, they have quite a “breadth,” too! Doing one IDE is hard enough, now they have three. Linux & Windows. C#, Python, JavaScript, and more. Intel & ARM. That they have enough engineers to cover them all and still turn profit is almost unbelievable.

Fully-integrated

This is one of those “Microsoft DNA,” I think. So many things they do leverage each other, in such a way that once you get hooked into their ecosystem, it starts to make more and more sense to use the rest of the Microsoft stuff. Whether you see that a good thing or a bad thing, I’m not sure.

As a case in point, it felt like every developer platform from robotics to gaming, from serverless to ML, have IDE and Azure DevOps integration. “Run this command and we’ll generate your CI/CD pipeline” was a common theme. Another example is their ML development platform leveraging their BI platform for visualization.

Or when you use their PaaS , monitoring is integrated out of the box. It’s a far cry from deploying Kubernetes, install Datadog agents, and wire up Datadog with PagerDuty.

Microsoft has been great to Azure ISVs like CloudBees in the past few years, but seeing this “everything is fully integrated” DNA, I understood why — it’s the “land & expand” play. They are so confident with their ability to “expand” to gradually increases their share of the IT spend of a customer, they don’t mind to “land” with ISVs to meet customers where they are today.

Hybrid Cloud

One key misunderstanding I fixed in this trip is what Microsoft means with “private cloud” or “hybrid cloud.” I used to think this term was coined by vendors like VMware/Oracle around tech like OpenStack who wanted their customers to believe you can have something comparable to public cloud in your own data center. That is a complete hallucination, so I dismissed the idea of hybrid cloud, too.

I must have missed the memo, but the industry (certainly Microsoft) doesn’t use the term that way anymore. It’s more analogous to Mac and iPhone working well together, or Windows desktop and Windows servers. Different things that together covers different parts of a spectrum, with consistency where possible. And Microsoft is pushing this lineup even further down to the smaller scale by adding micro-controller to Azure brand. That makes sense.

I bet, however, that enough customers are still living in the hallucination, and Microsoft seems to be benefiting from this. Those customers just want a better data center control plane, and they assume that the productivity gain from that is sufficient. They are wrong on that last point, but by then Microsoft has “landed.”

Combined with Microsoft’s enterprise salesforce and customer relationship, I think this is a convincing story for Microsoft to win. I’m curious how well AWS is meeting this challenge. From where I see, it looks to be a no match.

SNY02988

Developer Eyeballs

Another Microsoft DNA that is alive and kicking is their obsession to developer eyeballs. They simply don’t let other companies set up a toll booth between themselves and developers. Compare that with companies like Google, Amazon, and Oracle.

They have 2 IDEs (legacy Visual Studio and Visual Studio code) that are AFAIK of completely different codebase. They are both very popular and still adding tons of features. Then no top of that, in this event, they announced a web IDE.

On top of all that, they have Azure DevOps and GitHub to complete the picture. That’s how obsessed they are. No wonder Microsoft paid top price for GitHub.

I already talked about how so many of what they do leverage these IDEs and developer services. AWS should be really nervous about the fact that people are developing with VSCode to deploy to AWS.

Conclusion

There’s so much more to write about, but this post is already getting too long.

Being an old-timer and ex-Sun, I can’t help but feel a deja-vu of the Microsoft vs Java war to some extent. Microsoft is still leading with the value of more integrated products that work better together (used to be Windows, Exchange, & SharePoint, now Azure), backed by enterprise salesforce and relationship building & nurturing. A tried & tested winning formula for decision makers.

Meanwhile, developers and disrupters are building loosely connected OSS projects on top of a platform they control (used to be Linux, now Kubernetes) and they are happy to wire those on their own. Presumably, the difference in how better the technology is compared to the fully integrated solution make up for the cost of wiring them on your own.

But there are difference in this round, too. Now those two worlds aren’t mutually exclusive. The latter world can happily run on Azure.

As so many people have already written, I think Microsoft is being pretty successful at the moment. But to me, the question is why Microsoft is not even more successful.

SNY02886

Dockercon 2019

I was at the Dockercon last week. TL;DR is that +1 for the event team, their OSS strategy, and ARM, but -1 for the CTO’s stage performance, and their product value.

SNY02807

Docker is clearly focused on developer experience (DX) that hides rapidly evolving “implementation details” (think Kubernetes, Helm, etc.) As the OSS strategy, that made sense because Docker is already in every developer’s laptop. The problem is that they overly focus on what devs do from their laptops, ignoring (or unaware of?) the reality that much of the software development is around making those things happen automatically at the right time. For example, one of the demos was developers making a change on their laptop and pushing that to production, but in the real world, nobody deploys to production from their laptops. Instead, you design a flow around environments and branches. Something Jenkins X gets right.

On the exhibit floor, it was clear the money is in operations (security, monitoring, & policy.) Curiously, Docker products don’t seem to do any of those. In fact, I came out of two keynotes feeling entirely unclear what their product values are. It was almost like they forgot to talk about products completely. They clearly sell support, they seem to sell “simplicity” (but isn’t that the top-of-the-funnel OSS thing? since when enterprise is simple?), and there was one mention of “SAML authentication,” but what else? IMHO, that was a fail on their part.

SNY02805

The most brilliant strategic move during the whole show was by ARM. Through Docker, they made it drastically easy for developers to build equivalent images simultaneously for amd64 and arm64. They worked with Amazon that arm64 is available cheaply on AWS (50% cost reduction was claimed.) The story they are working toward is “15 minutes will save you 50% on your EC2” and I think that’s very compelling. On top of that, they seem to be working toward bringing container-based app development to “edge computing,” aka their home turf. If the single development & deployment paradigm can span everything from embedded devices to servers, I think that’d be amazing. Docker on embedded devices would be a tremendous boost to the sorry state of typical embedded app development.

Two keynotes were well produced and demo were well scripted. IOW, their event team and the product team did a great job. Demo wasn’t just a proof that something works, it told a story. But two key speakers on the stage were a little disappointing. The CEO was OK but clearly reading from a teleprompter. The CTO was worse — he was lacking energy and conviction, at times apologetic, and his interaction with demo staff lacked chemistry. To me, it’s a sign that they didn’t put enough effort into practicing. I’m pretty sure the CTO is smart, thoughtful, and quiet guy; it’s just that that combo doesn’t make a great keynote speaker by default, so he has to make up for that by practicing more. That’s something I have to be careful myself.

Lastly, as a speaker, I was impressed by the touch that the Docker team provided for speakers. They reviewed outline with you, made sure you prepare slides early on, put you through dry-runs, and provided useful feedback. That must have taken tons of efforts. Hats off to the team.

SNY02803

On technology leadership

What does it mean to be a technology leader? This has been and still is a perpetual question for me. In this post, I’d like to reflect on what I learned about technology leadership in my experience as the creator of Jenkins and as the CTO of CloudBees.

Leading by Code

When a software engineer rises in prominence through the popularity of their work, the first phase of the leadership is what I call “leading by code.” People around you notice that you are doing some great work, and they start paying more attention to you. By this point, you own a lot of code, meaning you’ve probably written quite a bit of code of the said software yourself, including the heart of the design and the architecture. Owning code gives you upper hand in many ways, which you can use to influence others:

  • You probably are the most knowledgeable person for many parts of the system, so whenever somebody tries to build something anywhere, they want your input, approvals, or maybe even some matching change from you.
  • Your code, coding style, and how you produce it all become the baseline for everyone. In trying to do “monkey see monkey do,” they seek your input and approvals.

In essence, you lead by being “a better doer.” I remember that this felt great. Engineers close to me did not need a title change to know the influence I was radiating, but now my boss, my distant friends, and people who are looking at my LinkedIn resume knows that I’m a leader! I also obviously believed that my way of doing something is great, and I was helping others do it better. I wasn’t really changing anything I was doing before – I kept writing code, just more often together with other people. When I wanted something to be done in a certain way, it was usually followed. Everyone wanted my input. What’s not to like!?!?

Limit of Leading by Code

I think “leading by code” comes naturally to everyone. I’ve seen many senior engineers who perpetually stayed in this comfortable mode. But today I believe it breaks down at a certain size of a team. I now think, perhaps selfishly, trekking beyond this comfort zone is what differentiates a technology leader from a better software engineer.

Why does “leading by code” break down? Simply put, you become a bottleneck.

  • Bandwidth: if everyone needs your attention and focus to get their job done, then you run out of time. When people start doing things increasingly on their own as a result, you feel threatened, because you feel your control is slipping away and other people are developing deep expertise in various pockets.
  • Creativity: when you influence others by basically controlling what & how they do, either directly by telling so, or indirectly by controlling a tribal custom, it naturally limits other people’s creativity. This is a hard one to see when you are in the middle of it, because there’s always some room of creativity that you can point to.

That is me in 2019 speaking. I didn’t have a perspective back then to think in these terms. So I’ve probably stayed in this mode of leadership for too long, feeling increasingly unhappy and disappointed at myself. There were a few signs that this leadership style was hitting a limit.

  • At one point, I wanted to design and write an interface between two subsystems myself, because “owning code” was how I was exercising the influence. People implementing the two subsystems weren’t happy at all that I was lagging behind, producing something different from what they wanted. It didn’t help me either because I lost some credibility in their eyes, rather than gaining influence!
  • I was in a design fight with somebody under me, and I couldn’t resolve it in an amicable manner and ended up just overriding him. I remember him saying “I’ll reserve the right to tell you I-told-you-so.” I had some reasons for doing that, but really that’s beside the point. He got demotivated, and it created “my way or your way” framing. He’s also now rooting for you to be wrong, and sooner or later, you WILL be wrong!

So I started exploring other mode of technology leadership.

Launching a New Project

One of the things that worked was to focus on launching a new project/product/effort. The idea is that you basically get embedded into a team for a while, then transition out when the time is right. I’ve done this several times, and I think it suited me well for several reasons:

  • I could “lead by code” with all its benefits without much of its downsides because new things start small. I could steer a team toward a general direction without making people feel like you are telling them what they do.
  • New things benefit from an executive sponsor who communicates progress upward, keeps unwanted disruptions at the bay, builds bridges to where necessary, brings input, context, and perspectives from other parts of the org.
  • When a new team is formed to take on a new thing, which I suspect is a norm in a startup, I can help the forming and norming phases. I bring in “how things are done around here” naturally thanks to the title, and speed up initial decision making. Just don’t forget that you will be wrong on some of those decisions, and you got to let the team take over and fix some of those. That used to feel embarrassing, but now I take it as a sign of a maturity of the team.
  • I suspect one of the things that keep the CTO role influential is being on the side of change, as opposed to being the defender of the reality on the ground. Being seen as a part of new projects is helpful with this regard.
  • And new things are fun, so it kept me engaged!

Storytelling

As the org grew, the number of people I needed to influence got larger, and I found myself leaning more and more toward a meaty, structured write-up, as opposed to email dialogs, or 1:1s with key individuals. This blog post itself is a good example of such a write-up, albeit too long.

A write-up can start with a recap of context, or cross references to earlier write-ups, which help larger number of people to “get it.” Writing down helps you organize your thoughts more clearly and think more thoroughly. Your message also won’t get lost in Chinese whisper game, when people forward your message to others. It takes more time to compose one, but the wider reach makes it worthwhile.

CEOs and managers of big orgs do this all the time, but I think technology leaders should do this more, too. And you don’t need to use the “unicorns and rainbows” voice that they use, either!

I started consciously doing this more and more after getting hooked to great storytelling podcasts like This American Life and Radiolab. Those podcasts got me thinking a lot about how to tell a story better. When it works, it makes your write-up more engaging, and it frames how people think about a certain thing. When you succeed in that, there’s no need to sell people a solution. I’ve sought some help from people who studied journalism, and I’m seriously wondering if there’s more training/class available in this area.

Aside from one-off write-ups, I’ve run two series of write-ups that have certain themes, and both had great tractions. One was a series of “portraits” of people who we are building our products for, and the other is things I’m noticing about what’s going on in the industry, such as event reports.

I’ve done most of this on Google Doc, and what I loved is that people can comment on my message to add more colors, which makes it a whole lot more valuable. OTOH, Google Doc sucked at helping people find your write-ups, and that took a lot of value away from it. If Confluence was more loveable experience, I would have switched.

1:1s with CTO

At around ~100 people, and as I developed a “storytelling” broadcast channels, my 1:1 interaction with engineers started to change in nature. It used to be “outbound” where I tried to influence key senior people. It now became more “inbound” where I learn great ideas, interesting efforts, and/or their challenges from people. People were much more willing to talk about that in 1:1 and all I needed to do was to prime the pump and listen attentively.

So I started to encourage them, help them get more time working on them, and so on. Basically, find a pocket of excellence and try to lend my credibility to grow them.

It’s much easier to make something happen when there’s already somebody who’s thinking about it and passionate about it, even considering the downside that is “it’s not exactly how I wanted it.” I’d also like to think it creates a workplace that engineers want to be in.

I’m still trying to figure out how best to do this. When you skip reporting chains and go directly to ICs, it can be disruptive. You can’t create a change without creating any friction, but you want to minimize that. Also, not every ground-up effort pans out, and when they fail for one reason or another, it can be really disappointing for the person I encouraged. Expectation setting can be tricky, and so much depends on individual context and the personality. When an effort does pan out, the next stage of the project needs more serious organizational commitment, and I often struggled to bridge that smoothly.

Toward the end, I started thinking about creating a rotating position to do these “incubation” projects by pulling engineers out of their regular positions and time-boxing projects. This felt like a promising idea – it makes the whole thing a repeatable program that other people would recognize better, and hopefully that encourage more people to bring ideas to me and help with the “next stage” problem.

Architecture Forum

I’ve seen this done in many places, but I’m not sure if I cracked the nut. I’d love to hear from people who has done it successfully. The idea is that you have many teams with defined territories, and you drive the alignment/collaboration between them through meetings where people from those teams come, present, and talk to each other.

  • I thought the way to make this valuable is to ask them questions. What about observability? Have you thought about this failure mode that happens all the time? Help them tap into organizational memories. Bring concerns and realities from the field. But I felt one-hour session is too short for that, and there’s always somebody from other teams who ends up in the “let me design it for you” mode who speaks too much.
  • When I invited one person from every team to be on this forum, some people thought “hey how come Alice is invited but not me?” and that made them unhappy.
  • I suspect many engineers just don’t have the mindset of “there’s always something to be learned from new perspectives” baked into them, and those people don’t feel a real value from this setup. When enough people do not feel value, it’s too easy for this to become a bureaucratic formality.

In retrospect, I think a better way to do this is eng-org wide continuous education. I could have identified people who can speak to those concerns, perspectives, and context, and perhaps I could have a dialog with them and capture that into a presentation, a document, a podcast, or something. That way I can amplify a voice of somebody while still creating a crafted message that I want the org to hear, and I can influence everyone, not just tech leads. And so long as it sticks to somebody in a team and they speak up, the team will be doing the right thing.

Conclusions

This post is really more of a progress report of mine than a guide book for others. Your mileage will vary.

That said, if there’s any take away for others, I think it’s that the PDCA cycle works. That is, you try something, notice that it seems to be working, so you build/adjust a mental model of why that works, then that helps you plot the next move. What started as a random search and blind shots gradually become more systematic hypothesis testing over time. You never really find answers (at least I didn’t find one), but you do find questions that are worth asking.

It was uncomfortable at the beginning, but once I felt that the cycle was going, I felt happier because I was making a progress. Then the rest became easy.

I hope you’ll find some of this useful, and I’d love to hear from people what worked for them and didn’t.

My personal journey around diversity

Today, in the software development industry, the issue of gender, racial, and other diversities is widely accepted. But I also think this is still largely understood as “social value,” as opposed to “business value.” As a result, the awareness is not leading to enough actions, because it loses out in the crucial fight over priority. Like all other good things, it never get acted on.

This is a story of my personal experience that helped me develop new perspectives on this issue, to more clearly see why it’s in your business interest to tackle this.

Where I started

I’m a left-leaning man born to Japanese parents, who spent equal time between the U.S. and Japan. In other words, I’m generally on the side of increasing diversity in any given team. I’m also a father to a daughter. So I’ve always felt very sympathetic to the gender diversity issue. it was obviously a good value to uphold.

I was also very proud of the geographical and cultural diversity of the Jenkins community and CloudBees. I remember in one dinner we had, I looked around the table and realized that this group of 8 people represented 6 countries and 6 native languages, and I loved it.

I’ve been feeling like I was a supporter of the cause, a part of the solution.

What made me pause

Then I hired Tracy Miranda as a new community manager to my team at CloudBees. She was going to help us grow the Jenkins community. She was very passionate about the issue of diversity, she led that effort in the Eclipse foundation. So it was no surprise that in one of the planning meetings, she proposed that our team prioritize tackling the gender diversity problem.

A surprise was my own reaction. I found myself not really buying into it. The question wasn’t whether that is a good thing to do. It obviously is. The question was whether it’s better than all the other good things we can do.

I didn’t feel like she gave me that rationale, I couldn’t answer this myself, and I felt other people in the team had the same question.

That’s when I realized maybe I wasn’t a part of the solution, I was a part of the problem, a part of the system that created inertia.

What I run into

That realization wasn’t a pleasant one to face, but it also gave me an important question — what is the business value of the diversity? Fortunately, within a relatively short period of time, I randomly came across two great stories that helped me explore this question.

  • The story of Arlan Hamilton: This series of podcasts tells a story of a gay black woman trying to start a venture capital. She makes the observation that startup founders are not reflecting the general population balance, and therefore her core investment thesis is that there’s a untapped business opportunity in investing in minority founders. It was interesting that she doesn’t see her work as a social responsibility or charity work. It’s simply a space the competition was failing to dominate, an untapped field, and she was going to win that space.
  • Curb cuts: This 99% invisible episode talks about ‘curb cut,’ which is a slope from a pedestrian walk to a road. This was originally done to make roads more accessible to wheelchairs, but when it happened, it turned out that everybody benefited. Parents with strollers, people with suitcases, etc. What I got thinking is that when we make a community accessible to one under-represented group, the effort will have a ripple effect to other under-represented groups. This is a very common thing in software engineering. When we solve one use case, we often solve it in a bit more generalized way, and that helps other use cases.

What I learned

From these stories, I started to see the issue of diversity differently.

It’s like opening a new bar and in a new town. You build a bar that you like, you hang the “open” sign, and you wait for people to show up. Some people wonder in, the place gets busier, your regulars give you feedback, and you improve your bar, which brings more customers. Things are going great.

But at some point, this kind of growth will plateau, because you are self-selecting your customer base. At some point, to ignite the next growth, you need to do something else.

What you do is to go out on the street, see people who are not coming to your bar today, then think about what it takes to make them your customers. You might notice that there are a lot of Hispanic people walking around in your neighborhood. Maybe you’ll prepare a menu in Spanish, or put pictures in the menu. And you repeat this cycle to win more customers.

If you think about it this way, it’s obvious. Don’t sit and wait. Reach out. In marketing, for example, people do this all the time. But among software companies or open-source projects, too many people are still in the “open a bar and hope people show up” mode, and not realizing the lost business opportunities.

And that’s because this is also hard. It’s hard to think about and understand people who are different, who you don’t see.

How I connected dots

And on the point of how hard it is to think about and understand people you don’t see, I have a lot of personal experience.

I grew up in Japan, so I’m well connected in the software development scene over there. I know countless faces of great software engineers, team leaders, and open-source people over there. It’s the number 3 economy in the world, after all! But here in the west, they are just not on the radar. The open-source people I interact in the U.S. and Europe simply don’t know that these people exist, and so they don’t know what they are missing. That’s something I felt personally passionate about, and I realized the structure of this is exactly the same with the gender diversity.

I had another “aha” moment of this exact same structure from another angle, when I visited China and saw its thriving tech scene. I realized many obstacles we unknowingly set up in the Jenkins community. Our mailing lists are on Google Groups. Too bad, Chinese firewall blocks them. Our official twitter feed was similarly blocked. If we hope to reach Chinese users, we need to be on their social network that none of us knew. Something that I was completely unaware of, that became totally obvious once I met them and talked to them.

Conclusion

As the software engineering community, we need more good engineers to join our teams, our communities. And a rational way to do so is to build a new muscle to seek out untapped engineers and win them. Find, talk, and understand a group of people different from you. Accommodate and win them. As with everything else, if you’ve never done it before, doing it for the first time is hard. But you will get a hang of it pretty quickly.

I think any group of people would do. When you solve it for one group, you are already halfway there solving it for other groups.

Gender just happens to be a relatively “obvious” one to tackle first. It just acts as a concrete goal of the otherwise abstract idea of “reaching out.” One, the total addressable market (TAM) is pretty big — it’ll basically double your current TAM. There’s no firewall, no time zone difference, no language barrier. It’s also a very visible group that everyone can see, which makes reaching out much easier. On top of that, it’s a well trodden path with a lot of collective experience to tap from.

So that’s where I stand in 2019. Hopefully my journey is useful to others going through the same journey. Looking forward to hearing from people who have thought more about this.