I’ve been thinking about scaling in my current role as the Manager of Managers for our Platform Group.

No, this isn’t about a ridiculously impressively large MongoDB cluster, or scaling MongoDB config servers with insufficient network bandwidth at scale, about finding the right abstractions for a tangled web of an area, or using wiredtiger’s zlib to reduce storage needs and no it isn’t about platform services with high QPS.

I’ve been thinking about scaling people and their units of operation.

We’ve been growing ~30-50% per year in my part of the Engineering organization and I see the inherent challenges of staying far enough ahead of that growth curve so that it’s a sustainable climb.

At this rate of growth:

1. What you start planning now needs to be the solution that will serve from now+6 months to now+18 months. Anticipate the execution lag. Watch the more distant horizon.
2. Hire leaders before you’ve filled your IC ranks, otherwise you’ll be ~6 mo or more behind and straining existing leaders.
3. Treat org design as you would distributed systems, anticipate the following challenges and be ready with redundancy and fallback positions:

• Attrition
• Promotion
• Re-organization
• Parental leave
• Bus factor
• Burnout

4. Believe in specialization of labor and use that as a way to horizontally scale leadership and to help IC engineers focus on their area of expertise. Hire PMs and EAs to scale out EMs, their bosses and the ICs.
5. Invest in recruiting org to enable hyper growth hiring. They need to be hired in advance of the true necessity or results will be 6 months delayed.
6. Hire additional EMs when your teams exceed 5 engineers so that the group can avoid scaling bottlenecks. By the time a new EM starts, you’ll have 8 engineers in that group and be blocked from scaling further without starting to stretch leadership thin.
7. Minimize as much friction in hiring as possible and get better at managing performance.

Introduction

2021, the second year of pandemic.

I joined a Fintech company at the recommendation of a friend who works there and during the interview process learned that I was VERY excited about leading a team dealing with their scaling problems. In hindsight, I should remember that all large and challenging tasks come with commensurate amount of stress :P.

I started with them in the spring, leading a team in their Platform division and ended the year by taking over leadership of the Platform division. I’ll spend 2022 guiding the managers in our division on projects that are the backbone of most other development in our engineering organization. By end of year, our division will be ~40 engineers. That’ll keep me busy with hiring, mentoring managers, and guiding engineers on key projects.

2021 was a good year personally in many ways but a hard one with isolation. I’ve been working remotely for a decade, in a combination of fully remote companies and hybrid companies. I’ve loved my remote work experience, except once the pandemic hit and cut my social life off outside work. This is the first role I’ve taken that’s leading a colocated group. Honestly, it was one of the few hesitations I had when taking this role. But because my group was colocated and happily so, I started leaving my house 3x/wk to commute to our office. It’s been wonderful! I’m happy to be out of my house, making social connections, and having new stimuli.

The commute itself is long and much earlier than I would normally think a good idea. But I’ve gotten accustomed to an earlier bedtime an 5:45am alarm clock… and having an hour or more in office before the day starts for my individual responsibilities adds to my happiness, accomplishments and satisfaction.

All in all, the year closed out well, went through rocky spots and I’m grateful that it was gentle to myself, friends and loved ones. I even learned that someone I had mentored not only was able to transition into working in tech from non-traditional background, but they recently started a role as a Lead Software Engineer. We met because they interviewed at my company in 2017 or 2018 for a junior role. They weren’t yet far enough along to hire, but I was impressed at their story and grit. So I turned them down for role and offered to mentor them as they pursued leading software development. I’m proud of them and their accomplishments and it was a wonderful surprise to hear about their next role during the Christmas holiday.

Accomplishments

• Wrote 2 plugins for Joplin Note system (auto tagger and auto alarm see link)
• Prototyped many MongoDB related systems/configurations/approaches to solve foundational issues
  • Terraform for Mongo configurations (unpublished)
  • Mongo cluster automation via terraform, auto-scale groups, consul link
  • A chunk splitter for MongoDB cluster on older version that lacks correct chunk size behavior when using many mongos (unpublished but open source potential)
  • MongoDB and I are currently on the “I know how to search source code for our version of it rapidly and have bookmarks for it in sourcegraph”.
• Solved challenging longstanding large db cluster scaling issues and improved cluster resilience and team operational knowledge. Proud of my team and of myself here, it’s a worthy and difficult feat.
• Joined our hiring committee at work and automated part of our process with a python script + google sheets + google docs as templates.
• Dabbled in learning/writing Rust link
• Started using earthly to unify my CI and local experience: eg. It’s the best docker experience I’ve had. Debugging and building up a commandlist is wonderful.
• Automated deployments of my blog using Github Actions + Earthly.
• Automated release builds of my Joplin plugins
• Setup shortcuts for blogging to reduce friction. Credit to @brandur for inspiration
• Started using “iA Writer” app on Mac for drafting blog posts in distraction free markdown. I use a ruby script to prefill the frontmatter, add file to git and then open file for writing in “iA Writer”.
• I tried three flavors of note taking systems at home and work, starting with neuron, followed by a vscode plugin driven neuron like experience with my own custom hashtagging, and finishing the year with Joplin.
  • Joplin strikes a nice balance of extensible and batteries included. As noted earlier, I’ve created two plugins of my own to add small quality of life improvements.
  • I appreciate Joplin’s proper due date tracking and that I can use plugins or my own code to determine best method for displaying those.
  • I appreciate that I can pop open my own editor, if I want more in-depth note taking. I have this set to vimr as my gui option.
  • I appreciate the export functionality to dump my notebook into markdown files with front-matter.
  • Joplin’s a simple version of what I want out of Evernote but with open source extensibility and a solid plugin architecture.
• Learned to make a solid milk foam for lattes and cappuccinos on our work espresso machine, while using the manual setting for the steam wand. I’ve been hawking my wares at work and rustling up takers of my latte efforts so I can more quickly grind barista xp.
• Lost ~8% body weight through portion control and eating nutrient rich but not calorie rich foods. It’s stayed steady at that level with negligible amounts of exercise and I plan to push it further in 2022.
• I became a more skilled motorcyclist and mechanic.

Posts

I had an inconsistent year of blogging in 2021 that included 12 posts.

My posts were focused around work topics and unsurprisingly I posted more when I had less demanding responsibilities (March while on holiday before starting new role, Nov around holidays and December around holidays).

I was able to post more easily this year because of automating my post-drafting mechanism and my deployment of the site. Those little reductions in friction were worthwhile optimizations and I posted more than I otherwise would have, especially my shorter micro posts.

Plans for 2022

• Do everything I can to delegate and empower others at work
• Utilize better specialization of labor to foster improved morale, velocity and satisfaction in work.
• Keep time in my work and personal schedule for heads down focused efforts
• Publish at least one substantial open source project, either at work or personally. Top of mind options:
  • the mongo chunk splitter
  • mongo deployment orchestration
  • mongo configuration via terraform style declarations
  • something entirely non-mongo, like a mongo wire-protocol compatible distributed database architecture built on sqlite
  • something genuinely unrelated to mongo and also not related to automating google docs as templates.
• Blog more times than there are months in the year
• Learn to ride a dirtbike
• Find ways to integrate incidental exercise into my life. Walking 1-1s at work are a good example of this but I want more vigorous pursuits that I enjoy outdoors.
• Be proactive in taking vacation
• Don’t let perfection be the enemy of done

I wrote two Joplin plugins today.

1. Auto create tags based on any content in title or body with a hashtag repo
2. Auto alarm creation based on natural language in title repo. With this I can type notes like, “Write blog post due at 4pm tomorrow” and have the alarm auto-set :).

The Joplin plugin interface was clean and simple once I browsed through the docs and a few example plugins.

PS - Thanks to @forcewake on Github for the plugin that accounts for 50% of the code in my auto-tag plugin repo

Introduction

I’m exploring Terraform to deploy MongoDB sharded clusters. Aka, binge coding Terraform, Packer and Consul was a delightfully obsessive way to spend the recent holidays. I’m learning a lot of Terraform, Consul and Packer along the way :). Terraform’s impressive and I’m enjoying architecting a robust system from AWS’s building blocks.

Design

The principle this system follows is self-healing and immutability.

Components:

1. Packer - builds server images of the core functionality (mongod shardsvr, mongod configsvr, mongos) on top of a base image of Ubuntu LTS with consul agents pre-configured.

• Terraform - deploys, updates and deletes AWS infrastructure:
  • SSH keys
  • Security groups
  • VPCs and Subnets
  • Auto-scale groups + Launch configurations
  • EBS Volumes (boot disks and db data storage)
• Consul Servers - these are the 3-5 servers which form the stable ring of consul server elements.
• Consul - each mongo server has consul setup with auto-join functionality (aka retry_join: [provider=aws...") based on aws tagging.
  • Used for dynamic DNS discovery in conjunction with systemd-resolved as a DNS proxy.
  • Will use consul-template to update config files on servers post launch. (I’m hoping this is an elegant solution for ASG booted instances that need final configuration after launch and a way to avoid having to roll the cluster for new configurations.)
• Auto-scale groups (ASG)
  • Each mongo instance is an auto-scale group of 1.
  • ASG monitors and replaces instances that become unhealthy
• Auto re-attach of EBS data volume
  • In the event that a mongo instance becomes unhealthy, ASG replaces the node but it will initially lack the db data bearing EBS volume.
  • That EBS volume is prohibitive to recreate on large volumes when considering the restoration time + needing to dd the full drive to achieve normal performance.
  • Instead of a new volume, a cronjob runs on each data configsvr and shardsvr that each minute tries to re-attach the EBS db data volume paired to this instance using metadata from the EC2 instance tags and the EBS volume tags.
  • The cronjob looks up the required metadata and executes aws-volume-attach.
  • If the volume is currently attached, aws-volume-attach is a no-op.
• EBS Volumes
  • DB Data volumes are separately deployed and persist after separation from their instance.
  • These will be in the terrabyte size range.
  • To replace a drive (corruption/performance issues)
    • Provision an additional drive from snapshot using terraform
    • Update the metadata of that shard’s replicaset member to point to the new drive’s name
    • terraform apply

My next steps are to automate the replicaset bonding and then shard joining. The open source tooling for this portion isn’t what I want, with the closest being mongo ansible. It’s an established tool but I want something more declarative and a simpler model of what it will do when executed. As a result, the answer might be a custom terraform provider to manage the internal configuration state of MongoDB. Philosophically the CRUD resource management and plan/deploy phase of Terraform matches what will give me confidence using this on production clusters.

I’ll open source the work if it gets to a mature spot. Right now the terraforming successfully spins up all the mongod nodes, networking, VPCs, security groups, ec2 instances, ebs volumes and they auto-join their consul cluster.

Credit for the concept of this approach belongs to multiple different blog posts, but the original idea of ASG + EBS re-attaching came from reading about how Expedia operates their sharded clusters. Thanks!

VSCode+Neovim wasn’t working for me today and I took the time to debug and find the patched issue.

When starting VSCode with Neovim plugin, my VSCode displayed an error code:

line   34:
E1208: -complete used without -nargs
line   10:
E1208: -complete used without -nargs

Which led me to first finding an bug report in vim-ripgrep and I assumed the problem was in vim-ack due to similarity of purpose.

I removed that plugin from my plugged directory and restarted VSCode, but not luck.

Next I discovered a vscode-neovim bug and manually patched the two files per diff:

# Note prefix directory will depend on your choice of nvim
# package manager. I'm using plug.
~/.vim/plugged/vim/vscode-file-commands.vim:34
~/.vim/plugged/vim/vscode-tab-commands.vim:10

VSCode worked on the next start and once a new release of vscode-neovim is released, it will auto-update 😊.

Ruby’s Bundler introduced a feature that I’m loving for prototypes and quick scripting

Inline Gemfiles

Introduced in v1.10, these avoid needing a separate declaration of a Gemfile and you write it directly into the script you’re creating.

It fixes the following ergonomic issues:

• Writing a separate Gemfile when your program is a single Ruby script is overkill
• Packaging your Gemfile to accompany aforementioned script is a hassle
• For single file scripts you can now rely on Gems!

How To Use It

require 'bundler/inline'

# Declare Gemfile in ruby script
gemfile do
  source 'https://rubygems.org'
  gem 'mongo'
  gem 'pry'
end

client = Mongo::Client.new(ENV.fetch('MONGO_URL'))

PS

I’m not writing much Ruby these days because I work leading engineers, but when I hack on things in my spare time I’m prototyping and want to iterate on an idea before converting it to Golang. I did this recently when iterating on how to speed up controlled MongoDB failovers and when spiking out a terraform-like system for MongoDB configuration management.

Bundler-inline is great for these situations because they’re a single Ruby script but require installing Gems :).

As an engineering leader, saying no is a superpower.

Counter: Saying yes and imagining what’s possible is a superpower.

Honorable mention: Avoiding becoming a gatekeeper or single point of failure is a superpower

TLDR: Build distributed and fault tolerant human systems. Reject what needs to be rejected and say yes to ambitious things you can’t yet conceive.

A good summary and collection of Rails security/anonymizing practices:

https://ankane.org/sensitive-data-rails

TLDR

When I build my own company, I’ll think back to working at a bootstrapped startup and the lessons I learned from our founder.

My business will embody these principles:

• Believe (obsessively and doggedly believe)
• Keep It Super Simple ^{1 2} (radical simplicity)
• Ship It (ship the bare minimum and get feedback, repeat)

Honorable mention:

• do things that don’t scale ³ (By the time you have scaling limits, you can pay to figure out scale.)

When I’m building features, finding customers, or planning technical architecture, I’ll think of her and say: WWMD.

My Startup Story

Small startups at a cesspool of ideas, experiences, stress, and joy and I learned a ton through my phases in smaller startups. Today, I want to highlight lessons learned from working at a small bootstrapped-ish startup. I’ve also been reading Ray Dalio’s book Principles ⁴.

The startup was ~50 people when I left and I was there for 3 years. I joined two years into their journey (~25 ppl total with 6 in engineer). Money was always tight and yet we were moving two digits of millions in revenue when I left.

Beyond learning from my time there, I learned from the 2 years before me… from the example set by out solo founder.

She had a year or two of professional experience in web development focused on design and frontend and had no business experience…. until she made her own experience! She became obsessed with a business problem encountered casually and from then on, lived, breathed, and fought for her business.

She built a truly minimum viable product… and was a shining example of what you can do with radical simplicity. Her busines started with a Google Form. She leveraged amazing productivity and value from off-the-shelf free tools. Turns out free tiers can be pushed REALLY FAR with the right urgency and necessity! If Airtable existed at the time, I think that would have been our database.

She learned what she needed to know about databases, servers, analytics but it was a means to an end of making the business work. She didn’t get lost in la-la land of falling in love with the tech. They were a tool she became proficient with but they didn’t rule her. Technology was a means to an end of building the business. My time with her honed my business approach to delivering the most value we could with available time and energy. We radically optimized for making the biggest bang for the buck, because there weren’t many bucks to go around.

What drew me to the company was her obsession, passion and success. Her fervor and obsession were infectious and I saw my own star rising with hers. I learned to see the world a more like she did, with less rules and boundaries. I bring that with me to my own engineering leadership and personal values in software engineering.

Recommended reading Blue Tape List