Introduction

I’m researching passing verified identity of an AWS user or role for a service and came across an approach that solves it using AWS STS get-caller-identity paired with presigned urls. I found this article about the technique by Bobby Donchev in AWS Lambda invoker identification

During this research, I discovered that Hashicorp Vault and AWS IAM Authenticator experienced a security vulnerability due to this pattern. In this post I summarize the underlying approach and the mitigations that Google’s Project Zero describe.

Use Case

Allow an AWS Lambda to verify the role of the invoker in order to execute commands that depend on knowing the role.

The technique is to presign an STS get-caller-identity API call on the clientside and send that presigned link to the Lambda. The lambda executes the presigned link via an HTTP GET request and validates the output which feeds into additional internal logic.

This technique is used in:

1. Hashicorp’s Vault
2. AWS Lambda Invoker Identification by Donchev
3. AWS IAM Authenticator in Kubernetes

Security Issues

I found documented and addressed security issues in the Github Tracker for AWS IAM Authenticator and the Google Project Zero post describing vulnerabilities in Hashicorp’s Vault product.

Hashicorp Vault’s Issues

The security problems described are:

• Golang’s surprising xml decoding behavior
  • Mitigation: require application/json
• Attacker supplied STS domain component of URL can be spoofed
  • Mitigation: use known good STS endpoints concatenated with with presigned payload
• Attacker can spoof Host header
  • Mitigation: allow-list certain headers and maintain control of what headers are used for the upstream GET
• Caller can presign various STS actions
  • Mitigation: validate that action is GetCallerIdentity

The fixes for Vault were allowlist of HTTP headers, restricting requests to the GetCallerIdentity action and stronger validation of the STS response ref

AWS IAM Authenticator Issues

For aws-iam-authenticator the issues discovered were:

• Regex for host is too lax
  • Mitigation: strict set math of known endpoints in regions
• HTTP Client allows redirects
  • Mitigation: Disallow redirects in client
• URL.Query() vs ParseQuery: silent drop of invalid params rather than erroring
  • Mitigation: use ParseQuery
• Request smuggling in Golang < 1.12
  • Mitigation: Build with Golang >= 1.12

Conclusion

When I prototype a service for using STS get-identity-caller via pre-signed links, I’ll keep in mind these security concerns which boil down to following security principles:

1. Distrust user content
2. Perform strict validations
3. Understand and limit behavior of libraries that could expose a wider surface area of attack

With knowing about the existing security constraints, both in specific and in principles involved, I’m confident about the ability to build a system that uses STS presigned get-identity-caller requests safely and pair it with an AWS lambda which has a second layer of IAM defenses for allow-listing a subset of ARN based invokers.

I read The Calculus of Service Availability: You’re only as available as the sum of your dependencies today and it summarizes some of the most salient wisdom in designing for reliability targets.

The takeaways are:

1. Humans have enough imperfections in their nearby systems that 4 or 5 9s of reliability is the maximum value worth targeting
2. Problems come from the the service itself or its critical dependencies.
3. Availability = MTTF/(MTTF+MTTR)
4. Rule of an extra 9 - Any service should rely on critical services that exceed their own SLA by 1x 9 of reliability, ie a 3x 9s service should only depend on 4x 9s services in its critical path. 5. When depending on services not meeting that threshold, it must be accounted for via resilient design 6. The math 7. Assuming a service has error budget of 0.01%. 8. They choose 0.005% error budget for service and the 0.005% for critical dependencies, then their 5 dependencies each get 1/5 of 0.005% or 0.001%, ie must be 99.999% available.
5. Frequency * Detection * Recovery constitute the impact of outages and the feasibility of a given SLA. 7. Thus the levers for improvement are: reduce frequency of outage, blast radius (sharding, cellular architectures, or customer isolation), and MTTR.

I’ve been evolving our Online Database Platform at work and the themes of “rule of an extra 9” and how to move quickly as well as safely with limited blast radius are top of mind.

We’ve made some major changes (cluster topology, upgrades in nosql and sql, automation tooling) that are moving our stack to a point where I’m proud of the accomplishments.

Hundreds of TB of online data and hundreds of clusters managed by a team that I can count on one hand :).

Deno, a modern JS platform, worked well tonight as a scripting language and ecosystem for building a tiny cli interface for paging in teammates.

I will expand my use of it and replace usage of zx + js with vl + ts + Deno.

I prototyped the cli with a focus on reducing human cognitive overhead during stressful operations to provide sound defaults and reduced choice when choosing how many teammates to bring in, what to use as incident message, and details.

A few things made this easy:

1. Confidence in deploying this as a standalone tool thanks to Deno without dependency management issues
2. npx allowing for lazy loading of the pagerduty-cli tool to shell out to
3. vl as a deno library that emulates behavior from Google’s SRE tooling of zx. These are javascript libraries that make for convenient bash-like scripting in a js/ts environment.

Prototype script below:

git smart-pull is a great tool for avoiding the messiness of git rebases when there’s changed content.

I long ago inlined the full ruby gem into a single executable file to avoid the hassle of installing it in various ruby environments. It’s worked well!

Ruby 3.2.0 broke my script in a tiny way and broke the underlying gem. The patch has sat unmerged for months and now with bundler/inline I have a better solution than keeping a spare inlined script… forking the project and pointing a wrapper script with bundler/inline at my own repo.

I’m applying patches from PRs in upstream (eg hub merge https://github.com/geelen/git-smart/pull/25).

It’s an elegant solution that I’ll re-use for other ruby scripts in my development environment.

We’re automating more of our cluster operations at work and here’s the procedure for warming an EBS volume created from a snapshot to avoid their first-read/write performance issues.

How it works

Follow instructions in gist for installing in root’s crontab as a @reboot run instruction. It uses an exclusive flock and a completion file to ensure idempotency.

I recently modified our failover protocol at work for MongoDB in a way that reduces the interruption from 14 seconds down to 3.5 seconds by altering election configurations ahead of controlled failovers. This was tested on a 3.4 cluster but should hold true up until modern versions. Starting in 4.0.2 it’s less valuable for controlled failovers but still useful as a tuning setting for uncontrolled failovers.

How it works

The premise is to make the shard call a new election as fast as possible by reducing electionTimeoutMillis and heartbeatIntervalMillis.

Procedure:

// on the primary
cfg = rs.conf()
cfg.settings["electionTimeoutMillis"] = 1000
cfg.settings["heartbeatIntervalMillis"] = 100
rs.reconfig(cfg)

// wait 60 seconds for propagation
rs.stepDown()

// wait for 60 seconds for election to settle
// connect to primary

cfg = rs.conf()
cfg.settings["electionTimeoutMillis"] = 10000
cfg.settings["heartbeatIntervalMillis"] = 1000
rs.reconfig(cfg)

This is valuable to tune also if you’re on high quality, low latency networks. You’re missing faster failovers in non-controlled circumstances every time mongo insists on waiting 10 seconds before allowing an election, even when receiving failing heartbeats.

PS - While browsing the docs I found this ^_^ which is non-intuitive since I would expect no writes to one shard but no impact to other shards. Presumably it’s a typo and cluster means replicaset.

During the election process, the cluster cannot accept write operations until it elects the new primary.

Ruby’s bundler inline installs to --system destination and does not respect BUNDLE_PATH (as I would expect).

Errors will be about permission errors and server requesting root access for the bundler install.

Digging around in github issues, this is desired behavior: https://github.com/rubygems/bundler/pull/7154

Solution:

export GEM_HOME=vendor/bundle

It’s 2022 and the macroeconomic environment is severely correcting from the easy days of cheap capital.

In this environment it’s wise on a personal and business level to consider expenditures and ensure they’re providing value. I’ve been through this with personal subscriptions and with family financial planning.

Today I spent a day off work helping a friend and colleague from a former startup in optimizing his company’s tech infrastructure spend. His business is a web application with a valuable service but low traffic levels ( < 10 RPS ) and running on Heroku.

Within the first 30 minutes of him screensharing and describing the business behavior of the app, I was able to recommend $200/mo (25%) savings on his plan.

With another 2 hours, I had a recommendation for how to save another $450, which is a great savings for an indie lifestyle business. That will save him $7,800 per year and reduce his Heroku bill from ~$800 down to $150 per month.

In debugging that, I discovered how he can also trim 20 seconds (ie 75%) off of a critical feature which will translate to increased conversion.

It was a fun problem to solve and fits well into my interest in either increasing business through technology or increasing efficiency and profit through technology. I’ve done similar projects on database and server spend which yields amazing results at scale (7 figures per year), on backend and frontend performance in small and medium size companies, and take pleasure in using my expertise at the intersection of technology and business.

I’ll test the waters to see if there’s demand for consulting to help companies outsource these efforts and reduce their AWS / Heroku / NewRelic / Datadog / etc bills.

The timing is right for this adventure.

I’ve upgraded Hugo, ditched webpack for esbuild and removed most javascript from blog 🍋. The best part is that it now does syntax highlighting on the server side and builds in < 500ms 🐎.

While reading docs, I learned that Hugo allows for custom type blocks such as the mermaid diagramming in a code block below that’s rendered into an image. Unsure if this will have practical use in blog, but it’s a problem in search for an answer ;).

Make your OSX setup reproducible by exporting settings using defaults cli tool:

$ defaults find com.apple.driver.AppleBluetoothMultitouch.trackpad

Found 24 keys in domain 'com.apple.driver.AppleBluetoothMultitouch.trackpad': {
    Clicking = 1;
    DragLock = 0;
    Dragging = 0;
    TrackpadCornerSecondaryClick = 2;
    TrackpadFiveFingerPinchGesture = 2;
    TrackpadFourFingerHorizSwipeGesture = 2;
    TrackpadFourFingerPinchGesture = 2;
    TrackpadFourFingerVertSwipeGesture = 2;
    TrackpadHandResting = 1;
    TrackpadHorizScroll = 1;
    TrackpadMomentumScroll = 1;
    TrackpadPinch = 1;
    TrackpadRightClick = 1;
    TrackpadRotate = 1;
    TrackpadScroll = 1;
    TrackpadThreeFingerDrag = 1;
    TrackpadThreeFingerHorizSwipeGesture = 1;
    TrackpadThreeFingerTapGesture = 0;
    TrackpadThreeFingerVertSwipeGesture = 0;
    TrackpadTwoFingerDoubleTapGesture = 1;
    TrackpadTwoFingerFromRightEdgeSwipeGesture = 3;
    USBMouseStopsTrackpad = 0;
    UserPreferences = 1;
    version = 5;
}

Then convert them into items in the format seen here: https://github.com/zph/zph/blob/master/home/.osx to create your own custom configuration to apply during next system setup.

Use defaults domains to find all the key domains.