Scrape full link set from site:

wget -r -l4 –spider -D blog.xargs.io http://blog.xargs.io

Analyze link set from site

tree -J -f blog.xargs.io | grep file | grep -o 'name.*' | \
  awk -F":" '{print $2}' | tr -d '",}' | sort -u

Curl to see which currently work

# Deal with multiple saved copies of same entry from wget
cat current_links.log | grep -v "\.[[:digit:]]*$" | \
sed 's/blog.xargs.io/http:\/\/blog.xargs.io/g' | \
  parallel -- \
    "curl -o /dev/null --silent --head --write-out '%{http_code} %{url_effective}\n' {}" | \
  sort -u | tail -r > current_links_master.log

Then run the results.csv through a processor to compare your staging site to your production site. Watch for those 404s and make sure your 302s look good.

cat current_links_master.log | sed 's/blog.xargs.io/localhost:5000/g' | \
  parallel -- \
    "curl -o /dev/null --silent --head --write-out '%{http_code} %{url_effective}\n' {}" | \
  sort -u | tail -r | grep -v "(200|302)"

Credit for these scripts:

• https://beeznest.wordpress.com/2012/07/01/spider-a-website-with-wget/
• http://stackoverflow.com/a/22379949

Log Architecture

I’ve been considering architectures lately that allow for realtime updates across many disparate systems. We use one of these at work and it allows for a near infinite number of subscribers to watch a Kafka stream(s) for updates. Many different systems feed into this pipeline and many systems consume the data. In case of rare/non-existent Kafka downtime all events are stored temporarily into S3.

The architecture that informed this system was documented in a LinkedIn technical article here: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying. Go spend twenty minutes reading and digesting the implications of that article.

The Problem

I was recently presented with the problem where many applications modify many records in various postgres databases that all need to be indexed by ElasticSearch. Bulk indexing is possible and actively done, but realtime updates are preferable.

The Solution

The solution to this problem rests with Posgres, NOTIFY/LISTEN, an intermediary application, a Kafka stream, and consumers who know how to update records from DB -> ElasticSearch.

Here’s how I prototyped the solution:

• Register functions that are called whenever a PG database table performs an INSERT/UPDATE/DELETE, aka a database trigger.
• Function calls NOTIFY <CHANNEL> <PAYLOAD> (or pg_notify(CHANNEL, PAYLOAD)) where channel is a string identifier of where to publish a stream of those events. Payload is a string of arbitrary data, which I set as table=<NAME>,action=<INSERT|UPDATE|DELETE>,id=<ROW_ID>. This payload configuration is similar in concept to a query string and passes just enough information that an event can be registered on Kafka stream.
• Intermediary application is registered to listen on <CHANNEL> and call a callback of its own for each message. This parses the message and encodes it in a more advanced/portable manner.
• Intermediary app posts that formatted event onto Kafka stream.
• Workers consuming stream pick up that event and fire off a re-index of the row.

For simplicity and because of current excitement about Elixir, I built the intermediary application using Elixir and Boltun. Elixir gives me a good degree of confidence in its reliability and uptime because of the built in OTP architecture with GenServer and Supervisor trees. It’s also approachable for other engineers, regardless of their current familiarity with Elixir.

# Requires setting DB connection details in config/config.exs per Boltun Readme

defmodule Listener do
  use Boltun, otp_app: :listener

  listen do
    channel "watchers", :my_callback
  end

  def my_callback(channel, payload) do
    # Send to Kafka and S3
    IO.puts channel
    IO.puts payload
  end
end

Listener.start_link

Database functions and triggers for NOTIFY/LISTEN

-- DROP TABLE example_table;

CREATE TABLE example_table (id serial primary key, name varchar);

-- create function for DELETE action
-- Uses OLD id instead of NEW because the ID after action will be null
CREATE FUNCTION delete_event() RETURNS trigger AS $$
DECLARE
BEGIN
  PERFORM pg_notify('watchers', 'table=' || TG_TABLE_NAME || ',action=' || TG_OP || ',id=' || OLD.id );
  RETURN OLD;
END;
$$ LANGUAGE plpgsql;

-- create function for INSERT/UPDATE action
CREATE FUNCTION insert_or_update_event() RETURNS trigger AS $$
DECLARE
BEGIN
  PERFORM pg_notify('watchers', 'table=' || TG_TABLE_NAME || ',action=' || TG_OP || ',id=' || NEW.id );
  RETURN new;
END;
$$ LANGUAGE plpgsql;

-- attach insert_or_update_event fn to the update_trigger
-- Which is how we specify to act on INSERT/UPDATE
CREATE TRIGGER updates_trigger BEFORE insert or update ON example_table
FOR EACH ROW EXECUTE PROCEDURE insert_or_update_event();

-- Attach delete_event to deletion_trigger for DELETE
CREATE TRIGGER deletion_trigger BEFORE delete ON example_table
FOR EACH ROW EXECUTE PROCEDURE delete_event();

Now start the Elixir application for watching the NOTIFY stream. Execute insert/updates/deletes.

At this point, notifications will go out via PG’s NOTIFY as:

NOTIFY 'watchers', 'table=example_table,action=INSERT,id=2'

You’ll see Elixir logging those events in realtime via NOTIFY. Which is the equivalent of the following function call.

Listener.callback('watchers', 'table=example_table,action=INSERT,id=2')

-- Sample code for running the above SQL and doing row modifications.
-- PSQL <DBNAME>
-- $ psql mytestdb
-- <DBNAME># \i complete_action.sql
-- <DBNAME># insert into example_table (name) VALUES ('something');
-- <DBNAME># insert into example_table (name) VALUES ('something_else');
-- <DBNAME># DELETE from example_table where id=1;
-- <DBNAME># DELETE from example_table where id=2;

Conclusion

With larger datasets and more realtime data streaming through tech companies, I look forward to seeing and working on more log based architectures. These systems provide resilience, fault tolerance, simplicity, and scalability. By funneling events such as PG’s row modifications into a Kafka stream, we build a robust system of keeping ElasticSearch in near-realtime sync with Postgres.

Feed in data from Postgres, from various server logs, from user events, and from business metrics. Soon the kafka firehose is a central river of data running through the organization.

tl;dr - Use a VPN (answer to many questions).

Verizon and other major cellphone carriers are inserting tracking tokens into all our requests made from smartphone. And if you tether your phone as a hotspot, it’s adding tracking tokens there as well.

Verizon states that the Opt-Out form on their website only disconnects your unique id being associated with demographics data.

There are a couple good websites for checking if your cell data is being injected with the tracking header. My favorite is by researcher Ken White: link.

But if you want to check it on your own hardware, here’s the tiny sinatra application I setup:

Start that up by installing Sinatra gem & then running ruby app.rb.

Next, go through the hassle of pointing your phone at it. I did this by using ngrok. Start that up with ngrok 4567 where 4567 is the default port for app.rb’s Sinatra Webrick server.

Note the output from ngrok 4567

We want the line that says Forwarding http://a5c92a1.ngrok.com. We’ll use that url when setting up proxy on Android cell phone.

Now, make sure you’re not on Wifi data, and visit the output from Ngrok in your cell phone’s browser (ie http://a5c92a1.ngrok.com for me, yours will be different).

Tada! Thanks Verizon for tagging every web request with a header :-/.

So, what can we do to protect against this?

First off, sites using HTTPS aren’t susceptible to this header inject. But that leaves a good chunk of the internet that’s vulnerable. Also, when using Wifi data Verizon can’t inject the headers because our data isn’t going through their network.

I tried two different methods for dealing with this, first was to setup a Squid Proxy that strips UIDH values from requests before passing them along. This worked, but isn’t as robust as the alternate solution, which is to use a VPN on cellphone. If you’re using an iPhone, GetCloak is a nice service run by good people. It will route your traffic through their VPN. I’m currently using their app on OSX for easy VPN that avoids leaking lots of data when in coffeeshops.

With an Android phone, you need to be a bit more savvy. Your convenient option is to ask GetCloak nicely for their unsupported/unofficial OpenVPN endpoint. I set it up on my cellphone by using OpenVPN.app. Very straight forward setup. The alternative would be to setup your own VPN endpoint, which was more work to get it properly forwarding all traffic.

So the short answer to Verizon spying on users? Use a VPN. And switch providers when there’s a company actively resisting such despicable practices.

A friend recently told me that they’d received a job offer that was 12% more than their current employee salary. Then they told me it was contract work.

This spurred a conversation about income that software developers should understand.

Employee salary != Contractor Income

Let talk about being an employee first (US-Centric) and attribute a dollar value to everything:

Employee Income:

+ Salary $90,000/yr
+ Company pays half of your social security tax and medicare tax (6.25% + 6.25% + 1.45% = 13% ~= $12,000/yr)
+ Healthcare benefits ($500+/mo, ~= $6,000/yr)
+ Sick days (5/yr @ $400/day ~= $2,000/yr)
+ Federal Holidays (10/yr @ $400/day ~= $4,000/yr)
+ Paid Time Off (10 days @ $400/day ~= $4,000/yr)
+ Company 401k retirement matching (up to 3% of salary/yr ~= $2,700/yr)
+ One software conference paid per year
  + 2 days paid * $400/day = $800
  + Flight = $400
  + Hotel  = $500
  + Conference Ticket = $400
+ No equipment cost for employee

= $123,425/yr Equivalent contractor income to $90,000 as employee

$90,000 is worth $123,425/yr when accounting for benefits.

So roughly, we can add 35% value to an employee income to get their contractor income.

Contractor Income

Now imagine it from the other side if this friend accepted the contracting job for a 12% raise over their employee salary:

+ Income $100,800
- Healthcare $6,000
- Sick Days  $2,000
- Holidays   $4,000
- PTO        $4,000
- Retirement matching $2,700
- Conference budget $2,100
- Equipment/office/cell/laptop $5,000+

== $75,000 equivalent employee salary

Yep, that’s an employee salary of $75,000 after paying for your own benefits.

Conclusion

Taking that job would be a loss of $15,000/yr, even with it being 12% more than employee salary, after factoring in lost benefits.

More things to consider

We did not take into account that employment can be more stable, depending on the laws of that state. We also ignored the benefits of contracting, such as running your own business, possible geographical flexibility, and excitement of working on new projects. These personal growth and lifestyle preferences can be thrown in as weighted values. How much are they worth?

Credit

Many of these calculations are based on http://rakkar.org/ContractPayCalculator.html. Thank you for clearly laying out the details of what needs to be accounted for in determining employee vs contractor income. To see a clojure re-implementation of the Js source of that website, checkout this gist:

For the last 10 months I’ve been working on a remote team of 30+ engineers and additional non-engineering teammates. We make it work quite well and I want to share some of the secrets for making it work.

Buy-In Requires Everyone

Structure all activities with the expectation that part of the team will be remote: meetings, sales peptalks, tech reviews, coding training, etc.

Encourage all staff to work from home 1x or more per week. This sets the tone and builds a shared understanding of how to communicate when you can’t walk into someone’s office.

Implement tools that support remote work, this means a central chat system, voice/video conferencing, screensharing solutions, etc. We use a combination of:

• skype (phasing out)
• HipChat (IM communication)
• Sococo (virtual office w/ screensharing and VOIP in intuitive package)
• Tmate (terminal sharing for pair programming)
• PivotalTracker (good for documenting and viewing progress asynchronously)

We do a fair bit of pair programming to limit the amount of knowledge silo’ing and this helps introduce new coders to the team. It also builds relationships so we know who we can ping for expert level Javacsript questions or who knows about the obtuse Oracle database behavior.

Occasional Physical Meetings

I’ve met the people on my team three times in 10 months. The first event was a week long onboard process. The next two events were company retreats where we left our normal responsibilities and descended on an unsuspecting vacation rental.

The retreats are a nice way to spend some non-traditional work time hacking on more creative projects. Also helps to build relationships. Plus it’s a nice thing to do to improve engineer retention.

Reasons to do remote hiring / build remote team

1. Wider selection of applicants.
2. Applicants bring a breadth of backgrounds, not just SF based-engineers.
3. Allows time-zone diversity in case critical tasks arise outside of PST. Early morning downtime? Get those east coasters on it!
4. Potentially better engineer retention if there’s time flexibility for dropping kids at school, lunch errands, etc. The guideline should be “did the work get done”.

Things that help me

• Keeping regular hours
• Overlap 5+ hours per day with our main timezone. I normally keep 8 hrs of overlap.
• Learning to be very forward with questions that other teammates might know. If I’ve spent 30 min banging my head, it’s time to ping someone else for a reality check.

First off: it’s a good idea. You have a chance to apply problem solving and creative skills on a daily basis. You can find amazing work conditions. Your skills will be in demand (depending on language and tool choices). You could do remote work from the comfort of home for a salary well beyond the average American income.

Let’s Talk

I know someone trying to make it into the programming world by way of DevBootCamp and she, Krystyna aka @Wimsy113 asked me about thoughts on learning other programming languages such as PHP or .NET.

She’ll be learning Ruby, Javascript, and maybe some SQL basics during her time in DevBootCamp.

What are my thoughts on other languages at this early stage?

I’m not a language purist and don’t do well with dogma. PHP / .NET / Ruby all have their place and all can be part of one’s career. They fulfill similar tasks, all being competent languages for web development. I’m biased towards Ruby and PHP because their ecosystem is based on Open Source Software, but they all get the job done.

Since it’s early in her learning, here’s what I think is the best choice for now:

• Stick with Ruby and leave PHP and .NET alone for the time being.

Ruby will form the backbone of her education at DevBootCamp and neither PHP nor .NET will prepare her as well as more plain Ruby. After she completes that intense training and has a basic footing in Ruby, checkout some other languages but realize that Ruby is a solid money-maker for the time being.

High Level View

What’s the high level view of how languages will enable her to solve problems and what priority should they get in her limited time to study? Items are ranked in general order of importance for her career advancement and taking into account that there’s a finite amount of hours she can study.

• Ruby is a solid backbone and glue for solving problems through building websites. Ruby’s a must have skill and proficiency will mean job advancement.
• HTML. It’s a must have though probably won’t occupy most of her time at work.
• Linux/Mac commandline. It’s not glorious but it’s a required skill for this kind of web development. It’s as close as web dev work gets to getting hands dirty, and it’s fun :).
• Javascript should be part of the education at DevBootCamp. This enables her to make websites more interactive and live, vs a straight Ruby application. It’s another must have skill nowadays for web development.
• SQL, it’s the nearly-universal language of relational databases. A basic understanding is good. Whether it’s highly valuable or of little value depends on the role and company. Some shops have dedicated database admins, others leave those responsibilities to software developers. It’s a nice to have but not necessary to master skill (at least early on). I’d wait on this until being more established as a developer.
• CSS. Good to know some but the heavy lifting might be done by fulltime designers. In the last 9 months of work, I haven’t needed to write a single line of CSS. Though I do a little bit on side-projects while freelancing.

Once she’s comfortable with Ruby/Commandline/Javascript, then branch out and dabble in other languages. Re-create something in a new language for fun: PHP, .NET, Clojure, Haskell, whatever! Learning new languages will improve her programming mind both in the new language and in her known languages.

And for a little bit of unsolicited advice, aka a few things I wish I knew when getting into software development:

1. Attend Meetups and Conferences related to your languages/tools of choice. For me this would mean Ruby, Elixir, and Functional Programming. These are some of your potential future co-workers. Share your excitement, they’ll love it.
2. Talk to everyone you can at conferences/meetups.
3. Talk to folks on twitter and get them engaged and excited about your journey. Chat with or research @joshuakemp01 for a good account of how this pans out.
4. Blog about the little details of your journey. Both the technical stuff and the problem solving tricks you learn.

Remember: it’s a marathon. So keep your head up, look after yourself, and catch your breath when necessary.

Help Krystyna Achieve Her Goals

Donate to her fund for DevBootCamp: http://www.gofundme.com/8i0mms

Looking for a way to run a command each time a file(s) changes that can work across multiple platforms?

Previously, I would have used guard but I’m trying to break my reliance on Ruby tools.

Enter entr a commandline utility written in C. It does the same thing as Guard but without the dependency on Ruby and without needing verbose configuration files.

Installing on OSX is simple: brew install entr and other platforms look like:

./configure
make test
make install

Using it is even simpler. I’m using it to rebuild a presentation which means re-running a make task each time the source file changes. If the command were being run manually it would be:

make reveal

With entr installed we can run:

find dir/ -name *.md | entr make reveal

Or for convenience we can wrap it in a shell script:

#!/usr/bin/env bash

readonly FILENAME=surviving_large_unfamiliar_codebases

(cd ${FILENAME} && \
  echo ${FILENAME}.md | entr make reveal)

Edit:

In case only a single file needs to be monitored, try out this BASH script:

2024

HTAP Summit Keynote “Driving TiDB Adoption at Plaid”

2015

1. “How to Conf” - StirTrek Conference Co-presented with Matt Darby
2. “How to Conf” - Kansas City Developer’s Conference Co-presented with Matt Darby
3. “Remotely Successful” - Kansas City Developer’s Conference Co-presented with Matt Darby

2014

1. “Surviving Large Unfamiliar Codebases: A Harrowing Tale of Legacy Apps and Oracle Databases” - 757rb User Group

2013

1. “Introduction to Elixir” - Pittsburgh Ruby User Group

I’ve recently gotten a little hooked on Golang. Strangely enough, I think it finally started to click for me as a result of one Haskeller and one friend Christopher Stingl.

Christopher was kind enough to link me to this wonderful Introduction To Golang. And as a result of the Haskell reading that I’ve done lately, Golang started to make sense.

For a little background, I’m coming from a Ruby development standpoint with no prior formal training in compiled statically typed languages.

First Impressions

package sack

import (
  "fmt"
  "github.com/codegangsta/cli"
)

func shellInit(c *cli.Context) {
  sh := `
    sack=$(which sack)

    alias S="${sack} -s"
    alias F="${sack} -e"
    `

  fmt.Println(sh)
}

func shellEval(c *cli.Context) {
  sh := "eval \"$(sack init)\""
  fmt.Println(sh)
}

I’ll come clean, first impressions weren’t good. I picked up the language a couple months ago and tried to implement a simple line parser. Didn’t go so well because I got caught up on the various statically typed bits of the language. Essentially, ‘what do you mean I have to know what’s going into my function and returning from it? I don’t even know what types are available’.

But after reading through that 5 Week Intro to Golang and doing some Haskell homework, it clicked. And I liked it.

So let me back up and explain the static typing for those readers that don’t have any background in such languages. In Go, you define functions (the rough equivalent of a method in ruby) as having specific stypes of inputs and outputs. So you might define something as follows:

func Version() string {
  return "0.3.0"
}

This function will take 0 arguments and return one argument of type string.

Or this function:

func executeCmd(term string, path string, flags string) []string {

  var lines []string
  _, err := exec.LookPath(agCmd)
  if err == nil {
    lines = agSearch(term, path, flags)
  } else {
    lines = grepSearch(term, path, flags)
  }

  return lines
}

Which will take three strings as arguments and return an array of strings.

Initially cumbersome during the learning phase, but incredibly helpful. I’ve found myself wishing for a similar safety measure in the language I use for my dayjob, which is Ruby.

And the best part is that the compiler won’t build the program unless all your type ducks are in a row! Which prevents the need to write about 30% of the tests that I see on a daily basis. In fact, many of the bugs that I’ve seen in production would be avoided in a strongly typed static language.

Advantages

• Static typing safety
• Cross compiled binaries
• Super easy deployment (b/c the whole go world is wrapped up into that binary)
• Fast runtimes
• Encourages less usage of mutable variables vs. Ruby (though not as good as pure FP language)

Outcome

After playing around with that tutorial, I couldn’t resist reimplementing a commandline tool in Golang. It’s a tool that started as a shell script, which I reimplemented in Ruby, and now have reimplemented in Golang. It’s called sack and you should use it as the glue that connects your silver-searcher or grep results with your $EDITOR. Project link.

The other outcome was that I worked on a project during our 10% time to build a golang web application that serves up a JSON feed of Whois information. It was amazing to me that 2 weeks into learning Golang I was able to put up something useful over the course of two days.

I’m looking forward to experimenting more with Golang, possibly by setting up a Goship server at work. It also affirms for me that I’m missing out on some wonderful programming paradigms in the Ruby world. I expect to see more Functional Programming in my future, whether that’s in the form of Erlang/Elixir/Clojure or a more mathematically pure language like Haskell.

As developers, we’re familiar with the basic software tools that grease the wheels of the programming world. Version control systems, deployment tools, notification hooks, CI servers: they all make our jobs easier.

But how can we work better as an organization when our startup exceeds 2, 4, 8, 16, 32, 64, 128 engineers?

There are three essential tools/patterns that need to be implemented for a development team to be successful.

1. Project Management Software
2. Git workflows and commit message standards
3. Cultivate your Developers’ creativity

Project Management Software

If you’re like me, you hate long running email threads that attempt to pin down the acceptance criteria and context of projects or stories.

Pull the email escape hatch and move into the 21st century!

A good start is having a unified interface for tasks, ie a Project Tracker. This gives the technical and non-technical folks a single point of interface for all the knowledge of “what’s happening” and “what timelines are we operating under?”. It’s incredibly important that the software is convenient to use & updated in realtime.

Though I don’t get any referral bonuses, my favorite thus far is Pivotal Tracker and if you’re an Emacs user check out Pivotal-Tracker-mode or my fork of that project. I prefer using the Emacs interface rather than their web interface when working in a large organization because it’s faster.

So now you have a single channel of communication and record keeping system for projects, YAY! Rambling email threads will (mostly) fade into your company’s past!

Git workflows and commit message standards

Next, come up with a style guide for Git commits and version control practices. Maybe you use ‘git-flow’. Maybe you come up with your own variant of an established workflow. But discuss it among the engaged members on the team.

I strongly recommend running all code changes through other members on the team. A spare pair of eyes is invaluable as is the unique context/domain knowledge that a team-member brings to the table. If you’re using a web based interface for Git such as Github, consider using “Pull Requests”. They’ve become a standard practice in the Github using community. If you have a team using an alternate git-based solution, consider submitting all code as patches through the Dev Leads. Doing so will catch some issues before they make it into production.

As for git commit messages, educate your developers on why they’re important.

Take the following commits:

Fixed thing that was wrong

or

[ZPH/KM][#3489634] BUG/Updated Jquery selector for dashboard login box

Dashboard login box functionality was broken (SHA a98adf91) due to a change in `dashboard.slim`.

Changed selector to $('.login_box') to match slim file.

Can you imagine which is easier to reference 6 months in the future? Say, when you’re investigating why the dashboard login is broken? Or maybe you’re a Dev lead looking for the commits related to a specific project story. If you’re on a team where stories are completed in pairs, it’s a good idea to include the initials of both parties who worked on the commits. Then it’s simple to answer the question of: “Didn’t Zander and Kerri do something with our dashboard last week?”.

If you have developers writing descriptive commits, finding answers to questions is as easy as git log -S dashboard for a list of each commit involving the word dashboard.

Care and Feeding of Developers

Last and certainly not least is related to the care and feeding of software developers. We might not all be beautiful and unique snowflakes (we in the Ruby community do largely use Apple computers after all) but we require regular care to stay sharp and engaged.

10% Time

Set aside dedicated time to have developers work on more creative projects. Why bother?

• Developers are valuable and losing them is expensive.
• Happy programmers are productive programmers.
• Who else could be better situated to see inherent flaws in existing software/tools/workflows in the organization? Let them scratch their own itch.
• Doing the same tasks day in and day out is a sure-fire recipe for dulling even the brightest minds. So knock it off!

Build in time each two weeks that’s dedicated to creative projects envisioned by the coders.

Where I’m currently working, we do this every 10th business day and it’s called our 10% time. On those days, we come up with interesting projects that advance the company’s interests without the same pressure of failure. These are great times to implement something in a new technology or setup those Git hook integrations with Pivotal Tracker. Fix things that are slowing you down or bothering you on a regular basis. What else could you work on?

• An idempotent way to setup new developer workstations (script based please).
• A unified script for firing up the application stack on a local machine (or stopping, reloading, updating, etc).
• Team chat integrations like Campfire + Hubot, HipChat + Github Merge notification.
• Dashboard for your income generating parts of software. Track those leads in realtime!
• Build tools to simplify your Quality Assurance teams’ lives. Or build tools so Product can be more efficient.

10% is the perfect time for fixing these nagging issues. The only requirement is that the developers generate most of the 10% ideas and get to vote with their feet (or digital feet) in choosing what tasks to work on. They’ll self organize into teams where they can be most efficient. Cultivate your developers and they’ll flourish! Also, you won’t have to engage recruiters because your retention will go through the roof. It’s a win-win situation.

Thoughts, comments or jeers are all encouraged! Come have chat w/ me on Twitter @_ZPH.