weblog.masukomi.org - geeky

The Thing About Local LLMs

2026-06-06T00:00:00-04:00

Table of Contents

Overview

Someone responded to yesterday’s post about the Pi coding agent, asking if I was using it with local models. I’ve done a lot of testing with them, and I can say with great confidence that the hype about using them is mostly just that.

At the same time, Pi is a truly remarkable tool that is not only your best option for working with local LLMs, but also - in my opinion - your best option for working with any LLM.

The Reality of Local Models

Unless you’re rich and can afford a beefy PC with multiple top-of-the-line GPU cards, there are very few edge cases where running a local LLM model makes sense.

The videos claiming you can run them on a normal laptop or a Raspberry Pi are only technically correct. Practically they are all lies. They have all had the delay between prompt and response edited out, and are probably running models that aren’t even remotely comparable to what you get from cloud providers.

You can run tiny models and get… bearable response times, but they’re 💩 for coding.

My wife has a gaming PC with 1 good GPU, and in theory we can get load Qwen3-Coder-Next 30b models into its memory, and get output similar to one of Anthtropic’s Sonnet models. I believe that, but what’s unclear is how bearable the delay will be.

She and her research partner Rhyannon (👋), have some things they want to investigate that involve or are facilitated by using local models, so we’re going to be seeing what that looks like and how usable it is.

The Mathematical Reality of No

So far, my experience has been that even on a maxed out mac laptop the only use cases that are practical are background processes where you don’t actually care when it finishes. Apple’s claims about their “Neural Engine” are just hype. Yes, it’s probably better than CPUs without something like that, but good models are large - like really large - and if you want to use that model interactively you need to load the whole thing into the GPUs VRAM.

The back-of-the-napkin equation is pretty straightforward: M = Np * Bp where

Symbol	Description
M	Memory in Gb
Np	Number of parameters in Billions
Bp	Bytes per parameter

Some pre-calculated examples for models you might want to use:

model	Model VRAM Size	Activation Mem	Total Needed
qwen3:4b	8GB	~3.2GB	11.2GB
quen3:8b	16GB	~6.4GB	22.4GB
llama3.2:3b	6GB	~2.4GB	8.4GB
llama3.2:1b	2GB	~0.8Gb	2.8GB

On top of that we need to consider “Activation memory” - temp data used during inference or training. That’s generally another ~20-40%. In the table above I’ve used 40% to make sure we have “enough”. Note that Training (not something we’re discussing) requires 2-4x the parameter memory. Note that that extra 20-40% is required for each agent and subagent hitting the system at the same time because they each have their own unique blob of text that inference needs to be run on. As soon as you exceed the VRAM of your GPU you start having massive performance penalties.

Quantization of models helps lower the VRAM requirements, but the fact of the matter is that you need way more VRAM dedicated to the GPU than any laptop can provide.

Pi Changes Things

I think Pi isn’t just the right paradigm / approach for coding agents. I also think that it’s minimalism is required for doing anything interactive with a local LLM. The things it can do to accommodate restrictive hardware end up working out a lot like accommodations for people with physical disabilities. Yes, it helps them, but it also helps everyone else.

All the other coding agents are financially motivated to use more of your tokens. Anthropic doesn’t care how wasteful Claude is of tokens as it does its work. The more it wastes the more you end up paying. Pi enables a lot of important things that minimize token usage:

trivially small & modifiable “system prompt”
encourages putting code in extensions not skills
ability to override everything with custom code
the ability to treat interactions with the LLM as a tree

The System Prompt

It’s been said that the thing that truly differentiates the Agentic platforms is not the models. It’s all the infrastructure and hidden prompts the vendors have built around their models.

The system prompt is one of those. It’s included at the start of every session and sucks down thousands of tokens. Claude’s changes between releases without warning, includes content dynamically generated by the agent (presumably more token usage) and there’s no easy way to see what it is. I’ve tried overriding it with the --system-prompt-file a_file.md flag and --system-prompt "some text" but it failed to follow the instructions I put in the prompt in both cases.

Pi’s system prompt is known, tiny, and modifiable.

Extensions over Skills

Agentic Skills are ultimately just Markdown documents that have to be read in and reasoned about before the system makes some non-deterministic choices about what to do, and what tools to use to do it with. You can’t actually control it’s actions. It’s like giving instructions to a grade-schooler. It doesn’t matter how precise those instructions are, you can never be 100% sure they’ll be interpreted correctly, or consistently every time they’re read. Also, that reading uses tokens.

Pi extensions are code. You invoke them via a prompt just like skills, but once its invoked it’s just a matter of waiting for the code to finish its run and either hand something back to pi for further processing, or tell it it’s done. It doesn’t matter how complex the extension is, it doesn’t use any tokens unless you’re explicitly asking it to interact with the model.

Overriding Everything

It’s not so much that you can override everything, because most things in Claude code - for example - seem possible to be overridden with custom versions given a sufficient application of Markdown (and prayers). It’s that you can override everything with deterministic code instead of throwing a block of Markdown text at it and hoping it interprets it correctly. Additionally, you don’t have to pay the token cost of processing those markdown instructions because - with Extensions - it doesn’t exist.

Conversation Trees

Models have no memory. The ever-growing history of your conversation with an LLM in each session keeps getting sent to the LLM with each successive prompt. It has to read in the entire thing from scratch with every new prompt. In addition to just being wasteful, this can be problematic when the human or the agent have gone down side quests that either didn’t pan out, or weren’t relevant to the primary task at hand.

A good example of this is the human asking the agent to explain something it just did. It’s useful to you, but each successive prompt to the LLM will have that useless blob of text to process.

Pi’s sessions are stored as trees. This means that, in the example above, you could run /tree and jump the context back to just before you asked it to explain. More significantly, if you and the Agent had been exploring some way of solving a coding problem only to find that your thinking was bad, or that it wasn’t possible, you could jump back so that those “bad” ideas were no longer things that had to be considered on each successive prompt.

Making Useful Structured Commits That Become Changelogs

2026-04-04T00:00:00-04:00

Table of Contents

Overview

git-com is a tool that makes it easy to create structured commit messages. It exists because of two roles I’ve played at work:

release manager
Leading a support engineering team tasked with putting out production fires.

Both of these jobs involved investigating what actually ended up in a release. With the support engineering role I needed to find out what got added that might have broken a thing. With the release manager role I needed to find out what actually got added as compared to what was promised, or never mentioned. Then, I needed to create a document to convey those changes.

Having a readable structured commit history would have been huge for me back then. Now that I do have one it’s proving invaluable.

This post discusses how to think about creating and utilizing structured commits in a way that will help you accomplish two goals:

finding specific changes in your history
conveying what changed to others via generated changelogs

None of this requires git-com, but using it will definitely facilitate the process, and it’s written from the perspective of someone using it.

What Should Be In Your Commit?

git-com is intentionally not opinionated about this. What is “useful” or “good” varies wildly from project to project.

The one guiding thing I can say with a lot of confidence is that you should never add structured data to your git commit messages that you don’t need. Leave off the “nice to have” items. Every “unnecessary” step you make someone take (even you) when creating a git commit is another tiny piece of built up resentment.

Titles

Getting the title “right” is the most important thing. There are many cases where the title is the only thing you’re going to see until you dig in for more info. Thus the title needs to convey the most critical information in order for you to know if you want to get more info. The title is frequently the only thing that gets written in a commit, and they’ll show up in your changelog, or become the starting point for what shows up.

There is, however a significant limitation on git titles: They should be ≤ 72 characters. Ideally they should be around 50.^[1] That makes for a tricky balance.

Additionally, you need to think about what will be useful to the tooling that will take the commit messages and convert them into a changelog.

I’ve addressed these problems in two ways.

Start with structure

All my git-com generated commit messages start with [<change type>]. This serves two purposes.

The square brackets draw your visual attention. They say “hey this is most critical bit of this line”
It’s not something I’ll bother with when making a quickie “corrected spelling errors” type commit, and thus it becomes easy to distinguish “stuff worth conveying to others” from “one of the many little changes that got made but aren’t worth talking about”

Because the contents of those square brackets come from a limited set of choices defined in my .git-com.yaml file, it also serves as a way for my changelog generator to easily distinguish between a changelog worthy commit and something that just happens to start with square brackets. This also means that you can have commits in your git log that, by virtue of having no structure don’t end up in the changelog.

Personally I use git commit -m "corrected spelling errors" for things not worth conveying, but if you want to build the habit of always using git-com you have two options.

use a specific change type that will be visible in your commits but excluded later. E.g. [meh]
make the change type option “skippable”. The skip option appears first so you can just whack the enter key if you don’t care about choosing something from the list.
```
    allow-empty: true
    empty-selection-text: Skip
```

Neither of these is “better”. I like the idea of having trivial commits start with [meh] It sounds entertaining. On the other hand, I can see the argument that you’d want to draw as little attention as possible to commits that aren’t noteworthy. That way humans scanning git log output don’t have their eyes drawn to unimportant things.

I’m firmly of the belief that the type of change is the most important thing to have in a commits title in order to help anyone who needs to look through the log to find something specific.

I also believe that change type is an excellent way of breaking down changelogs of software products. I don’t want features, fixes, and tweaks to all be mixed together in a jumble of a list. Sometimes I want to know what cool features a project has added. Sometimes I just want to know if they fixed a specific bug.

Keep it concise

Not because of the character limit, but as a way of being nice to future humans. The freeform portion of the title should contain only enough information for a reader to know if it’s worth their attention.

Ex. “full-text search refinements” not “reworked our use of faceted search to prioritize items we sell”

Additional details like that can be added to the commit body. Just give me enough information to be able to determine if this commit is the one I care about or need more details on.

Different Text for Different Audiences

Git logs and changelogs are things generated for different audiences. Git logs are for geeks looking into the details of a codebase. Changelogs are for people who use your tool, or maybe for stakeholders within your company who have been involved in guiding the tool. They have different needs.

Because of this, it’s important to keep in mind that the text that appears in your git logs isn’t necessarily the text you want to show in your changelogs. With regards to the structured text this is easily handled with aliases, and leads to some interesting possibilities.

Git wants short titles, so find short ways to represent things like change type in actual commits. “feature” is a lot longer than “add” or “feat”, and you don’t really need to indicate that “feat” is an abbreviation by adding a period (“feat.”).

Personally I’ve settled on

add
fix
clean-up
refactor

You’ll note that the last two could be shorter. I want them to be shorter, but “cleanup” annoys me, and “clean” feels wrong. I also haven’t thought of a shorter way to say “refactor”. What you end up will be the result of a highly subjective balancing act.

However, as I noted before, the thing in your git commit message doesn’t have to be identical to what’s displayed in the changelog. So you can put short things in the commit message that end up as longer things in the final communication. You can also put in things that will be excluded. Like fixes to a security issue no-one noticed yet.

My changelog generation tool reads in my .git-com.yaml file and leverages the fact that it supports the concept of “meta elements”. These are elements that exist within the config but aren’t for consideration as part of your commit message.

meta_element_changelog:
    display_aliases:
        fix: "fixed"
        add: "added"
        refactor: "refactored"
        clean-up: "cleaned"

When building the changelog it finds commits with [add] and groups them under an added heading because of the mapping in display_aliases, but this could be anything. Maybe “add” commits end up under “❇️ New Features”, or the other way around. Maybe ❇️ in a commit title indicates new coolness and gets grouped under “added”. Keep in mind that users of git-com are usually choosing from a pick list, so you don’t have to know how to type special characters like ❇️ to have them added in the commit message.

Other Representations

Note: square brackets aren’t the “right” solution here. They’re just a solution. Here’s some other ways you might handle the same thing. What’s important is that you find a way to represent things that you and your team (if working with others) can agree makes sense and isn’t annoying.

# <optional breaking change> <change type> <separator> <freeform text>
⚠️ (feature) → full text search
# nonbreaking version…
(feature) → lists recently completed items

# Minimal Emojis
# <indicator structure> <change type> <separator> <freeform text>
# ℹ️                    (fix, add, etc.) :           blah blah
ℹ️ fix: now supports linked worktree directories

# FULL EMOJIFICATION 🤪
# <indicator structure> <change type> <separator> <freeform text>
# ℹ️                    (❎, ❇️, ♻️ etc.) ▶️       blah blah
ℹ️❎▶️   now supports linked worktree directories

# maybe add special emoji to indicate a breaking change
# <indicator structure> <change type><optional breaking change><separator> <freeform text>
ℹ️❇️⚠️▶️ full text search
# Breaking because it might require a db upgrade or something noted in the commit body

Body

Here’s where things get extremely subjective.

Obviously you need block of freeform text for people to (optionally) describe what the commit is about, but the question of what - if any - structured data you should include is where things get complicated.

I’ve chosen to go with a multi-select for portions of the codebase that are involved in the change. The list of options for this is very different for each project I use git-com in, but what they have in common is that each item represents a significant distinct part of the codebase, and it’s mostly sorted so that the frequent items are near the top.

This data never shows up in changelogs, and I rarely ever read it. I do this because of my time spent hunting down production bugs at work.

For example, let’s say the look of our site had become borked. I’d have a pretty good guess it was changes related to the “css” portion of our codebase or maybe something in the “ui” layer. It would have been invaluable to me to be able to run a command like this and find commits since the last release that included changes to the relevant sections of the codebase.

git log v1.2.0..HEAD --grep " (css|ui)" --extended-regexp

Here’s what I see when I run that in a codebase that uses those two keywords in its “Code Sections” (see example below)

Commits since a release in matching sections of the codebase.

If a search like that overlapped with too much of the freeform text I could limit the results to the “Code Sections” metadata line by doing this.

git log v1.2.0..HEAD --grep "Code Sections:.*(css|ui)" --extended-regexp

I experimented with Tags and “Code Sections” it ended up feeling like unnecessary paperwork that I was resenting. Again, my advice is to only add sections that you have real use for.

Example Config

The list below is from the BackupBrain webapp which is currently focused on managing, and archiving bookmarks.

code-sections:
    type: multi-select
    allow-empty: true
    modifiable: true
    record-as: joined-string
    before-string: "\nCode Sections: "
    destination: body
    instructions: Which section(s) of the codebase?
    options:
        - bookmarks
        - archives
        - ui
        - ux
        - css
        - config
        - meta
        - tests
        - migrations
        - infrastructure
        - tags
        - oauth

Changelogs

I have plans for creating a Changelog generator that can read in a project’s .git-com.yaml and extrapolate the rules it needs to read and format a changelog. Until then though, you’ll have to approach this problem on your own. The changelog tool I’m currently using is called git-com-changelog and can be found here.

The goal

Creating changelogs from commit messages has been the most useful and meaningful thing I’ve done with git-com. It doesn’t just give me good changelogs. It’s also wired into my release tooling so that:

new version tags include the changelog in their body
my CHANGELOG file gets the latest info prepended to the top with markdown or org-mode formatting depending on the file-type.
my Github Releases inherit the changelog text from the tag to populate their bodies.

For some of my tools, cutting a new release means typing release v1.2.3 and that’s it. Changelogs get updated when needed, tags get cut when needed, releases are created with changelogs in their body, artifacts get built and attached to the release, homebrew formulas get updated and pushed.

Changelog example:

v1.7.0

[fixed]
- hitting ESC at a confirmation now exits
- tweaked how lefthook's go-tester works
- commits work in linked worktrees
- maintains yaml element ordering when adding an option

[added]
- final validation now leads to edit if NO is chosen
- added some lefthook files
- now supports include-empty attribute
- stores commit message for reuse when pre-commit hooks fail
- stores & offers up commit messages from failed commits

[refactored]
- faster startup
- hook output now streams live

[cleaned]
- backporting some notes from web site
- removed instructions.org

Making it Happen

Getting from commits raw commits to useful changelogs is surprisingly easy. There are three aspects to the processing.

read .git-com.yaml to figure out what goes into a prompt
optionally read the contents of the meta_element_changelog section to get processing rules & modifiers.
gather commits, filter, group (optionally) and output.

1. Figure out what to look for

Limiting your input

You don’t want to reprocess your entire git history every time. So, I recommend that your changelog tool can take standard parameters for limiting the commits it has to work through. I’ve chosen to go with a --from and an optional --to parameter. My tool just passes those limiting bounds on to git log when asking for commits to process. This is especially useful when you start building automated changelogs into your build process.
```
# everything between the v1.5.0 tag and the v1.7.0 tag
git com-changelog -f v1.5.0 -t v1.7.0
# everything from v1.5.0
git com-changelog -f v1.5.0
```

Knowing what ’s worth considering

The first step is knowing what structured data you can always count on existing in a commit worth including. In my case it’s a commit message whose title starts with a change-type indicator [foo] where “foo” is any of the currently supported change type options.

So, for me that means reading in the project’s .git-com.yml, and asking for the contents of the change-type element’s options.
```
change-type:
    destination: title
    type: select
    modifiable: true
    instructions: What kind of change?
    before-string: '['
    after-string: ']'
    format: "%-12s"
    options:
        - fix
        - add
        - refactor
        - clean-up
```

2. Apply modifiers

Note: This is completely custom, but I think it’s an idea worth copying.

git-com supports the inclusion of custom YAML in its config file as long as the top level element begins with meta_element_ I’ve been using meta_element_changelog to store project specific configuration / behavioral instructions for my changelog generator.

None of the following is built into git-com, but my changelog generation is based on the git-com configuration at the time of invocation, so it makes sense to store it all together. You can make your own meta_element_foo with whatever information you find useful for helping generate a great changelog.

I have 4 types of assorted modifiers.

display_aliases

These provide a display alias for the heading each type of commit will be grouped under. For example you can see that any commit with a change-type of “add” will be displayed under a heading of “added”, but it could be something fancier like “❇️ New Features”
```
meta_element_changelog:
    display_aliases:
        fix: "fixed"
        add: "added"
        refactor: "refactored"
        clean-up: "cleaned"
```

retired_aliases

Because ideas and conventions change over time, clean-up may get replaced with something like clean. This happened a lot as I was getting started with git-com and figuring out what worked, and didn’t work for me.

As a result, I’ve added in support for retired_aliases. So, if clean-up changed to clean then I’d change my meta_element_changelog section like this.
```
meta_element_changelog:
    retired_aliases:
        clean-up: "clean"
    display_aliases:
        fix: "fixed"
        add: "added"
        refactor: "refactored"
        clean: "cleaned"
```
Note that under display_aliases clean-up has been replaced with the new term which would have been added to the options of change_type at the top of the config. The new retired_aliases section tells the code that if it finds a line with clean-up as the change type it should replace it with clean and then process as if it was clean all along.

“permissive” mode

My changelog tool has a “permissive” mode that can be toggled with a command line argument. It basically means “hey, if find a change type that isn’t in the list, just consider it valid.” It’s good for testing but I don’t recommend it in practice.

exclusions

While I don’t use it, I’ve added support for an optional “exclusions” list. After applying any retired_aliases it checks if the current change-type is in the exclusions list and skips it if it is.
```
meta_element_changelog:
    display_aliases:
        fix: "fixed"
        add: "added"
        refactor: "refactored"
        clean-up: "cleaned"
    exclusions:
      - security
      - meh
```
Given the list above, and commit with a change-type of “security” or “meh” would be excluded from the changelog.

3. Generate Output

Most of you will be generating markdown flavored files, but if you want to use this with git tag, you need a way to generate useful output with no markdown headings. The problem is that any line that starts with # is considered a comment by git, and is completely ignored. This means all your markdown headings will be stripped.

I’ve solved this by having it generate output without markup by default (like in the example above). It also supports --markdown and --org flags. Most of you won’t need the org one.

Because I like grouping similar changes (again, see the example) I create a “hash” / “dictionary” with keys for each encountered change-type, then build up a list of all the commit titles as its value (after stripping off the change type).

Then I just iterate over the collection. I use any display_aliases I found to change the heading text. For example changing “fix” to “fixed”, or “add” to “added”. For each line I apply the appropriate markup if the user requested it. That’s about it.

Consistency Consistency Consistency

The more projects you work on the more critical it becomes to have a consistent structure to your commit titles across projects. I can leverage my changelog generator across all of my projects because they’re always the same.

[<change_type>]<space><freeform_text>
For example:
[fix]     commits work in linked worktrees

If I start using a different structure in one of my projects I’ll have to create a new changelog generator or make modifications to the existing one specifically to accommodate the second format.

Summary

Make commit titles include a small piece of consistent structured text that’s easy for humans and code to pick out. Having this be the change-type of the commit is a great place to start.
Don’t gather data you don’t need. Ignore temptation to add steps to your git-com config that make future you and collaborators enter / choose info that isn’t really needed. You’ll just get annoyed.
The more important it is for you to fix bugs quickly, the more important it becomes to leave yourself hints in the commit body that’ll help you track down problematic changes quickly
Creating a changelog generator is relatively easy, but you need something that understands the format that you’ve come up with for your commits.

Bonus

Using git-com’s format functionality makes it much easier to quickly skim changes because everything’s aligned.

❯ git log --oneline v1.6.0..v1.7.0
3382062 (HEAD -> main, tag: v1.7.0) changelog.org → CHANGELOG.org
8ddaa76 (origin/main, origin/HEAD) added a commented out lefthook pre-commit that'll always fail
23d3f40 [add]       stores & offers up commit messages from failed commits
2324d4e [add]       stores commit message for reuse when pre-commit hooks fail
1203fdd [refactor]  hook output now streams live
0a1ad2c [fix]       maintains yaml element ordering when adding an option
7489d58 [fix]       commits work in linked worktrees
3611a23 [refactor]  faster startup
906284d [fix]       tweaked how lefthook's go-tester works
0f01ecc [add]       added tag & release handling to homebrew tap script
5010937 [add]       added changelog.org file

Martijn Hols has a really nice blog post about How to write a good commit message. ↩

New Year New Tools

2026-01-05T00:00:00-05:00

Table of Contents

Wheeeeee
- Cleandiff
  - Why
- git-com
  - Why
- options_parser
  - Why
- oho (bonus)
  - Why?
Credit Where It’s Due

Wheeeeee

Ringing in the new year with new tools!

Cleandiff

CleanDiff is a graphical diffing tool that pays attention to word instead of lines.

Why

Two reasons:

I hate looking at diffs of changed lines and knowing something changed, but having to hunt for what that something is.
Too much noise. In addition to wanting to see the specific “word” or “character” that changed, I don’t want to have to scroll through the entire document to get to it, or the opposite problem where something like git diff shows you a tiny bit of context, but not enough.

git-com

git-com is a tool for creating structured commit messages that’s easy to configure for each project. A simple YAML document at the repo’s root drives an interactive UI with select lists and text fields to easily populate your commit message with structured, and repeatable elements.

[front-end] bug-fix: item listing works with large minimum font size

Text elements started overlapping each other when a user had a
minimum font size set in their browser.

Tags: accessibility, css

That commit message could have been created with the following elements

<code-section> <work-type>: <title>

<description>

<tags>

Why

This is yet another entry in a collection of tools to help answer questions about what’s changed in a codebase. Sometimes it’s that simple. Sometimes it’s trying to find where a bug introduced. Sometimes it’s to be able to have an accurate accounting for stakeholders.

Good commit messages have saved me countless hours of spelunking when I was in charge of a support-engineering team (putting out fires in production), and as a release manager trying to figure out what actually made it into the impending release.

In things like my dotfiles repo, they’ve been useful for just knowing what part of my system a commit pertains to. In my writing it lets me know which book I was making changes in, and if it was new writing, or edits.

Codebase Example

[fix] was erroring if run from a subdirectory
[fix] icon in readme on github
[fix] removed unecessary blank line from .git-com.yaml
[fix] removed flag from script to update homebrew tap
[add] added quick-start to docs
[clean-up] simplified .git-com.yaml

Writing Example:

[ji-woo] renamed 2 of the container kids
[thimble] finished telling taylor's dad about thimble
[thimble] few days worth of editing and adding
[worldbook] added info re cats barely growing
[worldbook] added / updated content warning stuff
[nicole] finished off day after meeting ji-woo

options_parser

options_parser is a new options parser for Ruby

It supports

auto-generated help docs
required & optional arguments
short and/or long variations
arguments that take values, or just act as flags
separating flag from value with space or equals
auto-conversion to integer or float as defined by the value_type
trailing arguments Ex. `command -n 4 – file_1.txt file_2.txt`

A simple example

options = {}
parser = OptionsParser::Parser.new(
            command: "myapp",
            description: "Does something useful") do |p|

  p.on(short: "-f",
       long: "--file",
       value_type: :string,
       help: "Input file path") do |value|
    options[:file] = value
  end
end
parser.parse(ARGV)

Oddly, this isn’t the first options parsing library I’ve written. I wrote one for Crystal that allowed you to create “sentence-like human interfaces to your apps”. It had a similarly creative name: sentence_options

Alas, everything pales in comparison to what Raku comes with in its standard library. 😿

Why

Because - IMNSHO - Ruby’s optparse is a clusterfuck of confusing code, bad docs, and limited functionality. And I was writing a script that took two arguments and processed commit messages created with git-com and converted them into a rough changelog for the next release of git-com. 😜

oho (bonus)

Not technically “new” but somewhere along the way oho (the world’s greatest tool for converting escape codes to HTML) developed bit rot. Two key regular expressions that definitely used to work, started either matching too much, or too little and completely hosing the output as a result.

I’ve updated it, and found - and fixed - a couple edge cases it wasn’t handling well.

Why?

oho was created because - as a release manager - I found myself maintaining a growing arsenal of CLI tools that could spelunk through our git logs and generate useful information for stakeholders.

Just copy-pasting from the terminal - and loosing all the colors - lost a lot of important context and made it hard for stakeholders to focus on what they needed to. I needed a way to capture the colors and formatted output my tools were producing.

Oho lets me go from colorful terminal output to HTML which can then be converted to PDF. People are weird about being given raw HTML files, but PDF is something they’re comfortable with.

Credit Where It’s Due

Cleandiff wouldn’t have been possible without my amazing wife bringing Fenestro back to life by porting it to Go, and without her creation of the tokendiff diffing algorithm (geeky details here) the results wouldn’t be nearly as nice.

Fenestro is a fancy web-view you can throw lots of HTML files at really quickly, and then switch between them by clicking on its sidebar. It used to be in Swift, but then it stopped compiling as the language moved forwards. Now it’s written in Go and leverages the Wails GUI framework.

To make Cleandiff I simply took her work with Fenestro, and Tokendiff, and combined it with everything I’d learned making, and refining the proof of concept for Cleandiff, and slapped it all together into one app.

The Joy Of No Batteries

2024-10-16T00:00:00-04:00

Table of Contents

Overview

A musing about my enjoyment of Scheme and having to work a little harder.

Unfamiliar with Scheme?

Side note for those who aren’t familiar with Scheme:

Scheme is an umbrella term for many languages. Most of them try and follow one of the official standards like R5RS, but some go off on their own and explore completely different paths like Racket. Schemes are considered dialects of Lisp, but Lisp too is a term for a collection of languages. I’ll use the terms somewhat interchangeably below.

Please also see the bit at the end before complaining about the parentheses to me.

Old Cycles & New Learnings

I’ve been coding professionally for just shy of 30 years now. All of the professional work has been web development.

At its core web development is taking form submissions, sticking them in a database, taking the stuff in the database and showing it on a web page to read, or a web page with a form to edit. That’s it. I don’t care if you’re using Rails, or Django, or Phoenix, or some homebrew creation. The back end is essentially the same. It’s not until you start approaching Google scale that what you do, or how the infrastructure supports that begins to change.

Most of us don’t deal with requests at that scale. Most of us are coding minor variations on the same thing again and again. I suspect that it’s the same for back-end developers in any vertical, be it mobile games, or radar systems. Even if you switch languages it’s the same stuff with slightly different syntax most of the time.

I suspect that’s why so many of us work on hobby projects, or open source libraries even though they’ll eat up our free time, and are unlikely to ever make us money. Those of us who love programming, love the problem solving, and figuring out how to express an idea with in a different set of constraints.

Batteries Not Included

A programming language that has “Batteries Included” is one that has a large / versitile “Standard Library”. Languages like Ruby could be said to come with not only “batteries”, but a spare generator, & solar panels. It’s got a web server, JSON parser, HTTP support, and lots more.

Scheme’s aren’t like that. Some Scheme’s - like Racket and Chicken Scheme - have a thriving ecosystem of third party libraries, but with the exception of the SRFIs^[1] those libraries are frequently written in ways that make it as difficult, or awkward to use them as to write your own implementation.

In the end, you end up needing to write a lot of your own “batteries”, and that means thinking about “simple” things that you’ve been using for years, but probably never thought about before.

Forced Thought

This is where Scheme comes in for me. Scheme is considered a multi-paridimatic language. It’s not explicitly functional, and there are various Object Orientation libraries that have been written for them. It’ll let you write code in a very OOP-like fashion, but everything seems to work a bit more smoothly if you embrace a more Lispy approach.

For most of us that means a different approach. For example, by default Scheme doesn’t have an method to break a loop or return early from a function, at least not like most languages^[2]. This isn’t a bug, or an oversight. You just don’t need them when you start thinking in a Lispy Way.

Lisp, and by extension Scheme, has a historical mystique to it. It’s a fundamental part of the history of machine learning and programming language creation.

I learned it because - at the time - I hadn’t done any functional programming and I wanted try something that would give me new challenges & force me to learn new ways to solve problems. Common Lisp^[3], & Scheme gave me that. Those learnings helped give me new solutions to problems in my day job too.

But I’ve been finding real joy - and frustration - in having to implement little things that we take for granted in other languages. For example, yesterday I needed to extract the first, or last N elements of an array. In Ruby I could do something simple like this.

long_array  = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
array_start = long_array[0,3]  # => [1, 2, 3]
array_end   = long_array[-3..] # => [8, 9, 10]

Scheme has a lot of list manipulation functions, but it’s also missing a lot of things we take for granted, so when I didn’t find a function to do this (see below), I figured I had to write my own. This is frustrating when you’re trying to get something done quickly, but it can also be joyful in that it makes you stop and think about simple things. “🤔 how do I efficiently select a subset of a list?”

My solution uses tail recursion, and Scheme’s ability to define a function within a function which lets me expose a wrapper function (first-n) to the world that doesn’t force the users to deal with passing in an accumulator, and doesn’t add a second top-level method to this module that future me will have to think about separately. It also doesn’t use any loop / iteration functionality. It’s radically different than how I’d solve the same problem in Ruby, but I think it’s also a pretty Lispy solution, that runs quickly.

  ; returns the first n elements of the list
  (define (first-n number a-list)
    (define (inner-first-n number inner-list accumulator)
      (if (or  (eq? (length accumulator) number)
               (null? inner-list))
          accumulator
          (inner-first-n number
                         (cdr inner-list)
                         (append accumulator (list (car inner-list))))))

    (if (> (length a-list) number)
        (inner-first-n number a-list '())
        a-list))

Scheme’s got a lot of oddly named functions, so occasionally you’ll not find a built-in one just because you don’t know what keyword to search for. In this case, I didn’t know the built-in methods for this behavior were called take and take-right, and I’d never heard anyone refer to that behavior as “taking”.

Even when I found those, I had to modify things, because those methods don’t behave well when you pass in a list that’s shorter than the number you’re trying to take, and Scheme doesn’t have built-in try/catch exception handling like most languages^[4]. There are libraries to do it of course, but the point is that even when there is a “battery” - built in or 3rd party - you frequently need to futz with it.

  ; returns the first n elements of the list
  ; or the entire list if its length is < number
  (define (first-n number a-list)
    (if (<= (length a-list) number)
        a-list
        (take a-list number)))

Thinking about this as I write it, this sounds terrible. Admittedly, sometimes it is damn annoying having to write so many wee low-level functions.

I’ve got a module that just a bunch of list manipulation functions to let me easily implement common tasks that aren’t built in, like sorting a list of strings. Every one of those represents a little bit of learning, a problem that I’d never have to solve in my day job.

Each one of those is also written in a way that matches my brain. That too is one of the arguments for, and against, Lisps. Codebases, it is argued, tend to end up looking like DSLs where everything is a function tailored to the specific purpose of the app. You can end up with a codebase that makes weirdly specific things incredibly easy, but also requires a longer spin-up time for new team members because so much of it is written using custom functions instead of generic “batteries” from the language.

Parting thoughts

When starting a new project, ask yourself if you want to get something written quickly, or if you want to learn new things. If you want to learn new things, maybe checkout a language like Scheme.

If you’re interested in learning a Scheme I’d highly recommend you check out Racket. It’s got a ton of 3rd party libraries so you don’t have to write everything from scratch, and the documentation of them tends to be very high quality. Chicken Scheme’s pretty good too, with a good ecosystem of 3rd party libraries^[5] but I find the documentation of most libraries to be simultaneously comprehensive, maddening, and nearly useless. For example, every function listed with a sentence or two describing what it does, but each description is based on a pile of domain knowledge that people new to the library, or new to the language don’t have.

Post Script About The Parentheses

Whinging about Lisp’s parenthesis is a lot like being fatphobic. It’s claiming something is “bad” or “lesser” because you don’t like how it looks without any real understanding of why that particular person / thing is the way it is.

The only people who whinge about Lisp’s parentheses are people who don’t code Lisp. Not only that, but Scheme is so damn flexible that there are multiple implementations that don’t use parentheses. Wisp (Whitespace to Lisp: SRFI-119), for example, uses indentation levels like Python. The Readable Lisp S-expressions Project uses parentheses, indentation, and curlies and resembles C.

So it’s not just “yucking someone’s yum” because you don’t like how it looks, it’s also really ignorant because you don’t even need to use them.

To me, Lisp’s parentheses are little hugs that keep my codey thoughts safe and secure. They help Lisp to be a homoiconic language, and every decent editor makes them easy to work with, and helpful. So stop trying to get rid of my hugs!

SRFI: Scheme Request For Implementation. These represent a couple hundred implementations of different useful things (like hash tables). They tend to be written in a generic way that can either be used directly in any Scheme that adheres to the de facto standard called the Revised Report on the Algorithmic Language Scheme (RnRS). E.g. R5RS. Or can be easily ported. ↩
Most schemes do have a call-with-current-continuation function (abbreviated as call/cc) that allows you to exit a function early. However, there’s a huge difference between just returning the result of your calculations directly (ex.: return "foo") and calling a function that takes a function that then gets executed in order to get your result out, but also behaves differently based on how and when it’s called. ↩
I started with Common Lisp, but parts of it frustrated me, and at the time the community seemed essentially dead, and the library options (number of 3rd party batteries) was almost non-existent. It’s gotten much better since the introduction of the Quicklisp library manager. ↩
Yes, of course there are libraries that implement modern try / catch style exception handling. They’re just not “batteries” that are included by default. ↩
Unfortunately Chicken Scheme uses a centralized Subversion repository for sharing “eggs” (libraries). The process around this is so fucking obnoxious, and requires so much obscure hoop jumping that I’m not bothering to update the one library that I did submit. ↩

Debugging Our Ambulance

2023-12-08T00:00:00-05:00

Table of Contents

Debugging Our Ambulance

tl; dr: two developers use standard debugging techniques to fix electrical problems in their ambulance.

Setting The Stage

We had to drop our car off at the dealership last night. In order to avoid walking home in the dark, on roads without sidewalks, in 27°F (-2.7°C) - or just the last two shortly after dawn, we needed to drive the 🚑 Ambulance. Unlike most years, I actually remembered that it had a block heater^[1] and had turned it on at the beginning of winter.

Because it was so cold, I went out in the morning and double checked that that the block-heater was turned on. It was. What I didn’t check was the voltmeter^[2].

The Problem

That evening - in the cold, and dark of winter - I hopped into the 🚑, turned the key, and… nothing. Literally nothing happened. I looked over to the voltmeter and understood why. Zero volts coming from the batteries. Nothing to power the ignition.

A debugging we will go

Gather the initial known truths

The ambulance has 2 separate electrical systems. There’s the 12v DC system (normal car stuff) and the 120v AC system (normal “house” electrical stuff).
An Inverter takes DC power from the batteries and converts it into AC power. This is not very efficient and you loose a fair amount of power along the way even with an expensive inverter like the ones they use in ambulances.
A Converter takes AC power and converts it to DC power.
There’s no reason you can’t have an Inverter and Converter in the same physical device.
The voltmeter reads 0 volts.
The system is acting as if there is no power.

We don’t know where the AC in from the side of the ambulance goes. I suspect it’s in our wiring diagrams^[3] but we haven’t hunted that bit down yet.

Propose Theories

the batteries are “dead”. They aren’t set up as some assume. See footnote.^[4]
the batteries are disconnected.
the ambulance isn’t getting power from the house.
- The house plug is bad.
- The cord is bad.
- The ambulance plug is bad.

Choose A Theory And Test It

Eliminate things you can test rapidly.

I started with the second theory - despite it being unlikely - for three reasons: First it could be eliminated in under a minute. Second, because I’d recently replaced the batteries and they live in a slide-out drawer - 🚑s are weird - and its possible one of the leads came loose. Third, because I was already outside and didn’t need to go inside to grab anything to test this. A battery lead being detached is very unlikely to be our problem, because even if it did, each lead is clamped into two batteries. So if it comes off of one post it’s going to still be clamped onto a second one. Two batteries gives us redundancy, reliability, and more available power in an emergency when the engine has stopped running, and thus the alternators aren’t generating any DC power.

The leads were very well attached to the batteries. I couldn’t guarantee that the far end of the leads were well connected because I don’t know where they terminate, but it’s very unlikely that anything has gone wrong there. They’re very thick wires and would need significant clamps to hold them in place.

Work on probable things.

The batteries being dead is the most likely. It’s so likely that we just worked with that as a default assumption, and moved on to the third theory.

The house plug had a self-test light, and GFCI test & reset switches that all seemed to be working. We also had a plug-in AC outlet tester which was easy to grab confirmed. So, it was reasonable to temporarily eliminate it as the problem.

It’s difficult to test if the plug on the ambulance is working because we don’t actually know what it connect to inside the ambulance. It’s probably in our wiring diagrams, but even if we did know, it’s unlikely that we could reach it without disassembling parts of the ambulance.

Next was the cord. Testing that required a multimeter. We couldn’t find our “good one”, but we had an ancient Radio Shack one that still had power. A quick trip to YouTube to make sure we were using it correctly for AC wall power, and we confirmed that the cord was transmitting power. We tested this inside in the basement which was both well-lit, and significantly warmer.

Because we couldn’t easily test that the 🚑 was receiving power via its plug, we had to just assume - for now - that that was still good.

Since we had the multimeter in hand, it made sense to eliminate some things that had just become quickly testable: the batteries themselves. They were - unsurprisingly - dead.

We tested the AC plug on the Inverter (for sending power out). It worked. Since the batteries were completely dead, and the outlet had full power, we could reasonably assume that the power was coming from the cord to the house, which meant that power was successfully entering the ambulance and being used by things in it.

Generate new theories
- State of Existing / Old Theories
  1. ✅ the batteries are “dead”.
  2. ⛔ the batteries are disconnected.
  3. ⛔ the ambulance isn’t getting power from the house.
    - ⛔ The house plug is bad.
    - ⛔ The cord is bad.
    - ⛔ The ambulance plug is bad.
- Evaluating What We Learned
  
  We have to assume that the power system from the socket on the wall of the 🚑 to whatever it goes to is fine. It’s been sitting still for months, and is unlikely to have had a problem. We’re also assuming that the other end of the battery leads are still connected. Both of these are too hard to test, and too unlikely to bother trying.
  
  The batteries being dead isn’t surprising. It explains why we couldn’t start. It also raises two questions
  - Why are the batteries dead?
  - What can we do to give them power?
  Normally, plugging it into the wall is all we need to do to give the batteries a trickle-charge that’s significant enough to overcome the normal phantom draw.
- Another Quick Test
  
  We have a lithium jump start battery pack for the car. It was our belief that it didn’t provide enough power to turn over the engine, but it was something we could test quickly. It’d be great to be wrong about it.
  
  When I connected it to the batteries, the ambulance lit up like a Christmas Tree. Thus confirming that the batteries were well connected to the system. Turning the key “worked”, but as suspected there wasn’t nearly enough power to “turn over” the engine. I left it connected so that it could help the batteries out when we figured out how to get power into them. It drained very quickly. These emergency lithium jump-start batteries work by providing a lot of power throughput, but they don’t actually have a lot of power stored. It’s just a short powerful burst until the engine starts and can take over.
- New theories needed
  
  New theory: Power is making it into the ambulance, but not making it into the batteries.
  
  Batteries don’t have separate “input” and “output” things. DC power is like water. It just flows through the system, self-levels, and leaks out any “holes” (things using power) you may have. Since the batteries were well connected (still assuming far end is good) it meant that the problem was likely to be something in the electrical cabinet.
  
  I believe the batteries also go directly to the starter mechanism, but we’re assuming that that’s good, and investigating questions around getting power into them, not getting power to the starter. We need power to exist before we can bother giving it away.
  
  The problem with this theory is that it is too generic. It helps narrow down the problem, but it doesn’t give us anything specific to test, and the system has thousands of wires, a wall of relays, and many other things.
  
  We went back into the warm, and well lit house to think.
  
  Dachary proposed that we might have blown a fuse. The problem with this is that there are surprisingly few fuses in this system. Because we believed (reasonably) that all the stuff to go from “shore power” to the batteries was in the electrical cabinet, the problem with that theory was that neither of us could recall any fuses in there. One wall of the cabinet has a panel that opens behind the driver’s seat, and we think there are some fuses there. Unfortunately the seats don’t fold forward and it’s a pain to get to even when you can see well. Also, we didn’t believe those fuses controlled anything relevant.
  
  New Theory: blown fuse.
  
  Currently not worth testing, because we didn’t have any likely or easily accessible location to look.
  
  So, we got out the wiring diagrams^[3].
  
  At this point we’re thinking that the Inverter and Converter are in the same box. However, I’ve been bothered by the fact that it only says “Inverter” on it. If there is a separate converter I don’t know where it is.
  
  Dachary digs into the wiring diagram while I test something I can’t remember - I’ve probably just messed up the order of something here. She finds the Inverter, but it doesn’t mention a converter. The rest of the diagram wasn’t very helpful, but there was a reference to another diagram number though. So, I start hunting for that one. There I find two things: First, a separate mention of the converter, implying that it’s a physically separate device. Second, an indication that it’s wired in via a standard AC plug. That would have to be one of the plugs in the electrical cabinet.
  
  I head back out, and start re-examining the cabinet. As I’m looking around at the outlets and plugs and wondering where the converter could be I notice a large metal box that I’ve mostly been ignoring. It says “converter” on it. 🤦‍♀️
  
  I start examining the converter and notice that it has 3 wires going into it. AC is 3 wires (positive, negative, and ground) and DC is 2. Because a Converter is AC to DC this must be where the power comes in. The connector they used has places you can stick multimeter leads and test if it’s getting power.
  
  New theory: Converter isn’t getting power.
  
  I also notice a single automotive blade-style fuse hidden behind where those wires come in. It’s impossible to tell if it’s blown, and really difficult to access.
- Closing In
  
  I grab pliers to try and extract the fuse, and a multimeter to test if it’s getting power.
  
  I dive back into the cold and dark. It’s getting power! I attack the fuse. It’s blown.
  
  It’s a 30 amp fuse too. That’s a lot. If it hadn’t blown then “Bad Things™” would have likely resulted. E.g. 🚑 + 🔥… and then maybe 🏠 + 🔥.
  
  I stand up and reach into the cabinet with spare fuses^[5], grab a new 30v fuse, slide it in and the whole ambulance lights up again! I check the voltmeter and it’s back at the standard charging & running level of 13v, which means the batteries are being fed power now!
  
  After a couple hours of changing, it started right up.

Summary

Practicing a considered, and well structured approach to debugging is far more efficient than flailing wildly at random ideas that flit through your brain. It’s also very useful outside of software.

At each step of the way I literally stopped and restated what we knew the known truths to be, and ask what new information we could derive from the theories we’d eliminated or confirmed.

While we didn’t do a formal “post-mortem” to the “production incident”, we did create “action items” from it.

Buy a new multimeter. It doesn’t matter that we have a good one if we can’t find it.
Increase the priority of the existing task to measure, model, and 3D print a custom shroud to improve the water resistance of the 🚑’s outside plug. Rainwater in there seems the most likely cause of a problem.

Good debugging and Incident Management procedures help in your everyday life too.

For a good guide to debugging start with “The Pocket Guide To Debuging”, by Julia Evans.

For a good guide to creating your own incident management processes, start with Graceful Disasters, by me.

Afterward

(based on internet commentary)

The electrical systems in modern ambulances are nothing like those in cars, diesel trucks, or RVs. Unless you’ve helped manufacture these things, or have spelunked in the wiring of one first hand, your assumptions about how things work are probably wrong. Additionally, each one is a literal one-off-custom build.

I assure you that it’s very easy for a brand new starter battery to go dead in 6 weeks of cold weather if an ambulance isn’t plugged in. This makes sense if you understand the regulations these must meet to give you the best chance of survival in some extreme environmental conditions. It makes sense if you understand how they’re used by the EMTs and Paramedics.

Trust me when I tell you, that once we get beyond “Electricity 101” and Ohm’s Law, you should throw your assumptions about how this is wired or constructed out the window. We aren’t guessing about how it’s wired. We have ridiculously detailed wiring diagrams for this specific vehicle. Every wire’s insulation says what that specific wire is for. We know how things are connected. We just don’t know where everything is connected. To put it another way. We know that two wires meet at a junction, we just don’t know which part of the wall each junction is hidden behind.

our ambulance dashboard

A block heater is basically an electric blanket for your engine. This is especially notable for large diesel engines for three reasons: ↩
1. diesel stops being liquid and turns to a gel when it gets cold enough. They actually change the formula in the pumps just before winter temperatures kick in. A diesel engine uses “glow plugs” to heat the fuel. I don’t know what they look like, but they’re essentially just little electric heaters in the fuel system. They take a lot of power, and a many minutes to make the diesel ignitable on cold days.
2. our engine is so large that even if you do manage to get it started in the cold it takes a long time to heat up all the metal.
3. diesel engines don’t have spark plugs. They combust the fuel by compressing it, which is harder to do when it’s a gel.
Ambulances have dashboards with a lot of switches (see below). There’s also a voltmeter (the radial-dial). Unsurprisingly an 🚑 has a lot of electrical stuff going on. Unlike most vehicles there’s a non-trivial phantom draw. It will drain its two AGM batteries if it just sits around for a while. This is acceptable, because it was designed to be used literally every day. It’s also got a 120v AC input on the side (with a weird plug). We don’t drive ours much, but we do keep it plugged in.

The voltmeter is important because if yours is low, the EMTs and Paramedics in the back are going to have trouble using things like IV bag warmers and having lights to see what they’re doing. You want to know early if there’s a problem. ↩
Unlike most vehicles every 🚑 has complicated custom wiring. Every ambulance is custom built. What features you get depends mostly on how much your town is willing to spend^[6]. Manufacturers need to keep track of what they put into each one so that they know how to fix it if it comes back for repairs. When we got ours, we called up the manufacturer, gave them our Vehicle Identification Number, and asked them to send us a PDF of the wiring diagram for our vehicle. ↩ ↩2
The ambulance has 2 AGM batteries wired in parallel. There is not a separate “house” battery and “starter” battery. Within the US there are strict regulations about being able to maintain temperature in the patient area of an ambulance. Basically you need to be able to keep it cool in the middle of Death Valley in the summer, and you need to be able to keep it warm in the northernmost part of Alaska in the winter. You’d need a massive battery to be able to power the heating and cooling systems to achieve this for hours on end. Ambulance manufacturer’s don’t bother. Instead they just guarantee that that will work while the engine is running. In my case they accomplished it by using two beefy alternators. ↩
Of course we have spare fuses on hand. A) it’s an ambulance with a billion cabinets. B) It would really really suck to be stuck on the side of the road just because you didn’t have a fuse. Especially in a long distance (thousands of miles) trip like the ones we take. B) of course the fuses are right next to the electrical cabinet. Why would you keep them anywhere else? ↩
A new Type III ambulance - the big boxy ones with a van shaped front - starts around $150,000. Most towns sell them after they hit 100,000 miles. That’s how we got ours significantly cheaper. ↩

Mirroring With Gitea

2022-09-26T00:00:00+00:00

Table of Contents

Overview

Following on the heels of my last post on why you should (not) self host your git repos, I went ahead and used Gitea to set up a local mirror of all my repositories, and all the repositories I don’t want to loose access to.

The results were surprising, and after reading this, you might want to do the same. This post will be a qick overview of how I did it, some tips that’ll help, and what I learned as a result.

Installation was pretty straightforward. I grabbed my old Raspberry Pi 3B and followed this spectacualar step-by-step guide for installation for installing Gitea. Afterwards a friend pointed out that I could have just installed installed it on our Synology with a Docker image.

The Goal

My goal was two-fold:

I wanted a backup of all my repos. Many of them only existed in GitHub.
I wanted to delete every fork I’d made just to guarantee I had a copy.

The Process

With that in mind I started working through my list of repositories^[1]. Gitea makes it trivial to mirror a repo. You just choose “new migration” from under the “+” menu, and then click on the service you’re migrating from. There are 3 things that can make your life easier here if you’re importing lots^[2].

Bookmark the form that comes up after choosing a service to migrate from.
If you’re going to be importing a lot you’ll probably want to use a personal access token (GitHub docs). That way you won’t get rate limited by the API, unless you’re migrating (not mirroring) something with many thousands of issues.
Use a clipboard manager. It doesn’t remember your API key so you’ll be pasting it into every form, along with the URL of the repo you’re importing. I use Maccy, and I would recommend it.

For each repo in my list of repos on github, I’d make a couple quick decisions.

for things I authored
- If it was something I am either maintaining or may poke in the future I’d just grab the url and tell Gitea to mirror it.
- If it was something I wandered off from years ago, and am unlikely to ever touch again, I’d mark it as archived.
for the backups of other people’s things
- I’d click through and see if i’d added anything, or if it was just an old copy.
  - If I’d added something, and the original repo hadn’t been touched, I’d mirror my fork.
  - If I’d added something, but the original had moved forwards some significant amount I just abandoned my changes, and mirrored theirs.
  - If mine was just an ancient copy of a repo that’d moved on, I’d go to the parent repo.
    - If the parent repo was archived I’d just tell Gitea to import it but not bother mirroring. There aren’t going to be updates.
    - If the parent repo indicated that it too was a mirror, I’d follow that back to the source repo and mirror that instead.
    - After successfully mirroring I’d delete my copy.

The Learnings

(In no particular order)

First off, Gitea is really impressive. Those devs have done a great job and should be proud of what they’ve built.

Along the way, I discovered that many of the repos that I’d forked had moved on significantly. I had copies that were so old they probably wouldn’t run with modern versions of things. They were also frequently missing lots of new features and bug fixes.

While this isn’t terribly surprising if you think about it, the notable learning is that that’s never going to be a problem again, because Gitea is mirroring them and regularly polling them for updates. If they disappear I’ll have the latest copy before they went away. Github doesn’t give me that option.

I’m not the only one forking repos just to have a copy. Multiple times I clicked through to the parent repo, and got curious. I’d see hundreds of forks, click into the insights page and see that essentially no-one was hacking on it except for the original authors. I guess lots of folks have gotten burned when they discovered that some repo they’d starred or bookmarked was gone when they went to check it out again.

Mirroring isn’t something you can turn on later in Gitea. You must set a repo up as a mirror from the beginning.

Gitea can’t mirror everything. Some repos (like this great command line time zone tool) always error out. Gitea provides no information as to why it failed to clone them down too. I think I encountered three repos like that. Also, I can’t figure out how to clone anything from Sourcehut^[3]. Even when I tried to clone repos manually their http links for cloning always error.

I realized that if i don’t just mirror the tools that I author, and want backups of, but also mirror the repos of tools that I need to reference the source code of regularly then I have a single place I can search for only stuff that’s relevant to me. For example, it’s not uncommon for me to know that one of the libraries I use frequently has a piece of functionality, but not remember which library. Searching Google or Github would be frustrating, because there are just too many responses. But now, I can just search my Gitea install.

The experience of doing this was very cathartic for me. I now know that I have up-to-date copies of all the useful tools I care about. I was also able to delete about 100 repositories from my list of repositories on Github. Now, if a future employer checks them out, they’re almost all things I have actively worked on to some degree. It’s also nice to know that from now on I have a trivially easy way to mirror any useful tool I care about. No more ancient frozen it time forks, and no more bookmarks & stars that cease to exist.

My next step? Going through all the repos I’ve bookmarked, but not forked, and see which of those are worth mirroring.

Before working through your list of repositories, be sure to sort them by name. Mine were sorted by recency and that made things a little harder towards the end. ↩
For a sense of scale I ended up mirroring or migrating 169 repos into Gitea. ↩
In addition to having a visual style from the 1990’s, and having UX from hell, Sourcehut appears to also just not fucking work as a git host. I don’t understand why anyone uses them, nevermind why anyone pays for them. ↩

Do (not) Self-Host your repos

2022-09-24T00:00:00-05:00

Once upon a time, GitHub was a successful geek enterprise. Then Microsoft bought it, and folks started arguing that you should abandon ship. You should self-host your repos they say.

I 100% agree, and 100% disagree. Let me explain.

GitHub’s been a benevolent host. When they bought NPM they went from being a de facto piece of internet infrastructure to an actual piece of critical infrastructure. At this point you may as well argue that we should sever our connection to the electric company.

But there’s a difference between needing the electric company and ignoring the fact that sometimes the power goes. Usually it’s temporary. Days isn’t uncommon in wintery climes. But every now and then, it’s more. In the case of internet SAAS apps, it’s sometimes permanent.

Why You Should Self-Host

The biggest argument, of course, is backups and continuity of service. If something hapened and you lost access to your GitHub account right now, what would you loose? If you maintain a project with many users and contributors, what would it do to your community, and your work?

GitHub isn’t just where your Git repos are hosted, but where all the metadata that surrounds them. This means years of Issues, Pull Requests, wiki pages, and frequently the project’s web site.

I’ve derived a ton of value from old tickets on GitHub projects. They regularly answer questions, and provid workarounds. The takedown off a popular repo could be a huge loss.

GitHub suspends accounts, and deletes repositories for a variety of reasons. If yours targeted for seemingly “no reason” it wouldn’t be the first.

Even if you believe it won’t, or can’t, happen to you, there’s another reason reason to consider self-hosting: Lock in. Right now (September 2022) GitHub is very good about letting you export everything. As far as I know, there’s no data you can put into GitHub that you can’t also get out via their APIs. In theory.

Our migration has been in the process for a while, but every time we run it we find that our attempts were missing some data, so we need to re-run to fetch missing data, if we had higher limits it wouldn’t take so long to find these issues. Github doesn’t provide tokens with higher limits, so we have to work around that. We even had people with [GitHub Enterprise] subscriptions ask, and they got the same response.

Gitea on Mastodon

Gitea is an open source GitHub alternative, whose creation started out on GitHub. Eventually, the software got really robust, and they decided to host their own repo and issues. The problem is, they’ve got over 20,000 Issues & PRs they want to export. Thousands of those are still open, and the longer it takes them to cut over, the larger those numbers get.

To be fair, 99% of GitHub users wouldn’t have this problem. We don’t have projects that are popular enough, but companies who use GitHub do, and there’s no advantage to GitHub to help large projects or customers leave. They may even consider locking that metadata down to guarantee they stay.

What about GitLab and other Competitors?

You can apply the same reasoning to GitHub’s hosted alternatives. Plus, none of them are as financially successful as GitHub. They’re far more likely to go out of business, and screw everyone who uses them.

Why You Shouldn’t Self-Host

Right now it’s incredibly improbable that you’ll have your account suspended. The stories of suspended accounts are rare. Most of them seem to be from government action,or “Terms of Service” violations that most geeks react to with “Yup. Seems fair.”.

There’s also the fact that self-hosting is work. But let’s set that aside for the moment. Let’s pretend there’s some magical free button you can click that takes care of everything. Boop. You’re self-hosted now.

Let’s also assume you’ve got a project that’s popular / useful enough to get Issues and the occasional Pull Request.

How is that going to happen on your new site? Well, first they’re going to need to set up an account on your server. Are they going to think it’s worth it? Worth it enough to give their email to yet another stranger? If they want to submit a fix / feature, is it also worth it to figure out how to fork it in this not-GitHub? Worth it to figure out how its PR equivalents work? These probably aren’t hard, but every little difference they have to navigate is another reason to not bother.

And then there’s discovery. GitHub isn’t only the default place to look for code. They’ve also got massive “website authority” in Google. It doesn’t matter if you’re the only place hosting your thing. Any project on GitHub that’s remotely similar to your search terms is likely to come up before your site.

But, even if you’re ok with all of that. There’s an even bigger reason you shouldn’t self host. A reason we shouldn’t encourage folks to self host. Sooner or later you’ll get sick of bothering, or you’ll die. One is guaranteed. The other has lots of historical precedence.

I’ve bookmarked thousands of useful web sites over the years. A huge chunk of those have been useful, and insightful blog posts about development. Hundreds of them don’t exist anymore. Mostly because the person who ran them simply didn’t feel like bothering with it anymore. How many millions of others have poofed out of existence?

But, a repo isn’t some just random thoughts. Almost every repo contains some useful tool or library. Sooner or later, someone’s going to need tha. There are thousands, probably millions, of little repos out there containing the only bit of code in existence, that solves that particular problem. Sometimes they’re just the only thing that solves it in that particular language. When you stop hosting it, for whatever reason, that useful tool ceases to exist. Even if it’s “just a port”. That’s a lot of work that someone will need to recreate, and debug.

So what’s a geek to do?

Before I answer that, let me note that this bit gets into implementation strategies. If you just came for the question of “should I, or shouldn’t I?” you can stop reading. Thanks for making it this far. Good luck whatever you choose.

–

So, what to do…

Short short version: If lots of people rely on your project, then mirror your code to a large competitor and accept that you may loose access to your issues, PRs, etc. . Stick a note in your README so that folks know where to find your mirror before lightning strikes.

If you’re like most open source developers, you’ve got some repos that are mostly ignored by the world, and receive essentially no issues or PRs. No-one’s going to be stressed if your repo disappears for a week. So, Install Gitea on your computer. Tell it to mirror your repos and forget about it. Or, just clone the all locally and set up a cron job to regularly pull down changes.

Longer version: I think that depends a lot on what, if any, metadata exists in your GitHub repos. Most devs have small repos that are mostly just used by them. If this sounds like you, then simply keeping an up-to-date copy of them on your home computer could be fine.

Things are different if you maintain some repo that other people actually care about enough to make issues, or even contribute code to.

In that case my personal advice is to host your stuff on Github, but have it mirrored to a second system. Ideally you’d want all your issues, PRs, etc. mirrored to it. But as far as I know, that’s not actually something you can do. It’s easy to keep the code synced to a mirror. It’s the metadata that’s a problem. To be clear, I’m not saying it’s not technically possible. I’m saying, I don’t think anyone’s actually written the code to keep those synced between GitHub and any open source alternative. It’s a hard problem because Issues, PRs, comments, and just about everything else can be edited.

So, that leaves you with regularly backing up your stuff. I’m going to assume you’re not someone with ~10k open & closed issues, plus another ~10k PRs like Gitea. So, exporting your issues is probably something that’s actually doable. Exporting. Not syncing. However, the more things you have to export, and the more frequently you want to back it up, the more likely you are to hit your rate limits.

If you go this route, you’ve got two options. You could use something like the gitea-github-migrator that can export everything. Tools like that are made to create a new repo in GitHub alternatives, not sync to them. You’d be constantly churning, deleting old repos and setting up new ones.

A better solution might be something like github-issues-mirror which can download all your issues as JSON files. Keep that running in the background and if worst comes to worst you can write an importer for wherever you’ve been mirroring your code.

What am I going to do?

I think I’m going to install Gitea on my Raspberry Pi. I’ll tell it to mirror all my repos, and all the repos of useful tools I want to always have a copy of. I do maintain some repos which are somewhat useful to other people if GitHub deletes them. No-one will be screwed, and I can take my time in uploading a backup somewhere else. I’ll stick it in the basement with the router and back it up to our NAS, which backs up to B2.

gh-url script to get GitHub url for a file

2022-08-15T00:00:00+00:00

The Problem

For the past decade or so, I’ve noticed a trend amongst my coworkers. When they need to look at the contents of a file that they’re not currently editing, they will go to GitHub, and click their way down through the folder structure until they eventually find the file they want to see the contents of.

On a related note, I believe that most of my coworkers don’t know how to take a relative path in a repo, and tell their text editors to open it.

For example, let’s assume we’re working in the same repo, and you’ve already got a file open in it in your editor. If I were to tell you to open foo/bar/baz.rb would you know how to tell your editor to open that file directly, without manually clicking to open the foo and bar folders and then clicking on the baz.rb file? For most developers the answer seems to be “no”, even though their editors can probably do it easily.

So, if I need to communicate the file I want you to look at, how do I do so in a way that makes your life as easy as possible, assuming you’re like most devs?

Easy, I give you a link to the file in GitHub. But how do i easily acquire that link?

The Solution

gh-url is a script I wrote to take a given path within a repo and generate a GitHub url for it, and optionally, open it in your browser.

So, for example i could say take that foo/bar/baz.rb path and say gh-url foo/bar/baz.rb and it would output a link to that file in GitHub. It takes care of making sure it links to the right branch, and if you have multiple remotes, it asks you which one. Just run it without any arguments to see the usage instructions.

On GitHub Achievements

2022-06-09T00:00:00+00:00

GitHub recently announced GitHub achievements. It’s a great idea, but I’m really left scratching my head by what the achievements are.

The “Pull Shark” is open pull requests that have been merged. I’ve got a “4x” version. 4x makes NO sense to me given the number of repos I’ve contributed to, but… ok. Maybe it just maxes out at 4x.

That’s it, though, except for “Arctic Code Vault Contributor” which … is more chance than anything else.

I don’t really care that I don’t have more. BUT I can’t figure out what the eff they’re expecting a person to do that they would use to “recogniz[e] the many stages of a developer’s coding journey” that I haven’t done in my GitHub account.

Doubly so because Coderwall already did this for GitHub profiles years ago. They give you “achievement badges” by assigning you to “Organizations” that represent your achievements, and then those show up as icons in your “organizations”

…and they’re cool. Really well thought out. I’m having difficulty finding a full list of the Coderwall achievements, and I’m not sure this is even something they do anymore, but here’s the ones I have. I’m not suggesting they should ALL be used as GH achievements, especially not the language specific ones like “Mongoose” (see below) BUT things like the “Charity” and the “Walrus” badges? Those would be great GH achievements.

Lemmings 100: …have open sourced at least one original repo that’s being watched by more than 100 devs.
Charity: …members that have forked and commited to someone’s open source project.
Opabinia: members that started social coding on GitHub within 6 months of its first signs of life.
Forked: …members that have open sourced a project valued enough to be forked by someone else.
Forked 20: members that have open sourced a project valued enough to be forked at least 20 times.
Komodo Dragon: members that have open sourced at least one original repo where Java is the dominant language.
Mongoose: members that have open sourced at least one original repo where Ruby is the dominant language.
Mongoose 3: (same as Mongoose > 3x)
Raven: …members that have open sourced at least one original repo where some form of shell script is the dominant language.
Velociraptor: …members that have open sourced at least one original repo where Perl is the dominant language.
Walrus: …members that use at least four different languages throughout their open source repos.

I guess my thinking is that I’ve done just about everything a person could do in GH except for having a repo used by millions BUT even on that I prolly contributed to one at some point. And yet, all i have is recognition for getting a PR merged?!?!

I know it sounds egotistical, and entitled, but the reality is that I’ve been using GitHub professionally and personally since roughly the time it began. Off the top of my head here are the things I’ve done in GitHub that are relevant to “the many stages of a developer’s journey”

I’ve got original and forked repos in piles of languages.
I’ve contributed to other people’s open source repos in the form of reporting issues, contributing to discussions, and submitting PRs.
I’ve taken over ownership of other people’s repos when they abandoned them.
I’ve done work in obscure languages, popular languages, and even Bash.
I’ve had a repo starred by hundreds
I’ve had repos forked my … many
I’ve got repos in compiled and interpreted languages
I’ve done all the normal, and most of the rare things you can do with git.
I’ve shared my dotfiles for others
I’ve contributed to languages, frameworks, and libraries
I’ve got repos with server code and repos with client code.
I’ve made private repos public, and public repos private.

What else is there in a “coding journey” that GitHub could be aware of that I haven’t done? I assume “YoLo” is some whimsical thing that maybe I haven’t done but setting aside random silly / quirky achievements, the fact that the only one I’ve got is “PR Shark” makes me wonder if they actually have any more than the 3 shown on their announcement page.

How To Register an Internationalized Domain Name

2022-05-26T00:00:00+00:00

( A guide for English speakers as of May, 2022)

I’m going to assume that this isn’t terrible if you speak a language which doesn’t look anything like English. I’m going to assume that your domain name registrar’s don’t have their heads up their butts.

Over here in the English speaking world they’re too anglocentric to notice anything that goes on in languages that have non-ascii characters. Since you’re reading this, you probably use a lot of software that was written by American companies. This means that even if you do succeed in buying an Internationalized Domain Name (IDN) there’s a very good chance that anything you try and stick it in that isn’t a browser will reject it. Especially if you try and use it for email.

If you still want to do it, the process isn’t terrible, if you know how things work.

Key Information

Not all Top Level Domains (TLDs) like .com, .org, and so on) allow IDNs.
The companies that control each TLD, sometimes require you to specify the alphabet type you’re registering (Japanese, Chinese, etc.). As far as I can tell this is completely unnecessary but they require it anyway.
Most registrars don’t allow you to specify the character set so it doesn’t matter if the TLD supports it or not because you can’t register it without specifying.
IDNs are an ugly hack, and aren’t actually unicode.

Finding A TLD That Supports Your Language.

I have no good advice for you here. All I can say is that if you’ve seen a domain in your language, then you know you can use whatever TLD it used. Searching for this is hard but the keywords you need are “IDN” and “registrar”. You can try searching for the domain on Gandi.net and hope there’s a globe next to the TLD you want in your results. If there is, you can try specifying the language you want and it’ll either accept it or tell you it’s not allowed. If there’s no globe, the TLD may or may not support IDNs but it probably won’t work if you try to buy it there even if it does. .com, for example, does support IDNs but has no globe, and gives you an error when you try and buy it even for languages that are allowed.

Getting A Punycode

Let’s address that last point first, because it’s critical to you being able to register an IDN. Let’s say you want to register バター.com You can’t actually do it (even if no-one else has). You have to convert it to “Punycode”. In this case the punycode is xn--5ck7a3i.com. You can, in theory, register that.

In practice, you still can’t. At least, not with most registrars, but I’ll get to that in a minute.

There are 2 ways to get your punycode:

use a punycode converter like https://www.punycoder.com/
just type the domain you want into Firefox. If it works, someone has it and you’ll need to think of something else. If it doesn’t it’ll give you an error message with the punycode version.

NOTE: Do not include the TLD in the punycode conversion. So, again, if we want バター.com we enter バター into the converter and it gives us xn--5ck7a3i then, take that and add the TLD like `.com to it.

Do not even bother trying to mix character sets like this バターสมศักดิ์. It’ll convert to Punycode just fine ( xn--12c2bwb2ai8bxa6ql661a3ea9s ) but you’ll hit the problem of the TLD owners wanting you to specify the character set and you can’t specify more than one.

Registering your Punycode Domain

Ok, now you’ve got the punycode. How do you register it?

In, theory, you register it just like any other domain name, but as noted above, you have to indicate the character set, or you have to find a registrar who can figure it out automatically and do that for you. In practice, there’s no way to know if it’ll work until you try and actually buy it.

Here’s my experience
( tl;dr: go to 101domain.com )

Hover.com has the problem that you can’t seach for バター.com. It deletes the バター part and thinks you’re searching for com. If you enter バター it thinks you’re searching for nothing. You can search for the punycode version though. After getting past that I got “order completed” “payment rejected” and some other error I’ve forgotten at checkout. I got different, or multiple, messages / errors on each attempt. It wasn’t the payment at all. It was Hover.

So I tried Gandi.net. There I noticed something interesting. For some TLDs it had a globe icon which gave you a form to choose a character set when you clicked it. There wasn’t one for .com, but when I tried to complete my order it give me a brief alert telling me I needed to specify the character set. It wasn’t possible with their site, but at least I had a clue what the problem was.

After some googling I found 101domain.com. I gave them a shot because they also support Internationalized TLDs. I figured if they did that, they must have a clue about the “normal” IDNs, and I was right. Even better, it handled unicode searches like バター, and when I went to buy my shiny new domain they handled specifying the character set silently.

They’re not my first choice is registrars for a number of reasons, but they don’t seem to be trying to scam anyone, checkout was relatively painless, and my new domain works. In 60 days I’ll try transferring it to Hover with my other domains.

What about Internationalized TLDs

These exist. For example .コム is a Japanese TLD. My advice, at this point in time, is to assume that these will not work anywhere except the browsers. The thing is, you enter your domain name in a lot of places that aren’t browsers. For example, every time you enter an email address, or paste a link in Slack.

You’re going to have an even harder time finding a registrar that supports those, but if you do they’ll probably handle the other problems well.

I have no idea how to convert a TLD like .コム to ASCII, but I assume it has to happen and that there’s a magic incantation there. If you know, contact me and I’ll update this.

Splitting an org-roam collection

2022-01-28T00:00:00+00:00

org-roam supports multiple directories and it should work fine if you start that way, but if you’ve already got an org-roam project that you need to split up, it’s a pain in the butt.

Here’s how I managed to divide my org-roam project into multiple directories after much trial and error and googling.

A note before continuing: If you’re googling around for this you’re going to find a bunch of old commands from when people were upgrading from v1 to v2. Many of these won’t work because their names have been changed.

Step 1 config cleanup

Remove these from your init.el (config.el in doom emacs)

 (setq org-roam-directory (file-truename "~/Documents/org-roam"))
 (org-roam-db-autosync-mode)

Setting a default org-roam-directory is just going to cause you headaches and accidentally writing to the wrong place. Leaving the autosync in there will just cause errors when you load.

Step 2 .dir-locals.el

create a .dir-locals.el file with contents like this. It should be in the directory specified by org-roam-directory

 ((nil . ((org-roam-directory . "/absolute/path/to/project")
 │        (org-roam-db-location . "/absolute/path/to/project/org-roam.db"))))
 (org-roam-db-autosync-mode)

Warning: emacs doesn’t actually read this file until you open something in the same directory or one of the subdirectories. This means that until you do that org-roam-directory is either unset, or set to whatever it was last.

So, always start an org-roam editing session by opening one of the files in the project.

Step 3

Copy over the subset of files that you want to be part of this project into the directory alongside the .dir-locals.el (dailies should be in a dailies subfolder).

Make sure you do not have the *.db file from the original project.

Step 4

Open every single one of those files. I tried every relevant command I could find that didn’t invole doing this. None worked.

Evaluate (org-roam-db-sync)

If you get errors about uniqueness constraint then run the following

(org-id-update-id-locations (org-roam-list-files))

Then re-evaluate (org-roam-db-sync)

No, I have no idea how it’s possible that the brand new sqlite database could possibly have uniqueness constraints but somehow this is a thing that happens.

Step 5

Wonder if maybe you should be using org-multi-wiki instead. I know I am…

Months Later…

I really wanted to like org-roam. The pitch is great, but I just kept having database related problems.

In the end I switched to Denote and have been very happy every since. He’s taken a unix style approach of using simple, proven, filesystem based tools, and making something really useful by combining them. Because of what it’s based on it “just works”.

Check out the author’s intro video on YouTube.

If org-roam is working for you, then stick with it. If it starts giving you headaches like it did me, then definitely check out Denote. I think it solves the same problem in a much simpler, and more reliable way.

Some of us need bigger text

2020-12-04T00:00:00+00:00

That sounds pretty obvious. It is pretty obvious. Anyone with any familiarity with older humans knows that they generally have trouble reading small text, or making out fine details. Every drug store has a rack of magnifying glasses. Everyone’s seen an older person doing the thing where they lift up their bifocals and start moving a thing closer and farther with their arm trying to find a spot where it both large enough to be readable but far away enough to be in focus. They’re not the only ones who need bigger text, of course, but they are the ones everyone should be familiar with, and likely has in their lives.

I’m bothering to tell you this, because you, my young friend, with your 20/20 vision, or something close, are building the web sites that I attempt to read, and you’ve forgotten to consider that not everyone has your beautiful and capable eyes.

This post is to give you a glimpse of what the internet looks like for us, & to teach you how to easily see it. Actually fixing the problem is usually quite easy once you can see it. This post won’t be addressing the things you need to do to help users with screen readers, but the techniques I do discuss will help many people who are functionally and legally “blind” but still make use of their screens.

What’s the problem

There are 2 parts to this problem.

The Fundamental Misunderstanding

The biggest, is the fundamental fallacy that the web designer has the final say on how a web page is displayed. They don’t, and this is a good thing. If you choose teeny tiny fonts that look beautiful to you, I can’t read them. If you choose a font with very thin lines, I’m going to still be squinting even when I increase the size. However on the web, the reader has the final say over fonts. We can say “oh, I need it bigger”, or “I like it smaller”, or “I need a font that helps with my dyslexia”. Whatever is comfortable for you is going to be unreadable by some of your visitors, and that’s ok.

The Design Problem

Almost every time I encounter this problem it is the result of a designer trying to put text in a box, but not allowing that box to grow to fit the content. The “box” can take many forms, and is frequently invisible. Sometimes designers allow the boxes to grow to fit the text, but then don’t allow them to re-flow, so they start overlapping each other, or slide off the edge of the screen instead of moving down the page.

Also, I need to specifically call out graphing libraries. Graphing libraries, in general, are terrible about how they handle the text in labels. They fail when it’s too big, or when there’s too much of it. Those same failing result in a product that’s very hostile to people with accessibility needs. In many cases it’s better to just make an image. We can zoom that. Yes, it’ll be terrible for totally blind folks, but most of the time your graph was completely unusable to them anyway.

Some examples:

I count at least 8 different errors in Amazon’s Navigation Bar right now. We’ve got forced box size, poorly handled overflow, assumptions about how tall a font is, and more.

Here’s a pretty typical example of not allowing a box to resize from the State of CSS 2020 web site. In these situations odds seem even that stuff will either be poking out the side like this, or just cut off and unreadable.

From the same site, we have an example of forcing a font size. You can see that regardless of what text size you need (big or small), the graph is imposing its own ideas on what is the “correct” size. The text above the graph, in both images, is the size the browser has been told to render text as. This means that the graph library is likely also overriding the designer’s choice of “correct” size for the page’s text.

The color key in this graph is detached from the normal “flow” of the text and didn’t shift over as the text beside it increased in size.

How to See What We See

“Zoom” is the enemy. Zoom, in its most common form, changes nothing. Your layout looks just as great “zoomed” as it does at the size you designed it at. This is because “zooming” the page is like taking a magnifying glass to it. It’s great for extremely low vision folks, but In the end it doesn’t meaningfully alter anything about it, and makes your stuff so big that much of it falls off the side of my screen.

Many of us don’t want or need your images zoomed. For example: imagine an image of someone holding their credit card out to a store clerk. I don’t need to be able to read the numbers on the card to know that that’s what they’re holding. I can extract all the relevant info from the picture without zooming “oh, it’s an example someone buying something in a store”. Similarly, I don’t want my monitor filled with your “hero image” when I came to read your text.

When a browser forces me to zoom to read the text it also forces me to work in a mode designed for incredibly low vision people (think “legally blind”). It’s great that that’s an option for them, but I just need to not squint at your text.

Here are the settings in Firefox, Chrome, and Safari. Note the “Zoom Text Only” checkbox in Firefox. Chrome doesn’t have this.* This is going to make your life, and mine, much easier. Check this in Firefox can now whack Command + (or Control + on Windows) a bunch to see what your page looks like with a fat font, and then Command + 0 to reset it. In Chrome and Safari Zoom will only magnify the whole page. Sadly Safari has no obvious way to set a default font size, only a minimum. If you want to zoom text only in Safari you have to hold down the option key, and click on the “View” menu. Then you will see “Make text bigger” instead of “Zoom in”**. I am unaware of a way to do this without the mouse.

All three have the ability to set a minimum font size. Firefox hides this under an “Advanced…” button. This is significant for designers, because it means that even if you do a great job and make all your font sizes relative to the browser’s base font size, the browser may still use a lower bound that is higher than you expected.

How to Deal

In general, the solution is to let go of the idea of pixel perfect web sites, and embrace “Responsive Design” and Flexbox layouts. It’s never going to look exactly the same on every computer and OS anyway.

The great thing about “responsive” web sites is that when I jack my font up to huge sizes, they just reset themselves as if i were on a device with a smaller screen, which in a way I am.

The other thing that will dramatically help, is to stop replacing the browser’s widgets with “prettier” CSS + JavaScript versions. Pull-downs and multi-selects are the most common example of this. These custom widgets are rarely designed with accessibility in mind, and frequently need to be tweaked to handle whatever your specific use case is. Those tweaks almost never take accessibility into account, because most developers don’t think about a11y. Firefox, Chrome, and Safari have all done a great job of making sure that their built-in widgets work really well for folks with accessibility needs.

As noted above, graphs are a problem. No-one wants to write their own graphing libraries, and few of the existing ones are good at accessibility. Try them out at varying font sizes and with lots of elements and see what happens. Unfortunately there’s not really a good answer here. Using a graph is frequently the right call, they’re notoriously problematic, and writing your own graphing library is almost always the wrong call.

As far as sticking text in boxes though. That’s not actually a problem. There are lots of good reasons to do it. Nav bars, grouping, highlighting a specific thing… These are all reasonable design decisions. If you need to use a box keep these things in mind:

Make boxes that expand in a controlled manner with the text within them.
If expanding the box isn’t an option, don’t just cut off text and make it impossible to read. Make it obvious that there is text that isn’t currently being displayed, and provide a mechanism to easily read it. For example, have scroll bars appear when contents overflow. They are there to help people read.
Avoid CSS’s overflow: hidden property when dealing with text, unless you provide scroll bars. It almost always results in cutting off the text and leaving people like me unable to read whatever you wanted us to read. text-overflow: ellipsis looks better, but the intended text is still cut off.
if clicking an item causes a sub-menu (or whatever) to be displayed, make sure that that sub-item is anchored to the boxes current location, not some hard-coded number of pixels or em spaces across the page.

Summary

There are millions of folks who have visibility issues, and you’re likely to become one as you age. So, embrace responsive design. Use native widgets, and never try to make “pixel perfect” designs.

If you’ve got more suggestions on how to improve sites for low vision folks please contact me and I’ll add it to this page.

Footnotes:
* Chrome doesn’t have the built-in ability to zoom text only, BUT there is a Zoom Text Only plugin which over 10,000 people have installed. There are probably many thousands of other users who didn’t know this was an option, or weren’t tech savvy enough to even look for something like this.
** Not it is not reasonable to expect anyone to “just know” that holding down the option key when clicking on a menu will change its contents.

First Principles & Scheme

2020-07-26T00:00:00+00:00

(Or, The Value of Working at Lower Levels of Abstraction.)

I’m loving working in Scheme because it forces me to work from First Principles.

There’s a huge value in the convenience functions that most languages wrap around those first principles, but it’s like buying and using a car vs. building the car you’re using.

The latter is more work but you’re going to really understand how that car works and you’re going to have the perfect car for your needs.

Some languages (cars) are built for speed (Rust, Go). Some are built for user happiness (Ruby). Some are built for making a specific task easy (SQL). Each has its pros and cons.

Scheme feels like it’s built to be buildable. It’s the Lego™ set of languages.

Like Lego™ if you want to quickly build some complex thing, it’s going to require a lot more time and effort unless you’ve got some pieces pre-built (by you or others). Like Lego™ there is very little limitation to what you can do with it.

The car you build won’t necessarily be the fastest possible car, but it will be the perfect car for your needs. Imagine making a pick-up truck that lets you safely chain additional truck-beds to it when you have big loads. No-one makes a car/truck like that. They make faster cars. They make better looking cars. They make cars your kids can drive. But they don’t make the perfect car for every use case.

Understanding first principles allows you to create that perfect vehicle for your needs. Most languages are pre-built cars. Some are highly specialized Formula-1 models that will send inexperienced developers hurtling into the wall at high speed. Some are comfortable Honda station wagons that’ll get you to your destination safely. Some are Porche 911’s: fun, relatively safe, go reasonably fast, but they’re terrible for taking the whole family and picking up groceries.

Most of the time, that’s perfect. You get the right kind of car for the type of stuff you’re doing. If you’re writing cutting edge video games it’s probably the Formula-1. It’ll be hard to steer, and easy to crash, but you can go really fast with it. If you’re writing a typical webapp it’ll be the Honda station wagon: safe, easy to drive, san handle a load of unexpected groceries, plenty of service stations.

From a productivity perspective it’s great to be able to hop into a “normal” car and just drive. You don’t have to worry about weather much because it’s got windshield wipers, heater, and AC built in. You just want to get to your destination relatively quickly and comfortably.

But, it’s terrible for ever learning how the car works, or how to make something more appropriate than the car they handed you.

And that brings us back to Lego™ and Scheme. Frankly, they’re both terrible for getting things done quickly if all you have is the raw pile of “Bricks”. But they’re spectacular for learning how to make things, and thus how to make a thing perfectly suited to your current needs.

Once you learn those first principles, AND you’ve chosen a scheme with good libraries,^[1] you’re left with a tool for building the perfect thing for your needs.

Scheme is a machine shop with a lathe and a cutter. Scheme + libraries is a machine shop with a lathe, a cutter, a CNC, and all the other machines people built with their machine shops.

Ruby + libraries (for example) is a woodshop with saw, chisel, hammer, and a bunch of useful Jigs built by other woodworkers.

Practical Realities

But.. that’s just a tool shop. I want a tool box 99% of the time

Python is a toolbox. It’s not the fastest & not the best. But it builds pretty quickly 🤷‍♀️

Not saying I wouldn’t learn Scheme. Just prioritizing trade offs. e.g., I learned C++ in 2018 for reasons. - Joshua Kehn

Josh’s got a good point. That type of thinking is embraced by basically every programmer trying to get something built in a timely manner. For example when my coworkers and I sit down to work we either use Ruby + Rails or Elixir + Phoenix. We’re not just working in high level languages, we’re also using libraries that give us skyscraper levels of abstraction. It’s all about using off-the shelf tools that’ll get us to our destination quickly. For us, making reliably things quickly translates either directly, or indirectly into money.

But, some folks who really understood first principles had to spend a lot of time building those abstractions. I’m incredibly grateful to them, but I don’t want to be limited to their abstractions, and I don’t want to be tied to their choice of language to use them.

For example. Let’s take the Maybe Monad. It’s incredibly useful, but Ruby didn’t have that baked in. So some rails developer who understood first principles added the .try method. It took until ~2016 for Ruby to add .dig.

.try and .dig are incredibly useful, but if you didn’t understand the core principles of programming you either wouldn’t think to add them, or you’d implement it in a worse way, because you aren’t aware that there are established and efficient patterns for implementing that stuff.

This brings me back to why I love Scheme. There are geeks who love reading academic and abstract programming guides that have no direct practical bearing on their day job, and critically can actually remember the stuff the next week. Then, there are the rest of us, who don’t remember any of that crap unless we actually use it, repeatedly.

Scheme forces me to use it. Scheme forces me to understand it. Every day I spend programming Scheme I better understand fundamental concepts that give me more options in my day job. Every day I use Scheme I build up a few more abstractions in code that are perfectly suited to my tasks and my way of approaching those tasks.

To put it another way, by having so little baked in, and having such great support for building good things, I’m building my own perfect framework.

Would it have been faster to use Ruby + Sinatra? Absolutely. Would the end result have met my goals as well? Not even remotely. Would I have learned anything along the way? Not a chance.

So, yes, I’m going to reach for Ruby or Elixir in my day job. But for my personal work, I’m going to keep working in Scheme and improving my libraries, and eventually I’ll have my own framework, that is more efficient than Rails or Phoenix for the things I’m trying to accomplish.

Racket and Chicken Scheme both have great collections of pre-built libraries. Racket’s language, and library ecosystem have some of the best documentation available for any language. ↩

Scheme Is My Vim

2020-07-26T00:00:00+00:00

Emacs is arguably the most powerful tool available to the modern programmer. Vim’s pretty close. Both require more effort to learn than say Atom or Sublime Text. But, the additional start-up effort pays off quickly.

Like Scheme, they both suck out-of-the-box. Unaltered they’re both horrible bare-bones skeletons of an editor. Their potential is incredible though. If you’re just going to do something quickly, and never spend the time to customize them, they are a terrible choice.

But, if you’re willing to invest a little time and tweak them to your needs (a macro here, a plugin-there) then you’ll end up with an amazing tool perfectly suited to your brain and your approach to programming.

Watching an experienced Atom / Sublime user with vs. an experienced Emacs / Vim user with a setup customized to their brain is the difference between watching 8 year olds attempting to play Soccer and watching the final match of the World Cup.

Scheme is my Vim. I’m still learning how to customize it, but I can see the possibilities.

Software That Lasts

2020-02-15T00:00:00+00:00

The Cloud Problem

When your data only exists in software that you don’t host yourself it is no longer yours. It can disappear at any moment. Its future is dependent upon a company’s continued profits and interests.

Imagine you’ve been building up a knowledge base for two years, but you fall on “hard times”. Maybe you can’t afford it. Maybe something happened and you’re literally unable to (accidents, medical issues, legal restrictions). What happens to all that data?

If you’re lucky the site has a good export feature, and you have the opportunity to use it, before its too late.

You can’t rely on a company’s VC money continuing to last. They might get bought and end their “incredible journey”. They may simply loose interest in the product.

If you’ve put years worth of effort into it. It would be a full time job to recreate it in another system. So, really you’re screwed.

The Local Problem

When the software lives locally, on your phone, or on your desktop, the situation a bit better. It should continue to work until your Operating System eventually upgrades enough to break it. If you know a breaking update is coming you could stop upgrading your device… forever. This problem is guaranteed to happen for every piece of unmaintained software.

The Solution

There are two parts to the solution.

The Data Solution

Software that manages text (documents, spreadsheets, etc) must be stored natively in a human readable, text format. You must be able to open up its files with a text editor, and with some motivation, be able to make sense of them.

Image and video data is harder, but the formats are generally public so we should be able to read them in 40 years.

Anything with DRM is guaranteed to stop working.

The Software Solution

The software must be open source. I don’t say this as an open source zealot. No company lasts forever. This is especially true for technology companies. Sooner or later, every single piece of proprietary software will die. The question is will it die before, or after you do. This sounds dramatic, but it’s actually a very real consideration. All proprietary software will eventually be abandoned. All of its creators will die.

If you’re going to put data into a piece of software, and you want that data to last for years, it must be open source software. You don’t need to be able to maintain it, but anyone who cares about it needs to have that option. In the worst case scenario, you will always be able to pay someone to make it work with your current computer.

But, being open source isn’t enough. Imagine, for example, that the world moves on from MS Word, and Microsoft decides to open source it. It’s an incredibly large and complex codebase. Many pieces are tied to the version of the operating system they were written for. It’s also, probably not documented very well internally. To keep MS Word alive and useful for the next 40 years will require a large team, working countless hours. A future government that really wants to read their digital archives, might be able to fund this. It’s not going to help normal mortals.

The software needs to be open source, documented, and reasonably focused in its scope. It must be something that a handful of dedicated fans can keep alive.

The considerations needed to design software that will last 40 years is another post entirely.

Choosing Well

For now I want you to consider a some things about the software choices you are making:

Will my livelihood be affected if the the developers who make this die, or suddenly kill this product?
If I keep using this, will I loose years of data that I care about when the developers die, or go out of business?
- Can I export my data?
- If so, would it be in a format I could do something with? For example: PDFs are nice, but they’re terrible for transferring data from one piece of software to another. They’re also annoying to extract data from.
If I suddenly become poor, or otherwise unable to pay my bills, will I loose years of data I care about?
- Note that many sites will delete your data to save space if you stop paying.
If I forget my password, or no longer have an old email address, will I loose years of data I care about?

How SSH Keys let you into servers - a metaphor

2019-06-17T00:00:00+00:00

SSH Keys can be a little confusing to new developers. Here’s a quick little metaphor to help you think about how they work.

tldr; Your public key is your business card. You can give it to people so that they can add it to the list of people authorized to access a machine or service. Your private key is a tool that proves that you’re the person associated with that business card.

…

Imagine that remote machines as members-only nightclubs. The bouncer will prevent anyone from entering if their name isn’t on the list.

You get your name added to the list by having a trusted member of the nightclub add it.

Once that happens, you go up to the door, prove that that name is really you, and then you are allowed entry.

With regards to SSH keys, every key is actually a key pair. There’s the public half (the .pub file) , and the private half. Frequently these are called id_rsa.pub and id_rsa. But, they could be called anything. Your public key is like a business card. You should feel comfortable giving it to anyone you think could use it.

In this metaphor, you gave your business card to a nightclub member and they attached it to the list, so that the bouncer would know to let you in. In real life, you give your public key to a system administrator and they add it to a literal list of keys of people/machines allowed to access something.

Your private key is super secret. You don’t let anyone see it. You don’t even let the bouncer see it. The private key is a tool that lets you prove to the bouncer that you’re the person on the list. The bouncer says “prove you’re XXX” and you use your private key to do just that.

…

There are different forms of encryption that can be used in generating your key pair, and different reasons for each, but that’s largely irrelevant to how they’re used.

Go will never belong to the community until they decide its future

2019-05-29T01:03:48-05:00

There was a recent post* about how Go is Google’s language, not ours. It was an opinionated post, but it provided some evidence to back up its claims.

Russ Cox (a Tech Lead for the Go language) posted a reasoned response to this which, I think basically tried to say that it wasn’t true, and they had regular meetings to discuss community proposals.

But for me, the telling bit of it was this:

We have talked in the past about whether to explicitly look for people outside Google to sit in our weekly meeting, but if that’s really important, then we are not doing our job right.

It doesn’t matter how many different types of people you let speak at a meeting if you don’t let those same types of people participate in the decision at the end.

There’s a huge difference between listening to people unlike yourself and truly understanding people unlike yourself. For example, I’m white. I try really hard to listen to members of the black community, but I am in no way equipped to truly understand what it’s like to be a black person in this country. I’m definitely not equipped to make decisions on their behalf.

This same thinking applies to any group. If the community is not participating in the decisions about their needs then they have no real voice, and whatever you’re discussing absolutely does not belong to them. It’s just something you’ve chosen to share with them, and let them talk about.

For a real world example of this, consider politicians. Their job is to listen to community proposals about what to do with the community. Recently 25 white men who just decided that they were better judges of what to do with women’s bodies than women. Similarly, politicians with stable incomes have been making it increasingly illegal to be homeless. This is what happens when the community gets to speak, but only one type of person gets to decide.

Until the community has a voice in the decisions about Go’s future Go is not theirs.

[UPDATE] Russ responded to my post on twitter with more thoughtful words. The core of it seemed to be that I was misunderstanding the purpose of the “room” discussed above. I may be. But in my experience, whenever an insular group is discussing things that affects others, and asks “should we allow them in here?” the answer should essentially always be “yes”.

* Recent as of May 29 2019

oho's Backstory

2019-05-23T01:03:48-05:00

I’m going tell you the tale of oho, a program which is arguably the world’s greatest ANSI text to HTML converter, and how it solved a real business problem. I’m sharing this because, as a geek, it’s important to remember that you can frequently solve work needs, while having fun creating open source tools that interest you.

Side note: ANSI escape sequences are the things that cause text to be colored, bolded, etc when displayed in a terimnal.

One of the roles I played at One Door was that of Release Manager. The core of that task was figuring out what code was included in, or missing from, a release - and then communicating that info to the Project Managers and Scrum Masters. Most importantly, I needed to make it clear to them if there were things they should be concerned about.

I accomplished this by writing a series of small, focused, command line tools that embodied the Unix Philosophy and worked together to extract data from Git, Github, and Jira, then combine it into a report which was transmitted to the interested parties via Slack.

The larger releases presented a problem though. Project Managers didn’t really want to read through the status of 300+ tickets, or contemplate what unexpected commits might mean to their release. It was too daunting of a task, and they had enough things on their todo list.

One of the strategies I took to help address this was to highlight unimplemented tickets, or unexpected commits, in different colors. Unfortunately, there’s no good mechanism to get terminal colors into Slack. Colored text just isn’t supported. For multiple reasons, the right answer seemed to be to give them a PDF version of what I was seeing in the output of my tools.

I didn’t find any extant ANSI to PDF converters, but that wasn’t surprising. I did find a number of HTML to PDF converters though, and I knew you could convert ANSI to HTML. So, I set up an ANSI -> HTML -> PDF pipeline and quickly discovered that my PDFs looked terrible. The problem was the HTML. Every existing ANSI to HTML converter had some aspect of it that made it just unusable. In one case, my half-page test grid of colors ended up being an unreadably small postage stamp in the corner once it made the final transition. Others didn’t actually support the various color codes very well.

Choosing a technology

aha (Ansi HTML Adapter) seemed the best of the available tools. It was written in C. I figured it was better (and faster) to debug and tweak something that was fast and mostly working, than it was to write something from scratch. Unfortunately, it’s source code was not up to my standards. It worked, but it wasn’t very readable, or commented, or well organized. It was more a stream of consciousness in code form. As I spent more time refactoring it, I realized that by the time I was done, I’d have a completely different codebase that was readable, and maintainable, but still not very extensible.

So, I began to write oho.

It was clear from my work refactoring aha that this was going to be a non-trivial codebase with multiple files. I wanted to write it in a language that would allow me to distribute it without people having to know anything about the ecosystem surrounding it. Ruby is terrible for that because you need to know about Gems, and Bundler, and most systems ship with old Ruby versions and… I wanted this to be simple for users. So, I needed a compiled language. Crystal met those requirements, and I’d been using it for a while and found it rather enjoyable, so that’s what I used.

Crystal produces Dynamically Linked executables (like most compiled languages) so distribution was just a matter of creating a Homebrew script (for Mac users) that told people’s computers where to download it and what dynamic libraries it dependend on. The same can be done for other systems with little difficulty.

Chicken Scheme would have been another choice I could have easily worked with, but beyond just producing a single executable, Crystal is less intimidating to most developers. Lisps are treated as exotic things with “too many parentheses.” I wanted something that could be easily enhanced by anyone in the Open Source community. Crystal looks very similar to Ruby (it’s essentially just Statically Typed Ruby), and organizes its code along lines that are familiar to most developers who use descendents of C (Ruby, Python, Java, etc). Crystal also has a decent testing framework built in. It could be better, but it gets the job done relatively painlessly.

So, I chose a language that would give me a single executable that required no knowledge of the language or ecosystem it was written in to install, would be easy to test, and easy to extend by people unfamiliar with the codebase. If I had it to do again, I might write it in Go, because of the Statically Linked executables it creates, along with easy cross-compilation.

In the end, I had a fast tool that I installed at work and began presenting the project managers with colorful PDFs for each release. Since then, I’ve added support for more esoteric escape sequences, and plan on continuing to do so in the future.

I also had fun writing it.

Quality Is Rarely Job 1

2019-03-03T15:13:58-05:00

In 1981 Robert Cox came up with a slogan for Ford; “Quality is Job 1”.

It has always stuck with me.

In the software industry there are few slogans could be further from the truth. C-level’s and other customer facing types frequently proclaim the “quality” of their products, but they aren’t the ones making the product. They’re frequently not even the ones using the product.

In software there are two viable ways to release quality software. You can release it when everything you want done is done, or you can release what you happen to have done at a specific date. You can’t combine the two, although almost every software company tries to.

People outside of engineering tell clients X will be ready on Y date. That date is inevitably too soon for the promised features to be completed.

Realistically, there is only one way to combine promised dates and promised features in software. You compromise the quality. People could, promise a date generously far in the future, but that’s not very helpful meeting to this quarter’s sales goals.

In order to deliver quality an engineer needs the support of management to not deliver code that isn’t done. “Done” of course doesn’t mean “It seems to work.” or “QA didn’t complain.” Done means “I have tests to prove it works.” Done means “QA can prove it works when users get ahold of it.” Done means “It’s intuitive for users.” For multi-user systems Done means “We can prove it works when tons of users try to use it at the same time.”

Done might not mean “It does everything that was requested.” There are always more features to add. It does mean that everyone, even the client, agrees that the features that are Done, are “enough” for now.

I believe that there are a few companies out there that actually deliver “quality” software that even their engineers believe in. I’ve never worked for someone else’s company that did this. Features sure, but overall product quality? No.

Libraries, Fameworks, and DSLs

2018-08-16T23:23:07-04:00

One of my coworkers was trying to understand the differences between libraries, frameworks, and DSLs and asked me

…how do I know what i’m using when all these things are

interacting and being used within each other, etc

To some degree, you don’t, and it doesn’t matter, but that’s not a very helpful answer. So let me step back and talk about what each of these are.

Libraries

Libraries are the simplest. A library is just a collection of code intended to be reused. They’re typically packaged using a package manager, but it doesn’t need to be “packaged”. It could be just some file of useful code that you load into your code like any other source file.

A library is the hardest to “know” you’re using because it generally provides some new classes or methods. Unless you happen to know where those clasess and methods came from you don’t know you’re using it. In languages like Ruby where you can alter code at runtime a library could change the behavior of existing code; no new methods, no new code, things just behave differently.

DSLs

DSLs are Domain Specific Languages. The greatest example, in my opinion is SQL. For example

select name from students where age > 7

Without any SQL knowledge you can probably guess that there’s a list of students somewhere, and running that line will return the names of each one who is older than 7.

SQL is very powerful, but it’s very focused on querying and manipulating tables of data. It makes that very easy, but it’s really crappy at everything else. That’s pretty typical of a DSL.

In the Ruby world we have a lot of DSLs but people tend to not recognize them as such. For example, an RSpec test might have something like this

describe "Math" do
	 context "double" do
	   it "should return twice the input" do
	     expect(Math.double(2)).to eq 4
	   end
	 end
end

Like SQL it’s pretty easy to guess what it does.

You might be thinking “But rspec’s just Ruby code. That’s just Ruby methods.” This is true. RSpec is a DSL written in Ruby. It’s not limited to Ruby though. I could port RSpec to most languages and maintain the same syntax.

You can think of a DSL as a collection of commands that simplify instructing the computer to do something in a focused domain.

How do you know when you’re code from a DSL instead of your core language? If you are dealing with a non-trivial set of commands, that aren’t part of the core language, and are all focused on helping with one specific aim, you’re probably using a DSL.

RSpec, is a DSL, that’s packaged up as a library, that can be added to your project. Loading the library teaches the Ruby compiler how to understand the RSpec DSL.

Frameworks

Neha Choudhary described it like this:

If I told you to cut a piece of paper with dimensions 5m by 5m, then surely you would do that. But suppose I ask you to cut 1000 pieces of paper of the same dimensions. In this case, you won’t do the measuring 1000 times; obviously, you would make a frame of 5m by 5m, and then with the help of it you would be able to cut 1000 pieces of paper in less time.

That sounds a lot like a DSL, and from some perspectives there is overlap. But a framework doesn’t make it easier to do only one thing. It provides a large infrastructure to facilitate the connection of things in many domains.

I like to think of them like skeletons. Skeletons are collections of bones. Some bones are for protection. Some facilitate locomotion. Some facilitate eating. Collected together they are a framework with areas specialized for holding digestive bits, and areas specialized helping us move around, and so on. Creatures can exist without them, but once you have a skeleton lots of things become easier to accomplish. Code can exist without frameworks, but frameworks make it easier to create code that does big complex things.

Let’s look at Rails (a web development framework) as an example.

Rails (like most web frameworks) provides infrastructure to:

facilitate object relational mapping to store the data you’ve collected about some object in your system in a database, and then take the data in a database and convert it back to objects in your system.
facilitate the handling of HTTP requests from a web server and make it easy to query all their relevant pieces (form data, headers, query string data, cookies, etc.)
facilitate the generation of HTTP responses to a web server with all their relevant pieces (headers, cookies, mime types, content (html, javascript, css, etc)
facilitate the handling of those http request objects it created for you
template management (in various templating DSLs, haml, erb, etc)
facilitating the connection of templates to the things handling the requests
logging
testing (of models, of controllers, of views)
and many, many more things.

You could say that almost all of those involve some DSLs. Making a class a descendent of ActiveRecord::Base connects it to a database table, gives it a getter and setter method that corresponds to every column of that database table, functions to update, validate, save, delete, etc. There are even methods for describing the relationship between other classes tied into tables like belongs_to :some_class

All those methods are part of a DSL for manipulating the data in database tables. Just like SQL. But, that DSL doesn’t know anything about manipulating HTTP requests, even though Rails does.

Rails is a Framework, made of a collection of DSLs, packaged in a Library.

Summary

So, how do you know when you’re using a Library, a DSL, or a Framework?

The only way you’re not using a library is if you’re writing something that only uses code from the smallest beating core of your language.

Most languages come with a “Standard Library” which is a collection of classes to save people from writing a lot of obnoxious low level code. For example: most languages have a Date class (or something like one). That’s part of the Standard Library. Compilers have no need of a Date class. They can do their job just fine without one. That’s a convenience library for the humans.

Are you using a DSL? If using methods defined as part of a Framework then there’s a good chance you are. Frameworks are frequently collections of things that could be classified as DSLs. If you’re using a set of commands that would be useless for anything other than one focused domain, then you’re probably using a DSL.

Are you using a framework? That’s the easiest. Did you load a framework? Are you calling methods defined in that framework? Then yes, you’re using a framework.

If you are using methods defined in a framework, that are also a DSL, then you’re using all three, because frameworks are collections of code intended to be reused.

Ultimately, it doesn’t matter what you’re using, but there is value in knowing where the commands you’re calling come from. When you start another project, you will know what libraries to pull in to give you those commands again without having to create them from scratch.

Converting HTML to PDF on the command line

2018-05-27T21:39:00+00:00

I recently needed to convert some HTML to PDF on the command line and went hunting down the options.

There numerous posts saying “X is great” “Y works great for me” but no-one gives examples that show you anything.

I’ve tried WeasyPrint, wkhtmltopdf, Pandoc and Google Chrome (yes via the command line). The test was simple. Take a simple color chart, made from pre-formatted text and render it as a pdf. This doesn’t test any fancy CSS grid layout, or sizing or … anything. Just straight pre-formatted text with some spans that only define colors.

Here is the original html and here’s what the conversions looked like.

The version rendered from Chrome is perfect. I have zero complaints.

The version from WeasyPrint is cut off on the right edge. Note that it is not the window that’s cutting it off. That’s the edge of the PDF itself.

The wkhtmltopdf is perfect if you zoom waaaaaay into it. Realistically though it’s unusable crap because it’s so small. Note that it has also added a border

Pandoc (not included in the screenshot above) seems to be really focused on text. The result was completely devoid of color, used a different font, and was centered in the page. It did maintain the formatting within the table though.

The winner, for me, is clearly Chrome. The magic incantation is as follows. Save it in a file called html_to_pdf and then you can convert your html with

html_to_pdf path/to/my/html_file.html

It will generate an output.pdf file in the current directory.

#!/bin/sh

# on linux everything before --headless can be replaced with just
# chrome
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \
  --headless \
  --disable-gpu \
  --no-margins \
  --print-to-pdf $1

This all came about because I’d recently written oho (an app for converting your colorful terminal output to HTML), and I wanted to convert that HTML to PDF to more easily share with my coworkers over Slack. Oho is perfect for sharing terminal output on a blog though. ;)

Zed Shaw's Utu: Saving the internet with hate

2018-03-25T13:51:38-04:00

A high level summary (and paraphrasing) of Zed Shaw’s talk at DEFCON 15 (in 2007) because I couldn’t find a good text version.

Utu is the Maori word for a system of revenge used by Maori society to provide social controls and retribution. Utu is also a protocol that uses cryptographic models of social interaction to allow peers to vote on their dislike of other peer’s behavior. The goal of Utu is to experiment with the effects of bringing identity, reputation, and retribution to human communications on the Internet. A secondary goal is wiping out IRC because apparently nobody really likes IRC.

The Utu project is now dead, but its source code can be found on Github. While reading this keep in mind that the core idea is a centralized chat server.

Part 1: The Idea

Philosophical Foundations

Strong Identity
Reputation
Retribution

If you have and combine those, you can start to implement some controls available in human communications.

Strong Identity

In order to have strong identity everyone has a cryptographic key, but because keys are so easy to create you have to be invited, and you have to have a handshake as part of the invitation because you need to care about who the invitee is. This doesn’t work if any jerk can invite any other jerk without consequence.

Zed certify that he wants to invite Joe.
Joe certifies that yes, that invite is from Zed.
Joe takes action to assert that they actually want the invite
Zed says yup, I really did want Joe to join us.

Reputation

Rather than associating likes with an identity or posts you associate “hate” with an identity, and over time that hate builds up from many people.

“Wouldn’t it be great to punish the dickheads?”

Retribution

(Keep in mind that Utu was intended as a chat network)

As a person builds up hate they have to do some cryptographic proof of work before they can do anything. The more hate they have the more work their computer has to do before they can say anything.

This accretion of hate gives you a weighted social network that you can crunch numbers on. You can figure out who “hates” who, who talks to who, where they come from, and we can start looking at the social problem.

You can say “Hey, if we change the hate model to be this way, or we change the hate model to be that way, how does the social network change? Do the jerks go further away?”

Another alternative to hate is distance, because people aren’t prevented from talking to you it just takes them longer.

Part 2: The Implementation

Geeky Details

The Protocol

Sends 2 bytes for size (htons(size)) (for framing)
Sends the data structure (“Stackish”)
- S-expressions with netstrings in a FORTH stack order
- [['4:test' 1234 child root is equivalent to this lisp (root (child (1234 '4:test')))
That’s it.

Semantics

Determined by the Hub (everyone talking to hub)
- routes things to different chat rooms
One Stackish node for header
Another for body

Data Encoding

Always ASCII text, but you can put anything you want in BLOBs.

Simplicity is Key

2 bytes for frame
Stackish for header and body
easy to implement

Security Preventions

Zed believes that you need to design your software with the assumption

..that everyone on the internet is a fucking ass hole and they need to be taken down…If it goes on the internet it’s social software. It shouldn’t be designed assuming everyone likes you… Eventually some person is going to hack it. Eventually someone is going to try and abuse it.

I’m not going to be able to stop everyone…but if the server and the protocol, like from the base of the protocol on up is designed to make it so it’s harder… it’s not making it impossible, just harder… then what I’m banking on is the predator concept. A predator, when picking two different types of prey, will pick the weakest one.

Client does all the crypto work to prove identity first and the server just validates it. If it’s wrong the server puts the identity in a blacklist and then boots them.
Client is considered hostile
Booted immediately
A Finite State Machine controls

The Cryptography

Never write your own. Let the pros do it.
Follow the rules
Use standard stuff

Details on exactly what it uses are at 20:48 in the talk.

Secure Coding Practices

All code “ruthelessly” run through a test suite on a loop, with running stats used to compare releases.

Statistical Quality Control

Collect Information on
- Valigrind erros
- Unit test errors
- Logged Errors
Analyzed the changes over time

Code Auditing

Review my code repeatedly.
Consistent “rules” I follow.
Assume it’s always broken.

C Coding Practices

Bstring library for strings
All functions have valiadtions
- Assert
- Preconditions / postconditions
- Extensive Unit Testing.

Secure Unix Server Practices

The Server Always chroots ALWAYS
Leave minimum resources open.
Minimum configuration.

Practical Demonstration

This starts at 30:41 in the talk and involves a fair number of geeky commentary. For non geeks a) why did you read this far b) it’s just a really simplistic chat client.

The Guy Who Loved His Work

2018-03-21T17:23:02-04:00

(or Why A Healthy Work-Life Balance Is Important)

During the dot com boom I worked at a company with a developer who loved his work. The problems were challenging, and we really valued the things he produced. Everyone who worked late late got dinner, and sometimes he’d work so late that he ended up sleeping by his desk. Bob (not his real name) wasn’t pushed to do this. He just really liked his job.

After about five years the dot com crash hit. A third of the company was laid off. Then another third of the company was laid off. Then everyone left was laid off and all the assets were sold to IBM.

Bob was left with a five year gap in his life. He had nothing to show for that time. No real relationships. No software he could point to. It had disappeared into the bowels of IBM. I assume he became a better programmer during this time, but everything else suffered.

Bob could have been spending those hours writing cool software for himself. He could have been helping open source projects that were important to him. He could have spent time with the people he cared about. He could have found new people to care about. He could have been falling in love.

Day after day, most of us help someone else’s company to thrive, and many of us forget that it’s not our company. It’s not our “baby”. We don’t get rich if it succeeds, and we suffer when the people in charge make bad decisions. Your company may get sold, or implode. The people in charge may decide to abandon the thing you’ve devoted your time to. Even in a healthy company, your work rarely makes the world a better place. It rarely makes the people in it healthier, or happier.

Our time on this planet is too limited, and most of our waking hours go to someone else. They aren’t devoted to improving our lives or our happiness.

Don’t let yourself wake up one day and discover you’re another Bob. Put in the hours you agreed to when you signed up, and then go do something for you.

Do Social Networks Need Companies?

2017-12-22T09:37:51-05:00

I’ve been thinking a lot about Scuttlebutt lately (see my Why Scuttlebutt post), and Srol just wrote a great post about how Mastodon makes the internet feel like home again.

There’s a lot of good reasons for people to use tools like them for socializing online, and I don’t want these services to just wither as their users wander off. I want there to be options that aren’t controlled by large companies, but at the same time services that require servers (like Mastodon) need someone to pay for those servers. And while Scuttlebutt doesn’t need servers, the “pub” servers do play an important role. I find myself wondering what is required for a social network to succeed? Are businesses a critical part of the equation?

I think yes, and no. I think sites like Patreon its like, have shown that people are willing to support the people and projects that are important to them. So, I think that from a purely financial standpoint that no we don’t need the corporate money to keep these spaces afloat. Also, hosting costs for distributed systems like this are dirt cheap, and getting cheaper. There are a million people using Mastodon on servers paid for entirely by people who just feel like it.

At the same time corporate presence helps with the legitimacy of young networks. When major businesses establish a presence on a network it reinforces the idea that this is a “real” place, and that it’ll probably be around for a while.

From the business perspective, social networks aren’t great for selling things (instagram ads seeming to be the major exception to this), but they’re great for supporting your customers and finding like-minded people. Think, for example about what REI has done with their Opt Outside campaign.

Small companies, like indie software developers, survive because of their social interactions. They can’t afford the big marketing. So they make human contact, and support their customers, and help them to succeed in their business, or play, or whatever. And the customers appreciate it. Many of us become fantastically loyal to a product or company specifically because of the social interactions we have with the people behind them.

I look forward to getting on to Scuttlebutt and seeing what new things people have had to say, just like Srol looks forward to getting on Mastodon. Having to then switch to Twitter to ask a question of some developer whose software I use… it’s not hard, but it is a barrier, and it also means i’m going to have to deal with whatever’s going on “over there”.

I love Twitter, but if I’m not speaking with the Esperanto people there I’m pretty much guaranteed to get stressed out with some new political BS, and Facebook has always annoyed me.

But, it takes a lot of effort to for a company to have an effective social presence on a network, all that effort isn’t cheap. There’s got to be a reason why it’s worth the effort to have a presence there. They’ll come, if the people are there, of course, but the question on my mind, is will it help the people to come, or stay, if the businesses are there?

Do the creators of need to think about how companies could derive value from their platform? Not for any financial benefit of the creators, but as a tool to help the platform thrive.

I don’t think McDonalds, for example, being on Twitter devalues the Twitter experience in any way. I don’t follow them. They don’t affect me. I do think that the fact the companies I do care about are on Twitter enhances the experience for me. Is there a downside to the users to making the network more useful to those companies?

To be clear, I am explicitly not talking about allowing them to put ads in front of users, or charge for anything, or directly recognize revenue in any way. I’m talking about helping them to know that their presence on the network provides benefit to their company. Maybe it’s just knowing they got the word out about something important. Maybe it’s supporting their users.

I think maybe we should be thinking about the companies, if we want to see these new networks survive to maturity.

How to use the same Scuttlebutt identity on multiple computers.

2017-12-19T22:43:35-05:00

v2.1

(Note: Manyverse / mobile users please see the warning at the end.)

Offically Scuttlebutt doesn’t support posting from the same identity on multiple computers (as of Dec 2017). Unofficially, it’s easy but requires a little bit of care. In practice this means never run the Scuttlebutt client on two computers at the same time.

The gotcha is that if you post from both computers before the changes of one have had a chance to replicate to the second via scuttlebutt one or both of your feeds will get screwed up and other people won’t see some of your own posts ever again.

Note: These instructions require using the command line. I’ll show you how to do it on a Mac. Hopefully that’ll be enough of a clue for Other OS’s too.

Step one.

On the computer where Scuttlebutt is already running, Take your ~/.ssb/secret and make a backup of it in some secure encrypted location like 1Password, because you really don’t want to loose that and you don’t want to give folks keys to your identity.

How?
Open your Terminal app and type

open ~/.ssb

(then return). A window will open up in the finder showing the contentns of your ~/.ssb directory. Copy the secret file from that directory and put it somewhere safe. Then send it and the gossip.json file to the 2nd computer.

Step two

On the 2nd computer create a ~/.ssb directory in there and put a copy of your secret and gossip.json files in there.

How?
Open up your terminal app and type:

mkdir ~/.ssb
open ~/.ssb

Now drag a copy of your secret and gossip.json file into the window that just opened.

Step three

Confirm Patchwork is off on the 1st computer. Boot Patchwork on the 2nd computer. It will ask you to fill in a profile. DO NOT FILL IT IN. Instead click “Cancel”. Then wait. It’s going to have to re-download everything you had before, including your profile. After it’s done syncing quit, then relaunch Patchwork to make it re-attempt to connect to other pubs. Currently (Dec 2017) it doesn’t always sync with all the pubs you were connected to the first time it loads.

WARNING

Do NOT make Patchwork an app that opens at startup. Do not start Patchwork immediately after waking up a computer. Always quit patchwork on one computer before opening it on the second one. Then wait for all the posts to finish syncing before you publish anything or update your profile.

Manyverse users

Currently the SSB Mobile client Manyverse is still very new. Mobile devices add a whole bunch of weird edge cases to how and when things function. My personal advise is to not try sharing your identity with a mobile device at this point in time. In theory, as long as you let the posts from your computer(s) sync to your mobile device before posting from there, and vice-versa, you should be ok. That being said, the consequences of this going wrong are being forced to abandon your old identity because 1/2 of the world can’t see any new posts from it. It’s just not worth the risk.

I don’t have an Android phone so I can’t test it. Instructions are available in Scuttlebutt though if you’re willing to take the risk.

Keeping a great Changelog

2016-07-03T13:03:55-04:00

Changelogs are an invaluable, and often neglected part of any software project. So, how do you do that?

A good changelog helps you users to understand:

Why they should care about your latest version
If any of your changes affect the problems or frustrations they’ve been having.
If there are any changes that might affect how they use your app / library.
Why your efforts are worth their continued support.

A great changelog does all that, and shows the personality of your team.

Shifty Jelly’s Pocket Casts is notorious for their fun, and informative changelog entries:

Equal Opportunity variable speed playback. Why should MP3s have all the fun? Welcome AAC & OGG! * Bzzz. Bzzz. You can now choose which podcasts to be notified of.

Slack’s been doing a pretty bang-up job too:

Fixed: The app now automatically scrolls down when you begin writing a file comment, giving your precious fingertips much needed beauty rest. * Fixed: Time zones were sometimes shown with incorrect deltas. Now your coworker who is 5 hours ahead of you will no longer simultaneously be 8 hours behind you.

I can’t overstate how valuable it is to have a Changelog with personality. As soon as you see that one of the apps you use has a great changelog you look out for them, and always stay on top of what they’ve fixed and added. Then you start spreading the word, because not only did they add good stuff, or fix that really annoying bug, they made you smile along the way.

A Changelog with personality helps build evangelism for the thing you’ve spent so much time building.

The guts of a good Changelog Entry

It must speak in terms users understand. Most of them won’t be intimately familiar with your source code. Don’t use internal jargon. If a bug is fixed, speak in terms of the actions that caused it, or the behavior it exhibited, but not the specific classes, unless you absolutely have to.
It must be knowledge that’s useful to your users. Speak to the features and flaws that affect your customers. Users don’t care that you refactored some unit test, or reformatted your comments.
It has to be at the right level of abstraction. Even if your writing an open source library. The Slack timezone bug above was handled perfectly. They could have mentioned which class & method had the bug, or the conceptual flaw that caused the bug, but that’s all too fine grained for a Changelog. Instead, they spoke at a higher level of abstraction.

Changes need to be grouped by type

Group your bug fixes. Group your features. Group “all the things!”

Imagine a user who’s been particularly frustrated by a bug in your app. They’re going to open your changelog and immediately try and find out if you’ve fixed it in this release. Help them. Don’t frustrate them further by making them hunt for it in a jumble of features, fixes, and changes.

Dates are important

Versioning your software is great, but dates are important too. When did version 6.0.0 come out? Did I have that problem before then or after?

Additionally, not everyone in the world keeps dates in the same format. 06-08-16. Is that August 6th or June 8th? It depends on who’s reading it. Year Month Day is the only one that makes any logical sense no matter where you are. XKCD even had a Public Service Announcement about it.

To summarize

Write at a high level of abstraction
Speak to things that affect users, not developers.
Group things so that people can find them easily.
Add some personality.

Want an easy tool to help manage this?

Check out changelog_manager. It’s a free and open source tool to help you easily generate high quality Changelog entries.

Why you can’t auto-generate your Changelog

Why you can't auto-generate your Changelog

2016-06-30T08:28:07-04:00

Let’s start by taking it as a given that a Changelog file is something very valuable that every product should come with. Even if your “product” is a library for other developers.

With that in mind, the question rises of “How can I make it really easy to generate one”. Many developers have had exactly that thought. There are many free and some paid solutions that will “Autogenerate your changelog from your git commits/tickets”. The simple fact is, that no matter how well they’re written, you shouldn’t use any of them.

A good Changelog entry has…

A good Changelog entry has three primary aspects

It’s at the right level of abstraction. Consumers of your product don’t care what files you tweaked or how. They care about the fixes and changes you’ve made that may impact them.
It needs to be knowledge that’s useful to the consumer of your product. It’s a simple judgement call, but it’s one that has to be made.
It needs to be written in a way that people who either aren’t geeks, or at least aren’t intimately familiar with your codebase can understand.

Changelogs from commits

Commits are at the wrong level of abstraction

A git log is filled with commits like: “ensured current_and_future_campaigns returns a relation”, or “cleaned up whitespace in User.rb”, or “fixed expected aspect name in update test for locations_controller_spec” These are useful to other developers on the project, but they aren’t things that are useful to the users of your product.

In short, they contain too much detail, at the wrong level of abstraction.

Commits are messages from one developer on a team to another one

They’re not messages to your users, and they shouldn’t be.

Commits frequently use jargon, and references to specific elements of code. This is absolutely fine, but it’s probably meaningless or useless to most of your consumers.

There’s no point

If I wanted a geek level listing of all the changes in your git history I could just ask git: git log --oneline Plus, git will let me slice and dice that information sixteen ways from Sunday.

You can’t edit them

As Commits From Last Night proves. Commit messages frequently aren’t written in a customer friendly way. Even if you’ve got very polite developers, everyone makes typos. You can’t edit the old commit. If you want a changelog that looks professional you’re going to have to manually edit it, the same mistakes every single time you generate it.

Tickets are at the right level of abstraction

…but they rarely have titles that are appropriate for a changelog.

How about “Item Finder - Page gets wonky after searching for content that doesn’t exist - plus 404”, or “Something’s eating up all the ram and taking down the servers” Not really appropriate for your changelog.

If you’re really following agile an agile methodology you have user stories that look like “As a I should be able to ___ because ___” and that’s no good for your changelog.

Some projects do take the time to curate their ticket titles. The results are almost usable as changelog entries too, but there are always exceptions. They’re generally worded well for the people who need to work on them, but not your consumers.

It’s a matter of human judgement

You want a changelog with the information that is useful to your consumers, and written at the right level of abstraction. This means people need to ask themselves some questions. “Do we need a changelog entry for this?”, and if so, “How do I summarize this change in a way that’s useful to our users?”

The first question sometimes eliminates information, through curation. That’s something tools can’t do. The second question changes the form of the information conveyed. It was directed internally, now it’s directed externally.

Want an easy tool to help manage this?

Check out changelog_manager. It’s a free and open source tool to help you easily generate high quality Changelog entries.

Keeping a Great Changelog

Adding Press and Hold Characters to macOS (OS X)

2015-12-13T00:00:00+00:00

EDIT: As of Monterrey Apple has completely locked down the files you need to edit.

There are 2 ways I know of to do this, and neither of them is as good.

use TextExpander. You can tell it that when you type cx to “expand” it to ĉ. You’ll probably have to copy the character you want from here or any other page with it. Then do the same for all the other characters that need ˆ. Here is a public snippet group with the Esperanto Diacritics that I’ve put together. Just type cx to get ĉ, and so on for all the characters that use the ˆ or ˘ diacritics.
use Keyman. They specificaly have a QWERTY Esperanto keyboard layout. However it’s not what you think. It doesn’t give you press-and-hold menu, and it doesn’t actually let you type a ĉ. what it does, is exactly the same thing as Text Expander, only with +h or +x. Basically, what you get is an incredibly limited free version of Text Expander, that only works with the QWERTY keyboard layout.

In short, buy Text Expander, because you’re going to get a ton of other cool functionality and it’ll work regardless of what keyboard layout you use.

OS X makes it easy to type accented characters by simply holding down the letter you want accented. If you hold down c (for example) you’ll see this:

Well, I do. You probably don’t see that 4th option: “ĉ”, and that’s what this post is about. I needed to add that, and others to support typing in Esperanto without switching to a different language keyboard. This technique is also useful if there is a language keyboard you want to use but it doesn’t have the Press and Hold support you want.

Here’s the deal though. The files you need to edit, are in an area of the system that Apple has locked down so hard that even the root user can’t touch it. Normally the root user can do anything it feels like. You’ll need to temporarily disable System Integrity Protection and run a few commands in the terminal to replace the XML file that defines what characters show up in the Press and Hold menus. You’ll need a developer’s text editor (not Text Edit), and knowledge of how to copy and move files on the command line.

If that’s all gobbledygook and/or scary to you, then you should probably stop here. I’m sorry. Apple’s made this a geek-only operation.

Still here? Good.

Here Be Dragons

There’s a system folder in which You’ll find a variety of files. We need to modify the file in there that corresponds to your default language. If you’re reading this your default language is probably English, so you’ll want to Keyboard-en.plist (“en” for English). If not, find the file with the 2 letter abbreviation that corresponds to your system’s language.

In El Capitan this folder is here

	/System/Library/Input\ Methods/PressAndHold.app/Contents/Resources/

In Sierra this folder is here

	/System/Library/Input\ Methods/PressAndHold.app/Contents/PlugIns/PAH_Extension.appex/Contents/Resources/

If you just want to add Esperanto character’s to your English Press and Hold menus then download this file (right click, save as Keyboard-en.plist) and skip down to the part about Disabling System Integrity Protection.

Modifying your Press and Hold plist file.

Once you’ve found the appropriate file copy it into another directory and start editing. You can’t edit it in place.

First find the <key>...</key> line that corresponds to the key you want to edit. When I went to add a capital C with a caret over it I found the <key>Roman-Accent-C</key> line, then edited the <dict>...</dict> block below it. I’m fairly certain that the first character must not be changed. The other characters are the ones that will show up in the Press and Hold pop-up. Modify them however you want. You can even put emoticons in there. I just modified to have the Ĉ you see below. Note that there are spaces between each character. What happens if you string multiple letters together without a space? Dunno. I’m betting you’ll get an entire word for an item in the menu.

	<key>Roman-Accent-C</key>
	<dict>
		<key>Direction</key>
		<string>right</string>
		<key>Keycaps</key>
		<string>C Ç Ć Č Ĉ</string>
		<key>Strings</key>
		<string>C Ç Ć Č Ĉ</string>
	</dict>

Once you’ve modified all the uppercase and lowercase versions of the letters you want the pop-up for it’s time to replace the original. I recommend making a back-up first. Just in case.

Temporarily disabling System Integrity Protection

You’ll probably want to print out / write down these instructions because you won’t be seeing them after restarting.

Restart your computer. Hold down the “Command” key and the “R” key when you hear the restart sound. You’ll see a progress bar as it boots you into Recovery Mode. Ignore the window listing backups and disk utility. Click on the “Utilities” menu, and choose Terminal.

Disable SIP and reboot (to make it take effect).

csrutil disable; reboot

sudo cp path/to/your/edited/Keyboard-en.plist /System/Library/Input\ Methods/PressAndHold.app/Contents/Resources/Keyboard-en.plist

or this for Sierra & High Sierra

sudo cp path/to/your/edited/Keyboard-en.plist /System/Library/Input\ Methods/PressAndHold.app/Contents/PlugIns/PAH_Extension.appex/Contents/Resources/

Reboot and go back into Recovery Mode. Open up the Terminal again, and run

csrutil enable; reboot

That’s it. You’re done. Now your customized popover will be available to any keyboard in your default language.

Software worth recommending

2015-11-14T00:00:00+00:00

There are lots of great tools out there. Far too many to try. Here are the ones that I’ve tried, and found worth recommending. Mostly they’re

OS X

Developer Things

Quiver is a “Programmer’s Notebook”. I’ve recently switched to it from CodeBox which seems to have been abandoned by its developer. I’ve got some minor quibbles with Quiver, but overall it’s pretty nice and I’ll be bringing all my code snippets and reference material over to it.

Dash is excellent for anyone who needs to search language / framework docs semi-regularly.

MacVim because I can’t fathom why anyone who wants to use vim on a mac would limit themselves to using it in a terminal when they don’t have to.

Kaleidoscope It’s a great visual diffing app. If you use diff on anything resembling a regular basis, you should give Kaleidoscope a try.

Markdown

Marked 2 for previewing Markdown files and exporting them to other formats, this is the app you want. That being said, I’d only recommend it to people who really love Markdown. People who only use it occasionally probably don’t care enough to justify buying this.

Deckset It’s presentation software, but you build your presentations with Markdown. Yay. Not as full featured as I’d like, but it’ll let you build very nice presentations.

Flow Charting

OmniGraffle is the king of the flow charting apps. It’s awesome. End of story.

But…
Monodraw has earned a special place in my heart. Recently I had to wrap my head around a complicated data flow. When I was done, I created a flow chart in Monodraw and pasted it into the source code so that the next time I have to look at that code I’ll have a nice graphic to guide me through it.

Miscellaneous

Noizio is great for noisy work environments. Turn it on, along with some music and you won’t make out anything your coworkers are blabbing about. And no I’m not suggesting making everything so loud as to drown them out.

Day One This is a really beautiful app, and the iPhone version is great. The companion app on the watch looks like a really nice addition too. I haven’t been using it much, but it’s the kind of app that makes you want to use it.

Pixelmator Screw Photoshop. Get Pixelmator. It’s affordable, powerful, and exceptionally well made.

DaisyDisk Helps you figure out what’s taking up your disk space. I don’t need this very often, but when I do it’s invaluable.

Skitch If you need to make and annotate screenshots, this is the app to use.

Bartender Helps address an excess of menu bar icons.

Transmit If you deal with FTP, SFTP, S3, or pretty much any other file transfer system on a regular basis, then you want this app.

Screenflow Excellent, easy to use screencast software, that lets you go back and add really useful annotations, highlights, and more.

Multiplatform

Spacemacs “The best editor is neither Emacs nor Vim, it’s Emacs *and* Vim!”. I love Vim. I envy the extensibility of Emacs. This is the first implementation of Vim (technically it’s Evil) on Emacs that has come even close to working. Normally they piss me off within 30 seconds. This has me wanting to take the time to figure out how to replicate my vim workflows in it.

Scrivener Writing anything of significant length? This is what you want. Be sure to go through the tutorial. There are a ton of powerful features hiding in plain sight. Out of the way until you need them.

1Password It’s the best password manager out there, and they’ve had their code independently audited by security experts. Don’t even get me started on how ignorant / stupid it is for people to not be using a password manager.

Vim plugins

The list of plugins I’m currently using is generally in flux. I’m always trying to improve my vim setup. Check my .vimrc for the lastest complete list. Here are some that I think are definitely worth sharing by all.

Vundle Best vim plugin manager I’ve found.

Fugitive Spectacular git integration. If nothing else, check out the GBlame command.

Tabular Align your text! Make things more readable. Check out this vimcast for a good idea of what you can do with this.

tcomment lets you easily toggle if a piece of text is commented out or not.

airline it’s the successor to Powerline.

expand region is a Vim plugin that allows you to visually select increasingly larger regions of text using the same key combination. Doesn’t seem like much but in practice has proven pretty nice.

vim-renumber For folks who like to edit markdown in vim. This will renumber ordered lists for you.

Command Line

Oh My Git The best prompt enhancement I’ve found to tell you what branch you’re on, if it’s been pushed or not, and tons more. That link’s to my fork of the project, which has some visual tweaks. The original is here.

git-wtf Helps you know wtf the differences are between your current branch and the upstream branch it’s tracking.

The Fuck Every time you type out some long command and then git is all “did you mean…” because you made a typo? Yeah well, The Fuck fixes that. It picks up on the suggested correction, and let you type fuck to execute the command you meant to type.

Static vs. Dynamic Blogging

2015-10-18T00:00:00+00:00

You’ve got a lot of software options when setting up a blog. Over the years. I’ve used or tried most of the options including, but not limited to: WordPress, Jekyll, Octopress, and at least 3 custom built systems.

What follows is my thinking on the pros and cons of each option, and why I’m switching back to a static blog system (Hugo this time).

Dynamic Blogs (WordPress, etc.)

Dynamic blogs, like WP, have a lot going for them:

Easy to use
Lots of plugins
Great media handling
Usable from any internet connected device with a browser.
Great selection of themes
Instant publishing
Scheduled posts
Multiple Authors

So, what’s the downside? It seems like every few months one of our WordPress blogs gets hacked and Google starts warning folks it has malware. You have to constantly log in to update it, and every time you do you run the risk of hosing your database (admittedly a rare occurrence), or breaking compatibility with one of your plugins (happened a few times to us).

Another big downside is when you do happen to write something popular, and thousands of people try hitting your site at once. Many a WordPress (and similar) blog has gone down under the strain of success.

Squarespace would be an excellent option except for the $8+ per month, per site price tag. $24 if you want to be able to actually modify the HTML, CSS, or JavaScript. Maybe, if you only have one, but I manage 11+.

Static Blogs

There are notably good things about static blogs:

The fastest possible page loads.
Can withstand crazy high request loads
Zero maintenance or security worries.
Minimal hosting requirements
Markdown support
- you may not care, but I demand it of a blogging platform.

There are a number of downsides though:

Can only blog from home computer (without bullshit hoop jumping)
- Prose is the only decent exception I know of. Tinypress may be an option too, but they’re too new to say.
Multistep publishing
- run build script
- sync files over scp / rsync / ftp / whatever.
Generally weak media support
- no auto-resizing
- no media players
- have to think about where the images live
- very limited selection of plugins (if any)
Typically poor selection of themes
Limited layout capabilities
- they all use markdown, and Markdown can only produce a restricted subset of HTML.
Slow site generation
- Hugo is the only exception I’ve seen to this rule.
Metadata gets mixed in with posts instead of stored in a separate database
- All static blogs seem to use Jekyll idea of Front Matter embeded at the top of your post.
Multi-author sites require either human organization or hacking. It’s not really supported well out of the box.
Geeky
- Very few of the static blog options seem particularly usable by the non-geeks out there.

Note

Being a geek, I’ve got some advantages. When I used Jekyll I wrote JekyllMail which lets you create blog posts via email. It even handles attached images. I could do that for any other platform too. That addresses what’s probably the biggest downside to static blogs. Again, using Prose would too, but that’d tie me to Jekyll on the back end, and I just wasn’t thrilled with it.

Why’d I Choose Static?

With so few upsides, and so many downsides, a dynamic blog would seem the obvious choice. So why would I switch back to a static blog?

At this point I require any dynamic system to also be a managed system (like Squarespace) that doesn’t cost a boatload of money to run a bunch of sites on. Also, I need to actually like the software. Ghost is a decent, and relatively affordable option but I’ve had problems with their import, and I’m not thrilled with the software.

Medium and Svbtle both have their proponents, but posting there doesn’t contribute to your success or credentials. It contributes to their’s.

Static blogs give me a few big things:

The ability to stop compensating for or worrying about hackers
The confidence my site will stay up when I post something popular
The ability to customize the look and feel
Unlimited blog sites for zero additional dollars.
- I just pay a monthly fee for hosting space with as many domains as I want.
- Or do it totally free by using GitHub pages…

Also, and I realize this is totally illogical, building a site with a static blog gives me a feeling of having actually been a part of it’s creation.

So, for now I’m using Hugo. I’m going to have to start converting my other blogs over because they’ve been hacked again. I just need to find a way to enable my wife to easily post to our shared blog at CorporateRunaways.

Want to debate this? Think I missed something? Hit me up on Twitter.

The Five Virtues of a Great Programmer

2014-10-03T00:00:00+00:00

In Programming Perl Larry Wall (in)famously suggested that programmers had three great virtues: Laziness, Impatience and Hubris. Over the years I’ve kept coming back to those because there’s a real truth to the idea as he originally presented it, but it’s limited, and his definition of “Hubris” has no relation to the actual word. I believe that those may be aspects of real programmers, a great programmer goes beyond that. Building on Larry’s idea, I present you The Five Virtues of a Great Programmer:

Laziness

The quality that makes you go to great effort to reduce overall energy expenditure. It makes you write labor-saving programs that other people will find useful and document what you wrote so you don’t have to answer so many questions about it.

Impatience

The anger you feel when the computer is being lazy. This makes you write programs that don’t just react to your needs, but actually anticipate them. Or at least pretend to.

Hubris

The belief that you can do a better job than the people who came before you.

Humility

The belief that despite your hubris, there are more talented coders than you, and anything you produce could be better. This leads to well tested code, because you might have screwed up.

Unselfishness

The quality that makes you document your code, and leave helpful comments and commit messages, so that others don’t have to puzzle out what you’ve done. It makes you take the time to help coworkers understand things, and change your code to take their needs into consideration.

Vampire Bug [Definition]

2014-10-01T00:00:00+00:00

Vampire Bug: n. something that worked when you went to bed at 2AM, but when exposed to the light of the next day dies horribly. Typically the exposure proves that it couldn’t possibly have been working at 2AM either.

How to enable GitHub 2 factor Authentication on new device / app

2014-09-30T00:00:00+00:00

( as of Sept 30th 2014 )

These are the instructions for how to do it if you’ve already got it configured and need to add a new app / device. If you don’t have it set up already, GitHub’s docs are… probably passable.

Go to Settings
Click on Security
In the “Two-factor authentication” section click “Edit”
- Yes, even though you don’t want to edit it.
Under “Delivery options” click “Reconfigure two-factor authentication
- Yes, even though you don’t want to reconfigure it.
Click “Set up using an app” or “Set up using SMS”

What would you want to do?

2014-07-23T00:00:00+00:00

This afternoon my intern asked me this simple question. She’s a new developer, and a friend of hers is working in a fresh codebase, with best practices. Everything is nice, and he can keep the entirety of it in his head. She’s working with my team, Support Engineering. We’re the front-line bug squashers at our company. We’ve got a legacy codebase with no tests and brain melting insanity around every bend.

So she asked me which environment I’d want to be working in if I was her. The question, and the answer speak a lot to what it takes to become a great programmer.

What I would want if I was her is simple. I’d want to be working in the new codebase. There’s no question. A new codebase is always more enjoyable to work on. But, what I would want is not what I would recommend with 19 years of professional programming experience.

My suggestion was that right now she was in the best place she could be: debugging a bad codebase with good instincts and helpful mentors, and that’s why I’m writing this.

I firmly believe that the most important skill you can learn as a programmer is how to debug: how to debug well, and how to debug fast.

New codebases are great, but they’re like kittens. One day they’re cute and fun, and the next thing you know they’ve grown into a tom cat who’s running around spraying your furniture. It’ll be too big to hold it all in your head, and large swaths of it will have been written by other people. Even if you’re really lucky, and people have written great tests, and some god has smiled down upon you and granted you the only significant codebase with no bugs, you’ll still need to use your debugging skills.

In the typical case you’ll need debugging skills to figure out where the problem is, and then what exactly is going wrong. In our mythical perfect codebase you’ll need to figure out where a feature is implemented and how it works in order to build on it. There’s really very little difference: hunt something down, understand it, make some changes, write some tests, move on.

People with bad-ass debugging skills get things done faster, in any industry.

That’s just the start of why our situation is a better one for her than the happy-fun-land her friend is in. New developers haven’t experienced pain yet. They’ve seen the patterns books but they haven’t internalized the problems that resulted in those patterns. They won’t know what problems those patterns were designed to avoid, so they won’t know when they’re about to create that problem. I think that even if they were to go read the best pattern book for their situation they wouldn’t be able to truly appreciate it because they don’t know the classes of problems they’ve got to deal with.

Debugging a legacy codebase though… If you’ve got good instincts, or good mentors (preferably both) you’ll be able to see all the ways to not do something. You’ll encounter things that just feel wrong, and have people who can explain to you why they’re wrong, or what piece of the puzzle you’re missing. Once you’ve found the bad, you’ll have to figure out how to make it good, and you’ll learn about tradeoffs, because professional coders rarely have the luxury of refactoring everything to be the best it could be, especially in crufty old legacy codebases. No, you have to find the balance. You have to figure out what you can do to improve the situation without spending so much time that the other bugs pile up. It’s a lesson I’m still learning nearly 20 years in.

When she eventually moves on from her job as a professional bug slayer she may not know the right way to write good code, but she’ll definitely know how to not write bad code, how to make code that other people can understand (because she had to wade through so much incomprehensible stuff), and why you don’t violate encapsulation.

A year from now her friend will just be starting to experience the consequences of his inexperienced decisions. A year from now she will be ready to tackle anything.

Writing good User Stories

2014-02-20T00:00:00+00:00

First, it should be noted that not all stories are “User Stories”. For example a developer might be tasked with manually running some script. The Story might simply be “run the fooberry script”.

For everything that effects the UI, use a template:

As a < type of user >
I want < to perform some task >
So that < I can achieve some goal >

Note that it’s all about what the user wants. This isn’t about instructing what change to make to a system. It’s about advising implementers on what the desires of the user are.

Sizing, details, content

Now, everyone says user stories should be “testable”, and “small”, but what does that mean?

Stories should be “Testable”

You’ve heard it before, but its a horrible way of expressing the intent. In general, a story is not directly testable. A Story is only testable through its Acceptance Criteria.

Acceptance Criteria?

Each Story should have a compliment of Acceptance Criteria which cover every interaction, and outcome of the Story. Think of it this way. If you write code that meets every Acceptance Criteria and nothing else your coding work should be done. You should not need to write anything else in order to satisfy the Product Owner. Furthermore, you should be able to write a unit or functional test for each acceptance criteria. If you find your test is testing multiple aspects of the system then your acceptance criteria is too large in scope.

For example: if you’ve added a new page to your app one of the acceptance criteria might be “Administrative users will see a link to the new page in the upper nav. on all pages” This acceptance criteria would pass regardless of where the new link appeared in the “upper nav.” If the Product Owner cares where it appears then the criteria isn’t specific enough. Note that it also specifies what pages it will appear on. Without that you could put it on the “wrong” page and still pass the criteria.

In XP (Extreme Programming) they use a 3“x5“ index card to create all their user stories. The story (in template form) goes on the front, and the acceptance criteria go on the back. The rule-of-thumb is that if the acceptance criteria don’t fit on the back of the card, then the story is almost guaranteed to be too big.

Story Sizing

How small is too small? A Story is too small if, by itself, it provides no business value, like this: “As an Administrative User I want to have a link to the new page so that I can navigate to it.” Unless someone’s already created and deployed the new page, it’s a link to nowhere. The next question, of course is how big is too big? I’d say it’s too big if it can still be decomposed into multiple smaller pieces of deliverable business value.

General Tips

Know your users, before you can write a story about what a user wants, you need to know who your users are. Put together, and commonly agreed upon set of user Personas, and use those persona names when writing the “As a… “ portion of your user story. E.g. “As an Accounts Manager I want…”
Focus on the user, not the product. Keep your stories focused on the user’s goals, and helping them to achieve them.
Start big, and decompose It’s generally much easier to write an “Epoch” level story (one that comprises many units of business value) and break that down into increasingly smaller stories, than it is to start with fine grained stories.
Use index cards (to start with at least). The physical limitations of the card really help make it obvious when a story is too large, because you can’t fit it, or it’s acceptance tests on one side of a card (unless you write in verrry small print, and that’s cheating).
Acceptance tests should be automatable If you can’t automate the acceptance test, it’s probably not a good one. They should be very focused, and very precise.

Finding the Github Pull Request for a topic branch

2013-12-24T00:00:00+00:00

Finding the Github pull request associated with a branch.

Work on a large enough project, with other people and sooner or later you’re going to find some commit, or branch, and want to know what was in the pull request that merged it in. Maybe you want to see what other commits got merged over at the same time. Maybe you want to see what the diff was at that point in time. But, how do you get from a branch name to a pull-request.

The key to solving this is knowing that pull requests are actually branches. They’re just… hidden branches. So, first you start tracking them…

Locate the section for your github remote in the .git/config file. It looks like this:

[remote "origin"]
    fetch = +refs/heads/*:refs/remotes/origin/*
    url = git@github.com:joyent/node.git

Now add the line fetch = +refs/pull/*/head:refs/remotes/origin/pr/* to this section. Obviously, change the github url to match your project’s URL. It ends up looking like this:

[remote "origin"]
    fetch = +refs/heads/*:refs/remotes/origin/*
    url = git@github.com:joyent/node.git
    fetch = +refs/pull/*/head:refs/remotes/origin/pr/*

Now, after a git fetch origin you can say git checkout pr/999 to checkout pull request number 999 as a local branch and diff it, or do pretty-much anything else with it.

Once you’ve got that set up, it’s just a matter of comparing your local topic branch to the other branches in your system to find other branches that point to whatever branch you’re interested in.

I use a shell script called git-find-twins which iterates over the known branches (including pull request branches) and finds any whose tree-ish matches that of the current branch. You can find the latest source here in Github.

[Note: My apologies for the disappearing code on the right side in these examples. Just grab the source from the link above.]

    #!/bin/sh
    # Finds other git branches whose HEADs point to the same 
    # treeish as this branch.

    # thanks to mockinterface on 
    # http://stackoverflow.com/a/20756047/13973
    # for showing how to do this.

    echo "Looking for branches pointing to: $(git rev-parse HEAD)"
    git for-each-ref --format="%(refname)" | \
        xargs -I refname   \
        sh -c '[[ $(git rev-parse HEAD^{tree}) == $(git rev-parse refname^{tree}) ]] && echo refname'
    echo "DONE"

Add that to your path, and make it executable then check out the branch you want to find the pull request for, and run your new shell script with git find-twins (note the space) and you’ll see output like this.

$ git find-twins
Looking for branches pointing to: cb3c2dcf606a0a26107131697dd37aef53d44aad
refs/heads/9284_vzw_bug_fixtures_without_fixture_types
refs/remotes/origin/9284_vzw_bug_fixtures_without_fixture_types
refs/remotes/origin/pr/530
DONE

That last one is what you want. 530 the number of the pull request, which you can put into an URL to see the pull request. Now, you’re probably saying “Yeah, but what URL?” well, one way to get the correct url is to just bring up any other pull request in the current project and change the number in the url. Or you can add this shell script, which I call url_for_pr which generates an url for the pull request number in the current repo. It’s based on the assumption that “origin” is the Github repo. If it isn’t you’ll need to tweak what remote it gets its info from. You can find the latest source here on Github

[Note: My apologies for the disappearing code on the right side in this example. Just grab the source from the link above.]

    #!/bin/sh
    # when run from a git repository, and passed the number 
    # for a pull request, it will generate an url for 
    # where to find it on Github. This assumes that
    # "origin" is pointing to the Github repo.
    # This will work with Github Enterprise too.

    if [ $# -eq 1 ]; then
        ROOT_URL=$(git config --get remote.origin.url | \
            sed s/.*@// |sed s/\.git// | sed s/\:/\\//)
        PR_NUM=$1
        echo "https://$ROOT_URL/pull/$1"
    else
        echo "Please pass in a Pull Request number"
        echo "Note that this should be run within a git repository"
    fi

Bonsai Coding

2013-12-06T00:00:00+00:00

Writing code is a lot like maintaining a Bonsai Tree. If you stop pruning it it’ll stop being a Bonsai and turn into a bush. Little tweaks, frequently aesthetic ones, will help to keep it beautiful and under control. It will still grow in unexpected directions, as other developers make changes, but careful pruning will keep it balanced, and healthy.

What is “careful pruning” then?

Each file is a branch on our tree. The methods, are leaves. We come in to work and start examining one of the branches. Some days we need to encourage it to grow a specific way by adding a feature. Some days we make little snips to correct a bug. But what happens if, upon examining your branch for other reasons, you discover that it’s grown into spirals and knots. You can’t reach your clippers in to snip the leaves you need. You can’t do much of anything without difficulty and frustration, and frankly, it looks like crap. The unexpected spirals and knots need to go.

This, however, is the point where you’ll hear other developers saying “No! Don’t do that!” They’re upset, because your task wasn’t related to the knots and spirals. You were just there to trim some leaves. If you go and get rid of the “other” problems then how will they be able to tell where you trimmed?

To abandon the metaphor for a minute, they’re concerned that the diffs will be so filled with aesthetic changes that they can’t see the changes that were relevant to the bug fix, or feature add. And, they have a point. Being able to look back and understand the changes another developer made is a very valuable tool. But some developers get so obsessed with this idea, or so beaten down by others who are obsessed with it that they say things like this,

“The rule for controlled development is that if the line is not relevant to the fix, you do not modify it. If we want to go through and fix all the lines in our product that are over 80 characters, then we log a bug for it.” - A rule you shouldn’t follow

This solution will never leave us with a beautiful tree. I don’t know of any professional developer who has the time to deal with tickets that are that low priority, and no-one wants to anyway.

If you accept that there is value in making the line / file readable, then you must also accept that there will be change unrelated to any bug fix or feature request. You must also accept, that if those changes get put off for “later” our tree will become a bush, and require so much work that it will be almost unrecognizable the day after the change.

So, how do we balance these two concerns? How do we make readability changes to our code without obscuring the functional changes? My recommendations are as follows.

It should be acceptable for whatever method you’re directly trying to enhance or fix to be refactored for readability. Even if you’re refactoring a 20 line method, for a one line fix. To do otherwise is like having a child make their bed but leave their clothes and toys strewn about the floor. Don’t do a half-assed job. Leave the method better than you found it.
Separate other aesthetic changes out into separate commits. It would be nicer if you could commit the fix, then commit the aesthetic changes, but we frequently need to make the aesthetic changes in order to make the code readable enough to understand it. So, make it readable, but leave it broken. Commit that change. Then fix it and commit that change.

The downside to 2 is that you have to instruct reviewers to review specific commits, rather than the changeset as a whole. So, an alternative is to make the aesthetic changes, create a pull request / request review of your branch to be merged, and then, once that’s done create another pull request for the bug fix. I think it’s a judgement call, and the appropriate choice depends on the situation.

As per usual, if you’ve decent test coverage everything becomes much easier. Code reviewing the aesthetic commits can be just a matter of seeing if the tests pass, and giving it a quick glance to make sure the other developer didn’t go all crazy-pants with their changes, like changing the indentation style to something no-other files use. You don’t have to stop and consider the funcional impact of the changes, because there shouldn’t be any. “Yup, still compiles, still passes the tests, looks more readable. Approved!”

Summary Codebases are like Bonsai Trees. Regular aesthetic changes are as

important a part of keeping your codebase healthy as regular refactorings. Some trivial changes to how you commit your work can elimanate the diffing problems that arise when combining fixes / features with aesthetic changes.

Git push is not what you think

2013-03-13T00:00:00+00:00

tldr;

git’s default configuration with regards to push is potentially very dangerous.
make sure you’ve run git config --global push.default current
There are other options for push.default but make sure you read the docs before setting them.
setting current as your default behavior means no more complaints about setting upstream when pushing.

Perception vs. Reality vis-à-vis git push

When it comes to git push most people think “It pushes my current branch’s updates up to the remote server” but that’s only a small part of what’s happening, and ignorance about the rest can leave you with very upset coworkers. I know, because that’s exactly what happened to me today when I ran git push -f on a coworker’s computer that happened to have the default configuration.

I mistakenly believed that “Hey, git push just pushes the current branch up, so git push -f will just force push this current branch up. I want to wipe out history on this topic branch, so no worries.”

git push

“Updates remote refs using local refs, while sending objects necessary to complete the given refs.”

Note that nothing in that statement is singular: refs… plural.

push (in its default configuration) is effectively the opposite of fetch. Fetch pulls down all the changes from branches you’re tracking. push uploads all your changes to branches you’re tracking. Now, normally this won’t get you in trouble. The branches you’re not currently working are either: older (or current) versions of what’s on the remote repo (in which case nothing happens), or incompatible versions of what’s up there (in which case they get rejected), or possible new commits that can safely be updated. The latter case can be annoying, but it’s rarely that bad, and can be easily backed out.

The problem is when you add -f to push in a default configuration and don’t specify what repo and branch you’re pushing to. This essentially says “Hey remote repo. All these branches I’m tracking? Yeah, make yours match. No. I don’t care that I’m blowing away work. You can take those rejections and shove them where the sun don’t shine.”

The more remote branches you’re tracking, the more damage it can do. And, you’re probably tracking master, which is usually the most important branch in any repo, which means there’s a good chance you’ll be doing some notable damage.

But, if you change your default configuration by running git config --global push.default current then the default behavior stops being “everything I can find a matching name for” and starts being “the current branch” which is what everyone generally assumes is happening anyway. People don’t want, or expect this “reverse fetch” behavior. They expect something akin to “my branch will go up”.

Making this change to push’s default behavior also means that when you do do git push on a brand new branch it won’t complain about you not having set upstream. Instead it will just create a new remote branch for you that matches the current (new) branch.

What to do when you’ve run push -f

In the default configuration?

pray
apologize profusely to everyone who’s working on your repo.
ask very nicely if every could please do a push to all the branches they’re tracking.
thank the deit(y/ies) that someone came up with a distributed version control system where everyone’s got a legitimate copy of the repo.

The only circumstance under which 3 will fail to recover the data is if you did the most recent work that was pushed to a remote branch, then reset or rebased your local copy of it to some prior state, then force pushed before anyone else had a chance to download your changes. Alternately, everyone hates you and refuses to push their copy in order to force you to recreate all the work you blew away. If that’s the case you should probably go look for another team.

Setting the Atomic Clock

2013-02-21T00:00:00+00:00

This morning’s shower brought me an interesting series of thoughts that I thought you might appreciate, and it all started with the simple question of “How do you set The Atomic Clock?”

My first thought was that at some point you have to find some other clock and precisely sync up with it. Then again, they may have said “fuck it” and just had Bob press a button when some other clock flipped over, but then I wondered “How do we know what time it is in the first place?”

Well, our calendar is based on the birth of Jesus, and we claim we know what day that was, and what town that was in. I’m ignoring that the details in the Bible point to a date that’s nowhere near Dec. 25th. Anyway… The first moment of the first day would, I suspect, be associated with the moment at which dawn broke in Jerusalem that morning. Or maybe some more esoteric moment that day. Regardless, let’s stick with dawn for the sake of attempting to figure out what time it is.

So, dawn, in Jerusalem, on day 1. Great we can work forwards from there. But, wait… how do we know how long a year is?

A year is the time it takes for the Earth to travel around the Sun, but how do we know when we are at the same place in the rotation we were exactly one year ago? You could try and position yourself relative to other heavenly bodies “When star X is directly behind the Sun.” But our orbit’s not circular, it’s elliptical. Actually, that’s not true either “The Earth travels around the sun in an elliptical orbit that varies in shape, with time from nearly circular to distinctly elliptical.” Then there’s the fact that every galaxy in the universe is shooting out away from the center of it.

If you ignore the orbital complication and say “ok, we can compensate for orbital weirdness”, how do you verify it with observation? Well you’d look up on… wait a minute. How the fuck do know where the same up is? In order to calculate our point in the Earths solar orbit we have to first be able to look up at the same point on different days. Which means we have to calculate exactly how long a day is, but “The tilt of the earth changes cyclically between 21 3/4 degrees and 24 1/4 degrees. The period of cyclicity is 42,000 years.”. On top of that we’re going around the sun.

So, even if the Earth wasn’t wobbling, and even if I knew exactly how long it took to spin 360° when I looked up the sun would be in a different place because we’ve moved around it. The stars would be in a different place because we’ve moved around the sun, and even if we hadn’t done that, the contents of the universe is still shifting outwards.

Back to figuring out how long a day is. So, we look up. We find something in the universe that we’re not rotating around to mark as a point of reference. We can compensate for the universe’s expansion by observing its Redshift (i think). So, we look up but once 23 hours have passed and we’re ready to start looking straight up again to observe the exact instant we’re lined up with that distant star (or whatever) the planet has wobbled on its own axis and now “up” will be pointed in a different direction than it was exactly one day ago. So, now we have to calculate the the wobble of the planet…

At that point I needed to get out of the shower. But I do know that the wobble’s going to be affected by the fact that we’re on a large ball of rock that’s got a molten core, which means liquid sloshing around in the center (for some definition of “sloshing”). I’ve no clue how one would determine that we even have a liquid center. Maybe by the fact that we’re wobbling?

To summarize:

In order to calculate the correct time you must:

Figure out the speed at which the universe is expanding so that we can use a distant star as a reference point to observationally confirm all the other things we need to figure out.
Figure out the Earth’s eccentric wobble. No clue how to do that. My shower ended. Once you’ve got that you can then figure out how long a day is.
Now store away what you’ve learned about our wobbling rock with a liquid center. We’ll need that.
Once you know how long a day is, and what direction you’re pointing after a given span of time, you can then start observing our orbit by marking the location of the sun relative to you and some distant star. So, now you can figure out how long it takes you to get around the sun. Unfortunately that’s going to change every single year as our orbital path changes.
So observe those changes for enough years to calculate how our orbit changes. Then factor in the effects of a wobbling mass on an orbit.

Once you’ve done all that you can then work backwards to calculate out where the hell the sun was over Jerusalem on December 25th 2013 years ago, and then you can work forwards to figure out what time it is.

You’re still going to have the problem of actually syncing the clock with the math though.

P.S. How was your shower?
P.P.S. Those links? Missouri Department of Natural Resources, Division of Energy. “Global Climate Change: Effect of the Earth’s Orbit”
P.P.P.S. I’m not a Physicist. I just play one on the Internet.

Some useful Vim plugins

2013-02-20T00:00:00+00:00

There are two things that make using vim awesome… no there are about 200,000 but most of them involve adding a few lines to your .vimrc to enable them, or installing a plugin. My .vimrc is just over 300 lines after all these years of use and customization. But, rather than go into all that, I figured some of the vim geeks out there might appreciate a pointer to some of the plugins I use. I’d also be happy to hear suggestions any alternatives to the ones I am running.

I should note that some of these rely on the use of ctags which seems to be a dying art these days. So many people are unaware of the awesomeness contained therein. If you don’t know about ctags, go read this.

Vim Plugins I use

AutoComplPop (acp) Autocomplete popup. It’s a little too aggressive. I need to find something that’s mare accepting of me not wanting what it’s offering.
AutoTag Automatically reruns ctags on a source file when you save it.
bclose Deletes a buffer without closing the window.
bufexplorer allows you to quickly and easily switch between buffers
FuzzyFinder (fuf) allows you to search for a file by name or via ctags… has a tendency to totally hang vim far too often.
indent_guides visually displays indent levels in code. I don’t use it much, but it’s nice to have sometimes.
powerline makes the vim statusline awesome. For extra awesome install the modded font.
rails I have never used this, although I probably should.
Fugitive Install this, if for no other reason than to have awesome Git Blame integration
taglist I never use this but really ought to. There’s a video on it here
gitgutter Puts a little indication of what lines have been added, modified, or deleted in the left hand gutter. Read the FAQ after install if gutter section for non-modified lines looks ugly.

Plugins I’ve installed but should probably get rid of

scratch I’ve never used this.
vcscommand for interacting with version control systems. I should probably kill this as there are better ones that are git specific like…
gitv gitk for vim… I can’t remember using this and I’ve got Kaleidoscope and SourceTree which are awesome so I don’t really need it.
l9 I have no clue why I installed this, and I’m somewhat unclear on what it does (documentation sucks).

Rebasing A Pull Request on GitHub

2013-02-07T00:00:00+00:00

It’s generally good practice to rebase commits on a topic branch into a single commit before merging. It results in a much cleaner commit history, and makes rollbacks easier.

The Question

However, the question was raised: what happens if you…

fix a bug (commit 1)
create a Pull Request
get feedback via the Pull Request
fix the bug fix (commit 2)
rebase those two commits together (new tree-ish)
push that back to GitHub (requires push -f )

The answer is based on understanding that a GitHub pull request has two forms of commenting: * comments on the pull request itself * comments on the commits that the pull request encapsulates. These are the comments made on specific lines in the diff.

The Answer

When you rebase and then force that changed version of history up onto GitHub you are effectively throwing out the old commit (the old tree-ish no longer exists) and replacing it with a new one.

As a result any comments created on the diffs (and thus tied to a specific commit) will be lost, but the comments on the pull request itself will be preserved.

So, the proposed rule of thumb is to simply NOT rebase once you have made a pull request. Instead, just add commits to your topic branch. This will maintain the comment history for future people curious about this piece of code.

Alternate Solutions

There are alternate solutions, of course, but the consequences of them tend to not be worth it.

Rebase Anyway

Decide that you don’t care about maintaining the inline comments, then rebase and push -f back up to the server. The downside here is that at this point someone has already started looking at your code and has likely pulled it down. They will now be forced to blow away their old branch (or at least reset --hard back to a point in its history before your commit), which is a pain and not obvious unless you tell them directly.

Make A Second Topic Branch

You could close the first pull request after having gotten the set of commits to an approvable state, and then open a new pull request from a separate topic branch with all those commits having been rolled up into one. The problem here is that if your new pull request is merged in then git blame will point to the tree-ish of the new one and no-one will ever see the comments on the old pull request unless they search, but when they do they may find that the commits in the original request don’t exist in the codebase (they’re all under the rebased commit).

Git: pushing and pulling from multiple repos

2013-01-31T00:00:00+00:00

Lets assume you’ve already cloned a remote repo and have been working with it. Now, someone has set up a second repo out there for the same codebase, and you’d like to interact with both.

*Please note: The following is based on the assumption that you have write privileges to the second repo, but don’t worry, you do essentially the same thing if you don’t and I’ll cover the differences at the end. *

git remote add <local_name_for_remote_repo>
git checkout -b
<local_name_for_remote_repo>/master
- could be any branch on that repo, not necessarily master

Now, your new local branch is now set up to track (push and pull to/from) the remote repo’s “master” branch. All your pre-existing branches will continue to track whatever they were tracking previously

A practical example:

git remote add repo2  git@github.com:masukomi/kudos.git
git checkout -b repo2_kudos -t repo2/master

Workflow

Let’s say you’ve decided to add a new feature in the kudos repo we just connected to.

Checkout the repo2_kudos branch
git pull to get the latest code from repo2/master
git checkout -b adding_feature_x
- this new topic branch is not yet tracking anything so if you attempt to push or pull it won’t work, because it won’t know what endpoint you want to interact with.
Make your changes on this topic branch.
When you’re ready run git push -u repo2 adding_feature_x
- This will do three things:
  1. It will create a new branch in the repo2 repository named “adding_feature_x”
  2. It will push the contents of your adding_feature_x branch into the corresponding one on repo2.
  3. It will configure things so that your local adding_feature_x branch is now tracking the remote adding_feature_x branch so push and pull will work without additional arguments. You must pass the -u parameter for tracking to be set up here. If you fail to pass the -u parameter at this point then push and pull won’t work without additional parameters. You can, however, set up tracking after the fact with git branch -u repo2/adding_feature_x

Now that you’ve pushed your changes to the new branch on the remote repo you can initiate pull requests from your topic branch into the main branches (if you’re using github) or you can simple tell others about it so that they can interact with it.

Keep in mind that it’s best practice to regularly pull from your main branch into your topic branch (repo2_kudos into adding_feature_x in this case) so as to avoid large merge conflicts. Alternately you could pull from the corresponding main branch on the second repo (repo2/master in this case) since they should be equivalent.

When you don’t have write privileges

If you don’t have write privileges to the second repo things change slightly. The difference is in where you push your new topic branch (adding_feature_x) to. You can either push as a new branch on whatever repo your local repo was cloned from (if you’ve got write privileges to that) or you can create a third remote repo that you control, and push the branch to that. The third repo would be configured in the same way as the second, and you should be able to push to it in the same way we pushed to the second. Git makes it easy to flow code from one repo to another, or through multiple repos.

Never register a domain name with your hosting provider

2012-11-19T00:00:00+00:00

A smart developer I respect recently asked my why I didn’t just register my domain names through my hosting provider. I hoped he was joking, that he knew why this was a horrible idea. He did not, and I know some other smart people who register domains with their hosting providers. Education is needed.

The problem is simple: conflict of interest. Should you ever decide to switch to a different hosting provider it’s in their best interest to prevent you from moving your domain. Why? Because if you point it to some other hosting provider you stop having to pay them.

It doesn’t matter how great your current hosting provider is. It doesn’t matter how great their morals, customer support, or promises are. There is no guarantee they won’t be bought by an ass-hole tomorrow. There’s no guarantee some greedy concepts won’t start to permiate their management.

Your domain name is your online identity, or that of your company. Do you really want to risk the possibility of having it stuck at a company with rapidly declining quality of service or support? Or, maybe a company that starts jacking prices through the roof? If your hosting provider controls your domain name’s registration you are taking this risk.

The solution, however, is simple, and cheap. Transfer your domain to a company that is primarily a registrar today. You don’t need to change your hosting setup, and you don’t need to wait until your current domain regstration is about to expire. When you transfer a domain name you get the current time remaining on your registration plus the year, or more, you choose to pay for at the new registrar.

Currently I’m recommending people go to Hover.com. I don’t get anything for referring you there. They’re just refreshingly easy to use and don’t try and sell you any other services.

Before you say “my hosting company would never prevent me from leaving.” I would remind you of the famous saying

Those who cannot remember the past are condemned to repeat it.

George Santayana

While it may not happen much these days, back when hosting providers started offering registration services this practice of locking customers in was actually quite common. I encountered quite a few people who were forced to abandon their domain name because they wanted to leave a corrupt, or at least bad, hosting company.

There are some measures in place to help prevent this, but you will still have a miserable time getting your domain away from a hosting provider that doesn’t want you to go and tries to thwart your exodus.

If you have any domain names that are currently managed by your hosting provider, please transfer them away ASAP. Most of you won’t have a problem. Some of you will. For those who will, keep in mind that it’s much better to learn your hosting provider is an ass now, then when you actually need to move it at some point down the road.

Update: A response to the comments

The key to this happening is that hosting providers your register a domain name through will frequently set themselves as the administrative contact, because, they would argue, they are the ones administering the domain. You’ve paid them to do so. BUT the administrative contact also holds the keys to the kingdom as it were. It’s their email address that will be involved in the confirmation process during a transfer, and their approval of the transfer that will be required.

It’s absolutely true that the situation has gotten better, and that you’re safer with the large, well established hosting companies, but I’m not trying to spread FUD here. This has happened in the past to many companies and individuals, and I would not be surprised to hear that it’s still happening. Yes, there are higher authorities you can go to, but in the end most people who encounter this problem fight with their evil host for a month or more, and don’t know who that higher authority is, know how to report it to them, or have enough fight left in them to bother starting the whole process over again.

Some helpful folks have mentioned some other great registrars in the comments, and I do have domains at Gandi.net but I’m recommending Hover because they don’t try and sell you anything else and the’ve got a nice uncluttered and easy to figure out UI.

Blog post dates: Precision, or freshness. What's important?

2012-08-24T00:00:00+00:00

Not too long ago I sent out a question. I asked people when, and why, dates were important to them on blog posts. The responses were revealing, both for what they did, and did not contain.

There are some situations where having a date on your blog posts is obviously needed. If you write about anything techy you absolutely need them. I come across tons of sites with perfectly good code examples, that have been obsolete for years. Ditto for how-to’s related to software and operating systems. The other place where dates are critical is anything related to news or politics.

Basically, if you’ve written something that will be used as research material later on, you need a date. Everyone seems to agree on that. But, what about pieces where the passage of time really doesn’t have much of an effect. For example, I wrote a post about a paper-based task management system. Read it today, read it two years from today… it doesn’t impact the usefulness or relevancy of the content at all. And that was the basis of my questioning. If you had a blog filled with posts like that, would there be any real value in including dates on the post?

The responses were overwhelmingly “YES”, but not for the reasons I’d expected.

Freshness and Precision

While they used different words, everyone was in agreement that an indication of freshness was very important to them. Sometimes, it was just to help them know if the blog had been abandoned or not. More often, it was simply to know if the information in the posts was still relevant or not.

What was interesting, was that when pressed for details, almost everyone claimed that a simple freshness indicator (e.g. “three months ago”) wasn’t sufficient. They claimed that they needed to-the-day precision. They claimed that having this level of precision allowed them to compare it to what was happening in their lives at the time; they could look compare the date to their calendar, and see what they were doing then.

These claims seemed… suspect to me. They felt like the type of answers people give when they’re trying to explain something they’ve accepted as true, or necessary, but have never actually thought about. Like, the idea that going out in the cold will lead to catching a cold.* So, I pressed for details. I wanted someone to give me a real world situation when they would look at a date on a blog post and actually do what they claimed. When would they ever compare the date of one to what was going on in their lives at that time, or to what was going on in the world at large at that point. The response, was a resounding silence.

Conclusions

As far as I can tell, if you exclude the types of posts where dates are obviously required (news, politics, content that will become obsolete) no-one actually needs dates with any real specificity, but because everyone believes they need them, their absence makes them uncomfortable, or at the very least, slightly annoyed.

I believe that relative dates are a much closer match to what people need, and much easier to actually use. “Three months ago” translates quickly to “recent, but not too recent”, whereas 5/24/12 requires you to figure out what month 5 is (“let’s see that’s…May”), compare the day of that month (24) to the day of the current month (“Are they the same? Is that one earlier in the month than this one? Does that make it more time or less, and is that enough to bother worrying about?”) before you can actually leverage the information in a meaningful way. But, what we need, and what makes sense logically, isn’t always what we want.

I believe that Kirk’s proposal of “mid-June 2011” is probably the best compromise between usability and need. It’s easier for a human to parse than 6/15/11. It feels much more useful than “over a year ago” or “15 months ago”, even though once you get that far from a dates post your true need for even to-the-month specificity is doubtful.

With that said, the more recent something is, the more precision seems to matter to people. So, you could probably improve on Kirk’s suggestion by a building a system where dates are written with to-the-day specificity for one month, and then switching to “early May 2012” style dates after that.

Closing Thoughts

When pondering dates in blogging, context is everything, and many, if not most, blogs have posts from multiple contexts. Hardly anyone writes a blog where the value of each post is unaffected by time. With that in mind, a blog’s posts should be dated using a method that supplies the maximum level of precision required. If you write a blog of fictional stories, but occasionally comment on news that affects writers you need to-the-day precision, either that, or use a different level of precision on every post, but really, no-one wants to go through that much trouble.

* As it so happens, it’s probable that being exposed to the cold for a while will increase your odds of catching one. Some scientists did an experiment where they had some college students stand in buckets of cold water for a while, and compared which of them got colds soon afterwards. The kids who stood in the cold water got more colds than the kids who didn’t. I just wish I could find the research again…

An offer for Web Geeks

2012-08-17T00:00:00+00:00

I’d like to make a simple offer to the web geeks out there:

One hour of one person’s skills, to make the web a little bit more awesome, and raise $75 for your favorite environmental or medical charity.

Information Architects (the people behind iAWriter) have come up with a really spectacular JavaScript / CSS widget. When you load one of their long articles (like this one) you’ll see a widget in the top right corner that says “Older | Newer” and links you to the previous or next article. No big deal. What’s cool, is that as you gradually scroll through the article the widget disappears, then returns, hovering beside your scroll bubble (wherever that may be in its track); its message now telling you how many minutes it’ll take you to finish the article, at an average reading speed. When you get to the bottom it reverts back to the “Older | Newer” state.

This is something that I’d love to see flourish.

The CSS and JavaScript that powers it isn’t particularly complicated, but I don’t have time at the moment to break it out into a reusable library the way I did with Kudos. So, here’s my offer. For the first person to create a public domain or MIT licensed repository on Github that makes it trivial to add similar functionality to any blog, I’ll donate $75 to the environmental or medical charity of their choice.

I’m thinking a repo with one JavaScript file, one CSS file, an example HTML file, and a README. Simple. Less than an hour’s work for a good front-end geek.

Please note: IA’s stuff is copywritten so, while you can take inspiration for them, you can’t just extract (steal) their code. If you did, the MIT License wouldn’t be valid.

Major MObtvse Update

2012-08-08T00:00:00+00:00

I’ve just merged the experimental branch of MObtvse into master. This represents a huge update and in addition to the feature list below there’s a nice update to the default theme, great new editor, improved Kudos integration on the admin screen, and a number of more subtle improvements. I’m really happy with the progress I’ve been able to make on MObtvse in my free time, and if you’ve been considering it, now is definitely the time to grab it from Github and give it a spin.

It should be noted that while it’s fairly easy to install, and the README attempts to cover all the details, people who’ve never installed a Rails app before, or used the command line, may feel a bit out of their depth.

Sorry non-geeks, we’ve got a little ways to go before installation is totally painless for you, but I’m working on it.

Current Feature List (as of Aug 7, 2012):

Kudos
Live Markdown preview
Drafts
Scheduled posts
Image uploads via S3
Disqus integration
Google Analytics integration
Archives page with Tags and Kudos integration (example here)

Pull requests will be happily accepted. ;)

Want Some Kudos?

2012-05-13T00:00:00+00:00

I think Dustin Kurtis’ idea of “Kudos” is spectacular. A simple tool for people viewing your post to say “I really appreciated this.” You can see it in action in the upper-right corner of every blog post in the Svbtle blogging network.

I really want to bring it to MObtvse, but first I had to figure out how it worked. So, I’ve put together an example implementation of Svbtle-style Kudos that can be incorporated into your blogging software with a few easy changes.

I want to see Kudos everywhere. I want readers like me to have a way to tell great bloggers like you when you’ve written something great. So, go grab the code and add it to your blog system.

UPDATE:

MObtvse now has Kudo support built in. This blog has a different UI implementation from Svbtle’s (and the example repo), but it’s still backed by MObtvse’s new Kudo functionality.

P.S. If you’re looking to add Kudos to your WordPress install checkout the wordpress-svbtle theme from scavone. It’s got them built in, and is where I went to figure out how this works.

P.P.S If you’ve got good CSS chops I’d love to see some improvements on the texts that’s displayed during the animation in the example implementation of Svbtle-style Kudos. Send me a pull request and I’ll happily incorporate it.

Introducing MObtvse

2012-05-01T00:00:00+00:00

Today, I would like to introduce you to MObtvse. It’s a fork of Nate Wienert’s Obtvse, a Markdown based blogging system written with Ruby on Rails. Obtvse is itself, a reverse engineering of the Svbtle blogging platform / network.

The notable differences between MObtvse and Obtvse are that:

MObvtse uses MongoDB via MongoID
MObtvse allows posts to be tagged. Readers can click the tags to see all other posts with the same tag, and Administrators can use the tag cloud to help find specific, or related, posts.
MObtvse uses Haml (partially implemented)
Responsive CSS layout via the Foundation framework.
MObtvse has big plans for the future.

MObtvse doesn’t currently generate static pages, but support for statically generated pages is a top priority. The faster serving helps SEO and allows a single server to handle far greater numbers of readers.

There are a number of planned features like Disqus integration for comments, Mixpanel for stats (it already supports Google Analytics), easy image uploading, and much more.

Reasoning

Why another blogging system?

I know. I know. But, the reasons are pretty simple:

I love blogging with Markdown. It has totally changed my perception of blogging.
Wordpress is a bloated behemoth that can’t withstand serious load without a really good hosting package, and all the Markdown plugins for it suck in one way or another.
Blogging with command line tools like Jekyll and Octopress have serious limitations, frustrations, and complications.

Why a new fork?

I created a totally separate project, instead of working to extend the common Obtvse codebase, because there are a number of changes that I have made, and intend to make, that are far beyond what Nate appears interested in doing with Obtvse, which is fine. I’ll contribute back as I can, but MObtvse is quickly moving to a point where most of the new code simply won’t be compatible with Obtvse’s.

I started with Obtvse because writing something from scratch would have taken too much of my limited time, and Rails’ ecosystem allows me to rapidly introduce many of the features I want without having to code them all from scratch.

Why MongoDB?

Simple, by switching to MongoDB users can now throw an instance of MObtvse up on Heroku and use one of the free MongoDB hosting plans to store their data. These plans generally cap their storage but there are very few blogs out there that would even approach the limits they impose.

Why not stick with Octopress?

I’ve really enjoyed the switch to blogging with Markdown, but Octopress just hasn’t been cutting it for me. I spend almost all my working days staring at terminals, so it’s not that I mind getting down and dirty with the command line, but from the perspective of wanting to write, the process kind-of-sucks:

bundle exec rake new_post[“some title”]
select and copy the new file path it generates
open said file with a text editor (I’ve been using Byword lately).
skip the chance to preview what it would look like in the final version because that would require regenerating the entire site which takes forever.
add it to git.
push that to the server, which will then background the generation of the entire site for me via a git post-receive hook. Most people have to sit around twiddling their thumbs waiting for it to finish generating and then upload via FTP / SCP / Rsync / carrier pigeon.
wait a few minutes for the entire site to be regenerated on the server (even when you just upload one post).
View the post and hope you don’t spot any typos.

And if you want to write a quick blog post from work? No good way to do it without downloading your entire blogging system to your work computer or install and configure JekyllMail. Even then you can’t make an update. Actually… JekyllMail makes blogging with Octopress way easier. Just send an email, with attached photos and placeholder image tags and you’re all good.*

The YAML data structure embedded in the top of my Markdown posts drives me nuts (Octopress inherited this from Jekyll) even though Marked will auto-filter that out on preview for me. This is just wrong. Markdown’s supposed to be about simplicity and ease of writing. Embedding an incomptable, and manually managed raw data structure at the top of each post instead of a centralized metadata solution that would allow the app to make intelligent decisions about what needs to be regenerated and what doesn’t… UGH!

The problem, of course, is that right now Octopress is one of, if not the most feature-packed Markdown specific blogging platforms there is.

How about looks?

MObtvse will always strive to keep its interface clean, simple, and easy to use. The admin section is fairly close to that of Obtvse with the addition of tag support, responsive css, and modifications to better support mobile devices.

The Reading page currently (May 2012) takes many of its cues from weblog.masukomi.org but I expect that to change in the near future.

The Admin Page

Creating or Editing a Post

Reading a post

Like what you see?

Go to MObtvse.com for more details, or head directly over to MObtvse’s Github repo and grab (or fork) a copy. MObtvse is all about integrating kick-ass new features, so your pull-requests will be warmly met.

* Footnote: I’m mostly done with a complete refactoring of JekyllMail which fixes some bugs and makes it way more extensible. I just need to test it before releasing it into the wild.

Using Git Bisect to Crush Your Enemies

2012-04-10T00:00:00+00:00

Using Git Bisect

…to crush your enemies and/or bugs

Or, how to save countless hours and find out where things broke

Git bisect is the most awesome, and most poorly publicized feature of git. It allows git to walk through your branch and quickly find out which commit broke things.

The usage is simple. You point git to a bad commit ( usually the most recent one ) and you point it to a good commit (the most recent one you know of when things were working). So, if, for example, things were working on Tuesday morning, you bring up git log and scroll until you find one from Tuesday morning or maybe late Monday and copy its hash.

$ git bisect start
$ git bisect bad
	# that tells it that the current commit is bad
	# but you could say git bisect bad 48476f7b15022526393e6c4f44f610f552736fdc
$ git bisect good 71f45f2a0302b5fb2331c268ee2bfb3cfde452ab
	# that's the hash of some commit you know to have been good

git will then find the commit mid-way through and wait for you to tell it if it’s good or bad (git bisect good or git bisect bad). If it’s bad it knows the offending commit was earlier. If it’s good it knows the offending commit was later. It continues on in that vein, bisecting the commits with each step until it eventually says something like 54cb86bd35c226dc0df83dc2e2d4c8702fcf2c04 is the first bad commit

When you’re done you run the following to reset your branch to its original state

$ git bisect reset

Full Automation

But it’s even better than that. You can fully automate git bisect. Just kick it off and have it test each branch. All you have to do is write a script that can be executed from the shell and exits with 0 if the commit is good, and a number from 1-124 if it is bad.

You only have to do one thing differently. Once you’ve told it a good and a bad commit it will bisect the commits as usual and give you a system prompt and wait for you to tell it if it is a good or bad commit, but instead of telling it if it’s good or bad, you tell it what it needs to run to test it.

$ git bisect run my_test_script.sh

For example: I’m working on a rails project at the moment and just used this simple rake task which would blow up if the .fixtures method wasn’t loaded correctly:

# lib/tasks/location_test.rake
namespace :location_test do

	desc "tests location.fixtures classload issue"
	task :run => :environment do
		begin 
				Location.first.fixtures
				$stderr.puts "it worked"
		rescue Exception=>e
				$stderr.puts "it died: #{e}"
				exit(1)
		end
		exit(0)
	end
end

So, my run looked like this:

$ git bisect start 
$ git bisect good bebcc378d16d8fe74736c763c144753b2a382933
$ git bisect bad
$ git bisect run bundle exec rake location_test:run

And git bisect proceeded about its merry way until it found the offending commit. Then I ran git bisect reset and went about fixing the bug.

Pro-Tip

The Git Book adds this great note:

Note that the version which git-bisect checks out for you at each point is just a suggestion, and you’re free to try a different version if you think it would be a good idea. For example, occasionally you may land on a commit that broke something unrelated; run

$ git bisect visualize

which will run gitk and label the commit it chose with a marker that says “bisect”. Choose a safe-looking commit nearby, note its commit id, and check it out with:

$ git reset --hard fb47ddb2db...

then test, run “bisect good” or “bisect bad” as appropriate, and continue.

Obviously this won’t work if you’re running a fully automated bisect, and it points to the one gotcha of git bisect. If there are multiple bugs that inhibit your ability to test it ( either manually or via a more automated means) it can leave you with no obvious candidate. Most of the time, this isn’t a problem, especially when you have done your best to chose the most recent “good” commit that you are confident in.

How to Merge Specific Files From Another Branch in Git

2012-02-28T00:00:00+00:00

There are many ways to get specific files from another git branch into your current git branch (overwriting the ones in your current branch), but this is the only method I’ve been able to find to merge those files into your branch en-masse. With this method you’ll be able to pull in any file, or files based on the name of the file or containing folder. If you need to merge files in multiple folders on different subdirectories you can simply rerun step two with a pattern that matches each of the different portions of your tree that you wish to merge.

Please note that this was originally an answer on Stack Overflow, but I felt it could do with a reposting since it’s a commonly asked question and there are lots of poor answers out there that don’t really answer it.

So, without further ado, the simple way, to actually merge specific files from two branches, not just replace specific files with ones from another branch:

Step one: Diff the branches

git diff branch_b > my_patch_file.patch

Creates a patch file of the difference between the current branch and branch_b

Step two: Apply the patch on files matching a pattern

git apply -p1 --include=pattern/matching/the/path/to/file/or/folder

useful notes on the options

You can use * as a wildcard in the include pattern.

Slashes don’t need to be escaped.

Also, you could use –exclude instead and apply it to everything except the files matching the pattern, or reverse the patch with -R

The -p1 option is a holdover from the *unix patch command and the fact that the patch file’s contents prepend each file name with a/ or b/ ( or more depending on how the patch file was generated) which you need to strip so that it can figure out the real file to the path to the file the patch needs to be applied to.

Check out the man page for git-apply for more options.

Step three: there is no step three

Obviously you’d want to commit your changes, but who’s to say you don’t have some other related tweaks you want to do before making your commit.

If you found this useful you might be interested in some of the other git posts here.

The thing about Mock Objects

2012-01-14T00:00:00+00:00

You can tell weather or not someone really “gets” unit testing by asking them one simple question, “Do you use mock objects?” Almost invariably, they will say “no”. Even people who have totally gotten the testing religion. It’s like watching someone pray to a statue of Jesus; totally oblivious to the fact that Jesus himself is standing four feet away reading a book.

This is partially due to the fact that most geeks don’t actually know what a unit test is. They think that testing the methods of a specific class constitutes a unit test, but that’s only part of the story. A unit test test is when you test the methods of a specific class in isolation, and the difference is critical. You know how some people call us “computer scientists”. Yeah, well this is the science part.

Imagine you were creating a new medicine, and wanted to test it out on some cells in a petri dish. The weather was so nice you decided to take your dish outside and enjoy the sunlight while you ran your test. It’s a resounding success! Or… is it? How do you know it’s wasn’t just the UV rays from the sunlight? Or maybe there was something floating in from the nearby blossoms? You don’t, and you can’t. You can’t even say, with any confidence, that the medicine worked. Something worked. Maybe the medicine had a part in it, maybe it didn’t. You can’t tell.

The same thing applies to unit testing. If you’re not testing in isolation there is no way to know that the thing your are testing was actually effective or if it just happened to work in the current environment, and when the test fails there’s no way to know (without digging) if your code was responsible, or if something it relies on has broken and passed on its problems. Mock objects are the best tool we have for addressing this problem.

A Mock What?

You probably already know, but just in case, mock objects…

“…are simulated objects that mimic the behavior of real objects in controlled ways. A programmer typically creates a mock object to test the behavior of some other object, in much the same way that a car designer uses a crash test dummy to simulate the dynamic behavior of a human in vehicle impacts.” - Wikipedia

In practice it is a functional guarantee that a foreign object/method/API, will behave in a specific way regardless of its implementation. You figure out exactly what a call should return and then you encode that in your mock.

Why?

Let’s look at an example. In a banking application a User would have one or more Accounts, and it’s not uncommon for the User and Account objects to each have methods that reference the other. Maybe, the User object has an available_funds() method that adds up all the available funds in each of the Accounts. User is depending on each Account object to correctly calculate the current balance, minus any outstanding charges. Problems arise when there’s a bug in how Account is doing its calculation. Now, the test of your User’s available_funds() method is failing, but you can’t tell if it’s because it was performing a bad calculation or Account was. Even worse, it could be that there’s bad data in the database, or that someone changed your test data but didn’t update the tests. All you know is that the number expected isn’t the one received.

The problem with this setup is that you’re no longer testing your User object when you do this. You’re testing that the code in your User class works if the code in the Account class also works. The typical solution of carefully chosen test data in a database does not solve this problem in any way. The only thing it does is add in another source of potential failures and misdirections, and provides yet another thing to maintain.

Using mock objects allows you to test that the User class works regardless of how broken the Account class may or may not be, with no database installed anywhere, and no separate fake data to maintain. You create a real User object and mock up the calls to Account to return mocked account objects, mocked data, whatever else is appropriate.

If you’re not doing this, you are not unit testing. That’s not meant to deride what you’re doing. I mean that as a literal statement of fact. Anyone, for example, who is using the built in Rails “unit” testing framework with fixtures (or FactoryGirl fixtures) is guilty of this. No test that relies on a separate class, or API, or interfaces with a database in any way is a unit test. It is an integration test (some people call them functional test), and when something goes wrong it is just a matter of time before it misleads you.

Now, integration tests are great. They serve the very useful function of guaranteeing that your code works within the context of the expected ecosystem, but they’re not very good about telling you exactly what has broken.

People will disagree with me on this. They’ll argue that testing with the db and other classes isn’t bad, that they can generally tell what’s broken very quickly. And it’s true that a developer with a good knowledge of the system should be able to figure out what’s causing a test to fail pretty quickly, but until they start digging there’s no way to be 100% confident that the bug is where the unit test claims it is because of all the other things that could affect it.

But that’s not all

The process of creating mocks to support your tests will very frequently reveal unexpectedly tight coupling’s between classes and systems. Because you need to mock up every interaction with a foreign anything, you end up creating a bunch of additional lines in your test. Each one is an expenditure of effort and each one represents a coupling. If the object you were testing wasn’t coupled to anything else you wouldn’t need to mock anything. Adding a few lines of mocking to your test is no big deal, but when you have 20+ lines of mocking in a single test it’s a strong indication that something is wrong in your system. If you find you can’t figure out how to mock something up it either means that you don’t really understand what the thing you need to mock does, or that it suffers from a notably poor design.

Mocking should always be easy, and quick. The only times it isn’t are when you’ve got a poor design, or an insufficient grasp of what you’re attempting to test. These are both very good things to have revealed.

Sometimes, the mocks will reveal bad designs, or excessive coupling’s in an library you didn’t even write. Part of the reason I started writing this article was because of just that type of situation. I was trying to unit test a Rails controller. Typically people just don’t unit test rails controllers. They do functional tests that tie in the controller methods, the database, and the view generation, which again, is useful, but doesn’t do a great job of telling you exactly what is broken when the test fails until you start digging. Anyway, the methods I was testing required an the user viewing them to be logged in, but try as I might I could not find a way to successfully mock it so that it never touched the database.

I assumed that I simply didn’t understand the library, and began looking and source code and googling around. After hours of trying, and failing, to make it work I eventually started Googling from a different angle and discovered that the problem was that it simply wasn’t possible (by any reasonably definition of the word) to test this library with mock objects, even leveraging its test helper classes. The thing was so tightly coupled to the underlying database that it demanded the presence of a test database with appropriately structured fake data to even create a test login.

I had to make the choice of either giving up on any hope of decoupling my tests from the database or throwing out a working authentication system. I care about my users, and I care about being able to guarantee that my sites work for them, so I spent two evenings throwing out that system, learning and implementing a new one, figuring out how to test it without touching the database, and then making all my controller tests work again.

It was worth every minute.

Reasons not to use mocks

The downside to mocking everything is the inverse of not mocking anything. If you don’t do pure unit testing, and don’t mock anything you can never be 100% confident that your objects work in isolation. If you don’t do integration testing and do mock everything you can’t be 100% confident that your objects work in concert with each other.

A combination of unit tests and integration tests is always best. There’s no reason not to build both. It just makes your system more robust and catches more bugs before they are released. But, if I had to choose one only one method, I would choose pure unit tests with mock objects for two reasons:

The objects you test in unit tests constitute the foundation, the building blocks, of your system. If they have problems then everything has problems. I want the maximum possible confidence that each of those building blocks is reliable and well crafted.
The process of mocking up the interactions with other objects gives me a far better understanding of how my system works and frequently reveals design flaws in my code, and the libraries I’ve chosen to use. Exposing and correcting those flaws helps improve the maintainability of my code.

I realize this is not the popular choice, and many developers, would argue that if you’re going to write only one test it should be an integration test. There are good arguments for that, but there are no good arguments for doing either without the other.

A moment for FactoryGirl

This one is specifically for the Ruby geeks, but I’m sure there are similar tools for other languages. FactoryGirl is a system that allows you to easily create named fixtures with the data you need.

“When you invoke a factory, factory_girl uses your definitions to compile a list of attributes that should be assigned to that instance, as well as any associated factories… To create an instance, it calls new without any arguments, assigns each attribute (including associations), and then calls save!”

In other words, it’s a nice wrapper around what is an inherently bad idea for unit tests. In fact, using it guarantees that you are not writing unit tests.

From what I’ve seen FactoryGirl is almost always used as a bad stand-in for mock objects. It is a good solution for your integration tests, where you actually want to test that items can be sent to and retrieved from your database from the other parts of your system, but it should never be called from a unit test because that is precisely what you don’t want happening in that situation.

Chris Parsons has a good writeup with plenty of code examples about FactoryGirl vs mocking that you should check out.

The Takeaway

There is immense value to testing in isolation and the process of creating mock objects to support that testing. If you’re not testing in isolation, you’re not writing unit tests.

If you haven’t been using mocks, take the tiny amount of time it takes to learn how one of the popular mock object frameworks for your language, and start updating your unit tests to be true unit tests. Make sure that every new one is a true unit test, and every time you have to touch an old one take a moment to replace the fixtures with mocks. It will be worth it in the long run.

Resources for Ruby Geeks

In the Ruby world we have a good number of mock object frameworks.

Mocha is probably the best known, and from what I’ve seen most widely used.
Flexmock is another option, but I have essentially zero knowledge about it.
RSpec has a built in mocking library called RSpec-Mocks. I have spoken with a number of RSpec users who prefer to use other mock libraries like Mocha instead of RSpec-Mocks. I’ve never gotten the RSpec religion so I can’t comment one way or the other.
RR (Double Ruby) is a lesser-known option. The syntax is very nice (not nearly as verbose) and it offers some nice features like Proxies and Spies that other frameworks don’t.

Personally I’m a fan of Mocha and RR.

My New Favorite Interview Question

2012-01-09T00:00:00+00:00

I interview a fair number of geeks every year and usually spend my alotted time going over one programming challenge. Lately I’ve been looking for a new one that was simple, but still big enough to give me a glimpse into their thinking. I think I’ve found it.

Why I Like This

I really like this question because:

A good solution involves recursion but you could approach it in multiple ways.
The seemingly simple Array throws a monkey-wrench into the whole thing.
It tests their ability to follow written instructions.
It should give an idea of how willing they are to ask questions.
It’s small, but with 3 separate, but connected, things to handle should be enough to give insight into how they work through things.
In my experience you’ll spend ~40 minutes on this regardless of experience level. I’ve only had one person do it in 5. If folks finish a little early I try and find out what kind of job would make them happy.

The Question

You’re presented with the following YAML file which you need to convert to a useful data structure, but for whatever reason you don’t have a YAML library. You do however, have something that has converted it into a tree of nodes.

Each node has some useful methods you can call:

node.type returns Array, Hash, or String
node.name returns the name of the current node or null/nil if you’re looking at a node in an array.
node.value returns a node, or a String
node.next returns the next node in the tree at the current level or null/nil (assume there are no empty nodes)
node.children will return all the child nodes.

The goal

Using the following YAML as an example, create a method that will take a “node” and return a useful data structure. In the case of the example YAML, that structure would look something like this (Ruby notation)

# en-us is just the name we've assigned to the method's return value.

en-us['date']['formats']['default']   #=> contains "%Y-%m-%d"
# en-us is a hash, with a key 'date', whose value is a hash, 
# ... with the key 'formats' whose value, in turn, is a hash
# ... with the key 'default' whose value is a string.
# So, hash->hash->hash->string

en-us['date']['day_names'][1]       #=> contains "Monday"
# hash->hash->array->string

Notes

Keep in mind that Arrays items can themselves be Hashes, Strings, or other Arrays, and they can be nested, or empty.

Notes for Inerviewer

I present the interviewee with “The Question” and “The Goal” one one side, with the YAML printed on the back. I point out the YAML and tell them that I’ll go over it with them when they reach that point. You can find a PDF version of it here.

I also tell them that “If you ask questions I’ll give you helpful answers. If you don’t ask questions I won’t. Please ask questions.” and let them know that I’m not looking for perfect syntax, optomized code, or anything like that. I only want to see them work through the problem. That is 100% true.

It is very common for people to go down the wrong route before solving this. If someone is stuck on it for a while, I’ll tell them that if they can verbalize what they’re thinking, or ask me questions I can help. Every now and then you’ll get someone who absolutely refuses the help. That’s ok. You don’t want someone who refuses to work with helpful coworkers when they’re stuck.

The test is not to see if they can do it entirely themselves. The test is to see how they’ll work through a problem, and how they’ll leverage the resource of someone who knows more than them.

People who give up are an automatic fail for me. If you can’t work through a problem with a helpful questionner at such a critical time I can’t trust you in a crunch.

Help folks who ask questions. Give them honest answers and leading hints. Don’t give them the answer, obviously, but work with them like the future coworker they might. Help them to figure it out by building on the thoughts in their head.

Note that Junior / Intern level people may not have encountered the concept of recursion. That’s ok. At some point you may have to ask them if they’re familiar with it. Walk them through the idea and see if they can solve it then. Some mid-level people have it in their heads that recurion is evil for “performance reasons” and should be avoided at all costs. Assure them you don’t care about performance.

About the YAML

When they’re ready to look at the YAML I tell them that each line is a “node”. I then tell them I’m going to ask them some really “stupid” questions just to make sure you understand everything. I assure them that it’s not a trick. And then I ask them what the expected return values would be for each node type and make sure they understand that if i give them the - Tuesday node they’d give me the string "Tuesday", day_names becomes an array of seven strings, and so on. I also note that there can be nested data structures and that I don’t want a solution that would work for any tree of data structures, not just this tree.

The Yaml

"en-US":
  date:
    formats:
      default: "%Y-%m-%d"
      short: "%b %d"
      long: "%B %d, %Y"

    day_names:
      - Sunday
      - Monday
      - Tuesday
      - Wednesday
      - Thursday
      - Friday
      - Saturday
  time:
    formats:
      default: "%a, %d %b %Y %H:%M:%S %z"
      short: "%d %b %H:%M"
      long: "%B %d, %Y %H:%M"
    am: "am"
    pm: "pm"

Serving Octopress From a Self-hosted Git Repository

2011-12-19T00:00:00+00:00

There are two good reasons to serve Octopress from a self-hosted git repo.

It provides you with an off-site backup in case your local copies go up in flames.
It gives you an environment where you can integrate secondary scripts and libraries that allow you to do things like e-mail posts to Octopress.
git provides a very efficient, and atomic, means of uploading your files.

Complicating factors:

Not all ISPs have ssh access or the latest version of ruby, and may not have git or Bundlr installed.
Because of the above you may not be able to regenerate html files from markdown on the server.
You need to have a basic familiarity with navigating directories and editing files over ssh.

These instructions are not for people uncomfortable with using the command line in a *nix environment. Of course, if you were uncomfortable with that you probably wouldn’t be using Octopress in the first place.

In order to do this you must have ssh access to your server, and have git installed on it. In order to generate new files on the server you must also have Ruby 1.9.2 (or higher) and Bundlr installed You must also have a basic understanding of how to get around on a Linux command line. This is not uncommon these days, but in order to check just ssh in and type:

which git
which bundle

If they both return a path to the executable you’re all set. If git is installed we can proceed without problem. If git and bundler are installed and ruby is at 1.9.2 we can generate the html on the server. For those not already familiar, which xxxx will print out the path of the executable if it is installed and on your path. If it returns immediately with no output it is either not installed or not on your path.

Next check the ruby version:

ruby --version

Octopress requires a recent version of Ruby to run, so if it says anything lower than 1.9.2 you will not be able to generate files on the server. Talk to your ISP about installing a more recent version of Ruby. You will, however, still be able to use git to transfer files to and from your server, serve from its generated directory, and act as a backup.

There are a couple ways to make this work. You could just host a non-bare repo on the server and set receive.denycurrentbranch to ignore so that you can push to it, but for security reasons, and safer removal of deleted posts it’s better to use a detached work tree.

SSH to your server, and create a new bare git repo somewhere that will not be served to the public.

#on the server
mkdir octopress_blog.git
cd octopress_blog.git
git init --bare
# core.worktree should point to a directory where 
# everything will be checked out into, but not 
# visible to the public. You must create this 
# directory. Git will not create it for you.
git config core.worktree /path/to/staging/directory

Once you’ve done that we need to set up the post-receive hooks. Create a hooks/post-receive file (in your bare repo) and make it look like this.

#!/bin/sh
# checkout the files after they've been pushed here
GIT_WORK_TREE=/path/to/staging/directory git checkout -f

Next, make it executable

chmod 755 hooks/post-receive

Later we’ll tweak hooks/post-receive to auto-generate the site (if you’ve got the pre-requesites on your server) but for now we just want to make sure we can push to it successfully.

Back on your local computer:

cd my_octopress_blog
git remote add live me@example.com:/path/to/octopress_blog.git

Now try pushing to it:

git push live master

It doesn’t have to be the master branch but that’s what most people use so we’re working from that assumption. It should push without problem, but pay careful attention to lines starting with “remote:”. If it does have problems, you should be able to figure out what’s wrong from the error message. At the end of the push you should see a message from remote saying “remote: Checking out files:…”

Back on your server take a look in the directory you pointed core.worktree at and see if your blog has been checked out there. From here on out we have to divide the instructions for people who can generate files on the server, and people who can’t.

To generate server-side or not….

Generating HTML server-side

Generating the html files server-side is good because you can set up things like JekyllMail to allow you to post to your Octopress without having to have a local checkout (great for blogging from work). Another benefit is that your “public” directory doesn’t have to be a subdirectory of your Octopress checkout. You can tweak your _config.yml to generate it anywhere on the server which can make server configuration easier for some. If you do change the destination attribute in your _config.yml on the server to a path that differs from your local version you’ll have to be careful to not commit any local changes to it.

Caveat Emptor: On shared servers there is one gotcha that you’ll have to be aware of. Long-running scripts are frequently auto-killed by the server so that one user doesn’t consume an unfair amount of resources. Until such a time as Octopress is capable of regenerating only the new posts there is the possibility that large blogs with hundreds of posts may exceed this limit and the generate script may be killed before completion.

Generating HTML locally

The only notable advantage to generating locally and pushing to a server-side repo is it works on servers where you have git but can’t run Octopress’ generate script for whatever reason.

Serving from a self-hosted git repo without generating HTML files server-side.

This is the simplest way to go, but it requires one change to the standard Octopress workflow, and the ability to point your server at a specific directory to serve your site from. In your workflow you’ll need to commit everything in your public directory each time your run rake generate, then push to the server. You need to be sure to add & commit any files deleted from your public directory after generation too, or else a deleted post will not get deleted on the server. Typically you can just say git add public each time and it’ll take care of managing the deleted files too.

Since the HTML files are going to be generated in a subdirectory of your Octopress repo (typically called “public”). You’ll need to point your server to the public directory under the directory that your core.worktree option is pointed at. Feel free to rerun the command above that set it if you want to change it to a different place now that you know everything works.

Once you’ve pointed your server to the appropriate directory all you will need to do to update your site and create an offsite backup of the original markdown files (and everything else) is to locally run

git push live master

Serving from a self-hosted git repo and generating HTML files server-side.

You’re going to follow the same steps as serving without generating files except you won’t be committing the “public” directory’s contents.

You’ll also need to add the following to the end of your hooks/post-receive

cd /path/to/staging/directory bundle install bundle exec rake generate

Note, if you’re using RVM or a custom gem path you may need to add those to the start of the hooks/post-receive script so that git will have it in its environment when it runs. For example, on Dreamhost you would have to add something like this to the top (assuming you’d already installed RVM and the current version of Ruby):

[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm" # Load RVM into a shell session *as a function*
GEM_PATH=$GEM_PATH:/home/my_username/.gems
PATH=$PATH:/home/my_username/.gems/bin

Make a commit locally, push it to the new repo, and confirm that your changes have been pushed to the staging directory and that rake generate has done its job and placed the new html files in your “public” directory (or wherever your _config.yml has destination pointing to).

Once you’ve confirmed that that works you can (optionally) change the /path/to/staging/directory to point to somewhere that isn’t just for testing in the hooks/post-receive file and in the core.worktree setting. All that’s left is pointing your server at whatever directory your _config.yaml’s destination is pointed at.

From then on you just run

git push live master

on your local machine to update your blog.

Open Source Wednesday

2011-12-15T00:00:00+00:00

This is a simple idea for every web development company (small or large) that owes its existence to open source software. I’m going to use Ruby on Rails as an example, but this is just as applicable to all of the other frameworks and tools we use daily.

On the first Wednesday of every month all of your developers work on bugs or needed features in one of the frameworks or tools that your company can’t live without.

The thinking is simple, and entirely self-serving:

A typical SAAS (Software As A Service) shop, or web development house is constantly depending on the quality of frameworks like Rails, but these frameworks are riddled with bugs and lack features that would really help.

No-one worth their paycheck would allow their own products to remain in a state like that, and yet we typically do nothing about those things in the frameworks and tools we are utterly dependent upon. Spending one day per month working on tools like these is not an altruistic move. It is a form of self defense. It makes your product stronger, and helps to guarantee that the tools you’ve built your company around will not fall by the wayside.

Improving the quality of the tools we all use helps to keep them popular, and popular tools get maintained and enhanced by more developers.

Contributing to the tools you work with regularly also helps build up your company’s reputation with the developers you most want to hire.

Open Source Wednesday helps your business, your customers, and your ability to attract the best talent.

How exactly is this done

Implementing Open Source Wednesday is simple, and there are a couple ways to approach it. One way is to simply point your developers at the issue tracker for the tool you want to contribute to and have them each choose a bug or feature request they think they can knock out in a day. Get together in the morning, make sure everyone’s working on something different, or intentionally joining forces, and set them loose.

If there’s a particular bug, or missing feature, that’s been impacting your daily work you might want to have some of your developers specifically attack that.

The thing to keep in mind here is to not take bugs that can’t be addressed in a day if you’re only going to work on them once a month.

Open Source Wednesday also serves as a useful change of pace. Letting your developers work on things that interest them outside of the tasks they’re faced with every other day helps to refresh the mind. And working in the different codebase is likely to teach them new techniques that can be applied to your business’ code.

What else?

That’s it. Go. Write something cool.

Why Wednesday?

I was looking for a day of the week that was somehow connected to the idea of “open source” and realized that “open” sounds like Odin who used to be known as Wōden and Wednesday is Odin’s day / Wōden’s day / Oden Dag/. Open -> Odwin -> Odwinsday -> Wednesday.

His name is related to ōðr, meaning “fury, excitation,” “mind,” or “poetry.” And what is code if not mind poetry? Wednesday seemed just about perfect.

Why There?

2011-12-14T00:00:00+00:00

Where you spend your time developing is an important decision for an open source developer. Partly we do it for personal satisfaction, partly we do it to give us a tool we want, but there’s always a part of us that wants others to use and enjoy our work. I want to talk about that, and I want to talk about the frustrations that people who use those open source projects have, but first I need to set the stage. Paul Ruoget (@paulrouget) has been working on a cool live CSS editor for Firefox which should be out in FF 11.

When he proudly mentioned this on Twitter I responded in a less than helpful manner:

That’s great, and something i’d find useful… but right now all i really care about is seeing FF be as fast as Chrome.

I’m happy to say that Christian Heilmann (@codepo8) called me on it.

can you imagine how frustrating such a tweet is when you spent a few month building this great tool?

and later…

The point is that sweeping statements like that don’t help anyone. Your tweet might make people move, building good dev tools not

I think he was trying to say that stupid statements like mine, borne out of frustration from something only tangentially related can lead developers to simply stop contributing at all. He’s totally right, and I’m sorry I said that Paul.

At the same time, I am frustrated, and while it’s not even remotely Paul’s fault this frustration is shared by many many webdevs and it does affect Paul, and everyone else who wants to develop cool things for Firefox.

Firefox is slow. Once upon a time it was the king of the hill. We switched to it because it was soo much faster than anything else, and then tools like Firebug came along and FF became a joy to use for developers.

Somewhere along the line that changed. For reasons I can’t explain FF has become notably slower than the alternatives… and that is the only comparison that matters. A slow browser is fine if there aren’t any speedy ones you could use instead, but Google has put Chrome out there and, by comparison, it feels blazingly fast. I can feel the delay every time I open a new window in Firefox.

I just tested it. It just took nearly five seconds for FF to open a new blank window. I regularly see delays of one to five seconds. It always takes a fraction of a second for Chrome to do it and I’ve always got way more open in Chrome. Honestly, I don’t like Chrome… not when compared to FF. But every single action we perform in our browsers is affected by the underlying speed of that browser. For a web developer with 20+ tabs open with research, test, and reference pages open all day long this is a huge deal.

And that’s how we end up with people like me saying stupid shit to people like Paul who are trying to make our lives better.

But… I am a developer too. I’ve got a FF extension, a useful Greasemonkey script, and a few cool Ubiquity tools that are all sitting abandoned for the simple fact that I am unwilling to add that kind of delay that Firefox demands to thousands of actions I perform every day. No, I’m not saying that everything in FF takes 5 seconds longer, but I am saying that everything in FF feels like it takes longer than the alternative.

I look at what Paul is doing and think that
A) there’s already live css editing in the Web Developer extension.
It’s probably not as good, but it’s there) and Firebug has it to some degree too.
B) Here’s a guy building something cool that developers like me are never going to use because the environment he’s chosen to do it in is slow, and loosing people like me to the competition at an astonishing rate.

So why is he writing it at all?

Firefox developers have been making progress in speeding up FF, but it’s a hard problem, it’s taken them a long time, and it still feels like they’ve got a long way to go before they catch up. Now, I say “feels” here because that’s what’s important. Numbers don’t matter to end users. It’s all about perception.

Consider this little story from the guys at Panic regarding a metting they had with Apple.

Jeff Robbin then asked us a funny question that had obviously been percolating for a while: “Does Audion do any kind of special filtering?” You see, since the beginning of time, press reviews in magazines, websites, etc., had consistently said that Audion simply sounds “better” than SoundJam, without question. It wasn’t until that declaration showed up in a very respectable, high-end British Hi-Fi magazine that people started to take it even more seriously. Audion was always noted for being “richer” or “sweeter”, and we were tremendously proud of our incredible results in this regard.

The only problem was: we didn’t do anything. We have no idea why people heard Audion as sounding better.

I try and put myself in Paul’s shoes and have to wonder if I’d think it was worth it. Sure, he loves FF, and maybe that’s good enough reason to stick with it. But, I just like Firefox, and wish that it was keeping up because I love the tools I used to use with it. If I wanted to build a kick-ass live CSS editor I can’t say that I would be willing to build it for FF right now.

And that’s a question we all have to ask when we’re developing something that’s completely dependent on some larger project beyond our control. Is loving the platform enough? Even when thousands of the people who would benefit from your work are leaving it?

Saying “be faster” is like saying “be better” Of course we are working on performance. But we can’t work only on perf

Paul

He’s right, and yet, at the same time I think he isn’t. Firefox has better tools, better extensions, better extensibility, better everything, but every action you take on it either is slower or feels as if it is. Would Firefox really be hurt in any notable way if it actually did apply every resource at their disposal to speed improvements? I realize this isn’t realistically possible, because not every developer has the skills needed to address the problem, and no-one wants to see the other aspects stagnate, but I think it’s still a valid question.

Christian is right that those of us sitting on the sidelines shouldn’t disparage those who are doing good work, even if we think they’re doing it on a sinking ship. I really hope that FF can come back from setback that Chrome has presented it, but in the meantime where does a developer put their energy and what is the best way to keep them from not contributing at all.

I should have said something like this:

That live CSS editor looks great.

… but how do I tell Paul, and people like him that as much as I appreciate what they’re doing, and as much as I really want to use it, I’m saddened to and frustrated to think that thousands of people like me may never touch it, because of something it’s tied to that he has no control over? Is there any way to do that without discouraging them? Is it better to just shut up and leave them to their dreams?

Maybe…

I’m sorry that this post, and my earlier tweets are probably disparaging to Paul. That’s a shit thing, but even if I had the polite social interaction thing down, it seems like in the end the only way to avoid future occurrences is for people like me to not ask “why?”, or at least not publicly discuss what comes out of those thoughts.

The uncomfortable state of things doesn’t have to be the end of it though. People like Paul love Firefox enough to devote serious amounts of time to making it better. I think it sucks that we are in a situation where asking “Why is he bothering?” has some legitimate value. But maybe if more of us stopped taking the easy route of jumping ship to whatever competing tool is “better” at the moment we would actually take a moment to help bring the tools we “loved” back into fighting shape.

I may not have the skills needed to make FF faster, but I’ve got some decent JavaScript skills that could probably be used to help improve the test JavaScript test suite. If you’re a geek like me who’s gone to the land of Chrome, but lament all the great tools you’ve left behind, consider swinging over to the Firefox Bug Tracker and see if there’s a small task or two you can help out with. If nothing else there’s probably a bunch of bugs in there needing confirmation.

Maybe, if we get off our asses, we can help get Firefox back to being the most awesome browser around. I know I’d love to have my default browser be the same one with all the cool tools.

gVim / MacVim drag command for base64 encoding images

2011-08-25T00:00:00+00:00

The idea is that it can be very useful to base64 encode an image directly into your css file instead of referencing a separate file, but doing so usually involves dropping to the command line, calling openssl, copy-pasting the output, specifying the mime-type, etc… Bret’s Terpstra distilled all of that into one drag-and-drop command for Textmate.The following is simply a generalization and instructions for using the drag and drop in MacVim / GVim

First, take the following and make it into an executable somewhere in your path. That way you can use it from any app that can interact with the command line. I called the fill css_image

#!/bin/bash openssl base64 -in
"$1" | awk -v ext="${1\#\*.}" '{ str1=str1 $0 }END{ print
"background:url(data:image/"ext";base64,"str1");" }'

It’ll generate something like this (only much longer):

background:url(data:image/png;base64,iVBORw0KGgoA...AAAAASUVORK5CYII=);

Now, to call it from MacVim and have the output inserted into your file simply switch to command mode and type

:r ! css_image

leave a trailing space and drag the file into MacVim and hit enter. Dragging it on when in this state will cause the path to the image to be inserted. Enter causes the command to be executed, and the output to be inserted into your file.

JekyllMail: Posting to Jekyll from Email

2011-08-18T00:00:00+00:00

There are a few problems with Jekyll / Octopress though that would, realistically, make me less inclined to use it. First, you need to have your entire blog checked out on whatever box you’re posting from, and that is simply not something I’m willing to do on a work computer, and not something I necessarily can do when on a borrowed computer. Secondly, the user interface sucks. Well, there really isn’t one.

Being able to send e-mail to your blog and have it posted solves both of these problems. I can e-mail from anywhere, and there are tons of e-mail clients with good user interfaces, and there are good Markdown apps on the iOS that will e-mail off your Markdown formatted writings. I wouldn’t be surprised if they handled Textile too, but I wouldn’t know.

So, I did a little hunting around and found a script by Ted Kulp that works with procmail to post a Markdown, Textile, or HTML formatted post to a Jekyll blog. Procmail’s not an option for many of us, but POP3 is. So, I did a little hacking, added POP3 support, made some configuration and anti-spam tweaks and JekyllMail was born.

In addation to importing your post it will also save your image attachments. I heartily encourage folks to fork the repo on Github and send in pull requests.

Cleaning Wordpress of some Malware

2011-04-26T00:00:00+00:00

The other day the Googlebot swung by to check my site for updates and found Malware. Almost immediately, it seemed, people’s browsers were warning them off from my sites. Malware! Bad Things (TM)! There be Dragons here! and so on. Fortunately a friend dropped me a Tweet shortly after it started and thus the hunt began with one clue:

Malicious software is hosted on 1 domain(s), including globalpoweringgathering.com/.

Sadly, that was ALL I had to go on, and when I told the browser I was ok with the risk (Windows malware can’t hurt us Mac folks) I was unable to find any calls to JavaScript to files on my blog. I couldn’t find any that were encoded either. I was stumped. Poking around on masukomi.org, which is just plain HTML files I did find they had all been prepended with an evil script tag, but that was easy enough to replace as there were only a few files. weblog.masukomi.org though… I was stumped. I’m still not sure where exactly it was coming through to the browser, but I did find the culprit.

I should note here that Securi’s free malware checker should definitely be commended. It helped narrow things down and confirm that the site was clean in the end. You should definitely take a minute to plug your site into there and double check that all is ok. Unfortunately as they make their money by fixing malware problems for people, and monitoring for them, they don’t actually tell you how to get rid of any of it. Anyway, at the top of twenty-four of the core Wordpress files (wp-*, index.php, and xmlrpc.php) was a single line 3,209 characters long that started like this

 <?php /**/ eval(base64_decode("aWYoZnVuY3Rpb25fZXhpc3RzKCdvYl9zdGFydCcpJ

For the non-geeks amongst you, that’s a piece of source code that’s been Base64 encoded so you can’t just search for the globalpowirenggathering.com domain, or anything else that a human would recognize. The PHP then decodes it with base64_decode and then executes it with eval. It decodes to a bunch of stuff that looks like it’s targeting Internet Explorer users. Seriously people? Chrome, Firefox 4, or Safari… Anything but Internet Exploder. Geeks: just run the following code on your server’s command line to get a list of infected files. If it doesn’t pull up anything then replace the gibberish in quotes with a section of the Base 64 encoded text from one of your infected files.

grep -rc "aWYoZnVuY3Rpb25fZXhpc3RzKCdvYl9zdGFydCc" * | grep -v ":0"

Non-geeks: If you don’t have a geek, or shell access to your site that you can hand over to a geek you trust then your best bet is to simply delete your WordPress install and reinstall it from scratch. Write down the plugins you’re using first so that you can re-download and reinstall them, and don’t forget that your WordPress install creates an “assets” directory at the top level for the various files (images) you’ve uploaded for your posts. Some viruses do infect images, but the particular malware I’m referring to here does not. So, if you replace the entire WordPress install (just to be safe) be sure you have a copy of your assets folder that you can re-upload.

Reinstalling it from scratch really is the safest way to address the problem and it’s actually pretty easy because WordPress installs are very painless these days. As always, be sure to backup your database first.

Speaking of… backing up your database to a text file is actually a reaaaally good idea in this case because some variants of it inject themselves into all the posts in your wp-posts table. If you back up your db to a text file you can then search it for any of the offending domains, “eval” statements, “unencode” statements, or PHP that you didn’t put there. If they’re in there, you’ve got your work cut out for you, but this does emphasize just how important it is to do regular backups of your database. If it gets infected you can just restore from a recent one.

I was lucky. It was just the PHP files on my blog and a handful of html files on the main site that were infected. Cleaning up one line from twenty some-odd files is not so hard. In fact, once I knew what WordPress files, I just replaced them all with the latests ones from a fresh WordPress download. Before you do that though make sure you’ve grabbed the settings from the wp-config.php file on your server. Don’t worry. Reading the file with a text editor can’t infect you even with the malicious code in it. Once you’ve replaced those files you can hop over to Google’s Webmaster’s Tools and request a review. The review doesn’t happen immediately, but it bumps up your priority and apparently doesn’t count against you if you still fail, but shows that you are taking action to eradicate the stuff.

Also, change your passwords. Change the password on your account with your ISP, and change your password in WordPress. It is possible that this password has been compromised. If you use the same password anywhere else (bad idea) then be sure to change the password on those sites too. Note: other domains that might show up from this malware infection: globalpoweringgathering.com lessthenaminutehandle.com lessthenaseconddeal.com welcometotheglobalisnet.com

Geeks: If you know git, I’d suggest creating a new git repo at the root of your blog. Add and commit all the WP files into it. You just need to remember to update it when you update a plugin, theme, or wp itself which is generally infrequent. Next time the malware strikes (it will) all you’ll have to do is

git reset --hard HEAD

Check for any strange new files with

git status

and delete them.

You won’t have to worry about what malware it is, or what it has or has not infected.

Why you should never indent code with spaces

2010-07-26T00:00:00+00:00

Tabs vs. spaces

It’s a big back-and-forth between developers, with lots of us swearing by the use of spaces, but I’m here to explain to you why everyone who uses spaces instead of tabs for indentation is not only wrong but seriously inconsiderate of their fellow developer.

First though, we must consider what a tab is and what a space character is. A tab is a typographical element specifically designed for the indentation of text to various levels. A space is a typographical element specifically designed to separate individual characters so as to distinguish words from each other.

Next we must consider the individual coders we work with. Each of us has a personal preference about how much indentation looks good, and is most readable to our eyes. Many of us like about four spaces worth per indentation level, but some prefer two or eight. What’s important here is that what’s readable and good looking to you is almost guaranteed to not look good to someone else working in the same codebase.

When you use spaces you are misusing a typographical element to emulate one that already exists. Not only that but you are enforcing your personal aesthetic on your co-contributers. However, if you were to use a tab for each indentation level then everyone else who works in the same codebase would see each level of indentation at a depth that they prefer. Every code editor out there allows you to specify how many spaces of indentation each tab should emulate. People who like it wide can set it to eight. People who like it skinny can set it to two. Everyone gets to see the code indented in a way that is aesthetically pleasing and readable to them.

There’s only one thing worse (in this discussion) than enforcing your indentation aesthetic on your co-contributers, and that’s combining tabs and spaces. When you do that it’s almost guaranteed to look screwed up in other people’s text editors. So turn on invisible characters in your editor (or at least have it display tabs). In summary:

Tabs were invented for indentation, so stop misusing the damn space bar for something it was never intended to do.
What looks good to you doesn’t look good to everyone elese.
What’s easily readable to you isn’t easily readable to everyone else.
Everyone gets to have indentation at a level they want if you use the freaking tool that was designed to indent instead of the one that was designed to separate words.
Turn on invisible characters so you know when some idiot’s mixed tabs and spaces.

git status-report

2010-07-16T00:00:00+00:00

Every week I, and millions of developers like me, have to put together a status report for our bosses, letting them know what we’ve been up to for the previous week. Like most of the developers I’ve encountered I’m always a little unsure of what *exactly* I was working on, and typically I just open up git to see what commits I made, and try to remember any non-code stuff I’ve Thinking it was silly to keep wading through everyone’s commits for the past week to see what I worked on I’ve put it all together in a script (in Ruby) called git status-report, which you can grab from github here.

It outputs a bullet list of all your commits for the past n days, or the commits of one of your fellow contributors. I think I’ll find the latter a great way to keep up on what other folks on my team are working on at the moment.

Usage: git status-report [-e <email>][-d <days>]
Generates a status report for what you worked on for the past n days.
    -a, --all                        Displays output for all contributers sorted by person
    -b, --branches <branch name>     Limit results to work found in the specified branch(es). 
                                         Multiple branches can be comma separated
    -c, --current                    Generates status report for the current repo only
    -d, --days <num days>            The number of days to generate the report for. Defaults to past 6 days (one week)
    -e, --email <email address>      The email address of the committer to generate a report for. 
                                         Defaults to your "git config user.email" address. 
                                         Multiple email addresses can be comma separated.
    -f, --files                      Display a list of files involved in the commit(s)
    -h, --help                       Displays this message
    -r, --reverse                    Displays output in reverse chronological order
    -s, --simple                     Simpler output. Doesn't include the dates
    -v, --verbose                    Verbose output includes details on each commit

Please don’t hesitate to fork the repo on github and make improvements. I’ll happily merge them in to the main one. git-status-report supports will check for commits in all the branches and supports multiple repositories. Check the README for full details.

Git Rebase: why, and when, you'd use it.

2010-05-08T00:00:00+00:00

Rebase is one of the most powerful tools in Git’s arsenal, but it can trip up people coming from centralized version control systems. This is just a quick example of why, and when, you’d want to use it.

Let’s say we’ve got a team of three developers. Monday morning they all come in, Bob makes a quick commit, and shares it with everyone. They all do a git pull and suck it into their repos.

Tuesday Mary does some work, but Bob and Tom are taking the day off and don’t pull in her changes yet.

Wednesday Tom makes a change and everyone pulls it into their repos.

Now depending on what changes Mary has made Bob and Tom may be able to pull them in without problem. Then again it could result in a conflict. But Mary’s a pro who thinks of her teammates, so she’s going to rebase her repository and reorder the commits so that her change is on top of Tom’s. Now, if Mary’s changes were such that they’d cause a conflict when Bob and Tom pulled them in then they’ll cause a conflict for Mary when she rebases, but they’re Mary’s changes so she’s the one most capable of resolving those quickly. Also, if Mary takes care of it in the rebasing then no-one else is going to have to deal with the conflict. It’s much better to have the person who wrote the change resolve it than have everyone else on the team have to deal with it.

So, Mary reorders her commits with git rebase, and if there were any conflicts she resolves them. When the others pull down her changes it’ll look like she wrote them on top of the changes they already had.

Another reason Mary might want to rebase is if she made a bunch of partial commits along the way. She might do this to give herself multiple roll-back points, or maybe she committed every time she finished working on one of the files related to the current feature she’s implementing, just so she wouldn’t have to worry about loosing anything. Now, she wouldn’t want to share those partial commits because they’d probably break something in other people’s system. So, when she finishes with her work, she uses rebase to squash the partial commits into one nice clean commit and then tells people to pull that, or maybe pushes it to some common repo they all push changes to.

Rebase is an incredibly powerful tool, but there’s one thing you must remember to never do and that’s rebase commits that have already been shared with others. For example: Git wouldn’t prevent Mary from rebasing the “Bob - Monday” commit up after all the other commits, or merge it into the commit she made on Tuesday, but doing so is guaranteed to cause problems for anyone who tries to pull from her. So rebase your own work before you set it free, but once others have it you can’t touch it.

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

If you found this useful you might be interested in some of the other git posts here.

Why tinyurl.com and its cousins are a blight upon the internet

2009-12-27T00:00:00+00:00

Every web page on the internet has an URL that is a unique address (that’s why it goes in the “address” bar), and in the beginning everyone used that. But early e-mail clients kinda sucked, and some of the current ones still do, and those addresses were so long they’d wrap, or had some funky characters in them that the e-mail client wasn’t expecting, and so it’d break the URL in such a way that you’d have to copy and paste both parts of it into the address bar instead of just clicking on it.

This kinda sucked.

Then Twitter came along and said “You only get 140 characters because that’s all that’ll fit in an SMS”. And, trying to write a useful little post in under 140 characters with some big-ass url eating up a third of them really sucked.

So, people turned to tinyurl.com and it’s cousins. Because they promised short urls that saved you characters and would always be clickable in crappy email clients so long as you ignore the fine print. Fine Print? What fine print? Well, imagine, if you will, that an URL is a bridge to a distant place. A normal URL works like a normal bridge, but tinyurls…

When you cross a normal bridge you know where you’re going to end up. I never have to worry about crossing the Mississippi and finding myself in Nigeria. With a tinyurl bridge, you just might. There’s no way to tell where it goes before you get there. KinkyPornThatWillGetYouFired.com ? Could be. Oh, and the bridge doesn’t go straight to the mystery land. No, all tinyurl bridges curve through the country of tinyurl. So, if they’re having a power outage, or an earthquake, you’re not getting there today. If they experience an economic catastrophe and cease to be then no tinyurl bridge that’s ever been built will ever make it past the once powerful, but now non-existant, country of tinyurl.com and you won’t be able to get to the desired location manually because there’s no way to know where that bridge ended up. Then, there’s the problem of Google (and its cousins). Google watches every bridge that’s built, and notes where it goes. The more bridges people build to a place, the more important that place must be. But guess what? When you use a tinyurl bridge no-one can see where that bridge goes. So that place you want to spread the word about? You’re not helping it. If anyone is getting any Google-juice out of the process it’s Tinyurl.com, but I suspect the Google geeks know better than to count those urls.

Then there’s the question of longevity. Not only shouldn’t you count on tinyurl.com and it’s cousins staying around, you should count on them going away. Almost all of them are totally free services with no advertising that just cost their owners money. The more you use it, the more bandwidth they consume, and more disk space they need. No, you should definitely count on these sites going away. It’s just a matter of time. Sure, as long as there’s a need another one will probably pop up to take it’s place, but all the millions of urls that used the old one will break when it goes away.

But, what about those old fashioned long urls? When I give someone the url to this page https://weblog.masukomi.org/2009/12/27/why-tinyurl-co…n-the-internet It not only gets them to the post but it also gives them some additional information. They can tell for example that it’s from my blog ( weblog.masukomi.org ) and not some porn site. In my urls (and many other blog URLs) there’s also an indication of when it was written ( 2009/12/27 ). This is especially useful when you need current information not something that was written five years ago. You can also tell if it’s something you’ve already seen. And last, but not least, the end of the URL ( why-tinyurl-co…n-the-internet ) gives you a hint at what the article’s about.

But what about Twitter?

Yeah, I hear you. It’s easy enough to stop using URL shorteners in everything else, especially since modern e-mail clients are generally pretty good about not breaking up urls, but you need to include URLs in your tweets from time-to-time. There are two simple simple solutions and both are really easy to implement. The only problem is that Twitter has to do it.

It’s probably safe to say that Twitter’s not going to give up the 140 character limit even when all phones have gone beyond that limit for texting. Those 140 characters have become part of Twitter’s ethos. If the limit was extended much it would transform what what Twitter is and how it’s used. So, what are these solutions?

The best would be if Twitter were to set up their own tinyurl clone. [Update: they have now] There are plenty of free tinyurl implementations out there that are easy to install. You’d still have the downside of not being able to tell where the url lead to, and you wouldn’t be able to get there if Twitter was down, and if twitter went out of business they’d all stop working BUT, except for not seeing where you’re going these are all ok because if Twitter was down you wouldn’t see the tinyurl in the first place, and if Twitter went out of business the site would go away, and take all those now non-functional links with it.

If Twitter were to simply not count any characters contained in an url it would solve the problem on the web, and only require a minor code change and a simple database update. It has the downside that older phones would have trouble with the extra length and would need an url shortener, of course, if your phone is so old it can’t handle more than 140 characters then odds are you’re not going to be clicking on any links that come in a Tweet anyway.

While neither of these are hard they do both have a financial impact on a company that seems incapable of figuring out how to make any money. Storing those few extra characters may not seem like much but I assure you that they add up, and the servers and hard drives required to make them keep working under a Twitter level of load (and nigh-constant hard drive failure) would not be cheap. But, there’s a good reason for Twitter to go through the effort of fixing these. Over time the URL shortening sites will die out, and as they do millions of links in old tweets will start to break, and that makes twitter look bad. How much will it suck when most all of the links in old tweets stop working and people stop bothering or wanting to click on links in tweets because they’re so frequently broken?

Summary:

In summary, Tinyurl.com and it’s cousins are bad because:

You can’t see where the URL takes you
The URL won’t work if the tinyurl is down temporarily
The URL will break permanently when (not if) tinyurl ceases to be.
It takes longer to get to the site because you have to first detour through the Tinyurl servers (literally adding thousands, sometimes tens of thousands, of miles to every request).
You’re not helping promote the page you’re trying to spread the word about because Google can’t tell what you’re linking to.

Please note that it’s not tinyurl.com that’s specifically the problem. They’re just the most popular one. ALL of the URL shortening sites have these same problems.

Localization for Struts Freemarker users

2009-03-12T00:00:00+00:00

Because it took me freaking forever to find instructions on how to do this…

You do NOT need a message-resource tag in your struts configuration files. Those are outdated instructions for old versions of Struts.You do not need to edit ANY xml at all.
Your Action needs to implement Freemarker’s TemplateMethodModel interface
You need a package.properties file (the default locale) and a then another one for each other locale / language you want to support (ex. package_en_US.properties). These should be located in the same package as the Struts action that will be needing them. You can also do ClassName.properties if you want to tie some to a particular class.
Add the code snippet below to your Action. There’s a backup here if it’s borked below.

Insert localized text into the page with a call like this ${text(“property.name”)} Struts will take care of figuring out the default language from the user’s browser. If you have a better way of determining this there is documentation out there on how to programatically specify the locale you want to be used

public class MyAction extends ActionSupport implements TemplateMethodModel { /*Your functionality here */

 /**
 * when you need to display a localized piece of text in
 * the ftl just use  ${text("some.property.name")}
 * there should be a package.properties or ClassName.properties
 * file in the same source package directory as the Action that
 * will be using them. There *is* a way to set a global properties file
 * but I don't know the details.
 * Obviously the text for other languages would be specified in files like
 * package_en_US.properties
 * @return an implementation of TemplateMethodModel which takes a
 * one parameter method, the parameter is the name of the localized property
 * you want to access.
 */
public TemplateMethodModel getText(){
    return this;
}
public TemplateModel exec(List args) throws TemplateModelException {
    if (args.size() != 1) {
        throw new TemplateModelException("Wrong arguments");
    }
    return new SimpleScalar(getText((String)args.get(0)));
}

}

ecgn adds the following notes about actions:
There has to be an action class involved in the Struts/Freemarker action.

Example – the following simple page view won’t work:

<action name=”ShowLogin”>
   <result type=”freemarker”>Login.ftl</result>
</action>

To get it to work, I have to insert a “dummy” action into the process:

<action name=”ShowLogin” class=”actions.DummyAction”>
<result type=”freemarker”>Login.ftl</result>
</action>

Keywords / Google Food: localization, internationalization, i18n, freemarker, struts

Why your tiered password scheme is flawed, and what to do about it.

2009-02-27T00:00:00+00:00

First, let me explain what I mean by “tiered password scheme”. Many perfectly smart people I know have one strong password they use for one or two online banking type sites. They’ll then have a “medium security” password they use on sites that kind of important to them (maybe those sites have their credit card info stored), but not critical to day to day stuff. Then they’ll have one or two passwords they use on all the other sites like Twitter, Yahoo!, Facebook, GMail, etc.

Obviously they’re being relatively careful about the important stuff and that’s good, but the flaw in this system is in the perception of pain. People think “wow, it would massively suck if my bank account password got out, but it’s not such a big deal if my Twitter password gets compromised. I can always make another Twitter account.” Or, they’ll say “Why would anyone care about hacking my account? There’s nothing special about it.” And while there’s nothing inaccurate about those thoughts, they’re also totally missing the point.

The thing is that many of those low value sites aren’t even remotely careful with your data. They’ll store your passwords unencrypted on their servers. And while people are unlikely to try and go after your account in particular, it’s not unthinkable that someone might try to hack Twitter (just an example), or even better, one of the myriad tools that you have to enter your Twitter username and password into and are much more likely to be insecure.

Now, they’ve got a list of thousands, or maybe millions, of usernames and passwords and because so many people use a tiered password scheme (or worse, only one password) those passwords are going to work all over the place. Now, most people I talk to say “I can just make another Twitter account.” but It’s not just *that* account. How many hundreds of sites have you used that low value password at? Do you really want to go and change it on all of them? And how would you even know you need to change it on all of them. Considering how many you’ve probably used it at, it’s a safe bet that someone could be using your name and spamming people, or linking to virus ridden sites, or whatever from your account and you might not find out about it for months, if ever. Not because it’s your account, but because your username and password just happened to be in the list of thousands they stole from some site. It is simply not realistic to claim that, if you somehow knew the password had, or might have been compromised, you are going to change it on all the sites you’ve used it at. And worse, if any of your accounts that got compromised, and probably killed by administrators, were ones where you’d made friends, built up lists of contacts, etc., then you’re going to be experiencing some serious pain and frustration when it comes to figuring out how to contact all those people, telling them that no, that wasn’t really you trying to sell them male enhancement pills, and that you’ve got a new account now…. Gods forbid this happens to your e-mail account.

So, what are you supposed to do? Well, the “simple” answer is that you’re going to have to start using a different password on every site, and, because you’re human, you’re going to have to use a password manager to keep track of them all. There are a ton of password managers out there and while they’ll all get the job done securely, the usability of these apps varies wildly. Typically the process is: load a web page that needs a password, launch your password manager (if it isn’t running already), find the entry for this site in a list, copy the password to your clipboard, switch back to the browser, select the password field, and paste in the password. That’s six steps, two applications, and possible wait time while one of them launches, just to enter a password!

“But Kay,” you say, “my browser remembers my passwords.” And that’s great. It’s going to save you from having to do this all the time, but sooner or later your browser’s cache is going to get emptied, and you’re going to have to go through the process again. Even worse, what if you’re at someone else’s computer?� The latter is a strong argument for having a password manager on your smartphone, but the usability is even worse when you’re talking about an entirely different device that you can’t copy the password from to paste into your browser.

So, again, what are you supposed to do? Well, first off, you need a password manager that’s not going to make you leave your browser. This means we’re talking about browser extensions, or maybe a web based password manager. And while there are a ton of choices, some of them are wildly complicated and most of them fail when it comes to usability. If you’re going to stick with this, and protect your identity and passwords the way you should it is absolutely imperative that you use a tool that makes it easy, easy, easy, to get your passwords in there. The act of retrieving your password and entering it into the site you want can’t be frustrating or annoying to you. [Update: A number of people have felt that I’m trying to promote Passpack below. Passpack just happens to be one of the few services that meets all of my personal criteria, but there are other services out there and I encourage you to explore them. Furthermore, web based services aren’t right for everyone. Some of us prefer Firefox Extensions, or apps that live on our phones.� Really, I just want you to go find a tool that will help you to practice good password hygene, and am offering up what I’ve discovered whilst searching for tools for me.]

So far I’ve only found one that I’m really willing to recommend: Passpack. Unlike most web-based password managers it can work offline, if you have Google Gears installed. It’s got a ton of features (probably far more than you need), and while it’s interface can be a bit overwhelming with all it’s options and it’s big grid of icons, it appears relatively easy to use, and most of the time you’re just going to click its bookmarklet when you want to log in to some site you’ve stored the password for. Once you’ve saved a password for a site it’s two clicks the next time you need to retrieve your password and log in. I definitely recommend checking out their video tour. [Update: the icon grid, can be turned off, and will be off by default in the next version.]

When it comes to browser extensions I simply haven’t seen any that get the usability right. Some have daunting management screens, or just bad tools for initiating the action of getting the password and almost all of them finish by copying the password to the clipboard and making you paste it in to the form, which is just brain-dead considering that extensions have access to essentially every feature of the browser. Many (most?) of them are going to leave you screwed if you need to access a site from someone else’s computer. The same can be said for desktop apps. The things the extensions do typically do right is generate good passwords for you, store them securely, and save you from having to open another app. But, all of the ones I’ve seen simply aren’t painless enough when it comes to usability, or management, or both. Actually, there was one exception that got the usability right, but ultimately encouraged users to go with something resembling a tiered password scheme.

Is that even running?

2009-02-20T00:00:00+00:00

It’s not uncommon for me to wonder if some app is running on my linux or OS X box, and while I could pipe together ps and a couple greps it felt silly to keep doing it after a while. So, I applied my admittedly limited bash skills and came up with the following script which I throw that in an executable called “got”. Now I can just type “got tomcat?” (the question-mark is optional). If anything is running with “tomcat” in it’s command it’ll give me the skinny on it. Otherwise it’ll let me know it wasn’t found.

Then I threw it up on github and others helped improve it.

So, if the idea of being able to ask your system if it’s “got tomcat?” (or anything else) sounds good to then Check out the code on github and fork it if you think of a way to improve it.

Get any URL onto your phone with a QR Code Bookmarklet

2009-02-13T00:00:00+00:00

If you’re like me you find yourself sitting at your computer and need to go away, but there’s some page you’d like to read, or continue reading, on your phone. Well, if you’ve got an Android Phone or essentially any phone in Japan you can just use your phone to scan in a QR Barcode from your computer screen and then open the url on your phone. I know for a fact that there are other phones in the US that can read QR Code, but you’ll have to Google around to see if your phone is one of them. Sound good? Then go here, see the screenshots, grab the bookmarklet, and let me know what you think. P.S. If you’re interested in seeing what’s going on with 2D barcodes as we start to catch up to Japan you might be interested in checking out the 2d code news site.

Developershare [definition]

2009-01-13T00:00:00+00:00

Developershare: adj. The percentage or proportion of the total available pool of developers that is coding for a particular product or platform.

Example: Regardless of how good the Palm Pre is, Palm will be hard-pressed to steal any of iPhone’s developershare.

4 1/2 Killer Mac Apps

2009-01-08T00:00:00+00:00

Yesterday I was discussing the fact that I need a new laptop and how much I wanted to get an HP Mini 1000 (cheap, ultra light, good manufacturer), but couldn’t because of those damn independent Mac developers. They keep making incredible apps I simply won’t give up. Unsurprisingly, he asked me what my killer apps were for the Mac, and I thought you might be interested too. But, before I start the list, I just have to give a major shout-out to the indie developers for OS X. You guys make the most creative, useful, and beautiful software on any platform. If it weren’t for you I would have given up my Mac years ago.

Scrivener Scrivener is a writer’s Swiss Army Knife from the future. I say “from the future” because unlike a modern Swiss Army Knife, and MS Word, it does’n’t throw every possible option in front of you. No, it keeps them subtly out of view, waiting until you need them. Unfortunately, this means that unless you actually walk through the tutorial it comes with, you’ll only notice the superficial goodness, and not the myriad Pieces Of Awesome that are hidden in plain sight.It’s filled with tools to help you organize your writing, reference material, thoughts, etc. Honestly, I can’t even begin to encompass how amazing this tool is for writers. The only thing I wish it had was bibliography support.
OmniGraffle I admit it. I like making flow charts, and nothing makes flow-charting as fun, easy, or beautiful as OmniGraffle. I constantly get comments on the look of my OmniGraffle flow charts. It’s also great for web mock-ups (just grab some of the free stencils). Yeah, I’m sure Visio has more features, but Visio is ugly, no fun to use, and I really don’t give a damn if my flow charting software can’t map my LAN.
DeliciousLibrary Nothing makes cataloging your media easier or more beautiful than DeliciousLibrary. There are lots of competing apps on every platform, but nothing is even remotely as nice to look at, or as easy to use, which is sad, because the company behind DL sucks when it comes to adding new features**, and it leaves out some obvious ones, like you can lend out books, but not borrow them. The HTML export is beautiful, (there is another prettier, but less functional output too) but gives you very little control over the end result. But, even with all its little annoyances it’s still twice as good as anything else out there. And, they’ve just released an iPhone app that syncs with your desktop, which is probably cool, but does me no good because I have an Android phone.
LineForm
[Update] Lineform is no-longer updated and prone to crashing in the current versions of OS X. I don’t have a recommended replacement..
For the average person I wouldn’t normally classify LineForm as a killer app. It’s a good vector art package, but it’s simple. On the other hand, it’s $80 not $600 like Illustrator*. In spite of it’s simplicity I’ve come to really enjoy this app, and because I use it five days a week to create things like this, switching to anything else that could produce the same quality of results would be painful, or ridiculously overpriced. So, for me it’s a de-facto killer app. I did a write-up before about why I chose LineForm.
VoodooPad Ok, technically, VoodooPad isn’t a “killer app” for me. It alone wouldn’t keep me from leaving the Mac, but it is freaking awesome! Like many of these apps, the thing that makes it awesome is it’s subtle power. On the surface it’s just a personal wiki. But, it is so well implemented that it just makes you want to use it more and more. In my opinion none of the competiton comes anywhere close to VoodooPad. It’s great for personal knowedge bases, building up research, and 42,001 other uses.

All of these apps have free trials. If any of them sound interesting to you, please download them and give them a try. They are each worth every penny. * Yes, I could pirate Illustrator, but as a software developer it’s important to me to only use software that’s free or has been legitimately paid for. I wouldn’t want anyone pirating software I wrote so I won’t do it to them. And yes, Inkscape is arguably more powerful than Lineform, and I like it, but I can’t use Inkscape on a Mac because OS X’s X server doesn’t support pressure sensitivity, and it tends to slow everything down, which screws up long flowing lines. ** They have updated Delicious Library with new features since it was originally released, and they are quite good, but it took years to actually get them the last time around. And it still can’t @$#% let me borrow a book! Also, there was lots of frustration on their forums and very little interaction from the company when the last version was months late in coming out. So, the app rocks but I’m not thrilled with the support or the co.

On coding for fun

2008-12-14T00:00:00+00:00

I love programming. I really do. It’s one of the few things that really gets my brain buzzing. In my twenties I’d go to work, program my ass off, then come home and repeat. Or, when I worked for myself, I’d just not stop. But, as I make my way through my thirties I’ve found that most days I come home and simply don’t want to look at code anymore.

I code the back ends to web sites you see, and I’ve been working on web dev. for over twelve years now, and honestly, there’s not a lot of innovation going on in that area. Every now and then someone comes out with a sweet new framework like Rails, but after the initial fun of learning it, and trying to convince everyone you know to use it too, you realize that you’re almost always doing the same damn thing: taking form data and shoving it into a database, then taking crap in a database and shoving it to the screen. The only difference is which crap you’re shoving around, and where it happens to be stored.

It seems that the mid-life crisis comes a bit early to coders, as a number of the ones I’ve talked to have found themselves feeling a bit ground down as they start in on their thirties, wondering if maybe they shouldn’t open a greenhouse or do something else equally removed from the world of code. I don’t blame them. I’ve felt the same way, but every now and then the universe reminds me that coding can be an absolute blast. That no matter how much I might enjoy some other simpler profession, there’s nothing that’d get my brain buzzing like this.

John Bender has come up with five simple rules to keep programming fun. And while they’re all good, I, like you, look at it a bit differently. So here are my rules for keeping programming fun.

1. Remember that nothing is more important than living a full life.
When I worked at iPhrase there was this one guy who worked on the back end. He probably averaged 16+ hour days because he was really into his piece of the project. It really got his brain going. But, eventually, he, like everyone else, got laid off. And I always remember him, because when he looks back on those years of his life, there’s nothing there. There was not time for a social life, and IBM now owns (and has probably mothballed) all of the code he wrote. Code is guaranteed to become obsolete. Pretty much anything you work on will be abandoned sooner or later. Don’t get so wrapped up in your code that you forget to spend time with the people who matter to you. There will always be more code to write. That weekend getaway with your friends is a little once-in-a-lifetime experience you can’t replicate. Sometimes, it’s even simpler than that. When you’re sitting there coding away and your child, wife, or even pet comes up and asks you to play, or spend time with them, do it. Finish that function, hit save, and walk away from the computer. Do not make them wait more than a few minutes.

2) Small, self-useful, projects.
John talks about small triumphs, and while those are very important, I’d take it even further. I get the most joy out of small projects. Writing some cool little app for my phone that I’ll use every day, or some helpful little tool. While John’s right that “You’re rarely going to write the next big thing, or anything really meaningful in a single day’s work.” To me it’s so self evident as to be a non issue. Not only that, if you attempt to write the “next big thing” or anything “really meaningful” it’s going to be work, not play. And I want to look back and think that ,while I may have been spending some of my free time coding, I was coding something fun that put a smile on my face. As such I try and work on very small projects, or very focused features of someone else’s project. Making each feature it’s own little one day project.

Let me give you an example. I’ve been hacking on a fork of Photostream because I thought it was a great proof of concept, but one that could use a lot of improvements. For example, it doesn’t support Flickr contacts. So, while I can add people in manually, I can’t say “load in all the people I’m following on Flickr.” So, one night’s project was to make it support adding a person, their contacts, or them and their contacts. A small accomplishment, like John says. But, I like to, whenever possible, make it so that I could walk away, after a nights work, and have something that would be useful as is, without ever touching it again.

If I wasn’t forking the whole project I would have just submitted that change and felt I’d accomplished something useful, that others could benefit from. But, as I make my way through the day at work I think “I wish it would…” and every couple nights I’ll boot up Eclipse and hack in another feature. But, if my dog decides I’ve been spending too much time at the computer, and too little time with him, I remember rule number one, finish the current thought, or leave myself enough comments to pick it up tomorrow, and go play with him. But, I’ve found that it’s not just about making small goals.

Time and again I’ve found myself getting the most enjoyment out of the smallest projects. A tiny Flickr app, an app that hollers if your servers go down, something that graphs the lines of code per package in a Java project, a distributed bug tracking system, or a library that’ll save me tons of frustration on other projects.

These won’t change the world. They aren’t “really meaningful” but they’ve all been useful to me, and because they’re never so large as to have any chance of becoming work, they’re always lots of fun. Instead of hacking the next great foo, consider adding that feature you always wished that app you use every day had. It doesn’t matter if you never contribute another thing to that project. It doesn’t even matter if you release your code. What matters is that it’s a fun little challenge that you’ll appreciate the results of afterwards, and hopefully learn something interesting along the way.

3) It’s all about you baby.
Never code for anyone else in your free time. If you do, it’s work. Rick Miller left a great example of this in the comments on my art-blog. He said,

“I know from experience that something you love doing can become a pain in the ass when you feel like you have to do it. For a while, I was making flutes out of any kind of pipe, hose or hollow plant stalks that I could get my hands on. I used to carry them with me and play them when I had a few spare moments. Sometimes when someone would make a positive comment or if a child was curious about the flute I would hand it to them and say, “It’s yours.” Sometimes I’d give them away just on a whim. Once I even gave one out through my car window to a girl in another car on the freeway at 60MPH. Then I agreed to make a flute for a Spanish fellow for $20. My flutemaking stopped. When I finally forced myself to make the damned flute, it was nothing to be proud of. I was glad that my bank couldn’t figure out how to draw on a check from the boonies of Galacia because I’d have felt bad if the guy had actually paid for such a piece of crap.”

You’ve never hear of a child “playing for someone else”. Why? Because it’s simply not possible to play for someone else. You can play by yourself, or with others, but never for others.

4) Prioritization.

John says

“Write whatever you need to put a useful piece of software out, then add on the frills later. This will let you release earlier, possibly get feedback and, if anyone is interested in your project, maybe even get help.”

And while that’s a good plan, it’s a bit too close to work for me. I prioritize my personal coding like this:

It’s only when I’m about to release a new project that I even begin to consider what anyone else wants, and even then it’s pretty minimal. Because it doesn’t matter to me if you use my code or not. I had fun writing it and that’s the only thing that’s important. However, I’d feel bad if my code caused you misery or frustration, so, before the release of a new project I make sure that I’ve got all the basics covered.

For example, my Photostream fork isn’t released yet because if you tried to load junku’s 3,700+ contacts, you’d almost assuredly run out of ram. So, I need to add some paging to the list of your buddies. And, it just wouldn’t be right to use the same icon, so I need to change that too, but, both of these things are things I want to do for me anyway, and thus they’ll be fun, but even if they weren’t fun, the knowledge that it would make others unhappy would make me unhappy. So I’ll do a little work to avoid sadness, and increase everyone’s happiness.

5) Share freely.
Talking about “advice” John says:

“I find myself afraid of asking questions for fear that someone else will steal my idea or think I’m an idiot, and this keeps me from heading to the an irc channel when I need help. The bottom line is, 99 percent of the people who come up with really cool ideas never finish them and most never even start, so if you need to divulge a little in the interest of getting the thing done, do it.”

I say, screw that. You’re doing this for fun right? If you’re playing it so close to the vest that you don’t want anyone finding out what exactly you’re doing, then you’re working not playing. I remember reading a post by a novelist who talked about how there were all these people with great ideas for a novel who were afraid to mention them for fear some author would steal them. But professional authors don’t hesitate to share their ideas because they know that cool ideas are a dime a dozen, and, like John says, executing on one of them is something hardly anyone ever bothers to do.

Even if someone took your idea and ran with it, what they’d end up with would be radically different from what you would end up with. Plus, there’s the fact that all the stories have pretty much been written already. New novels are just old stories processed through different people’s imaginings. As with writing, there are very few new software ideas. Mostly, we’re just making iterative improvements to existing ideas.

Personally, I’ve got notebooks filled with ideas for projects, and about half of them would be profitable businesses, but I’m not going to implement 99% of them (and if any of my geek friends was in need of a business idea they’re there for the taking).

So ask your questions. Discuss your ideas. “Worst” case scenario someone else does most, or all, of the work for you, and you get that fun app you wanted without having to lift a finger, which leaves you more time to work on that other fun app. Or maybe they point out that you really are an idiot, but, wouldn’t it be better to discover that you’re doing something stupid *before* you spend weeks coding on it? Best case scenario, you get free suggestions on how to make your thing better, and if you release it open source then maybe someone will send in a patch with a nifty new feature, or fix that bug you never got around to addressing.

Remember, as soon as you start charging for it, you have responsibilities to the users, and those can very quickly turn into work. And if you’re not charging for it there’s no good reason not to release it as open source. Plus, every time you get a patch or a bug fix sent in it’s like a mini-christmas. Some surprise present, and we all like presents.

In summary, for me, the key to having fun with coding is to focus on the fun, the little bits of joy, and always keep sight of the life that’s going on around you. Be part of it, and make sure that when you look back on any part of your life, you’ve got something to show for it beyond some code that’ll probably be replaced or abandoned in 5 years.

Handling and Avoiding Conflicts in Git

2008-07-12T00:00:00+00:00

John Kelvie said:

[To] me the fundamental challenge with existing version control systems is the difficulty of merging change sets from multiple developers across the same set of code. To me, this issue comes down to the diffing/merging functionality provided by the software, and I haven’t seen or heard of anything that really improves the state of the art. How does GIT address this? How does it make it easier to do? Are there specific branching and merging tools it provides? Is through the use of more atomic commits (which I could see helping to an extent, but only so far as it allows for changes to be small enough that there is no overlap, thus sidestepping the problem).

There are two concepts that you must understand to really take advantage of git. The first is the index / staging area. A full description would require its own post but for this discussion you can think of it as a temporary branch where you put everything before committing it. You can diff between it and the last commit, between it and the working directory, etc. The second is that a git repo can have (and usually does) have multiple branches in the same location on disk.

The idea of multiple branches in the same folder, is critical to understand when trying to address conflicts. In most version control systems you have one checkout for every branch, each in a different location on your disk, and while this is an option in git, you can also have multiple branches in the same folder, just like the server side of most version control systems.

So, we’re going to need an example. Let’s assume I’ve got a git repo with foo.rb. In the master branch foo.rb looks like this:

if (value == "test") puts "I'm in!"
end

On branch_one we’ll change that test to

if (value == TEST\_CONSTANT)

and on branch_two we’ll make it

if (value =="test" || value == "test2")

Now, if master, branch_one, and branch_two are all in the same repo we could stay in the same directory and “checkout” whichever branch we wanted to be working on. All the would change in place without you ever leaving the directory. So, if you checkout master and cat foo.rb you’ll get the first example. Checkout branch_two and cat foo.rb (without changing directories) and you’ll see the last example. I’m sorry if that was obvious, but it’s critical that you understand that, and some newbs just aren’t aware of it yet.

So, back to John’s questions. How does git make branching and merging easier to do, and does it provide any specific tools for managing it? Well, first off the commands are freaking simple:

to create a branch in the same repo git branch branch_name or, more commonly git checkout -b branch_name The latter creates the branch then checks it out so you can immediately start working on it.
to merge in a branch in the same repo git merge branch_name
to merge in a branch from a remote repo you’ve got git fetch and git pull More on these two in a minute, because they do dramatically change how you manage conflicts.

As for specific tools for managing it. Yes and no. “…if you rather use GUI tools to merge files instead of editing file with conflict markers (like shown in example), you can use git-mergetool, which would call GUI tool of your choice; currently supported out of the box are: kdiff3, tkdiff, meld, xxdiff, emerge, vimdiff, gvimdiff, ecmerge, and opendiff.” BUT, regardless of GUI tools, the ability to have multiple branches in the same local repo changes everything.

Here’s an example. Same files as above, except I’m using a different repo for each instead of different branches in the same repo, because that’s the way it would be when pulling from other people, and since we’re dealing with people, we’ll say branch_one is Mary’s branch and branch_two is Bob’s (pretend that the. The first thing you’d do if you were pulling from these guys regularly is to add them to your repos list of remote directories

$ git remote add marys\_branch ../branch\_one/
$ git remote add bobs\_branch ../branch\_two/ 
$ git remote show bobs\_branch marys\_branch

And here’s where the difference between pull and fetch come in to play. fetch tells git “hey, go get this remote data, and shove it into a “remote” branch in my repo. pull says, “fetch it, and also merge it with my current branch.” Using fetch we can suck in data from two sources, knowing full well it would conflict if we tried to merge it, but still have no problems.

$ git remote update
Updating marys_branch
remote: Counting objects: 5, done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From ../branch_one
* [new branch] master -> marys_branch/master
Updating bobs_branch
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), remote: reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From ../branch_two
* [new branch] master -> bobs_branch/master

$ cat foo.rb
if (value == "test")
puts "I'm in!"
end

So, now, we’ve pulled in two people’s conflicting changes, but not applied either. So, lets assume I don’t suspect a conflict (hopefully they’re not normal occourrances for you).

$ git merge marys_branch/master
Updating 7d6f564..5713885
Fast forward
foo.rb | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)

$ git merge bobs_branch/master
Auto-merged foo.rb
CONFLICT (content): Merge conflict in foo.rb
Automatic merge failed; fix conflicts and then commit the result.

Damn, a conflict! Who would have guessed? Now, I do what git suggests, fix the problem and commit it, but I’m neither the one who broke it nor the one who knows most about it. So, I’m going to kick this back to Bob or Mary, but, I may want to see what’s going on, to help choose who to kick this to.

One thing I can do is see specifically which commits are conflicting:

$ git log --merge
commit f068f89e88e9174b50a0bc5622875dd4d8e21bc8
Author: Kay Rhodes
Date: Sat Jul 12 09:26:44 2008 -0400

switched to testing for test or test2 in branch_two

commit 5713885c9adc5689b6d8222b2650d4b3ad0dbc42
Author: Kay Rhodes
Date: Sat Jul 12 09:25:52 2008 -0400

switch test to TEST_CONSTANT on branch_one

If the commit messages aren’t enough info I can actually diff the two branches that conflicted totally separately from anything in MY branch.

$ git diff marys_branch/master bobs_branch/master
diff --git a/foo.rb b/foo.rb
index dac3b0f..220011b 100644
--- a/foo.rb
+++ b/foo.rb
@@ -1,3 +1,3 @@
-if (value == TEST_CONSTANT)
+if (value == "test" || value == "test2")
puts "I'm in!"
end

I could have also diffed the specific commits in conflict if I’d wanted, instead of the entire branches. In this case it would give the same result. But, not knowing what TEST_CONSTANT is (maybe it’s a closure, maybe it’s a predefined variable) I can either make an educated guess that whatever TEST_CONSTANT is it’s probably a better choice from a maintenance perspective and tell Bob to go fix his stuff. Or, I could just send an e-mail to both of them and let them figure it out.

It’s also important to point out that the “index in git contains on conflict *all* versions of a file: ‘ours’ i.e. the version in the branch you merge into, ‘theirs’ i.e. the version in the branch you are merging, and ‘base’ i.e. the version in the common ancestor ot the branches, and also version with conflict markers.”

Another option is to just fix it myself. The conflicting file currently looks like this:

<<<<<<< HEAD:foo.rb

if (value == TEST_CONSTANT)
=======
if (value == "test" || value == "test2")
>>>>>>> bobs_branch/master:foo.rb
puts "I'm in!"
end

HEAD is git slang for “the branch you’re currently working on” which, in this case, has Mary’s changes successfully merged in (actually, by default it’s pointing to the last commit on the current branch). So, I decide that all three tests need to be there and make the edit, and then update the index. You need to update the index because git has already added Bob’s changes to the index but knows that there was a conflict that needs resolving.

$ git status
foo.rb: needs merge
# On branch master
# Changed but not updated:
# (use "git add ..." to update what will be committed)
#
# unmerged: foo.rb
#
no changes added to commit (use "git add" and/or "git commit -a")

Updating the index is simply a matter of getting the file how you want it and saying:

> $ git add foo.rb

at which point committing would get you a commit message template that started like this:

> Merge commit 'bobs\_branch/master' Conflicts: foo.rb \# \# It looks
> like you may be committing a MERGE. \# If this is not correct, please
> remove the file \# /Users/krhodes/temp/trunk/.git/MERGE\_HEAD \# and
> try again. \#

The basic take-away from all this fetch / pull stuff is that it’s generally a good idea to fetch first when dealing with other people’s work, because having a local copy of their branch in your repo gives you a number of additional options if things go wrong.

And yes, atomic commits go a huge way towards avoiding the problem of conflicts, and handling them when they crop up. Obviously this applies to centralized systems too. They avoid them by keeping your commits on topic. If one commit is for bug X you generally don’t have to worry about it conflicting with a commit for bug Y. And, when you discover that two people you’re pulling from both have patches for bug X that conflict you have many more options.

Another key technique for avoiding conflicts is to merge constantly. Every single day you should be pulling any work that’s been committed upstream. That way, when things do conflict, you only have to resolve a little tiny thing (if the commits are atomic then it’s even tinier), and, if you choose to kick the issue back to someone else it’s still fresh in their heads. This, of course, applies to centralized systems to.

Another way that git (and every other distributed system) helps to avoid conflicts is a social change that comes out of the distributed nature of the repos. Because you’re constantly pulling in from multiple sources, having atomic commits becomes a requirement to participation. If Bob is a twit and only makes massive sprawling commits you’re simply not going to pull from him. It’s not like a centralized system where you’re forced to take all the changes out there. Whoever is in charge of the main trunk would be stupid to accept Bob’s changes because if there was a problem in one of them it’d be a pain in the butt to extract the bits that fixed bug A from the bits that didn’t really fix bug B. No, they’d kick it back to Bob, telling him to clean up his stuff and make two separate patches.

This becomes even more important when you’re working on a project that manages patches via e-mail. Just check out the git mailing list. People’s patches are incredibly focused, enabling other git developers to take just the bits and pieces they choose. Because the patches are uniquely identifiable in git, it doesn’t matter if some people take a change and some don’t and you, as the maintainer, pull from all of them. Git won’t think they’re two different but identical changes. It’ll know, “Oh, that’s change x. I’ve got that already.” Git also doesn’t care how a patch got there. It could be from a commit you pulled over the lan, or something someone e-mailed. Because it’s guaranteed cryptographically unique, it’s the same thing.

[Update 2] Some additional tips from Jakub Narebski

If you want to see what some file looks like on other branch (or for example at specified point of time, for example at tagged revision, marking some released version), you don’t need to use “git checkout ‘branch’ && cat ‘file’”; you can use “git show ‘branch’:’file’” (see “Examples” section in git-show(1) manpage, and “Specifying revisions” section in git-rev-parse(1) manpage).

Below there is example how index looks like during conflicted merge. The example uses yet another way of specifying object names, i.e. “:’stage’:’file’” (we could alternatively use SHA-1 or shortened SHA-1 of object shown in git-ls-files output, e.g. “git show 2f096cc”).

$ git ls-files –abbrev –unmerged
100644 2f096cc 1 foo.rb
100644 89f36fe 2 foo.rb
100644 d3ea75d 3 foo.rb
$ git show :1:foo.rb
if (value == “test”)
puts “I’m in!”
end
$ git show :2:foo.rb
if (value == TEST_CONSTANT)
puts “I’m in!”
end
$ git show :3:foo.rb
if (value == “test” || value == “test2″)
puts “I’m in!”
end

Try it yourself!
I find that reading how to do things like this is much enhanced by actually having something to test it on, especially when there are so many variables. So, I’ve uploaded the temp dir with the three repos I used in writing this. If you go into the trunk repo you’ll see that the other ones have already been configured as remote repos. And, if you go into branch_one or branch_two you’ll see that they’re clones of trunk and thus both already know that their upstream is trunk and how to fetch from it without having to configure anything. You can also see what the difference would be in pulling and fetching from one of them.

Please let me know if there’s still something that’s a little unclear about dealing with conflicts in git and I’ll add it in.

If you found this useful you might be interested in some of the other git posts here.

Why you should use a distributed version control system

2008-06-28T00:00:00+00:00

If you’ve ever:

made a commit and then realized you forgot “one little change”.
made a commit and regretted it.
wished you could combine some the past couple days worth of commits into one nice combined commit in the main branch.
wished you could commit just part of a file.
needed to drop work on one task and switch tracks to another one without having to make commits with unfinished changes, or commits with changes for one issue and a little of another.
wanted to make a test spike with version control and without polluting the public repo.
managed an open source project.
wanted the security of knowing that there was a valid backup of your revisions on many other peoples boxes, or even just your own.
been frustrated with branch namespacing issues
been frustrated with how difficult branching and merging is in most centralized version control systems.
wished you could just create branches to work on a feature or a bug without worrying about the consequences to the main repo.
wondered which branch a bug applied to.
wanted to use version control when you were offline.
wished you could quickly compare versions of entire trees.
wished you could easily release everything in the current branch “except that”.
been concerned about how to scale a system to support hundreds, or thousands, of users.
been concerned about what would happen if your main repo box died.

…then distributed version control is worth your consideration.

Now, there are a variety of DVCS to choose from, Git and Mercurial (Hg) are main contenders, and both are really good. My experience is primarily with Git so I’ll be speaking as a Git user, but Mercurial can do most everything Git can, and a lot of this is general to all Distributed Version Control systems.

Quite possibly the most powerful, world altering thing about DVCSs is how they handle branches. Because of their distributed nature branching and merging absolutely must be simple effective, and avoid conflicts whenever possible. As a result people who’ve gotten the DVCS religion tend to make a branch for practically every bug or feature. We call these “topic branches”. If you’ve been working on one task for a while and a bug comes in that has to be address NOW it’s not an issue. Commit your changes in the current branch (or “stash” them away) and make a new branch for the bug that just came in. I doesn’t matter if you commit unfinished work in one of these branches, or if the work totally breaks the build, because it’s your branch. When you’ve finished working on a feature or bug you can “rebase” all your interim commits on that task into one nice clean one, or maybe generate a patch to apply to the main branch.

The fact that you can as many of your branches as you want solves a huge number of problems. Combine that with the ability to “rebase” you changes, by bringing up a list of past commits to merge, reorder, or exclude, as you see fit, and …well it’s like heaven. The only caveat is that you can’t rebase commits that others have already pulled. But that’s not a big deal if you simply do your work in personal branches that others aren’t going to be pulling from.

Because most DVCSs allow you to cherry-pick exactly which commits you want to be in a branch you can easily build a release branch with just the parts that you want. There’s one caveat to cherry-picking in most distributed and centralized systems : to do it well you must have commits that are a self contained and atomic as possible. If you make a commit with multiple features/fixes in it and later on you decide not to include one of them there’s no simple way to exclude just the bit you don’t want, because it’s mixed in with all the others. And this, brings us back to the idea of “topic branches.” If you’re already working with topic branches then you can easily merge the changes in one topic branch into the main branch as one atomic commit that can be excluded or included individually. However, as Aristotle pointed out in the comments

“the power of [Git’s] rebasing allows you to go back, split the commit sensibly into several, and then transplant the rest of your history on top of that. I guess whether this is simple depends on your definition of simplicity, but git has enough tooling to support this procedure directly. In other DVCSs you have to painstakingly monkey around with far less powerful tools, and in centralised VCSs it is for all intents and purposes impossible to do it at all.”

Yes, there will always be some changes that are dependent upon some prior commit, but by having nice little atomic commits you can limit the tree of dependencies to a bare minimum.

One of the best features of DVCSs is speed, but unfortunately it’s a lot like the Matrix. No-one can be told how much of a difference it makes. You have to experience it for yourself. I never complained about the speed of the centralized systems I worked with until I switched to Distributed Version Control. I found it hard to believe that “instant” commits would make a difference. Or that the time it took to diff against past versions was an issue. But I swear to you that the effect on how you work is almost as radical as the improved branching, and it flows over into the branching too. Now branching and merging is not only painless, it’s instant. I create, merge, and destroy branches throughout the day without hesitation because no matter what operation I do it comes back so fast that I frequently am incapable of telling that any time passed. That’s means that many (most?) of my operations are happening in the sub 100ms range. Imagine being able to diff versions of entire trees in tenths of seconds.

Scaling to thousands of users is not an issue with Distributed Version Control Systems because one box doesn’t have to support all the simultaneous operations that would normally be going on. Commits, checkouts, reverts, branches, diffs…. all of it happens on peoples local machines. They rarely need interact with the canonical / central repo. This means that large companies like Google wouldn’t need to invest in massive boxes with huge processors and piles of ram just to keep their version control system usable.

Speaking of the canonical repo… workflow doesn’t have to change at all. If you’ve got an established workflow with a centralized system there’s absolutely no reason it can’t continue on after a switch to Distributed Version Control. But, it does open up some other ways of working that you may want to explore.

In a centralized system if the box your main repo lives on goes down you’re pretty much screwed. If your IT people are really on the ball you’ve got it syncing to another box every few minutes (or more) and everything will fail over… bringing the primary box back up and resycing it may cause them to pull out their hair but at least no-one would be affected. Unfortunately it is a very rare company that’s that well prepared. And if you’re hosting an open source project you just have to hope that the people you’re hosting with have good backups and don’t go down.

Most software companies, and open source projects, simply grind to a halt when the version control system goes down. Changes can’t be shared, branches can’t be made, bugs can’t be patched, releases can’t be made, developers start building up a backlog of changes that will end up mooshed together into one commit when things do come back up, etc., etc., etc. But teams using DVCSs simply don’t care. It’s a non issue. “The heads crashed on the drives of the main repo, and a fuel truck plowed into our offsite backups”. It simply doesn’t matter. Sure your IT guys are going to have to deal with fixing that, but the impact to development is essentially zero. Someone stands up, or sends out an e-mail, and says “the main repo is down, just use my box instead”, and people do, and that’s the end of it. When the main box is repaired IT simply pulls from the box everyone’s been using in the interim and once again stands up, or sends out an email saying “the main box is back up”. Automatically syncing a remote box for fail-over becomes trivially simple with DVCS. Just set up a cron job to pull from the main box every minute and you’re done.

When it comes to open source projects there is one factor that absolutely sucks and that’s giving out write permissions. You don’t want to give them out to just anyone, but if you don’t then people who you don’t trust yet either have to work on your code without any revision control, or as is frequently the case, they just fork it into a version control repo they do have control over and loose the ability to easily merge their changes back into the main repo. You could just give them out to anyone but that’s not a choice that many project managers are comfortable with. With DVCSs it simply isn’t an issue. Everyone’s got their own repo(s) everyone can commit, branch, etc.. And when they’ve got something worthwhile they can ask you to pull from them, or send you a patch that they know will work with your repo because that’s where their repo originated and they’ve been pulling down your changes and merging it in to their work.

One problem that’s been an annoyance for me for years is that most bug trackers have no concept of branches. When you go into your bug tracker it shows you all your bugs but there’s no way to tell what branches they exist in, and what happens when you fix a bug, and get QA to sign off on it, in one branch but it hasn’t been patched in another one? Is the bug fixed or isn’t it? And what about the people who don’t have an intimate knowledge of that bug? How are they supposed to know where it does and doesn’t exist?

Distributed Issue Trackers solve this problem. You file a ticket in the branch the bug exists in. Then that ticket follows along with and branches of that branch until someone closes the ticket. As they merge the fixes back into parent branches the closed ticket merges in too. As a result you can always tell if a bug is or isn’t fixed in the current branch. If you commit your closed ticket with your patch then even more possibilities open up. If, for example, it was decided that your cure was worse than the cause and excluded that commit from a release then the ticket would re-open itself in the release too (because you got rid of the fix).

To date the best Distributed Issue Tracker, without question is Ditz, but the field is still very young. I’ve personally been working on a fork of Ditz that makes some dramatic changes and tightly integrates itself with Git, and I’ll update this page and make an announcement as soon as it’s released (a week or two). Personally I wouldn’t recommend anything other that Ditz right now. The projects are either too immature or abandoned and buggy. Ditz doesn’t have a lot of features yet but it’s reliable and gets the job done.

Another cool thing about Distributed Version Control Systems is that you don’t need anyone’s permission, or cooperation, to start using them. I work at a company that uses Perforce, and I think that Perforce is the devil, but it doesn’t matter, because I have Git. I do all my work in Git; constantly branching and merging and mooshing commits, and when I’m ready I just submit the completed changes back to Perforce. And this is actually better for everyone. Sysadmins don’t have to deal with the consequences of me making topic branches all over the place, the commits I make for coworkers are generally cleaner, and I can commit as often as I want, even break the build, without having to worry about the impact on others. Now, it happens that Git has a number of tools for working with the major centralized version control systems (it’s even got a CVS proxy so that CVS people too resistant to change can keep working the way they’re used to). But, if your DVCS of choice doesn’t have a tool to bridge the gap to whatever centralized system you’re forced to deal with, you can always use Tailor.

It sounds too good…

Some of you are probably thinking this sounds too good to be true, or maybe that I’m DVCS zealot. But really, it isn’t, and I’m not. I’m a huge fan of DVCS, it’s true, but I’m not about to claim that they will solve all your problems. To really get the full benefits of them you’ve got to start making atomic commits. This is actually true of centralized systems too, but since centralized systems tend to be such a pain to use, and frequently can’t do cool things like cherry-picking, people don’t even bother to do anything advanced with them. With a distributed system the advanced stuff becomes trivial every-day stuff, and it becomes more annoying to have to deal with those developers who won’t stop making ginormous commits with changes to multiple bugs as well as a new feature or twelve. Fortunately, you don’t have to accept their patches of you don’t want to. ;)

Guis have come a long way since this was first posted, especially for Git and Mercurial with quality that rivals the best tools for centralized systems.

On the command line some DVCSs like Darcs shine with totally intuitive commands, others have a bit more of a learning curve. Git definitely does because of how it stages things before commits. It appears similar enough to the way centralized systems work that many newbs expect it to work the same, and get frustrated when their expectations collide with its significantly different paradigm. Not only does Git stage things before commits but it doesn’t even think in terms of files, although, unless you’re paying attention, it may seem that it does.

Scaling to handle extremely large repos with years of history is something that Git handles very well (Mercurial probably does too), but some systems, like Darcs, have serious problems with.

In short, Distributed Version Control systems are totally bad-ass, and can really help with a lot of common development problems, but they’re still fairly new, and while the big ones are good, and reliable, they may not have all the polish and GUI widgets that you’re used to with your centralized system. My advice is to go get Git, unless the majority of your developers are on Windows, in which case Mercurial may be a better choice… for the moment.

typelation [Definition]

2008-06-12T00:00:00+00:00

typelation: n. The act of converting speech, or thoughts, into text for a text-based conversation (e-mail, instant messages, etc.). Example: The joke lost something in the typelation.

Angle-bracket Operator [Definition]

2008-06-09T00:00:00+00:00

Angle-bracket Operator: n. A person who manipulates HTML professionally. Also see Web-designer.

If Patton was a coder...

2008-05-19T00:00:00+00:00

General Patton said that “A good plan violently executed now is better than a perfect plan executed next week.” If he was a coder he might have said “An automated test violently executed now is better than a perfect test next week.”

How to create a Test Suite in Perl's Test::Unit v0.25

2008-04-09T00:00:00+00:00

If your Test Case is a package whose goal is to test all aspects of a particular class then a Test Suite is something which kicks off a collection of related Test Cases. As with most things in Perl’s Test::Unit it’s really easy to do and also terribly documented. So, without further ado… You need something to kick off all your tests:

    use Test::Unit::HarnessUnit;
    use My::Test::Suite::Package;

    my $testrunner = Test::Unit::HarnessUnit->new();
    $testrunner->start("My::Test::Suite::Package");

Next you need the test suite it’s going to kick off:

    package My::Test::Suite::Package;
    use base qw/Test::Unit::TestSuite/;

    # returns an array of the fully qualified names of the TestCase
    # based classes you wish to run.
    sub include_tests() {
        return ('Some::Test::Case',
            'Some::Test::Case2',
            'Some::Test::Case3);
    }

    1;

That’s it. Happy testing.

LazyWeb Idea: Gravatar + Network of sites = Gravanetric

2008-03-20T00:00:00+00:00

Pretty much everyone agrees that Gravatar rocks. A global avatar that shows up wherever you make a comment on a blog (sometimes even in your desktop apps). And I don’t think anyone other than naive VC guys wants another “social networking” site, so I’m not going there. But, imagine what would happen if every time you made a comment on a blog that used Gravatars a ping was sent off to the “Gravanetric” servers with two bits of information the hash of your e-mail and the root url of the site you posted too.

Then you could start to graph shared interests. Imagine a simple force-directed graph with the Gravatars of people who posted to them arrayed around a central site node, or vice-versa, or a graph of the sites with lines of different thickness between them to denote how much of their community is shared. Then you could look up a site you liked and see what other sites people thought were interesting enough to comment on, which would mean it more likely that you’d want to read, and maybe comment on, what’s at that site too. Or, you could enter in your e-mail address and it could show you other people like you, or maybe graph the network of sites you’ve visited. If you added in the ability for a person to associate an URL with their Gravatar then you could go find out more about the people similar to you.

Visualizing this would be trivial with something like JsViz. The only trick would be deciding how much to visualize. There’s really not a lot to the app itself either, although you’d have to plan for some pretty huge database tables.

Throw in an API for people to query x degrees of separation of sites, or people or anything else interesting and you’ve got a giant ball of coolness. The API could allow them to overlay it with links to the people who were signed up to their TrueLoveForever.com type site and you’ve got some interesting possibilities. Maybe make use of the API free for < 100 requests a day like Google, and other sites do, but charge if anyone wants to make some significant number of queries…. Plus you can always throw banner ads on the site. Heck, you might even make some money off of this.

Obviously this would be easiest for the Guys over at Automattic since they’ve already got the Gravatar infrastructure in place and know exactly what kind of load they’d need to handle before going into it.

Getting just the tip of a Git repo

2008-03-13T00:00:00+00:00

Sometimes you just want to distribute the source code without its history, and that’s where git-archive comes in. git-archive will create an archive of the files at any point in the history and wrap them all up for you in a tar or zip (defaults to tar). You can even make an archive from a remote repo by using the

—remote=<repo>

option in the administrator has enabled it.

You’ll typically use git-archive like this:

git archive <tree-ish> > my_new_archive.tar

And that will create an archive of everything in your repo. If you just want an archive of some specific files you can simply pass in the path(s) to the file(s) you want in the archive after your tree-ish. If you wanted to create a zip file instead of a tar you’d simply pass it the “—format=zip” option.

Many people wonder if there is a way to check out just the tip of the repo so that they don’t have to download the entire revision history. This is called a “shallow” clone and is possible by passing the “--depth <some\_number\>” but it has a number of limitations: you can’t clone, or fetch from it and you can’t push from or into it, but it’s useful if you only want to look at, or near, the tip of a large project with a long revision history and would want to send in your fixes or features as patches.

Squeak By Example (first impressions)

2008-03-13T00:00:00+00:00

I’m reading through Squeak By Example because I’ve got some ideas rumbling around in my head that might be nice to do in Smalltalk. Having an integrated visual environment where everything is an object opens a lot of data visualization possibilities. Anyway, it has been probably two years since I’ve touched Smalltalk, and even then it was pretty brief. So I needed a refresher course.

I’ve been flipping through it looking for random bits of information I was interested in and found them all. Then, I went back and started from the beginning, following all the instructions, doing all the examples…. They’ve done a great job explaining things, it’s really easy to follow, and gives you an excellent step-by-step introduction to Squeak’s IDE, and that’s a very good thing if you’re not familiar with Smalltalk.

And then I hit section 1.10 and my jaw dropped. Test Driven Development…right there, at the start of things too. Many of my friends know how annoyed I am by the fact that books don’t teach about writing unit tests as part of the language. Most language books teach you how to program in some new language and then, if you’re lucky, at the end go “oh yeah, and you should write unit tests.” Which, I feel, results in a very similar attitude amongst the developers who use the language. Code code code, “oh yeah, unit tests are ‘good’ but, whatever…” code code code.

But, here I am (in a book from the land where unit tests were invented (sUnit)) and for the first time in my life I’m seeing unit tests being treated as first class citizens in a language. And they’ve already mentioned version control in passing.

Holy, fucking, shit.

So, 22 pages in (plus however many random pages I read while skipping around over the past couple days) and I have to say I am really impressed. Well written, easy to follow, good example graphics. It’s still pretty preliminary of course but… If you’re interested in learning Smalltalk I recommend checking out Squeak By Example. I’ll be grabbing the soft-cover version shortly but you can download the PDF version for free and check it out.

And, please, don’t be turned off by the cutesey iconography Squeak uses. A lot of the development that’s gone into Squeak has been to make it approachable and easy to use by children, but that doesn’t mean it isn’t still a powerful development platform. If you’re not convinced just check out DabbleDB. It’s written in Smalltalk with the very cool Seaside web framework.

P.S. Many have heard me say there’s no joy in Java, but there’s much joy to be found in Perl, Ruby, and Python. Well, there is in Smalltalk too. In all my Smalltalk excursions so far I’d say that Smalltalk is “nice”. I don’t know if I’ll personally come to find it joyous (I certainly don’t find Python joyous but I understand how others can), but it is, at the very least, nice. :)

Sharing a public Darcs repo over HTTP [flow chart]

2008-03-11T00:00:00+00:00

The same friend who wondered about how to share a Git repo over HTTP dared to suggest that “It’s so easy with Hg…”. While I happen to disagree that it’s easier in Hg than Git, I think this flow chart successfully demonstrates that, on this front, Hg and Git both suck ass when compared to Darcs.

P.S. Yes, while 100% factual, the graph is also intentionally silly.

[Update] After considering it more I think think it probably is easier to share a repo with Hg than Git, but I dislike the fact that it involves mucking about with things outside of the repo. You have to set up a CGI. You have to alter a config file that tells it what repos to share and where…. Yes, there’s a built in server you could use but that’s really an edge case. Most of the time you’ll want to leverage an existing web server.

Sharing a public Git repo over HTTP [flow chart]

2008-03-11T00:00:00+00:00

There is also an SVG version of this flow, which is more readable (but poor IE folks will have issues). Notes: This is a simplest possible configuration. Be sure to check out the docs for git-remote to see how to, optionally, designate specific local or remote branches. Many of the initial commands could be performed locally and then just uploaded to the server. This particular sequence guarantees that all the connection pieces are in place and working correctly.

Much thanks to Tim Toolman for getting this info into two easy to follow posts: Sharing git repositories via OS X’s built-in web sharing and Setting up a new remote git repository.

However, this doesn’t feel elegant to me. If anyone can come up with a simpler / more elegant way to do this please add a comment.

Where's the "main" repo when using Git?

2008-03-11T00:00:00+00:00

Sivaram said: “I have been using CVS on and off for a long time; so using git is a bit confusing.

If all the repo clones are equivalent, how does one know one is the ‘clean’ repo? On CVS, there is a centralized repo lying somewhere. I can’t seem to wrap my head around the decentralized model.“

This question trips up a lot of people when they’re introduced to the concept of distributed version control systems. But the answer is exactly the same as in the centralized world. It is wherever the project maintainers tell you it is. Let me give you an example:

If you were joining a new company or project that used SVN (or any centralized system) you’d come in and say “Hey, I’m new. Where’s the repo? “ and they’d respond, “We use SVN. The repo’s over there. Here’s the url.”

If you were joining a new company or project that used Git (or any other distributed system) you’d come in and say “Hey, I’m new. Where’s the repo?” and they’d respond, “We use Git. The repo’s over there. Here’s the url.”

See the difference? Neither do I. But in case you’re not convinced, consider the fact that at any point in time your sysadmin could move your SVN repo to another machine with a different URL. Everyone would change the URLs they connected to and that would be the end of it. The same thing applies in the distributed world.

Where your canonical repo lives, in any version control system, is nothing more than the place people happened to have agreed to keep it.

The only real difference is that in the distributed world you also have the option of pulling from non-cannonical repositories.

For a real world example we can look to Git itself. Where’s the main git repo? It’s where the Git homepage says it is. Compare this with Apache, which uses Subversion; their main repo is where their homepage says it is too. In both cases it might change tomorrow, but everyone would be really annoyed if it did so the people in charge are going to do their best to avoid it.

Sometimes people contributing to Git will send an e-mail to the list asking others to pull some feature from their personal repo (instead of just sending a patch). Some people pull, some people don’t. It really doesn’t matter. All that really matters is what changes get pushed to the main repo, just like in the centralized world.

Deciding what gets pushed to the main repo, or who gets to push to it is a project management decision that has very little, if anything, to do with what version control software you happen to use. Some systems enforce those business rules with software. Some don’t.

Importing an existing codebase into Git (a flowchart)

2008-03-08T00:00:00+00:00

A simple flow chart showing the steps you should take to add an existing codebase to Git. This assumes you don’t have revision history that you wish to migrate from another version control system. Some notes about the flow:

When adding file paths you can use wildcards like “git add /path/to/images/*.jpg”
This is one of the few times when you’d want to use “git rm –cached “ to un-add a changes in the index. After the first commit the recommended way to un-add changes to the index (staging area) is to “git reset HEAD “.
Note that the only file that is edited during this process is .gitignore When you call “git add” you are adding the current content of a file to Git to the index. If you change a file after you add it Git won’t commit the additional changes (unless you add them too). So, it’s a good idea to run “git add .gitignore” just before you commit in order to make sure Git has the most recent version of it.
You’d only, obviously, “add” files you wanted. “git add .” (note the dot) tells Git to add everything in, and below, the current directory.

[Update] P.S. It’s just a matter of time before someone tries to use this as evidence that Git is complex or hard to use, but before you do, please do an equivalent flow chart for your version control system that includes ignore files, adding selected files, adding all files, removing accidental adds, and everything else. You’ll find your chart is just as “complex”. The shortest path here is only 4 commands and that includes setting up the new repository to check the code into, and an optional sanity check of what’s going to be committed.

Review of Edward Tufte's Presenting Data and Information course

2008-03-07T00:00:00+00:00

First, let me set the stage. I’ve been reading stumbling across interesting data information articles by Edward Tufte for years now, have been interested in getting his books for a while now, and was excited when my manager offered to send me to his one day course, and am quite grateful to have had the opportunity to go. So, I definitely went into this with good expectations.

30 Second Summary:

The first two thirds were not bad. The second two thirds sucked. The type of people who would appreciate this course the most are ones akin to the woman in front of me who wore red velvet pants, a scarf that probably cost $60 and from the Museum of Fine arts, dangly earrings of semi-precious stones, and, were you to talk to her, would be sure to let you know that she’s “an artist.” She loved it. You, on the other hand, should buy his books and skip the course.

[Update] Actually, skip buying his books, unless you really want them. Everything you wish was in his course can be found in The Back of the Napkin: Solving Problems and Selling Ideas with Pictures. I simply can not recommend that book highly enough to people wanting a practical guide to the visual presentation of data.

Details:

The first two hours were pretty informative, and essentially all of my notes that didn’t involve expletives came from that portion of the talk. But, there were two quotes (taken completely out of context here) that pretty much summed up his course

“…I prefer text.” and “Well this is all very interesting but…”

After the first intermission Edward made sure to let everyone know about his wonderful four hundred year old books by by Euclid and Gallileo which he held aloft, and flipped the pages of with his oily ungloved hands, thus guaranteeing they will not survive the next four hundred years as they have the last. He even made sure to carry it down a couple of aisles and show it to the people on the end, whilst frustrating and / or boring the other 95% of the audience.

After lunch he talked about visual interfaces which was comprised almost entirely of him telling you how cool his iPhone was. According to Mr. Tufte the only major thing that Apple got wrong with the iPhone was using icons on the main interface instead of text. Just imagine how many more items you could fit on that screen if it used text instead of icons! Oh the joy of having to read through text lists to find the fucking calendar app instead of just poking a fingertip sized picture of a calendar. Gallileo was so cool. Did you see this section of my book by Gallileo? To be fair he pointed out some valid ways they could improve the weather and stock apps.

Much time was spent making fun of PowerPoint, “bullet grunts”, and “chartoonists”. A grand total of zero minutes were spent discussing the pros or cons of any standard graphing technique: bar graphs, line graphs, area graphs, pie charts, scatter plots, etc.. Zero minutes were spent discussing what graphing techniques were best suited for what kinds of data (unless you count the 3 minutes spent on vertical scaling of graphs). Did you see the picture of Gallileo? There aren’t many pictures of him but I have an etching of him in my 400 year old book right here.

In between jabs at PowerPoint he discussed his technique for speeding up time in meetings, which is to start out with a “massive data dump” (giving the participants stuff to read before you talk), and then to ask if there are any questions. He guarantees this will make your meetings at least 30% faster. I guarantee that if you ever try that with a potential customer you won’t make the sale. He repeatedly used this technique on the audience by saying “now turn to page x in my book titled y” and waiting silently until everyone in the audience had read the whole page. Oh have you seen my book by Euclid?

He repeatedly advised people to clarify confusing data by adding more detail. While there are a handful of situations where this may actually be good advice it’s incredibly bad advice to give to an audience without clarifying when, why, and how to do so, which, of course, he never did. I will note that I found at least one good example of this technique while flipping through his books. The iPhone is so cool, but AT&T’s Edge network is the reason it isn’t cooler.

Edward also seems to be laboring under the mistaken impression that people actually want to understand all of the details of all the data you’re presenting to them. It is my experience, and many concur with me, that most people want to be shown the high-level “bullet grunts” of whatever data you have so that they can have a clue what the hell they’re looking at. Only then do they actually want nitty-gritty details, and usually they want the details on just a subset of the information. Aren’t Gallileo’s etchings of these sun-spots wonderfully detailed?

Documentation. All(?) graphs need documentation. The names of everyone who contributed to them should be present in big bold letters just like Gallileo did in this really old book. He even put his picture in it. Isn’t Gallileo cool? Also, descriptions: lots of descriptions, and annotations, and annotated descriptions, It’s fine if it the text required to describe the graph requires the same, or more, space than the graph itself. One of the techniques that he mentioned multiple times for making data more approachable was using annotations, which is great if your data isn’t going to change the next time the person looks at it (any stats on constantly changing things), but doesn’t seem to grasp how incredibly difficult it is to write software that can intelligently find a point to annotate that is either interesting or indicates some causality, never-mind trying to write software that will generate useful text about it. As an example, check out the Google Trends for “git”. When this was written only one of the annotated points had any causal effect on the data that’s being graphed (the one about Git software being released).

Resolution. He makes a really big deal about resolution. He applies it to media (screens, paper, etc.), as well as the resolution of information in a graph. He makes a lot of valid points about resolution, then talks about how some typographers designed their fonts at a resolution of two thousand grid lines per inch, and how his “Sparklines” have “two significant digits of resolution”. But he utterly fails to grasp that there is a point at which adding resolution is useless. In print his Sparklines may very well be precise to within a hundredth of an inch, but no fucking reader can tell the difference.

Imagine a typical ruler with precision down to a sixteenth of an inch. Now put ten times as many tick marks in each inch and you’re roughly at one half the resolution of print on good paper. Even at half the resolution of print you’re still going to have a really hard time telling the difference between 23/160ths of an inch and 21/160ths. But, after telling you how amazingly high the data resolution is in his Sparklines he proceeds to tell you it’s not enough and that he’s started work on displaying information in “High-def” Which bewilders the mind when you consider that you’re very lucky if your “High Def” screen is one third of the resolution of paper. Have you seen the resolution in this etching of Gallileo’s sun-spots? By giving the iPhone a higher resolution screen they more than doubled the amount of information they could fit on it relative to a normal phone! If only Gallileo had of had an iPhone…

Throughout the presentation I arrived at the following conclusions (potentially wrong) about Edward Tufte.

He has great distain for software developers.
He has no clue about the problems involved in displaying dynamic data.
He has no clue about how to run a presentation.
He has no clue why software works the way it does and berates it for not working the way he thinks it should.

BUT It didn’t all suck. As I said there were a number of good points in the first two hours, and all attendees got a box with all four of his books in it. You will need these if you attend the course because a third of the time he’s referring to pages in them like a typical college professor, and the other two thirds of the time you’ll need them to entertain yourself while he tells you about his four hundred year old books and his iPhone. I’ve read a few articles that went into one of them and I’ve skimmed through the rest and I do think they’re actually worth the cover price. He makes a lot of good points in the books and has a lot of good examples, although I don’t think that all of his suggestions actually play out in the real world.

Notes from the course (in case anyone cares):

Focus on causality when presenting information.
Data visualizations should be content driven not method driven.
Strive to invoke content responses / questions in your users.
Data should tell a story
Provide a context for the data.
Annotations: choose a point, explain it, then tie tangentially related facts to that explanation (not sure I agree with that last part)
Try and make the “smallest effective difference” when trying to emphasize some point.
Order data by the most important aspect, not alphabetically.
Find a way to connect your data to your users on a personal level (for example when dealing with data on neighborhood fires plot it on a satellite image of the town so that your users can find their house and thus connect to it personally).
Try to eliminate legends. You don’t want your users to have to look up details of your graph. It should all be integrated. For example if you’ve got a line graphing Bananna sales put “Bananna” on the line.
Keep related items adjacent in space.
It’s easiest to see variations in a slope when it is roughly at a 45 degree angle.
Make comparisons.
Attempt to overcome recency bias.

Also, some paraphrased questions to ask when consuming presentations:

What’s their story?
Can I believe them?
What is the “domain specification” of this presentation.
What do I want to see? Is that what I was shown?

Some questions to answer when presenting information:

What is the problem I’m addressing?
Who is it relevant to, and why?
What is the solution I am presenting?

Mr. Tufte, if you’re reading this, I appreciate and respect the work you have done, but you need some serious lessons on how to give a course. I’d recommend spending some of the millions people have given you for these courses on some private lessons with Tony Robbins. Whether or not you agree with what he has to say it’s hard to deny that that man knows how to convey lots of useful techniques to his audience in a way that is both memorable and entertaining.

P.S. Gallileo did, in fact, rock. However, if I wanted to see a historically significant book I’d go to the museum or the library (Boston happens to have a kick-ass library with regular, and free, exhibitions). The iPhone also rocks, and is a triumph of interface design, but if I wanted to see a video of someone using it I would go to Apple.com.

Some thoughts about Git

2008-02-05T00:00:00+00:00

Not too long ago I decided to start writing a book about distributed version control. I was originally going to focus on Mercurial (Hg) because it’s quite good and of the two leading systems it was the only one that ran on every OS (because it was written in Python). The fact that it could also run under Windows meant that I could help spread the word about distributed version control to more people, and it slightly increased the chance that I might actually make some money in the process.

There was just one problem: Linus was right.

“Hg is pretty good, but Git is better.” - Linus Torvalds

When I heard him utter those words in his talk to the folks at Google I took them with a grain of salt. Almost every project owner believes that their software is “better” than the competition, especially when they’re so similar, but it turns out that it wasn’t hubris on his part. When you sit down and compare the functionality Git simply lets you do more.

I had actually started writing the book about Hg but time and again I found myself confronted with something that was either easier to do in Git, or simply not possible in Hg. It happened enough that I decided that I didn’t want to keep writing about, and arguably helping, out a product that I not only wouldn’t choose to use myself but also felt was dramatically inferior from a usage standpoint.

And then I joined the Git mailing list. Linus mentioned that it had a high signal to noise ratio but… holy shit. I have been on a lot of mailing lists for open source projects over the years and I have never seen anything like this. Almost every e-mail is a patch, or a good discussion of a patch, or a good discussion of some new feature and how to go about implementing it. Currently, one of the topics they’re discussing is how to allow people to push to a Git repo based on GPG keys stored in, or accessible by, Git. What’s amazing is that it isn’t just one or two people going back and forth or someone trying to proclaim the “one true way” to do things. It’s exactly what you would expect if you took a bunch of smart geeks, stuck them in a room, and watched them all work together to solve some common problem. That alone is impressive, but when you combine that with the fact that it’s a pretty high traffic list and you’ve got a lot of really smart people pushing a tool forwards at an incredible pace.

Now, all is not sunshine and roses in Git land:

The documentation is actually pretty good, but when you want to do a specific task and don’t know the Git specific terminology for it it can be a little difficult to find what you need, and sometimes the commands take a dizzying array of options and could really do with more examples. That works out well for me, because it’ll be all the more reason for people to buy my book, but it’s obviously not great for attracting new users. I still think that, for most use cases, Darcs does a better job than anyone at making it easy to ferret out what command you need to accomplish a task.

Git is a lot like C. It’s incredibly flexible and powerful but it is also incredible flexible and powerful. If you want to shoot yourself in the foot Git will point you to the gun rack and show you how to load the bullets.

Under the covers Git is a collection of small C apps surrounded by shell scripts, Perl scripts, Python scripts…. whatever the author chose to get the job done. From a usability standpoint this is irrelevant. Calling “git foo” is going to work just as well regardless of what foo was written in and will have no impact on the user, but I’d personally prefer to hack on a project that was all written in one language.

It also means that the msysgit team has a lot of work ahead of them in trying make it install easily under Windows. If you’d like to see git working under Windows, and are familiar with coding for Windows please check these guys out and see if there’s anywhere you can help out. Even just taking on a little task here and there is something that would be greatly appreciated. Part of me wonders if it wouldn’t be better to just port everything in Git to C.

I think that if the Git book goes over well I may go back and retool it for Mercurial, but in the meantime I have to agree with Ted Tso:

“…the main reason why I’ve come out in favor if Git is that I see its potential as being greater than [that of] Hg, and… in the long run I think it has ‘more legs’ than Hg…”

Just to be totally up front. Ted did have some complaints about ease of use and documentation, but I think that since his post came out Git has made some really significant improvements in those areas.

Pidgins Aren't DSLs

2008-01-19T00:00:00+00:00

Piers Cawley just posted about Martin Fowler’s attempt to write a book about DSL’s actually, “internal DSLs”. Piers calls these “Pidgins” and I think it’s a pretty good term for them.

These are the sorts of languages where you don’t write a lexer or parser but instead build a family of objects, methods, functions or whatever other bits and pieces your host language provides in order to create a part of your program that, while it is directly interpreted by the host language, feels like it’s written in some new dialect. - Piers Cawley

Unfortunately ever since Ruby on Rails came out everybody (except Piers) seems to have forgotten what a real DSL is and it’s been driving me batty. Rails is not a DSL it’s a Pidgin. And it seems as if every post about writing DSLs since RoR has actually been a post about writing Pidgins. Which sucked because I was actually interested in writing DSLs but the only references I could find essentially said “you need to write a lexer and a parser, maybe generate an AST. Go use YACC or Bison”. Which didn’t help me one bit because while I knew what a lexer and a parser were I certainly didn’t want to reinvent the wheel about how to generate them and even now I’m still a little unclear on what an AST is under the covers.

I know what they let you do but not how they’re stored or accessed. Also, using YACC always seemed like it would involve journeys into unix obscurity and uber-geekdom. It may actually be trivial to use, I have no idea. And then, thanks to a friend, I was introduced to ANTLR.

OMFGWTFBBQ. ANTLR is the bees knees. It’s got a very nice IDE (ANTLRWorks) in which you write out your grammar using an EBNF-like notation. It will show you the Syntax Diagram of the rule you’re currently working on, and has an interpreter and debugger. Then, for most simple DSLs, you just hook a little code to be executed or returned at various parts of your grammar, click “Generate Code”, and you’re done. I’m not kidding. It’s that freaking simple.

ANTLR builds you a lexer and a parser from your grammar, inserts the native method calls you specified and outputs classes for you to stick in the app that’s going to be reading in the DSL. Now by default ANTLR’s focused on Java, but according to the docs you can also generate Python, C#, C++, and more. AND, if you’re writing a language that’s going to do some pretty heavy duty work (like Python for example) then you can also have ANTLR generate an AST for its parser to use which will speed up processing dramatically. I haven’t explored this part of ANTLR yet so I can’t comment on it. If you’re interested in writing your own DSL (a real DSL not a fucking Pidgin) I highly recommend that you grab the ANTLR book (it’s very well done) and start playing with ANTLR. If you’re not going to be leveraging your DSL from Java you may need to grab the latest version of ANTLR from SVN and you’ll definitely have to read some more of the docs on the site about making it output code in your language of choice.

I was able to get my head around ANTLR, write a grammar for my small language, and then hook in the code to generate a parser that actually did something in roughly one work day. Next week we’re having a brainstorming session at work to see just how far we can take this language and what systems we’d like to retool to take advantage of it. So please people, stop confusing DSLs with Pidgins and stop writing about Pidgin’s as if they were DSLs. There are so many good reasons to write real DSLs to help in whatever problem domain your applications are addressing, and no good reason to limit yourself to a Pidgin. Real DSLs, like SQL for example, can be leveraged from code in any language and be incredibly useful in their domain.

P.S. You should really go read Piers’ article. It makes some really good points and is covering a different, although related, topic to this one.

Convert textual RSS feeds into podcasts

2008-01-08T00:00:00+00:00

[EDIT] Odiogo is now some sort of Japanese Car site. As such, this page has been obsoleted by yet another cool proprietary product disappearing.

Odiogo will take your blog’s rss feed and run it through a text-to-speech converter so that people can subscribe to it as a podcast. It’ll, obviously, have the same quirks as any other text-to-speech converter and is, probably, limited to English but it’s a pretty nifty idea, even if their name is a total rip-off (Odiogo makes rss into podcasts Odeo manages rss feeds of podcasts). Also, they seem to have done a really good job with the intonation of the computer voice.

Ironically I found out about this from a deaf blog. Creating a podcast via normal means would be tricky for a deaf person (you’d have to edit exclusively by sound wave patterns), and, with no disrespect meant, people born deaf tend to sound a little off when speaking, and I would assume that most of them would lack confidence in their speech. I’m know I would. So, this is a great way for deaf bloggers to offer podcasts. Plus, the idea of listening to a deaf blog is just so wonderfully contradictory.

Follow your bliss, then write your tests

2007-12-27T00:00:00+00:00

_why suggested that

…chaos is an essential component of writing code. The system is too big for you to fathom. So you are always finding yourself in unfamiliar territory. And once you fathom the system, it becomes too boring and tedious to pay attention to details…

…Unit testing, in particular, is designed to reel in spontaneous hacking. It is like framing a picture before it has been painted. Hacking, at heart, will continue to be something of spontaneous order, something of anarchy, and the landscape of hacking is something which comes from human action but is not of human design.

I, as you already know, am a huge advocate of unit testing, constantly poking my coworkers to get off their asses and cover all of ours by writing the tests I believe our code so desperately needs…

But unit testing doesn’t have to be an either or proposition, and I am in full agreement with the essence of _why’s message. Some of the greatest hacks, like some of the greatest stories, have come from simply opening yourself up to possibilities and seeing where they take you. There are many good arguments to be made for Test Driven Development, but when I’m writing the really cool code, I frequently haven’t a clue where the methods will take me. But that’s not a valid excuse for blowing off tests. Sure, follow your bliss. See where it takes you, but, when you get there you will have time to look back upon your creation while it’s still fresh in your mind and decide if it’s something worth keeping. If it is, then it’s probably worth making sure it works correctly, especially if it was written in the midst of an endorphin high.

And… if, when looking back upon your creation, you think it’s good enough to share, is it not ever more worth testing, so that your entirely human errors don’t trip others up too? So that you can be sure that things really work for others the way you claimed they did? How inconsiderate of your fellow men would it be to foist untested and potentially buggy code upon them?

There is no binary in this world; not in code or computers. Beauty and motion lie not within the ones or zeros but the infinite layers of transition between them.

Don't be afraid to look like an idiot

2007-12-15T00:00:00+00:00

The other day I posted a rant about “Alphabetical != ASCIIbetical”, which, much to my surprise, got picked up in a couple places and brought thousands of readers. As with any post that gets thousands of readers, some of them are going to call you an idiot.

…I don’t know what you call this sorting order, but it most definitely is not alphabetical. Maybe you should make sure you aren’t being a dumbass before you climb atop your own soapbox of delusional self-importance. - Dave G.

And Dave’s right, about that part. It wasn’t “Alphabetical” sorting that I was after. I’ve heard it called “Alphanumeric”, “Natural”, and “Lexicographical” sorting. I haven’t a clue what the correct term is, or if there really is one for what I’m looking for^[1]. I knew going into that post that it wasn’t really “Alphabetical” because numbers aren’t in the English “alphabet”, but I also knew that most people would know what I meant if I called it “Alphabetical”. But does my ignorance of the correct word mean I should have not said anything? Or that I should have written some meek apologetic post saying “I’m terribly sorry I don’t know what the word for this is but you should probably be aware that…” Fuck no! Nobody would have read it, or if they did they wouldn’t really think about it. The fact that that post made it is far as it did was because I was willing to stand up on my soap box and call people on what I believe to be a pervasive problem.

But you know what? I could have been totally effing wrong. Someone could have commented “Masukomi, you idiot, it’s the Foo.alphanumSort() method and it’s been there for six years. All major languages have something similar.” or more succinctly “RTFM!” I would have looked like a complete ass for telling people they should “be ashamed” of themselves.

But even if that happened it would have been worth it because my ignorance on how to address this problem would have been removed AND I would have a nice trusted library to work with. Plus, because I have absolutely no problem with apologizing when my ignorance leads me to say wrong-headed things.

We are all ignorant of far more than 99.999% of the things that go on in this world. There are just too many of them. So don’t be ashamed of your ignorance. Embrace it. Try and recognize what you do and don’t know, and don’t be afraid to say “Hey, I don’t know wtf this is but I know it’s fucked up.” Maybe you’re right, maybe you’re not. If you’re right, improvements may result. If you’re wrong you’ll learn how it works. And don’t be afraid to say “I am such a freaking idiot sometimes.” because you are. We all are. It is only a matter of time before each of us says something that’s based in ignorance and makes us look like an ass. But so what? As long as you’re willing to say “Oh my God. You’re right. Thank you! I was an ass. I’m sorry.” It’s no big deal, or it shouldn’t be. If the person you’re talking to can’t respect you for owing up to your mistakes then they’re not someone you want to associate with anyway.

I should also note that I didn’t write that post, or any other, with the goal of getting lots of people to read it, but I do recognize that calling people out frequently results in more people paying attention, and engenders discussion. I posted it because I believed there was an issue that needed addressing. And I know there are a handful of people who read this who might want a heads-up, and / or be able to empathize with my frustration. I know that I find it comforting to read other developer’s blogs and see them struggling, or being annoyed with, the same things I am.

This is my soap-box. Sometimes my leaflets are wrong, but I’d rather risk being wrong than never say anything.

Looking for: a sorting algorithm that treats numbers in strings as the numbers they represent and not the digits that happen to be used to represent them. 1000 is not four digits that need to be sorted. It’s a representation of the number one-thousand which needs to be sorted correctly relative to other digit based number representations that get mixed in with letters of the English alphabet. ↩

Alphabetical != ASCIIbetical

2007-12-11T00:00:00+00:00

[BEGIN RANT]

Partially this is a case of Java community being populated by idiots, but people seem to be wholly ignorant on this issue in other languages too. Google for java alphabetical sorting capitalization or any combination of words you can think of that might get you an algorithm that sorts a collection alphabetically. You will find hundreds of wrong responses and no correct ones. Most of them say to use the Arrays.sort(..) or Collections.sort(..) methods. But both of those use natural order (or ASCIIbetical as I like to call it) not alphabetical order so 1 is followed by 10 not 2 and things starting with a capital letter aren’t beside things with the lowercase version of the same letter.

Some people think of calling .toLowerCase() on everything first so that at least they eliminate the latter issue but I have yet to see a single example of an actual alphabetical sort in Java (or any other language I’ve checked).

Silly me, I just figured that alphabetical sorting was such a common need (judging by the number of people asking how to do it I’m not wrong either) that I wouldn’t have to write the damn thing. But I didn’t count on the stupid factor. Jesus Christ people. You’re programmers. You’re almost all college graduates and none of you know what the fuck “Alphabetical” means. You should all be ashamed.

If any of you are using your language’s default sort algorithm, which is almost guaranteed to be ASCIIbetical (for good reason) to get alphabetical sorting you should proceed to the nearest mirror and slap yourself repeatedly before returning to your desks and fixing your unit tests that didn’t catch this problem.

[END RANT]

[Update] There is apparently one person who knows the freaking difference. Someone get that man a cookie!

[Update 2] Alphabetization/Alphanum/whatever you want to call it is entirely language dependent. Obviously an algorithm that works for English is unlikely to work for other languages. Attempting to create a generic purpose one that handles special characters of multiple languages is brain-dead because there may be other rules that affect alphabetization in them.

[Update 3] Jeff Attwood points out that this is called “Natural sorting” not “Alphabetical” sorting. Unfortunately the Javadocs for Collections.sort(…) talk about sorting things according to their “Natural ordering” which may be technically true but just adds to the confusion.

SSCM 0.4 Released

2007-12-09T00:00:00+00:00

Some of you may be interested to know that SSCM v 0.4 has been released. Notable changes: supports move operations, fixed a bug with perforce support, allows you to live dangerously and just accept all detected changes into the repo without asking.

The two things I’d like to get in there now are branching and merging all the known repos with one command each. Should be relatively trivial for the distributed clients, but the centralized ones will be a little work. Anyone feel like pitching in?

Disovery coding through tests

2007-12-08T00:00:00+00:00

Testing as a process of discovery

The other day a coworker said,

Some times you get situations where the specification for the unit or module you are writing just are not available. The code writing is a discovery process as much as anything else. Moreover, some of the packages and methods being called don’t have predictable or documented behavior. That’s ugly and horrible, and I don’t know how that’s allowed, but nonetheless, from the perspective of someone who wants to do unit testing in such an environment, can you give any tips? I mean, do you mock up approximations to what you think these external things should be doing if you really don’t know what they are doing? Do you do your best, updating mocks and tests, “in the face of adversity”?

Really we’ve got two questions here: How do you address the desire to write tests when you don’t know what exactly you need / want yet?

How do you handle external libraries where you’re just not confident in what the expected behavior is?

How do you write tests for something when you don’t know what

exactly that something is yet?

Let’s look at coding a simple User Story and Acceptance Test and tackle them in a way that allows for maximum flexibility and exploration while coding.

The User Story (for a cash register): Keep a running receipt with a short description of each item and its price.

Its Acceptance Test: Setup: The cashier has a new customer. Operation: The cashier scans a toothbrush for $2.25. Verify: The receipt has a description of the toothbrush and its price.

Let’s assume we’ve written absolutely no code. But we want to take a test driven approach and we don’t want to write tests for the bits we haven’t really figured out yet, because, well… we haven’t figured them out and we want to keep the possibilities open. So let’s write the bare minimum required to test this. From the user story and acceptance test above we can tell exactly what classes and methods we’ll need to write and test. So we’ll need:

ICashier (cashier interface)
ICustomer (customer interface)
IRegister (register interface)
IToothbrush (toothbrush interface)
IReceipt (receipt interface)

As for methods we’re going to need:

IToothbrush

-   getPrice()
-   getShortDescription()

IRegister

-   scanItem()

IReceipt

-   addItem()
-   print()

Now, there are three approaches we could take here:

Write a full interface and tests for all of these with all the other methods we expect them to need (like a getUPC() method on IToothbrush)
Write an interface and tests for all the methods we know we’ll need (the ones listed above)
Write an interface and tests just for the items we are using RIGHT NOW.

I’d been doing the latter for a little while and was pleased as punch to find out, when listening to a podcast interview with Martin Fowler, that this is exactly what he’s been doing lately too. I think that this is the best possible approach because it keeps you focused on what you actually need, it gives you much smaller and more quickly obtainable goals, and it gets things working faster. If you took this approach you might proceed something like this (again, assuming no code has been written at this point):

Write the IToothbrush interface.

Add the method you want to address first (get price)

Write a unit test for that method.

check that a toothbrush never has a null price
assert that the price is going to be a decimal number
check that the price is never less than zero.

Now that you’ve got a properly failing test you write a Toothbrush implementation with just a getPrice() method.

In your Toothbrush implementation you’d probably have a no parameter constructor that set up a default price of $0.00, or your getPrice() method would be smart enough to guarantee a non-null price was returned, either way, your test would pass.

I’d say just repeat for each item in the list but it’s not quite that simple. When you get to the getShortDescription() you’ll probably have a test that it’s not null, but we’ve got no method in our spec for actually setting a description so we can’t make an implementation where that test passes unless we create a method to set the description or allow it to be set in the constructor. XP people advocate doing the simplest possible thing that will work, which in this case makes sense and would be to set it in a constructor. So, now we’ve got to add a unit test for the constructor, and it’s reasonable to think that we should set the price in the constructor too. So, in order to really test the getShortDescription() method we also need to write a test for the Toothbrush constructor. So, testing getShortDescription() leads us on to the next bits we need RIGHT NOW.

Add the getShortDescription() method to the IToothbrush interface

Add the testGetShortDescription() unit test

check that the description isn’t null

check that the description isn’t empty

check that it’s actually “short”

this could, conceivably lead you to want create a generic Item interface that has a validateShortDescription(...) and / or truncateDescription(...) method, but I’d recommend holding off until you finish with the items required for this acceptance test. While a good idea, and something to plan on, it’ll be nicer to actually get through this initial acceptance test before diving down the rabbit hole of nice-to-haves.

Add testConstructor() method to your ToothbrushTest test case

test that we can instantiate an object with a valid price and short description
test that we can’t instantiate an object with an invalid price and valid short description
test that we can’t instantiate an object with a valid price and invalid short description
test that we can’t instantiate an object with an invalid price and short description
test that the price and short description come out through the two getters we have in our interface.

Add a constructor that takes a price and a short description to Toothbrush

Add getShortDescription()`` to toothbrush``.

By taking such a focused approach we haven’t constrained what could happen with any of the other functionality in toothbrush. Let the emerging specification define if you need a getColor() method on your toothbrush. If it’s not there now and none of the methods you’ve worked on so far have implied the need don’t go there at all… otherwise when you finally do get there you’re likely to find that “blue” wasn’t an acceptable thing to for it to return at all what it really needed to return was an array of RGB values: [0,0,255]. My advice would be to add a comment to the interface noting that you think a getColor() method might be useful and what you think it should return, that way as you keep coming back and updating IToothbrush you’ll keep that in mind and won’t forget or accidentally code yourself into a situation where it’s difficult to derive the color.

How do you handle external libraries where you’re just not confident

in what the expected behavior is?

The same approach we just discussed for writing the unit test should guide what we mock up of untrusted external libraries AND how we proceed in working with them. Normally I try and work under the assumption that the libraries I depend on are dependable and never write unit tests for them, but sometimes you need to find out what you can depend on or what exactly some methods do because they’ve got no docs. So first you look at the library and ask yourself what methods, based on your limited understanding, do you absolutely need to use? Then you write unit tests that test that confirm that when using it the way you plan on using it it does what you think it would do. You don’t have time to test their entire library or all the possible edge cases for their code so you just test that the bits you will be using work the way you think they do when called in the way you’ll be calling them. For example if your code can’t possibly pass in a negative number then you don’t need to worry if their classes correctly handle negative numbers or not. You just need to make sure that the class handles the full range of numbers you could be passing it correctly and nothing else. Now you know that it does (or doesn’t do) what you expected and you know some things that you can safely pass it.

Depending on how untrusted the library is you may also want to make sure that everyone in your group knew to not use anything in that library that they didn’t have a minimal test case for.

Mike Clark wrote an article wherein he describes a similar approach but for a language instead of a library. His idea was that when you start learning a new programming language you just don’t know how exactly things work. You assume things may work one way but that’s not always the case. He set out to write test cases for all the things he learned how to do in the language. This had a few good side effects: it got him familiar with the testing framework of the language, it got him to actually use all the language constructs (doing is far better for memory retention than just reading) and he now has a test suite of his mental beliefs of how the language works. If a new version of the language comes out he’ll know right away if his mental model of how things works needs to change because his tests will fail.

Pretty graphs you can't show customers

2007-12-05T00:00:00+00:00

This projects has been rolling around in my brain for a while but I haven’t tackled it yet because I have too damn many other projects in process. So I’m putting it here in the hopes that maybe someone will pick it up and run with it.

I want to put together a collection of javascript based graphing tools that generate pretty SVG graphs of your data in a way that’s fun to look at for people who have to work with it every day but not necessarily something you’d ever want to try and explain to a customer. I want to do this because we deal with a crapload of really interesting data at work, but a lot of it is just internal and only of interest to geeks. Also I’d like a visually interesting way to keep an eye on the status of our systems and the data flowing through them. Stacked bar charts and line graphs get old fast.

Here’s my first idea:
Imagine you have a wooden beam. Hanging down from that beam are strands of ivy. Each strand of ivy represents a thing you’re watching (customer actions, a product’s sales, a website, whatever). The length of the strand represents the number of hits to / quantity of that thing. Now, imaging that that thing has some quality goodness/badness, happyness/saddness, good profit margin / bad profit margin, etc.. As each item is added to the end of the ivy the ivy curves to the right or the left based on how much goodness/badness (or whatever) this item contains. If every item you added had some goodness the line would angle off to the right. Bad items would, similarly, curve it off to the left. If you have a relatively even distribution your lines will simply curve back and forth in a wavy line like a real piece of dangling ivy.

Originally the ivy was just a metaphor to help people visualize it but brainstorming with John brought up the idea that when an item’s goodness/badness crosses a threshold a leaf is added to the line. Maybe the leaf could have an onClick event to take you to a page with details. Maybe the leaves could be different colors or sizes to emphasize how important the event is (relative to how far over the threshold it is).

Imagine how cool it would be to have a whole collection of fun little graphs like this to keep an eye on your data! The beauty of it is this is actually really easy to implement. I just won’t let myself do it until I release the next version of SSCM out the door (with merging and all that goodness), and release the source code to TicketEverything!.

There’s a good tutorial on creating svg objects and you could pre-generate complex items, like the leaves, with Inkscape, or any other vector based illustration app that can export svg.

Is it testable?

2007-11-06T00:00:00+00:00

Is it testable?

Apparently some people are having trouble with determining if the code they’ve written is testable. So I’ve put together this flow chart to help you navigate through this complex decision making process. The image is distributed under the WTFPL license so please feel free to use wherever and however you want.

Should I test it now?

2007-11-06T00:00:00+00:00

Yesterday’s flow chart was designed to help with the complex issue of determining if a particular piece of code is testable. Today’s flow chart helps with the equally complex problem of determining when to write, or run, your tests. Like the last one it is distributed under the WTFPL license so please feel free to use wherever and however you want.

Unit Testing 101 Presentation v2

2007-10-24T00:00:00+00:00

I just put together a new unit testing presentation for the folks at work and you. Although it may need to be edited here and there for your coworkers… maybe mine too….

Anyway, Unit Testing 101 (v2) requires Firefox and I recommend you move your mouse up to the top edge and click on the icon to the left of the slider where you’ll get a menu of all the chapters and slides. Yes, there are a lot of slides, but it’s Takahashi method so they go really fast.

As always feedback is not only welcomed, but encouraged..

Branching for atomic patches and cherry-picking

2007-09-10T00:00:00+00:00

The best thing about Distributed Source Control Managers ( IMNSHO ) is how quick and easy it is to branch and merge. The problem is that most of us cut our teeth on centralized systems that couldn’t even hope to take advantage of cherry picking, which is, in short the ability to take a single patch out of the middle of a sequence of patches, or every patch but one from a sequence. Just imagine knowing that there was a bug introduced in a specific patch and being able to prune it from your repository but not any of the patches around it. Or, plucking one little feature out of a mass of others that should wait until the next release. You can, but if you don’t make the effort to keep your patches as atomic as possible you’ll find that that patch you want to remove or extract is dependant upon another one, or more, that you may not want to involve.

Which brings me to my hierarchy of branches. Now, I’m still working with this concept so if you can improve upon it please let me know. The idea is that most of the time you’re not adding one little atomic feature, you’re adding a number of features that can be conceptually clustered. So, you make a branch for the cluster, but do no actual development directly in it. Then you branch from there for each atomic feature (be sure to scroll down to the graph below the fold here).

While you are working on a feature feel free to commit as often as you want. Check in broken or unfinished code. Pollute that branch’s revision history with as much crap as you want. Do whatever it takes in there to make sure you can roll back to any point you want, and never loose work. Have a cron job check in any uncommitted changes to this micro-branch every night in case you forget to. It doesn’t matter because you’re not going to push the revision history back up to the branch for the feature cluster. Instead you diff from the branch point and patch the cluster branch and commit it. This way you’ve got one atomic revision with the combined changes for that one small feature and if you need to prune it for any reason (maybe it’s broken, or decided you should wait on adding it) you can easily do so without worry of dependencies. Your micro branch has all the details of the feature evolution if you really feel you need it, but most of the time you’ll just delete it after creating the patch and applying it to the cluster. Eventually the cluster will have all the related features completed and checked in and it’s revision history will consist of a series of atomic commits. The cluster’s revision history is then pushed upstream to your personal development branch. Occasionally you’ll have a small feature to work on with no related ones to cluster. Just branch from your personal development trunk and then patch back into it as you would have into a feature cluster.

Bugs should be treated in exactly the same way as features, although you’ll find yourself with many more small bugs that don’t need a parent cluster and can just be worked on in their branch and then used to patch your personal development branch.

People working farther upstream now have the option of cherry picking features, feature clusters, bugs, and bug clusters, and will be able to do so with much less worry of getting caught up with patches that depend on other patches.

Code Underwriters

2007-08-31T00:00:00+00:00

Code Underwriters

Lloyds of London is able to do what they do thanks to the concept of underwriters. The simplistic version is that a risk is spread amongst a group of underwriters. If nothing goes wrong they get a cut of the profits relative the the percentage of the risk they took on. If things go wrong they take pay for whatever portion of the risk they agreed to take on.

It occurred to me that being a software developer is in many ways like being an underwriter, except, in most cases, without the big payoff. You see, each developer on a project tends to take responsibility for a portion of the code-base. We talk about spreading this responsibility around by increasing the bus count but the fact of the matter is that this rarely ever happens.

Most code-bases tend to be divided into sections each with a bus count of one or two. When something goes wrong the developer(s) responsible for the responsible piece of code are generally expected to do whatever it takes to make it work now. In some cases this means being woken at 3AM, hauling your butt into work, and remaining there until the bug is squashed. Just like an underwriter at Lloyds, the more risk you take responsibility for the more screwed you’ll be when things eventually go wrong. Unlike a Lloyds underwriter we don’t get a cut of the profits relative to the amount of responsibility we take on.

It is, obviously, quite debatable just what portion of the profits would constitute a “fair” portion of the profits, but in general we just get paid our salary. The first employees of a company frequently do get the payout, but most developers aren’t them. I’ve also heard tale of magical things called “bonuses” but it seems rare that developers actually get them or that they’re enough to buy you more than one box under the Christmas tree if you do see them. So, for the most part, we can treat these things as non-existent, which brings us back to our salary. Which is, in many ways, like insurance. While you don’t get a huge portion of the profits, you do get a guarantee of sorts from your employer that you’ll continue to get paid well even if things don’t work out. Which makes your employer a fellow underwriter in the development of your software.

There are three things that need to be underwritten in any given software project: The effort that will be required to fix things when they break, the money that will be required to finance the fixing of broken things, the money that will be required to finance the creation of potentially broken things. There’s also the money required to finance the project(s) needed to compensate for the projects that didn’t bring in more money than they cost to create and/or maintain but that’s a bit too recursive for this example.

From a financial standpoint, I think it’s understandable why developers don’t get a larger cut of the profits. While the worst case scenario for us is rather unpleasant, it’s also very short lived. But, we still get paid quite well during all those months and years when the project wasn’t bringing in anything notable, and even when it tanks.

Unfortunately developers are human, and humans are emotional creatures. We tend to not pay attention to the thing that are present in our every day lives, like regular paychecks, and focus on the atypical large events. Like having a fear of flying. If you’re afraid of flying then your brain is not paying fair attention to the boring everyday fact that millions of people are whizzing about the skies without any problems. It’s just that when a plane goes down it does so in such an atypically spectacular way, and that’s what we focus on.

So I’d like to propose a couple things for my fellow code underwriters to help mitigate our risk, and thus minimize the atypical large events and bring our perceptions more in line with the small portion of the risk that we’re actually taking on.

Write your tests: unit, integration, and system. It’s like getting underwriters for the risk you’ve underwritten yourself. They’ll put in the hours of work patrolling your code and making sure it isn’t broken so that you don’t have to put in the hours fixing the code later.
Increase your bus count. The bigger the bus count the more people you can throw at the problem so as to fix it faster and / or the greater the likelihood that the problem can be assigned to someone other than you.
Practice Defensive Development. You don’t wear a bicycle helmet because you’re afraid of falling over. You wear a bicycle helmet because you’re afraid someone else will make you fall over.

Defensive Programming 101

2007-08-30T00:00:00+00:00

Defensive Programming 101

For any given programmer the following statement should always be treated as truth:
My code sucks, but your code sucks more - Dave Astels [deleted post]

Good version control habits and test coverage will get you out of most jams related to your own code but we rarely write apps that are comprised of just our code. There are almost always libraries from other people code that you’ll include to save yourself from having to re-invent the wheel. Obviously you don’t want to start writing unit tests for code from other projects (you’d never finish) but there are some basic steps you can take to minimize your chances of failure.\

Untrustable code

The fact is that no matter how good your code is you just can’t trust anything you didn’t write. Furthermore, you can’t trust what people will do with their code.

You can’t trust other people’s code because it rarely has good test coverage or documentation so you can’t be sure if it is broken, or if it works the way you think it does. And unless it’s a huge open source project you can generally assume that not many people have looked at it either.

You can’t trust what they will do with their code because it’s their code not your code. Whoever “they” are they’ve got their own ideas about where it’s going next and how to get it there, and they may not jive with how you want to use it.

Your project must never rely on the successful build of another project, because it may depend on the successful build of yet another project, or even worse, end up in a sea of circular dependencies.

The results of another project’s “current” build should similarly not be trusted. It’s probably not tested well, the implementation could change completely tomorrow, and it may just be buggy. Whenever possible only use release builds. If you can’t use a release build at least try and use one that other people have had success with for a little while.

Untrustable people and environments

Even if you take all the steps possible to defend against other people’s bad code there’s still the fact that humans are incredibly fallible.

You can not assume that the correct libraries will be where they’re “supposed to be” or that the version that is there is 100% compatible with the version you coded against. Even if it is you can’t assume that someone won’t go and “upgrade” it to a version that isn’t compatible. Always bring your own libraries and keep them local to your app. Disk space is cheap and unless you’re on an embedded device it’s better to bring along copies of the libraries you coded against. It doesn’t matter if the ones you bring along are completely outdated relative to the latest version; what’s important is that you can guarantee that the version you coded against works.

Your project must never rely on another project or its libraries. Period. You can not assume that the other project will be on every box your project is installed on or that the version on those boxes be in a state that is either working or compatible with yours. Take whatever you need from their project and copy it into yours. If you need libraries that it generates or contains, copy them. If you must use source code copy it but be very careful. You don’t want to end up forking it and having to maintain your own version. Instead, copy it in but do whatever it takes to make it blazingly obvious that the code should not be edited unless absolutely necessary. When you do copy over make sure to copy the smallest amount possible. If you can, build it into a separate library file that you can update when the need arises.

You also can’t assume that someone isn’t going to screw up a config file, and that includes you. Just like libraries, bring your own. Maybe you’re working on a webapp which has some global configuration file that’s shared by other webapps on the same box. Don’t trust it. Yes use it so that you’ll be in sync with the other apps but since you can’t guarantee that some sysadmin won’t accidentally delete or change some piece of it that’s crucial to your app bring along a fallback with some bare bones settings that should work under any circumstance you can prepare for. This is exactly why your operating system has a “safe mode”. The operating system developers know you can’t trust the users not to accidentally screw something up or install an app that screws something up. Have a fallback.

If your app is configured differently for different deployments (maybe you’ve got a dev box, a QA box, and a production box, each with their own different database settings) then have a separate config file for each deployment. Don’t have a single config file with different sections that get commented or uncommented based on which deployment it is. You simply can’t guarantee that you, or your coworkers, won’t forget to comment or uncomment the right bits when deploying.

99 Lines of code on the wall...

2007-08-20T00:00:00+00:00

99 lines of code on the wall.
99 lines of code.
You look around, refactor it down…
98 lines of code on the wall.

98 lines of code on the wall.
98 lines of code.
You look around, refactor it down…
97 lines of code on the wall.

Or, alternately

    function singVerse(numLines){
    
        if (numLines \> 0){
            document.write("" + numLines + " lines of code on the wall.\\n");
            document.write("" + numLines + " lines of code.\\n");
            document.write("You look around, refactor it down...\\n");
            numLines -= 1;
            document.write("" + numLines + " lines of code on the wall.\\n");
            singVerse(numLines);
        } else if (numLines == 0) {
            document.write("Totally bug free code on the wall\\n");
        } else {
            document.write("Need more tests for the code on the wall.\\n");
        }
    }
    singVerse(99);

The best argument for compiled languages

2007-08-19T00:00:00+00:00

I keep thinking back to a short comment at BarCamp Manchester in the Unit Testing talk. When asked if anyone had written an app with 100% code coverage the guy beside me raised his hand. Now I’ve been advocating for a while now that it’s essentially wasted time to bother testing your getters and setters, and when I mentioned something to this effect he said, “How do you know you haven’t made a typo in a variable name?”

…

My mind just kind of spun for a second on that one for a second. I guess I’ve been in Java land for so long that the concept of having such a thing exist for more than a few seconds just doesn’t enter my brain. I mean first off, the IDE will bitch at you, because it can’t compile it, and even if you’re not using an IDE the compiler will bitch at you. I just responded that “I use a compiled language so that’s never something I have to worry about.”

Now, as many of you know I’m still a huge fan of languages like Perl and Ruby so I work in them fairly often. And I still say that it’s not worth your time writing tests for your getters and setters even in them because unlike Java these languages don’t have IDEs that happily spit out a getter and setter for every variable in your class regardless of if you need it or not. As a result, you only create the ones you need. And if you need them, then they’re going to be used, and if they’re used, you can pretty much guarantee that you’ll see the results of any typos show up really quickly when you test the methods that actually do something noteworthy, because they’ll use the getters and setters (otherwise why did you write them?) and either blow up when the compiler hits them or not work as expected, which is why you’re writing a test.

Who Cares About Performance

2007-07-13T00:00:00+00:00

Justin Jamesasks why nobody seems to care about performance anymore. He talks about how the performance hit you get from using a “slow” language directly translate into increased hardware and electricity costs just to maintain the same kind of performance you would have had if you’d used a “fast” language or spent more time optimizing your code.

All his points are good, but they’re also all irrelevant. You see, for most applications the performance hits you get from slow languages or non-optimized code just don’t matter. Your system will still be responsive enough that no-one will be bothered. People don’t care because it just doesn’t have any noticeable effect on the end product and some languages make coding far more enjoyable and productive. Their productivity gains far outweigh performance hits that are almost unnoticeable by end users.

This is exactly why Ruby on Rails is kicking ass and taking names. It’s not terribly performant. Ruby is unquestionably a slow language. The huge infrastructure of RoR makes it even slower… but it makes developers incredibly productive and the resulting sites are “fast enough”, especially if you take the time to host them with the correct tools.

Before I started working at Akamai I was one of those people who just didn’t care about performance and you would have been hard pressed to convince me that coding for performance really mattered for anyone who wasn’t coding something like a router or OS Kernel. That was until I started being presented with numbers that just blew my mind. Our group is currently just looking at a very small subset of the data that flows through Akamai’s pipes and we still have to process hundreds of millions of log lines per hour. I remember amazed by a graph showing Yaws handling 80,000 concurrent connections per second while Apache keeled over at 4,000. “80,000!” I thought. “With performance like that we could blow through those logs. Now lets see… Hundreds of millions divided by 80,000 is…. oh… not… even… close.” Erlang could be an incredible tool for some of the tasks here but even a language with incredible concurrency support like Erlang isn’t, alone, enough to deal with the sheer scale of things here. You have to care about performance at Akamai. It’s not just what we sell to our customers, it’s the only way for us to survive what we sell to our customers. I still believe that for most developers language and framework choices are more important that worrying about performance, but now I understand what a sheltered view of the world I had and how that view fits into the larger world. Joining Akamai combined with a recent series of small performance related events has opened my eyes to just what a dramatic choice it can make when you do start taking performance into account. Yes things are generally “fast enough” but you will kick your competitors asses when their products are “fast enough” but yours is instantaneous. “Lightning fast” is a more compelling feature than you would ever guess. It’s pointless market-speak in a feature list but once you experience it it’s damn hard to walk away.

Java: The Mediocre Date

2007-07-01T00:00:00+00:00

I’ve been programming in Java professionally for years now, and while I’ve become good at it, it’s never grown into a language I’ve been passionate about. It’s powerful, has tons of good libraries and tools but… It’s like going out on a nice date with someone but having no desire to ask them out for a second. You wouldn’t mind another dinner with them, and since you’ve got a common circle of friends you probably will, but you’re never going to get the butterflies in the tummy when you think about seeing them again. I’ve met a lot of people who’ve been out on a date with Java, and so far none of them seem to have tummy butterflies either.

Veritable religious wars break out over languages like Python, Perl, and Ruby, even C++ has it’s zealots but Java? People use Java because it’s stable, it’s got good tools, it can scale, and when you mention it to your customers they’ve heard of it and can nod as if they have a clue. Java is the Big Blue of languages. “Nobody ever got fired for choosing IBM.” they would say. People learn Java because they can get jobs in it. People learn Ruby because she’s hot, and when you start fooling around she makes some incredible noises.

I’m sure Java and I will keep having dinners together but we both know it’s just business. In the meantime I’m going to see about getting another date with that Perl chick. Everyone else has written her off like Hermione, because of her frizzy hair. But I know the truth. She’s gorgeous when she wants to be and under all that frizz lies a dam sexy brain.

Why you shouldn't dismiss Perl so quickly

2007-06-23T00:00:00+00:00

A reader commented that

…the syntax of Perl is so hideous and mysterious it produces unmaintainable code (I know having maintained a multi-tier Perl webapp).

And, sadly, he’s not the only one that shares that belief. Perl has been written off by many talented developers because essentially every piece of Perl code they’ve ever encountered is, well, crap. But, when you get right down to it you’re basing your opinion of a language based on what people write with it. It’s like saying that the English language sucks because there are so many vile and crappy things written in it.

Perl is quite possibly the most flexible modern programming language. But, as with anything, great power comes with great responsibility. Perl geeks believe “There’s more than one way to do it.” where as Python geeks follow a slightly different way

The Python philosophy [is] that “There should be one– and preferably only one –obvious way to do it.” That is, I can write a multitude of programs that do exactly the same thing in python in different styles and methods, but there is only one way that is Pythonic… What Python does is emphasize a standard quality and method for doing various tasks instead of just hacking it together. - Alexander Botero-Lowry

So, if you’re working with undisciplined idiots, there’s no question, choose a language like Python. But, what happens when the Pythonic way of expressing an idea doesn’t quite mesh with the idea you are trying to express?

In Python you’ll generally just have to deal and find a way of coding what you wanted to express that isn’t very “Pythonic” or that ends up being more verbose. In Perl there’s probably another handful of ways to express the same concept. It’s like always having a kick ass thesaurus available.

Yes, this means you’ll eventually speak with words other people aren’t familiar with yet or do things in ways others may not be so kosher with. But Perl is all about letting you work in whatever way / style feels best to you. This can obviously lead to situations where different people don’t agree with how something’s been implemented even if it’s very sensible. What you need to keep in mind though, is that the readability and maintainability are the direct result of the experience, discipline, and consideration given for other developers by the person writing the code.

The problem with Perl is that it makes it incredibly easy for a person with no discipline or limited experience to express themselves badly… very badly. Combine that with the fact that it’s so easy to get something that works in Perl when you haven’t a clue how to code well in any language and the problem just compounds itself.

It is very easy to write readable and maintainable Perl code, but to do so requires discipline. It’s the difference between giving a kindergartner a generous palette of oil paints and giving one to Michelangelo. The paint is capable of producing a masterpiece in his hands, but the kindergartner is just going to make a finger-paintey mess. Don’t blame the paint. Likewise, don’t blame Perl for the crap that idiots and newbs produce with it. Instead, go buy them a copy of Code Complete and make every-one’s life a little bit better.\

And yes, before anyone mentions it, Perl’s OOP tools seem a bit hackish and can feel bolted on. But once you start to understand how they work they’re incredibly flexible and let you do things that you couldn’t hope to touch in Java. And yes, Perl does have a number of funky looking operators like $# but you really only need to know a few of them. If you don’t like them you’ll be happy to know that they weren’t too thrilled with the number of them when they started planning Perl 6. Speaking of which, there are some truly awesome things that have been put into Perl 6 and they’ve cleaned up a lot of the syntactic issues people frequently bitch about…but there will still always be more than one way to do “it” so you’ll still be able to write ugly code if you don’t know any better.

A note about Python: Python rocks. It just has a very different mindset to Perl. Python isn’t a language for “undisciplined idiots” as I implied above, but it is a better language to have undisciplined idiots coding in because while their code may still be crap you’ll at least be able to read it because of the formatting limitations and the intentionally limited number of choices for how to do any given thing.

O'Reilly: a publisher with a brain

2007-06-21T00:00:00+00:00

O’Reilly has just agreed to assign

…the full copyright in the book “Perl 6 and Parrot Essentials” to The Perl Foundation. The text is out-of-date, but can be updated much more rapidly than it can be rewritten from scratch.

Three cheers for O’Reilly. I wish more publishers would stop being so damn stupid when it comes to the copyright on old books that they have no intention of reprinting. Especially geek books which are frequently outdated and no-one would want them if they were reprinted (not rewritten).

Geeks Don't Touch

2007-06-19T00:00:00+00:00

Have you noticed? Geeks don’t touch. I noticed this in myself a while ago, and have since been watching other geeks to confirm my theory. I was at the Boston Ruby User’s Group a week ago and essentially no-one touched. When two geeks are introduced, unless it’s a somewhat formal introduction like an interview, we don’t shake. When we encounter each other, or go our separate ways, there’s no casual touching. Many will actually wave at each other in greeting from a few feet apart so as to avoid the simple touch that normal people would expect. I think this is a byproduct of the fact that so many developers are autistic regardless of if they realize it or not.

…Some may even be pathologically sensitive to loud noises (as some people with AS have hyperacusis), strong smells, or dislike being touched; for example, certain children with AS exhibit a strong dislike of having their head touched or their hair disturbed while others like to be touched but dislike loud noises. - Psychology Wiki

I wouldn’t say that most geeks are “pathologically sensitive” to touch but please don’t think that the geek in your life is being antisocial, or intentionally rude, or anything like that if they eschew the common social touches. I think that most of us just don’t like to touch anyone who isn’t particularly close to us emotionally.

Programming books for newbs

2007-06-14T00:00:00+00:00

If you’re reading this blog there’s a fair chance you’re a programmer and that means that from time to time you’ll encounter people who want advice on leaning how to program. Unfortunately, it’s hard to point them in the right direction because we generally don’t want to spend the time to teach them ourselves and even if we did most of the learning to program books just plain suck.

So, I’d like to recommend two books. The first is Learn to Program by Chris Pine.This is the best intro to programming that I’ve ever seen. It’s not concerned so much with how to do things in a specific language as it is with teaching people the basic principles of programming although it uses Ruby to do so. It’s based on a series of tutorials that are still online but have been improved on, and expanded upon greatly in the book.

There’s also JavaScript: A Beginner’s Guide by John Pollock which does a surprisingly good job of teaching basic programming concepts with JavaScript. It’s not that JavaScript makes this hard, it’s that most of the books get sidetracked on doing html specific things instead of actually teaching people to program. Yes, the Rhino book (JavaScript: The Definitive Guide ) is undeniably a good JavaScript book but it’s not good for people who don’t know how to program.

What developers can learn from ancient stadium builders

2007-06-04T00:00:00+00:00

I just read an article that compared the crowd management techniques used in the stadium in ancient Pompeii to modern techniques, and while it’s an interesting read all by itself, and I do recommend you read it, I got to wondering about how similar ideas would apply to software design.

Build a bigger bathroom.

He’s actually talking about the bathrooms and concession stands but there are a couple applicable bits here: The bathrooms are an incredibly important part of any stadium. They also have nothing to do with the core purpose of a stadium. Keith recommends three things in regards to their creation:

Keep them separate from the main stadium because mixing them in just causes congestion and increases anxiety.
Make them big. In a stadium environment your bathrooms are generally accessed in waves. The average number of people who use it per minute may be quite load and lead you to think you don’t need big ones, but the usage pattern is anything but steady.
Make sure you have plenty of space to get to and from them for the same reason you need to make them big.

The software analogies are fairly obvious. Most applications have support systems that are completely unrelated to the core function of the app. Lets look at Last.fm for example. They’ve got an underlying system(Audioscrobbler) that manages the incoming data from peoples music apps. Without this, Last.fm wouldn’t work but nobody says “Oh I want to use Audioscrobbler.” Hell, most people don’t even know what it is. No, people say “I want to see what the people I know are listening to.” or, “I want to share my playlists with people.”

Last.fm just wants to show people lots of interesting music stuff so that they can sell advertising and convince you to buy things on Amazon for which they get a cut. If they could accomplish this without having to write plugins for music apps or tons of bandwidth costs from people sending in track data I’m sure they would.

In the meantime though, Last.fm has to be careful that there’s a big pipe between Audioscrobbler and the web site. It needs to be separate because it’s NOT the core product and issues that effect one shouldn’t have a negative impact on the other. They’ve also got to make sure that the Audioscrobbler servers are big enough to handle the surges. If not people will stop using the main site. They’ve also got to be easily accessible. This means good APIs so that other people will find it easy to implement and big pipes so that when the morning rush hits they’ll be up to the challenge.

At Pompeii, he says, “they clearly planned for the rush that would occur at the end of a spectacle. You had the same human needs”—to visit rest rooms—“but the layout and design made the whole dynamic of moving to and from much better.”

Separate Queues from Promenades.

This is actually tied to the bathrooms and concessions, but the point is that you don’t want the traffic that’s central to your app getting bogged down by data that’s queued up for a completely different, but related, system. This doesn’t just apply to bandwidth and cycles though. It also applies to design. Break your functionality down into appropriately separated bits. You shouldn’t have a method that handles moving objects around two sections of the system when it’s much more sensible to have one for each.

Open Open Open!

Here’s a challenge: Look for areas of Pompeii’s stadium where bottlenecks might occur, where the crowd could overwhelm a space. Still says you won’t find them. Seats are at the optimum viewing angle, and seating “packing densities are to comfort, not cost.”

Similarly it’s important when writing code to keep in mind that functionality might expand beyond your original intent. Keep that in mind and you’ll not have to worry about being straight-jacketed into convoluted or restrictive pieces of code to implement new functionality. Make sure your code has room to breathe.

Build a Big Road

I think this mostly speaks to bandwidth and processor cycles and remembering that we are rarely so lucky as to have constant usage patterns.

Limit Corners

Modern stadiums often maintain the oval seating but then put blocky concourses around it. They also use switchback walkways and stairs. All of that creates corners. Corners force people to slow down and encourage congestion. Pompeii’s concourses were elliptical; few corners exist to slow people down. Still says this also evened out flow to the vomitories [entrancies piercing the banks of seats of a theater, amphitheater, or stadium] as people could, like liquid, choose the path of least resistance easily without interrupting their pace.

To me this speaks to good APIs and good GUIs. Ruby on Rails is all about minimizing corners in the APIs. The goal is to make the process of developing a webapp a simple easy flow. You don’t have to stop and deal with some complexity here, or some thing unrelated to what you’re trying to work on. No, Rails lets you just keep moving forward with your development, impeding you as little as possible, and each successive version of the app has minimized more and more corners.

When it comes to good GUIs you need to apply the same principle. Users shouldn’t have to stop and do something only tangentially related to accomplish their task. Consider where people are going and make it so that they can get there with as few course changes as possible.

Limit Options

In crowd management, the maxim called Braess’ paradox states that more options equals decreased performance. That is, if you give people many routes to choose from, crowd traffic will slow down because of indecisiveness and selfish behavior when choosing one of the paths.

The guys over at 37Signals have been preaching this and implementing it to great success for years now. It’s similar to limiting corners because each corner is work and when there’s work people make a conscious or unconscious decision regarding whether or not it’s worth the work. When we limit options we’re minimizing the number of decisions a user has to make to accomplish their task, and as a result, we’re minimizing corners. Users frequently request every conceivable option and configuration control but when you actually give it to them they don’t want it. It would be like an mp3 player with 37 switches 12 sliders and 4 knobs. Sure it would let you get your music exactly how you wanted it but it would be too much of a pain in the ass to use so you’d trash it and go buy an iPod.

I’d recommend checking out Getting Real by the 37Signals guys if you haven’t already, because limiting options and corners is central to the process they describe.

Anxiety Control

Still says facility managers “can shift the behavior of a crowd. Good signage and lighting, for example, will reduce anxiety. People need information before they approach the crowd. If one person has to ask where their seat is, then 140 people have to ask. Now there’s a backup and people are frustrated. Now those frustrated people sense disorganization and start acting out. Others take that cue and the anxiety feeds on itself. People say it’s the crowd’s fault. No. As the facility managers, you shape the behavior.

In our world anxiety doesn’t grow into a stampede, it turns into bad word of mouth and users who won’t use your software. So, make sure your users have all the information they need to accomplish their tasks but don’t overwhelm them with tons of docs. Imagine if you were at the stadium and had to read through five paragraphs of signage just to figure out where to pee. Obviously you can help reduce anxiety by limiting options and corners too.

Nuceria vs. Pompeii

During games between bitter rivals Pompeii and neighboring Nuceria in A.D. 59, the historian Tacitus writes of an altercation that “arose out of a trifling incident at a gladiatorial show. During an exchange of taunts…abuse led to stone-throwing, and then swords were drawn.”… This might seem to disprove Still’s notion of best practices in ancient crowd control. To the contrary, Still says. “Think of the fact they could have a sword fight in the stands, what that meant about how they had very free movement in the stands. And because of the space, people could cluster away from the small pockets of danger, preventing small incidents from becoming bigger ones.” The violence, in other words, was not a stampede or a crush. Today, nearly all crowd incidents affect the innocent, who simply can’t escape.

We can learn a few lessons from this.

Make sure that one user’s craziness can’t force itself onto others.
Make sure your app can handle the fact that some people will come in carrying unexpected items (ideas of usage and/or data).
Make sure your entire app doesn’t come crashing down when problems break out in one or more sections. Yeah, you won’t be able to use that one piece but maybe you should be able to use the other pieces without it.

Respect Personal Space

In ancient Pompeii they made sure that everyone had the appropriate amount of elbow-room in each section of the stadium. As developers we need to recognize in advance which portions of the app will be working hardest and plan accordingly. Mostly this speaks to optimization and scaling. Early optimization is generally a bad idea but that doesn’t mean you should ignore optimizing sections where it’s obvious you’ll need good performance and if there’s a good chance you’ll need to spread some section of your app over multiple boxes then you should plan for it from the start.

Excellent tip for interviewers of geeks

2007-06-01T00:00:00+00:00

It’s so obvious you have to wonder why people haven’t been doing this for years now…

Witten proposes an incredibly simple and good way to evaluate a potential employee’s code skills during the interview. Instead of asking arbitrary code questions ask them to send in a code example they’re proud of before they come in and actually go over that code with them during the interview.

This is a much more realistic test for exhibiting coding prowess than some sort of artificial string manipulation problem. Additionally, and this is a hugely important point when performing any interview, it tends to put the candidate at ease to talk about their very own pre-written code, so they’re not sweating bullets and thereby giving you an inaccurate reading.

When you do go over someone’s code with them during an interview, here are the sorts of questions to ask:

How would you explain this code to someone unfamiliar with it?

Why did you make this particular design decision here?

What are some trade-offs involved with this design decision?

How would you go about profiling this code?

Can you think of a way to optimize this particular section?

How would you change things if the requirements were altered in this particular way?

Seriously, man, tabs for indentation?

Basically, ask all the same sorts of questions you would normally ask after a candidate solves a stupid string manipulation problem on the whiteboard, but instead ask those questions about the candidate’s own pre-submitted code, written on a proper computer with the use of a proper compiler.

There’s another benefit to this that Witten doesn’t mention in his article and that’s the opportunity to observe what happens when you challenge that persons decisions. How do they react when you say “do you really think that was the best way to tackle that? Wouldn’t it have been better to …” Even if you don’t really believe it better to do it a different way it’s a situation that will come up and you want to know that the person is going to be able to handle criticism well and be willing to work on alternate solutions to things.

Like I said, why haven’t we been doing this for years?

I think I know why people don't value tests

2007-05-26T00:00:00+00:00

I think I understand why people tend to not write tests. Because they believe that tests aren’t something that’s either needed or important.

“Duh,” I hear you say, but bear with me.

Why don’t people believe that tests are something that’s either needed or important? Well, I think one of the biggest contributing factors to WHY is that essentially zero of the learn to program in language FOO books ever mention unit testing. Unit testing has been around in a formal sense since the creation of SUnit back in 1994 ! 1994 I say! That’s thirteen years now. Thirteen years and I could probably count on one hand the number of introductory language, or language reference, books that not only mention unit tests but actually explain why their important and how to use them. Even worse, most languages don’t have unit tests tools built into their core libraries. All the modern languages have fairly comprehensive test coverage but they have to use external tools to write those tests. How crazy is that? We have this common programming task that we all agree is critical to releasing a stable version of the language but it’s not important enough to build into the language. Wha?!?! The end result is that since we don’t teach tests as being even noteworthy when teaching a language no-one learns that they are important. For the most part people just don’t seem to understand the value of tests until they’ve been in the industry so long that their feet are riddled with holes.[1]

Mike Clark, and others, suggest [writing “Learning Tests”](http://clarkware.com/cgi/blosxom/2005/03/18#RLT1 “writing “Learning Tests”“) as a way, not only to learn a new language, but as a way to accrete a repository of what you’ve learned about a language. I think this is a GREAT idea. Imagine if every book that taught a new programming language showed you not only how to do something but then followed it up with how to confirm that you didn’t screw it up by demonstrating how to write a test for it? People would start to see test writing as a standard part of the software writing process. It would be “just what you do.”

Imagine the impact that including unit testing as a standard part of the learning process would have on the software industry! Sure it might take five to ten years before we started to see the results from it but wouldn’t it be worth the wait?

[1] From having shot themselves in the foot on many prior occasions.\

The power of tests...

2007-05-25T00:00:00+00:00

If you look at Mingle, the project management tool we’ve been working on… On that tool I happen to know that their test base is twice as much as their code base. So, two-thirds of the code in that product is tests, and that allows them to do quite violent things. I know that a couple of months ago they made a very fundamental change to the database scheme. I mean, we’re talking, utterly to the guts of the database scheme. And they did that and… it wasn’t even an event worth talking about. And, when they were planning to do it they were saying ‘Yeah, yeah, we’ve got to fundamentally alter the core tables in this application… Yeah we’ll do that, and it’s not a big deal.’“ -Martin Fowler (paraphrased)

Getting some agility in your workplace. A flow chart.

2007-05-24T00:00:00+00:00

The software development industry is plagued with bad practices even though so many of us know better. A HUGE portion of this problem is that to really start, and continue, working the way we know we should requires buy-in from our managers and coworkers. And it’s not just a conceptual buy in that we need. People need to really get the religion. But, you and I both know that we can’t realistically expect the rest of the company to change everything at once.

This flow chart addresses what I think are the three biggest changes we can make that will dramatically improve our companies. Find your biggest pain point and work on that. Once you’ve got that working really well in your group and you’re confident that people are going to stick with it you can choose the next biggest pain point and address that one, remembering what things worked best to get your employees to buy into the last change.

Each of the major pain points could easily get their own graph for implementing solutions at a similar level of complexity, but those are graphs for another day.

As always you can click on the image for a full sized version and / or download the Dia file too.

This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.
\

Unit testing your JavaScript

2007-05-24T00:00:00+00:00

Most web developers will agree that unit tests are great, and some even write them…but I know very few developers who write unit tests for their JavaScript, but it’s not really their fault. Most don’t know of good unit test systems for JavaScript and / or don’t write their JavaScript in such a way that you even could test well. This means breaking all the functionality into discreet functions and objects instead of writing old-school procedural crap. There’s also the obvious problem that most of your JavaScript is tied to the browser and the current page. So how do you test stuff in the page? Well, JsUnit lets you do just that and, seeing as I’ve just added a javascript implementation to the FizzBuzz Overthink you can run over and see how to do it for your apps too.

The limitation of JsUnit is that it really wants a complete file to load into the test harness but it’s bean ages since I’ve worked on a site where at least some portion of every page was not dynamically loaded. Fortunately the workaround is fairly simple: grab the source to the JsUnit Test Runner ( testRunner.html ) and extract the form but leave out the the file chooser. Put all that into a template file that will be dynamically loaded into your app when you view the page as a specific test user or with a magic cookie. Otherwise it’s just not there.

Now every page that needs a JavaScript unit test will have a test harness down at the bottom. Click the button and see if it passes. In order to run all the tests on all the pages just have your Selenium tests also check that the unit tests run correctly. Usually you’ll want to have this be a separate test run so that you can log in as the user who gets to see the tests and because you’ll frequently want to populate your forms and such with data that WILL break things just to test that your JavaScript doesn’t blow up in people’s faces.

Getting the most out of version control for hosted web apps.

2007-05-23T00:00:00+00:00

Another graph for another friend who asked for a flow chart of the branching and merging described in Best Practices for Web Development html
pdf.

Update:

Michael says: I’m a little unsure, from you diagram, how, if your trunk contains two completed and merged features that aren’t yet live (video upload and REST API, say) you put one feature live without putting the other live. It looks like code only gets to the live branch via the trunk, but it seems from your diagram that the trunk could contain all manner of complete and semi-complete features.

He’s right and I cover this in more detail in my Best Practices For Web Developers Essay html / pdf , but the basic idea is this: Your dev trunk holds the bug fixes and features you want to see in the next release. If, as in Michael’s example, you’ve got the video upload written, but maybe you don’t have the hardware in place to support storing and serving gigs upon gigs of uploaded videos, so you decide to hold off on deploying that. So you leave it be in it’s little branch, remembering to update it at least once a week until you’re ready to include it in the general codebase.

Have you come up with a better solution to Michael’s problem? Please let me know if you have.

I’m really not thrilled with this particular chart though. It’s got no flow. If you can improve upon it please do so and send me a copy. I’ve got some ideas but nothing great. I really liked the Using Darcs with SVN / CVS flow chart much better.

As always you can click on the image for a full sized version and / or download the Dia file too.\

This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.

Using Darcs with SVN / CVS Flow Chart

2007-05-23T00:00:00+00:00

A flow chart for a friend demonstrating, step by step, how you’d go about using Darcs (or any other distributed version control system) with SVN / CVS. He needs to do it for the most common reasons:

it’s a pain to branch and merge with CVS.\
he’d have to coordinate with other departments like QA to get them working off of whatever the current release branch happened to be at the moment and convince them that it was a good idea.
currently his co., like so many, is working off of a single development trunk. There are no other branches.

Click on the image to get a full sized version. Or download the dia file here.

This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.

Some earlier related posts:

FizzBuzz Overthink as a Teaching Aid

2007-05-21T00:00:00+00:00

UPDATE: I’ve created a new subdomain for the FizzBuzz Overthink project ( http://fbo.masukomi.org ), updated the links below to reference it instead and modified the Readme in the project (and the default page of the site) to reflect the new goals of the project

UPDATE 2: There’s a JavaScript example in there now too which, of course, includes unit testing. More details here.

I was talking with someone the other day who mentioned, in passing, how they wished they had some good examples of how to do unit tests for his co-workers. There are, of course, plenty of examples of how to do unit tests but I haven’t seen many online that show them in the context of a complete but simple application. This conversation made me remember my FizzBuzz Overthink (FBO) that I’d written in Java, and it occoured to me that it would make a great teaching application. Translating an existing FBO into a language you’re trying to learn is also a great way to get your head around it.

The FBO demonstrates how to break down an application into useful methods, how to write unit tests for those methods, and how to document them.

So, I’ve added a Perl version of the FBO, with docs and unit tests, although I used Natural Docs for the documentation instead of standard POD because ND is just light-years ahead of POD in my opinion. You can grab the new Java + Perl versions as a zip file or better yet check it out with darcs from http://fbo.masukomi.org/code/FizzBuzzOverthink/ so that you can improve upon it and / or add your own language to it to help make it a more useful teaching aid. Just use darcs send to send me a patch with your changes ( masukomi at masukomi dot org ). I think there are a lot of people who would benefit from a PHP version and I’d personally love to see Haskell and Erlang versions.

Heuristically Driven Development Flow Chart

2007-05-18T00:00:00+00:00

This flow chart will only make superficial sense unless you’ve read the Triage section of my Best Practices for Web Developers essay, which I happen to have just updated with an expanded Triage section.

Click on the thumbnail for the full-sized verison. If you’d like, the Dia file is also available if you feel like tweaking it.

How Greasemonkey can save your webapp time and money

2007-05-12T00:00:00+00:00

For the most part web developers spend our time guessing what features people might want and how said features should be integrated. Frequently we guess wrong and that wastes time and sometimes frustrates users. But there are tons of users out there who also happen to be developers and they’re actively implementing new features for their favorite sites with Greasemonkey. UserScripts.org has over 6,000 scripts. That’s 6,000 features with working implementations for popular sites at no cost to those sites! Free I say.

Each one is a feature someone wanted so badly that they figured out how a site worked and then wrote working code to modify that site to implement that feature. If you’ve got a somewhat popular webapp you owe it to yourself to watch userscripts.org and google for other greasemonkey scripts for your site. Check out what people have done, see how it works, maybe the rest of your users would like that too…

Consider offering some free goodies to people who send you cool greasemonkey scripts. Even if it’s just a free t-shirt (go to CafePress and whip one up if you have to). Maybe even put up a page of user submitted scripts. Some of your own developers may even make them to implement features they wanted that may not have been appropriate for all users of your site. Just make sure to emphasize that they’re unsupported. ;)

Side note: this post was inspired by the creation of ~~my first Greasemonkey script~~ [EDIT: userscripts.org went down in 2014]

It’s something for people who use Reddit and del.icio.us. Not useful to Reddit in it’s current form but the idea behind it might be very good for Reddit if expanded to support more bookmarking sites.\

LazyWeb Hardware Idea

2007-04-18T00:00:00+00:00

If you build this I will buy it.: Ergonomic Dvorak keyboards are effing expensive. Dvorak keyboards are hard to find in general and software remapping in the OS has a variety of…. issues. I want a USB dongle that remaps keystrokes from a QWERTY keyboard to Dvorak. Even better put a switch on it to turn off the remapping. I will pay up to $25 for such a device.

What if webapps worked like pinball machines?

2007-04-12T00:00:00+00:00

Currently web frameworks are all about the page. This is starting to change a little thanks to Ajax but mostly just towards the idea of page snippets. But why? “Pages” are a metaphor taken from books, but books are linear and the page serves to restrict the layout. Things like Yahoo Widgets have shown us that there’s no need for layout to be bound to the “page” metaphor and we’ve known for a long time now that webapps are rarely linear, and when they are people don’t like it. Essentially we’re writing glorified “choose your own adventure” games that still say “To open the door on the left go to page 65. To open the door on the right go to page 27. Rails, and it’s clones, even number the “Pages” like that. We need a new metaphor. What if, instead of writing frameworks that are all about serving and managing pages, we started to think of webapps more like pinball machines. There are 5 major parts in every modern pinball machine (other than the ball):

The Flippers. You use these to force the ball in some direction You want, as opposed to the direction it wants.
The Bumpers. Posts that actively push the ball off in another direction when hit.
The Posts and Rubbers. These stick out of the Playfield with rubber that passively bounce the ball away.
The Playfield. The board that defines the scope of the game.
Gravity.

For this metaphor let’s imagine the ball represents the user and their session as they bounce around your site.

The Flippers forcefully redirect your users to do something you want, as opposed to what they wanted, and that generally appreciated. But sometimes you have to force them to do things like go to a login page. Of course, every time you have to use the flippers you risk loosing the ball down the outhole.
The Post annd Rubbers are simple no real processing required responses to a user’s actions. Maybe it’s a request for a css file. Maybe it’s a help page. Either way it’s an, essentially static, file or straight database read that provides the user with some information that alters their trajectory.
The Bumpers involve work, some active processing based on the data imparted by the user hitting it. They alter the user’s trajectory like a Post but also impart energy to the user as they send them off.
Gravity is like time. It’s what’s trying to get the ball to leave the playfield of your webapp.

In general you want to just let the user do whatever they want. The pinball machine doesn’t care about the where the ball just came from or where it’s going. Maybe it’ll keep a running log of the “Score” but in general it is only concerned with if the ball is on the Playfield or not. A bumper doesn’t care what other Bumper or Post the ball came from or where the ball is going after, although it will participate in recalculating the ball’s trajectory. They just sit around in their own stupid little world listening for input from balls.

The Ball…. The Ball is where the magic is. Since the ball represents our user we can assume it has a brain…even if they, like Pooh, they are “a bear of very little brain.” It knows where it’s been. It sees where it’s headed. And, it is actively trying to get a good score, or get off the Playfield. That point’s never been quite clear. Either way it’s the ball that changes it’s direction when affected by gravity. It’s the ball that is running around your site. It’s the ball that triggers the bumpers and it’s the ball that makes the game worth playing.

So what if we stopped focusing on pages (Bumpers / Posts) and started focusing on the Ball. What if the Bumpers and Posts were just listeners who knew nothing about each other. If you like technologies like Mule you could think of them as listeners on a queue. Or you could think of it as a publish subscribe model where they’re all listeners registered with the ball. Regardless of how i architect it why should my “Address” Bumper know anything about the “Company” Bumper. Why should it have to know that that’s where to force you next. What if you let the ball decide where it wants to go. Sure the Bumper will give it a push but maybe the ball just doesn’t have enough energy to care about going to the Company page/Bumper after filling out an Address. Maybe it’s feeling the influence of Gravity and just wants to get off the Playfield.

If everything on the Playfield is truly independent we add and remove them without concern for the effect on other ones because there isn’t any direct coupling. More Posts? No problem. Since Posts don’t know or care where you came from or are going it doesn’t matter if the Post that was there yesterday is still there or not. More balls? Again, no problem. Yes they might collide but that doesn’t affect the Posts or Bumpers, and interesting things happen when users interact with each other.

A friend said,

“I’m still a little unclear on how pages are tightly coupled to each other, and how the bumper idea fixes that.”

So let’s look at a concrete example. Lets imagine you were buying some stock. You place a buy order (the ball hits a Bumper). Now, it’s important to the user that they see an acknowledgment that the order was successfully placed but the Bumper / Controller that processed that form doesn’t and shouldn’t care if you see an acknowledgment or not. It’s done it’s job by processing your buy order. It’s the User that cares about that acknowledgement. Where you bounce off to is their concern not the Bumper’s (unless of course there was a problem). But a lot of frameworks actually make you put that routing information in there with the form processor (Ruby on Rails being a perfect example).

What if User A is an advanced user and should be shown some complex acknowldgement page but User B is a computer neophyte who should be shown something simple and to the point. Do you really think it’s appropriate for those routing decisions to be implemented in the controller that submits stock purchase orders? No, of course not. It should be handled in the object that actually cares about what’s displayed next; the User object. The Ball. The ball knows what kind of user it represents. It can decide. Sure it’ll take suggestions from the Bumper as hitting a bumper does affect a pinballs trajectory but the Bumper shouldn’t be tied to any of the other Bumpers / Pages.

Some frameworks like Struts handle this by having the controller send you to the success or error page and let a central point in the framework decide what that is, and while this is definitely better it’s still the wrong place. That’s like having the bumper ask

The Playfield where to send the Ball. It makes no sense. It is the combination of the spin, speed and initial direction of the ball, it’s angle of impact, and the force of the bumpers push plus a little bit of Gravity and other more negligable factors.

The Playfield itself plays almost no part in the equation. It’s almost entirely dependent upon the state of the ball. In a webapp it’s all about what the User wants not what the framework wants. So why would you have the framework that binds all the Posts and Bumpers to the playfield decide where the ball should go?

And before you ask… No, I’m not suggesting you actually ask the user if they want to see an acknowledgement to their buy order. Im suggesting that the Ball should act as their proxy. It should know, or at least be able to make an educated guess at, what the user will want to see after a given action based on where they came from and where they were headed.

In most cases this would be akin to the success page / error page routing table in Struts only moved to the User’s control. But, imagine what could happen if you wanted to get fancy. You could throw in some Bayesian filtering or other statistical analysis to watch how a user interacts with Posts and Bumpers. After a few times on your site it would know that they generally want to go from Post A straight to Bumper D.

In summary: Why are we using web frameworks to tell users where to go? Shouldn’t web frameworks be listening to where the user wants to go? And why should different pages and form processors in our webapps be tightly coupled?

Using Darcs WITH Subversion / CVS

2007-03-30T00:00:00+00:00

Using Darcs with Subversion / CVS

Some of you have gotten the distributed version control religion (If you haven’t, you should read my Best Practices essay) but are stuck with Subversion (or CVS) either because that’s what they use at work or because some part of your deployment systems use it. You may also want to combine them simply because of the power of svn externals which lets you pull in some of your code from constantly updated , Subversion Based, 3rd party repositories. Using SVN with a distributed version control system also gets you a cannonical, no doubt about it, central repo, instead of just a repo that everyone agrees to call the central one, plus you can utilize all those nifty notification and stats tools people have written for svn.

What makes this work so easily is the fact that Darcs stores everything in one _darcs directory instead of the SVN / CVS approach of throwing files in every single directory. There are a ton of reasons to use Darcs (or it’s competitors) when working in a corporate env (even one that uses SVN / CVS) but my favorite is that you get the full benefit of version control on all your work and can check in all the buggy crappy unfinished code you want without polluting the central repo that affects other developers (until you want to). It’s my practices to do a checkin when I leave work every day, no matter if I’m finished with what I was working on or not. With Darcs it doesn’t matter, because I won’t break anyone elses unit tests because it’s just my personal version control system that’s effected. When I’m done with whatever I was working on and it passes all the tests I can push it back out to other developers. Along the way I’ve been able to make as many revisions of the files as I wanted to without screwing anyone else over.

Sneaking darcs in under the radar:

For those of you who have to use Subversion / CVS at work sneaking Darcs in for your personal benefit it trivial. Just follow these steps for each branch (in svn / cvs ) that you need to work on.

check out a copy of the branch / trunk / tag / whatever that you’ll have to coordinate with others.

assuming you want to version control everything that got checked out you’d cd into that directory and

darcs add -r *
darcs record -am “Initial Import from svn version XXX”
Don’t worry. Darcs is smart enough to ignore all of your subversion files.

From now on whenever you want to start working on something for this branch you just move to wherever you want to checkout a new repo for a feature spike / bug fix / whatever and do a “darcs get”

Use Darcs as you normally would, checking in whenever you want without worry.

When you’re done with your feature spike / bug fix / whatever do a “darcs push” back into the dir where you started, then cd to that directory and do a “ svn commit -m “some comment here” “ (remember you haven’t added the _darcs dir to svn so you don’t have to worry about it getting checked in).

Snicker in the knowledge that you’ve beaten “The Man”.

Combining forces for everyone:

So you’ve convinced everyone to use Darcs. Yay, but for whatever reason you’ve decided to keep SVN. Just follow steps 1-5 above. You could theoretically check in your _darcs dir but I suspect that might just lead to more headaches than it’s worth, especially when you consider how easy it is to create a new darcs repo (steps 2.1-2.2) and the fact that you only have to do this the first time you check something out, which is to say, almost never.

If you’ve tried checking your _darcs dir into svn please add a comment so that I, and others, can learn from your experience.

[Update] Related posts:
Using Darcs with SVN / CVS Flow Chart

Why you should be using a distributed source control system

Best Practices For Web Developers

2007-03-29T00:00:00+00:00

I wrote this essay a while ago and have been tweaking it based on the feedback from those I’ve sent it to (thanks guys). There are a few things I’d like to change still but I’m going to go with the “Release early. Release often” mantra on this one. I think it’s more important that it get out there than it being perfect.

Best Practices for Web Developers introduces the idea of Heuristically Driven Development ( HDD ) as it applies to web development, but honestly almost everything in there is applicable to all forms of development. There’s an HTML version and a PDF version and your feedback would be greatly appreciated and help to make the next iteration of the document better.

I’d include it here but it’s just too long for a blog post.

FizzBuzz Overthink

2007-03-10T00:00:00+00:00

Raganwald suggests, with good reasons, why you shouldn’t over-think FizzBuzz. Obviously I took a different view of it and yet I still agree with all of his points.

So, I’d like to counter with some reasons why you should over-think FizzBuzz, and why I did, because I don’t think I did a great job of explaining that in my last post. But first, lets assume you’re not being asked to solve it on a whiteboard while the interviewer waits. In that case my 400(ish) line solution is completely inappropriate. In fact, anything but a quicky solution is inappropriate. Lets also assume that in submitting an over-thought solution to FizzBuzz you make it perfectly clear that you know it’s over-thought and what you were hoping to achieve by going to such ridiculous lengths in your creation.

As I mentioned in FizzBuzz rethink hundreds of programmers rushed to solve FizzBuzz in the comments of the various posts about it and many of them got it wrong. I suggested that this was a classic example of what’s wrong with this industry. And the reason I believe that is exactly why I think FizzBuzz should be over-thought.

As sad as it is that there are so many “programmers” out there who can’t successfully solve the FizzBuzz problem I believe it’s even sadder how many companies are allowing their programmers to take a half-assed approach to software development. There are far too many who still think that writing unit tests isn’t worth the time, or that since they can’t possibly test for every possible scenario they’ll test for none. Some don’t even use version control. These companies, like the people who rushed to post FizzBuzz solutions without really reading the instructions, allow their developers the slap together the first thing that comes to mind and rush it out the door without any real clue if it’ll break anything else when deployed, if it really meets the clients needs, or, in the case of the posters, if it will make them look like an idiot for not even being able to correctly solve FizzBuzz.

With the previously stated assumptions in mind an over-thought FizzBuzz solution is excellent because it tells the employer that:

you’ll actually apply some thought to the problems they set before you.
you recognize the value of things like testability (you do have unit tests right?)
you are familiar with and prepared for the common changes that clients will ask for.
you are self-motivated and not the type of person who go above and beyond what’s asked of you.

It’s great for the interviewee because if the employer starts bikeshedding you know you don’t want to work for them. If the interviewer does “grill you over why your solution doesn’t use the Y Combinator to implement `compose”, as Raganwald suggests, then you know these people may actually have brains, and I would much rather work for people with brains who will challenge me. If the challenge comes in the form of suggesting you’re a loon for such an extreme solution it’s an opportunity to defend why you think it’s good and see how they deal with a person willing to stand up for their craziness. You’ll quickly learn if they’re the kind of company where they expect you to roll over and acquiesce to their suggestions or if they are willing to have a real debate with you. Either way you learn a lot about the employer that will help you decide if you’d even want the job. It’ll also give them a good idea of how you handle your peers suggesting that you’ve done something wrong.

I’m not sure I want to work for someone who can’t see the value of an intentionally over-thought FizzBuzz like mine.

FizzBuzz Rethink

2007-03-09T00:00:00+00:00

Or maybe overthink would be more appropriate….

[UPDATE] I’ve written a followup about why an over-thought solution like the one proposed here can be / is a good thing. And just to be blazingly, obviously, painfully clear on the matter. I do not think that every problem should be over-thought to the degree I took this. My solution represents a crazy amount of code for such a small problem. It’s intended as an example of how I’d mentally approach a real and complex problem presented by a client.

The FizzBuzz test was conceived by Irman and posted here: http://tickletux.wordpress.com/2007/01/24/using-fizzbuzz-to-find-developers-who-grok-coding/

It was to be used as a method for weeding out applications for developer positions who couldn’t code anything and the original test instructions were to:

Write a program that prints the numbers from 1 to 100. But for multiples of three print “Fizz” instead of the number and for the multiples of five print “Buzz”. For numbers which are multiples of both three and five print “FizzBuzz”.

ANY developer should be able to write a solution to this and many did in the comments of the original post and other posts that referenced it. Unfortunately, as a number of bloggers pointed out, many of the people who rushed to post solutions GOT IT WRONG! Mostly this was a result of just not reading those three sentences carefully enough. I know, you’d think you wouldn’t have to re-read those…

The solutions people were posting were inadvertently providing a classic example of what’s wrong with how so many developers approach professional software projects, which means that FizzBuzz is a better interview test than anyone seems to realize. How a person approaches this problem for an interview is a great example of how they’ll approach other problems they’re faced with should you decide to hire them.

I’ve been coding professionally for over a decade now and along the way I’ve learned a number of things; things that I’d like to see addressed either in code or by the questions an interviewee asks in response to the test. So lets set the stage for something a little different.

First, we’re going to assume that it’s been explained that the FizzBuzz test is based upon a children’s game of the same name.

In this game a group of children sit around in a group and say each number in sequence, except if the number is a multiple of three (in which case they say “Fizz”) or five (when they say “Buzz”). If a number is a multiple of both three and five they have to say “Fizz-Buzz”.

Second, you have to remember that this is for an interview so the response should not only display that they CAN code but, hopefully, give an idea of how they’d approach the task of writing software for one of the interviewer’s clients because if someone’s hiring you to code then there’s a client (even if it happens to be the same company). Seeing if the interviewee can make this logical leap without being told is another part of the test.

Third, after explaining the game the test is based on and with the assumption that the person knows this is for an interview use the same instructions.

Continue reading to see my solution, or pretend you’re an interviewee and write something to impress then see how our solutions compare.

This is my solution. It is above and beyond what is needed to address the instructions. But, it’s goal was to show an interviewer how I would address the problem if it were for a client of theirs and assumes typical clients. I do not believe that every simple problem needs to be expanded to this extreme. It’s written in Java because that’s what I have been coding in all day today and you should be able to import it as an existing Java project (compiled against Java 1.6).

It addresses the following issues:

Clients change their minds. Constantly. Just because it’s 1-100 today doesn’t mean that it won’t need to be -1 to 1000 tomorrow.
While specs change (see above) wording changes even more. Three days after deploying the code the clients wife / husband will decide that FizzBuzz is a “horribly offensive” term or a foreign student will realise that it means “Dung Beetle Sex” in some obscure language. The words will have to be mutable.
If it’s even remotely useful it’ll be re-purposed. Which means the output you thought was appropriate will now need to be usable by the web version, the version for mobile devices, the Blackberry (even though the customer doesn’t own any they’ve demanded Blackberry support).
Documentation. It’s got comments. It’s got Javadocs. It’s got this document here.
Readability and maintainability. This is for an interview. The code should be readable not an example of your “leet” mastery of obscure syntax. Furthermore, if this was a “real” app it’s just a matter of time before someone else will have to pick up where you left off.
Unit testing. It’s got to be done especially for a mission critical component and I consider successfully obtaining a good job to be a critical mission so this app has got to run correctly. It should also show that I know how to write a good test.

Unfortunately it doesn’t address these issues but I would hope that my addressing them in this document will suffice:

The error messages are NOT internationalized but would definitely need to be in a “real” app. It’s far easier to internationalize in the beginning than to re-work an existing app but doing so would have required a bit more work than I thought was appropriate for this app.
I’ve also chosen to leave out logging because I wanted this to be runnable without any additional libraries other than jUnit.
There are no functional tests. Given that the app has no GUI and was to be run from the command line I’ll trust that the operating system developers of the world have correctly implemented argument passing to command line apps and that the JVM correctly prints to the console when it needs to. Were this a “real” app with a GUI it would require functional / integration testing.
It doesn’t enable modification of which numbers you’d call out Fizz, Buzz, and FizzBuzz on. This is because the next three prime numbers are 7, 11, and 13. Considering that this is a children’s game to teach multiplication none of those would come up with enough regularity to make the game worth playing and non-prime numbers would come up too often.

The unit test assumes you have jUnit 1.4 in your classpath and an easy way to run it, like eclipse.

Why you should be using a distributed source control system

2006-12-29T00:00:00+00:00

I was reading some articles yesterday that finally made the light bulb go off about distributed source control management (scm) and why we should be using them. First off, a distributed scm, unlike CVS or Subversion, has no central repository that all others pull from. It’s possible to set one up and say that it’s the master and tell people to pull from and push to it but that’s more a matter of convention. What’s truly unique about these systems is that each checkout is it’s own self-contained ecosystem. And there are many reasons this is a good thing:

You can make as many changes as you want, check in unfinished code, and explore new functionality without ever affecting other developers. It only impacts other users when you sync your current working environment with others.
You don’t have to have net access, or be connected to any other box to get work done while still taking advantage of version control. Working on your laptop? Stuck in the boonies with just dial-up? Commit changes, roll-back to previous versions, make branches, none of it is dependent upon some central server. When you do get back online you can sync your changes with others.
“You don’t need to set up and manage a central SCM host with sufﬁcient disk space, compute power, bandwidth, and backup to support the concurrent SCM operations of your entire development community.” *

Real world uses:

The open source project:

If Mozilla has taught us anything it’s that you’re NOT going to get thousands of developers working on your project. So, while huge numbers of developers is something that distributed scm handle exceedingly well I think the point is moot. But, in my experience, a developer interested in some OSS (Open Source Software) project will download the source from a traditional scm, poke around to understand it, and if they’re smitten with the idea will start customizing it for their needs. But, the are forced to either work without the benefits of version control or they have to check it into their own personal scm. If they check it into their own you can pretty much forget about ever getting patches from them because they’re no longer able to sync with your tree and it would be way more work than they generally want to do to get synced and give you a patch that was useful. If they work without scm (because they don’t have commit rights to your scm) you may get a patch from them but for them it’s like climbing a rock face without lines and harnesses.

If you were to use a distributed scm each developer would be working off of their own personal copy of the scm that could be synced at any point in time no matter how many changes, revisions, or commits they have made. They’re generally not going to check it into their own scm system because that’s work that doesn’t get them any real benefit. They already have version control via their checkout and the fact that it’s a distributed system. End result, they don’t need to work without the safety and security of an scm and they will generally always be working on a system they can easily send you changes from when they’re ready.
Web developers with sites others depend on:

If you’re really working like a pro, you’re constantly branching and merging your code. You’ve got a live site branch. You’ve got a branch for each new functionality exploration. You’ve got a trunk branch that everyone merges into their other branches regularly. In the real world you probably have a live site branch and the trunk and cross your fingers that each developers functional exploration doesn’t screw over any other developer’s pokings. Or, in far too many cases, you just have the trunk. The problem comes (for those of us who hate dealing with the pain in the ass that is the current state of merging on most traditional SCMs and have minimal branches) when you decide you want to integrate one developers functional exploration into the live branch. How do you extract that code from the other code in the trunk (or any other common branch)?

If you were using a distributed scm it would go something like this: Everything starts with the live site repo (wherever you happen to keep that). Everyone’s local repo(s) started as a copy of that. It’s not uncommon to check out another copy to go explore some new feature set in. Maybe you sync it with the live. Maybe you trash it. It doesn’t matter no other repo was affected by your fiddling. You don’t have to make a new branch in some central repo that gets stored forever even if your fiddlings proved insignificant or were just abandoned. If everyone decides that the features you worked out in a particular repo are worth keeping they just sync with it. If you decide to wait to put it into a later release that’s fine too because there’s no need to untangle it from other work you, or others, have been doing. This is because to “branch” you just do a simple checkout from any other repo. When it comes to merging you benefit from the fact that any decent distributed scm is designed to work with syncing sweeping changes to entire code-bases not just individual files. They all handle this a bit differently of course, and some better than others, but the point is that while you’re almost always going to have conflicts you’re working with tools that have to be better equipped to avoid them.
Projects with reeeeally large code-bases or many many developers:

I can’t really speak to those because I just haven’t worked on them but utilizing distributed scms does mean that you don’t have the huge demands on the server hosting your scm. Plus all the aforementioned benefits.

So what are your options? Well the main contenders appear to be darcs (written in Haskell), Mercurial (written in Python), and Git (written in c plus hooks into a bunch of other things I think). darcs seems to be the easiest to use, with some decidedly funky features, but isn’t great for projects with huge numbers of files (it can be ram intensive at times). Mercurial is going to be used by the newly open-sourced JDK, and Git is being used to manage the Linux kernel.

* My light bulb moment, and some of the quotes here are thanks in no small part to this blog post about the JDK moving to Mercurial by Mark Reinhold. Thanks Mark.

The niftiest little Rails plugin you never heard of

2006-12-09T00:00:00+00:00

Back in January of 2006 Ezra Zygmuntowicz came up with an exceptionally cool Rails plugin that, IMNSHO, should be in rails core. It’s called ez_where and it’s svn repo is here.

What’s so cool about ez_where? Well, the to really understand it’s beauty you have to step back to one of core concepts that’s at the heart of frameworks like Rails: Database Abstraction and Object Relational Mappings. Before these concepts really came into their own we were all writing raw SQL commands in our apps. Now we interact with a layer of abstraction that lets us work with the objects we’re actually programming with instead of database structures. Except, that’s not quite true with Rails. With rails you end up writing things that are possibly more complex than the SQL statements we’ve been trying to avoid. Here’s a particularly egregious example from a ticket system I’ve been working on:

     Ticket.find(:all,
         :conditions=>["closed = ? AND product_id in (?) AND user_id = ?  
             AND ticket_type_id=? ", product_ids, user_id, ticket_type_id],
         :order=>["weighted_vote_percentage DESC, ticket_id DESC"])

Personally I don’t see this as being particularly better than writing the raw sql since I am writing the raw sql. Yes Rails has saved me from writing the SELECT t.* from tickets t WHERE but then it forced me to write the entire where and order clauses and put them in a data contstruct that took more thinking and more characters than the SQL would have. How is this better? SQL is database agnostic anyway (unless you start adding in vendor specific additions ) so it’s not really buying me separation from that… So what is it buying me?

Ezra gets it though. If you use his plugin you’ll be able to use the power of ruby in your find statements. Yes, it may be more wordy than SQL but what it buys you is a true separation from your database. His blog posting on ez_where gives this great example:

 Model.find_with_conditons( :all, :limit => ..., :order => ... ) do 
   foo == 'bar'       # ["foo = ?", 'bar]
   baz <=> (1..100)   # ["baz BETWEEN ?AND ?", 1,100]
   woo =~ 'substri%'  # ["woo LIKE ?", 'substri%']
   id === (1..4)      # ["id IN (?)", [1,2,3,4]]
   fiz < 10           # lt, gt, leq, geq, etc., should all "just work" 
 end

Now, to be totally up front with you, I haven’t actually used this plugin yet, but, honestly, I don’t even care if it’s buggy. If it’s even remotely close to working I’ll be helping to fill in any holes that present themselves. For years experienced developers have been trying to get us to remove SQL from our code, and then Rails comes along and chucks that advice out the window. This, or something very much like it, is what we should be moving towards.

On having a mission

2006-11-21T00:00:00+00:00

The cranium of a good developer is filled with ideas for new applications. Most of them tend to bounce around with little energy and eventually succumb to entropy. Some ideas are made out of bouncier stuff and eventually reach escape velocity, at which point they are launched down the arms and funneled out the fingertips. You can tell just how cool an idea is by the speed of typing relative to the developers average words per minute.

A short while later there’s a working prototype, and, should things go well, a fully fledged app will follow. But, most of these apps have no goals, no mission in life. Sure they do something, hopefully well, but for the most part they’re like teenagers, and, like most teenagers, they’ll either figure out what they want to do in life or wander around aimlessly until they figure out what they want to do, start doing something nobody cares about, or simply keel over dead.

But what if they knew what they wanted out of life?
Your, application’s mission is it’s heart, it’s driving force. What is it’s purpose in life?

Google’s mission could be “To be the world’s best search engine.”, but how is that going to drive Google forward in any meaningful way? Where’s the vision? No, Google’s got bigger goals than that. Their mission is “to organize the world’s information and make it universally accessible and useful.” Admittedly it’s not a terribly rousing mission but take a second and think about the implications of it. To do anything useful with “the world’s information” you first have to have it. Just imagine what that means for it’s infrastructure! And how about making it useful? Well, that requires tools. Making it accessible is a combination of incredible search technology (we are talking about all the information in the world here) and tools that can handle the various types of information: text, video, audio, various forms of structured data like schedules, locations, etc.

Flickr ’s mission could be “The best place to store, sort, search and share your photos.” but honestly, how lame is that? Their mission is to be the “eyes of the world “:

That can manifest itself as art, or using photos as a means of keeping in touch with friends and family, “personal publishing” or intimate, small group sharing. It includes “memory preservation” (the de facto understanding of what drives the photo industry), but it also includes the ephemera that keeps people related to each other: do you like my new haircut? should I buy these shoes? holy smokes - look what I saw on the way to work! It let’s you know who’s gone where with whom, what the vacation was like, how much the baby grew today, all as it’s happening.

And most dramatically, Flickr gives you a window into things that you might otherwise never see, from the perspective of people that you might otherwise never encounter.

Flickr’s mission is all about vision, maybe a little to much so because it’s so non-specific. Stuart Butterfield obviously has a good grip on how it affects them, but anyone else reading just the mission wouldn’t really get it. But that’s ok. What’s important here is that there was a mission and that the people behind it understood it’s implications. What good is it to be the world’s eyes if nobody can see through them? Flickr is driven by it’s mission to build tools to let you see through those eyes.

But missions don’t just help you plan. They help keep you on track. Every nifty feature you want to add to your app should be run through the filter of your mission. Eyes aren’t about statistics or value or anything like that. They’re just about seeing, and that’s why you won’t find any analytics or ways to earn money form your images, or anything else like that on Flickr. If your app’s mission is to be “The easiest way to do X” you have to ask yourself if that new set of 20 different configuration options actually make things easier for people or just serves to confuse or intimidate them?

So build your rough prototype. Get that initial burst of creative energy out. Then, sit back and ask yourself what it really wants to be in life, and let that drive every new feature you add and provide you with the confidence to cut out everything else .

XML-RPC vs SOAP

2006-11-21T00:00:00+00:00

Update: This article was written years ago, however, the information still holds true. What I would note is that these days both XML-RPC and SOAP have excellent libraries which makes working with both fairly simple. This article is about the capabilities of the two technologies, however, in my experience the question is no longer one of capabilities. It’s a question of which one you need to talk to, or which one your framework has baked in. In most cases the answer is SOAP. However, if you’re considering writing your own API, my recommendation would be to seriously consider using XML-RPC. It’s far less complicated and bloated, and the combination will make your debugging efforts much easier in my not so humble opinion.

XML-RPC vs. SOAP

by k. rhodes masukomi@masukomi.org

A simple guide to choosing the best protocol for your XML Remote Procedure Call needs.

Within the world of XML there are two main ways to implement a Remote Procedure Call (RPC). XML-RPC and SOAP. This document will explore the differences between these two methods in order to help you decide which is best suited to your needs.

General information

XML-RPC http://www.xmlrpc.com

XML-RPC is “…a spec (http://www.xmlrpc.com/spec ) and a set of implementations that allow software running on disparate operating systems, running in different environments to make procedure calls over the Internet. It’s remote procedure calling using HTTP as the transport and XML as the encoding. XML-RPC is designed to be as simple as possible, while allowing complex data structures to be transmitted, processed and returned.” - xmlrpc.com

XML-RPC’s Goals

XML-RPC is very humble in its goals. It doesn’t set out to be the solution to every problem. Instead it seeks to be a simple and effective means to request and receive information.

“We wanted a clean, extensible format that’s very simple. It should be possible for an HTML coder to be able to look at a file containing an XML-RPC procedure call, understand what it’s doing, and be able to modify it and have it work on the first or second try… We also wanted it to be an easy to implement protocol that could quickly be adapted to run in other environments or on other operating systems.” - xmlrpc.com

Comments

The spec itself is roughly seven pages long, including examples and a FAQ, and is extremely easy to understand. Any competent programmer should find no difficulty whatsoever in implementing XML-RPC in their software after reading its spec.

SOAP 1.1 http://www.w3.org/TR/SOAP/

“SOAP is a lightweight protocol for exchange of information in a decentralized, distributed environment. It is an XML based protocol that consists of three parts: an envelope that defines a framework for describing what is in a message and how to process it, a set of encoding rules for expressing instances of application-defined datatypes, and a convention for representing remote procedure calls and responses.” - the SOAP spec.

SOAP makes extensive use of namespacing and attribute specification tags in almost every element of a message. For example when mixing data types within an array you have to set the SOAP-ENC:arrayType to indicate mixed data types within the array in addition to specifying the type of each element of the array.

SOAP’s Goals

SOAP tries to pick up where XML-RPC left off by implementing user defined data types, the ability to specify the recipient, message specific processing control, and other features.

Comments

Weighing in at 40 pages the SOAP spec is complex and filled with little gems like, “Using SOAP for RPC is orthogonal to the SOAP protocol binding.” If you ask me, it isn’t even remotely something I would call “lightweight” and they threw out the most important feature of XML-RPC, its ease of use.

Contrary to popular belief, SOAP has not been ratified by the W3C. As of this writing it is just a submission, which, essentially, means they thought it was nifty enough to start a discussion about it.

Features & Benefits

XML-RPC

Datatypes

Scalars:

Type	Example
32-bit signed integer	-12
boolean	0 or 1
ASCII string	hello world
double-precision signed floating point number	-12.214
date/time (iso 8601)	19980717T14:08:55
base64-encoded binary	eW91IGNhbid0IHJlYWQgdGhpcyE=

Structs: In XML-RPC structs define an anonymous set of name value pairs. “A contains s and each contains a and a . “The value(s) can be of any data type

Arrays:

XML-RPC arrays define an anonymous grouping of elements with no limitation mixing data types like integers and strings within the same array. “An contains a single element, which can contain any number of s.”

Stability

XML-RPC has been around for a while and, while it isn’t maintained by a standards committee, it is stable and open to community input.

Simplicity

XML-RPC’s simplicity is its greatest feature. It is extremely easy to understand, implement, and debug. The syntax is so uncomplicated that it is very easy to find, and avoid, mistakes.

SOAP

Datatypes

Scalars:

Type	Example
32-bit signed integer	-12
boolean	0 or 1
ASCII string	hello world
signed floating point number	-12.214
date/time	2001-03-27T00:00:01-08:00
base64-encoded binary	eW91IGNhbid0IHJlYWQgdGhpcyE=

Structs: SOAP structs define a set of name value pairs. Structs can be named.

Arrays: SOAP arrays define a grouping of elements with no limitation mixing data types like integers and strings within the same array. Arrays can be named.

Array of Bytes: An array of bytes MAY be encoded as a single-reference or a multi-reference value. The rules for an array of bytes are similar to those for a string.

In particular, the containing element of the array of bytes value MAY have an “id” attribute. Additional accessor elements MAY then have matching “href” attributes.“

Polymorphic Accessors: An accessor “…that can polymorphically access values of several types, each type being available at run time. A polymorphic accessor instance MUST contain an “xsi:type” attribute that describes the type of the actual value.“ Ex. 29.95

Enumerations: Based on the “XML Schema Part 2: Datatypes” (http://www.w3.org/TR/xmlschema-2/ ), which is still just a recommendation, a SOAP enumeration is “…a list of distinct values appropriate to the base type.” It is supported for all simple types except boolean.

User Defined Data-Types: Developers can define their own simple, or complex, data types.

Customizing

SOAP’s greatest feature is its ability to step past XML-RPC’s limitations and customize every portion of the message. This ability to customize allows developers to describe exactly what they want within their message. The downside of this is that the more you customize a message the more work it will take to make a foreign system do anything beyond simply parsing it.

Big Co. Support

SOAPs two biggest developers are Microsoft and IBM. Microsoft has incorporated SOAP into it’s latest OS and Visual Studio. As usual though, MS has decided that they don’t need to adhere to the whole spec. In their SOAP for Java SDK they modify the namespace (which breaks some other implementations), use a limited combination of 1.0 and 1.1 fault codes (and improperly document it), and include many serious bugs. IBM seems to have gotten almost everything right in their implementation.

Limitations

XML-RPC

Method Calls

XML-RPC calls methods via its methodName property which “may only contain identifier characters, upper and lower-case A-Z, the numeric characters, 0-9, underscore, dot, colon and slash.” For most purposes this is just fine, however it makes it particularly difficult when you need to pass an object as an argument.

Named Data Structures

The structs (hashes) and arrays are always anonymous. When passing multiple structs or arrays programmers need to rely on the order of the parameters ( ) they are contained in to differentiate between each struct or array. This isn’t a significant problem but there are cases when being able to name your structs and arrays would be nice.

Simplicity

As noted above, simplicity is also XML-RPC’s greatest limitation. Although most all of your RPC needs can be accomplished with it, there are some things that you just can’t do without bending over backwards, like passing an object as an argument to a function, or specifying which portion of a receiving application the message is intended for.

SOAP

Stability

Currently SOAP 1.1 is a submission to the W3C (http://www.w3.org ). This means it is not an official standard and its specifications can, and almost assuredly will, change at any time. There are a lot of cooks in the kitchen right now.

Documentation

Last updated in May the SOAP spec is filled with mistakes and contradictions and points to other specs which are themselves moving targets. According to Dave Winer, one of the spec’s authors, Microsoft and IBM have taken it upon themselves to add in the WSDL (http://www.w3.org/TR/wsdl ) layer without consulting the other authors. This doesn’t bode well for developers who need something they can count on remaining unchanged, or almost unchanged, for the forseeable future. Any current SOAP implementation may be incompatible with future revisions of the spec, although it should be noted that SOAP is far enough along that most of its spec should remain essentially the same.

Comparison

Feature	XML-RPC	SOAP
basic scalars	yes	yes
structs	yes	yes
arrays	yes	yes
named structs and arrays	no	yes
detailed fault handling	yes	yes
short learning curve	yes	no
Developers specified character set	no	yes (US-ASCII, UTF-8, UTF-16)
Developer defined data types	no	yes
Can specify recipient	no	yes
require client understanding	no	yes
message specific processing instructions	no	yes

When you get right down to it XML-RPC is about simple, easy to understand, requests and responses. It is a lowest common denominator form of communication that allows you to get almost any job done with a minimum amount of complexity. SOAP, on the other hand, is designed for transferring far more complex sets of information. It requires profuse attribute specification tags, namespaces, and other complexities, to describe exactly what is being sent. This has its advantages and disadvantages. SOAP involves significantly more overhead but adds much more information about what is being sent. If you require complex user defined data types and the ability to have each message define how it should be processed then SOAP is a better solution than XML-RPC (be sure to check out language specific solutions to this problem like java’s RMI (http://java.sun.com/products/jdk/rmi/ )). But, if standard data types and simple method calls are all you need then XML-RPC will give you a faster app with far fewer headaches.

“Make sure you don’t use and overly complex package to address what may be your rather simple needs” -Brett McLaughlin (brett@newinstance.com )

Sources & Other Links

XML-RPC

XML-RPC Specification http://www.xmlrpc.com/spec
Internet Scripting: Zope and XML-RPC http://www.xml.com/pub/a/2000/01/xmlrpc/

SOAP

SOAP specification http://www.w3.org/TR/SOAP/
SoapWare.org A busy developers guide to SOAP 1.1 http://www.soapware.org/bdg
Some considerations on SOAP http://www.xmlrpc.com/discuss/msgReader$1500?mode=day

Assorted other links

PI-RPC, the Platform Independent Remote Procedure Call http://www.blackperl.com/xml/PI-RPC.html is a proposed lightweight alternative to SOAP.
XML-RPC Extensions http://ontosys.com/xml-rpc/extensions.html add a value.
The Evolution of SOAP::Lite http://www.onlamp.com/pub/a/onlamp/2001/06/29/soap_lite.html
WSDL 1.1 http://www.w3.org/TR/wsdl

Financial Entropy of a Webapp Subscriber

2006-11-17T00:00:00+00:00

I was consumed with dreams about Stephen Hawking’s black hole entropy formula last night, which is frustrating because the math is, sadly, beyond me. But, I mention it to you today because, knowing so little about black holes my mind instead kept trying to change it into an formula to calculate the Financial Entropy of a webapp subscriber.

So, I put it to you, dear reader, have you, or any of your math enabled friends, come up with a formula for calculating the finincial entropy of a webapp subscriber? If you haven’t, but you could, there are many many entrepreneurs who would sing your praises and happily buy you a drink.

On a related note: a little Googling brings up this fascinating looking book whose price or $199.95 puts it beyond that of anything I’ll buy without having a damn good idea of what exactly is inside. Also, why is it that I never heard the phrase “Financial Entropy” until last night in my dreams?

[Edit]: 16 years later… The book is $4.99 on Amazon and I just ordered a copy.

Improved extract_fixtures

2006-11-14T00:00:00+00:00

I’m not sure where I originally came across the extract_fixtures rake task (maybe here)but there’s nothing better than using real data to run your Rails unit tests. Well, real in the sense that it was generated by actually using your app. But there’s a problem with extract_fixtures. Once you get some real data to base your tests on you don’t want it to change because it would break your tests. So, after the first run extract_fixtures becomes almost useless because it’ll wipe out the fixtures you’ve been working with.

So what’s a person to do? Well, in my case the answer is to beef it up a bit. The following version of extract_fixtures takes two optional parameters:

TABLES=foos,bars,other_foos
Tables takes a comma delimited (no spaces) list of table names that you want to extract. If you pass this in it will only extract those tables.
OUTPUT_DIR=/some/fully/qualified/non-relative/path
Tell it what directory to output to.

If you improve it further please drop me a line and I’ll add or link to your enhancements here.

desc "Create YAML test fixtures from data in an existing database.  " +
" Defaults to development database.  Set RAILS_ENV to override. " +
"\nSet OUTPUT_DIR to specify an output directory. Defaults to test/fixtures. " +
"\nSet TABLES (a coma separated list of table names) to specify which tables to extract. " +
"Leaving it blank will extract all tables."

task :extract_fixtures => :environment do
  sql  = "SELECT * FROM %s"
  skip_tables = ["schema_info"]
  ActiveRecord::Base.establish_connection
  if (not ENV['TABLES'])
    tables = ActiveRecord::Base.connection.tables - skip_tables
  else
    tables = ENV['TABLES'].split(/, */)
  end
  if (not ENV['OUTPUT_DIR'])
    output_dir="#{RAILS_ROOT}/test/fixtures"
  else
    output_dir = ENV['OUTPUT_DIR'].sub(/\/$/, '')
  end
  (tables).each do |table_name|
    i = "000"
    File.open("#{output_dir}/#{table_name}.yml", 'w') do |file|
      data = ActiveRecord::Base.connection.select_all(sql % table_name)
      file.write data.inject({}) { |hash, record|
        hash["#{table_name}_#{i.succ!}"] = record
        hash
      }.to_yaml
    puts "wrote #{table_name} to #{output_dir}/"
    end
  end
end

Why Rails Migrations are wrong headed

2006-11-05T00:00:00+00:00

Ever since migrations were introduced to Rails I’ve heard nothing but praise for them, and truth be told, they are a far better way of setting up your database than the standard raw sql import. But, that’s where the goodness ends.

The problem is in the concept of going up or down in database versions. The core concept is great, to be able to roll back to a previous version of the database, but the implementation is completely out of sync with the version control systems we use to manage the codebase that depends on that database. I’ll use subversion as an example because (for those of you still stuck in CVS land) every time you do a check in the system gets tagged with a new revision number.

Everything starts out fine. The initial migration reflects the needs of the initial codebase. But after that they’re never the same. There’s no way to now what migration corresponds to what version of the codebase. What if in checkin 400 I add a new migration that changes the schema. A few weeks later I need to roll back the codebase to version 380. What migration number should i roll the database back to if any? Unless you happen to remember what migration number you were on when the codebase was at 380 you’re screwed.

So what’s the solution?

Well, the solution starts with the schema.rb file that’s created every time you run a migration. If you’re like me, that gets checked in whenever it changes and the schema.rb file is always in sync with your codebase. So you’ve always got a representation of the appropriate database configuration for any revision of your codebase. That’s good. In fact that’s better than the actual migration scripts because it’s always in sync. But there’s one more problem…

Priming the database
Most webapps don’t function well with a completely empty database. There are usually some default settings, maybe an admin account to log in with the first time, things like that. And there are a few rake tasks out there to bootstrap your database with the contents of some .yaml files. There are others to dump your database into the same files for later import, or generating real data to run your tests against.

Putting it all together
If you make sure that your your .yaml files for bootstrapping are always in sync (you should be keeping them in sync anyway) then you can avoid the problem entirely by just making your own schema.rb file (it’s just a migration file for all the tables at once) and never running rake db:migrate again. Instead run your schema.rb (make sure it forces table creation) and then rake db:bootstrap. When you need to modify your database schema don’t make a new migration script. It’ll only confuse matters down the road. Instead, modify your schema.rb and make sure your .yaml files are in sync (you’d have to do that even if you were using migrations).

Damien Tanner points out that migrations are very useful when migrating a live site. And I have to agree. My proposed solution is seriously lacking in that department. What would most likely be best is a blending of both concepts. Don’t use migrations to muck about with your database during development. Instead, create migration scripts only for migrating data on live sites. Don’t title them “create_foo_table” or “add_foo_column” but instead name them something like “live_site_version_4” or something like that that makes it clear exactly what it’s for and how it pertains to your site.