Git on weblog.masukomi.org

Github Rubocop Workflow

Mon, 19 Jun 2023 00:00:00 -0400

It took me a while to figure out the correct collection of magical incantations required to make RuboCop run in a GitHub workflow, but ONLY on the files that were changed within the PR. This is a useful configuration if you have a codebase that has not yet been modified to satisfy all your “Cops”. I’ve also included a version of the same file that you can use when you’re ready to have RuboCop run on all non-excluded files.

The result is that rubocop will only block your PR if some of the files touched in the PR don’t conform to your new standards. Once your codebase is

The on: [pull_request] means that it will run when you make the pull request and any time you push to it (with or without force).


# A GitHub workflow to run rubocop, but ONLY on changed files
# honors any exclusions specified in your .rubocop.yml
#
# store this file in .github/workflows/rubocop.yml
# note: the Checkout Branches step is required
# because by default what's checked out is the branch
# that represents the future merging of the PRs branch
# with the target branch.

name: RuboCop

on: [pull_request]
jobs:
  rubocop:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v2
      - uses: ruby/setup-ruby@v1
        with:
          ruby-version: .ruby-version
          bundler-cache: true
      - name: Install gems
        run: |
          bundle config path vendor/bundle
          bundle config set without 'test tools functional'
          bundle install --jobs 4 --retry 3
      - name: Checkout Branches
        run: |
          git fetch origin ${{ github.head_ref }}
          git fetch origin ${{ github.base_ref }}
      - name: Run RuboCop
        run: bundle exec rubocop --parallel --force-exclusion $( git diff --name-only --ignore-submodules=all --diff-filter=ACMR origin/${{ github.base_ref }}..origin/${{ github.head_ref }} )

Once your codebase has finally been cleaned up to a point that RuboCop is happy with it you can change it to the following. This is strongly recommend because it will catch when code made it through to the target branch without getting checked (merged directly, or a PR that was merged despite failing checks). The time difference is not noteworthy. Spinning up the container and checking out the gems takes much longer than the actual RuboCop run.

# store this file in .github/workflows/rubocop.yml
# this will run rubocop on ALL the files in your repo
# that have not been excluded in your main .rubocop.yml
# rubocop configuration file.

name: RuboCop

on: [pull_request]
jobs:
  rubocop:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v2
      - uses: ruby/setup-ruby@v1
        with:
          ruby-version: .ruby-version
          bundler-cache: true
      - name: Install gems
        run: |
          bundle config path vendor/bundle
          bundle config set without 'test tools functional'
          bundle install --jobs 4 --retry 3
      - name: Run RuboCop
        run: bundle exec rubocop --parallel --force-exclusion

Mirroring With Gitea

Mon, 26 Sep 2022 00:00:00 +0000

Table of Contents

Overview

Following on the heels of my last post on why you should (not) self host your git repos, I went ahead and used Gitea to set up a local mirror of all my repositories, and all the repositories I don’t want to loose access to.

The results were surprising, and after reading this, you might want to do the same. This post will be a qick overview of how I did it, some tips that’ll help, and what I learned as a result.

Installation was pretty straightforward. I grabbed my old Raspberry Pi 3B and followed this spectacualar step-by-step guide for installation for installing Gitea. Afterwards a friend pointed out that I could have just installed installed it on our Synology with a Docker image.

The Goal

My goal was two-fold:

I wanted a backup of all my repos. Many of them only existed in GitHub.
I wanted to delete every fork I’d made just to guarantee I had a copy.

The Process

With that in mind I started working through my list of repositories¹. Gitea makes it trivial to mirror a repo. You just choose “new migration” from under the “+” menu, and then click on the service you’re migrating from. There are 3 things that can make your life easier here if you’re importing lots².

Bookmark the form that comes up after choosing a service to migrate from.
If you’re going to be importing a lot you’ll probably want to use a personal access token (GitHub docs). That way you won’t get rate limited by the API, unless you’re migrating (not mirroring) something with many thousands of issues.
Use a clipboard manager. It doesn’t remember your API key so you’ll be pasting it into every form, along with the URL of the repo you’re importing. I use Maccy, and I would recommend it.

For each repo in my list of repos on github, I’d make a couple quick decisions.

for things I authored
- If it was something I am either maintaining or may poke in the future I’d just grab the url and tell Gitea to mirror it.
- If it was something I wandered off from years ago, and am unlikely to ever touch again, I’d mark it as archived.
for the backups of other people’s things
- I’d click through and see if i’d added anything, or if it was just an old copy.
  - If I’d added something, and the original repo hadn’t been touched, I’d mirror my fork.
  - If I’d added something, but the original had moved forwards some significant amount I just abandoned my changes, and mirrored theirs.
  - If mine was just an ancient copy of a repo that’d moved on, I’d go to the parent repo.
    - If the parent repo was archived I’d just tell Gitea to import it but not bother mirroring. There aren’t going to be updates.
    - If the parent repo indicated that it too was a mirror, I’d follow that back to the source repo and mirror that instead.
    - After successfully mirroring I’d delete my copy.

The Learnings

(In no particular order)

First off, Gitea is really impressive. Those devs have done a great job and should be proud of what they’ve built.

Along the way, I discovered that many of the repos that I’d forked had moved on significantly. I had copies that were so old they probably wouldn’t run with modern versions of things. They were also frequently missing lots of new features and bug fixes.

While this isn’t terribly surprising if you think about it, the notable learning is that that’s never going to be a problem again, because Gitea is mirroring them and regularly polling them for updates. If they disappear I’ll have the latest copy before they went away. Github doesn’t give me that option.

I’m not the only one forking repos just to have a copy. Multiple times I clicked through to the parent repo, and got curious. I’d see hundreds of forks, click into the insights page and see that essentially no-one was hacking on it except for the original authors. I guess lots of folks have gotten burned when they discovered that some repo they’d starred or bookmarked was gone when they went to check it out again.

Mirroring isn’t something you can turn on later in Gitea. You must set a repo up as a mirror from the beginning.

Gitea can’t mirror everything. Some repos (like this great command line time zone tool) always error out. Gitea provides no information as to why it failed to clone them down too. I think I encountered three repos like that. Also, I can’t figure out how to clone anything from Sourcehut³. Even when I tried to clone repos manually their http links for cloning always error.

I realized that if i don’t just mirror the tools that I author, and want backups of, but also mirror the repos of tools that I need to reference the source code of regularly then I have a single place I can search for only stuff that’s relevant to me. For example, it’s not uncommon for me to know that one of the libraries I use frequently has a piece of functionality, but not remember which library. Searching Google or Github would be frustrating, because there are just too many responses. But now, I can just search my Gitea install.

The experience of doing this was very cathartic for me. I now know that I have up-to-date copies of all the useful tools I care about. I was also able to delete about 100 repositories from my list of repositories on Github. Now, if a future employer checks them out, they’re almost all things I have actively worked on to some degree. It’s also nice to know that from now on I have a trivially easy way to mirror any useful tool I care about. No more ancient frozen it time forks, and no more bookmarks & stars that cease to exist.

My next step? Going through all the repos I’ve bookmarked, but not forked, and see which of those are worth mirroring.

Before working through your list of repositories, be sure to sort them by name. Mine were sorted by recency and that made things a little harder towards the end. ↩︎
For a sense of scale I ended up mirroring or migrating 169 repos into Gitea. ↩︎
In addition to having a visual style from the 1990’s, and having UX from hell, Sourcehut appears to also just not fucking work as a git host. I don’t understand why anyone uses them, nevermind why anyone pays for them. ↩︎

Do (not) Self-Host your repos

Sat, 24 Sep 2022 00:00:00 -0500

Once upon a time, GitHub was a successful geek enterprise. Then Microsoft bought it, and folks started arguing that you should abandon ship. You should self-host your repos they say.

I 100% agree, and 100% disagree. Let me explain.

GitHub’s been a benevolent host. When they bought NPM they went from being a de facto piece of internet infrastructure to an actual piece of critical infrastructure. At this point you may as well argue that we should sever our connection to the electric company.

But there’s a difference between needing the electric company and ignoring the fact that sometimes the power goes. Usually it’s temporary. Days isn’t uncommon in wintery climes. But every now and then, it’s more. In the case of internet SAAS apps, it’s sometimes permanent.

Why You Should Self-Host

The biggest argument, of course, is backups and continuity of service. If something hapened and you lost access to your GitHub account right now, what would you loose? If you maintain a project with many users and contributors, what would it do to your community, and your work?

GitHub isn’t just where your Git repos are hosted, but where all the metadata that surrounds them. This means years of Issues, Pull Requests, wiki pages, and frequently the project’s web site.

I’ve derived a ton of value from old tickets on GitHub projects. They regularly answer questions, and provid workarounds. The takedown off a popular repo could be a huge loss.

GitHub suspends accounts, and deletes repositories for a variety of reasons. If yours targeted for seemingly “no reason” it wouldn’t be the first.

Even if you believe it won’t, or can’t, happen to you, there’s another reason reason to consider self-hosting: Lock in. Right now (September 2022) GitHub is very good about letting you export everything. As far as I know, there’s no data you can put into GitHub that you can’t also get out via their APIs. In theory.

Our migration has been in the process for a while, but every time we run it we find that our attempts were missing some data, so we need to re-run to fetch missing data, if we had higher limits it wouldn’t take so long to find these issues. Github doesn’t provide tokens with higher limits, so we have to work around that. We even had people with [GitHub Enterprise] subscriptions ask, and they got the same response.

Gitea on Mastodon

Gitea is an open source GitHub alternative, whose creation started out on GitHub. Eventually, the software got really robust, and they decided to host their own repo and issues. The problem is, they’ve got over 20,000 Issues & PRs they want to export. Thousands of those are still open, and the longer it takes them to cut over, the larger those numbers get.

To be fair, 99% of GitHub users wouldn’t have this problem. We don’t have projects that are popular enough, but companies who use GitHub do, and there’s no advantage to GitHub to help large projects or customers leave. They may even consider locking that metadata down to guarantee they stay.

What about GitLab and other Competitors?

You can apply the same reasoning to GitHub’s hosted alternatives. Plus, none of them are as financially successful as GitHub. They’re far more likely to go out of business, and screw everyone who uses them.

Why You Shouldn’t Self-Host

Right now it’s incredibly improbable that you’ll have your account suspended. The stories of suspended accounts are rare. Most of them seem to be from government action,or “Terms of Service” violations that most geeks react to with “Yup. Seems fair.”.

There’s also the fact that self-hosting is work. But let’s set that aside for the moment. Let’s pretend there’s some magical free button you can click that takes care of everything. Boop. You’re self-hosted now.

Let’s also assume you’ve got a project that’s popular / useful enough to get Issues and the occasional Pull Request.

How is that going to happen on your new site? Well, first they’re going to need to set up an account on your server. Are they going to think it’s worth it? Worth it enough to give their email to yet another stranger? If they want to submit a fix / feature, is it also worth it to figure out how to fork it in this not-GitHub? Worth it to figure out how its PR equivalents work? These probably aren’t hard, but every little difference they have to navigate is another reason to not bother.

And then there’s discovery. GitHub isn’t only the default place to look for code. They’ve also got massive “website authority” in Google. It doesn’t matter if you’re the only place hosting your thing. Any project on GitHub that’s remotely similar to your search terms is likely to come up before your site.

But, even if you’re ok with all of that. There’s an even bigger reason you shouldn’t self host. A reason we shouldn’t encourage folks to self host. Sooner or later you’ll get sick of bothering, or you’ll die. One is guaranteed. The other has lots of historical precedence.

I’ve bookmarked thousands of useful web sites over the years. A huge chunk of those have been useful, and insightful blog posts about development. Hundreds of them don’t exist anymore. Mostly because the person who ran them simply didn’t feel like bothering with it anymore. How many millions of others have poofed out of existence?

But, a repo isn’t some just random thoughts. Almost every repo contains some useful tool or library. Sooner or later, someone’s going to need tha. There are thousands, probably millions, of little repos out there containing the only bit of code in existence, that solves that particular problem. Sometimes they’re just the only thing that solves it in that particular language. When you stop hosting it, for whatever reason, that useful tool ceases to exist. Even if it’s “just a port”. That’s a lot of work that someone will need to recreate, and debug.

So what’s a geek to do?

Before I answer that, let me note that this bit gets into implementation strategies. If you just came for the question of “should I, or shouldn’t I?” you can stop reading. Thanks for making it this far. Good luck whatever you choose.

–

So, what to do…

Short short version: If lots of people rely on your project, then mirror your code to a large competitor and accept that you may loose access to your issues, PRs, etc. . Stick a note in your README so that folks know where to find your mirror before lightning strikes.

If you’re like most open source developers, you’ve got some repos that are mostly ignored by the world, and receive essentially no issues or PRs. No-one’s going to be stressed if your repo disappears for a week. So, Install Gitea on your computer. Tell it to mirror your repos and forget about it. Or, just clone the all locally and set up a cron job to regularly pull down changes.

Longer version: I think that depends a lot on what, if any, metadata exists in your GitHub repos. Most devs have small repos that are mostly just used by them. If this sounds like you, then simply keeping an up-to-date copy of them on your home computer could be fine.

Things are different if you maintain some repo that other people actually care about enough to make issues, or even contribute code to.

In that case my personal advice is to host your stuff on Github, but have it mirrored to a second system. Ideally you’d want all your issues, PRs, etc. mirrored to it. But as far as I know, that’s not actually something you can do. It’s easy to keep the code synced to a mirror. It’s the metadata that’s a problem. To be clear, I’m not saying it’s not technically possible. I’m saying, I don’t think anyone’s actually written the code to keep those synced between GitHub and any open source alternative. It’s a hard problem because Issues, PRs, comments, and just about everything else can be edited.

So, that leaves you with regularly backing up your stuff. I’m going to assume you’re not someone with ~10k open & closed issues, plus another ~10k PRs like Gitea. So, exporting your issues is probably something that’s actually doable. Exporting. Not syncing. However, the more things you have to export, and the more frequently you want to back it up, the more likely you are to hit your rate limits.

If you go this route, you’ve got two options. You could use something like the gitea-github-migrator that can export everything. Tools like that are made to create a new repo in GitHub alternatives, not sync to them. You’d be constantly churning, deleting old repos and setting up new ones.

A better solution might be something like github-issues-mirror which can download all your issues as JSON files. Keep that running in the background and if worst comes to worst you can write an importer for wherever you’ve been mirroring your code.

What am I going to do?

I think I’m going to install Gitea on my Raspberry Pi. I’ll tell it to mirror all my repos, and all the repos of useful tools I want to always have a copy of. I do maintain some repos which are somewhat useful to other people if GitHub deletes them. No-one will be screwed, and I can take my time in uploading a backup somewhere else. I’ll stick it in the basement with the router and back it up to our NAS, which backs up to B2.

Keeping a great Changelog

Sun, 03 Jul 2016 13:03:55 -0400

Changelogs are an invaluable, and often neglected part of any software project. So, how do you do that?

A good changelog helps you users to understand:

Why they should care about your latest version
If any of your changes affect the problems or frustrations they’ve been having.
If there are any changes that might affect how they use your app / library.
Why your efforts are worth their continued support.

A great changelog does all that, and shows the personality of your team.

Shifty Jelly’s Pocket Casts is notorious for their fun, and informative changelog entries:

Equal Opportunity variable speed playback. Why should MP3s have all the fun? Welcome AAC & OGG! * Bzzz. Bzzz. You can now choose which podcasts to be notified of.

Slack’s been doing a pretty bang-up job too:

Fixed: The app now automatically scrolls down when you begin writing a file comment, giving your precious fingertips much needed beauty rest. * Fixed: Time zones were sometimes shown with incorrect deltas. Now your coworker who is 5 hours ahead of you will no longer simultaneously be 8 hours behind you.

I can’t overstate how valuable it is to have a Changelog with personality. As soon as you see that one of the apps you use has a great changelog you look out for them, and always stay on top of what they’ve fixed and added. Then you start spreading the word, because not only did they add good stuff, or fix that really annoying bug, they made you smile along the way.

A Changelog with personality helps build evangelism for the thing you’ve spent so much time building.

The guts of a good Changelog Entry

It must speak in terms users understand. Most of them won’t be intimately familiar with your source code. Don’t use internal jargon. If a bug is fixed, speak in terms of the actions that caused it, or the behavior it exhibited, but not the specific classes, unless you absolutely have to.
It must be knowledge that’s useful to your users. Speak to the features and flaws that affect your customers. Users don’t care that you refactored some unit test, or reformatted your comments.
It has to be at the right level of abstraction. Even if your writing an open source library. The Slack timezone bug above was handled perfectly. They could have mentioned which class & method had the bug, or the conceptual flaw that caused the bug, but that’s all too fine grained for a Changelog. Instead, they spoke at a higher level of abstraction.

Changes need to be grouped by type

Group your bug fixes. Group your features. Group “all the things!”

Imagine a user who’s been particularly frustrated by a bug in your app. They’re going to open your changelog and immediately try and find out if you’ve fixed it in this release. Help them. Don’t frustrate them further by making them hunt for it in a jumble of features, fixes, and changes.

Dates are important

Versioning your software is great, but dates are important too. When did version 6.0.0 come out? Did I have that problem before then or after?

Additionally, not everyone in the world keeps dates in the same format. 06-08-16. Is that August 6th or June 8th? It depends on who’s reading it. Year Month Day is the only one that makes any logical sense no matter where you are. XKCD even had a Public Service Announcement about it.

To summarize

Write at a high level of abstraction
Speak to things that affect users, not developers.
Group things so that people can find them easily.
Add some personality.

Want an easy tool to help manage this?

Check out changelog_manager. It’s a free and open source tool to help you easily generate high quality Changelog entries.

Why you can’t auto-generate your Changelog

Why you can't auto-generate your Changelog

Thu, 30 Jun 2016 08:28:07 -0400

Let’s start by taking it as a given that a Changelog file is something very valuable that every product should come with. Even if your “product” is a library for other developers.

With that in mind, the question rises of “How can I make it really easy to generate one”. Many developers have had exactly that thought. There are many free and some paid solutions that will “Autogenerate your changelog from your git commits/tickets”. The simple fact is, that no matter how well they’re written, you shouldn’t use any of them.

A good Changelog entry has…

A good Changelog entry has three primary aspects

It’s at the right level of abstraction. Consumers of your product don’t care what files you tweaked or how. They care about the fixes and changes you’ve made that may impact them.
It needs to be knowledge that’s useful to the consumer of your product. It’s a simple judgement call, but it’s one that has to be made.
It needs to be written in a way that people who either aren’t geeks, or at least aren’t intimately familiar with your codebase can understand.

Changelogs from commits

Commits are at the wrong level of abstraction

A git log is filled with commits like: “ensured current_and_future_campaigns returns a relation”, or “cleaned up whitespace in User.rb”, or “fixed expected aspect name in update test for locations_controller_spec” These are useful to other developers on the project, but they aren’t things that are useful to the users of your product.

In short, they contain too much detail, at the wrong level of abstraction.

Commits are messages from one developer on a team to another one

They’re not messages to your users, and they shouldn’t be.

Commits frequently use jargon, and references to specific elements of code. This is absolutely fine, but it’s probably meaningless or useless to most of your consumers.

There’s no point

If I wanted a geek level listing of all the changes in your git history I could just ask git: git log --oneline Plus, git will let me slice and dice that information sixteen ways from Sunday.

You can’t edit them

As Commits From Last Night proves. Commit messages frequently aren’t written in a customer friendly way. Even if you’ve got very polite developers, everyone makes typos. You can’t edit the old commit. If you want a changelog that looks professional you’re going to have to manually edit it, the same mistakes every single time you generate it.

Tickets are at the right level of abstraction

…but they rarely have titles that are appropriate for a changelog.

How about “Item Finder - Page gets wonky after searching for content that doesn’t exist - plus 404”, or “Something’s eating up all the ram and taking down the servers” Not really appropriate for your changelog.

If you’re really following agile an agile methodology you have user stories that look like “As a I should be able to ___ because ___” and that’s no good for your changelog.

Some projects do take the time to curate their ticket titles. The results are almost usable as changelog entries too, but there are always exceptions. They’re generally worded well for the people who need to work on them, but not your consumers.

It’s a matter of human judgement

You want a changelog with the information that is useful to your consumers, and written at the right level of abstraction. This means people need to ask themselves some questions. “Do we need a changelog entry for this?”, and if so, “How do I summarize this change in a way that’s useful to our users?”

The first question sometimes eliminates information, through curation. That’s something tools can’t do. The second question changes the form of the information conveyed. It was directed internally, now it’s directed externally.

Want an easy tool to help manage this?

Check out changelog_manager. It’s a free and open source tool to help you easily generate high quality Changelog entries.

Keeping a Great Changelog

Finding the Github Pull Request for a topic branch

Tue, 24 Dec 2013 00:00:00 +0000

Finding the Github pull request associated with a branch.

Work on a large enough project, with other people and sooner or later you’re going to find some commit, or branch, and want to know what was in the pull request that merged it in. Maybe you want to see what other commits got merged over at the same time. Maybe you want to see what the diff was at that point in time. But, how do you get from a branch name to a pull-request.

The key to solving this is knowing that pull requests are actually branches. They’re just… hidden branches. So, first you start tracking them…

Locate the section for your github remote in the .git/config file. It looks like this:

[remote "origin"]
    fetch = +refs/heads/*:refs/remotes/origin/*
    url = git@github.com:joyent/node.git

Now add the line fetch = +refs/pull/*/head:refs/remotes/origin/pr/* to this section. Obviously, change the github url to match your project’s URL. It ends up looking like this:

[remote "origin"]
    fetch = +refs/heads/*:refs/remotes/origin/*
    url = git@github.com:joyent/node.git
    fetch = +refs/pull/*/head:refs/remotes/origin/pr/*

Now, after a git fetch origin you can say git checkout pr/999 to checkout pull request number 999 as a local branch and diff it, or do pretty-much anything else with it.

Once you’ve got that set up, it’s just a matter of comparing your local topic branch to the other branches in your system to find other branches that point to whatever branch you’re interested in.

I use a shell script called git-find-twins which iterates over the known branches (including pull request branches) and finds any whose tree-ish matches that of the current branch. You can find the latest source here in Github.

[Note: My apologies for the disappearing code on the right side in these examples. Just grab the source from the link above.]

    #!/bin/sh
    # Finds other git branches whose HEADs point to the same 
    # treeish as this branch.

    # thanks to mockinterface on 
    # http://stackoverflow.com/a/20756047/13973
    # for showing how to do this.

    echo "Looking for branches pointing to: $(git rev-parse HEAD)"
    git for-each-ref --format="%(refname)" | \
        xargs -I refname   \
        sh -c '[[ $(git rev-parse HEAD^{tree}) == $(git rev-parse refname^{tree}) ]] && echo refname'
    echo "DONE"

Add that to your path, and make it executable then check out the branch you want to find the pull request for, and run your new shell script with git find-twins (note the space) and you’ll see output like this.

$ git find-twins
Looking for branches pointing to: cb3c2dcf606a0a26107131697dd37aef53d44aad
refs/heads/9284_vzw_bug_fixtures_without_fixture_types
refs/remotes/origin/9284_vzw_bug_fixtures_without_fixture_types
refs/remotes/origin/pr/530
DONE

That last one is what you want. 530 the number of the pull request, which you can put into an URL to see the pull request. Now, you’re probably saying “Yeah, but what URL?” well, one way to get the correct url is to just bring up any other pull request in the current project and change the number in the url. Or you can add this shell script, which I call url_for_pr which generates an url for the pull request number in the current repo. It’s based on the assumption that “origin” is the Github repo. If it isn’t you’ll need to tweak what remote it gets its info from. You can find the latest source here on Github

[Note: My apologies for the disappearing code on the right side in this example. Just grab the source from the link above.]

    #!/bin/sh
    # when run from a git repository, and passed the number 
    # for a pull request, it will generate an url for 
    # where to find it on Github. This assumes that
    # "origin" is pointing to the Github repo.
    # This will work with Github Enterprise too.

    if [ $# -eq 1 ]; then
        ROOT_URL=$(git config --get remote.origin.url | \
            sed s/.*@// |sed s/\.git// | sed s/\:/\\//)
        PR_NUM=$1
        echo "https://$ROOT_URL/pull/$1"
    else
        echo "Please pass in a Pull Request number"
        echo "Note that this should be run within a git repository"
    fi

Rebasing A Pull Request on GitHub

Thu, 07 Feb 2013 00:00:00 +0000

It’s generally good practice to rebase commits on a topic branch into a single commit before merging. It results in a much cleaner commit history, and makes rollbacks easier.

The Question

However, the question was raised: what happens if you…

fix a bug (commit 1)
create a Pull Request
get feedback via the Pull Request
fix the bug fix (commit 2)
rebase those two commits together (new tree-ish)
push that back to GitHub (requires push -f )

The answer is based on understanding that a GitHub pull request has two forms of commenting: * comments on the pull request itself * comments on the commits that the pull request encapsulates. These are the comments made on specific lines in the diff.

The Answer

When you rebase and then force that changed version of history up onto GitHub you are effectively throwing out the old commit (the old tree-ish no longer exists) and replacing it with a new one.

As a result any comments created on the diffs (and thus tied to a specific commit) will be lost, but the comments on the pull request itself will be preserved.

So, the proposed rule of thumb is to simply NOT rebase once you have made a pull request. Instead, just add commits to your topic branch. This will maintain the comment history for future people curious about this piece of code.

Alternate Solutions

There are alternate solutions, of course, but the consequences of them tend to not be worth it.

Rebase Anyway

Decide that you don’t care about maintaining the inline comments, then rebase and push -f back up to the server. The downside here is that at this point someone has already started looking at your code and has likely pulled it down. They will now be forced to blow away their old branch (or at least reset --hard back to a point in its history before your commit), which is a pain and not obvious unless you tell them directly.

Make A Second Topic Branch

You could close the first pull request after having gotten the set of commits to an approvable state, and then open a new pull request from a separate topic branch with all those commits having been rolled up into one. The problem here is that if your new pull request is merged in then git blame will point to the tree-ish of the new one and no-one will ever see the comments on the old pull request unless they search, but when they do they may find that the commits in the original request don’t exist in the codebase (they’re all under the rebased commit).

Git: pushing and pulling from multiple repos

Thu, 31 Jan 2013 00:00:00 +0000

Lets assume you’ve already cloned a remote repo and have been working with it. Now, someone has set up a second repo out there for the same codebase, and you’d like to interact with both.

*Please note: The following is based on the assumption that you have write privileges to the second repo, but don’t worry, you do essentially the same thing if you don’t and I’ll cover the differences at the end. *

git remote add <local_name_for_remote_repo>
git checkout -b
<local_name_for_remote_repo>/master
- could be any branch on that repo, not necessarily master

Now, your new local branch is now set up to track (push and pull to/from) the remote repo’s “master” branch. All your pre-existing branches will continue to track whatever they were tracking previously

A practical example:

git remote add repo2  git@github.com:masukomi/kudos.git
git checkout -b repo2_kudos -t repo2/master

Workflow

Let’s say you’ve decided to add a new feature in the kudos repo we just connected to.

Checkout the repo2_kudos branch
git pull to get the latest code from repo2/master
git checkout -b adding_feature_x
- this new topic branch is not yet tracking anything so if you attempt to push or pull it won’t work, because it won’t know what endpoint you want to interact with.
Make your changes on this topic branch.
When you’re ready run git push -u repo2 adding_feature_x
- This will do three things:
  1. It will create a new branch in the repo2 repository named “adding_feature_x”
  2. It will push the contents of your adding_feature_x branch into the corresponding one on repo2.
  3. It will configure things so that your local adding_feature_x branch is now tracking the remote adding_feature_x branch so push and pull will work without additional arguments. You must pass the -u parameter for tracking to be set up here. If you fail to pass the -u parameter at this point then push and pull won’t work without additional parameters. You can, however, set up tracking after the fact with git branch -u repo2/adding_feature_x

Now that you’ve pushed your changes to the new branch on the remote repo you can initiate pull requests from your topic branch into the main branches (if you’re using github) or you can simple tell others about it so that they can interact with it.

Keep in mind that it’s best practice to regularly pull from your main branch into your topic branch (repo2_kudos into adding_feature_x in this case) so as to avoid large merge conflicts. Alternately you could pull from the corresponding main branch on the second repo (repo2/master in this case) since they should be equivalent.

When you don’t have write privileges

If you don’t have write privileges to the second repo things change slightly. The difference is in where you push your new topic branch (adding_feature_x) to. You can either push as a new branch on whatever repo your local repo was cloned from (if you’ve got write privileges to that) or you can create a third remote repo that you control, and push the branch to that. The third repo would be configured in the same way as the second, and you should be able to push to it in the same way we pushed to the second. Git makes it easy to flow code from one repo to another, or through multiple repos.

Using Git Bisect to Crush Your Enemies

Tue, 10 Apr 2012 00:00:00 +0000

Using Git Bisect

…to crush your enemies and/or bugs

Or, how to save countless hours and find out where things broke

Git bisect is the most awesome, and most poorly publicized feature of git. It allows git to walk through your branch and quickly find out which commit broke things.

The usage is simple. You point git to a bad commit ( usually the most recent one ) and you point it to a good commit (the most recent one you know of when things were working). So, if, for example, things were working on Tuesday morning, you bring up git log and scroll until you find one from Tuesday morning or maybe late Monday and copy its hash.

$ git bisect start
$ git bisect bad
	# that tells it that the current commit is bad
	# but you could say git bisect bad 48476f7b15022526393e6c4f44f610f552736fdc
$ git bisect good 71f45f2a0302b5fb2331c268ee2bfb3cfde452ab
	# that's the hash of some commit you know to have been good

git will then find the commit mid-way through and wait for you to tell it if it’s good or bad (git bisect good or git bisect bad). If it’s bad it knows the offending commit was earlier. If it’s good it knows the offending commit was later. It continues on in that vein, bisecting the commits with each step until it eventually says something like 54cb86bd35c226dc0df83dc2e2d4c8702fcf2c04 is the first bad commit

When you’re done you run the following to reset your branch to its original state

$ git bisect reset

Full Automation

But it’s even better than that. You can fully automate git bisect. Just kick it off and have it test each branch. All you have to do is write a script that can be executed from the shell and exits with 0 if the commit is good, and a number from 1-124 if it is bad.

You only have to do one thing differently. Once you’ve told it a good and a bad commit it will bisect the commits as usual and give you a system prompt and wait for you to tell it if it is a good or bad commit, but instead of telling it if it’s good or bad, you tell it what it needs to run to test it.

$ git bisect run my_test_script.sh

For example: I’m working on a rails project at the moment and just used this simple rake task which would blow up if the .fixtures method wasn’t loaded correctly:

# lib/tasks/location_test.rake
namespace :location_test do

	desc "tests location.fixtures classload issue"
	task :run => :environment do
		begin 
				Location.first.fixtures
				$stderr.puts "it worked"
		rescue Exception=>e
				$stderr.puts "it died: #{e}"
				exit(1)
		end
		exit(0)
	end
end

So, my run looked like this:

$ git bisect start 
$ git bisect good bebcc378d16d8fe74736c763c144753b2a382933
$ git bisect bad
$ git bisect run bundle exec rake location_test:run

And git bisect proceeded about its merry way until it found the offending commit. Then I ran git bisect reset and went about fixing the bug.

Pro-Tip

The Git Book adds this great note:

Note that the version which git-bisect checks out for you at each point is just a suggestion, and you’re free to try a different version if you think it would be a good idea. For example, occasionally you may land on a commit that broke something unrelated; run

$ git bisect visualize

which will run gitk and label the commit it chose with a marker that says “bisect”. Choose a safe-looking commit nearby, note its commit id, and check it out with:

$ git reset --hard fb47ddb2db...

then test, run “bisect good” or “bisect bad” as appropriate, and continue.

Obviously this won’t work if you’re running a fully automated bisect, and it points to the one gotcha of git bisect. If there are multiple bugs that inhibit your ability to test it ( either manually or via a more automated means) it can leave you with no obvious candidate. Most of the time, this isn’t a problem, especially when you have done your best to chose the most recent “good” commit that you are confident in.

git status-report

Fri, 16 Jul 2010 00:00:00 +0000

Every week I, and millions of developers like me, have to put together a status report for our bosses, letting them know what we’ve been up to for the previous week. Like most of the developers I’ve encountered I’m always a little unsure of what *exactly* I was working on, and typically I just open up git to see what commits I made, and try to remember any non-code stuff I’ve Thinking it was silly to keep wading through everyone’s commits for the past week to see what I worked on I’ve put it all together in a script (in Ruby) called git status-report, which you can grab from github here.

It outputs a bullet list of all your commits for the past n days, or the commits of one of your fellow contributors. I think I’ll find the latter a great way to keep up on what other folks on my team are working on at the moment.

Usage: git status-report [-e <email>][-d <days>]
Generates a status report for what you worked on for the past n days.
    -a, --all                        Displays output for all contributers sorted by person
    -b, --branches <branch name>     Limit results to work found in the specified branch(es). 
                                         Multiple branches can be comma separated
    -c, --current                    Generates status report for the current repo only
    -d, --days <num days>            The number of days to generate the report for. Defaults to past 6 days (one week)
    -e, --email <email address>      The email address of the committer to generate a report for. 
                                         Defaults to your "git config user.email" address. 
                                         Multiple email addresses can be comma separated.
    -f, --files                      Display a list of files involved in the commit(s)
    -h, --help                       Displays this message
    -r, --reverse                    Displays output in reverse chronological order
    -s, --simple                     Simpler output. Doesn't include the dates
    -v, --verbose                    Verbose output includes details on each commit

Please don’t hesitate to fork the repo on github and make improvements. I’ll happily merge them in to the main one. git-status-report supports will check for commits in all the branches and supports multiple repositories. Check the README for full details.

Git Rebase: why, and when, you'd use it.

Sat, 08 May 2010 00:00:00 +0000

Rebase is one of the most powerful tools in Git’s arsenal, but it can trip up people coming from centralized version control systems. This is just a quick example of why, and when, you’d want to use it.

Let’s say we’ve got a team of three developers. Monday morning they all come in, Bob makes a quick commit, and shares it with everyone. They all do a git pull and suck it into their repos.

Tuesday Mary does some work, but Bob and Tom are taking the day off and don’t pull in her changes yet.

Wednesday Tom makes a change and everyone pulls it into their repos.

Now depending on what changes Mary has made Bob and Tom may be able to pull them in without problem. Then again it could result in a conflict. But Mary’s a pro who thinks of her teammates, so she’s going to rebase her repository and reorder the commits so that her change is on top of Tom’s. Now, if Mary’s changes were such that they’d cause a conflict when Bob and Tom pulled them in then they’ll cause a conflict for Mary when she rebases, but they’re Mary’s changes so she’s the one most capable of resolving those quickly. Also, if Mary takes care of it in the rebasing then no-one else is going to have to deal with the conflict. It’s much better to have the person who wrote the change resolve it than have everyone else on the team have to deal with it.

So, Mary reorders her commits with git rebase, and if there were any conflicts she resolves them. When the others pull down her changes it’ll look like she wrote them on top of the changes they already had.

Another reason Mary might want to rebase is if she made a bunch of partial commits along the way. She might do this to give herself multiple roll-back points, or maybe she committed every time she finished working on one of the files related to the current feature she’s implementing, just so she wouldn’t have to worry about loosing anything. Now, she wouldn’t want to share those partial commits because they’d probably break something in other people’s system. So, when she finishes with her work, she uses rebase to squash the partial commits into one nice clean commit and then tells people to pull that, or maybe pushes it to some common repo they all push changes to.

Rebase is an incredibly powerful tool, but there’s one thing you must remember to never do and that’s rebase commits that have already been shared with others. For example: Git wouldn’t prevent Mary from rebasing the “Bob - Monday” commit up after all the other commits, or merge it into the commit she made on Tuesday, but doing so is guaranteed to cause problems for anyone who tries to pull from her. So rebase your own work before you set it free, but once others have it you can’t touch it.

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

If you found this useful you might be interested in some of the other git posts here.

Handling and Avoiding Conflicts in Git

Sat, 12 Jul 2008 00:00:00 +0000

John Kelvie said:

[To] me the fundamental challenge with existing version control systems is the difficulty of merging change sets from multiple developers across the same set of code. To me, this issue comes down to the diffing/merging functionality provided by the software, and I haven’t seen or heard of anything that really improves the state of the art. How does GIT address this? How does it make it easier to do? Are there specific branching and merging tools it provides? Is through the use of more atomic commits (which I could see helping to an extent, but only so far as it allows for changes to be small enough that there is no overlap, thus sidestepping the problem).

There are two concepts that you must understand to really take advantage of git. The first is the index / staging area. A full description would require its own post but for this discussion you can think of it as a temporary branch where you put everything before committing it. You can diff between it and the last commit, between it and the working directory, etc. The second is that a git repo can have (and usually does) have multiple branches in the same location on disk.

The idea of multiple branches in the same folder, is critical to understand when trying to address conflicts. In most version control systems you have one checkout for every branch, each in a different location on your disk, and while this is an option in git, you can also have multiple branches in the same folder, just like the server side of most version control systems.

So, we’re going to need an example. Let’s assume I’ve got a git repo with foo.rb. In the master branch foo.rb looks like this:

if (value == "test") puts "I'm in!"
end

On branch_one we’ll change that test to

if (value == TEST\_CONSTANT)

and on branch_two we’ll make it

if (value =="test" || value == "test2")

Now, if master, branch_one, and branch_two are all in the same repo we could stay in the same directory and “checkout” whichever branch we wanted to be working on. All the would change in place without you ever leaving the directory. So, if you checkout master and cat foo.rb you’ll get the first example. Checkout branch_two and cat foo.rb (without changing directories) and you’ll see the last example. I’m sorry if that was obvious, but it’s critical that you understand that, and some newbs just aren’t aware of it yet.

So, back to John’s questions. How does git make branching and merging easier to do, and does it provide any specific tools for managing it? Well, first off the commands are freaking simple:

to create a branch in the same repo git branch branch_name or, more commonly git checkout -b branch_name The latter creates the branch then checks it out so you can immediately start working on it.
to merge in a branch in the same repo git merge branch_name
to merge in a branch from a remote repo you’ve got git fetch and git pull More on these two in a minute, because they do dramatically change how you manage conflicts.

As for specific tools for managing it. Yes and no. “…if you rather use GUI tools to merge files instead of editing file with conflict markers (like shown in example), you can use git-mergetool, which would call GUI tool of your choice; currently supported out of the box are: kdiff3, tkdiff, meld, xxdiff, emerge, vimdiff, gvimdiff, ecmerge, and opendiff.” BUT, regardless of GUI tools, the ability to have multiple branches in the same local repo changes everything.

Here’s an example. Same files as above, except I’m using a different repo for each instead of different branches in the same repo, because that’s the way it would be when pulling from other people, and since we’re dealing with people, we’ll say branch_one is Mary’s branch and branch_two is Bob’s (pretend that the. The first thing you’d do if you were pulling from these guys regularly is to add them to your repos list of remote directories

$ git remote add marys\_branch ../branch\_one/
$ git remote add bobs\_branch ../branch\_two/ 
$ git remote show bobs\_branch marys\_branch

And here’s where the difference between pull and fetch come in to play. fetch tells git “hey, go get this remote data, and shove it into a “remote” branch in my repo. pull says, “fetch it, and also merge it with my current branch.” Using fetch we can suck in data from two sources, knowing full well it would conflict if we tried to merge it, but still have no problems.

$ git remote update
Updating marys_branch
remote: Counting objects: 5, done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From ../branch_one
* [new branch] master -> marys_branch/master
Updating bobs_branch
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), remote: reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From ../branch_two
* [new branch] master -> bobs_branch/master

$ cat foo.rb
if (value == "test")
puts "I'm in!"
end

So, now, we’ve pulled in two people’s conflicting changes, but not applied either. So, lets assume I don’t suspect a conflict (hopefully they’re not normal occourrances for you).

$ git merge marys_branch/master
Updating 7d6f564..5713885
Fast forward
foo.rb | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)

$ git merge bobs_branch/master
Auto-merged foo.rb
CONFLICT (content): Merge conflict in foo.rb
Automatic merge failed; fix conflicts and then commit the result.

Damn, a conflict! Who would have guessed? Now, I do what git suggests, fix the problem and commit it, but I’m neither the one who broke it nor the one who knows most about it. So, I’m going to kick this back to Bob or Mary, but, I may want to see what’s going on, to help choose who to kick this to.

One thing I can do is see specifically which commits are conflicting:

$ git log --merge
commit f068f89e88e9174b50a0bc5622875dd4d8e21bc8
Author: Kay Rhodes
Date: Sat Jul 12 09:26:44 2008 -0400

switched to testing for test or test2 in branch_two

commit 5713885c9adc5689b6d8222b2650d4b3ad0dbc42
Author: Kay Rhodes
Date: Sat Jul 12 09:25:52 2008 -0400

switch test to TEST_CONSTANT on branch_one

If the commit messages aren’t enough info I can actually diff the two branches that conflicted totally separately from anything in MY branch.

$ git diff marys_branch/master bobs_branch/master
diff --git a/foo.rb b/foo.rb
index dac3b0f..220011b 100644
--- a/foo.rb
+++ b/foo.rb
@@ -1,3 +1,3 @@
-if (value == TEST_CONSTANT)
+if (value == "test" || value == "test2")
puts "I'm in!"
end

I could have also diffed the specific commits in conflict if I’d wanted, instead of the entire branches. In this case it would give the same result. But, not knowing what TEST_CONSTANT is (maybe it’s a closure, maybe it’s a predefined variable) I can either make an educated guess that whatever TEST_CONSTANT is it’s probably a better choice from a maintenance perspective and tell Bob to go fix his stuff. Or, I could just send an e-mail to both of them and let them figure it out.

It’s also important to point out that the “index in git contains on conflict *all* versions of a file: ‘ours’ i.e. the version in the branch you merge into, ’theirs’ i.e. the version in the branch you are merging, and ‘base’ i.e. the version in the common ancestor ot the branches, and also version with conflict markers.”

Another option is to just fix it myself. The conflicting file currently looks like this:

<<<<<<< HEAD:foo.rb

if (value == TEST_CONSTANT)
=======
if (value == "test" || value == "test2")
>>>>>>> bobs_branch/master:foo.rb
puts "I'm in!"
end

HEAD is git slang for “the branch you’re currently working on” which, in this case, has Mary’s changes successfully merged in (actually, by default it’s pointing to the last commit on the current branch). So, I decide that all three tests need to be there and make the edit, and then update the index. You need to update the index because git has already added Bob’s changes to the index but knows that there was a conflict that needs resolving.

$ git status
foo.rb: needs merge
# On branch master
# Changed but not updated:
# (use "git add ..." to update what will be committed)
#
# unmerged: foo.rb
#
no changes added to commit (use "git add" and/or "git commit -a")

Updating the index is simply a matter of getting the file how you want it and saying:

> $ git add foo.rb

at which point committing would get you a commit message template that started like this:

> Merge commit 'bobs\_branch/master' Conflicts: foo.rb \# \# It looks
> like you may be committing a MERGE. \# If this is not correct, please
> remove the file \# /Users/krhodes/temp/trunk/.git/MERGE\_HEAD \# and
> try again. \#

The basic take-away from all this fetch / pull stuff is that it’s generally a good idea to fetch first when dealing with other people’s work, because having a local copy of their branch in your repo gives you a number of additional options if things go wrong.

And yes, atomic commits go a huge way towards avoiding the problem of conflicts, and handling them when they crop up. Obviously this applies to centralized systems too. They avoid them by keeping your commits on topic. If one commit is for bug X you generally don’t have to worry about it conflicting with a commit for bug Y. And, when you discover that two people you’re pulling from both have patches for bug X that conflict you have many more options.

Another key technique for avoiding conflicts is to merge constantly. Every single day you should be pulling any work that’s been committed upstream. That way, when things do conflict, you only have to resolve a little tiny thing (if the commits are atomic then it’s even tinier), and, if you choose to kick the issue back to someone else it’s still fresh in their heads. This, of course, applies to centralized systems to.

Another way that git (and every other distributed system) helps to avoid conflicts is a social change that comes out of the distributed nature of the repos. Because you’re constantly pulling in from multiple sources, having atomic commits becomes a requirement to participation. If Bob is a twit and only makes massive sprawling commits you’re simply not going to pull from him. It’s not like a centralized system where you’re forced to take all the changes out there. Whoever is in charge of the main trunk would be stupid to accept Bob’s changes because if there was a problem in one of them it’d be a pain in the butt to extract the bits that fixed bug A from the bits that didn’t really fix bug B. No, they’d kick it back to Bob, telling him to clean up his stuff and make two separate patches.

This becomes even more important when you’re working on a project that manages patches via e-mail. Just check out the git mailing list. People’s patches are incredibly focused, enabling other git developers to take just the bits and pieces they choose. Because the patches are uniquely identifiable in git, it doesn’t matter if some people take a change and some don’t and you, as the maintainer, pull from all of them. Git won’t think they’re two different but identical changes. It’ll know, “Oh, that’s change x. I’ve got that already.” Git also doesn’t care how a patch got there. It could be from a commit you pulled over the lan, or something someone e-mailed. Because it’s guaranteed cryptographically unique, it’s the same thing.

[Update 2] Some additional tips from Jakub Narebski

If you want to see what some file looks like on other branch (or for example at specified point of time, for example at tagged revision, marking some released version), you don’t need to use “git checkout ‘branch’ && cat ‘file’”; you can use “git show ‘branch’:‘file’” (see “Examples” section in git-show(1) manpage, and “Specifying revisions” section in git-rev-parse(1) manpage).

Below there is example how index looks like during conflicted merge. The example uses yet another way of specifying object names, i.e. “:’stage’:‘file’” (we could alternatively use SHA-1 or shortened SHA-1 of object shown in git-ls-files output, e.g. “git show 2f096cc”).

$ git ls-files –abbrev –unmerged
100644 2f096cc 1 foo.rb
100644 89f36fe 2 foo.rb
100644 d3ea75d 3 foo.rb
$ git show :1:foo.rb
if (value == “test”)
puts “I’m in!”
end
$ git show :2:foo.rb
if (value == TEST_CONSTANT)
puts “I’m in!”
end
$ git show :3:foo.rb
if (value == “test” || value == “test2″)
puts “I’m in!”
end

Try it yourself!
I find that reading how to do things like this is much enhanced by actually having something to test it on, especially when there are so many variables. So, I’ve uploaded the temp dir with the three repos I used in writing this. If you go into the trunk repo you’ll see that the other ones have already been configured as remote repos. And, if you go into branch_one or branch_two you’ll see that they’re clones of trunk and thus both already know that their upstream is trunk and how to fetch from it without having to configure anything. You can also see what the difference would be in pulling and fetching from one of them.

Please let me know if there’s still something that’s a little unclear about dealing with conflicts in git and I’ll add it in.

If you found this useful you might be interested in some of the other git posts here.

Why you should use a distributed version control system

Sat, 28 Jun 2008 00:00:00 +0000

If you’ve ever:

made a commit and then realized you forgot “one little change”.
made a commit and regretted it.
wished you could combine some the past couple days worth of commits into one nice combined commit in the main branch.
wished you could commit just part of a file.
needed to drop work on one task and switch tracks to another one without having to make commits with unfinished changes, or commits with changes for one issue and a little of another.
wanted to make a test spike with version control and without polluting the public repo.
managed an open source project.
wanted the security of knowing that there was a valid backup of your revisions on many other peoples boxes, or even just your own.
been frustrated with branch namespacing issues
been frustrated with how difficult branching and merging is in most centralized version control systems.
wished you could just create branches to work on a feature or a bug without worrying about the consequences to the main repo.
wondered which branch a bug applied to.
wanted to use version control when you were offline.
wished you could quickly compare versions of entire trees.
wished you could easily release everything in the current branch “except that”.
been concerned about how to scale a system to support hundreds, or thousands, of users.
been concerned about what would happen if your main repo box died.

…then distributed version control is worth your consideration.

Now, there are a variety of DVCS to choose from, Git and Mercurial (Hg) are main contenders, and both are really good. My experience is primarily with Git so I’ll be speaking as a Git user, but Mercurial can do most everything Git can, and a lot of this is general to all Distributed Version Control systems.

Quite possibly the most powerful, world altering thing about DVCSs is how they handle branches. Because of their distributed nature branching and merging absolutely must be simple effective, and avoid conflicts whenever possible. As a result people who’ve gotten the DVCS religion tend to make a branch for practically every bug or feature. We call these “topic branches”. If you’ve been working on one task for a while and a bug comes in that has to be address NOW it’s not an issue. Commit your changes in the current branch (or “stash” them away) and make a new branch for the bug that just came in. I doesn’t matter if you commit unfinished work in one of these branches, or if the work totally breaks the build, because it’s your branch. When you’ve finished working on a feature or bug you can “rebase” all your interim commits on that task into one nice clean one, or maybe generate a patch to apply to the main branch.

The fact that you can as many of your branches as you want solves a huge number of problems. Combine that with the ability to “rebase” you changes, by bringing up a list of past commits to merge, reorder, or exclude, as you see fit, and …well it’s like heaven. The only caveat is that you can’t rebase commits that others have already pulled. But that’s not a big deal if you simply do your work in personal branches that others aren’t going to be pulling from.

Because most DVCSs allow you to cherry-pick exactly which commits you want to be in a branch you can easily build a release branch with just the parts that you want. There’s one caveat to cherry-picking in most distributed and centralized systems : to do it well you must have commits that are a self contained and atomic as possible. If you make a commit with multiple features/fixes in it and later on you decide not to include one of them there’s no simple way to exclude just the bit you don’t want, because it’s mixed in with all the others. And this, brings us back to the idea of “topic branches.” If you’re already working with topic branches then you can easily merge the changes in one topic branch into the main branch as one atomic commit that can be excluded or included individually. However, as Aristotle pointed out in the comments

“the power of [Git’s] rebasing allows you to go back, split the commit sensibly into several, and then transplant the rest of your history on top of that. I guess whether this is simple depends on your definition of simplicity, but git has enough tooling to support this procedure directly. In other DVCSs you have to painstakingly monkey around with far less powerful tools, and in centralised VCSs it is for all intents and purposes impossible to do it at all.”

Yes, there will always be some changes that are dependent upon some prior commit, but by having nice little atomic commits you can limit the tree of dependencies to a bare minimum.

One of the best features of DVCSs is speed, but unfortunately it’s a lot like the Matrix. No-one can be told how much of a difference it makes. You have to experience it for yourself. I never complained about the speed of the centralized systems I worked with until I switched to Distributed Version Control. I found it hard to believe that “instant” commits would make a difference. Or that the time it took to diff against past versions was an issue. But I swear to you that the effect on how you work is almost as radical as the improved branching, and it flows over into the branching too. Now branching and merging is not only painless, it’s instant. I create, merge, and destroy branches throughout the day without hesitation because no matter what operation I do it comes back so fast that I frequently am incapable of telling that any time passed. That’s means that many (most?) of my operations are happening in the sub 100ms range. Imagine being able to diff versions of entire trees in tenths of seconds.

Scaling to thousands of users is not an issue with Distributed Version Control Systems because one box doesn’t have to support all the simultaneous operations that would normally be going on. Commits, checkouts, reverts, branches, diffs…. all of it happens on peoples local machines. They rarely need interact with the canonical / central repo. This means that large companies like Google wouldn’t need to invest in massive boxes with huge processors and piles of ram just to keep their version control system usable.

Speaking of the canonical repo… workflow doesn’t have to change at all. If you’ve got an established workflow with a centralized system there’s absolutely no reason it can’t continue on after a switch to Distributed Version Control. But, it does open up some other ways of working that you may want to explore.

In a centralized system if the box your main repo lives on goes down you’re pretty much screwed. If your IT people are really on the ball you’ve got it syncing to another box every few minutes (or more) and everything will fail over… bringing the primary box back up and resycing it may cause them to pull out their hair but at least no-one would be affected. Unfortunately it is a very rare company that’s that well prepared. And if you’re hosting an open source project you just have to hope that the people you’re hosting with have good backups and don’t go down.

Most software companies, and open source projects, simply grind to a halt when the version control system goes down. Changes can’t be shared, branches can’t be made, bugs can’t be patched, releases can’t be made, developers start building up a backlog of changes that will end up mooshed together into one commit when things do come back up, etc., etc., etc. But teams using DVCSs simply don’t care. It’s a non issue. “The heads crashed on the drives of the main repo, and a fuel truck plowed into our offsite backups”. It simply doesn’t matter. Sure your IT guys are going to have to deal with fixing that, but the impact to development is essentially zero. Someone stands up, or sends out an e-mail, and says “the main repo is down, just use my box instead”, and people do, and that’s the end of it. When the main box is repaired IT simply pulls from the box everyone’s been using in the interim and once again stands up, or sends out an email saying “the main box is back up”. Automatically syncing a remote box for fail-over becomes trivially simple with DVCS. Just set up a cron job to pull from the main box every minute and you’re done.

When it comes to open source projects there is one factor that absolutely sucks and that’s giving out write permissions. You don’t want to give them out to just anyone, but if you don’t then people who you don’t trust yet either have to work on your code without any revision control, or as is frequently the case, they just fork it into a version control repo they do have control over and loose the ability to easily merge their changes back into the main repo. You could just give them out to anyone but that’s not a choice that many project managers are comfortable with. With DVCSs it simply isn’t an issue. Everyone’s got their own repo(s) everyone can commit, branch, etc.. And when they’ve got something worthwhile they can ask you to pull from them, or send you a patch that they know will work with your repo because that’s where their repo originated and they’ve been pulling down your changes and merging it in to their work.

One problem that’s been an annoyance for me for years is that most bug trackers have no concept of branches. When you go into your bug tracker it shows you all your bugs but there’s no way to tell what branches they exist in, and what happens when you fix a bug, and get QA to sign off on it, in one branch but it hasn’t been patched in another one? Is the bug fixed or isn’t it? And what about the people who don’t have an intimate knowledge of that bug? How are they supposed to know where it does and doesn’t exist?

Distributed Issue Trackers solve this problem. You file a ticket in the branch the bug exists in. Then that ticket follows along with and branches of that branch until someone closes the ticket. As they merge the fixes back into parent branches the closed ticket merges in too. As a result you can always tell if a bug is or isn’t fixed in the current branch. If you commit your closed ticket with your patch then even more possibilities open up. If, for example, it was decided that your cure was worse than the cause and excluded that commit from a release then the ticket would re-open itself in the release too (because you got rid of the fix).

To date the best Distributed Issue Tracker, without question is Ditz, but the field is still very young. I’ve personally been working on a fork of Ditz that makes some dramatic changes and tightly integrates itself with Git, and I’ll update this page and make an announcement as soon as it’s released (a week or two). Personally I wouldn’t recommend anything other that Ditz right now. The projects are either too immature or abandoned and buggy. Ditz doesn’t have a lot of features yet but it’s reliable and gets the job done.

Another cool thing about Distributed Version Control Systems is that you don’t need anyone’s permission, or cooperation, to start using them. I work at a company that uses Perforce, and I think that Perforce is the devil, but it doesn’t matter, because I have Git. I do all my work in Git; constantly branching and merging and mooshing commits, and when I’m ready I just submit the completed changes back to Perforce. And this is actually better for everyone. Sysadmins don’t have to deal with the consequences of me making topic branches all over the place, the commits I make for coworkers are generally cleaner, and I can commit as often as I want, even break the build, without having to worry about the impact on others. Now, it happens that Git has a number of tools for working with the major centralized version control systems (it’s even got a CVS proxy so that CVS people too resistant to change can keep working the way they’re used to). But, if your DVCS of choice doesn’t have a tool to bridge the gap to whatever centralized system you’re forced to deal with, you can always use Tailor.

It sounds too good…

Some of you are probably thinking this sounds too good to be true, or maybe that I’m DVCS zealot. But really, it isn’t, and I’m not. I’m a huge fan of DVCS, it’s true, but I’m not about to claim that they will solve all your problems. To really get the full benefits of them you’ve got to start making atomic commits. This is actually true of centralized systems too, but since centralized systems tend to be such a pain to use, and frequently can’t do cool things like cherry-picking, people don’t even bother to do anything advanced with them. With a distributed system the advanced stuff becomes trivial every-day stuff, and it becomes more annoying to have to deal with those developers who won’t stop making ginormous commits with changes to multiple bugs as well as a new feature or twelve. Fortunately, you don’t have to accept their patches of you don’t want to. ;)

Guis have come a long way since this was first posted, especially for Git and Mercurial with quality that rivals the best tools for centralized systems.

On the command line some DVCSs like Darcs shine with totally intuitive commands, others have a bit more of a learning curve. Git definitely does because of how it stages things before commits. It appears similar enough to the way centralized systems work that many newbs expect it to work the same, and get frustrated when their expectations collide with its significantly different paradigm. Not only does Git stage things before commits but it doesn’t even think in terms of files, although, unless you’re paying attention, it may seem that it does.

Scaling to handle extremely large repos with years of history is something that Git handles very well (Mercurial probably does too), but some systems, like Darcs, have serious problems with.

In short, Distributed Version Control systems are totally bad-ass, and can really help with a lot of common development problems, but they’re still fairly new, and while the big ones are good, and reliable, they may not have all the polish and GUI widgets that you’re used to with your centralized system. My advice is to go get Git, unless the majority of your developers are on Windows, in which case Mercurial may be a better choice… for the moment.

Getting just the tip of a Git repo

Thu, 13 Mar 2008 00:00:00 +0000

Sometimes you just want to distribute the source code without its history, and that’s where git-archive comes in. git-archive will create an archive of the files at any point in the history and wrap them all up for you in a tar or zip (defaults to tar). You can even make an archive from a remote repo by using the

—remote=<repo>

option in the administrator has enabled it.

You’ll typically use git-archive like this:

git archive <tree-ish> > my_new_archive.tar

And that will create an archive of everything in your repo. If you just want an archive of some specific files you can simply pass in the path(s) to the file(s) you want in the archive after your tree-ish. If you wanted to create a zip file instead of a tar you’d simply pass it the “—format=zip” option.

Many people wonder if there is a way to check out just the tip of the repo so that they don’t have to download the entire revision history. This is called a “shallow” clone and is possible by passing the “--depth <some\_number\>” but it has a number of limitations: you can’t clone, or fetch from it and you can’t push from or into it, but it’s useful if you only want to look at, or near, the tip of a large project with a long revision history and would want to send in your fixes or features as patches.

Sharing a public Git repo over HTTP [flow chart]

Tue, 11 Mar 2008 00:00:00 +0000

There is also an SVG version of this flow, which is more readable (but poor IE folks will have issues). Notes: This is a simplest possible configuration. Be sure to check out the docs for git-remote to see how to, optionally, designate specific local or remote branches. Many of the initial commands could be performed locally and then just uploaded to the server. This particular sequence guarantees that all the connection pieces are in place and working correctly.

Much thanks to Tim Toolman for getting this info into two easy to follow posts: Sharing git repositories via OS X’s built-in web sharing and Setting up a new remote git repository.

However, this doesn’t feel elegant to me. If anyone can come up with a simpler / more elegant way to do this please add a comment.

Some thoughts about Git

Tue, 05 Feb 2008 00:00:00 +0000

Not too long ago I decided to start writing a book about distributed version control. I was originally going to focus on Mercurial (Hg) because it’s quite good and of the two leading systems it was the only one that ran on every OS (because it was written in Python). The fact that it could also run under Windows meant that I could help spread the word about distributed version control to more people, and it slightly increased the chance that I might actually make some money in the process.

There was just one problem: Linus was right.

“Hg is pretty good, but Git is better.” - Linus Torvalds

When I heard him utter those words in his talk to the folks at Google I took them with a grain of salt. Almost every project owner believes that their software is “better” than the competition, especially when they’re so similar, but it turns out that it wasn’t hubris on his part. When you sit down and compare the functionality Git simply lets you do more.

I had actually started writing the book about Hg but time and again I found myself confronted with something that was either easier to do in Git, or simply not possible in Hg. It happened enough that I decided that I didn’t want to keep writing about, and arguably helping, out a product that I not only wouldn’t choose to use myself but also felt was dramatically inferior from a usage standpoint.

And then I joined the Git mailing list. Linus mentioned that it had a high signal to noise ratio but… holy shit. I have been on a lot of mailing lists for open source projects over the years and I have never seen anything like this. Almost every e-mail is a patch, or a good discussion of a patch, or a good discussion of some new feature and how to go about implementing it. Currently, one of the topics they’re discussing is how to allow people to push to a Git repo based on GPG keys stored in, or accessible by, Git. What’s amazing is that it isn’t just one or two people going back and forth or someone trying to proclaim the “one true way” to do things. It’s exactly what you would expect if you took a bunch of smart geeks, stuck them in a room, and watched them all work together to solve some common problem. That alone is impressive, but when you combine that with the fact that it’s a pretty high traffic list and you’ve got a lot of really smart people pushing a tool forwards at an incredible pace.

Now, all is not sunshine and roses in Git land:

The documentation is actually pretty good, but when you want to do a specific task and don’t know the Git specific terminology for it it can be a little difficult to find what you need, and sometimes the commands take a dizzying array of options and could really do with more examples. That works out well for me, because it’ll be all the more reason for people to buy my book, but it’s obviously not great for attracting new users. I still think that, for most use cases, Darcs does a better job than anyone at making it easy to ferret out what command you need to accomplish a task.

Git is a lot like C. It’s incredibly flexible and powerful but it is also incredible flexible and powerful. If you want to shoot yourself in the foot Git will point you to the gun rack and show you how to load the bullets.

Under the covers Git is a collection of small C apps surrounded by shell scripts, Perl scripts, Python scripts…. whatever the author chose to get the job done. From a usability standpoint this is irrelevant. Calling “git foo” is going to work just as well regardless of what foo was written in and will have no impact on the user, but I’d personally prefer to hack on a project that was all written in one language.

It also means that the msysgit team has a lot of work ahead of them in trying make it install easily under Windows. If you’d like to see git working under Windows, and are familiar with coding for Windows please check these guys out and see if there’s anywhere you can help out. Even just taking on a little task here and there is something that would be greatly appreciated. Part of me wonders if it wouldn’t be better to just port everything in Git to C.

I think that if the Git book goes over well I may go back and retool it for Mercurial, but in the meantime I have to agree with Ted Tso:

“…the main reason why I’ve come out in favor if Git is that I see its potential as being greater than [that of] Hg, and… in the long run I think it has ‘more legs’ than Hg…”

Just to be totally up front. Ted did have some complaints about ease of use and documentation, but I think that since his post came out Git has made some really significant improvements in those areas.

Branching for atomic patches and cherry-picking

Mon, 10 Sep 2007 00:00:00 +0000

The best thing about Distributed Source Control Managers ( IMNSHO ) is how quick and easy it is to branch and merge. The problem is that most of us cut our teeth on centralized systems that couldn’t even hope to take advantage of cherry picking, which is, in short the ability to take a single patch out of the middle of a sequence of patches, or every patch but one from a sequence. Just imagine knowing that there was a bug introduced in a specific patch and being able to prune it from your repository but not any of the patches around it. Or, plucking one little feature out of a mass of others that should wait until the next release. You can, but if you don’t make the effort to keep your patches as atomic as possible you’ll find that that patch you want to remove or extract is dependant upon another one, or more, that you may not want to involve.

Which brings me to my hierarchy of branches. Now, I’m still working with this concept so if you can improve upon it please let me know. The idea is that most of the time you’re not adding one little atomic feature, you’re adding a number of features that can be conceptually clustered. So, you make a branch for the cluster, but do no actual development directly in it. Then you branch from there for each atomic feature (be sure to scroll down to the graph below the fold here).

While you are working on a feature feel free to commit as often as you want. Check in broken or unfinished code. Pollute that branch’s revision history with as much crap as you want. Do whatever it takes in there to make sure you can roll back to any point you want, and never loose work. Have a cron job check in any uncommitted changes to this micro-branch every night in case you forget to. It doesn’t matter because you’re not going to push the revision history back up to the branch for the feature cluster. Instead you diff from the branch point and patch the cluster branch and commit it. This way you’ve got one atomic revision with the combined changes for that one small feature and if you need to prune it for any reason (maybe it’s broken, or decided you should wait on adding it) you can easily do so without worry of dependencies. Your micro branch has all the details of the feature evolution if you really feel you need it, but most of the time you’ll just delete it after creating the patch and applying it to the cluster. Eventually the cluster will have all the related features completed and checked in and it’s revision history will consist of a series of atomic commits. The cluster’s revision history is then pushed upstream to your personal development branch. Occasionally you’ll have a small feature to work on with no related ones to cluster. Just branch from your personal development trunk and then patch back into it as you would have into a feature cluster.

Bugs should be treated in exactly the same way as features, although you’ll find yourself with many more small bugs that don’t need a parent cluster and can just be worked on in their branch and then used to patch your personal development branch.

People working farther upstream now have the option of cherry picking features, feature clusters, bugs, and bug clusters, and will be able to do so with much less worry of getting caught up with patches that depend on other patches.

Why you should be using a distributed source control system

Fri, 29 Dec 2006 00:00:00 +0000

I was reading some articles yesterday that finally made the light bulb go off about distributed source control management (scm) and why we should be using them. First off, a distributed scm, unlike CVS or Subversion, has no central repository that all others pull from. It’s possible to set one up and say that it’s the master and tell people to pull from and push to it but that’s more a matter of convention. What’s truly unique about these systems is that each checkout is it’s own self-contained ecosystem. And there are many reasons this is a good thing:

You can make as many changes as you want, check in unfinished code, and explore new functionality without ever affecting other developers. It only impacts other users when you sync your current working environment with others.
You don’t have to have net access, or be connected to any other box to get work done while still taking advantage of version control. Working on your laptop? Stuck in the boonies with just dial-up? Commit changes, roll-back to previous versions, make branches, none of it is dependent upon some central server. When you do get back online you can sync your changes with others.
“You don’t need to set up and manage a central SCM host with sufﬁcient disk space, compute power, bandwidth, and backup to support the concurrent SCM operations of your entire development community.” *

Real world uses:

The open source project:

If Mozilla has taught us anything it’s that you’re NOT going to get thousands of developers working on your project. So, while huge numbers of developers is something that distributed scm handle exceedingly well I think the point is moot. But, in my experience, a developer interested in some OSS (Open Source Software) project will download the source from a traditional scm, poke around to understand it, and if they’re smitten with the idea will start customizing it for their needs. But, the are forced to either work without the benefits of version control or they have to check it into their own personal scm. If they check it into their own you can pretty much forget about ever getting patches from them because they’re no longer able to sync with your tree and it would be way more work than they generally want to do to get synced and give you a patch that was useful. If they work without scm (because they don’t have commit rights to your scm) you may get a patch from them but for them it’s like climbing a rock face without lines and harnesses.

If you were to use a distributed scm each developer would be working off of their own personal copy of the scm that could be synced at any point in time no matter how many changes, revisions, or commits they have made. They’re generally not going to check it into their own scm system because that’s work that doesn’t get them any real benefit. They already have version control via their checkout and the fact that it’s a distributed system. End result, they don’t need to work without the safety and security of an scm and they will generally always be working on a system they can easily send you changes from when they’re ready.
Web developers with sites others depend on:

If you’re really working like a pro, you’re constantly branching and merging your code. You’ve got a live site branch. You’ve got a branch for each new functionality exploration. You’ve got a trunk branch that everyone merges into their other branches regularly. In the real world you probably have a live site branch and the trunk and cross your fingers that each developers functional exploration doesn’t screw over any other developer’s pokings. Or, in far too many cases, you just have the trunk. The problem comes (for those of us who hate dealing with the pain in the ass that is the current state of merging on most traditional SCMs and have minimal branches) when you decide you want to integrate one developers functional exploration into the live branch. How do you extract that code from the other code in the trunk (or any other common branch)?

If you were using a distributed scm it would go something like this: Everything starts with the live site repo (wherever you happen to keep that). Everyone’s local repo(s) started as a copy of that. It’s not uncommon to check out another copy to go explore some new feature set in. Maybe you sync it with the live. Maybe you trash it. It doesn’t matter no other repo was affected by your fiddling. You don’t have to make a new branch in some central repo that gets stored forever even if your fiddlings proved insignificant or were just abandoned. If everyone decides that the features you worked out in a particular repo are worth keeping they just sync with it. If you decide to wait to put it into a later release that’s fine too because there’s no need to untangle it from other work you, or others, have been doing. This is because to “branch” you just do a simple checkout from any other repo. When it comes to merging you benefit from the fact that any decent distributed scm is designed to work with syncing sweeping changes to entire code-bases not just individual files. They all handle this a bit differently of course, and some better than others, but the point is that while you’re almost always going to have conflicts you’re working with tools that have to be better equipped to avoid them.
Projects with reeeeally large code-bases or many many developers:

I can’t really speak to those because I just haven’t worked on them but utilizing distributed scms does mean that you don’t have the huge demands on the server hosting your scm. Plus all the aforementioned benefits.

So what are your options? Well the main contenders appear to be darcs (written in Haskell), Mercurial (written in Python), and Git (written in c plus hooks into a bunch of other things I think). darcs seems to be the easiest to use, with some decidedly funky features, but isn’t great for projects with huge numbers of files (it can be ram intensive at times). Mercurial is going to be used by the newly open-sourced JDK, and Git is being used to manage the Linux kernel.

* My light bulb moment, and some of the quotes here are thanks in no small part to this blog post about the JDK moving to Mercurial by Mark Reinhold. Thanks Mark.

Git on weblog.masukomi.org

Github Rubocop Workflow

Mirroring With Gitea

Overview

The Goal

The Process

The Learnings

Do (not) Self-Host your repos

Table of Contents

Why You Should Self-Host

What about GitLab and other Competitors?

Why You Shouldn’t Self-Host

So what’s a geek to do?

What am I going to do?

Keeping a great Changelog

The guts of a good Changelog Entry

Changes need to be grouped by type

Dates are important

To summarize

Want an easy tool to help manage this?

Related Posts:

Why you can't auto-generate your Changelog

A good Changelog entry has…

Changelogs from commits

Commits are at the wrong level of abstraction

Commits are messages from one developer on a team to another one

There’s no point

You can’t edit them

Tickets are at the right level of abstraction

It’s a matter of human judgement

Want an easy tool to help manage this?

Related Posts:

Finding the Github Pull Request for a topic branch

Rebasing A Pull Request on GitHub

The Question

The Answer

Alternate Solutions

Rebase Anyway

Make A Second Topic Branch

Git: pushing and pulling from multiple repos

Workflow

When you don’t have write privileges

Using Git Bisect to Crush Your Enemies

Using Git Bisect

Full Automation

Pro-Tip

git status-report

Git Rebase: why, and when, you'd use it.

Handling and Avoiding Conflicts in Git

Why you should use a distributed version control system

Getting just the tip of a Git repo

Sharing a public Git repo over HTTP [flow chart]

Some thoughts about Git

Branching for atomic patches and cherry-picking

Why you should be using a distributed source control system

Real world uses: