weblog.masukomi.org - programming

Amazon To Bookwyrm Browser Extension

2025-12-28T00:00:00-05:00

Table of Contents

Overview

I’ve created a Firefox extension to make it easy to populate BookWyrm’s “Create Book” form with information from Amazon.

This post describes the approach taken, and provides info for anyone who wishes to create a fork that can extract data from other web sites.

If you just want the extension, you can find it here the Firefox Addons site: https://addons.mozilla.org/amazon-to-bookwyrm

⚠️ Note: This post is being written before the extension has made its way through the approval process. If that link doesn’t work, you can either try tomorrow or - if you’re a geek - install it from the source code (see below).

Why this exists

BookWyrm is a wonderful federated book review platform, but adding a new book that doesn’t already exist in the database requires manually entering all the book’s metadata. This is tedious and time-consuming, especially when all that information already exists on Amazon.

Most of the books I read aren’t in BookWyrm’s database yet, and needing to manually look up and enter all the data for a book has prevented me from leaving a lot of reviews that I really wanted to make.

So, I created a browser extension to copy the data from Amazon.

What it does and why

It would be lovely to use Amazon’s books API to get nice structured data for books. However, that API requires AWS account credentials, and isn’t free. Running a service that used their API to populate a site like BookWyrm would probably violate their Terms of Service too. We shouldn’t expect to see this kind of functionality within BookWyrm itself, because it’s likely its creators would get in trouble with Amazon.

So, this plugin just reads the HTML of an Amazon product page you’re viewing, and checks known locations for the information you’ll need to populate the “Create Book” form on your BookWyrm instance.

It does not submit the form, because you really need to verify the data it extract. There are lots of edge cases that can lead to odd results. Series - for example - is the biggest offender. When looking at a book in a series, some authors/publishers will set up the book data such that it only shows the series name, some include which book in the series, or how many books with different formatting, some put it in the title. Using Brandon Sanderson’s “The Final Empire” as an example, here are all the ways - so far - that I’ve seen a book be recorded.

title: The Final Empire, series: Mistborn
title: The Final Empire, series: Book 1 of 7: Mistborn
title: The Final Empire, series: Part of: Mistborn (7 books)
title: The Final Empire: Mistborn Book 1, series: Book 1 of 7: Mistborn
title: The Final Empire (Mistborn Book 1), series: Book 1 of 7: Mistborn

And all sorts of other combinations of the above. Extracting the “correct” data is pretty hard when there’s no consistency in how that data appears in the source.

We don’t want to be filling your BookWyrm instance with bad data. So, please take the time to validate all of the data that’s been copied over before submitting, and obviously search for the book on your BookWyrm instance before jumping to the creation of a new one. Much better to use / improve an existing book that others have been reading / reviewing than to create a duplicate.

Why Only Amazon?

It doesn’t currently support other sites for one simple reason: tons of indie authors only publish their books on Amazon. This is largely because it’s a requirement if you want your book to be available via Kindle Unlimited. There are lots of reasons this is terrible for authors, and publishers, and readers, but that’s a separate discussion. What’s important for the purposes of this plugin is that regardless of how you feel about them, Amazon is the one place where you can consistently go and find the book you want to import to BookWyrm.

It currently only supports amazon.com and amazon.co.uk because most - or all - of the other amazon sites are in languages I can’t read and thus I can’t verify the extension will work on them.

Why only Firefox?

Because that’s what I use. Well, technically I use Waterfox because Mozilla seems determined to make Firefox awful. I might make a Chrome extension, but - for many reasons - I don’t like using Chrome, so I’m not strongly motivated to go through all the work. 😉

If you’re a motivated geek, I’ll happily accept a Pull Request that creates a “chrome” folder with a chrome extension that reuses the core code, or just fork it and change it to be just a Chrome extension. Have fun.

Geekery

The GitHub repo can be found here: https://github.com/masukomi/firefox_amazon_to_bookwyrm/

I’ve endeavored to put all the info you’ll need to get started with modifying this to work on other sites.

Please fork it, make changes, submit Pull Requests, all the good stuff.

The Joy Of No Batteries

2024-10-16T00:00:00-04:00

Table of Contents

Overview

A musing about my enjoyment of Scheme and having to work a little harder.

Unfamiliar with Scheme?

Side note for those who aren’t familiar with Scheme:

Scheme is an umbrella term for many languages. Most of them try and follow one of the official standards like R5RS, but some go off on their own and explore completely different paths like Racket. Schemes are considered dialects of Lisp, but Lisp too is a term for a collection of languages. I’ll use the terms somewhat interchangeably below.

Please also see the bit at the end before complaining about the parentheses to me.

Old Cycles & New Learnings

I’ve been coding professionally for just shy of 30 years now. All of the professional work has been web development.

At its core web development is taking form submissions, sticking them in a database, taking the stuff in the database and showing it on a web page to read, or a web page with a form to edit. That’s it. I don’t care if you’re using Rails, or Django, or Phoenix, or some homebrew creation. The back end is essentially the same. It’s not until you start approaching Google scale that what you do, or how the infrastructure supports that begins to change.

Most of us don’t deal with requests at that scale. Most of us are coding minor variations on the same thing again and again. I suspect that it’s the same for back-end developers in any vertical, be it mobile games, or radar systems. Even if you switch languages it’s the same stuff with slightly different syntax most of the time.

I suspect that’s why so many of us work on hobby projects, or open source libraries even though they’ll eat up our free time, and are unlikely to ever make us money. Those of us who love programming, love the problem solving, and figuring out how to express an idea with in a different set of constraints.

Batteries Not Included

A programming language that has “Batteries Included” is one that has a large / versitile “Standard Library”. Languages like Ruby could be said to come with not only “batteries”, but a spare generator, & solar panels. It’s got a web server, JSON parser, HTTP support, and lots more.

Scheme’s aren’t like that. Some Scheme’s - like Racket and Chicken Scheme - have a thriving ecosystem of third party libraries, but with the exception of the SRFIs^[1] those libraries are frequently written in ways that make it as difficult, or awkward to use them as to write your own implementation.

In the end, you end up needing to write a lot of your own “batteries”, and that means thinking about “simple” things that you’ve been using for years, but probably never thought about before.

Forced Thought

This is where Scheme comes in for me. Scheme is considered a multi-paridimatic language. It’s not explicitly functional, and there are various Object Orientation libraries that have been written for them. It’ll let you write code in a very OOP-like fashion, but everything seems to work a bit more smoothly if you embrace a more Lispy approach.

For most of us that means a different approach. For example, by default Scheme doesn’t have an method to break a loop or return early from a function, at least not like most languages^[2]. This isn’t a bug, or an oversight. You just don’t need them when you start thinking in a Lispy Way.

Lisp, and by extension Scheme, has a historical mystique to it. It’s a fundamental part of the history of machine learning and programming language creation.

I learned it because - at the time - I hadn’t done any functional programming and I wanted try something that would give me new challenges & force me to learn new ways to solve problems. Common Lisp^[3], & Scheme gave me that. Those learnings helped give me new solutions to problems in my day job too.

But I’ve been finding real joy - and frustration - in having to implement little things that we take for granted in other languages. For example, yesterday I needed to extract the first, or last N elements of an array. In Ruby I could do something simple like this.

long_array  = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
array_start = long_array[0,3]  # => [1, 2, 3]
array_end   = long_array[-3..] # => [8, 9, 10]

Scheme has a lot of list manipulation functions, but it’s also missing a lot of things we take for granted, so when I didn’t find a function to do this (see below), I figured I had to write my own. This is frustrating when you’re trying to get something done quickly, but it can also be joyful in that it makes you stop and think about simple things. “🤔 how do I efficiently select a subset of a list?”

My solution uses tail recursion, and Scheme’s ability to define a function within a function which lets me expose a wrapper function (first-n) to the world that doesn’t force the users to deal with passing in an accumulator, and doesn’t add a second top-level method to this module that future me will have to think about separately. It also doesn’t use any loop / iteration functionality. It’s radically different than how I’d solve the same problem in Ruby, but I think it’s also a pretty Lispy solution, that runs quickly.

  ; returns the first n elements of the list
  (define (first-n number a-list)
    (define (inner-first-n number inner-list accumulator)
      (if (or  (eq? (length accumulator) number)
               (null? inner-list))
          accumulator
          (inner-first-n number
                         (cdr inner-list)
                         (append accumulator (list (car inner-list))))))

    (if (> (length a-list) number)
        (inner-first-n number a-list '())
        a-list))

Scheme’s got a lot of oddly named functions, so occasionally you’ll not find a built-in one just because you don’t know what keyword to search for. In this case, I didn’t know the built-in methods for this behavior were called take and take-right, and I’d never heard anyone refer to that behavior as “taking”.

Even when I found those, I had to modify things, because those methods don’t behave well when you pass in a list that’s shorter than the number you’re trying to take, and Scheme doesn’t have built-in try/catch exception handling like most languages^[4]. There are libraries to do it of course, but the point is that even when there is a “battery” - built in or 3rd party - you frequently need to futz with it.

  ; returns the first n elements of the list
  ; or the entire list if its length is < number
  (define (first-n number a-list)
    (if (<= (length a-list) number)
        a-list
        (take a-list number)))

Thinking about this as I write it, this sounds terrible. Admittedly, sometimes it is damn annoying having to write so many wee low-level functions.

I’ve got a module that just a bunch of list manipulation functions to let me easily implement common tasks that aren’t built in, like sorting a list of strings. Every one of those represents a little bit of learning, a problem that I’d never have to solve in my day job.

Each one of those is also written in a way that matches my brain. That too is one of the arguments for, and against, Lisps. Codebases, it is argued, tend to end up looking like DSLs where everything is a function tailored to the specific purpose of the app. You can end up with a codebase that makes weirdly specific things incredibly easy, but also requires a longer spin-up time for new team members because so much of it is written using custom functions instead of generic “batteries” from the language.

Parting thoughts

When starting a new project, ask yourself if you want to get something written quickly, or if you want to learn new things. If you want to learn new things, maybe checkout a language like Scheme.

If you’re interested in learning a Scheme I’d highly recommend you check out Racket. It’s got a ton of 3rd party libraries so you don’t have to write everything from scratch, and the documentation of them tends to be very high quality. Chicken Scheme’s pretty good too, with a good ecosystem of 3rd party libraries^[5] but I find the documentation of most libraries to be simultaneously comprehensive, maddening, and nearly useless. For example, every function listed with a sentence or two describing what it does, but each description is based on a pile of domain knowledge that people new to the library, or new to the language don’t have.

Post Script About The Parentheses

Whinging about Lisp’s parenthesis is a lot like being fatphobic. It’s claiming something is “bad” or “lesser” because you don’t like how it looks without any real understanding of why that particular person / thing is the way it is.

The only people who whinge about Lisp’s parentheses are people who don’t code Lisp. Not only that, but Scheme is so damn flexible that there are multiple implementations that don’t use parentheses. Wisp (Whitespace to Lisp: SRFI-119), for example, uses indentation levels like Python. The Readable Lisp S-expressions Project uses parentheses, indentation, and curlies and resembles C.

So it’s not just “yucking someone’s yum” because you don’t like how it looks, it’s also really ignorant because you don’t even need to use them.

To me, Lisp’s parentheses are little hugs that keep my codey thoughts safe and secure. They help Lisp to be a homoiconic language, and every decent editor makes them easy to work with, and helpful. So stop trying to get rid of my hugs!

SRFI: Scheme Request For Implementation. These represent a couple hundred implementations of different useful things (like hash tables). They tend to be written in a generic way that can either be used directly in any Scheme that adheres to the de facto standard called the Revised Report on the Algorithmic Language Scheme (RnRS). E.g. R5RS. Or can be easily ported. ↩
Most schemes do have a call-with-current-continuation function (abbreviated as call/cc) that allows you to exit a function early. However, there’s a huge difference between just returning the result of your calculations directly (ex.: return "foo") and calling a function that takes a function that then gets executed in order to get your result out, but also behaves differently based on how and when it’s called. ↩
I started with Common Lisp, but parts of it frustrated me, and at the time the community seemed essentially dead, and the library options (number of 3rd party batteries) was almost non-existent. It’s gotten much better since the introduction of the Quicklisp library manager. ↩
Yes, of course there are libraries that implement modern try / catch style exception handling. They’re just not “batteries” that are included by default. ↩
Unfortunately Chicken Scheme uses a centralized Subversion repository for sharing “eggs” (libraries). The process around this is so fucking obnoxious, and requires so much obscure hoop jumping that I’m not bothering to update the one library that I did submit. ↩

Rewriting Hey

2022-12-18T00:00:00-05:00

Table of Contents

Overview

“Hey!” started as an Interruption Tracker, and now supports Time Tracking too. It has been through 3 iterations: Chicken Scheme, Crystal, and now Raku. This post is a high-level developer’s diary of what I wrote, why I rewrote it, and what I learned along the way.

In the Beginning

Once upon a time I was a tech lead, team lead, and eventually principal engineer. My days were filled with interruptions. They were making it impossible to get anything done, and more importantly to me, they were a source of a lot of stress.

I decided that if I was going to get a handle on the interruptions I needed to understand them. I needed to know when they were happening so that I could plan focus time. I needed to know who was interrupting me so that maybe I could talk to them about it. I needed to know why I was being interrupted so that I could find things I could document better to preempt those interruptions.

It started out as a little Chicken Scheme script that grew organically. It quickly became a hard-to-maintain hairball of code.

But it worked, and it was good.

Final Totals

~1300 lines of Scheme
~2500 lines of JavaScript (mostly library stuff) to support some fancy “graph it via the web” stuff that never really worked out well.
~1000 lines of HTML to hold the JavaScript graphing stuff.
GitHub Repo Here

Writing in Chicken Scheme

I love Scheme. It really works for my brain. Chicken Scheme is one of the better schemes. It’s got a large collection of useful libraries. It also transpiles to C (or JavaScript actually). The end result is a fast tiny binary. As a bonus, you can generate C files so that your libraries can be called from almost any other language.

As an example, my private_comments API web server runs in less than 7MB of RAM and I didn’t even try to make it small.

Unfortunately figuring anything out in Chicken Scheme felt like pulling teeth. There’s a ton of documentation, but it’s mostly useless. It assumes a ridiculous amount of knowledge and doesn’t explain anything. It feels like “Here’s the method signature. Have fun figuring out what it does.”

Getting anything done in Chicken Scheme feels like a major accomplishment. It’s also incredibly draining because it’s so damn hard to figure out if you’re new. It’s not the language’s fault. The language is trivially easy, like all Schemes. It’s the documentation.

These days I use Chicken Scheme very strategically. For example, my CLI heatmap generation tool is written in Chicken Scheme because I wanted something very fast. I also liked that folks could use the core library in any other language if they felt like dealing with their language’s C interface.

Delusions of Sharability

I should also note that I was working under a mistaken belief. I believed that it would be possible to generate a binary for an OS that I could just hand to someone without them having to know anything about Chicken Scheme, or Go, or whatever it was written in.

The truth is, you can’t. It’s theoretically possible. If you could fully statically link a binary, you’d have no worries. Unfortunately it turns out that that, practically speaking, that’s impossible on macOS. They really, really want you to use dylibs. Excluding apps with a “hello world” level of complexity, I’m not sure you can do it.

I spent a lot of time and energy trying to make this happen, but ultimately had to abandon the hope.

Crystal Lang

I’m a Ruby dev by trade, and Crystal is, essentially, just a compiled ruby with Static Typing. Or, it was at the time. These days it’s added concurrency and parallelism and is growing into its own thing.

I wanted to rewrite it in a nice, organized, maintainable way. At the time Crystal was still pre-1.0 volatile, and didn’t have a ton of users or projects. There were, and I think, still are, multiple ORMs (Object Relational Mappings) competing with no clear winner.

Unfortunately, they were very immature tools. So, I took one that I thought was good, and added in relations like “has many” and “belongs to”.

I thought. I planned. I organized. I Object Oriented the 💩 out of it. I wrote and used a sparkline library for Crystal. I wrote a CLI bar chart library for Crystal.

I added a CLI bar-chart graph of interruptions by hour so that I could easily see when I was being interrupted.

$ hey report interrupts_by_hour
Interrupts By Hour:

                                          ▁
                                          █
                                 ▁     ▁  █
                                 █     █  █
                                 █     █  █     ▁
                           ▁     █     █  █  ▁  █
                           █     █     █  █  █  █
                           █     █     █  █  █  █
                           █  ▁  █     █  █  █  █
                           █  █  █     █  █  █  █
                           █  █  █     █  █  █  █
                           █  █  █     █  █  █  █
                           █  █  █     █  █  █  █
▁  ▁  ▁  ▁  ▁  ▁  ▁  ▁  ▁  █  █  █  ▁  █  █  █  █  ▁  ▁  ▁
00 01 02 03 04 05 07 07 08 09 10 11 12 13 14 15 16 17 18 23

Honestly? It was pretty damn good.

But then, my life changed. I was completely burnt out from the constant interruptions. I was having random Panic Attacks over trivial things. My wife was burnt out too. We quit our jobs, cashed in the 401k and lived off of savings for the better part of a year. It set back our retirement, but gave us enough time to begin recovering.

I didn’t need Hey! anymore, so I wandered off.

Final Totals

~1600 lines of Crystal in the core tool (including tests)
~300 for the bar chart lib
~50 for the sparkline lib
~400 for the cli table generator
~150 lines for the custom arguments parser
~300 lines of additions to the DB library (mostly macros)
Total ~2800 lines of Crystal
GitHub Repo Here

Writing in Crystal

Crystal’s a great language. I have nothing bad to say about it. I happily supported them on Open Collective for a long time. If you want to see what I think is a pretty well designed Crystal app, go look at oho. It’s the world’s greatest tool for converting colored terminal output to beautiful HTML. I’m pretty proud of it.

These days though, Crystal isn’t my happy-place. I’ve spent too many years in Ruby. I want something different for my personal coding.

Fast-Forward…

Fast-forward from 2016 to December 2022. I’m unemployed again, like so many of us (Hire Me!), and while I’m actually getting a lot done, I don’t feel like I am. I’m being too hard on myself, and not giving myself the credit I deserve for accomplishing the things I have. I decide to start time tracking. I figured if I could look back and see how I actually spent the day I could have a better feeling for what really happened. “I didn’t get anything done today on my work, but that’s because I spent a ton of time on all these other things that needed doing.”

It turns out, I was right. In addition, the act of tracking my time makes me think about how I’m spending it, as I’m spending it.

I didn’t want to write a new time tracker. I just wanted to track my time. I knew there were multiple, good, off-the-shelf CLI time trackers.

I tried using 3 of them, but they all frustrated me for one reason or another. Some tools wanted me to interface with it in a way that felt awkward or annoying. Some had annoying technical things.

Watson came very close, but every time i needed to backdate a timer i had to type in explicit 4 digit 24 hour time. E.g. 08:30 not 8:30 and 18:47 not 6:47. It’s a little thing but I was doing it at least ten times a day and it annoyed me every time. Also, the ability to edit a timer was just broken.

Then I realized… I already have a time tracker that works with my 🧠. It just doesn’t support end times.

I checkout the Crystal repo. I compile. It goes 💥. It goes 💥 here. It goes 💥 there. It goes 💥 everywhere. The language has changed in notable ways which break my code, and break the code in libraries I was depending on.

In Crystal’s defense, it was 0.33.0 when I ported Hey! to it and they were very clear about the fact that things would be changing before they hit 1.0.

The compiler was very helpful in telling me what changes to make. Eventually though, I hit a case where it was complaining that I wasn’t implementing a method defined in an abstract class, and I definitely was. At that point I gave up. I just don’t have the emotional spoons to go fixing problems I didn’t create in a side-project that I didn’t even want to write in the first place. I don’t particularly want to be writing in Crystal these days anyway.

Maybe the Chicken Scheme version?

Chicken Scheme has moved from v4 to v5 in the intervening years, but Chicken’s upgrade is relatively painless. I’ve already done it for a couple of projects. It’s mostly just a matter of changing some import statements.

20 minutes later and I’ve got a working version of Hey! It’s a worse codebase, but it complies!

Hack. Hack. Hack. Bug. Bug. Bug. I’m fighting with Chicken Scheme again.

“Arrgh! I do not need this. This isn’t even a good codebase to begin with.”

So, I step back. I ask myself what the bare minimum feature set would be that would be good enough to use and not annoy me:

the ability to start and stop a timer
the ability to tag a timer
the ability to backdate a timer because I frequently forget to start or stop them.
no !@#$ 24hr time.
a log of time spent.

That’s not actually that much…

And then Raku

I create a new directory and start hacking. I’m still new to Raku, but overall I’m loving it.

I started by writing about ~250 lines of ugly-ass no-apologies code. I had one goal: get it working in the shortest possible time. 250 lines gets me a DB layer and the ability to start and stop timers (no backdating).

I try to run it. It goes 💥, but that’s to be expected. I have no tests & I’ve literally never compiled it. To my great surprise, the bugs are all little things like missing commas or semicolons. There’s nothing notable.

This is huge. If a Raku newb like me can write ~250 lines of code and not have any notable bugs, it means they have done a damn good job of putting together a language.

I’m chuffed. I keep hacking. Along the way, I’m noticing that lots of little nice-to-have’s are trivial to add while I’m there.

I don’t bother with an ORM. I don’t bother with any OOP (Object Oriented Programming). It’s just a collection of functions that query a SQLite DB with raw SQL and pass the resulting hashes around.

The ecosystem of libraries gets me nice things like time durations as strings “2 hours and 33 minutes” or “2h33m”, and pretty tables. The pretty tables are my fork of another library, but I’m not including that in the totals because I didn’t write it for this project.

I still don’t have the command line graphs implemented, and it looks like I’ll have to port that from Crystal. There isn’t a Sparkline library yet either, so add another ~300 lines of code before I’m done.

Final Totals

~800 lines of Raku
~100 lines of bash_unit tests
~300 lines of future code
Silly levels of satisfaction
GitHub Repo Here

Writing in Raku

Notable things: I wrote precisely zero lines of code to process the command line arguments in a natural language fashion. I wrote zero lines of code to generate or wire up the CLI help doc shown below. Raku gave it to me “out of the box”.

It took me about 5 lines to include a starter db with the package, check if you need it, and put it in place. That’s a combination of the language having a concept of resources in a bundle, a library that understands XDG Base Directory stuff, and my XDG::GuaranteeResources library that combines the two.

Raku’s still got a very small community, but they are “knocking it out of the park” with useful libraries. And the language has been incredibly well thought out. I have complaints, but they’re all nitpicks.

Maybe it’s just that I started my professional programming career with Perl5. Whatever the reason Raku really works for my 🧠.

Usage:
  hey start [<start_args> ...] -- Start a new timer
  hey stop [<stop_args> ...] -- stop an existing timer
  hey log <number> <duration> -- see a log of recent timers
  hey log-interrupts <number> <duration> -- see a log of recent interruptions
  hey running -- lets you know if there are any timers running & what they are for
  hey <name> [<start_args> ...] -- Record an interruption
  hey kill timer <id> -- Remove an unwanted timer.
  hey kill <name> -- Remove an unwanted person / thing from interruptions

    [<start_args> ...]    optional time adjustment, project(s), & optional tags
    [<stop_args> ...]     optional id, and optional time adjustments (e.g. 4 minutes ago)
    <number>              number of duration units
    <duration>            duration string. E.g. minutes, hours, days, etc.
    <name>                name of person / thing that interrupted you
    <id>                  the id of the timer to delete.

Reflecting on the rewrites

Looking back at all this work. I find myself coming to an interesting conclusion. With regards to code written specifically for you:

I believe that choosing subjectively good technology is more important than objectively good technology.

You have limited time to work on projects for you. Don’t spend them in languages you don’t love. Write them with tools, and languages you do love. Write them with tools that are exciting you. Rewrite them when you’ve moved on and you want to use them, but the idea of maintaining the old stuff is just draining.

99% of the tools you write for you will never be looked at by anyone else. Even less of it will be used by someone else. Don’t worry about what’s “good”. Do what makes you feel good.

Chicken Scheme made me feel good. Sometimes it still does. Crystal allowed my to do good things quickly, but it was never Joyful to me. I’m long past the days of Rubyish syntax being Joyful. I like it. It’s good, but I’ve written many many thousands of lines of it and the wonder and joy of it has worn off.

[UPDATE] Since writing this I’ve thought a lot about the choices we make when coding specifically for ourselves vs coding for others. The result is the Duct Tape & Baling Wire Methedology.

Was it worth rewriting all those times?

100%

The first rewrite (Crystal) was in a language I was enjoying, but more importantly it was desperately needed. The original codebase was spaghetti. It had grown without planning, as so many tiny-but-useful tools do.

The second rewrite (Raku) was because I didn’t have the emotional spoons to deal. Fixing problems someone else created, in a tool i didn’t want to be writing, in a language I didn’t want to use… ugh. I didn’t need that.

However, this rewrite also ended up reinforcing my thoughts about how awesome Raku is, and teaching me some new tricks.

If you liked that…

If you found that interesting, you’re probably the kind of geek who writes a bunch of useful command line tools. If so, you should check out TooLoo. That’s my Open Source tool that helps you document your tools. Find them later. Use them more.

P.S. What about the email app with the same name?

Hey! was created before the email service. I figure this project is never going to become popular enough that I’ll have to worry about brand confusion. ;)

I used AI to generate products & write copy for my store

2022-12-06T00:00:00+00:00

Table of Contents

Quick Summary

I used Midjourney to generate art that I threw on t-shirts, coasters, and almost everything else RedBubble offers. Then I used OpenAI to generate copy for it, and combined that with a handful of custom scripts to generate a product site called Bed Bath & The Beyond.

It took about 40 hours of work to put all that together for 13 products, but that will be faster next time. In the end, this is a surprisingly viable option, and the only real restriction on profit is how much effort you’re willing to put into marketing what you make.

It started as a joke…

A friend threw this out on Mastodon.

screenshot of mastodon posts

For a giggle, I asked Midjourney for “UNSPEAKABLE HORRORS AT UNSPEAKABLE PRICES. ALSO, BATH MATS.” and this came out

And they looked “real” enough that I wondered if I could actually make a “cool” bath-mat.

lovecraft & tentacles bath mat

A little editing, and I’d extracted it from the “bath mat”, and thrown it on Redbubble.

the same image on a floor pillow

That’s when things started to spiral out of control.

The work Begins…

Generating images

I spent that night working through images on Midjourney. It was hours of work, but as I was being pretty lenient with Midjourney I was able to get usable results pretty quickly. At the bottom of this post is a screenshot of the stream of images I generated. I’d put it here, but it’s really long. ;)

Prepping images

Then came the editing. By default an upscaled image from midjourney with a 3:2 aspect ratio is only 1536 x 1024 which is fine on your screen, but not so hot for printing.

Fortunately, I’ve actually got a background in print and fabrics are not a high-resolution, or high-fidelity medium. So, upscaling artifacts shouldn’t be a big deal on most things. I’m a bit unsure about the comforters and shower curtains though. Because of this, the acceptable resolution is much lower. Also, RedBubble has this handy page that suggests resolutions for each of their products.

Combine that with Pixelmator Pro’s ML Super Resolution and the upscaling went pretty well.

Most of the images required very little editing.

RedBubble

At this point I’ve sunk at least 10 hours into the process. I’ve made two or three product pages on RedBubble for them. Creating the RedBubble product pages was … work. There’s just no getting around it. It’s a time sink.

In their defense, it requires a lot of subjective human decision making. They’ve also done a ton of work to help folks to promote things. For example, when I click on “Download Images” on the Desk Mat for Billy and Gog’s Grand Adventures I get this excellent collection of promo images to use to promote it.

screenshot of desk mats

Upload, choose, set default colors, save, see what you missed, edit the page to tweak default fabric colors again, repeat.

The site

Hugo is my default static site generator. It’s fast, it’s easy, and I’m familiar with it. So I start making pages, and it… is… taking… forever.

The process was

grab the product url
make a markdown link
choose 2 or 3 items to highlight
download images of the image on those products
make a markdown link to each of those product pages
embed the image in the page with a markdown link to the product page around it.
come up with some witty title and ridiculous copy for the page

I punted on the last one. The process doesn’t sound like much when you say it, but in practice it was hugely time consuming, mentally draining, boring, and soul sucking.

Know Thyself

I knew that if I forced my way through, I could finish the 13 or so products I had. I also knew that I’d never add anything else to the site if I did. I’d definitely not make another site if some other entertaining idea came along.

So, I wrote some code. We’re talking ugly, get it done, do the first thing that’ll work kind a code.

Along the way I discover that for some reason Firefox doesn’t bother to store session cookies in its SQLite cookie database, and it turns out I need session cookies. So, right click, inspect the network call to their GraphQL API (totally undocumented) and “Copy as curl”.

Then write a bash script to read in the clipboard, extract the cookies and throw them into some ancient Netscape Navigator format that curl apparently knows how to handle. This file doesn’t need to be regenerated for …. days? I dunno.

Then copy the url for the product promotion page (the one with links to downloadable images) and run that through another script that extracts the product id, gives it to curl, which utilizes the new cookie file, and hits their GraphQL endpoint to extract the JSON (same stuff as on the web page). It’s more JSON than I need but refining the query would have been wasted time.

Run that JSON through a third script (this one in Raku) that presents me with a list like this. Now I just need to scan the RedBubble promo page to find images I want to use, and type in their numbers.

...
[33] Shower Curtain
[34] Spiral Notebook
[35] Sticker
[36] Tall Mug
[37] Tapestry
[38] Throw Blanket
[39] Throw Pillow
[40] Transparent Sticker
[41] Travel Coffee Mug
[42] Wood Mounted Print
[43] Zipper Pouch
[44] iPad Skin
[45] iPad Snap Case
[46] iPhone Skin
[47] iPhone Snap Case
[48] iPhone Soft Case
[49] iPhone Tough Case
[50] iPhone Wallet
Choose some items (comma separated):

The script considers my choices, downloads the default product image, runs it through imagemagic for resizing, takes my original artwork, resizes that, asks me for the name of the product and ….. voilà I’ve got a markdown file with all the soul-sucking grunt work done for me.

And yes, I have a way to add new products without overwriting the page.

That Sweet Marketing Copy

I never lost sight of the fact that this was, ultimately, a joke. I expect to, over time, make tens of dollars from this. If we consider my hourly rate it’s very unlikely this thing will ever pay for the labor that went into it.

So, because I’m not going to get rich on this, and I’m probably going to get a Cease and Desist letter from a Megacorp who likely lacks a sense of humor and thus shan’t be named. I’m not worried about “real” copy.

However. We’re a day and a half of work into this thing and frankly my ability to write entertaining micro-fiction about each of these is… shall we say, “diminished”.

But along the way I’ve been keeping my friend’s discord ( the one responsible for this stupidity ) in the loop. There someone points to an article about using AI for marketing copy, and I remember that I’d seen a bunch of hype about OpenAI on Mastodon lately. I don’t really want great “marketing copy”. I want creative stories, that happen to also be marketing copy.

So, I make an account there, and see what comes out. I started by creating some copy for this image, which I imagined having come from some old children’s book.

Elouise offers flowers to a tentacle montster

Folks… Text AIs are good.

OpenAI text output screenshot

In the end I edited the Elouise Makes A Friend text down to this:

This image from Elouise Makes a Friend is a timeless reminder of friendship and kindness. It shows Elouise bravely offering a flower to the Tentacle Monster, who in turn offers a kind smile. Whether you choose a the sleeveless top, the throw pillow, or a coffee mug, this image will comfort you and remind you of the power of friendship.

Bring this touching moment from Elouise Makes a Friend into your home. Our products featuring this image are perfect for gifting to that special someone, or for yourself.

Everything required heavy editing, but that didn’t matter. It’s radically easier to edit two to three paragraphs of copy than to come up with some ridiculous story that you then have to edit anyway.

Over time, I got better and guiding the AI, and increasingly impressed with what it would come up with. For example, I let it create almost everything on the About page.

Final Thoughts

Knowing then what I know now, would I do it again? 100% It was fun. I’m ridiculously proud of it, and I genuinely want many of the products.

Would I recommend it to others?

Yes, but with caveats. You really need tooling to get from “I threw something on RedBubble” to “I have a site for my products”. Otherwise your brain will melt. I guess you could outsource that work, but the profit margin is so small here that that’d just put you farther in the hole.

There are some existing tools on GitHub but a quick glance suggested they might be abandoned, and frankly it was faster to write ~100 lines of Raku than figure out and tweak someone else’s stuff.

Also, I really need to find a prettier template. I’d hire someone to make one but there’s no way this thing is going to cover the cost of my time to make it, and I’m not currently employed (hire me!) so that’s not going to happen.

Will I add to the site? Almost guaranteed. You can Follow Bed Bath & The Beyond on Mastodon to see the latest items as I add them.

Maybe? It’s kinda crap, and could use a lot of refinement, and frankly not many folks have realized the joys of Raku yet so it’s probably not in a language you want to poke anyways.

What’s my favorite?

Lovecraft & The Octopus. Something about this just tickles my funny-bone. Also it looks unreasonably good on everything I put it on. I can not explain it. Like, how can you not want this ridiculous clock?

a clock with an image of lovecraft and a giant octopus thing behind him

So Many Images

Working With AI Image generators is like herding cats. This is, honestly, a surprisingly small number of images to get 13 usable products. See A Journey With Midjourney if you’re curious about the process.

Some of these are good, but I haven’t gotten around to incorporating in the site yet. A handful of these are pixelated, because I’ve got another blog post idea involving them… maybe more t-shirts. ;)

Holler if there’s anything there that isn’t on the site, that you’d actually buy. I’ll add a page for it. I’m @masukomi@connectified.com

midjourney image stream screenshot

Mirroring With Gitea

2022-09-26T00:00:00+00:00

Table of Contents

Overview

Following on the heels of my last post on why you should (not) self host your git repos, I went ahead and used Gitea to set up a local mirror of all my repositories, and all the repositories I don’t want to loose access to.

The results were surprising, and after reading this, you might want to do the same. This post will be a qick overview of how I did it, some tips that’ll help, and what I learned as a result.

Installation was pretty straightforward. I grabbed my old Raspberry Pi 3B and followed this spectacualar step-by-step guide for installation for installing Gitea. Afterwards a friend pointed out that I could have just installed installed it on our Synology with a Docker image.

The Goal

My goal was two-fold:

I wanted a backup of all my repos. Many of them only existed in GitHub.
I wanted to delete every fork I’d made just to guarantee I had a copy.

The Process

With that in mind I started working through my list of repositories^[1]. Gitea makes it trivial to mirror a repo. You just choose “new migration” from under the “+” menu, and then click on the service you’re migrating from. There are 3 things that can make your life easier here if you’re importing lots^[2].

Bookmark the form that comes up after choosing a service to migrate from.
If you’re going to be importing a lot you’ll probably want to use a personal access token (GitHub docs). That way you won’t get rate limited by the API, unless you’re migrating (not mirroring) something with many thousands of issues.
Use a clipboard manager. It doesn’t remember your API key so you’ll be pasting it into every form, along with the URL of the repo you’re importing. I use Maccy, and I would recommend it.

For each repo in my list of repos on github, I’d make a couple quick decisions.

for things I authored
- If it was something I am either maintaining or may poke in the future I’d just grab the url and tell Gitea to mirror it.
- If it was something I wandered off from years ago, and am unlikely to ever touch again, I’d mark it as archived.
for the backups of other people’s things
- I’d click through and see if i’d added anything, or if it was just an old copy.
  - If I’d added something, and the original repo hadn’t been touched, I’d mirror my fork.
  - If I’d added something, but the original had moved forwards some significant amount I just abandoned my changes, and mirrored theirs.
  - If mine was just an ancient copy of a repo that’d moved on, I’d go to the parent repo.
    - If the parent repo was archived I’d just tell Gitea to import it but not bother mirroring. There aren’t going to be updates.
    - If the parent repo indicated that it too was a mirror, I’d follow that back to the source repo and mirror that instead.
    - After successfully mirroring I’d delete my copy.

The Learnings

(In no particular order)

First off, Gitea is really impressive. Those devs have done a great job and should be proud of what they’ve built.

Along the way, I discovered that many of the repos that I’d forked had moved on significantly. I had copies that were so old they probably wouldn’t run with modern versions of things. They were also frequently missing lots of new features and bug fixes.

While this isn’t terribly surprising if you think about it, the notable learning is that that’s never going to be a problem again, because Gitea is mirroring them and regularly polling them for updates. If they disappear I’ll have the latest copy before they went away. Github doesn’t give me that option.

I’m not the only one forking repos just to have a copy. Multiple times I clicked through to the parent repo, and got curious. I’d see hundreds of forks, click into the insights page and see that essentially no-one was hacking on it except for the original authors. I guess lots of folks have gotten burned when they discovered that some repo they’d starred or bookmarked was gone when they went to check it out again.

Mirroring isn’t something you can turn on later in Gitea. You must set a repo up as a mirror from the beginning.

Gitea can’t mirror everything. Some repos (like this great command line time zone tool) always error out. Gitea provides no information as to why it failed to clone them down too. I think I encountered three repos like that. Also, I can’t figure out how to clone anything from Sourcehut^[3]. Even when I tried to clone repos manually their http links for cloning always error.

I realized that if i don’t just mirror the tools that I author, and want backups of, but also mirror the repos of tools that I need to reference the source code of regularly then I have a single place I can search for only stuff that’s relevant to me. For example, it’s not uncommon for me to know that one of the libraries I use frequently has a piece of functionality, but not remember which library. Searching Google or Github would be frustrating, because there are just too many responses. But now, I can just search my Gitea install.

The experience of doing this was very cathartic for me. I now know that I have up-to-date copies of all the useful tools I care about. I was also able to delete about 100 repositories from my list of repositories on Github. Now, if a future employer checks them out, they’re almost all things I have actively worked on to some degree. It’s also nice to know that from now on I have a trivially easy way to mirror any useful tool I care about. No more ancient frozen it time forks, and no more bookmarks & stars that cease to exist.

My next step? Going through all the repos I’ve bookmarked, but not forked, and see which of those are worth mirroring.

Before working through your list of repositories, be sure to sort them by name. Mine were sorted by recency and that made things a little harder towards the end. ↩
For a sense of scale I ended up mirroring or migrating 169 repos into Gitea. ↩
In addition to having a visual style from the 1990’s, and having UX from hell, Sourcehut appears to also just not fucking work as a git host. I don’t understand why anyone uses them, nevermind why anyone pays for them. ↩

Do (not) Self-Host your repos

2022-09-24T00:00:00-05:00

Once upon a time, GitHub was a successful geek enterprise. Then Microsoft bought it, and folks started arguing that you should abandon ship. You should self-host your repos they say.

I 100% agree, and 100% disagree. Let me explain.

GitHub’s been a benevolent host. When they bought NPM they went from being a de facto piece of internet infrastructure to an actual piece of critical infrastructure. At this point you may as well argue that we should sever our connection to the electric company.

But there’s a difference between needing the electric company and ignoring the fact that sometimes the power goes. Usually it’s temporary. Days isn’t uncommon in wintery climes. But every now and then, it’s more. In the case of internet SAAS apps, it’s sometimes permanent.

Why You Should Self-Host

The biggest argument, of course, is backups and continuity of service. If something hapened and you lost access to your GitHub account right now, what would you loose? If you maintain a project with many users and contributors, what would it do to your community, and your work?

GitHub isn’t just where your Git repos are hosted, but where all the metadata that surrounds them. This means years of Issues, Pull Requests, wiki pages, and frequently the project’s web site.

I’ve derived a ton of value from old tickets on GitHub projects. They regularly answer questions, and provid workarounds. The takedown off a popular repo could be a huge loss.

GitHub suspends accounts, and deletes repositories for a variety of reasons. If yours targeted for seemingly “no reason” it wouldn’t be the first.

Even if you believe it won’t, or can’t, happen to you, there’s another reason reason to consider self-hosting: Lock in. Right now (September 2022) GitHub is very good about letting you export everything. As far as I know, there’s no data you can put into GitHub that you can’t also get out via their APIs. In theory.

Our migration has been in the process for a while, but every time we run it we find that our attempts were missing some data, so we need to re-run to fetch missing data, if we had higher limits it wouldn’t take so long to find these issues. Github doesn’t provide tokens with higher limits, so we have to work around that. We even had people with [GitHub Enterprise] subscriptions ask, and they got the same response.

Gitea on Mastodon

Gitea is an open source GitHub alternative, whose creation started out on GitHub. Eventually, the software got really robust, and they decided to host their own repo and issues. The problem is, they’ve got over 20,000 Issues & PRs they want to export. Thousands of those are still open, and the longer it takes them to cut over, the larger those numbers get.

To be fair, 99% of GitHub users wouldn’t have this problem. We don’t have projects that are popular enough, but companies who use GitHub do, and there’s no advantage to GitHub to help large projects or customers leave. They may even consider locking that metadata down to guarantee they stay.

What about GitLab and other Competitors?

You can apply the same reasoning to GitHub’s hosted alternatives. Plus, none of them are as financially successful as GitHub. They’re far more likely to go out of business, and screw everyone who uses them.

Why You Shouldn’t Self-Host

Right now it’s incredibly improbable that you’ll have your account suspended. The stories of suspended accounts are rare. Most of them seem to be from government action,or “Terms of Service” violations that most geeks react to with “Yup. Seems fair.”.

There’s also the fact that self-hosting is work. But let’s set that aside for the moment. Let’s pretend there’s some magical free button you can click that takes care of everything. Boop. You’re self-hosted now.

Let’s also assume you’ve got a project that’s popular / useful enough to get Issues and the occasional Pull Request.

How is that going to happen on your new site? Well, first they’re going to need to set up an account on your server. Are they going to think it’s worth it? Worth it enough to give their email to yet another stranger? If they want to submit a fix / feature, is it also worth it to figure out how to fork it in this not-GitHub? Worth it to figure out how its PR equivalents work? These probably aren’t hard, but every little difference they have to navigate is another reason to not bother.

And then there’s discovery. GitHub isn’t only the default place to look for code. They’ve also got massive “website authority” in Google. It doesn’t matter if you’re the only place hosting your thing. Any project on GitHub that’s remotely similar to your search terms is likely to come up before your site.

But, even if you’re ok with all of that. There’s an even bigger reason you shouldn’t self host. A reason we shouldn’t encourage folks to self host. Sooner or later you’ll get sick of bothering, or you’ll die. One is guaranteed. The other has lots of historical precedence.

I’ve bookmarked thousands of useful web sites over the years. A huge chunk of those have been useful, and insightful blog posts about development. Hundreds of them don’t exist anymore. Mostly because the person who ran them simply didn’t feel like bothering with it anymore. How many millions of others have poofed out of existence?

But, a repo isn’t some just random thoughts. Almost every repo contains some useful tool or library. Sooner or later, someone’s going to need tha. There are thousands, probably millions, of little repos out there containing the only bit of code in existence, that solves that particular problem. Sometimes they’re just the only thing that solves it in that particular language. When you stop hosting it, for whatever reason, that useful tool ceases to exist. Even if it’s “just a port”. That’s a lot of work that someone will need to recreate, and debug.

So what’s a geek to do?

Before I answer that, let me note that this bit gets into implementation strategies. If you just came for the question of “should I, or shouldn’t I?” you can stop reading. Thanks for making it this far. Good luck whatever you choose.

–

So, what to do…

Short short version: If lots of people rely on your project, then mirror your code to a large competitor and accept that you may loose access to your issues, PRs, etc. . Stick a note in your README so that folks know where to find your mirror before lightning strikes.

If you’re like most open source developers, you’ve got some repos that are mostly ignored by the world, and receive essentially no issues or PRs. No-one’s going to be stressed if your repo disappears for a week. So, Install Gitea on your computer. Tell it to mirror your repos and forget about it. Or, just clone the all locally and set up a cron job to regularly pull down changes.

Longer version: I think that depends a lot on what, if any, metadata exists in your GitHub repos. Most devs have small repos that are mostly just used by them. If this sounds like you, then simply keeping an up-to-date copy of them on your home computer could be fine.

Things are different if you maintain some repo that other people actually care about enough to make issues, or even contribute code to.

In that case my personal advice is to host your stuff on Github, but have it mirrored to a second system. Ideally you’d want all your issues, PRs, etc. mirrored to it. But as far as I know, that’s not actually something you can do. It’s easy to keep the code synced to a mirror. It’s the metadata that’s a problem. To be clear, I’m not saying it’s not technically possible. I’m saying, I don’t think anyone’s actually written the code to keep those synced between GitHub and any open source alternative. It’s a hard problem because Issues, PRs, comments, and just about everything else can be edited.

So, that leaves you with regularly backing up your stuff. I’m going to assume you’re not someone with ~10k open & closed issues, plus another ~10k PRs like Gitea. So, exporting your issues is probably something that’s actually doable. Exporting. Not syncing. However, the more things you have to export, and the more frequently you want to back it up, the more likely you are to hit your rate limits.

If you go this route, you’ve got two options. You could use something like the gitea-github-migrator that can export everything. Tools like that are made to create a new repo in GitHub alternatives, not sync to them. You’d be constantly churning, deleting old repos and setting up new ones.

A better solution might be something like github-issues-mirror which can download all your issues as JSON files. Keep that running in the background and if worst comes to worst you can write an importer for wherever you’ve been mirroring your code.

What am I going to do?

I think I’m going to install Gitea on my Raspberry Pi. I’ll tell it to mirror all my repos, and all the repos of useful tools I want to always have a copy of. I do maintain some repos which are somewhat useful to other people if GitHub deletes them. No-one will be screwed, and I can take my time in uploading a backup somewhere else. I’ll stick it in the basement with the router and back it up to our NAS, which backs up to B2.

gh-url script to get GitHub url for a file

2022-08-15T00:00:00+00:00

The Problem

For the past decade or so, I’ve noticed a trend amongst my coworkers. When they need to look at the contents of a file that they’re not currently editing, they will go to GitHub, and click their way down through the folder structure until they eventually find the file they want to see the contents of.

On a related note, I believe that most of my coworkers don’t know how to take a relative path in a repo, and tell their text editors to open it.

For example, let’s assume we’re working in the same repo, and you’ve already got a file open in it in your editor. If I were to tell you to open foo/bar/baz.rb would you know how to tell your editor to open that file directly, without manually clicking to open the foo and bar folders and then clicking on the baz.rb file? For most developers the answer seems to be “no”, even though their editors can probably do it easily.

So, if I need to communicate the file I want you to look at, how do I do so in a way that makes your life as easy as possible, assuming you’re like most devs?

Easy, I give you a link to the file in GitHub. But how do i easily acquire that link?

The Solution

gh-url is a script I wrote to take a given path within a repo and generate a GitHub url for it, and optionally, open it in your browser.

So, for example i could say take that foo/bar/baz.rb path and say gh-url foo/bar/baz.rb and it would output a link to that file in GitHub. It takes care of making sure it links to the right branch, and if you have multiple remotes, it asks you which one. Just run it without any arguments to see the usage instructions.

Quality Is Rarely Job 1

2019-03-03T15:13:58-05:00

In 1981 Robert Cox came up with a slogan for Ford; “Quality is Job 1”.

It has always stuck with me.

In the software industry there are few slogans could be further from the truth. C-level’s and other customer facing types frequently proclaim the “quality” of their products, but they aren’t the ones making the product. They’re frequently not even the ones using the product.

In software there are two viable ways to release quality software. You can release it when everything you want done is done, or you can release what you happen to have done at a specific date. You can’t combine the two, although almost every software company tries to.

People outside of engineering tell clients X will be ready on Y date. That date is inevitably too soon for the promised features to be completed.

Realistically, there is only one way to combine promised dates and promised features in software. You compromise the quality. People could, promise a date generously far in the future, but that’s not very helpful meeting to this quarter’s sales goals.

In order to deliver quality an engineer needs the support of management to not deliver code that isn’t done. “Done” of course doesn’t mean “It seems to work.” or “QA didn’t complain.” Done means “I have tests to prove it works.” Done means “QA can prove it works when users get ahold of it.” Done means “It’s intuitive for users.” For multi-user systems Done means “We can prove it works when tons of users try to use it at the same time.”

Done might not mean “It does everything that was requested.” There are always more features to add. It does mean that everyone, even the client, agrees that the features that are Done, are “enough” for now.

I believe that there are a few companies out there that actually deliver “quality” software that even their engineers believe in. I’ve never worked for someone else’s company that did this. Features sure, but overall product quality? No.

Bonsai Coding

2013-12-06T00:00:00+00:00

Writing code is a lot like maintaining a Bonsai Tree. If you stop pruning it it’ll stop being a Bonsai and turn into a bush. Little tweaks, frequently aesthetic ones, will help to keep it beautiful and under control. It will still grow in unexpected directions, as other developers make changes, but careful pruning will keep it balanced, and healthy.

What is “careful pruning” then?

Each file is a branch on our tree. The methods, are leaves. We come in to work and start examining one of the branches. Some days we need to encourage it to grow a specific way by adding a feature. Some days we make little snips to correct a bug. But what happens if, upon examining your branch for other reasons, you discover that it’s grown into spirals and knots. You can’t reach your clippers in to snip the leaves you need. You can’t do much of anything without difficulty and frustration, and frankly, it looks like crap. The unexpected spirals and knots need to go.

This, however, is the point where you’ll hear other developers saying “No! Don’t do that!” They’re upset, because your task wasn’t related to the knots and spirals. You were just there to trim some leaves. If you go and get rid of the “other” problems then how will they be able to tell where you trimmed?

To abandon the metaphor for a minute, they’re concerned that the diffs will be so filled with aesthetic changes that they can’t see the changes that were relevant to the bug fix, or feature add. And, they have a point. Being able to look back and understand the changes another developer made is a very valuable tool. But some developers get so obsessed with this idea, or so beaten down by others who are obsessed with it that they say things like this,

“The rule for controlled development is that if the line is not relevant to the fix, you do not modify it. If we want to go through and fix all the lines in our product that are over 80 characters, then we log a bug for it.” - A rule you shouldn’t follow

This solution will never leave us with a beautiful tree. I don’t know of any professional developer who has the time to deal with tickets that are that low priority, and no-one wants to anyway.

If you accept that there is value in making the line / file readable, then you must also accept that there will be change unrelated to any bug fix or feature request. You must also accept, that if those changes get put off for “later” our tree will become a bush, and require so much work that it will be almost unrecognizable the day after the change.

So, how do we balance these two concerns? How do we make readability changes to our code without obscuring the functional changes? My recommendations are as follows.

It should be acceptable for whatever method you’re directly trying to enhance or fix to be refactored for readability. Even if you’re refactoring a 20 line method, for a one line fix. To do otherwise is like having a child make their bed but leave their clothes and toys strewn about the floor. Don’t do a half-assed job. Leave the method better than you found it.
Separate other aesthetic changes out into separate commits. It would be nicer if you could commit the fix, then commit the aesthetic changes, but we frequently need to make the aesthetic changes in order to make the code readable enough to understand it. So, make it readable, but leave it broken. Commit that change. Then fix it and commit that change.

The downside to 2 is that you have to instruct reviewers to review specific commits, rather than the changeset as a whole. So, an alternative is to make the aesthetic changes, create a pull request / request review of your branch to be merged, and then, once that’s done create another pull request for the bug fix. I think it’s a judgement call, and the appropriate choice depends on the situation.

As per usual, if you’ve decent test coverage everything becomes much easier. Code reviewing the aesthetic commits can be just a matter of seeing if the tests pass, and giving it a quick glance to make sure the other developer didn’t go all crazy-pants with their changes, like changing the indentation style to something no-other files use. You don’t have to stop and consider the funcional impact of the changes, because there shouldn’t be any. “Yup, still compiles, still passes the tests, looks more readable. Approved!”

Summary Codebases are like Bonsai Trees. Regular aesthetic changes are as

important a part of keeping your codebase healthy as regular refactorings. Some trivial changes to how you commit your work can elimanate the diffing problems that arise when combining fixes / features with aesthetic changes.

Disovery coding through tests

2007-12-08T00:00:00+00:00

Testing as a process of discovery

The other day a coworker said,

Some times you get situations where the specification for the unit or module you are writing just are not available. The code writing is a discovery process as much as anything else. Moreover, some of the packages and methods being called don’t have predictable or documented behavior. That’s ugly and horrible, and I don’t know how that’s allowed, but nonetheless, from the perspective of someone who wants to do unit testing in such an environment, can you give any tips? I mean, do you mock up approximations to what you think these external things should be doing if you really don’t know what they are doing? Do you do your best, updating mocks and tests, “in the face of adversity”?

Really we’ve got two questions here: How do you address the desire to write tests when you don’t know what exactly you need / want yet?

How do you handle external libraries where you’re just not confident in what the expected behavior is?

How do you write tests for something when you don’t know what

exactly that something is yet?

Let’s look at coding a simple User Story and Acceptance Test and tackle them in a way that allows for maximum flexibility and exploration while coding.

The User Story (for a cash register): Keep a running receipt with a short description of each item and its price.

Its Acceptance Test: Setup: The cashier has a new customer. Operation: The cashier scans a toothbrush for $2.25. Verify: The receipt has a description of the toothbrush and its price.

Let’s assume we’ve written absolutely no code. But we want to take a test driven approach and we don’t want to write tests for the bits we haven’t really figured out yet, because, well… we haven’t figured them out and we want to keep the possibilities open. So let’s write the bare minimum required to test this. From the user story and acceptance test above we can tell exactly what classes and methods we’ll need to write and test. So we’ll need:

ICashier (cashier interface)
ICustomer (customer interface)
IRegister (register interface)
IToothbrush (toothbrush interface)
IReceipt (receipt interface)

As for methods we’re going to need:

IToothbrush

-   getPrice()
-   getShortDescription()

IRegister

-   scanItem()

IReceipt

-   addItem()
-   print()

Now, there are three approaches we could take here:

Write a full interface and tests for all of these with all the other methods we expect them to need (like a getUPC() method on IToothbrush)
Write an interface and tests for all the methods we know we’ll need (the ones listed above)
Write an interface and tests just for the items we are using RIGHT NOW.

I’d been doing the latter for a little while and was pleased as punch to find out, when listening to a podcast interview with Martin Fowler, that this is exactly what he’s been doing lately too. I think that this is the best possible approach because it keeps you focused on what you actually need, it gives you much smaller and more quickly obtainable goals, and it gets things working faster. If you took this approach you might proceed something like this (again, assuming no code has been written at this point):

Write the IToothbrush interface.

Add the method you want to address first (get price)

Write a unit test for that method.

check that a toothbrush never has a null price
assert that the price is going to be a decimal number
check that the price is never less than zero.

Now that you’ve got a properly failing test you write a Toothbrush implementation with just a getPrice() method.

In your Toothbrush implementation you’d probably have a no parameter constructor that set up a default price of $0.00, or your getPrice() method would be smart enough to guarantee a non-null price was returned, either way, your test would pass.

I’d say just repeat for each item in the list but it’s not quite that simple. When you get to the getShortDescription() you’ll probably have a test that it’s not null, but we’ve got no method in our spec for actually setting a description so we can’t make an implementation where that test passes unless we create a method to set the description or allow it to be set in the constructor. XP people advocate doing the simplest possible thing that will work, which in this case makes sense and would be to set it in a constructor. So, now we’ve got to add a unit test for the constructor, and it’s reasonable to think that we should set the price in the constructor too. So, in order to really test the getShortDescription() method we also need to write a test for the Toothbrush constructor. So, testing getShortDescription() leads us on to the next bits we need RIGHT NOW.

Add the getShortDescription() method to the IToothbrush interface

Add the testGetShortDescription() unit test

check that the description isn’t null

check that the description isn’t empty

check that it’s actually “short”

this could, conceivably lead you to want create a generic Item interface that has a validateShortDescription(...) and / or truncateDescription(...) method, but I’d recommend holding off until you finish with the items required for this acceptance test. While a good idea, and something to plan on, it’ll be nicer to actually get through this initial acceptance test before diving down the rabbit hole of nice-to-haves.

Add testConstructor() method to your ToothbrushTest test case

test that we can instantiate an object with a valid price and short description
test that we can’t instantiate an object with an invalid price and valid short description
test that we can’t instantiate an object with a valid price and invalid short description
test that we can’t instantiate an object with an invalid price and short description
test that the price and short description come out through the two getters we have in our interface.

Add a constructor that takes a price and a short description to Toothbrush

Add getShortDescription()`` to toothbrush``.

By taking such a focused approach we haven’t constrained what could happen with any of the other functionality in toothbrush. Let the emerging specification define if you need a getColor() method on your toothbrush. If it’s not there now and none of the methods you’ve worked on so far have implied the need don’t go there at all… otherwise when you finally do get there you’re likely to find that “blue” wasn’t an acceptable thing to for it to return at all what it really needed to return was an array of RGB values: [0,0,255]. My advice would be to add a comment to the interface noting that you think a getColor() method might be useful and what you think it should return, that way as you keep coming back and updating IToothbrush you’ll keep that in mind and won’t forget or accidentally code yourself into a situation where it’s difficult to derive the color.

How do you handle external libraries where you’re just not confident

in what the expected behavior is?

The same approach we just discussed for writing the unit test should guide what we mock up of untrusted external libraries AND how we proceed in working with them. Normally I try and work under the assumption that the libraries I depend on are dependable and never write unit tests for them, but sometimes you need to find out what you can depend on or what exactly some methods do because they’ve got no docs. So first you look at the library and ask yourself what methods, based on your limited understanding, do you absolutely need to use? Then you write unit tests that test that confirm that when using it the way you plan on using it it does what you think it would do. You don’t have time to test their entire library or all the possible edge cases for their code so you just test that the bits you will be using work the way you think they do when called in the way you’ll be calling them. For example if your code can’t possibly pass in a negative number then you don’t need to worry if their classes correctly handle negative numbers or not. You just need to make sure that the class handles the full range of numbers you could be passing it correctly and nothing else. Now you know that it does (or doesn’t do) what you expected and you know some things that you can safely pass it.

Depending on how untrusted the library is you may also want to make sure that everyone in your group knew to not use anything in that library that they didn’t have a minimal test case for.

Mike Clark wrote an article wherein he describes a similar approach but for a language instead of a library. His idea was that when you start learning a new programming language you just don’t know how exactly things work. You assume things may work one way but that’s not always the case. He set out to write test cases for all the things he learned how to do in the language. This had a few good side effects: it got him familiar with the testing framework of the language, it got him to actually use all the language constructs (doing is far better for memory retention than just reading) and he now has a test suite of his mental beliefs of how the language works. If a new version of the language comes out he’ll know right away if his mental model of how things works needs to change because his tests will fail.

Code Underwriters

2007-08-31T00:00:00+00:00

Code Underwriters

Lloyds of London is able to do what they do thanks to the concept of underwriters. The simplistic version is that a risk is spread amongst a group of underwriters. If nothing goes wrong they get a cut of the profits relative the the percentage of the risk they took on. If things go wrong they take pay for whatever portion of the risk they agreed to take on.

It occurred to me that being a software developer is in many ways like being an underwriter, except, in most cases, without the big payoff. You see, each developer on a project tends to take responsibility for a portion of the code-base. We talk about spreading this responsibility around by increasing the bus count but the fact of the matter is that this rarely ever happens.

Most code-bases tend to be divided into sections each with a bus count of one or two. When something goes wrong the developer(s) responsible for the responsible piece of code are generally expected to do whatever it takes to make it work now. In some cases this means being woken at 3AM, hauling your butt into work, and remaining there until the bug is squashed. Just like an underwriter at Lloyds, the more risk you take responsibility for the more screwed you’ll be when things eventually go wrong. Unlike a Lloyds underwriter we don’t get a cut of the profits relative to the amount of responsibility we take on.

It is, obviously, quite debatable just what portion of the profits would constitute a “fair” portion of the profits, but in general we just get paid our salary. The first employees of a company frequently do get the payout, but most developers aren’t them. I’ve also heard tale of magical things called “bonuses” but it seems rare that developers actually get them or that they’re enough to buy you more than one box under the Christmas tree if you do see them. So, for the most part, we can treat these things as non-existent, which brings us back to our salary. Which is, in many ways, like insurance. While you don’t get a huge portion of the profits, you do get a guarantee of sorts from your employer that you’ll continue to get paid well even if things don’t work out. Which makes your employer a fellow underwriter in the development of your software.

There are three things that need to be underwritten in any given software project: The effort that will be required to fix things when they break, the money that will be required to finance the fixing of broken things, the money that will be required to finance the creation of potentially broken things. There’s also the money required to finance the project(s) needed to compensate for the projects that didn’t bring in more money than they cost to create and/or maintain but that’s a bit too recursive for this example.

From a financial standpoint, I think it’s understandable why developers don’t get a larger cut of the profits. While the worst case scenario for us is rather unpleasant, it’s also very short lived. But, we still get paid quite well during all those months and years when the project wasn’t bringing in anything notable, and even when it tanks.

Unfortunately developers are human, and humans are emotional creatures. We tend to not pay attention to the thing that are present in our every day lives, like regular paychecks, and focus on the atypical large events. Like having a fear of flying. If you’re afraid of flying then your brain is not paying fair attention to the boring everyday fact that millions of people are whizzing about the skies without any problems. It’s just that when a plane goes down it does so in such an atypically spectacular way, and that’s what we focus on.

So I’d like to propose a couple things for my fellow code underwriters to help mitigate our risk, and thus minimize the atypical large events and bring our perceptions more in line with the small portion of the risk that we’re actually taking on.

Write your tests: unit, integration, and system. It’s like getting underwriters for the risk you’ve underwritten yourself. They’ll put in the hours of work patrolling your code and making sure it isn’t broken so that you don’t have to put in the hours fixing the code later.
Increase your bus count. The bigger the bus count the more people you can throw at the problem so as to fix it faster and / or the greater the likelihood that the problem can be assigned to someone other than you.
Practice Defensive Development. You don’t wear a bicycle helmet because you’re afraid of falling over. You wear a bicycle helmet because you’re afraid someone else will make you fall over.

Defensive Programming 101

2007-08-30T00:00:00+00:00

Defensive Programming 101

For any given programmer the following statement should always be treated as truth:
My code sucks, but your code sucks more - Dave Astels [deleted post]

Good version control habits and test coverage will get you out of most jams related to your own code but we rarely write apps that are comprised of just our code. There are almost always libraries from other people code that you’ll include to save yourself from having to re-invent the wheel. Obviously you don’t want to start writing unit tests for code from other projects (you’d never finish) but there are some basic steps you can take to minimize your chances of failure.\

Untrustable code

The fact is that no matter how good your code is you just can’t trust anything you didn’t write. Furthermore, you can’t trust what people will do with their code.

You can’t trust other people’s code because it rarely has good test coverage or documentation so you can’t be sure if it is broken, or if it works the way you think it does. And unless it’s a huge open source project you can generally assume that not many people have looked at it either.

You can’t trust what they will do with their code because it’s their code not your code. Whoever “they” are they’ve got their own ideas about where it’s going next and how to get it there, and they may not jive with how you want to use it.

Your project must never rely on the successful build of another project, because it may depend on the successful build of yet another project, or even worse, end up in a sea of circular dependencies.

The results of another project’s “current” build should similarly not be trusted. It’s probably not tested well, the implementation could change completely tomorrow, and it may just be buggy. Whenever possible only use release builds. If you can’t use a release build at least try and use one that other people have had success with for a little while.

Untrustable people and environments

Even if you take all the steps possible to defend against other people’s bad code there’s still the fact that humans are incredibly fallible.

You can not assume that the correct libraries will be where they’re “supposed to be” or that the version that is there is 100% compatible with the version you coded against. Even if it is you can’t assume that someone won’t go and “upgrade” it to a version that isn’t compatible. Always bring your own libraries and keep them local to your app. Disk space is cheap and unless you’re on an embedded device it’s better to bring along copies of the libraries you coded against. It doesn’t matter if the ones you bring along are completely outdated relative to the latest version; what’s important is that you can guarantee that the version you coded against works.

Your project must never rely on another project or its libraries. Period. You can not assume that the other project will be on every box your project is installed on or that the version on those boxes be in a state that is either working or compatible with yours. Take whatever you need from their project and copy it into yours. If you need libraries that it generates or contains, copy them. If you must use source code copy it but be very careful. You don’t want to end up forking it and having to maintain your own version. Instead, copy it in but do whatever it takes to make it blazingly obvious that the code should not be edited unless absolutely necessary. When you do copy over make sure to copy the smallest amount possible. If you can, build it into a separate library file that you can update when the need arises.

You also can’t assume that someone isn’t going to screw up a config file, and that includes you. Just like libraries, bring your own. Maybe you’re working on a webapp which has some global configuration file that’s shared by other webapps on the same box. Don’t trust it. Yes use it so that you’ll be in sync with the other apps but since you can’t guarantee that some sysadmin won’t accidentally delete or change some piece of it that’s crucial to your app bring along a fallback with some bare bones settings that should work under any circumstance you can prepare for. This is exactly why your operating system has a “safe mode”. The operating system developers know you can’t trust the users not to accidentally screw something up or install an app that screws something up. Have a fallback.

If your app is configured differently for different deployments (maybe you’ve got a dev box, a QA box, and a production box, each with their own different database settings) then have a separate config file for each deployment. Don’t have a single config file with different sections that get commented or uncommented based on which deployment it is. You simply can’t guarantee that you, or your coworkers, won’t forget to comment or uncomment the right bits when deploying.

99 Lines of code on the wall...

2007-08-20T00:00:00+00:00

99 lines of code on the wall.
99 lines of code.
You look around, refactor it down…
98 lines of code on the wall.

98 lines of code on the wall.
98 lines of code.
You look around, refactor it down…
97 lines of code on the wall.

Or, alternately

    function singVerse(numLines){
    
        if (numLines \> 0){
            document.write("" + numLines + " lines of code on the wall.\\n");
            document.write("" + numLines + " lines of code.\\n");
            document.write("You look around, refactor it down...\\n");
            numLines -= 1;
            document.write("" + numLines + " lines of code on the wall.\\n");
            singVerse(numLines);
        } else if (numLines == 0) {
            document.write("Totally bug free code on the wall\\n");
        } else {
            document.write("Need more tests for the code on the wall.\\n");
        }
    }
    singVerse(99);

Programming books for newbs

2007-06-14T00:00:00+00:00

If you’re reading this blog there’s a fair chance you’re a programmer and that means that from time to time you’ll encounter people who want advice on leaning how to program. Unfortunately, it’s hard to point them in the right direction because we generally don’t want to spend the time to teach them ourselves and even if we did most of the learning to program books just plain suck.

So, I’d like to recommend two books. The first is Learn to Program by Chris Pine.This is the best intro to programming that I’ve ever seen. It’s not concerned so much with how to do things in a specific language as it is with teaching people the basic principles of programming although it uses Ruby to do so. It’s based on a series of tutorials that are still online but have been improved on, and expanded upon greatly in the book.

There’s also JavaScript: A Beginner’s Guide by John Pollock which does a surprisingly good job of teaching basic programming concepts with JavaScript. It’s not that JavaScript makes this hard, it’s that most of the books get sidetracked on doing html specific things instead of actually teaching people to program. Yes, the Rhino book (JavaScript: The Definitive Guide ) is undeniably a good JavaScript book but it’s not good for people who don’t know how to program.

Getting the most out of version control for hosted web apps.

2007-05-23T00:00:00+00:00

Another graph for another friend who asked for a flow chart of the branching and merging described in Best Practices for Web Development html
pdf.

Update:

Michael says: I’m a little unsure, from you diagram, how, if your trunk contains two completed and merged features that aren’t yet live (video upload and REST API, say) you put one feature live without putting the other live. It looks like code only gets to the live branch via the trunk, but it seems from your diagram that the trunk could contain all manner of complete and semi-complete features.

He’s right and I cover this in more detail in my Best Practices For Web Developers Essay html / pdf , but the basic idea is this: Your dev trunk holds the bug fixes and features you want to see in the next release. If, as in Michael’s example, you’ve got the video upload written, but maybe you don’t have the hardware in place to support storing and serving gigs upon gigs of uploaded videos, so you decide to hold off on deploying that. So you leave it be in it’s little branch, remembering to update it at least once a week until you’re ready to include it in the general codebase.

Have you come up with a better solution to Michael’s problem? Please let me know if you have.

I’m really not thrilled with this particular chart though. It’s got no flow. If you can improve upon it please do so and send me a copy. I’ve got some ideas but nothing great. I really liked the Using Darcs with SVN / CVS flow chart much better.

As always you can click on the image for a full sized version and / or download the Dia file too.\

This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.

weblog.masukomi.org - programming

Amazon To Bookwyrm Browser Extension

Overview

Why this exists

What it does and why

Why Only Amazon?

Why only Firefox?

Geekery

The Joy Of No Batteries

Overview

Unfamiliar with Scheme?

Old Cycles & New Learnings

Batteries Not Included

Forced Thought

Parting thoughts

Post Script About The Parentheses

Rewriting Hey

Overview

In the Beginning

Final Totals

Writing in Chicken Scheme

Delusions of Sharability

Crystal Lang

Final Totals

Writing in Crystal

Fast-Forward…

And then Raku

Final Totals

Writing in Raku

Reflecting on the rewrites

If you liked that…

P.S. What about the email app with the same name?

I used AI to generate products & write copy for my store

Quick Summary

It started as a joke…

The work Begins…

Generating images

Prepping images

RedBubble

The site

Know Thyself

That Sweet Marketing Copy

Final Thoughts

Will I share my code?

What’s my favorite?

So Many Images

Mirroring With Gitea

Overview

The Goal

The Process

The Learnings

Do (not) Self-Host your repos

Table of Contents

Why You Should Self-Host

What about GitLab and other Competitors?

Why You Shouldn’t Self-Host

So what’s a geek to do?

What am I going to do?

gh-url script to get GitHub url for a file

The Problem

The Solution

Quality Is Rarely Job 1

Bonsai Coding

Summary Codebases are like Bonsai Trees. Regular aesthetic changes are as

Disovery coding through tests

How do you write tests for something when you don’t know what

How do you handle external libraries where you’re just not confident

Code Underwriters

Code Underwriters

Defensive Programming 101

Defensive Programming 101

Untrustable code

Untrustable people and environments

99 Lines of code on the wall...

Programming books for newbs

Getting the most out of version control for hosted web apps.