weblog.masukomi.org - raku

Rewriting Hey

2022-12-18T00:00:00-05:00

Table of Contents

Overview

“Hey!” started as an Interruption Tracker, and now supports Time Tracking too. It has been through 3 iterations: Chicken Scheme, Crystal, and now Raku. This post is a high-level developer’s diary of what I wrote, why I rewrote it, and what I learned along the way.

In the Beginning

Once upon a time I was a tech lead, team lead, and eventually principal engineer. My days were filled with interruptions. They were making it impossible to get anything done, and more importantly to me, they were a source of a lot of stress.

I decided that if I was going to get a handle on the interruptions I needed to understand them. I needed to know when they were happening so that I could plan focus time. I needed to know who was interrupting me so that maybe I could talk to them about it. I needed to know why I was being interrupted so that I could find things I could document better to preempt those interruptions.

It started out as a little Chicken Scheme script that grew organically. It quickly became a hard-to-maintain hairball of code.

But it worked, and it was good.

Final Totals

~1300 lines of Scheme
~2500 lines of JavaScript (mostly library stuff) to support some fancy “graph it via the web” stuff that never really worked out well.
~1000 lines of HTML to hold the JavaScript graphing stuff.
GitHub Repo Here

Writing in Chicken Scheme

I love Scheme. It really works for my brain. Chicken Scheme is one of the better schemes. It’s got a large collection of useful libraries. It also transpiles to C (or JavaScript actually). The end result is a fast tiny binary. As a bonus, you can generate C files so that your libraries can be called from almost any other language.

As an example, my private_comments API web server runs in less than 7MB of RAM and I didn’t even try to make it small.

Unfortunately figuring anything out in Chicken Scheme felt like pulling teeth. There’s a ton of documentation, but it’s mostly useless. It assumes a ridiculous amount of knowledge and doesn’t explain anything. It feels like “Here’s the method signature. Have fun figuring out what it does.”

Getting anything done in Chicken Scheme feels like a major accomplishment. It’s also incredibly draining because it’s so damn hard to figure out if you’re new. It’s not the language’s fault. The language is trivially easy, like all Schemes. It’s the documentation.

These days I use Chicken Scheme very strategically. For example, my CLI heatmap generation tool is written in Chicken Scheme because I wanted something very fast. I also liked that folks could use the core library in any other language if they felt like dealing with their language’s C interface.

Delusions of Sharability

I should also note that I was working under a mistaken belief. I believed that it would be possible to generate a binary for an OS that I could just hand to someone without them having to know anything about Chicken Scheme, or Go, or whatever it was written in.

The truth is, you can’t. It’s theoretically possible. If you could fully statically link a binary, you’d have no worries. Unfortunately it turns out that that, practically speaking, that’s impossible on macOS. They really, really want you to use dylibs. Excluding apps with a “hello world” level of complexity, I’m not sure you can do it.

I spent a lot of time and energy trying to make this happen, but ultimately had to abandon the hope.

Crystal Lang

I’m a Ruby dev by trade, and Crystal is, essentially, just a compiled ruby with Static Typing. Or, it was at the time. These days it’s added concurrency and parallelism and is growing into its own thing.

I wanted to rewrite it in a nice, organized, maintainable way. At the time Crystal was still pre-1.0 volatile, and didn’t have a ton of users or projects. There were, and I think, still are, multiple ORMs (Object Relational Mappings) competing with no clear winner.

Unfortunately, they were very immature tools. So, I took one that I thought was good, and added in relations like “has many” and “belongs to”.

I thought. I planned. I organized. I Object Oriented the 💩 out of it. I wrote and used a sparkline library for Crystal. I wrote a CLI bar chart library for Crystal.

I added a CLI bar-chart graph of interruptions by hour so that I could easily see when I was being interrupted.

$ hey report interrupts_by_hour
Interrupts By Hour:

                                          ▁
                                          █
                                 ▁     ▁  █
                                 █     █  █
                                 █     █  █     ▁
                           ▁     █     █  █  ▁  █
                           █     █     █  █  █  █
                           █     █     █  █  █  █
                           █  ▁  █     █  █  █  █
                           █  █  █     █  █  █  █
                           █  █  █     █  █  █  █
                           █  █  █     █  █  █  █
                           █  █  █     █  █  █  █
▁  ▁  ▁  ▁  ▁  ▁  ▁  ▁  ▁  █  █  █  ▁  █  █  █  █  ▁  ▁  ▁
00 01 02 03 04 05 07 07 08 09 10 11 12 13 14 15 16 17 18 23

Honestly? It was pretty damn good.

But then, my life changed. I was completely burnt out from the constant interruptions. I was having random Panic Attacks over trivial things. My wife was burnt out too. We quit our jobs, cashed in the 401k and lived off of savings for the better part of a year. It set back our retirement, but gave us enough time to begin recovering.

I didn’t need Hey! anymore, so I wandered off.

Final Totals

~1600 lines of Crystal in the core tool (including tests)
~300 for the bar chart lib
~50 for the sparkline lib
~400 for the cli table generator
~150 lines for the custom arguments parser
~300 lines of additions to the DB library (mostly macros)
Total ~2800 lines of Crystal
GitHub Repo Here

Writing in Crystal

Crystal’s a great language. I have nothing bad to say about it. I happily supported them on Open Collective for a long time. If you want to see what I think is a pretty well designed Crystal app, go look at oho. It’s the world’s greatest tool for converting colored terminal output to beautiful HTML. I’m pretty proud of it.

These days though, Crystal isn’t my happy-place. I’ve spent too many years in Ruby. I want something different for my personal coding.

Fast-Forward…

Fast-forward from 2016 to December 2022. I’m unemployed again, like so many of us (Hire Me!), and while I’m actually getting a lot done, I don’t feel like I am. I’m being too hard on myself, and not giving myself the credit I deserve for accomplishing the things I have. I decide to start time tracking. I figured if I could look back and see how I actually spent the day I could have a better feeling for what really happened. “I didn’t get anything done today on my work, but that’s because I spent a ton of time on all these other things that needed doing.”

It turns out, I was right. In addition, the act of tracking my time makes me think about how I’m spending it, as I’m spending it.

I didn’t want to write a new time tracker. I just wanted to track my time. I knew there were multiple, good, off-the-shelf CLI time trackers.

I tried using 3 of them, but they all frustrated me for one reason or another. Some tools wanted me to interface with it in a way that felt awkward or annoying. Some had annoying technical things.

Watson came very close, but every time i needed to backdate a timer i had to type in explicit 4 digit 24 hour time. E.g. 08:30 not 8:30 and 18:47 not 6:47. It’s a little thing but I was doing it at least ten times a day and it annoyed me every time. Also, the ability to edit a timer was just broken.

Then I realized… I already have a time tracker that works with my 🧠. It just doesn’t support end times.

I checkout the Crystal repo. I compile. It goes 💥. It goes 💥 here. It goes 💥 there. It goes 💥 everywhere. The language has changed in notable ways which break my code, and break the code in libraries I was depending on.

In Crystal’s defense, it was 0.33.0 when I ported Hey! to it and they were very clear about the fact that things would be changing before they hit 1.0.

The compiler was very helpful in telling me what changes to make. Eventually though, I hit a case where it was complaining that I wasn’t implementing a method defined in an abstract class, and I definitely was. At that point I gave up. I just don’t have the emotional spoons to go fixing problems I didn’t create in a side-project that I didn’t even want to write in the first place. I don’t particularly want to be writing in Crystal these days anyway.

Maybe the Chicken Scheme version?

Chicken Scheme has moved from v4 to v5 in the intervening years, but Chicken’s upgrade is relatively painless. I’ve already done it for a couple of projects. It’s mostly just a matter of changing some import statements.

20 minutes later and I’ve got a working version of Hey! It’s a worse codebase, but it complies!

Hack. Hack. Hack. Bug. Bug. Bug. I’m fighting with Chicken Scheme again.

“Arrgh! I do not need this. This isn’t even a good codebase to begin with.”

So, I step back. I ask myself what the bare minimum feature set would be that would be good enough to use and not annoy me:

the ability to start and stop a timer
the ability to tag a timer
the ability to backdate a timer because I frequently forget to start or stop them.
no !@#$ 24hr time.
a log of time spent.

That’s not actually that much…

And then Raku

I create a new directory and start hacking. I’m still new to Raku, but overall I’m loving it.

I started by writing about ~250 lines of ugly-ass no-apologies code. I had one goal: get it working in the shortest possible time. 250 lines gets me a DB layer and the ability to start and stop timers (no backdating).

I try to run it. It goes 💥, but that’s to be expected. I have no tests & I’ve literally never compiled it. To my great surprise, the bugs are all little things like missing commas or semicolons. There’s nothing notable.

This is huge. If a Raku newb like me can write ~250 lines of code and not have any notable bugs, it means they have done a damn good job of putting together a language.

I’m chuffed. I keep hacking. Along the way, I’m noticing that lots of little nice-to-have’s are trivial to add while I’m there.

I don’t bother with an ORM. I don’t bother with any OOP (Object Oriented Programming). It’s just a collection of functions that query a SQLite DB with raw SQL and pass the resulting hashes around.

The ecosystem of libraries gets me nice things like time durations as strings “2 hours and 33 minutes” or “2h33m”, and pretty tables. The pretty tables are my fork of another library, but I’m not including that in the totals because I didn’t write it for this project.

I still don’t have the command line graphs implemented, and it looks like I’ll have to port that from Crystal. There isn’t a Sparkline library yet either, so add another ~300 lines of code before I’m done.

Final Totals

~800 lines of Raku
~100 lines of bash_unit tests
~300 lines of future code
Silly levels of satisfaction
GitHub Repo Here

Writing in Raku

Notable things: I wrote precisely zero lines of code to process the command line arguments in a natural language fashion. I wrote zero lines of code to generate or wire up the CLI help doc shown below. Raku gave it to me “out of the box”.

It took me about 5 lines to include a starter db with the package, check if you need it, and put it in place. That’s a combination of the language having a concept of resources in a bundle, a library that understands XDG Base Directory stuff, and my XDG::GuaranteeResources library that combines the two.

Raku’s still got a very small community, but they are “knocking it out of the park” with useful libraries. And the language has been incredibly well thought out. I have complaints, but they’re all nitpicks.

Maybe it’s just that I started my professional programming career with Perl5. Whatever the reason Raku really works for my 🧠.

Usage:
  hey start [<start_args> ...] -- Start a new timer
  hey stop [<stop_args> ...] -- stop an existing timer
  hey log <number> <duration> -- see a log of recent timers
  hey log-interrupts <number> <duration> -- see a log of recent interruptions
  hey running -- lets you know if there are any timers running & what they are for
  hey <name> [<start_args> ...] -- Record an interruption
  hey kill timer <id> -- Remove an unwanted timer.
  hey kill <name> -- Remove an unwanted person / thing from interruptions

    [<start_args> ...]    optional time adjustment, project(s), & optional tags
    [<stop_args> ...]     optional id, and optional time adjustments (e.g. 4 minutes ago)
    <number>              number of duration units
    <duration>            duration string. E.g. minutes, hours, days, etc.
    <name>                name of person / thing that interrupted you
    <id>                  the id of the timer to delete.

Reflecting on the rewrites

Looking back at all this work. I find myself coming to an interesting conclusion. With regards to code written specifically for you:

I believe that choosing subjectively good technology is more important than objectively good technology.

You have limited time to work on projects for you. Don’t spend them in languages you don’t love. Write them with tools, and languages you do love. Write them with tools that are exciting you. Rewrite them when you’ve moved on and you want to use them, but the idea of maintaining the old stuff is just draining.

99% of the tools you write for you will never be looked at by anyone else. Even less of it will be used by someone else. Don’t worry about what’s “good”. Do what makes you feel good.

Chicken Scheme made me feel good. Sometimes it still does. Crystal allowed my to do good things quickly, but it was never Joyful to me. I’m long past the days of Rubyish syntax being Joyful. I like it. It’s good, but I’ve written many many thousands of lines of it and the wonder and joy of it has worn off.

[UPDATE] Since writing this I’ve thought a lot about the choices we make when coding specifically for ourselves vs coding for others. The result is the Duct Tape & Baling Wire Methedology.

Was it worth rewriting all those times?

100%

The first rewrite (Crystal) was in a language I was enjoying, but more importantly it was desperately needed. The original codebase was spaghetti. It had grown without planning, as so many tiny-but-useful tools do.

The second rewrite (Raku) was because I didn’t have the emotional spoons to deal. Fixing problems someone else created, in a tool i didn’t want to be writing, in a language I didn’t want to use… ugh. I didn’t need that.

However, this rewrite also ended up reinforcing my thoughts about how awesome Raku is, and teaching me some new tricks.

If you liked that…

If you found that interesting, you’re probably the kind of geek who writes a bunch of useful command line tools. If so, you should check out TooLoo. That’s my Open Source tool that helps you document your tools. Find them later. Use them more.

P.S. What about the email app with the same name?

Hey! was created before the email service. I figure this project is never going to become popular enough that I’ll have to worry about brand confusion. ;)

I used AI to generate products & write copy for my store

2022-12-06T00:00:00+00:00

Table of Contents

Quick Summary

I used Midjourney to generate art that I threw on t-shirts, coasters, and almost everything else RedBubble offers. Then I used OpenAI to generate copy for it, and combined that with a handful of custom scripts to generate a product site called Bed Bath & The Beyond.

It took about 40 hours of work to put all that together for 13 products, but that will be faster next time. In the end, this is a surprisingly viable option, and the only real restriction on profit is how much effort you’re willing to put into marketing what you make.

It started as a joke…

A friend threw this out on Mastodon.

screenshot of mastodon posts

For a giggle, I asked Midjourney for “UNSPEAKABLE HORRORS AT UNSPEAKABLE PRICES. ALSO, BATH MATS.” and this came out

And they looked “real” enough that I wondered if I could actually make a “cool” bath-mat.

lovecraft & tentacles bath mat

A little editing, and I’d extracted it from the “bath mat”, and thrown it on Redbubble.

the same image on a floor pillow

That’s when things started to spiral out of control.

The work Begins…

Generating images

I spent that night working through images on Midjourney. It was hours of work, but as I was being pretty lenient with Midjourney I was able to get usable results pretty quickly. At the bottom of this post is a screenshot of the stream of images I generated. I’d put it here, but it’s really long. ;)

Prepping images

Then came the editing. By default an upscaled image from midjourney with a 3:2 aspect ratio is only 1536 x 1024 which is fine on your screen, but not so hot for printing.

Fortunately, I’ve actually got a background in print and fabrics are not a high-resolution, or high-fidelity medium. So, upscaling artifacts shouldn’t be a big deal on most things. I’m a bit unsure about the comforters and shower curtains though. Because of this, the acceptable resolution is much lower. Also, RedBubble has this handy page that suggests resolutions for each of their products.

Combine that with Pixelmator Pro’s ML Super Resolution and the upscaling went pretty well.

Most of the images required very little editing.

RedBubble

At this point I’ve sunk at least 10 hours into the process. I’ve made two or three product pages on RedBubble for them. Creating the RedBubble product pages was … work. There’s just no getting around it. It’s a time sink.

In their defense, it requires a lot of subjective human decision making. They’ve also done a ton of work to help folks to promote things. For example, when I click on “Download Images” on the Desk Mat for Billy and Gog’s Grand Adventures I get this excellent collection of promo images to use to promote it.

screenshot of desk mats

Upload, choose, set default colors, save, see what you missed, edit the page to tweak default fabric colors again, repeat.

The site

Hugo is my default static site generator. It’s fast, it’s easy, and I’m familiar with it. So I start making pages, and it… is… taking… forever.

The process was

grab the product url
make a markdown link
choose 2 or 3 items to highlight
download images of the image on those products
make a markdown link to each of those product pages
embed the image in the page with a markdown link to the product page around it.
come up with some witty title and ridiculous copy for the page

I punted on the last one. The process doesn’t sound like much when you say it, but in practice it was hugely time consuming, mentally draining, boring, and soul sucking.

Know Thyself

I knew that if I forced my way through, I could finish the 13 or so products I had. I also knew that I’d never add anything else to the site if I did. I’d definitely not make another site if some other entertaining idea came along.

So, I wrote some code. We’re talking ugly, get it done, do the first thing that’ll work kind a code.

Along the way I discover that for some reason Firefox doesn’t bother to store session cookies in its SQLite cookie database, and it turns out I need session cookies. So, right click, inspect the network call to their GraphQL API (totally undocumented) and “Copy as curl”.

Then write a bash script to read in the clipboard, extract the cookies and throw them into some ancient Netscape Navigator format that curl apparently knows how to handle. This file doesn’t need to be regenerated for …. days? I dunno.

Then copy the url for the product promotion page (the one with links to downloadable images) and run that through another script that extracts the product id, gives it to curl, which utilizes the new cookie file, and hits their GraphQL endpoint to extract the JSON (same stuff as on the web page). It’s more JSON than I need but refining the query would have been wasted time.

Run that JSON through a third script (this one in Raku) that presents me with a list like this. Now I just need to scan the RedBubble promo page to find images I want to use, and type in their numbers.

...
[33] Shower Curtain
[34] Spiral Notebook
[35] Sticker
[36] Tall Mug
[37] Tapestry
[38] Throw Blanket
[39] Throw Pillow
[40] Transparent Sticker
[41] Travel Coffee Mug
[42] Wood Mounted Print
[43] Zipper Pouch
[44] iPad Skin
[45] iPad Snap Case
[46] iPhone Skin
[47] iPhone Snap Case
[48] iPhone Soft Case
[49] iPhone Tough Case
[50] iPhone Wallet
Choose some items (comma separated):

The script considers my choices, downloads the default product image, runs it through imagemagic for resizing, takes my original artwork, resizes that, asks me for the name of the product and ….. voilà I’ve got a markdown file with all the soul-sucking grunt work done for me.

And yes, I have a way to add new products without overwriting the page.

That Sweet Marketing Copy

I never lost sight of the fact that this was, ultimately, a joke. I expect to, over time, make tens of dollars from this. If we consider my hourly rate it’s very unlikely this thing will ever pay for the labor that went into it.

So, because I’m not going to get rich on this, and I’m probably going to get a Cease and Desist letter from a Megacorp who likely lacks a sense of humor and thus shan’t be named. I’m not worried about “real” copy.

However. We’re a day and a half of work into this thing and frankly my ability to write entertaining micro-fiction about each of these is… shall we say, “diminished”.

But along the way I’ve been keeping my friend’s discord ( the one responsible for this stupidity ) in the loop. There someone points to an article about using AI for marketing copy, and I remember that I’d seen a bunch of hype about OpenAI on Mastodon lately. I don’t really want great “marketing copy”. I want creative stories, that happen to also be marketing copy.

So, I make an account there, and see what comes out. I started by creating some copy for this image, which I imagined having come from some old children’s book.

Elouise offers flowers to a tentacle montster

Folks… Text AIs are good.

OpenAI text output screenshot

In the end I edited the Elouise Makes A Friend text down to this:

This image from Elouise Makes a Friend is a timeless reminder of friendship and kindness. It shows Elouise bravely offering a flower to the Tentacle Monster, who in turn offers a kind smile. Whether you choose a the sleeveless top, the throw pillow, or a coffee mug, this image will comfort you and remind you of the power of friendship.

Bring this touching moment from Elouise Makes a Friend into your home. Our products featuring this image are perfect for gifting to that special someone, or for yourself.

Everything required heavy editing, but that didn’t matter. It’s radically easier to edit two to three paragraphs of copy than to come up with some ridiculous story that you then have to edit anyway.

Over time, I got better and guiding the AI, and increasingly impressed with what it would come up with. For example, I let it create almost everything on the About page.

Final Thoughts

Knowing then what I know now, would I do it again? 100% It was fun. I’m ridiculously proud of it, and I genuinely want many of the products.

Would I recommend it to others?

Yes, but with caveats. You really need tooling to get from “I threw something on RedBubble” to “I have a site for my products”. Otherwise your brain will melt. I guess you could outsource that work, but the profit margin is so small here that that’d just put you farther in the hole.

There are some existing tools on GitHub but a quick glance suggested they might be abandoned, and frankly it was faster to write ~100 lines of Raku than figure out and tweak someone else’s stuff.

Also, I really need to find a prettier template. I’d hire someone to make one but there’s no way this thing is going to cover the cost of my time to make it, and I’m not currently employed (hire me!) so that’s not going to happen.

Will I add to the site? Almost guaranteed. You can Follow Bed Bath & The Beyond on Mastodon to see the latest items as I add them.

Maybe? It’s kinda crap, and could use a lot of refinement, and frankly not many folks have realized the joys of Raku yet so it’s probably not in a language you want to poke anyways.

What’s my favorite?

Lovecraft & The Octopus. Something about this just tickles my funny-bone. Also it looks unreasonably good on everything I put it on. I can not explain it. Like, how can you not want this ridiculous clock?

a clock with an image of lovecraft and a giant octopus thing behind him

So Many Images

Working With AI Image generators is like herding cats. This is, honestly, a surprisingly small number of images to get 13 usable products. See A Journey With Midjourney if you’re curious about the process.

Some of these are good, but I haven’t gotten around to incorporating in the site yet. A handful of these are pixelated, because I’ve got another blog post idea involving them… maybe more t-shirts. ;)

Holler if there’s anything there that isn’t on the site, that you’d actually buy. I’ll add a page for it. I’m @masukomi@connectified.com

midjourney image stream screenshot

weblog.masukomi.org - raku

Rewriting Hey

Overview

In the Beginning

Final Totals

Writing in Chicken Scheme

Delusions of Sharability

Crystal Lang

Final Totals

Writing in Crystal

Fast-Forward…

And then Raku

Final Totals

Writing in Raku

Reflecting on the rewrites

If you liked that…

P.S. What about the email app with the same name?

I used AI to generate products & write copy for my store

Quick Summary

It started as a joke…

The work Begins…

Generating images

Prepping images

RedBubble

The site

Know Thyself

That Sweet Marketing Copy

Final Thoughts

Will I share my code?

What’s my favorite?

So Many Images