Debugging Our Ambulance

Debugging Our Ambulance

tl; dr: two developers use standard debugging techniques to fix electrical problems in their ambulance.

Setting The Stage

We had to drop our car off at the dealership last night. In order to avoid walking home in the dark, on roads without sidewalks, in 27°F (-2.7°C) - or just the last two shortly after dawn, we needed to drive the 🚑 Ambulance. Unlike most years, I actually remembered that it had a block heater^[1] and had turned it on at the beginning of winter.

Because it was so cold, I went out in the morning and double checked that that the block-heater was turned on. It was. What I didn’t check was the voltmeter^[2].

The Problem

That evening - in the cold, and dark of winter - I hopped into the 🚑, turned the key, and… nothing. Literally nothing happened. I looked over to the voltmeter and understood why. Zero volts coming from the batteries. Nothing to power the ignition.

A debugging we will go

Gather the initial known truths

• The ambulance has 2 separate electrical systems. There’s the 12v DC system (normal car stuff) and the 120v AC system (normal “house” electrical stuff).
• An Inverter takes DC power from the batteries and converts it into AC power. This is not very efficient and you loose a fair amount of power along the way even with an expensive inverter like the ones they use in ambulances.
• A Converter takes AC power and converts it to DC power.
• There’s no reason you can’t have an Inverter and Converter in the same physical device.
• The voltmeter reads 0 volts.
• The system is acting as if there is no power.

We don’t know where the AC in from the side of the ambulance goes. I suspect it’s in our wiring diagrams^[3] but we haven’t hunted that bit down yet.

Propose Theories

1. the batteries are “dead”. They aren’t set up as some assume. See footnote.^[4]
2. the batteries are disconnected.
3. the ambulance isn’t getting power from the house.
   • The house plug is bad.
   • The cord is bad.
   • The ambulance plug is bad.

Choose A Theory And Test It

• Eliminate things you can test rapidly.

  I started with the second theory - despite it being unlikely - for three reasons: First it could be eliminated in under a minute. Second, because I’d recently replaced the batteries and they live in a slide-out drawer - 🚑s are weird - and its possible one of the leads came loose. Third, because I was already outside and didn’t need to go inside to grab anything to test this. A battery lead being detached is very unlikely to be our problem, because even if it did, each lead is clamped into two batteries. So if it comes off of one post it’s going to still be clamped onto a second one. Two batteries gives us redundancy, reliability, and more available power in an emergency when the engine has stopped running, and thus the alternators aren’t generating any DC power.

  The leads were very well attached to the batteries. I couldn’t guarantee that the far end of the leads were well connected because I don’t know where they terminate, but it’s very unlikely that anything has gone wrong there. They’re very thick wires and would need significant clamps to hold them in place.

• Work on probable things.

  The batteries being dead is the most likely. It’s so likely that we just worked with that as a default assumption, and moved on to the third theory.

  The house plug had a self-test light, and GFCI test & reset switches that all seemed to be working. We also had a plug-in AC outlet tester which was easy to grab confirmed. So, it was reasonable to temporarily eliminate it as the problem.

  It’s difficult to test if the plug on the ambulance is working because we don’t actually know what it connect to inside the ambulance. It’s probably in our wiring diagrams, but even if we did know, it’s unlikely that we could reach it without disassembling parts of the ambulance.

  Next was the cord. Testing that required a multimeter. We couldn’t find our “good one”, but we had an ancient Radio Shack one that still had power. A quick trip to YouTube to make sure we were using it correctly for AC wall power, and we confirmed that the cord was transmitting power. We tested this inside in the basement which was both well-lit, and significantly warmer.

  Because we couldn’t easily test that the 🚑 was receiving power via its plug, we had to just assume - for now - that that was still good.

  Since we had the multimeter in hand, it made sense to eliminate some things that had just become quickly testable: the batteries themselves. They were - unsurprisingly - dead.

  We tested the AC plug on the Inverter (for sending power out). It worked. Since the batteries were completely dead, and the outlet had full power, we could reasonably assume that the power was coming from the cord to the house, which meant that power was successfully entering the ambulance and being used by things in it.

• Generate new theories

  • State of Existing / Old Theories

    1. ✅ the batteries are “dead”.
    2. ⛔ the batteries are disconnected.
    3. ⛔ the ambulance isn’t getting power from the house.
       • ⛔ The house plug is bad.
       • ⛔ The cord is bad.
       • ⛔ The ambulance plug is bad.

  • Evaluating What We Learned

    We have to assume that the power system from the socket on the wall of the 🚑 to whatever it goes to is fine. It’s been sitting still for months, and is unlikely to have had a problem. We’re also assuming that the other end of the battery leads are still connected. Both of these are too hard to test, and too unlikely to bother trying.

    The batteries being dead isn’t surprising. It explains why we couldn’t start. It also raises two questions

    • Why are the batteries dead?
    • What can we do to give them power?

    Normally, plugging it into the wall is all we need to do to give the batteries a trickle-charge that’s significant enough to overcome the normal phantom draw.

  • Another Quick Test

    We have a lithium jump start battery pack for the car. It was our belief that it didn’t provide enough power to turn over the engine, but it was something we could test quickly. It’d be great to be wrong about it.

    When I connected it to the batteries, the ambulance lit up like a Christmas Tree. Thus confirming that the batteries were well connected to the system. Turning the key “worked”, but as suspected there wasn’t nearly enough power to “turn over” the engine. I left it connected so that it could help the batteries out when we figured out how to get power into them. It drained very quickly. These emergency lithium jump-start batteries work by providing a lot of power throughput, but they don’t actually have a lot of power stored. It’s just a short powerful burst until the engine starts and can take over.

  • New theories needed

    New theory: Power is making it into the ambulance, but not making it into the batteries.

    Batteries don’t have separate “input” and “output” things. DC power is like water. It just flows through the system, self-levels, and leaks out any “holes” (things using power) you may have. Since the batteries were well connected (still assuming far end is good) it meant that the problem was likely to be something in the electrical cabinet.

    I believe the batteries also go directly to the starter mechanism, but we’re assuming that that’s good, and investigating questions around getting power into them, not getting power to the starter. We need power to exist before we can bother giving it away.

    The problem with this theory is that it is too generic. It helps narrow down the problem, but it doesn’t give us anything specific to test, and the system has thousands of wires, a wall of relays, and many other things.

    We went back into the warm, and well lit house to think.

    Dachary proposed that we might have blown a fuse. The problem with this is that there are surprisingly few fuses in this system. Because we believed (reasonably) that all the stuff to go from “shore power” to the batteries was in the electrical cabinet, the problem with that theory was that neither of us could recall any fuses in there. One wall of the cabinet has a panel that opens behind the driver’s seat, and we think there are some fuses there. Unfortunately the seats don’t fold forward and it’s a pain to get to even when you can see well. Also, we didn’t believe those fuses controlled anything relevant.

    New Theory: blown fuse.

    Currently not worth testing, because we didn’t have any likely or easily accessible location to look.

    So, we got out the wiring diagrams^[3].

    At this point we’re thinking that the Inverter and Converter are in the same box. However, I’ve been bothered by the fact that it only says “Inverter” on it. If there is a separate converter I don’t know where it is.

    Dachary digs into the wiring diagram while I test something I can’t remember - I’ve probably just messed up the order of something here. She finds the Inverter, but it doesn’t mention a converter. The rest of the diagram wasn’t very helpful, but there was a reference to another diagram number though. So, I start hunting for that one. There I find two things: First, a separate mention of the converter, implying that it’s a physically separate device. Second, an indication that it’s wired in via a standard AC plug. That would have to be one of the plugs in the electrical cabinet.

    I head back out, and start re-examining the cabinet. As I’m looking around at the outlets and plugs and wondering where the converter could be I notice a large metal box that I’ve mostly been ignoring. It says “converter” on it. 🤦‍♀️

    I start examining the converter and notice that it has 3 wires going into it. AC is 3 wires (positive, negative, and ground) and DC is 2. Because a Converter is AC to DC this must be where the power comes in. The connector they used has places you can stick multimeter leads and test if it’s getting power.

    New theory: Converter isn’t getting power.

    I also notice a single automotive blade-style fuse hidden behind where those wires come in. It’s impossible to tell if it’s blown, and really difficult to access.

  • Closing In

    I grab pliers to try and extract the fuse, and a multimeter to test if it’s getting power.

    I dive back into the cold and dark. It’s getting power! I attack the fuse. It’s blown.

    It’s a 30 amp fuse too. That’s a lot. If it hadn’t blown then “Bad Things™” would have likely resulted. E.g. 🚑 + 🔥… and then maybe 🏠 + 🔥.

    I stand up and reach into the cabinet with spare fuses^[5], grab a new 30v fuse, slide it in and the whole ambulance lights up again! I check the voltmeter and it’s back at the standard charging & running level of 13v, which means the batteries are being fed power now!

    After a couple hours of changing, it started right up.

Summary

Practicing a considered, and well structured approach to debugging is far more efficient than flailing wildly at random ideas that flit through your brain. It’s also very useful outside of software.

At each step of the way I literally stopped and restated what we knew the known truths to be, and ask what new information we could derive from the theories we’d eliminated or confirmed.

While we didn’t do a formal “post-mortem” to the “production incident”, we did create “action items” from it.

• Buy a new multimeter. It doesn’t matter that we have a good one if we can’t find it.
• Increase the priority of the existing task to measure, model, and 3D print a custom shroud to improve the water resistance of the 🚑’s outside plug. Rainwater in there seems the most likely cause of a problem.

Good debugging and Incident Management procedures help in your everyday life too.

For a good guide to debugging start with “The Pocket Guide To Debuging”, by Julia Evans.

For a good guide to creating your own incident management processes, start with Graceful Disasters, by me.

Afterward

(based on internet commentary)

The electrical systems in modern ambulances are nothing like those in cars, diesel trucks, or RVs. Unless you’ve helped manufacture these things, or have spelunked in the wiring of one first hand, your assumptions about how things work are probably wrong. Additionally, each one is a literal one-off-custom build.

I assure you that it’s very easy for a brand new starter battery to go dead in 6 weeks of cold weather if an ambulance isn’t plugged in. This makes sense if you understand the regulations these must meet to give you the best chance of survival in some extreme environmental conditions. It makes sense if you understand how they’re used by the EMTs and Paramedics.

Trust me when I tell you, that once we get beyond “Electricity 101” and Ohm’s Law, you should throw your assumptions about how this is wired or constructed out the window. We aren’t guessing about how it’s wired. We have ridiculously detailed wiring diagrams for this specific vehicle. Every wire’s insulation says what that specific wire is for. We know how things are connected. We just don’t know where everything is connected. To put it another way. We know that two wires meet at a junction, we just don’t know which part of the wall each junction is hidden behind.