tl; dr: two developers use standard debugging techniques to fix electrical problems in their ambulance.
Setting The Stage
We had to drop our car off at the dealership last night. In order to avoid walking home in the dark, on roads without sidewalks, in 27°F (-2.7°C) - or just the last two shortly after dawn, we needed to drive the 🚑 Ambulance. Unlike most years, I actually remembered that it had a block heater1 and had turned it on at the beginning of winter.
Because it was so cold, I went out in the morning and double checked that that the block-heater was turned on. It was. What I didn’t check was the voltmeter2.
The Problem
That evening - in the cold, and dark of winter - I hopped into the 🚑, turned the key, and… nothing. Literally nothing happened. I looked over to the voltmeter and understood why. Zero volts coming from the batteries. Nothing to power the ignition.
A debugging we will go
Gather the initial known truths
The ambulance has 2 separate electrical systems. There’s the 12v DC system (normal car stuff) and the 120v AC system (normal “house” electrical stuff).
An Inverter takes DC power from the batteries and converts it into AC power. This is not very efficient and you loose a fair amount of power along the way even with an expensive inverter like the ones they use in ambulances.
A Converter takes AC power and converts it to DC power.
There’s no reason you can’t have an Inverter and Converter in the same physical device.
The voltmeter reads 0 volts.
The system is acting as if there is no power.
We don’t know where the AC in from the side of the ambulance goes. I suspect it’s in our wiring diagrams3 but we haven’t hunted that bit down yet.
Propose Theories
the batteries are “dead”.
They aren’t set up as some assume. See footnote.4
the batteries are disconnected.
the ambulance isn’t getting power from the house.
The house plug is bad.
The cord is bad.
The ambulance plug is bad.
Choose A Theory And Test It
Eliminate things you can test rapidly.
I started with the second theory - despite it being unlikely - for three reasons: First it could be eliminated in under a minute. Second, because I’d recently replaced the batteries and they live in a slide-out drawer - 🚑s are weird - and its possible one of the leads came loose. Third, because I was already outside and didn’t need to go inside to grab anything to test this. A battery lead being detached is very unlikely to be our problem, because even if it did, each lead is clamped into two batteries. So if it comes off of one post it’s going to still be clamped onto a second one. Two batteries gives us redundancy, reliability, and more available power in an emergency when the engine has stopped running, and thus the alternators aren’t generating any DC power.
The leads were very well attached to the batteries. I couldn’t guarantee that the far end of the leads were well connected because I don’t know where they terminate, but it’s very unlikely that anything has gone wrong there. They’re very thick wires and would need significant clamps to hold them in place.
Work on probable things.
The batteries being dead is the most likely. It’s so likely that we just worked with that as a default assumption, and moved on to the third theory.
The house plug had a self-test light, and GFCI test & reset switches that all seemed to be working. We also had a plug-in AC outlet tester which was easy to grab confirmed. So, it was reasonable to temporarily eliminate it as the problem.
It’s difficult to test if the plug on the ambulance is working because we don’t actually know what it connect to inside the ambulance. It’s probably in our wiring diagrams, but even if we did know, it’s unlikely that we could reach it without disassembling parts of the ambulance.
Next was the cord. Testing that required a multimeter. We couldn’t find our “good one”, but we had an ancient Radio Shack one that still had power. A quick trip to YouTube to make sure we were using it correctly for AC wall power, and we confirmed that the cord was transmitting power. We tested this inside in the basement which was both well-lit, and significantly warmer.
Because we couldn’t easily test that the 🚑 was receiving power via its plug, we had to just assume - for now - that that was still good.
Since we had the multimeter in hand, it made sense to eliminate some things that had just become quickly testable: the batteries themselves. They were - unsurprisingly - dead.
We tested the AC plug on the Inverter (for sending power out). It worked. Since the batteries were completely dead, and the outlet had full power, we could reasonably assume that the power was coming from the cord to the house, which meant that power was successfully entering the ambulance and being used by things in it.
Generate new theories
State of Existing / Old Theories
✅ the batteries are “dead”.
⛔ the batteries are disconnected.
⛔ the ambulance isn’t getting power from the house.
⛔ The house plug is bad.
⛔ The cord is bad.
⛔ The ambulance plug is bad.
Evaluating What We Learned
We have to assume that the power system from the socket on the wall of the 🚑 to whatever it goes to is fine. It’s been sitting still for months, and is unlikely to have had a problem. We’re also assuming that the other end of the battery leads are still connected. Both of these are too hard to test, and too unlikely to bother trying.
The batteries being dead isn’t surprising. It explains why we couldn’t start. It also raises two questions
Why are the batteries dead?
What can we do to give them power?
Normally, plugging it into the wall is all we need to do to give the batteries a trickle-charge that’s significant enough to overcome the normal phantom draw.
Another Quick Test
We have a lithium jump start battery pack for the car. It was our belief that it didn’t provide enough power to turn over the engine, but it was something we could test quickly. It’d be great to be wrong about it.
When I connected it to the batteries, the ambulance lit up like a Christmas Tree. Thus confirming that the batteries were well connected to the system. Turning the key “worked”, but as suspected there wasn’t nearly enough power to “turn over” the engine. I left it connected so that it could help the batteries out when we figured out how to get power into them. It drained very quickly. These emergency lithium jump-start batteries work by providing a lot of power throughput, but they don’t actually have a lot of power stored. It’s just a short powerful burst until the engine starts and can take over.
New theories needed
New theory: Power is making it into the ambulance, but not making it into the batteries.
Batteries don’t have separate “input” and “output” things. DC power is like water. It just flows through the system, self-levels, and leaks out any “holes” (things using power) you may have. Since the batteries were well connected (still assuming far end is good) it meant that the problem was likely to be something in the electrical cabinet.
I believe the batteries also go directly to the starter mechanism, but we’re assuming that that’s good, and investigating questions around getting power into them, not getting power to the starter. We need power to exist before we can bother giving it away.
The problem with this theory is that it is too generic. It helps narrow down the problem, but it doesn’t give us anything specific to test, and the system has thousands of wires, a wall of relays, and many other things.
We went back into the warm, and well lit house to think.
Dachary proposed that we might have blown a fuse. The problem with this is that there are surprisingly few fuses in this system. Because we believed (reasonably) that all the stuff to go from “shore power” to the batteries was in the electrical cabinet, the problem with that theory was that neither of us could recall any fuses in there. One wall of the cabinet has a panel that opens behind the driver’s seat, and we think there are some fuses there. Unfortunately the seats don’t fold forward and it’s a pain to get to even when you can see well. Also, we didn’t believe those fuses controlled anything relevant.
New Theory: blown fuse.
Currently not worth testing, because we didn’t have any likely or easily accessible location to look.
At this point we’re thinking that the Inverter and Converter are in the same box. However, I’ve been bothered by the fact that it only says “Inverter” on it. If there is a separate converter I don’t know where it is.
Dachary digs into the wiring diagram while I test something I can’t remember - I’ve probably just messed up the order of something here. She finds the Inverter, but it doesn’t mention a converter. The rest of the diagram wasn’t very helpful, but there was a reference to another diagram number though. So, I start hunting for that one. There I find two things: First, a separate mention of the converter, implying that it’s a physically separate device. Second, an indication that it’s wired in via a standard AC plug. That would have to be one of the plugs in the electrical cabinet.
I head back out, and start re-examining the cabinet. As I’m looking around at the outlets and plugs and wondering where the converter could be I notice a large metal box that I’ve mostly been ignoring. It says “converter” on it. 🤦♀️
I start examining the converter and notice that it has 3 wires going into it. AC is 3 wires (positive, negative, and ground) and DC is 2. Because a Converter is AC to DC this must be where the power comes in. The connector they used has places you can stick multimeter leads and test if it’s getting power.
New theory: Converter isn’t getting power.
I also notice a single automotive blade-style fuse hidden behind where those wires come in. It’s impossible to tell if it’s blown, and really difficult to access.
Closing In
I grab pliers to try and extract the fuse, and a multimeter to test if it’s getting power.
I dive back into the cold and dark. It’s getting power!
I attack the fuse. It’s blown.
It’s a 30 amp fuse too. That’s a lot. If it hadn’t blown then “Bad Things™” would have likely resulted. E.g. 🚑 + 🔥… and then maybe 🏠 + 🔥.
I stand up and reach into the cabinet with spare fuses5, grab a new 30v fuse, slide it in and the whole ambulance lights up again! I check the voltmeter and it’s back at the standard charging & running level of 13v, which means the batteries are being fed power now!
After a couple hours of changing, it started right up.
Summary
Practicing a considered, and well structured approach to debugging is far more efficient than flailing wildly at random ideas that flit through your brain. It’s also very useful outside of software.
At each step of the way I literally stopped and restated what we knew the known truths to be, and ask what new information we could derive from the theories we’d eliminated or confirmed.
While we didn’t do a formal “post-mortem” to the “production incident”, we did create “action items” from it.
Buy a new multimeter. It doesn’t matter that we have a good one if we can’t find it.
Increase the priority of the existing task to measure, model, and 3D print a custom shroud to improve the water resistance of the 🚑’s outside plug. Rainwater in there seems the most likely cause of a problem.
Good debugging and Incident Management procedures help in your everyday life too.
For a good guide to creating your own incident management processes, start with Graceful Disasters, by me.
Afterward
(based on internet commentary)
The electrical systems in modern ambulances are nothing like those in cars, diesel trucks, or RVs. Unless you’ve helped manufacture these things, or have spelunked in the wiring of one first hand, your assumptions about how things work are probably wrong. Additionally, each one is a literal one-off-custom build.
I assure you that it’s very easy for a brand new starter battery to go dead in 6 weeks of cold weather if an ambulance isn’t plugged in. This makes sense if you understand the regulations these must meet to give you the best chance of survival in some extreme environmental conditions. It makes sense if you understand how they’re used by the EMTs and Paramedics.
Trust me when I tell you, that once we get beyond “Electricity 101” and Ohm’s Law, you should throw your assumptions about how this is wired or constructed out the window. We aren’t guessing about how it’s wired. We have ridiculously detailed wiring diagrams for this specific vehicle. Every wire’s insulation says what that specific wire is for. We know how things are connected. We just don’t know where everything is connected. To put it another way. We know that two wires meet at a junction, we just don’t know which part of the wall each junction is hidden behind.
A block heater is basically an electric blanket for your engine. This is especially notable for large diesel engines for three reasons:
diesel stops being liquid and turns to a gel when it gets cold enough. They actually change the formula in the pumps just before winter temperatures kick in. A diesel engine uses “glow plugs” to heat the fuel. I don’t know what they look like, but they’re essentially just little electric heaters in the fuel system. They take a lot of power, and a many minutes to make the diesel ignitable on cold days.
our engine is so large that even if you do manage to get it started in the cold it takes a long time to heat up all the metal.
diesel engines don’t have spark plugs. They combust the fuel by compressing it, which is harder to do when it’s a gel.
Ambulances have dashboards with a lot of switches (see below). There’s also a voltmeter (the radial-dial). Unsurprisingly an 🚑 has a lot of electrical stuff going on. Unlike most vehicles there’s a non-trivial phantom draw. It will drain its two AGM batteries if it just sits around for a while. This is acceptable, because it was designed to be used literally every day. It’s also got a 120v AC input on the side (with a weird plug). We don’t drive ours much, but we do keep it plugged in.
The voltmeter is important because if yours is low, the EMTs and Paramedics in the back are going to have trouble using things like IV bag warmers and having lights to see what they’re doing. You want to know early if there’s a problem.
Unlike most vehicles every 🚑 has complicated custom wiring. Every ambulance is custom built. What features you get depends mostly on how much your town is willing to spend6. Manufacturers need to keep track of what they put into each one so that they know how to fix it if it comes back for repairs. When we got ours, we called up the manufacturer, gave them our Vehicle Identification Number, and asked them to send us a PDF of the wiring diagram for our vehicle. ↩︎↩︎
The ambulance has 2 AGM batteries wired in parallel. There is not a separate “house” battery and “starter” battery. Within the US there are strict regulations about being able to maintain temperature in the patient area of an ambulance. Basically you need to be able to keep it cool in the middle of Death Valley in the summer, and you need to be able to keep it warm in the northernmost part of Alaska in the winter. You’d need a massive battery to be able to power the heating and cooling systems to achieve this for hours on end. Ambulance manufacturer’s don’t bother. Instead they just guarantee that that will work while the engine is running. In my case they accomplished it by using two beefy alternators. ↩︎
Of course we have spare fuses on hand. A) it’s an ambulance with a billion cabinets. B) It would really really suck to be stuck on the side of the road just because you didn’t have a fuse. Especially in a long distance (thousands of miles) trip like the ones we take. B) of course the fuses are right next to the electrical cabinet. Why would you keep them anywhere else? ↩︎
A new Type III ambulance - the big boxy ones with a van shaped front - starts around $150,000. Most towns sell them after they hit 100,000 miles. That’s how we got ours significantly cheaper. ↩︎
