Fault Tolerant House
I am a software engineer working on massively scalable distributed systems. One of the central design principles is high level
of fault tolerance. When many people use systems like Office 365 for their daily work, failure is not an option.
However, underlying components of such systems are very failure-prone - and there is nothing we can do about it. Computers
fail. Network routers fail. Software has bugs. We are not trying to eliminate these failures - it is impossible. We are, however,
trying to create architectures which are highly tolerant of such failures - where when failure occurs it is taken care of
quickly and automatically and the user does not notice.
There are several principles for such designs.
First is embracing the failure. Assume that everything that can break, will. Prepare for the failure. Design for it.
Second, failover automatically. When one of the systems that you've built upon fails, a standby system must exist to take over,
and the switch to it must be quick and automatic, with no perceptible downtime for the user.
Third, the failover system must have uncorrellated failure modes with the system it backs up. If you have system A which
needs system B for execution, and system C is being a backup system, the backup better not depend on system B, because if
A is down because of B, C is no good as a recovery mechanism because it will be down, too.
Four, though auto-recovery is king, some failures will not be recoverable - and for that we need constant monitoring.
Humans need to know of a problem that requires their action ASAP so they can come and fix things.
If you have a house in which you live episodically - especially during the winter, many things can fail and the failure
can have drastic consequences.
The most obvious failure is electric service. If it's -20F ourside, electric service is down, the water pipes freeze and burst.
Then the services comes back, the house warms up, and the water gushes from the broken pipes and floods the house. That's
an example of a catastrophic failure that can occur. The way most houses are set up today, if the owner lives in Seattle
they two weeks later they will arrive to a house that is thoroughly destroyed.
Another common problem is someone breaking in the house and trashing the place.
A broken sump pump, a leaking pipe, an electric appliance that was forgotten in an "on" stace - all can result
in really bad things happening to the house where an owner is away for extended periods of time.
Luckily, technology exists to solve all these problems.
Heat backup
Most mainstream heating technologies available today involve electricity one way or another. Even propane
furnaces still need electricity for air circulation. When electric service stops, heat pumps and furnaces
stop working.
The first line of defense against electric service failure is a standby generator using propane as fuel.
Many companies make those - we have ones from Generac. The automatic transfer switch fails over to generator
power when it senses the utility failure, and goes back to utility power once it is restored.
Residential standby generators top out at 22kW. A modern house typically has 200 amp service with 240V
current (two leads, 120V each, in anti-phase to each other), which generally is equal to 40-50kW total
available power.
The code in WA requires that the electrician gets the power profile of the house for the entire yearly
cycle and the generator must be capable of sustaining the peak use. If the peak use is above the capacity
of the generator, the circuits must be separated into two groups - one protected by the generator,
and one not protected.
In my experience, a number of electricians were not aware of the peak use requirement and tried to insist
that the circuits are separated anyway - which makes the project a lot more expensive (and really
unnecessary, because for peak power consumption you have to max out all your outlets, AND have the
refrigerator on at the same time as the heat pump and the pool heater and the electric range - which
in practice never happens).
The best solution is to just move to a more experienced person,
but out here, in the rural areas, a more experienced person might not be available. Suggest that the
electrician discusses the power requirements with the L&I inspector. This should clear the misunderstanding.
As of this writing, a 22kW generator is approximately $5k. If the propane service is not available, a buried
propane tank would cost about as much, with excavation work included - and the electrical work will be
on the order of $2k.
Even if the house does not require the maximum power, I think it is wise to just get the most powerful generator
anyway. The difference between 13kW - which is barely adequte for even a very small service - and 22kW as of this
writing is $1500 - just over 10% of the total cost of the project.
If the house has resistive heat, and many houses do, a generator would work for a very short period of time,
but it is spectacularly inefficient way to heat the house by burning propane to convert it to electricity and
then burn the electricity to generate heat. The efficiency of this process is approximately 20% - you waste
80% of BTUs stored in propane.
As an emergency measure this would do, but if power is down for a week, it will be very, very expensive.
Ideally, the house would have a heat pump with liquid propane backup for heating. This gives you the best
of both worlds - it will operate in an efficient heat pump mode when practical, and switch to propane-fired
when not. Trane makes appliances like this. With installation, the costs pushes $20k.
An alternative, cheaper - but less efficient - solution is a gas stove. A modern gas stove can be had for $2000-$3000,
including the installation. It requires fan to operate efficiently, but will work sufficiently well to protect
the house from freexing even without the fan if electric service fails.
In our house we have installed the gas stoves even before the generator, as a first line of defense, before the second
winter in the house. Even though we did not have a power outage that winter, the peace of mind was well worth it.
In addition to gas heating stoves, they make wall-mounted propane heaters. They are cheaper, and take less space,
and may fit the profile of a small cabin better.
Do not - DO NOT - buy a ventless propane heater, however. A regular, properly installed gas stove takes the air
needed for combustion from outside, and returns combustion by-products to the outside using a dual-walled vent.
no inside ait is consumed, and no CO2 is released into the house.
Ventless heaters exist because they are cheap, and they are cheap to install. But they burn propane using the
inside air, and CO2 is released back into the house. These heaters depend on drafts to bring sufficient amount of
oxygen from the outside, and they increase the levels of CO2 in the house - dramatically. Increased levels of
CO2 result in cognitive impairment.
In an office environment, 1000ppm is generally considered the maximum acceptable concentration. I have a gas range
which, when operating without the exhaust fan, increases the concentration to beyond 1000ppm in a 1600 sq ft
home in under 10 minutes. I shudder to think what a ventless heater would do.
If you are in position to design an ideal house a new, here are the heating components I would recommend.
- If a house have a large space - a living room, a common space in the center of the house - a wood stove is
the cheapest, most efficient way to heat it quickly. A lot of land in Eastern WA come with inexhaustible
supply of firewood - which you can harvest yourself, and you cannot beat "free".
- An alternative, or even a complement to a wood stove is a pellet stove. Pellet stove is more expensive
than wood - but it is still much cheaper than propane or electric heat. Pellet stove needs a bag of pellet
dumped into it daily, and the ash emptied out of it every few days, but is capable of producing even,
realiable, and inexpensive heat for about a day unattended. In a cabin, a pellet stove can be the main
source of heat in the winter when people are present.
- A heat pump with propane backup, and a standby generator covering power outages.
Water
One of the realities of living in rural area is having to maintain one's own well. On one hand, this means
not having to pay for water - but on the other hand, if any of the complex machinery of your water system
breaks - it is extremely unpleasant.
To understand what can break, we should first understand how water system works.
A typical water system consists of a vertical shaft that goes into the ground, encased in steel pipe.
The depth of it, depending on the water table, can be anywhere from 50ft to 250ft.
Into that hole the pump is lowered on a pipe, usually 1-2" diameter, which is made of 20ft
individual segments. This is used to both support the pump and also pipe the water to the
surface.
The pipe hangs on a special adapter that is welded to the side of the casing which sits below the frost line.
On the other side of the casing it is connected to the water main.
The water main leads to the house where - usually - a pressure tank would be installed. Buried right by the
water main there is the electric cable that feeds the pipe. That goes to the pump controller.
While usually the pressure tank and the pipe controller are in the house, this does not have to be the case.
On my property there is a well house where all this equipment is housed, and there is another well which
has the pressure tank and the pump controler in an underground structure located right by the pump.
The function of pressure tank is to maintain the water pressure in the house without the pump running.
Pumps used in modern wells are very, very powerful. It makes no sense to turn them on every time one
needs a cup of water. Instead, there is a reservoir of water - 10-50 gallons, depending on the
pressure tank size - which is kept under pressure by the bubble of air at the top of the tank.
When the tap is open, the air expands and pushes the water out of the tank and into the tap. When
the tank is near exhausted, the pressure-activated switch turns the pump back on and refills the tank
(thus compressing the air inside it and creating durable pressure in the system).
This is a pressure tank.
The two electrical components that every well has is a pressure switch and a pump controller. The pressure switch
is almost always mounted right by pressure tank. It turns the system on when the pressure drops below certain level,
and turns it back on when it reaches certain level. These levels are usually 30psi and 50psi.
This is a typical pressure switch. And this is how it is typically installed:
The pump itself is driven by something called a "pump controller". Control box has a couple of capacitors and relays
that ensure that during startup - when the power demand of an electric motor peaks - the pump has enough power.
It basically charges the capacitor, and when capacitor gets enough charge, flushes that charge into the motor.
Pump control box is always matched with the pump - the voltage and wattage (or horsepowerage :-)) are extremely
important. So when buying a new control box ALWAYS know what pump you have (or just buy the exact same version as
you already have).
This is a pump control box. Capacitors are the cylinders on the top right.
The control box shown above is a "deluxe" version of the control box. Regular control box is connected to the
pressure switch in series. When the pressure switch triggers, it powers the control box, which after a tiny period
of time (when capacitors charge) starts powering the pump. When the pump is shut off, the control box is not powered.
The "deluxe" version of the control box has the power wired to it, and then it has the separate wire pair that go
to the pressure switch. When the pressure switch triggers, the relays in control box connect the power to the pump.
Control boxes have some primitive protections about overloads - if pump stalles and the current through it exceeds
some value, they will disconnect. There is a tiny button (or two, depending on the design) at the bottom of the
control box that allows resetting it.
Sometimes you will see another box wired into the pump circuit called "pump saver". It will look like this.
Pump saver contains a computer that analyzes the current going through the pump. The current fluctuates a little
depending on how pump rotates. When the water in the well runs low, the pump will rotate faster than usual, and
if the pressure in the system is too high, the pump will rotate faster than usual. The computer inside the pump saver
senses this, and cuts the power to the pump.
Ok, so now that we know how the water system works, let's talk about what can go wrong.
The absolutely worst of the worst that can happen is the rupture of the water lines. If this happens somewhere
underground, I shudder to think what the repairs would be. Digging up the entire line from the well to the house?
Especially in winter it sounds extremely unpleasant.
The good news is - the water pipes are usually made of extremely resilient polymer and they almost never, ever
rupture. And when they do, this happens near the connectors, not in the middle.
The water line can rupture inside the house, however, if the house is left to freeze in the winter. We have already
discussed the protection from this above.
The next most unpleasant potential breakage is the water pump. If the pump dies - and all mechanical devices die
eventually - you have to get a pump company to come, pull out the pump from 100 feet below, and replace it.
The big problems with this are two - (a) if it is winter, the path to the pump has to be plowed (and boy it is
an unpleasant task for the poor worker to pull out the wet, freezing pipes from underground in -20F weather!), and
(b) in rural areas it could take days - sometimes, a week - for the pump company to even show up. And it is expensive,
as their billed hours start from when they leave their building.
Basically, the catastrophic failure in the water system as above could lead to days or even weeks without water.
If this is just a cabin in the woods, you can just choose to stay home. But if you have animals and actually have to
show up (and water them), that's another story. If you have a garden that depends on the water system, it will probably
be dead by the time the water is restored.
So what can be done to survive? Basically, you have to keep an emergency supply of water on hand. There are two types
of water that is used on a farm - potable water for human consumption, and water for irrigation and animals.
For the first kind, it is useful to keep a 55 gallon barrel of water in the garage.
Believe it or not, Amazon sells them! But the best source for those is not Amazon - it's Craigslist. Search for "55 gallon"
on Craigslist - https://www.craigslist.org/search/sss?query=55+gallon - and voila!
For $20-$30 per drum you can store all the water you want!
A few things though.
First, make sure that provenance of the barrel is known and what was there before is also well known. I usually buy mine
from a local bakery. They get the sweetener in them - so I am absolutely sure that they are food grade (they are in
fact still sticky with syrup when I get them). All kinds of chemicals come in these drums, and you don't want to have
your water supply in a barrel that used to store some liquid version of rat cancer.
Second, when you pump the water in and out of it, remember that garden hoses and various garden faucets contain lead,
and should not be used for potable water. You will probably also want a spigot that you can put at the bottom of
the barrel to get to the water - AND THAT ALSO NEEDS TO BE LEAD FREE.
Obviously, thank gods for Amazon:
Potable water hoses also do exist and can be found in most large stores which have an RV section. They are usually white
to distinguish them as being for potable water. But not always.
If you can put this barrel somewhere high (but remember - it is 400lb of water! Make sure it does not crash through the
rafters if you put it in the attic!) and then when necessary just connect the hose to it to gravity-drain it where you
need it.
An alternative solution is a demand pump such as the ones that are used in RVs and boats. Like this:
This pump turns on automatically once the faucet is opened. It has pressure switch and a very small pressure cavity
built in. You can then even use vynil piping to get the water out to your kitchen or laundry sink.
For a while I nursed an idea to connect the 55 gallon barrel directly into the household water system. It's actually
pretty simple - one can shut off the water main and connect the hose to the drain bib of the pressure switch assembly, and
have another pressure switch and a small pump.
But eventually I buried this idea - and decided to just have a separate vinyl water line to the laundry sink, because
it is probably impossible to completely prevent the entering of bacteria into the supplemental tank, and I did not want
to inject that into the home water system.
So instead I just have a parallel water system fed from this 55 gallon drum in the cellar.
So now on to the irrigation/animal water.
I have a creek going through my property, so for me irrigation backup does not require water storage - just a pump
capable of pumping water from the creek and into the garden. Animals drink from that creek anyway.
If I didn't, Home Depot has a bunch of very large water storage tanks, for example, this:
They have big storage tanks in various colors - you want black because it limits the algae growth.
The alternative "money is not an issue" solution is to bury a large cistern up the hill and plug it into the water
system on the property. My hills are probably too steep for that.
There are two other spots in the water system that can break - but the breakage is less drastic in this case
and much easier to repair.
The pressure tank has a membrane inside that separates the air bubble in it from the water below. If this membrane breaks,
the air eventually dissolves in water and the water level raises gradually reducing the effectiveness of the pressure tank.
Eventually the bubble becomes so small that the pump cycles all the time.
This condition is called "waterlogged pressure tank".
This condition can be tested fairly easily. Pressure tanks have a valve similar to that on the automotive tires - so
the tire pressure gauge can be used for the presence of air. To test it, turn off the pump and let the water flow out through
any open faucet. When the pressure in the system is zero, the pressure in the tank should be 2psi below the turn-on value
of the pressure switch. So for example if the switch is set to 30-50, the pressure in the tank should be 28.
If the pressure is lower, add air to the tank through the same valve you use to measure the pressure using a compressor,
until the pressure is as necessary. If over time the system depressurizes again - time to replace the tank.
This is not an emergency, however, because the tank with ruptured membrane still functions - it's just the air needs to
be pumped back periodically. You can schedule a plumber's appointment at your laisure.
The other things that can break are pump control components - the pump saver, the control box, or the pressure switch.
These are super easy to replace - they are just plug-in components. Since the control box and pressure switch
are cheap, I just keep one of each on hand.
Home security
Though not typically considered a component failure, malicious human activity can cause a disaster at your rural house.
Since you may be away for weeks on end, someone breaking into the house and doing something really bad - leaving a faucet open,
or shutting off heat in the winter - can result in the same sort of problem as a failure of a major component, left
undiagnosed.
Here, monitoring is key, and a good security system is essential. Not rural homes have cell coverage - I consider it
an important selection criterion when buying property - but if yours does, SimpliSafe makes
a pretty comprehensive security system that not only would notify you of the break-in, but will also alarm on water leaks
and freeze conditions. If there is no cell coverage but thre is Internet access, it can be plugged into Ethernet network.
If there is no regular Internet, you can usually install satellite broadband through Hughes. This is a desperate, last resort option,
because this type of the connection is so bad, almost nothing works with it. The problem is not the bandwidth, but latency - a
ping response time is almost a second. Most applications require many roundtrips per transaction - and they simply time out.
I am not sure which security systems do work with Hughes, but almost no security cameras do. As of a couple years ago, Arlo was
a notable exceptions - the latency to connect to Arlo devices was so bad, it drowned out the limitation of the network. I am
not sure if they "fixed" this since - but it used to work.
On its own, Arlo is a pretty poor option. It takes forever to establish the connection to the cameras. It's "security" feature starts
recording AFTER this forever - and long after whatever triggered this recording is long gone. The batteries last... non-deterministic
amount of time. In some cases it could be weeks, in other, months. Some types of batteries last days. There is a lot of discussion
about the dangers of buying batteries on Amazon, because apparently a lot of them are fakes.
I found that Yi 1080p dome cameras are pretty good. I have on the order of 50 of them on my properties, and in 3 years since
I started using them only one failed. They are pretty inexpensive, reasonably reliable, and they do this circular recording
thing that ensures that whatever produced the security events has been captured.
Because of tilt and pan, the point of view can be changed remotely to surveil the "neighborhood" - so fewer cameras are needed.
One big disadvantage of Yi cameras is they depend on the cloud service for everything - so when they go out of business,
all the millions of devices they've sold will instantly become useless. Also, they can record locally, but the alerting only
only works if you subscribe to their web service, which only supports 5 cameras.
An ideal camera system should have a local DVR and a way to access it through a regular browser, without a web service or
a proprietary app. There are very few of those remaining these days.
Regardless of what cameras you chose, some duplication would make system more reliable - going with two or more different
providers is ideal. I have Yi cameras which I use most of the time, Arlo cameras because that's the only thing that worked
when I was on the satellite internet, and a couple of cheap Chinese DVRs scattered around the property.
I also have a bunch of Z-wave devices and SmartThings hubs installed throughout the house and a few barns that monitor
temperature and motion. Mostly they are for automatic light control - but they double as a parallel security system.
There is also a Z-wave water shutoff valve that allows me to turn the water off in the house remotely.
Fire
Wildfires are very common in the areas around our farm. The first year we've bought our property the whole area around it was evacuated.
When it happened, the fire started across the river - which is probably about 50ft wide in that area - and I thought there would
be no way it could reach us - and went shopping. At the store I got a call from my daughter who told me that the sheriff is here
and they are asking everyone to leave.
I ran out of the store, and from 40 miles away it looked like an underground nuclear explosion - a wide column of smoke. Luckily, the wind
changed and our little valley was unaffected - but I learned that the fire can jump 50 ft of water and the road beside it and not even notice.
The year after the smoke was all around Washington all summer.
In the past the fires were always suppressed. Recently the forest management policy changed - they let the fires burn and try to protect
the property, but because of 100 years of accumulated material, the next few years will be particularly brutal.
So how do you fireproof the house?
Most houses in WA have metal roofs. Structures with metal roofs and metal sidings don't burn, unfortunately, vast majority of structures
have wood siding. But if you are building something from scratch, consider cement board. It looks very nice - just like wood - it paints
well - and it does not burn.
Most of us, though, have litte choice in the matter - the houses were already built. For those, spring cleanings - getting rid of dry grass,
needles, dry pine cones, and other material. Ideally the winter accumulation of whatever falls from the trees should be removed every
spring.
If the land around the house is well cleaned, there would be far lower probability of it to catch on fire, and much easier for firefighters
to save it.
Neighbors
Thre will be times where something will fail terribly - and you would need to get something done before you can make
it to the house. In our case, this is almost always about goats finding a way to escape from their pasture.
In rural areas people rely on neighbors far more than they do in urban areas. Establishing and maintaining good relationships
with your neighbors is critical. Help them when they need help, and they will help you when you need it!