Sunday, September 29, 2019

Colorado Startup Life: Weeks 52 to 55

Whew, September 1st to 28th. I meant to blog a lot more this month, I even took a full week vacation, but I do this for free and I kind of have to be in the mood to sit there and write and edit it. In short the highlight of the month was watching the first 19 episodes of Designated Survivor with my uncle look alike Kiefer Sutherland. Just kidding. But it is a fun little tv series.

The first week in September was a bit stressful because we made a major change (15% of the product parts) to our product and we made the hardware change before we documented it in our CAD system or in our production system. It was stressful standing there in a group of the people who needed to build one of our products, and as the configuration engineer, not knowing the configuration that was in testing and needed to be built. Fortunately we kept working at it, I worked that Saturday and we managed to build the product.

The second week in September we tested the product. (I kind of enjoy generalizing what I work on so much that I just call it a product, and not the thing it actually is or even the industry I am in.) The testing went very well and then Thursday the 12th, for the first time in the year I have been here, we shipped our product to a customer! We stayed at work until 7:30 pm wrapping up documentation that we wanted to have before we shipped it. Not super late by any means, but the group of maybe seven people who were milling around, there was an energy and a seriousness. It was exciting in a slightly stressful sort of way.

Friday the 13th was an emotional hangover. We put so much effort into getting this thing out there, and it was the first time we have shipped in a long time, by far the most capable product we have ever shipped (and only second in the company's history). I was worn out, emotionally, by the the last two months of senior employee drama, the push to qualify and ship a product, the reorganization, training new people, and last minute design changes. So I took a week of vacation. I knew that once we shipped our product, there would be a good time to get out of the office, and I planned my vacation around our shipping schedule. I'll give a day by day rundown.

Saturday the 14th: rock climbing at white mountain cliff just a mile or two from Silerthorne right off I-70 then some bicycling near Independence Pass.

Sunday the 15th: Kit Carson and Challenger Peak, 15.2 miles, 7,000 feet of elevation gain and loss, 8.6 hours. The cairns on Kit Carson could be a little better, but it was in general a not so hard 3rd class 14er.

Monday the 16th:  Uncompaghre and Wetterhorn, 18.6 miles, 7,400 feet of elevation gain and loss, 8.3 hours. Uncompaghre looks big and intimidating, but you could probably run 95% of it. Wetterhorn has some pretty cool 3rd class the last 200 vertical feet, quite exposed actually. I was up there around 1 PM, and alone. It was beautiful weather, but not a place you would want to be if there was rain or hail.
Summit of Wetterhorn looking at Uncompaghre 9-16-2019
Tuesday the 17th: Handies Peak, and I need to post a video of this. I borrowed my friend's 2006 lifted Sequoia with 32 inch wheels and drove the rough 24 miles up Engineer Pass and slept at the 11,600 foot trailhead on the sweet bed she built for her Toyota Sequoia. It was cold! In the middle of the night it started raining, with a little hail, and then snowing up above 12,000 feet. When I woke up I took a weather forecast from my inReach that said the weather would clear later in the day and decided to wait a couple hours before starting. So I went to start the car to warm everything up and it would not start!

After five start attempts I opened the hood in the 35 degree rain to see if I had damaged anything, nothing found. There were no indications of leaking oil or gas underneath. Everything looked good. The battery and alternator sounded good, like they had plenty of life in them. So I kept just turning the key to get it to start. On about the 10th try, after 15 minutes of mild panic that I would have to get a tow truck out here 24 miles down a very rough all wheel drive road, it started! Way back in high school a mechanic friend of mine, on a trip to Colorado, told me that in the first minute of an engine running the fuel injection system will use the air pressure to determine the fuel air mixture ratio. So at the time he recommended stopping half way up a big pass and just shutting the car off, and then starting it again to adjust for the lower air pressure. As part of that, the theory goes, the car will use the last start's air pressure as the baseline to start the engine. So, being up at 11,600 feet maybe I just had flooded the engine with a rich mixture and it took some turning over to get the correct mixture, and then in that first minute adjusting to the air pressure conditions. I have no idea if that is actually how the fuel injection control system works, but if it is, that makes perfect sense why I would have trouble starting it at that high of an altitude.

Then at 10:30 I went and hiked Handies, 5.4 miles, 2,500 vertical feet, 2.5 hours round trip. It was snowing as I went up, and the winds on top were gusting to 40 miles per hour. It was pretty rough. I had the clothing to handle it just fine, but it was cold! I really need to post the summit video on youtube. It was a total shift from the sunshine the day before.

Wednesday the 18th: Castleton and Connundrum (which is not a "real" mountain but a bump on a ridge), 7.3 miles, 3,500 vertical feet, 4.3 hours. This was a pleasant hike after a little four wheel driving up a steep road. Descending there was a little loose rock and some snow, but overall a very pleasant hike.

Thursday the 19th: Capitol Peak... 17 miles, 5,300 vertical feet, 8.2 hours, and no summit. I made it up to "K2" at 13,600 feet and looked at the ridge, and you can see a picture of it below. I had not seen a single person in 4.5 hours and even though it was a very nice day, attempting to solo that ridge very alone, if anything happened there would be no one else to press my inReach SOS button. It was risk I wasn't willing to take, so I went down. Kind of disappointing, I was way up here in March 2017 and turned around that time too in beautiful weather at 2:30 PM. Capitol will require a third attempt from me.
Capitol from "K2" 9-17-2019
Friday the 20th: I finally took a day "off". Watched Designated Survivor, bicycled up to the Maroon Lakes, read some. Sat on my hosts deck and drank wine.
Life is good.
Saturday the 21st: Maroon Peak, 12.6 miles, 5,500 vertical feet, 7.9 hours, and a pure delight! I did no attempt the traverse to North Maroon because no one else appeared to be doing it when I was on top and it looked a bit scary to solo. However, the standard route on Maroon Peak was a lot of fun, better rock that I was expecting, and had lot of cairns to follow. I would gladly do it again, and in fact it might be not so hard in winter actually because I think it's too steep to hold deep snow.
Top of Maroon Peak looking at Pyramid Peak and Maroon Lake on the far left.
Sunday the 22nd: Snowmass Mountain, 12.3 miles, 4,300 vertical feet, and 6 hours, and again no summit. Once again I didn't see a single person for 3.5 hours, got off the trail for a solid 45 minutes and when I got to the "class 3" part at 12,600 feet I did what might have been 5.0 with some loose rock and decided that was too much to do alone, so I turned around and went down. There are old climbers and bold climbers but not old bold climbers. I made the decision a long time ago I was going to be an old climber. Every time I read about an "experienced" climber dying or getting seriously injured on a 14er or in the mountains, I am very aware I don't want that to be me. If it happens, it happens, I know it's a risk, but I take steps to mitigate it so that on that day when I am involved in an accident and need a rescue, hopefully I am one helping to extricate the injured person and not the one on the stretcher.
Snowmass Mountain, I turned around just above to the left of the small green patch in the center of the photo on the major cliff band 3/4 of the way up the photo.
Finally returning to work last week, it was really good. I won't lie, with all of the work drama the past few months I debated with myself on vacation if I wanted to stay in this startup and vest more options or try to go back to a big corporation. It wasn't a terribly long discussion in my head, yes I want to be doing what I am doing. However it was good to again choose the chaos and stress of more responsibility than I would have in a larger company. We're doing something unique and hard and with a fairly small team. It's a cool opportunity!

Saturday the 28th I hiked Mt. Sneffels, 13.2 miles, 5,400 vertical feet, 8 hours with three friends including J who used that mountain as his finishing mountain for climbing all the 100 highest mountains in Colorado and the 14ers. It was a very good day! I'm up to 41 official Colorado 14ers and only 12 official ones left to go.

Finally I get asked about my dating life, well, it's not what I would like, it would be great to skip to the committed relationship stage, and not have to go on a bunch of dates and text and get essentially nowhere, but I am meeting new women, and honestly, there are a lot of great women out there! The vast majority of the women I have dated this year are awesome and they are going to be just fine, but let's face it, I'm not normal and I'm not looking for normal, so it's not a fast or easy process. I am confident, as a Christian, that God has a plan for my life and whatever that plan is it is better than my own desires, even, or especially, when I don't understand why I'm 33 and very single. Funny story, I thought coming out to Colorado that there would be more active women I would mesh with and be able to go hiking and climbing with, but I still seem to intimidate many women with the things I have done and do.

Saturday, September 7, 2019

Recovery Setbacks and Resilience

Ugh. After over compressing my ankle descending Crestone Peak two weeks ago, and then before I was totally recovered from that on Monday trying to run five miles. This has not been a great two years for me athletically. I seem to do well on a six month long cycle and I'm now in the process of missing my fall 2019 cycle, the fourth in this chain of physical issues.

It's been depressing at times. A big part of me just wants to throw in the towel and not dream about doing big cool things in the future. For example, I had thought that this would be the year I go and crush Nolan's 14, but with my ankle, that's not the case.

I think sometimes people look to me as resilient. As a person that continues to overcome this and that, unemployment, the 2014 Everest disaster, and more running injuries than I care to admit. When I coached at UD track and field I would say I had "buckets of motivation" and having spent years thinking about it, I still don't know where it all came from or why I have it and others do not. And that's the struggle, how do I describe it to others? I'm sitting in a coffeeshop about to go on a longer bicycle ride this Saturday, because at least I can do that and yet, why?

One thing I like about running and mountain climbing are how accessible or democratized or informal they are. Where as if I was an NFL player or a golfer, you are either in the big leagues, or you aren't. However, there is a spectrum. I stood near the top of Everest with national heroes from other countries and I ran the first mile of the 2013 Chicago Marathon faster than Rita Jeptoo (who was doping). I think maybe that's part of the motivation. Maybe my leg speed is gone and I'll probably never run under 4:40 for a mile again, but if I can get my durability up there are 48 hour races and six day races...

Again, I don't have any magical answer. Sometimes I feel like I have some super deep insecurity that I am trying to fill through athletics that keeps me motivated to recover, but I don't know what would be. Maybe we get up and try to recover and get back to where we have been because, that's just what we do and it's easier to pursue the goal than debate it. I'm off to bicycle.

Monday, September 2, 2019

Changing Orders of Magnitude

As my little startup grows I’ve had some time to think about what sort of scaling I like. I think about orders of magnitude as a good indicator of the challenges. I'm not talking about number of employees, as I have bounced this concept off of some of my coworkers that's what they all assume, but rather number of products made. So to get into it:

First is zero to one. That in many respects is the hardest order of magnitude because you go from nothing to something. Advantages: you don’t have to worry about tolerances, because as long as it fits it works, if not pull out the die grinder.  Disadvantages: performance might be all over the map, it will probably take a long time to make it, it might fail quick, like within 120 feet like the Wright brothers first airplane crashed after four flights. 

At this stage even things I consider engineering and science are really in large part an art project. At my old company we called these builds "mules" because they were cobbled together to test a very specific set of variables. Typically only one was built, although sometimes two were. While we then qualified and warrantied our eventual products to 8,000 hours, a mule might only see 100-200 hours of operation to confirm that some big technical issue worked. 

A fun story, which was stressful at the time, back in the fall of 2012 I believe I was part of a big project, a $150 million dollar program, as a very junior engineer. Prior to my joining two mules had been made to test a new transmission, one for vehicle type A and one for vehicle type B. Vehicle type A was a success, the transmission worked well, but vehicle type B the same transmission just did not have the shift timing required, so about six months before we were planned to build 24 prototype machines, we made a program change to a different transmission. It was a big change that required a lot of design updates, analysis, and tooling changes. It was the right decision and shows the value of "failing" when you make a sample of one.

The next step is from one to ten. Suddenly tolerances start to matter, but you can still pull out a die grinder and make it fit if needed. There are small economies of scale, amazingly the difference between ordering one part and two parts is about 25%. In other words, to buy just one might cost $100, but to buy two might cost $150, and three might be $190. 

The same program I mentioned earlier, in November of 2013 when we had the 24 prototype machines I think six were in the field collecting hours. Suddenly we had a failure that we had not seen before. Turns out I had failed to account for a side load in my FEA the year before. So in the space of two days I confirmed the FEA with the new load case, we made some steel plates, and on the third day flew to Canada to oversee the weld repairs to fix the machine. We ended up having to rework basically all the machines with that particular feature, about 12 in all. 

At this point you can joke about being in production, although even if you try to keep things the same there will be inevitable differences, which might amount to several percent of performance. 

From 10 to 100. This is my favorite order of magnitude to transition across. Tolerances really matter at this point. If you screw it up and have to rework 100 parts, suddenly that’s a big deal. You get sizable discounts on machined parts. Castings and forgings are possible and may be financially justified at this scale depending on the application. At this stage problems are more expensive. 

I like this transition because in addition to the problems typically being more complex (like tolerance stack-ups with 15 parts) there is a level of consistency and volume that gives it a feeling of a real business. A big upside is that certain things become routine, as in every product achieves a basic level of performance. 

As an example, in the winter of 2014-2015 that $150 million program went to production. I was responsible for a very large welded structure. I reviewed the dimension inspection reports for each of the first 20 structures, and we were seeing 100+ points out of tolerance, I went through and listed things that were unacceptable and things that were acceptable, in a four page email. One of the dimensions was the overall length of the structure. It was up to 12 mm too short. I didn't know if that was bad or good. Turns out the other parts that mounted to this structure did not have the tolerance stack up correct fit. We ended up grinding hundreds of holes larger to make the parts fit, until we realized that adjusting the welding fixture on the large structure would solve the problem. 

100 to 1000. At this point you are probably outsourcing some operations to companies that will be doing them full time. Companies that make one of your components will make that during all working hours because the volumes are that high. At this point mistakes become quite expensive. My most expensive mistake was failing to correct an interference between two parts that ended up having over $500,000 in warranty claims in 2015 and 2016 before the design and tooling was updated in late 2015. 

At this stage, things start to happen fast. Mistakes are propagated uncomfortably fast. Making a change, after 100+ have been made an old way is hard and expensive. There is still an air of improvisation, most product lines will have individual product deviations from perfect and dimensions that are out of tolerance (unless the designers are amazing). Production is low enough that if you have to stop the assembly line for a day or a week or even a month, it's not the end of the world. Most likely profit margins are still high enough to tolerate that kind of pause. 

1000 to 10,000. This is where automation starts to become a bigger factor. To get programs set up and running without constant stoppages takes a longer ramp up. At lower levels of manufacturing humans can basically assemble it all, but at this point there will likely be fixtures and keep out cages to keep people away from interfering with the robot operation. 

At this point there will be many continuous improvement changes that at a lower level of production might have been recalls or reworks for all of the parts built. This is why most engineers with some experience don't like to buy the first model year of a new vehicle. There are dozens of changes that need to get ironed out that you don't even know about until you get to a higher volume. Unfortunately sheetmetal and plastic parts are often one of the issues. A skilled assembler will be able to make the part fit by loosening and tightening various mating parts, but then a new person will try it and nothing will fit. Inevitably the design will need to be changed to make it more robust. Similarly, long term drivetrain issues often start to appear at this magnitude. Gears and bearings are often robust enough to last thousands of hours, but minuscule misalignment will often destroy them. In other words, this is the reliability of vehicles in the 1960s, 1970s. 


10,000 to 100,000. This is the limit of my experience. I’ve never worked on a part with over 60,000 used per year. Stamping is huge at this point, because it’s super fast and repeatable. Mistakes are different at this stage. Programs are typically large enough with enough testing that the really low hours, gross errors, don't happen any more. Improvements are typically based on warranty claims at this point. It may be millions of dollars in warranty claims over years, but it can take awhile for those to show up. They are often due to specific conditions. The largest warranty issue I worked on had something like $2 million in warranty claims over seven years and it took us more than half a year to figure out how to replicate it in the lab. It was quite debatable on how to go about fixing it. 

At this level statistics becomes far more important. At lower levels of production when there is an issue, you just go fix it. You might have to parallel path multiple different fixes if you aren't totally sure what will do it, but at this level there are random failures that statistically are not worth the time and effort to go and fix. For example, I have only ever heard of one belay loop breaking on a harness ever. He was a famous climber, so it's a somewhat well known event. As a result the climbing companies did some testing and determined it was so rare no big changes were needed. In the automotive world, this is basically the range of statistical confidence that manufacturers get into when they are doing their prototype testing. It becomes much more expensive to fix these "little" issues or even find them when testing 100 prototype cars. 

From 100,000 to 1 million is beyond anything I have worked on. However, I am familiar with a couple stories, when bolts have a bad steel lot for example, or the base steel material is defect, it can be nearly impossible to determine when the problem began. When you see huge automotive recalls this is because they are getting into corner cases of corner cases. The Takata airbag saga is still another level or two beyond this level, but it is illustrative of the issues at this level. Somewhere around 20+ people died and 250+ injured worldwide and there are 45 million affected vehicles that had those airbags. Finding that kind of error in a typical testing program is not guaranteed. I'm sure after one or two people died the engineers and managers scratched their head and wondered, is it a bad design or a random occurrence? Kind of like Teslas having crashes while on autopilot. It's hard to say from such a small sample size with much confidence what the real issue is. People slip and fall and die in the shower all the time, but we don't quit taking showers. 

In short, while I really like statistics, these huge orders of magnitude do not really interest me because the problems can be so complex with so many variables that a solution can easily be worse the the original problem, even with additional testing. That doesn't mean we shouldn't try for improvement, rather that the reality is we will never achieve it. In other words, there will always be engineering jobs, even if something happened to reset the world back to zero and we had to start over again. 

Sunday, September 1, 2019

Colorado Startup Life: Weeks 46 to 51

July 21st to August 31st. Woof. Work has been tumultuous. First the bad, get the elephant out of the room. After our CTO was fired, about a month later two of the "original six" had a rather public fight. Trust was broken with both of them amongst us little people. The week after, one of them quit. Related, but not exactly related, a third person of the "original six" quit citing wanting to diversify financially, which makes sense after spending nearly four years at a little startup. Point being, we lost a lot of history and experience in the space of a little over a month, and are down to three of the original six people at the company. In particular, we lost experience on the design and virtual side of the business. Our production team is doing just fine, and our test team is rapidly growing into it's own responsibilities, but our design side has struggled. On top of all that we reorganized, which I am very optimistic about. I again have a first time manager for the fourth time in my career, who is going to go far, he's pretty awesome, he's also younger and every time I have a younger manager I ask myself, "What did I do wrong that my career has not reached that level?" Anyway, both him and our now most experienced design engineer both had babies on Wednesday and are out on paternity leave, so suddenly the design team is looking very small and inexperienced. Not to mention that we have made a relatively large number of changes (about 15% of the product) in the last month and have to ship our product to our first customer in two weeks.

Trying to get everyone on the same page, and get all of the work done, is difficult. It's interesting, because we have high standards, both individually and as a company, there are things that honestly, we don't have to do to meet our customer obligations, however, doing them is clearly the right thing, and that's what's causing me stress. Essentially, as configuration engineer, I want to deliver a packet that more or less says, 'this is what you bought'. Even if we just keep it internally, it's important, if anything goes wrong, that we know exactly what we shipped. The design matches the assembly and the testing.

On the scale of turmoil that is possible, we're actually not doing bad at all. It's interesting to see the new people, I think we are up to 52 people total, come in and have no idea both of what we have done, and what we haven't done. It's easy to think from the limited public information out there that we are more advanced than we are. I struggle because despite making huge progress in the past year, and especially this past week, I know we have a long way to go to even justify our current valuation, let alone any future dreams of big success (profitability).

There is a part of me that thinks I should just go work for one of the big beaurcratic companies where I just navigate the system to get things done and they move super slow, but have lots of stability and a lower emotional investment because of my lower status in the company relative to it's overall trajectory. That being said, everyone knows startups go through tough times. I feel a bit like Jony Ive at Apple in the 1990s. Meaning, I think this is the place to be, but at the moment with all of the chaos, I don't know. Our success as a company with our products is not guaranteed at all. At my old company, you knew your market, you knew the volumes through good times and bad times, you knew the problems, and you knew how to deliver a new product successfully. We're just not there yet. We're not ready to ship the Mac or the Apple II or the iMac, those cool things that ultimately were precursors to really big changes like the iPod and then the iPhone.

There's an old saying among professional runners, I think one of the British guys from the 1960s or 1980s, that if you don't question pursuing this goal, you didn't aim high enough. There are lots of bad days as an athlete. There are lots of mundane average days too. Then occasionally there are those days where you just float and PR by 19 seconds in the 5k. The second part of the professional runner quote is that ultimately, you can question pursuing the goal, but the decision to pursue the goal has already been made. You're going to go run 18 miles in the 50F degree rain, because to get where you are going every workout matters, even when they aren't so fun.

I think this stress is just a short term thing really, and I plan to take a full week of vacation in September after this little push the next two weeks. We have unlimited vacation but I've only taken six days this year, that's just less than one day per month, and we don't have a lot of paid holidays, unlike my last company.

Running and climbing are more or less going well. I twisted my ankle last Saturday coming down Crestone Peak. However I still did it, including class 3 parts, for 20.2 miles, 7600 feet of elevation gain and loss, in 9:07 which is pretty good. I even "raced" a pick up truck down the last 2.5 miles of the four wheel drive road, and won, by like two minutes. However it's taken most of the week for my ankle to recover. I've now done 34 Colorado 14ers and have 19 remaining. I also did Dallas and Teakettle two technical 13,000 foot mountains part of the 100 highest in the state. I've done the third flatiron twice in the last weeks weeks after work, it's a delight! I've even plain run 8 miles over trails from 5800 up to 6800 feet in the my longest continuous run in the recovery. It's been five months now and I'm happy with what I can do, but frustrated that my ankle recovery isn't fast. I just want to cry sometimes.

In other news, I am renewing my lease to stay in Longmont another year. At church I've befriended a somewhat recent widow in her 80s and most weeks she will have me over for dinner. Totally strange I think, but I like it and frankly, there is a loneliness epidemic in my generation and little things like this help mitigate it. Dating is going. I'll just leave it there.