871 stories

Shipping Software Should Not Be Scary


On twitter this week, @srhtcn noted that “Many incidents happen during or right after release” and asked for advice on ways to fix this.

And he’s right!  Rolling out new software is the proximate cause for the overwhelming majority of incidents, at companies of all sizes.  Upgrading software is both a necessity and a minor insanity, considering how often it breaks things.

Image result for deploy production memeI’m not going to recap the history of continuous integration and delivery, suffice it to say that we now know that smaller and more frequent changes are much safer than larger and less frequent changes.

But it’s still risky.  And most issues are still caused by humans and our pesky need for “improvements”.  So what can be done?

It’s not ok for software releases to be scary and hazardous

First of all: If releasing is risky for you, you need to fix that.  Make this a priority.  Track your failures, practice post mortems, evaluate your on call practices andImage result for test in production meme culture.  Know if you’re getting better or worse.  This is a project that will take weeks if not months until you can be confident in the results.

You have to fix it though, because these things are self-reinforcing.  If shipping changes is scary and fraught, people will do it less and it will get even MORE scary and treacherous.

Likewise, if you turn it into a non-cortisol inducing event and set expectations, engineers will ship their code more often in smaller diffs and therefore break the world less.

Fixing deploys isn’t about eliminating errors, it’s about making your pipeline resilient to errors.  It’s fundamentally about detecting common failures and recovering from them, without requiring human intervention.

Value your tools more

As an short term patch, you should run deploys in the mornings or whenever everyone is around and fresh.  Then take a hard look at your deploy pipeline.

In too many organizations, deploy code is a technical backwater, an accumulation of crufty scripts and glue code, forked gems and interns’ earnest attempts to hack up Capistrano.  It usually gives off a strong whiff of “sloppily evolved from many 2 am patches with no code review”.Image result for test in production meme

This is insane.  Deploy software is the most important software you have.  Treat it that way: recruit an owner, allocate real time for development and testing, bake in metrics and track them over time.

If it doesn’t have an owner, it will never improve.  And you will need to invest in frequent improvements even after you’re over this first hump.

  • Signal high organizational value by putting one of your best engineers on it.
  • Recruit help from the design side of the house as well.  The “right” thing to do must be the fastest, easiest thing to do, with friendly prompts and good docs.  No “shortcuts” for people to reach for at the worst possible time.  You need user research and design here.  Image result for deploy production meme
  • Track how often deploys fail and why.  Managers should pay close attention to this metric, just like the one for people getting interrupted or woken up, and allocate time to fixing this early whenever it sags.  Before it gets bad.
  • Allocate real time for development, testing, and training — don’t expect the work to get shoved into people’s “spare time” or post mortem cleanup time.  Make sure other managers understand the impact of this work and are on board.  Make this one of your KPIs.Image result for deploy production meme

In other words, make deploy tools a first class citizen of your technical toolset.  Make the work prestigious and valued — even aspirational.  If you do performance reviews, recognize the impact there.

(Btw, “how we hardened our deploys” is total Velocity-bait (&& other practitioner conferences) as well as being great for recruiting and general visibility in blog post form.  People love these stories; there definitely aren’t enough of them.)

Turn software engineers into software owners

The canonical CI/CD advice starts with “ship early, ship often, ship smaller change sets”.  That’s great advice: you should definitely do those things.  But they are covered plenty elsewhere.  What’s software ownership?

Software ownership is the natural end state of DevOps.  Software engineers, operations engineers, platform engineers, mobile engineers — everyone who writes code should be own the full lifecycle of their software.

Software owners are people who:

  1. Write codeImage result for deploy production meme
  2. Can deploy and roll back their own code
  3. Are able to debug their own issues in prod (via instrumentation, not ssh)

If you’re lacking any one of those three ingredients, you don’t have ownership.

Why ownership?  Because software ownership makes for better engineers, better software, and a better experience for customers.  It shortens feedback loops and means the person debugging is usually the person with the most context on what has recently changed.

Some engineers might balk at this, but you’ll be doing them a favor.  We are all distributed systems engineers now, and distributed systems require a much higher level of operational literacy.  May as well start today.

Fail fast, fix fast

This is about shifting your mindset from one of brittleness and a tight grip, to one of flexibility where failures are no big deal because they happen all the time, don’t impact users, and give everyone lots of practice at detecting and recovering from them.

Here are a few of the best practices you should adopt with this practice.

Make operability a high-value skill set.  Never promote someone to “senior engineer” if they can’t deploy and debug their Image result for test in production memeown code.

Software engineers don’t have to become operational experts.  They do need to know the bare basics of instrumentation, deploy/revert, and debugging.

Everyone who puts software in production needs to understand and feel responsible for the full lifecycle of their code, not just how it works in their IDE.

Baking: it’s not just for cookies

Shipping something to production is a process of incrementally gaining confidence, not a switch you can flip.

You can’t trust code until it’s been in prod a while, Image result for deploy production memeuntil you’ve seen it perform under a wide range of load and concurrency scenarios, in lots of partial failure modes.  Only over time can you develop confidence in it not being terrible.

Nothing is production except production.  Don’t rely on never failing; expect failure, embrace failure.  Practice failure!  Build guard rails around your production systems to help you find and fix problems quickly.

The changes you need to make your pipeline more resilient are roughly the same changes you need to safely test in production.  These are a few of your guard rails.

  • Use feature flags to switch new code paths on and offImage result for test in production meme
  • Build canaries for your deploy process, so you can promote releases gracefully and automatically to larger subsets of your traffic as you gain confidence in them
  • Create cohorts.  Deploy to internal users first, then any free tier, etc in order of ascending importance.  Don’t jump from 10% to 25% to 50% and then 100% — some changes are related to saturating backend resources, and the 50%-100% jump will kill you.
  • Have robots check the health of your software as it rolls out to decide whether to promote the canary.  Over time the robot checks will mature and eventually catch a ton of problems and regressions for you.

The quality of code is not knowable before it hits production.  You may able to spot some problems, but you can never guarantee a lack of then.  It takes time to bake a new release and gain incremental confidence in new code.

In summary.

  1. Get someone to own the deploy software
  2. Value the work
  3. Create a culture of software ownership
  4. LOOK at what you’ve done after you do it
  5. Be suspicious of new versions until they prove themselves

Image result for deploy production meme

Two blog posts in one weekend!  That’s definitely never happened before.  Thanks to Baron for asking me to draft this up following the weekend’s twitter thread: https://twitter.com/mipsytipsy/status/1030340072741064704.



Read the whole story
1 day ago
Share this story

Game dev suffers broken teeth, skull fracture after trying to film arrest

2 Comments and 3 Shares

A longtime video game developer was arrested at a South Carolina gas station on Wednesday afternoon after suffering significant injuries as the result of a tackle by a police officer. The arrest followed the woman's attempt to film officers arresting fellow shoppers at a QuikTrip store in Rock Hill. Soon afterward, both sides of the arrest told vastly different stories of what exactly happened.

Former Disney, Turbine, and Ubisoft developer Patricia Pizer posted images from her hospital stay on Thursday morning on her Facebook page. A text post attached to the images, apparently written by Pizer's husband, included the following list of injuries that she sustained following her altercation with police: "Broken teeth (five that we know of), dislocated shoulder, several lacerations, bruised hip, fracture of the skull, concussion."

The news of Pizer's arrest and injuries quickly spread throughout the game development community, with industry peers such as Brenda Romero (Wizardry) and Robin Hunicke (Journey) sharing Pizer's text and video updates (along with a link to a GoFundMe fundraiser to cover her medical bills). Pizer's game-industry credits include creative director for Asheron's Call 2, senior designer for Disney's Club Penguin series, and senior design analyst for MMOs such as The Matrix Online and Uru: Ages Beyond Myst.

“Punched the officer in the head twice”

In a statement offered to Charlotte TV station WBTV, Rock Hill Police alleged that "the officer involved made a lawful arrest and used force necessary to effect the arrest of Ms. Pizer." This was determined, RHPD said, after reviewing an officer's body-cam footage and the QuikTrip store's camera footage (neither of which are currently publicly available). A separate report from HeraldOnline clarified that no dash cam footage of the arrest, which occurred near a police car, was captured, as the original arresting officer did not "turn on his blue lights" when arriving at the scene.

The RHPD statement alleged that Pizer "physically interjected" herself during the officer's arresting attempts, then alleged that Pizer became increasingly physical:

The officer approached Ms. Pizer to place her in handcuffs when Ms. Pizer punched the officer in the head twice and kicked the officer once. The officer then attempted to put Ms. Pizer on the ground, however she grabbed the handle of the door to the store. The officer pulled Ms. Pizer to break her grip on the door. The officer then put Ms. Pizer on the ground and continued to attempt placing her in handcuffs.

A police report filed after the arrest included an allegation that Pizer threw a drink at an officer's face. That allegation did not appear in the statement later offered to the media.

“Cop hits her like a football player”

Pizer's husband offered a different account of what happened on his wife's Facebook account on Thursday evening, based on the QuikTrip security camera footage that he was allowed to see by store management:

Video of the incident shows OFFICER Hernandez Talking to Patricia. He tells her to Walk away or be Arrested. She walks away and starts filming with her phone in Selfie mode. Cop runs up behind her and hits her like a football player. She leaves her feet and goes face first into the glass door of the gas station[.] cop then punches her in the head, Picks her up and slams her. AT NO FUCKING TIME DID SHE HIT THIS BITCH ASS COP. UGH!!!!!! Her teeth were on the ground. I actually found one on the scene[;] a piece of tooth. Blood as well [sic]

One part of the story that all parties appear to agree on is that Pizer turned on her smartphone's camera and began filming the incident. RHPD's statements do not mention

Pizer's allegation to WBTV

that the arresting officer "saw that I was filming him, and he came up to me, and he smashed my camera out of my hand, and it went flying."

Pizer is awaiting legal counsel before deciding whether or not to publicly release whatever video was captured by her phone during the incident. Pizer, a resident of Miami, has not indicated when she will return home; she was in South Carolina as part of a vacation with her husband.

Read the whole story
4 days ago
The participatory panopticon continuing to speak truth to the abuse of power.
New York, NY
3 days ago
Share this story
1 public comment
3 days ago
The Good Guys™

It’s a war over education

1 Share

Keep this in mind when you see so-called intellectuals like the gang at PragerU, or Jordan Peterson, or just about any Republican, demonizing universities as the domain of cultural Marxists with entire disciplines in the humanities and social sciences that need to be razed to the ground. They’re not stupid. They know exactly what they’re doing. Wrecking the electorate’s ability to think and process information is what keeps them rich and in power.

Read the whole story
4 days ago
Share this story


1 Comment and 2 Shares


Read the whole story
5 days ago
Share this story
1 public comment
5 days ago
Apparently true according to Snopes
Waterloo, Canada

The affirmation I need


Do you need one, too?

I want to face the day like this every day.

Read the whole story
5 days ago
Share this story

Saturday Morning Breakfast Cereal - Intelligence

2 Comments and 6 Shares

Click here to go see the bonus panel!

Call me, movie studios. I could write this script in 3 hours.

Today's News:
Read the whole story
5 days ago
Share this story
2 public comments
2 days ago
shared for the comments
seattle, wa
6 days ago
The real joke is of course that everyone goes for Einstein in a world where John von Neumann[0], Srinivasa Ramanujan[1] and Bertrand Russel[2] also exist (well, used to exist)


EDIT: I'm more than slightly annoyed by my male-centric top three here, so I tried to think of extremely prolific female geniuses. So my mind went to Marie Curie[3], Margaret Hamilton[4], Grace Hopper[5], Marilyn vos Savant[6], and Lynn Margulis[7] (my favourite - crazy contrarian conspiracy theorist near the end of her life, but punk AF).

They are all awesome of course, but more as "famous successful and important scientists" (except for vos Savant). None are quite so "polymathy" or extremely prolific.

I can explain that within the historical context of sexism and traditional gender roles but goddamn is it frustrating.

So I check Wikipedia. First the list of women in science. Full of amazing scientists, but nothing that helps me figure out who is *exceptional* among them. So onto the list of child prodigies we go[9]. Well.. ehm... again nobody quite as prolific... Anne-Marie Imafidon[10] is cool but it's too young to tell.

And then there is Hildegart Rodríguez Carballeira, whose tragic story is a stark reminder of what we're actually joking about here

> Hildegart Rodríguez Carballeira (December 9, 1914 in Madrid – June 9, 1933 in Madrid) was an activist for socialism and sexual revolution, born and raised by her mother as a model for the woman of the future. She spoke 6 languages when eight years old, finished law school as a teenager, and was a leader of the Spanish Socialist Workers' Party and afterward of the Federal Party.

> Hildegart was conceived in Ferrol by Aurora Rodríguez Carballeira and an undisclosed biological father chosen by her mother with eugenic intentions. When she was sure she was pregnant, she moved to Madrid, where Hildegart was born. Aurora set a clock to wake herself up every hour, allowing her to change her sleep position so blood could irrigate the fetus uniformly.

> Hildegart spoke 6 languages when eight years old

> In June 1928, at age 13½, she enrolled in the School of Law of the Complutense University of Madrid. She later gave lessons at its School of Philosophy during the Second Spanish Republic.

> She was a leader of the Spanish Socialist Workers' Party and afterward of the Federal Party.

> Hildegart was one of the most active people in the Spanish movement for sex reformation. She was connected to the European vanguard, corresponding with Havelock Ellis, whom she translated, and Margaret Sanger.

> By the time she was 17 years old and had become internationally known, her mother shot her to death.

> There were several hypotheses about the cause of the murder. Hildegart may have fallen in love. She intended to separate from her mother who, out of paranoia, threatened suicide. Aurora's explanation was that "the sculptor, after discovering a minimal imperfection in his work, destroys it." She shot her daughter twice in the head and twice in the heart


[0] https://en.wikipedia.org/wiki/John_von_Neumann

[1] https://en.wikipedia.org/wiki/Srinivasa_Ramanujan

[2] https://en.wikipedia.org/wiki/Bertrand_Russell

[3] https://en.wikipedia.org/wiki/Marie_Curie

[4] https://en.wikipedia.org/wiki/Margaret_Hamilton_%28scientist%29

[5] https://en.wikipedia.org/wiki/Grace_Hopper

[6] https://en.wikipedia.org/wiki/Marilyn_vos_Savant.

[7] https://en.wikipedia.org/wiki/Lynn_Margulis

[8] https://en.wikipedia.org/wiki/List_of_female_scientists_in_the_20th_century

[9] https://en.wikipedia.org/wiki/List_of_child_prodigies

[10] https://en.wikipedia.org/wiki/Anne-Marie_Imafidon

[11] https://en.wikipedia.org/wiki/Hildegart_Rodr%C3%ADguez_Carballeira
5 days ago
Oh and btw, why is it always the genes they focus on? The prenatal environment is also a huge factor, but you never hear anyone about creating a cloning vat copying the womb of mothers of geniuses https://en.wikipedia.org/wiki/Heritability_of_IQ#Maternal_(fetal)_environment
5 days ago
Also, Amish and Hutterites have their own genetic problems due to inbreeding. (Full disclosure: I'm a Anabaptist/Mennonite (but not a hereditary one).)
3 days ago
(Eh, my family name is Van der Zwan. Until three generations ago, they all lived in one single village in the west of the Netherlands so I can probably safely bet that half of my genes are the product of inbreeding: http://www.cbgfamilienamen.nl/nfb/detail_naam.php?gba_lcnaam=van%20der%20zwan&gba_naam=van%20der%20Zwan&nfd_naam=Zwan,%20van%20der&operator=eq&taal= )
Next Page of Stories