George V. Reilly https://www.georgevreilly.com/ george@reilly.org George V. Reilly tag:www.georgevreilly.com,2011-06-11:/atom/ https://www.georgevreilly.com/favicon.ico https://www.georgevreilly.com/feed-logo.png 2021-05-10T07:00:00Z acrylamid Passphrase Generators tag:www.georgevreilly.com,2021-05-10:/blog/2021/05/10/PassphraseGenerators.html 2021-05-10T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <a class="reference external image-reference" href="https://xkcd.com/936/"><img alt="Password Strength" src="https://imgs.xkcd.com/comics/password_strength.png"/></a> <p>I've been using <a class="reference external" href="https://en.wikipedia.org/wiki/Password_manager">password managers</a> for at least 15 years to keep track of all my passwords. I have separate, distinct, strong passwords for hundreds of sites, and I've only memorized the handful that I need to actually type regularly.</p> <p>I started out with the <a class="reference external" href="https://www.georgevreilly.com/blog/2006/02/06/200KeePassEntries.html">KeePass</a> desktop app originally, but I switched to the online <a class="reference external" href="https://www.georgevreilly.com/blog/2016/01/07/DicewareAndLastpass.html">LastPass</a> app about a decade ago. At work, we use <a class="reference external" href="https://1password.com/">1Password</a>.</p> <p>When I register for a site, LastPass generates a random password for me, such as:</p> <pre class="literal-block"> tV%5joS$U6^uY5xU T2oEUY!g70Iv1b&amp;I 8kNHg9*A5GMR9%8D </pre> <p>LastPass securely syncs my passwords between machines and devices. Its browser integration and its Android and iPhone apps mean that I rarely ever have to actually type any of those ugly messes in.</p> <p>But when I do have to type in such a password, it's unpleasant in a browser. It doesn't help that LastPass in some cases displays passwords in a sans-serif font that makes it easy to <a class="reference external" href="https://typography.guru/journal/letters-symbols-misrecognition/">misrecognize</a> letters such as <tt class="docutils literal">Il</tt>, <tt class="docutils literal">0O</tt>, <tt class="docutils literal">5S</tt>, or <tt class="docutils literal">8B</tt>. It's far more painful in an Android app, where you have to switch the keyboard in and out of symbol mode. It's usually even worse in iPhone apps, which rarely offer you an option to see your password in the clear as you're laboriously typing it, so it's easy to make a mistake. When I tried to use a remote control to enter my Netflix and Amazon Prime passwords into a new set-top box, I got so annoyed that I brought down a real keyboard and plugged it into the USB port.</p> <p><a class="reference external" href="https://theintercept.com/2015/03/26/passphrases-can-memorize-attackers-cant-guess/">Passphrases</a> have nice properties compared to random passwords: they're human readable, they're much easier—if longer—to type, and you can actually remember them if you have to. A passphrase of at least five words (chosen by a secure random generator) is computationally infeasible to crack.</p> <p>The ur-example of random passphrase generators is <a class="reference external" href="https://en.wikipedia.org/wiki/Diceware">Diceware</a> from 1995. There are various problems with the Diceware wordlist, which are rectified by more modern lists, such as the <a class="reference external" href="https://www.eff.org/deeplinks/2016/07/new-wordlists-random-passphrases">EFF Wordlists</a>.</p> <p>Which would you rather type? The <a class="reference external" href="http://www.catb.org/jargon/html/L/line-noise.html">line noise</a> above or one of these passphrases?:</p> <pre class="literal-block"> confident starfish aftermost elsewhere jasmine shun baggage chaps reward cuddle avenue rut pardon skating earlobe latter blissful snippet jolt corroding upstage-divinely-ninth-unfilled-skeleton SkimmingMachinistBlessHesitancyKissableRink </pre> <p>When I want to generate a random passphrase, I tend to use either the <a class="reference external" href="https://github.com/ulif/diceware">Python diceware</a> command-line tool or Glenn Rempe's JavaScript-based <a class="reference external" href="https://www.rempe.us/diceware/#eff">Diceware website</a>. Both use cryptographic random number generators to generate excellent passphrases.</p> <p>The <a class="reference external" href="https://1password.com/password-generator/">1Password Online Generator</a> (in Memorable Password mode) also generates passphrases, as do the desktop and browser versions of 1Password.</p> <p>My master password for LastPass is a passphrase, as is my laptop password. I'm also using <a class="reference external" href="https://authy.com/">Authy</a> for 2FA, but that's a post for another time.</p> <div class="admonition tip"> <p class="first admonition-title">Tip</p> <p>If you have to supply answers for one of those misbegotten <a class="reference external" href="https://www.okta.com/blog/2021/03/security-questions/">security questions</a>, such as your favorite movie or your first car, <em>do not answer truthfully</em>. Truthful answers increase your risk of identity theft. The answers are often guessable, can frequently be learned easily about you, and may be obtained through a password breach on another site.</p> <p>Instead, generate a passphrase as the &quot;answer&quot; <em>and store it and the question in the Notes field of your password manager</em>. If you have to supply the answer to a security question over the phone to a customer service rep, you'll be thankful that you chose something that you can clearly say aloud.</p> <p class="last">Also <a class="reference external" href="https://www.mentalfloss.com/article/522136/taking-facebook-quizzes-could-put-you-risk-identity-theft">Facebook quizzes</a> and memes like &quot;Your porn name is your middle name and the first car you had&quot; are trying to obtain your answers to common security questions. Don't answer them.</p> </div> Punctuating James Joyce tag:www.georgevreilly.com,2021-05-08:/blog/2021/05/08/PunctuatingJamesJoyce.html 2021-05-08T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <a class="reference external image-reference" href="https://www.writermag.com/improve-your-writing/revision-grammar/punctuation-bootcamp/"><img alt="Punctuation Boot Camp: Our ultimate grammar guide" src="https://cdn.writermag.com/2018/07/punctuationbootcamp_news-e1540567976133.jpg"/></a> <p>In <a class="reference external" href="https://lithub.com/the-punctuation-marks-loved-and-hated-by-famous-writers/">The Punctuation Marks Loved (and Hated) by Famous Writers</a>, Emily Temple relays a range of opinions from writers such as Tom Wolfe, Elmore Leonard, and Ursula K. Le Guin on periods, semicolons, hyphens and more.</p> <p>On commas:</p> <blockquote> <p>Listens to the sound of the sentence, and is always right, Bob: Toni Morrison</p> <blockquote> [On her editor, Bob Gottlieb, who famously “was always inserting commas into Morrison’s sentences and she was always taking them out”] We read the same way. We think the same way. He is overwhelmingly aggressive about commas and all sorts of things. He does not understand that commas are for pauses and breath. He thinks commas are for grammatical things. We have come to an understanding, but it is still a fight.</blockquote> </blockquote> <p>On periods:</p> <blockquote> <p>Tolerates it, if he must: Cormac McCarthy</p> <blockquote> <p>I believe in periods, in capitals, in the occasional comma, and that’s it.</p> <ul class="simple"> <li></li> </ul> <p>James Joyce is a good model for punctuation. He keeps it to an absolute minimum. There’s no reason to blot the page up with weird little marks. I mean, if you write properly you shouldn’t have to punctuate.</p> </blockquote> </blockquote> <p>My own prose tends towards longer sentences, often sprinkled with dashes, parentheses, and semicolons.</p> <p>Since 2004, I've adapted all of James Joyce's <em>Ulysses</em> for staged readings by the <a class="reference external" href="https://www.wildgeeseseattle.org/">Wild Geese Players of Seattle</a>, and I'm in the Morrison camp, not the McCarthy–Joyce one.</p> <p>Paragraphs like these work on the printed page. (More or less.)</p> <blockquote> <p>The tear is bloody near your eye. Talking through his bloody hat. Fitter for him go home to the little sleepwalking bitch he married, Mooney, the bumbailiff's daughter, mother kept a kip in Hardwicke street, that used to be stravaging about the landings Bantam Lyons told me that was stopping there at two in the morning without a stitch on her, exposing her person, open to all comers, fair field and no favour.</p> <p class="attribution">&mdash;Anonymous narrator, Episode 12, “Cyclops”, L400</p> </blockquote> <p></p> <blockquote> <p>Martin Cunningham forgot to give us his <a class="reference external" href="http://www.jjon.org/joyce-s-allusions/spellingbee-conundrum">spellingbee conundrum</a> this morning. It is amusing to view the unpar one ar alleled embarra two ars is it? double ess ment of a harassed pedlar while gauging au the symmetry with a y of a peeled pear under a cemetery wall. Silly, isn't it? Cemetery put in of course on account of the symmetry.</p> <p class="attribution">&mdash;Mr Bloom, Episode 7, “Aeolus”, L170</p> </blockquote> <p>But imagine trying to read those sentences <em>aloud</em> during a performance and bring the sense of the text to the audience.</p> <p>As an aide to my performers, I've introduced “cadence bars” (denoted by ‘≀’) to the scripts to augment Joyce's sparse punctuation and to bring out the individual fragments.</p> <blockquote> The tear is bloody near your eye. Talking through his bloody hat. Fitter for him go home ≀ to the little sleepwalking bitch he married, Mooney, the bum·bailiff's daughter, mother kept a kip in Hardwicke street, that used to be stravaging about the landings ≀ Bantam Lyons told me ≀ that was stopping there at two in the morning ≀ without a stitch on her, exposing her person, open to all comers, fair field and no favour.</blockquote> <p></p> <blockquote> Martin Cunningham forgot to give us his spelling·bee conundrum this morning. It is amusing to view the ≀ unpar ≀ one ar ≀ alleled ≀ embarra ≀ two ars is it? ≀ double ess ≀ ment ≀ of a harassed pedlar ≀ while gauging ≀ au ≀ the symmetry ≀ with a y ≀ of a peeled pear ≀ under a cemetery wall. Silly, isn't it? Cemetery put in of course ≀ on account of the symmetry.</blockquote> <p>I've also added some pseudo-hyphens (bum·bailiff, spelling·bee, what·do·you·call·him) to counteract Joyce's Germanic habit of stringing several words into one.</p> <p>This seems to help, though some of our readers have to fight a tendency to pause too much when they encounter a ‘≀’ symbol.</p> Now You Have 32 Problems tag:www.georgevreilly.com,2020-04-23:/blog/2020/04/23/regex-32-problems.html 2020-04-23T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <p></p> <blockquote> <p>Some people, when confronted with a problem, think “I know, I'll use regular expressions.” <a class="reference external" href="http://regex.info/blog/2006-09-15/247">Now they have two problems</a>.</p> <blockquote> — Jaime Zawinksi</blockquote> </blockquote> <p>A Twitter thread about <a class="reference external" href="https://twitter.com/nbashaw/status/1253186961482715136">very long regexes</a> reminded me of the <a class="reference external" href="https://www.georgevreilly.com/blog/2009/07/11/64bitWindows7.html">longest regex</a> that I ever ran afoul of, a particularly horrible multilevel mess that had worked acceptably on the 32-bit .NET CLR, but brought the 64-bit CLR to its knees.</p> <blockquote> <p>Whenever I ran our ASP.NET web application [on Win64], it would go berserk, eat up all 4GB of my physical RAM, push the working set of IIS's w3wp.exe to <em>12GB</em>, and max out one of my 4&nbsp;cores! The only way to maintain any sanity was to run <tt class="docutils literal">iisreset</tt> every 20&nbsp;minutes to gently kill the process.</p> <p>WinDbg and Process Explorer showed that the rogue thread was stuck in a loop in <tt class="docutils literal">mscorjit!LifetimesListInteriorBlocksHelperIterative&lt;GCInfoLiveRecordManipulator&gt;</tt>. I passed a minidump on to my former colleagues in IIS, who sent it to the CLR team. They said:</p> <blockquote> The only thing I can tell is that it is Regex, and some regex expression compiled down to a method with 456KB of IL. That is <em>huge</em>, and yes 12GB of RAM consumed for something like that is expected.</blockquote> <p>With that clue, I was able to track down the problem, a particularly foul regex, built from a 10KB string, with 32&nbsp;alternating expressions, each of which contains dozens of alternated subexpressions. The string is built from many smaller strings, so it's not obvious in the source just how ugly it is.</p> </blockquote> <p>I never wrote a followup post explaining how I dealt with this beast.</p> <p>The regex was used on the <a class="reference external" href="https://www.cozi.com/calendar/">Cozi calendar</a> to parse appointments in everyday language, such as “Ann/John Dinner out Friday at 8pm” or “John's birthday every Dec. 7”. These would get translated into (possibly recurring) <a class="reference external" href="https://tools.ietf.org/html/rfc5545">iCalendar</a> appointments.</p> <p>Some of the subexpressions mentioned above looked like:</p> <ul class="simple"> <li><tt class="docutils literal">ordinals = <span class="pre">&quot;1st|2nd|...|31st&quot;</span></tt></li> <li><tt class="docutils literal">short_days = <span class="pre">&quot;Sun|Mon|...|Sat&quot;</span></tt></li> <li><tt class="docutils literal">full_days = <span class="pre">&quot;Sunday|Monday|...|Saturday&quot;</span></tt></li> <li><tt class="docutils literal">short_months = <span class="pre">&quot;Jan|Feb|...|Dec&quot;</span></tt></li> <li><tt class="docutils literal">full_months = <span class="pre">&quot;January|February|...|December&quot;</span></tt></li> <li><tt class="docutils literal">recurrence = <span class="pre">&quot;((every|each)?</span> (first|second|third|fourth|fifth|last)? &quot; + &quot;(&quot; + short_days + &quot;|&quot; + full_days + &quot;)&quot; + ...</tt></li> </ul> <p>I've elided the intermediate values but they were spelled out in the original. Some of the simpler subexpressions were repeated several times, nested inside others.</p> <p>This all screamed <em>grammar</em> and <em>real parser</em> to me, but the test suite also screamed <em>here be dragons!</em></p> <p>I resisted the temptation to rewrite the appointment parser from scratch with a proper grammar, or to experiment with a real natural language parser, though it remained on my personal todo list for the rest of my time at Cozi. We were migrating from C# to Python at that point, and the legacy appointment parser was one of the few remaining pieces that prevented us from shutting down the .NET servers.</p> <p>Instead, I changed the appointment parser code so that it didn't attempt to match the entire 10KB monster in one go. I looped through each of the 32 top-level disjunctions, manually performing the alternation. If any one of those matched, then I had what I needed. Reducing the regexes to a few hundred characters each tamed the combinatorial explosion of backtracking state.</p> <p>Regexes definitely have a place, but do not try to implement a full grammar as a single regular expression.</p> Weirdest Birthday Ever tag:www.georgevreilly.com,2020-03-15:/blog/2020/03/15/WeirdestBirthdayEver.html 2020-03-15T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <p>When I said that Emma and I would be spending <a class="reference external" href="https://www.georgevreilly.com/blog/2019/11/22/Dublin2020.html">2020 in Dublin</a>, I could not possibly have anticipated what would be happening in Seattle while we were gone.</p> <p>Today is my 55th birthday and it's the weirdest birthday ever, in what must be the weirdest week that most of us have lived through. (So far.)</p> <p>COVID-19 is all that anyone can talk about: where it's spreading, how it's being handled, what comes next.</p> <p>I started working from home on Tuesday, March 10th. Emma's general health and immune system are not good. My parents, who live nearby, are now both 80 years old and neither is in great health. It seemed prudent to minimize my risk of passing something on to any of them. Since then, Stripe has closed most offices, as have many other companies.</p> <p>Ireland has closed schools, banned large gatherings, and is generally trying not to become like Italy. St Patrick's Day parades are cancelled. Most pubs have not yet closed and I wish they would, since they are now a public health risk.</p> <p>If Ireland institutes a full lockdown, we'll move in with my parents for the duration—assuming that none of us are showing signs of COVID-19.</p> <p>Seattle has been ground zero for coronavirus in the US. All of the initial deaths were there. Some hospitals are already overwhelmed, and I'm sure others will be as the number of cases rises exponentially. Governor Inslee and other state and city leaders have been doing a good job of managing the crisis.</p> <p>I wish I could say the same about the United States as a whole. It was <em>always</em> obvious that Trump was wholly unfit to be president, but it's never been more clear over the past month. I fear for my adopted country.</p> <p>Anyway, we're heading off to my parents soon, for a subdued birthday celebration.</p> Dublin for 2020 tag:www.georgevreilly.com,2019-11-22:/blog/2019/11/22/Dublin2020.html 2019-11-22T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <a class="reference external image-reference" href="https://www.irishcalendars.ie/products/dublin-calendar"><img alt="Dublin Calendar 2020" src="https://www.georgevreilly.com/content/binary/dublin-calendar-2020.jpg"/></a> <p>I left in the Eighties; I'm going back in the Twenties.</p> <p>I am transferring to a Dublin-based team at <a class="reference external" href="https://stripe.com/">Stripe</a> for a one-year rotation. Emma and I will be moving to Dublin just before Christmas. Emma has never lived in Ireland and I haven't lived there since January 1989. After 30 years in the US, I'm about to spend a year in my hometown.</p> <p>I grew up in Dublin, earned a Bachelor's degree in Computer Science at Trinity College Dublin in 1987, and moved to the US in 1989 to get a Master's degree in Comp Sci at Brown University in Providence, RI. Microsoft moved me to Seattle, WA in 1992, where I've lived ever since. Between 1992 and 2005, I worked at Microsoft three times for a total of ten years. I joined Stripe in Seattle in mid-2018, after eleven years at two startups, Cozi and Cookbrite/MetaBrite.</p> <p>Emma and I met in 1997, and married and bought a house in 2000. We are now frantically trying to get that house into shape to rent out during our year's absence. We've been working all-out for several weeks and now there's only four weeks to go. She carried the burden by herself last week, as I spent the week in Dublin, meeting my new team. I had done so much painting of interior rooms before I went to Dublin two weeks ago, that the fingerprint readers on my laptop and phones were rejecting my fingerprint. My fingerprint works again, but there's still a little painting to be done.</p> <p>We're excited and nervous. It's not easy to pick up and move after spending decades in one city. We have plenty of storage in our Seattle house and had little impetus—until now—to shed stuff. And there's lots and lots of stuff. Thousands upon thousands of books, the impedimenta of various hobbies, and twenty years of odds and ends. We've purged a lot but the end is not yet in sight. I'll be spending the second week of December in the San Jose and San Francisco, so I'll need to spend Thanksgiving working on the house.</p> <p>Stripe has been great in providing relocation assistance to us, providing services in both Seattle and Dublin. I lined up an apartment in Dublin the day before I returned. We'll be living at the top of a Georgian house on Adelaide Road. It's not too small, but it'll feel cramped compared to our Seattle house.</p> <p>A few weeks ago, I moved from an infrastructure team in the Stripe Seattle office to a new security team (Anti-Abuse) in the Stripe Dublin office. Not only is the team new, all of the team members are new to Stripe: most joined in August and the longest-tenured joined in March. They've already accomplished quite a bit, but they've left a few things for me to do.</p> <p>My family are excited too. I have two siblings living in Ireland, but they're both over two hours' drive from my parents' home in Dublin. My sister's in rural Cork and one of my brothers is in rural Mayo. We're all going down to Cork for a few days at Christmas.</p> <p>Ireland, here we come!</p> A Use for Octal: Calculating Modulo 36 from Modulo 9 tag:www.georgevreilly.com,2019-09-15:/blog/2019/09/15/use-for-octal.html 2019-09-15T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <!-- --> <blockquote> (I posted an <a class="reference external" href="https://weblogs.asp.net/george_v_reilly/284388">earlier version</a> of this in December 2004 on my old technical blog. A discussion at work last week about 36-bit computers at the <a class="reference external" href="https://livingcomputers.org/">Living Computers Museum</a> prompted me to write an updated post with improved explanations and much better typography.)</blockquote> <p>I've been programming in C since 1985 and C++ since 1991, but I've never found a use for <a class="reference external" href="https://en.wikipedia.org/wiki/Octal">octal</a> representation until [2004], aside from the permissions argument for <a class="reference external" href="http://en.wikipedia.org/wiki/Chmod">chmod</a>. Octal has always seemed as vestigial as a human appendix, a leftover from the early days of computers, when <a class="reference external" href="https://en.wikipedia.org/wiki/Word_(computer_architecture)">word sizes</a> were often a multiple of three: 6-, 12-, 24-, or 36-bits wide. All modern computers use word sizes that are powers of two—16-, 32-, or 64-bits wide—with 8-bit bytes, so octal is less useful than hex, which evenly subdivides bytes and words. I've done a lot of bit twiddling and hexadecimal has always been indispensable, while octal has remained a curiosity.</p> <p>The other day [in 2004], a mathematician friend described to me a problem that he had solved at a previous company. They were designing hardware that emulated some old <a class="reference external" href="https://retrocomputing.stackexchange.com/questions/11801/what-was-the-rationale-behind-36-bit-computer-architectures">36-bit computers</a>. For backward compatibility, the various shift instructions had to accept an arbitrarily large shift count, <span class="formula"><i>k</i></span>, and shift left or right by <span class="formula">(<i>k</i><span class="textrm"> mod </span>36)</span>. Now, divisions are not cheap to implement in hardware, so they needed to come up with an alternate approach to calculate the modulus.</p> <p>My friend tried to do something with the factors of 36: <span class="formula">4 × 9</span>. Four and nine are <a class="reference external" href="https://artofproblemsolving.com/wiki/index.php/Relatively_prime">relatively prime</a>: they have no common factors other than one. By the <a class="reference external" href="https://medium.com/@astartekraus/the-chinese-remainder-theorem-ea110f48248c">Chinese Remainder Theorem</a> therefore, the combination of <span class="formula"><i>k</i><span class="textrm"> mod </span>4</span> and <span class="formula"><i>k</i><span class="textrm"> mod </span>9</span> is enough to uniquely determine <span class="formula"><i>k</i><span class="textrm"> mod </span>36</span>. By inspection, this is true for the following table of “residues”. All the integers in the range <span class="formula">[0, 36)</span> appear exactly once.</p> <table border="1" class="docutils"> <colgroup> <col width="18%"/> <col width="9%"/> <col width="9%"/> <col width="9%"/> <col width="9%"/> <col width="9%"/> <col width="9%"/> <col width="9%"/> <col width="9%"/> <col width="9%"/> </colgroup> <thead valign="bottom"> <tr><th class="head">4 \ 9</th> <th class="head">0</th> <th class="head">1</th> <th class="head">2</th> <th class="head">3</th> <th class="head">4</th> <th class="head">5</th> <th class="head">6</th> <th class="head">7</th> <th class="head">8</th> </tr> </thead> <tbody valign="top"> <tr><td>0</td> <td>0</td> <td>28</td> <td>20</td> <td>12</td> <td>4</td> <td>32</td> <td>24</td> <td>16</td> <td>8</td> </tr> <tr><td>1</td> <td>9</td> <td>1</td> <td>29</td> <td>21</td> <td>13</td> <td>5</td> <td>33</td> <td>25</td> <td>17</td> </tr> <tr><td>2</td> <td>18</td> <td>10</td> <td>2</td> <td>30</td> <td>22</td> <td>14</td> <td>6</td> <td>34</td> <td>26</td> </tr> <tr><td>3</td> <td>27</td> <td>19</td> <td>11</td> <td>3</td> <td>31</td> <td>23</td> <td>15</td> <td>7</td> <td>35</td> </tr> </tbody> </table> <p>Calculating <span class="formula"><i>k</i><span class="textrm"> mod </span>4</span> is easy in hardware: it's the two least-significant bits.</p> <p>How to calculate <span class="formula"><i>k</i><span class="textrm"> mod </span>9</span> in hardware is not so obvious.</p> <div class="section" id="shifting-and-masking"> <h3>Shifting and Masking</h3> <p>Several programming languages now provide a <tt class="docutils literal">0b</tt> prefix for binary literals to go along with the <tt class="docutils literal">0x</tt> prefix for hex literals and the <tt class="docutils literal">0o</tt> prefix for octal literals. (Older languages use a <tt class="docutils literal">0</tt> prefix for octal and have no <tt class="docutils literal">0b</tt> prefix.) See the discussion in <a class="reference external" href="https://github.com/golang/proposal/blob/master/design/19308-number-literals.md">Go number literals</a> for more detail on <tt class="docutils literal">0b</tt>, including a list of languages that now support this notation.</p> <p><span class="formula">2<sup><i>n</i></sup></span>, written in binary, looks like <tt class="docutils literal">1</tt> followed by <span class="formula"><i>n</i></span> <tt class="docutils literal">0</tt>s. For example, <span class="formula">2<sup>3</sup> = 1000<sub>2</sub></span>. In C-like languages, <span class="formula">2<sup><i>n</i></sup></span> can be written as <tt class="docutils literal">1 &lt;&lt; n</tt>.</p> <p>Similarly, <span class="formula">2<sup><i>n</i></sup> − 1</span>, <tt class="docutils literal">(1 &lt;&lt; n) - 1</tt>, written in binary, looks like <span class="formula"><i>n</i></span> <tt class="docutils literal">1</tt>s. For example, <span class="formula">2<sup>5</sup> − 1 = 31<sub>10</sub> = 11111<sub>2</sub></span>.</p> <p>We can <strong>multiply</strong> an unsigned integer, <tt class="docutils literal">u</tt>, by <span class="formula">2<sup><i>n</i></sup></span> by <strong>shifting</strong> <tt class="docutils literal">u</tt> <strong>left</strong> by <span class="formula"><i>n</i></span> bits, <tt class="docutils literal">u &lt;&lt; n</tt>, introducing <span class="formula"><i>n</i></span>&nbsp;zeroes as the low-order bits. For example, using 8-bit numbers without loss of generality, written as modern Go/Rust number literals:</p> <pre class="code rust literal-block"> <span class="mb">0b_0001_0101</span><span class="w"> </span><span class="o">&lt;&lt;</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mb">0b_1010_1000</span> </pre> <p>Similarly, we can <strong>divide</strong> by <span class="formula">2<sup><i>n</i></sup></span> by <strong>shifting</strong> <tt class="docutils literal">u</tt> <strong>right</strong> by <span class="formula"><i>n</i></span> bits, <tt class="docutils literal">u &gt;&gt;&gt; n</tt>, which drops the <span class="formula"><i>n</i></span> low-order bits and introduces <span class="formula"><i>n</i></span> zeroes as the high-order bits.</p> <pre class="code rust literal-block"> <span class="mb">0b_0101_0110</span><span class="w"> </span><span class="o">&gt;&gt;&gt;</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mb">0b_0000_1010</span> </pre> <p>A sign-extending or arithmetic right shift introduces <span class="formula"><i>n</i></span> copies of the sign bit as the high-order bits. In some languages, such as Java and JavaScript, <tt class="docutils literal">&gt;&gt;</tt>&nbsp;means an arithmetic right shift and <tt class="docutils literal">&gt;&gt;&gt;</tt>&nbsp;means a zero-extending right shift. In other languages, including C, C++, and Go, there is only a <tt class="docutils literal">&gt;&gt;</tt> operator and sign-extension generally depends upon the type of the left operand, <tt class="docutils literal">signed</tt> or <tt class="docutils literal">unsigned</tt>. However, sign extension is not guaranteed in C/C++.</p> <pre class="code rust literal-block"> <span class="mb">0b_0101_0110</span><span class="w"> </span><span class="o">&gt;&gt;</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mb">0b_0001_0101</span><span class="w"> </span><span class="mb">0b_1001_0110</span><span class="w"> </span><span class="o">&gt;&gt;</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mb">0b_1110_0101</span> </pre> <p>Finally, we can find the <strong>remainder</strong> of dividing <tt class="docutils literal">u</tt> by <span class="formula">2<sup><i>n</i></sup></span> by <strong>masking</strong> <tt class="docutils literal">u</tt> with <span class="formula">2<sup><i>n</i></sup> − 1</span>, bitwise-and with <tt class="docutils literal">(1 &lt;&lt; n) - 1</tt>, to extract the <span class="formula"><i>n</i></span> low-order bits:</p> <pre class="code rust literal-block"> <span class="mb">0b_0101_0110</span><span class="w"> </span><span class="o">&amp;</span><span class="w"> </span><span class="mb">0b_0000_0111</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mb">0b_0000_0110</span> </pre> <p>In other words, <tt class="docutils literal">u % 8 == u &amp; 7</tt> and <tt class="docutils literal">u / 8 == u &gt;&gt; 3</tt>.</p> <p>Read MDN's page on <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Bitwise_Operators">bitwise operators</a> for more background.</p> </div> <div class="section" id="casting-out-nines"> <h3>Casting Out Nines</h3> <p>There's an old trick for checking the results of arithmetic operations, known as <a class="reference external" href="http://mathworld.wolfram.com/CastingOutNines.html">casting out nines</a> or <a class="reference external" href="http://web.archive.org/web/20060101140519/http://web.mit.edu/mwpstr/www/dropnine.htm">dropping nines</a>.</p> <p>Add up the decimal digits of each number. Apply the arithmetic operation to these digit sums. They should be congruent, modulo 9.</p> <p>For example, <span class="formula">12, 345 × 8, 765 = 108, 203, 925</span>.</p> <p>To check the multiplication, compute the <a class="reference external" href="https://en.wikipedia.org/wiki/Digit_sum">digit sum</a> of each number, by adding up each decimal digit:</p> <div class="line-block"> <div class="line"><span class="formula">1 + 2 + 3 + 4 + 5 = 15 ≡ 6 (<span class="textrm"> mod </span>9)</span></div> <div class="line">Note: <span class="formula">12, 345<span class="textrm">  mod </span>9 = 6</span></div> </div> <p>and</p> <div class="line-block"> <div class="line"><span class="formula">8 + 7 + 6 + 5 = 26 ≡ 8 (<span class="textrm"> mod </span>9)</span></div> <div class="line">Note: <span class="formula">8, 765<span class="textrm">  mod </span>9 = 8</span></div> </div> <p>Take the first two digit sums, modulo 9, and multiply them:</p> <div class="line-block"> <div class="line"><span class="formula">6 × 8 = 48 ≡ 3 (<span class="textrm"> mod </span>9)</span></div> <div class="line">Note: <span class="formula">15 × 26 = 390 ≡ 3 (<span class="textrm"> mod </span>9)</span></div> </div> <p>Check against the sum of the digits of the product:</p> <div class="line-block"> <div class="line"><span class="formula">1 + 0 + 8 + 2 + 0 + 3 + 9 + 2 + 5 = 30 ≡ 3 (<span class="textrm"> mod </span>9)</span></div> <div class="line">Note: <span class="formula">108, 203, 925<span class="textrm">  mod </span>9 = 3</span></div> </div> <p>This works because <span class="formula">10<sup><i>n</i></sup> ≡ 1 (<span class="textrm"> mod </span>9)</span>.</p> <p>Consider 758:</p> <div class="formula"> 758 = 7 × 100 + 5 × 10 + 8 </div> <div class="formula"> 758 = 7 × (9 + 1) × (9 + 1) + 5 × (9 + 1) + 8 </div> <div class="formula"> 758 = 7 × (9<sup>2</sup> + 2 × 9 + 1) + 5 × (9 + 1) + 8 </div> <p>Dropping the nines from each term leaves the digit sum, which is <em>congruent</em> to the original number modulo nine:</p> <div class="formula"> 7 × 1 + 5 × 1 + 8 = 7 + 5 + 8 = 20 ≡ 2 (<span class="textrm"> mod </span>9) </div> <p>Checking: <span class="formula">758<span class="textrm">  mod </span>9 = 2</span>.</p> <p><a class="reference external" href="https://www.math.nyu.edu/faculty/hausner/congruence.pdf">Congruences</a> have a number of useful properties.</p> </div> <div class="section" id="casting-out-elevens"> <h3>Casting Out Elevens</h3> <p>Let's use 11, instead of 9. Since <span class="formula">10 = 11 − 1</span>, then <span class="formula">10<sup><i>n</i></sup> ≡  − 1<sup><i>n</i></sup> (<span class="textrm">mod </span>11)</span>.</p> <p>Consider 5234:</p> <div class="formula"> 5234 = 5 × 10<sup>3</sup> + 2 × 10<sup>2</sup> + 3 × 10<sup>1</sup> + 4 × 10<sup>0</sup> </div> <div class="formula"> 5234 = 5 × (11 − 1) × (11 − 1) × (11 − 1) + 2 × (11 − 1) × (11 − 1) + 3 × (11 − 1) + 4 </div> <div class="formula"> 5234 = 5 × (11<sup>3</sup> − 3 × 11<sup>2</sup> × 1 + 3 × 11 × 1<sup>2</sup> − 1<sup>3</sup>) + 2 × (11<sup>2</sup> − 2 × 11 × 1 + 1<sup>2</sup>) + 3 × (11 − 1) + 4 </div> <p>Dropping the elevens from each term leaves the alternating digit sum:</p> <div class="formula"> 5 ×  − 1 + 2 × 1 + 3 ×  − 1 + 4 =  − 5 + 2 − 3 + 4 =  − 2 ≡ 9 (<span class="textrm"> mod </span>11) </div> <p>It's more convenient to proceed rightwards from the least significant digit, <span class="formula">4 − 3 + 2 − 5</span>.</p> <p>Checking: <span class="formula">5234<span class="textrm">  mod </span>11 = 9</span>.</p> <p>To cast out elevens, we calculate the <a class="reference external" href="https://en.wikipedia.org/wiki/Alternating_sum">alternating sum</a> <em>from right to left</em>.</p> <p>Casting out elevens catches some <a class="reference external" href="http://mathyear2013.blogspot.com/2013/01/casting-out-elevens.html">transposition errors</a>, unlike casting out nines. For more, see <a class="reference external" href="https://artofproblemsolving.com/wiki/index.php/Divisibility_rules/Rule_for_11_proof">divisibility rule for 11</a> and <a class="reference external" href="https://en.wikipedia.org/wiki/Divisibility_rule#Proof_using_basic_algebra">proof for alternating sum</a>.</p> </div> <div class="section" id="modulo-9"> <h3>Modulo 9</h3> <p>At last, we turn to base 8, octal. Nine bears the same relationship to eight in octal, as eleven does to ten in decimal: <span class="formula">9<sub>10</sub> = 11<sub>8</sub></span>, base plus one, and <span class="formula">8<sup><i>n</i></sup> ≡  − 1<sup><i>n</i></sup> (<span class="textrm">mod </span>9)</span>.</p> <p>We can calculate <span class="formula"><i>k</i><span class="textrm"> mod </span>9</span> in base 8 by alternately adding and subtracting the octal digits, from right to left. For example, <span class="formula">1234<sub>8</sub><span class="textrm">  mod </span>9 = 4 − 3 + 2 − 1 = 2</span>. This gives the right answer.</p> <p>Here's a simple, albeit incomplete, algorithm in Go. We're masking and shifting three bits at a time, which is tantamount to working with the octal representation of <tt class="docutils literal">k</tt>.</p> <pre class="code go literal-block"> <span class="kd">func</span> <span class="nx">Mod9</span><span class="p">(</span><span class="nx">k</span> <span class="kt">uint</span><span class="p">)</span> <span class="kt">uint</span> <span class="p">{</span> <span class="kd">var</span> <span class="nx">m</span> <span class="kt">int</span> <span class="p">=</span> <span class="mi">0</span> <span class="nx">sign</span> <span class="o">:=</span> <span class="o">+</span><span class="mi">1</span> <span class="k">for</span> <span class="nx">t</span> <span class="o">:=</span> <span class="nx">k</span><span class="p">;</span> <span class="mi">0</span> <span class="o">!=</span> <span class="nx">t</span><span class="p">;</span> <span class="nx">t</span> <span class="o">&gt;&gt;=</span> <span class="mi">3</span> <span class="p">{</span> <span class="nx">r</span> <span class="o">:=</span> <span class="nb">int</span><span class="p">(</span><span class="nx">t</span> <span class="o">&amp;</span> <span class="mi">7</span><span class="p">)</span> <span class="nx">m</span> <span class="o">+=</span> <span class="nx">sign</span> <span class="o">*</span> <span class="nx">r</span> <span class="nx">sign</span> <span class="p">=</span> <span class="o">-</span><span class="nx">sign</span> <span class="p">}</span> <span class="k">return</span> <span class="nb">uint</span><span class="p">(</span><span class="nx">m</span><span class="p">)</span> <span class="p">}</span> </pre> <p>What about <span class="formula">617<sub>8</sub></span>?</p> <div class="formula"> 7 − 1 + 6 = 12 ≡ 3 (<span class="textrm"> mod </span>9) </div> <div class="formula"> 617<sub>8</sub><span class="textrm">  mod </span>9 = 3 </div> <p>And <span class="formula">6172<sub>8</sub></span>?</p> <div class="formula"> 2 − 7 + 1 − 6 =  − 10 ≡ 8 (<span class="textrm"> mod </span>9) </div> <div class="formula"> 6172<sub>8</sub><span class="textrm">  mod </span>9 = 8 </div> <p>Almost there!</p> <blockquote> Casting out “octal-elevens” (<span class="formula">11<sub>8</sub> = 9<sub>10</sub></span>) in octal, by an alternating sum of the base-eight digits, computes a small number <em>congruent</em> to the original number number modulo nine.</blockquote> <p>The algorithm above is calculating numbers that are congruent to the correct answer modulo nine, but which may be outside the desired range. If the intermediate sum dips below zero or rises above eight, we have to add nine or subtract nine respectively to keep the running total in the range <span class="formula">[0, 9)</span>.</p> <p>Here's a complete algorithm for Modulo 9 in Go, computing the alternating sum of the octal digits:</p> <pre class="code go literal-block"> <span class="kd">func</span> <span class="nx">Mod9</span><span class="p">(</span><span class="nx">k</span> <span class="kt">uint</span><span class="p">)</span> <span class="kt">uint</span> <span class="p">{</span> <span class="kd">var</span> <span class="nx">m</span> <span class="kt">int</span> <span class="p">=</span> <span class="mi">0</span> <span class="kd">var</span> <span class="nx">negative</span> <span class="kt">bool</span> <span class="p">=</span> <span class="kc">false</span> <span class="k">for</span> <span class="nx">t</span> <span class="o">:=</span> <span class="nx">k</span><span class="p">;</span> <span class="mi">0</span> <span class="o">!=</span> <span class="nx">t</span><span class="p">;</span> <span class="nx">t</span> <span class="o">&gt;&gt;=</span> <span class="mi">3</span> <span class="p">{</span> <span class="nx">r</span> <span class="o">:=</span> <span class="nb">int</span><span class="p">(</span><span class="nx">t</span> <span class="o">&amp;</span> <span class="mi">7</span><span class="p">)</span> <span class="k">if</span> <span class="nx">negative</span> <span class="p">{</span> <span class="nx">m</span> <span class="o">-=</span> <span class="nx">r</span> <span class="k">if</span> <span class="nx">m</span> <span class="p">&lt;</span> <span class="mi">0</span> <span class="p">{</span> <span class="nx">m</span> <span class="o">+=</span> <span class="mi">9</span> <span class="p">}</span> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> <span class="nx">m</span> <span class="o">+=</span> <span class="nx">r</span> <span class="k">if</span> <span class="nx">m</span> <span class="o">&gt;=</span> <span class="mi">9</span> <span class="p">{</span> <span class="nx">m</span> <span class="o">-=</span> <span class="mi">9</span> <span class="p">}</span> <span class="p">}</span> <span class="c1">// assert(0 &lt;= m &amp;&amp; m &lt; 9) </span> <span class="nx">negative</span> <span class="p">=</span> <span class="p">!</span><span class="nx">negative</span> <span class="p">}</span> <span class="k">return</span> <span class="nb">uint</span><span class="p">(</span><span class="nx">m</span><span class="p">)</span> <span class="p">}</span> </pre> <p>Clearly, this algorithm can be implemented in much simpler circuitry than that required to compute a remainder through full-blown division.</p> </div> <div class="section" id="modulo-36"> <h3>Modulo 36</h3> <p>We now have enough to calculate <span class="formula"><i>k</i><span class="textrm"> mod </span>36</span> from <tt class="docutils literal">Mod9</tt> and the Chinese Remainder Theorem:</p> <pre class="code go literal-block"> <span class="kd">func</span> <span class="nx">Mod36</span><span class="p">(</span><span class="nx">k</span> <span class="kt">uint</span><span class="p">)</span> <span class="kt">uint</span> <span class="p">{</span> <span class="nx">Residues</span> <span class="o">:=</span> <span class="p">[</span><span class="mi">4</span><span class="p">][</span><span class="mi">9</span><span class="p">]</span><span class="kt">uint</span><span class="p">{</span> <span class="p">{</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">28</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="mi">24</span><span class="p">,</span> <span class="mi">16</span><span class="p">,</span> <span class="mi">8</span><span class="p">},</span> <span class="p">{</span> <span class="mi">9</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">29</span><span class="p">,</span> <span class="mi">21</span><span class="p">,</span> <span class="mi">13</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">33</span><span class="p">,</span> <span class="mi">25</span><span class="p">,</span> <span class="mi">17</span><span class="p">},</span> <span class="p">{</span><span class="mi">18</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">30</span><span class="p">,</span> <span class="mi">22</span><span class="p">,</span> <span class="mi">14</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">34</span><span class="p">,</span> <span class="mi">26</span><span class="p">},</span> <span class="p">{</span><span class="mi">27</span><span class="p">,</span> <span class="mi">19</span><span class="p">,</span> <span class="mi">11</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">31</span><span class="p">,</span> <span class="mi">23</span><span class="p">,</span> <span class="mi">15</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">35</span><span class="p">},</span> <span class="p">}</span> <span class="k">return</span> <span class="nx">Residues</span><span class="p">[</span><span class="nx">k</span> <span class="o">&amp;</span> <span class="mi">3</span><span class="p">][</span><span class="nx">Mod9</span><span class="p">(</span><span class="nx">k</span><span class="p">)]</span> <span class="p">}</span> </pre> <p>My friend says that he later learned that similar tricks were used in classic 36-bit hardware.</p> <p>I looked everywhere I could think of to see if I could find this algorithm to calculate modulo 9 described. I found something that hinted at it in Knuth's <a class="reference external" href="http://www-cs-faculty.stanford.edu/~knuth/taocp.html">Seminumerical Algorithms</a>, §4.4.C, discussing <a class="reference external" href="https://books.google.com/books?id=Zu-HAwAAQBAJ&amp;pg=PT532&amp;lpg=PT532&amp;dq=octal+cast+out+nines+modulo+36&amp;source=bl&amp;ots=9nglVlTuaU&amp;sig=ACfU3U0_RR51okwrvfY3WwC0xBudfLGhuw&amp;hl=en&amp;sa=X&amp;ved=2ahUKEwih44eUxc_kAhVVo54KHcgKDeEQ6AEwDXoECAgQAg#v=onepage&amp;q=octal%20cast%20out%20nines%20modulo%2036&amp;f=false">converting octal integers to decimal</a> by hand, where he mentions using casting out nines in octal and in decimal to check the result. There was no mention of it in Warren's marvelous <a class="reference external" href="http://www.informit.com/articles/article.asp?p=28678">Hacker's Delight</a> or in <a class="reference external" href="http://home.pipeline.com/~hbaker1/hakmem/hakmem.html">HAKMEM</a>.</p> <p>I tried to come up with an analytic way to calculate the elements of the <span class="formula">9<i>x</i>4</span> table. The best that I found is <span class="formula">(72 − 8 × (<i>k</i><span class="textrm"> mod </span>9) + 9 × (<i>k</i><span class="textrm"> mod </span>4))<span class="textrm">  mod </span>36</span>! The inner expression yields a number in the range <span class="formula">[0, 99]</span>, which can be reduced to <span class="formula">[0, 36)</span> by subtracting 36 at most twice. From <a class="reference external" href="http://www-cs-faculty.stanford.edu/~knuth/gkp.html">Concrete Mathematics</a>, mod 36 can be derived from mod 4 and mod 9 by looking at the [0][1] and [1][0] elements of the table: <span class="formula">(9 × (<i>k</i><span class="textrm"> mod </span>4) + 28 × (<i>k</i><span class="textrm"> mod </span>9))<span class="textrm">  mod </span>36</span>. It works, but it's even worse. A table lookup is clearly more efficient.</p> <p>Most, if not all, of the computer architectures designed in the last forty years use a word size that is a power of two. Useful relationships like shifting and masking are one big reason why non-power-of-two word sizes have gone out of fashion.</p> <p>Another big reason is the success of C and Unix, which have a bias towards 8-bit bytes. <a class="reference external" href="http://www.parashift.com/c++-faq-lite/intrinsic-types.html">C doesn't require 8-bit bytes</a>, but there's a lot of software which tacitly assumes that <tt class="docutils literal">char</tt> has exactly 8 bits.</p> <p>On systems with 9-bit bytes, like the 36-bit computers, octal is useful, since a 9-bit byte can hold all values up to <span class="formula">777<sub>8</sub></span> and the word size is a multiple of three.</p> <p>And there you have it: an unexpected use for octal notation. It's not exactly an important use, but then 36-bit computers aren't exactly important any more either.</p> </div> Decrypting Blackbox secrets at build time with Paperkey tag:www.georgevreilly.com,2019-09-02:/blog/2019/09/02/gpg-blackbox-paperkey.html 2019-09-02T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <blockquote> “Security is 1% technology plus 99% following the procedures correctly” — Tom Limoncelli</blockquote> <p>Having dealt with GPG last week at work, I remembered that I had intended to write a blog post about how we used <a class="reference external" href="https://gnupg.org/">GPG</a>, <a class="reference external" href="https://github.com/StackExchange/blackbox">Blackbox</a>, and <a class="reference external" href="http://www.jabberwocky.com/software/paperkey/">Paperkey</a> to store secrets in Git at my <a class="reference external" href="https://www.georgevreilly.com/blog/2018/12/31/2018-review.html">previous job</a>.</p> <p>We used Blackbox to manage secrets that were needed during development, build, deployment, and runtime. These secrets included AWS credentials, Docker registry credentials, our private PyPI credentials, database credentials, and certificates. We wanted these secrets to be under version control, but also to be secure.</p> <p>For example, we had a <tt class="docutils literal">credentials.sh</tt> that exported environment variables, which was managed by Blackbox:</p> <pre class="code bash literal-block"> <span class="c1"># Save current value of xtrace option from $-; disable echoing of executed commands </span><span class="o">{</span> <span class="k">if</span> <span class="nb">echo</span> <span class="nv">$-</span> <span class="p">|</span> grep -q <span class="s2">&quot;x&quot;</span><span class="p">;</span> <span class="k">then</span> <span class="nv">XT</span><span class="o">=</span><span class="s2">&quot;-x&quot;</span><span class="p">;</span> <span class="k">else</span> <span class="nv">XT</span><span class="o">=</span><span class="s2">&quot;+x&quot;</span><span class="p">;</span> <span class="k">fi</span><span class="p">;</span> <span class="nb">set</span> +x<span class="p">;</span> <span class="o">}</span> <span class="m">2</span>&gt;/dev/null <span class="nb">export</span> <span class="nv">AWS_ACCESS_KEY_ID</span><span class="o">=</span><span class="s1">'...'</span> <span class="nb">export</span> <span class="nv">AWS_SECRET_ACCESS_KEY</span><span class="o">=</span><span class="s1">'...'</span> <span class="nb">export</span> <span class="nv">PYPI_USER</span><span class="o">=</span><span class="s1">'build'</span> <span class="nb">export</span> <span class="nv">PYPI_PASSWD</span><span class="o">=</span><span class="s1">'...'</span> <span class="nb">export</span> <span class="nv">PIP_INDEX_URL</span><span class="o">=</span><span class="s2">&quot;https://</span><span class="nv">$PYPI_USER</span><span class="s2">:</span><span class="nv">$PYPI_PASSWD</span><span class="s2">&#64;pypi.example.com/pypi/&quot;</span> <span class="c1"># Restore previous value of xtrace option </span><span class="nb">set</span> <span class="nv">$XT</span> </pre> <p>The <tt class="docutils literal">XT</tt> prologue ensures that even if this script is <tt class="docutils literal">source</tt>’d with <a class="reference external" href="https://renenyffenegger.ch/notes/Linux/shell/bash/built-in/set/x">set -x</a> (debug tracing) enabled, that executing this script will not leak secrets into build logs. The epilogue turns the <tt class="docutils literal">xtrace</tt> option back on again if it was on at the start.</p> <p>We used <a class="reference external" href="https://www.lastpass.com/">LastPass</a> to manage personal credentials that were needed in a browser, but it wasn't suitable for automated use in CI.</p> <div class="section" id="how-blackbox-works"> <h3>How Blackbox Works</h3> <p>Blackbox builds on top of <a class="reference external" href="https://gnupg.org/">GNU Privacy Guard</a> (aka GnuPG aka GPG) to automate the secure management of a set of files containing secrets that are “encrypted at rest” and stored in a Version Control System (VCS), such as Git. These registered files are owned collectively by a set of administrators, each of whom has their own separate keypair (a public key and a private key) stored in their own keyrings. The administrators' public keys are also present in Blackbox's keyring, which is stored in the VCS. Using Blackbox's commands, any administrator can decrypt a file containing secrets, update the secrets in the file, encrypt the updated secrets file, and commit that encrypted file into the VCS. Administrators can be removed from a Blackbox installation, after which they will not be able to decrypt the updated secrets files<a class="footnote-reference" href="https://www.georgevreilly.com/blog/2019/09/02/gpg-blackbox-paperkey.html/#revocation" id="id1">[1]</a>.</p> <p>How does Blackbox encrypt a file so that any administrator can decrypt it? It uses GPG to encrypt the file for multiple recipients, say, <a class="reference external" href="https://en.wikipedia.org/wiki/Alice_and_Bob#Cast_of_characters">Alice, Bob, and Carol</a>.</p> <p>When GPG encrypts a file, it:</p> <ul class="simple"> <li>creates a random <em>session key</em> for <a class="reference external" href="https://www.ssl2buy.com/wiki/symmetric-vs-asymmetric-encryption-what-are-differences">symmetric encryption</a></li> <li>writes a header for each recipient, containing:<ul> <li>the ID of the recipient's public key</li> <li>the result of encrypting the session key with the recipient's public key</li> </ul> </li> <li>possibly signs the data</li> <li>compresses the (signed) data</li> <li>encrypts the compressed data with the session key</li> <li>writes the encrypted, compressed data</li> </ul> <p>Only the recipients have the private keys (in theory, at least). Therefore, only a recipient can decrypt the encrypted file.</p> <p>To decrypt the file for a recipient, GPG:</p> <ul class="simple"> <li>finds the encrypted session key packet whose keyID matches the recipient's public key</li> <li>decrypts the session key using the recipient's private key</li> <li>decrypts the encrypted, compressed data using the session key</li> <li>decompresses the decrypted data</li> <li>verifies the signature, if present</li> <li>writes the cleartext</li> </ul> <p>This is a hybrid scheme. Symmetric encryption is a lot faster than public key/private key asymmetric encryption, so it's used to encrypt the actual data. Furthermore, if the data were entirely encrypted with a recipient's public key, then encrypting for <em>N</em> recipients would mean that the size of the result would be proportional to the number of recipients times the length of the original data. With the hybrid scheme, the header grows a <a class="reference external" href="https://security.stackexchange.com/questions/8245/gpg-file-size-with-multiple-recipients">few hundred bytes</a> for each recipient but the data is encrypted only once, with faster encryption.</p> <p>Blackbox encrypts a registered file with all of the administrators as the recipients, so any administrator can decrypt the file.</p> <div class="figure"> <a class="reference external image-reference" href="http://www.cse.tkk.fi/fi/opinnot/T-110.5240/2009/luennot-files/Lecture%202.pdf"><img alt="Typical PGP Message" src="https://www.georgevreilly.com/content/binary/typical-pgp-message.jpg"/></a> <p class="caption">Typical PGP Message</p> <div class="legend"> (Figure from <a class="reference external" href="http://www.cse.tkk.fi/fi/opinnot/T-110.5240/2009/luennot-files/Lecture%202.pdf">Network Security: Email Security, PKI</a>, Tuomas Aura)</div> </div> <p>You can use <tt class="docutils literal">gpg <span class="pre">--list-packets</span></tt> to dump the contents of any GPG message. <a class="reference external" href="https://begriffs.com/posts/2016-11-05-advanced-intro-gnupg.html">An Advanced Intro to GnuPG</a> dives into the message format in more detail.</p> <p>Going back to my original example, <tt class="docutils literal">credentials.sh</tt> is a file registered in <tt class="docutils literal"><span class="pre">blackbox-files.txt</span></tt>. This file should never be committed to the VCS—add it to <a class="reference external" href="https://git-scm.com/docs/gitignore">gitignore</a> to prevent accidentally committing it. Instead, <tt class="docutils literal">credentials.sh.gpg</tt> is committed. Since the latter is a binary file, comparing two versions in cleartext is tricky.</p> <table class="docutils footnote" frame="void" id="revocation" rules="none"> <colgroup><col class="label"/><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="https://www.georgevreilly.com/blog/2019/09/02/gpg-blackbox-paperkey.html/#id1">[1]</a></td><td>If they have a snapshot of the VCS before their access was revoked, they will still be able to decrypt the secrets as they were then. In principle, you should be changing passwords and certificates every time someone's access is revoked.</td></tr> </tbody> </table> </div> <div class="section" id="private-keys-and-paperkey"> <h3>Private Keys and Paperkey</h3> <p>Administrators can encrypt and decrypt Blackbox'd files because they have their private key on a local keyring.</p> <p>Getting a private key onto other hosts can be tricky. We developed this technique when we were using Atlassian's hosted Bamboo CI service. We later used it with hosted Jenkins at Cloudbees. Because we were using a hosted Continuous Integration (CI) service, we had limited control over what we could install. If I remember correctly, Bamboo had support for secret environment variables, but did not provide a way to store a keyring file. There was also a limit on the length of the environment variables, I believe.</p> <p>We were able to get past this by using <a class="reference external" href="http://www.jabberwocky.com/software/paperkey/">Paperkey</a> to (de)serialize the secret key. Paperkey can extract just the secret part of a secret key: ‘Due to metadata and redundancy, OpenPGP secret keys are significantly larger than just the &quot;secret bits&quot;. In fact, the secret key contains a complete copy of the public key.’</p> <p>We created a keypair for the CI on a secure host, serialized the secret with Paperkey, and pasted the secret into the CI's UI to become an environment variable. At build time, we used Paperkey on the CI box to deserialize the secret key from the environment variable, before decrypting the secrets needed with Blackbox.</p> <p>To create the CI keypair, follow the portion of the <a class="reference external" href="https://github.com/StackExchange/blackbox#set-up-automated-users-or-role-accounts">Blackbox &quot;role accounts&quot; instructions</a> that create a sub-key with no password for <tt class="docutils literal">ci&#64;example.com</tt>.</p> <p>Then, serialize the public key and the secret with Paperkey:</p> <pre class="code bash literal-block"> <span class="nb">cd</span> /tmp/NEWMASTER gpg --homedir . --export ci&#64;example.com <span class="se">\ </span> <span class="p">|</span> base64 &gt; public_key.txt gpg --homedir . --export-secret-keys ci&#64;example.com <span class="se">\ </span> <span class="p">|</span> paperkey --output-type<span class="o">=</span>raw <span class="se">\ </span> <span class="p">|</span> base64 &gt; secret.txt </pre> <p>Copy and paste the contents of <tt class="docutils literal">public_key.txt</tt> to the <tt class="docutils literal">GPG_PUBLIC_KEY</tt> environment variable in the CI. Similarly, copy <tt class="docutils literal">secret.txt</tt> to <tt class="docutils literal">GPG_SECRET</tt>.</p> <p>Securely delete everything in <tt class="docutils literal">/tmp/NEWMASTER</tt>.</p> <p>We used a script like this on the CI to reconstitute the keypair and to decrypt the other secrets from Blackbox:</p> <pre class="code bash literal-block"> <span class="ch">#!/usr/bin/env bash </span> <span class="c1"># Run during a CI build to decrypt all Blackbox-encrypted files in this repo. # Can also be used interactively. </span> <span class="nb">set</span> -ex <span class="c1"># Root of Git working tree </span><span class="nv">SERVICES_DIR</span><span class="o">=</span><span class="s2">&quot;</span><span class="k">$(</span><span class="nb">cd</span> <span class="s2">&quot;</span><span class="k">$(</span>dirname <span class="s2">&quot;</span><span class="nv">$0</span><span class="s2">&quot;</span><span class="k">)</span><span class="s2">/..&quot;</span><span class="p">;</span> <span class="nb">pwd</span><span class="k">)</span><span class="s2">&quot;</span> <span class="k">if</span> <span class="o">[</span> <span class="s2">&quot;</span><span class="nv">$CI_BUILD</span><span class="s2">&quot;</span> <span class="o">=</span> <span class="s2">&quot;true&quot;</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> <span class="nv">GPG_HOMEDIR</span><span class="o">=</span><span class="s2">&quot;</span><span class="k">$(</span>mktemp -d -t gnupg.XXX<span class="k">)</span><span class="s2">&quot;</span> <span class="nv">SECRET_KEY_FILE</span><span class="o">=</span><span class="s2">&quot;</span><span class="nv">$GPG_HOMEDIR</span><span class="s2">/secret.key&quot;</span> <span class="nv">PUBLIC_KEY_FILE</span><span class="o">=</span><span class="s2">&quot;</span><span class="nv">$GPG_HOMEDIR</span><span class="s2">/public_key.gpg&quot;</span> <span class="c1"># this variable is how you can customize how GPG is used in Blackbox </span> <span class="nv">GPG</span><span class="o">=</span><span class="s2">&quot;gpg --homedir=</span><span class="nv">$GPG_HOMEDIR</span><span class="s2">&quot;</span> <span class="c1"># Remove secrets from filesystem on exit. </span> <span class="k">function</span> clean_up <span class="o">{</span> <span class="c1"># TODO: use shred, if available </span> rm -rf <span class="s2">&quot;</span><span class="nv">$GPG_HOMEDIR</span><span class="s2">&quot;</span> <span class="o">}</span> <span class="nb">trap</span> clean_up EXIT<span class="p">;</span> <span class="nb">echo</span> <span class="s2">&quot;Unpacking keys; exiting debug mode to redact...&quot;</span> <span class="nb">set</span> +x <span class="k">if</span> <span class="o">[</span> -z <span class="s2">&quot;</span><span class="nv">$GPG_PUBLIC_KEY</span><span class="s2">&quot;</span> -o -z <span class="s2">&quot;</span><span class="nv">$GPG_SECRET</span><span class="s2">&quot;</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> <span class="nb">echo</span> <span class="s2">&quot;Missing CI credential env vars for GPG key and secret&quot;</span> <span class="nb">exit</span> <span class="m">1</span> <span class="k">fi</span> <span class="c1"># unpack public key </span> <span class="nb">echo</span> <span class="s2">&quot;</span><span class="nv">$GPG_PUBLIC_KEY</span><span class="s2">&quot;</span> <span class="p">|</span> base64 --decode &gt; <span class="s2">&quot;</span><span class="nv">$PUBLIC_KEY_FILE</span><span class="s2">&quot;</span> <span class="c1"># unpack secret key </span> <span class="nb">echo</span> <span class="s2">&quot;</span><span class="nv">$GPG_SECRET</span><span class="s2">&quot;</span> <span class="p">|</span> base64 --decode &gt; <span class="s2">&quot;</span><span class="nv">$SECRET_KEY_FILE</span><span class="s2">&quot;</span> <span class="nb">echo</span> <span class="s2">&quot;Secrets unpacked...&quot;</span> <span class="nb">set</span> -x <span class="c1"># reconstitute and import full key into $GPG_HOMEDIR </span> paperkey --pubring <span class="s2">&quot;</span><span class="nv">$PUBLIC_KEY_FILE</span><span class="s2">&quot;</span> --secrets <span class="s2">&quot;</span><span class="nv">$SECRET_KEY_FILE</span><span class="s2">&quot;</span> <span class="se">\ </span> <span class="p">|</span> <span class="nv">$GPG</span> --import <span class="c1"># TODO: vendor Blackbox </span> <span class="nv">BLACKBOX_DIR</span><span class="o">=</span><span class="s2">&quot;</span><span class="k">$(</span>mktemp -d -t blackbox.XXX<span class="k">)</span><span class="s2">&quot;</span> <span class="nv">BLACKBOX_BIN</span><span class="o">=</span><span class="nv">$BLACKBOX_DIR</span>/bin <span class="c1"># Shallow clone of Blackbox with most-recent commit only </span> git clone --depth <span class="m">1</span> https://github.com/StackExchange/blackbox.git <span class="nv">$BLACKBOX_DIR</span> <span class="k">else</span> <span class="c1"># So that you only have to enter your password once when running interactively </span> <span class="nb">eval</span> <span class="s2">&quot;</span><span class="k">$(</span>gpg-agent --daemon<span class="k">)</span><span class="s2">&quot;</span> <span class="c1"># No custom GPG_HOMEDIR needed </span> <span class="nv">GPG</span><span class="o">=</span><span class="s2">&quot;gpg&quot;</span> <span class="nv">BLACKBOX_POSTDEPLOY</span><span class="o">=</span><span class="s2">&quot;</span><span class="k">$(</span><span class="nb">command</span> -v blackbox_postdeploy<span class="k">)</span><span class="s2">&quot;</span> <span class="o">||</span> <span class="nv">ret</span><span class="o">=</span><span class="nv">$?</span> <span class="k">if</span> <span class="o">[</span> -n <span class="s2">&quot;</span><span class="nv">$BLACKBOX_POSTDEPLOY</span><span class="s2">&quot;</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> <span class="c1"># Use the Blackbox that's on the path </span> <span class="nv">BLACKBOX_BIN</span><span class="o">=</span><span class="s2">&quot;</span><span class="k">$(</span>dirname <span class="nv">$BLACKBOX_POSTDEPLOY</span><span class="k">)</span><span class="s2">&quot;</span> <span class="k">else</span> <span class="c1"># Assume Blackbox is checked out in a sibling dir to $SERVICES_DIR </span> <span class="nv">BLACKBOX_BIN</span><span class="o">=</span><span class="s2">&quot;</span><span class="k">$(</span><span class="nb">cd</span> <span class="s2">&quot;</span><span class="nv">$SERVICES_DIR</span><span class="s2">/..&quot;</span><span class="p">;</span> <span class="nb">pwd</span><span class="k">)</span><span class="s2">&quot;</span>/blackbox/bin <span class="k">if</span> <span class="o">[</span> ! -f <span class="s2">&quot;</span><span class="nv">$BLACKBOX_BIN</span><span class="s2">/blackbox_postdeploy&quot;</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> <span class="nb">echo</span> <span class="s2">&quot;Can't find Blackbox binaries&quot;</span> <span class="nb">exit</span> <span class="m">1</span> <span class="k">fi</span> <span class="k">fi</span> <span class="k">fi</span> <span class="c1"># decrypt secrets in $SERVICES_DIR using custom GPG_HOMEDIR </span><span class="nv">GPG</span><span class="o">=</span><span class="s2">&quot;</span><span class="nv">$GPG</span><span class="s2">&quot;</span> <span class="nv">$BLACKBOX_BIN</span>/blackbox_postdeploy <span class="c1"># test that decryption worked </span>grep <span class="s1">'congrats!'</span> test_secret.txt </pre> <p>At the end of the build, run <tt class="docutils literal">blackbox_shred_all_files</tt> to destroy any decrypted files.</p> </div> <div class="section" id="more-reading"> <h3>More Reading</h3> <ul class="simple"> <li><a class="reference external" href="https://github.com/StackExchange/blackbox">Blackbox</a></li> <li><a class="reference external" href="http://www.jabberwocky.com/software/paperkey/">Paperkey</a></li> <li><a class="reference external" href="http://www.linux-magazine.com/Online/Features/Protect-your-Documents-with-GPG">Protect your documents with GPG</a></li> <li><a class="reference external" href="https://davesteele.github.io/gpg/2014/09/20/anatomy-of-a-gpg-key/">Anatomy of a GPG Key</a></li> <li><a class="reference external" href="https://alexcabal.com/creating-the-perfect-gpg-keypair">Creating the perfect GPG keypair</a></li> <li><a class="reference external" href="https://www.darkcoding.net/software/how-gpg-works-encrypt/">How GPG works: Encrypt</a></li> <li><a class="reference external" href="https://gist.github.com/chrisroos/1205934">GPG import and export</a></li> <li><a class="reference external" href="https://begriffs.com/posts/2016-11-05-advanced-intro-gnupg.html">An Advanced Intro to GnuPG</a></li> <li><a class="reference external" href="http://www.cse.tkk.fi/fi/opinnot/T-110.5240/2009/luennot-files/Lecture%202.pdf">Network Security: Email Security, PKI</a></li> </ul> </div> Review: Fire and Blood tag:www.georgevreilly.com,2019-01-03:/blog/2019/01/03/ReviewFireAndBlood.html 2019-01-03T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <a class="reference external image-reference" href="https://www.amazon.com/dp/152479628X/?tag=georgvreill-20"><img alt="Fire and Blood" class="right-float" src="https://images-na.ssl-images-amazon.com/images/P/152479628X.01.MZZZZZZZ.jpg"/></a> <div class="line-block"> <div class="line">Title: Fire and Blood</div> <div class="line">Author: George R.R. Martin</div> <div class="line">Rating: ★ ★ ★ ½</div> <div class="line">Publisher: Bantam</div> <div class="line">Copyright: 2018</div> <div class="line">ISBN: <a class="reference external" href="https://www.amazon.com/dp/152479628X/?tag=georgvreill-20">978-1524796280</a></div> <div class="line">Pages: 736</div> <div class="line">Keywords: fantasy</div> <div class="line">Reading period: 28 December, 2018–1 January, 2019</div> </div> <p>I've been waiting longer than most for George R.R. Martin to finish the <em>A Song of Fire and Ice</em> series: I read the first book when it was <a class="reference external" href="https://www.georgevreilly.com/blog/2005/11/20/AFeastForCrows.html">newly published in paperback</a> in 1997. <em>Fire and Blood</em> is a new addition to the series, but it is a prequel and does not advance the plot at all. This book is a history of the first half of the three hundred–year reign of the Targaryen dynasty, the dragon riders who conquered Westeros with their firebreathing dragons. The Game of Thrones series takes place fifteen years after the last Targaryen king was deposed.</p> <p>Instead of a standard novel told from multiple viewpoints, <em>Fire and Blood</em> purports to be a history assembled by a scholar from a variety of sources, decades after the events recounted. As such, it is much drier than the novels and less enjoyable. It fleshes out characters and events oft alluded to in the novels. The Targaryens are a nasty, backbiting lot. They are literally inbred, frequently marrying brother to sister. Different factions feud over succession; civil wars are fought; alliances are made and lives are broken.</p> <p>This book was once dubbed the <a class="reference external" href="https://en.wikipedia.org/wiki/Fire_%26_Blood_(book)">GRRMarillion</a>; I would recommend it mainly to completists.</p> Review: Watership Down (miniseries) tag:www.georgevreilly.com,2019-01-02:/blog/2019/01/02/ReviewWatershipDownMiniseries.html 2019-01-02T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <a class="reference external image-reference" href="https://en.wikipedia.org/wiki/Watership_Down_(miniseries)"><img alt="Watership Down" src="https://occ-0-1782-1007.1.nflxso.net/art/ec6ad/8da2469ea8d6ed01ce41c61a0dae6a7c26fec6ad.jpg"/></a> <div class="line-block"> <div class="line">Title: Watership Down (miniseries)</div> <div class="line">Director: Noam Murro</div> <div class="line">Rating: ★ ★ ★ ★</div> <div class="line">Released: 2018</div> <div class="line">Keywords: animation</div> <div class="line">Country: UK</div> <div class="line">Watched: 30 December, 2018–1 January, 2019</div> </div> <p>Two years ago, just after the death of Richard Adams, I reread the <a class="reference external" href="https://www.georgevreilly.com/blog/2017/01/01/ReviewWatershipDown.html">book</a> of <em>Watership Down</em> for the first time in many years, having originally discovered it when it was new in the mid-1970s. There's a beautiful new adaptation, an animated <a class="reference external" href="https://en.wikipedia.org/wiki/Watership_Down_(miniseries)">miniseries</a> made by the BBC and Netflix.</p> <p>This adaptation is largely faithful to the original book: The brave young rabbits striking out on their own before their home warren is destroyed; creating a new warren on Watership Down; the war with the totalitarian warren of Efrafa; the peaceful aftermath. One shortcoming is that although the voices are generally distinct, many of the rabbits look alike, and it's hard to tell one gray-brown rabbit from another.</p> 2018 ends tag:www.georgevreilly.com,2018-12-31:/blog/2018/12/31/2018-review.html 2018-12-31T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <p>2018 was a mixed year for Emma and me.</p> <p>At the start of the year, I was the principal engineer at MetaBrite. The year started out well initially, as we had moved to much larger offices at the end of 2017. In late January, a number of people were laid off, when it became apparent that the old business plan would no longer work. In late March, the company died abruptly when we lost our principal source of funding. I spent April looking for a job and received several offers.</p> <p>I joined <a class="reference external" href="https://www.stripe.com/">Stripe</a>'s Seattle office in June, where I work on the Edge team, which is &quot;ensuring Stripe’s continued existence on the Internet&quot;. It's been a bit of adjustment going from being a big fish in a small pond to at best a medium-sized fish in a much larger pond. Stripe is growing rapidly in Seattle and we just moved to a new space.</p> <p>I've been active in Toastmasters, continuing as an officer at my home club, <a class="reference external" href="http://freelyspeaking.org/">Freely Speaking Toastmasters</a>. We've struggled with finding new members. After a couple of years in South Lake Union, we have moved back to Capitol Hill. In July, I became an Area Director, responsible for several clubs in downtown Seattle.</p> <p>In June, the <a class="reference external" href="https://www.wildgeeseseattle.org/">Wild Geese Players of Seattle</a> put on our annual Bloomsday reading at the Seattle Central Library. In October, we arranged a 20th Anniversary Celebration of Irish Writing.</p> <p><a class="reference external" href="https://www.goodreads.com/user/show/3723742-george-reilly">GoodReads</a> says I read 113 books in 2018. Most of those books were on Kindle. I didn't bother to add most of the paperbacks and hardbacks that I read, so the total is probably closer to 140 books for the year.</p> <p>In the year leading up to my <a class="reference external" href="https://www.georgevreilly.com/blog/2017/01/17/KneeWalker.html">heel surgery</a> at the beginning of 2017, I lost the habit of cycling to work, which I had developed over the preceding decade. I have not yet rebuilt the habit. I'm going to work on it in 2019.</p> <p>Emma's health has not been good and she's had a rough year. She has a lot of abdominal problems, which are not well understood.</p> <p>It's been a poor year for the United States, with Trump in the Oval Office. I look forward to the new Democratic Congress which takes office later this week.</p> <p>Here's to 2019!</p> Review: The Heart's Invisible Furies tag:www.georgevreilly.com,2018-12-01:/blog/2018/12/01/ReviewHeartsInvisibleFuries.html 2018-12-01T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <a class="reference external image-reference" href="https://www.amazon.com/dp/152476079X/?tag=georgvreill-20"><img alt="The Heart's Invisible Furies" class="right-float" src="https://images-na.ssl-images-amazon.com/images/P/152476079X.01.MZZZZZZZ.jpg"/></a> <div class="line-block"> <div class="line">Title: The Heart's Invisible Furies</div> <div class="line">Author: John Boyne</div> <div class="line">Rating: ★ ★ ★ ★ ½</div> <div class="line">Publisher: Hogarth</div> <div class="line">Copyright: 2017</div> <div class="line">ISBN: <a class="reference external" href="https://www.amazon.com/dp/152476079X/?tag=georgvreill-20">152476079X</a></div> <div class="line">Pages: 592</div> <div class="line">Keywords: fiction, gay, irish</div> <div class="line">Reading period: 30 October, 2018</div> </div> <p>Before I begin to describe <em>The Heart's Invisible Furies</em> with abundant spoilers, let me say two things. Despite what I describe below, the book is very funny, as Cyril recounts his frequent fuckups. You would never know, from reading the back cover or the excerpted reviews inside, that Cyril is gay. Yet Cyril's sexuality is the central theme of the book. I can only assume that this is a marketing decision, with which I strongly disagree.</p> <p>16-year-old Catherine is forced out of her Cork village by the parish priest, when her pregnancy is discovered. It's impossible to be a single, unwed mother in Ireland in 1945, so she puts her son up for adoption. Cyril Avery grows up in an odd, well-off household in Dublin, always being told by his adoptive father that he's not a <em>real</em> Avery. At seven, he develops a fascination with Julian, the son of his father's lawyer, whom he meets one fateful day. They meet again at fourteen, sharing a room in boarding school, and Cyril develops a full-blown but entirely secret obsession with the golden boy, Julian.</p> <p>Cyril begins to realize that he is that most despised creature, a homosexual in 1950s Catholic Ireland. By twenty-one, he is frequently meeting other men for furtive, anonymous sex. There is no outlet for romance or love, just hurried encounters. Homosexual acts are illegal in Ireland and discovery would mean social and professional ruin, perhaps even prosecution. He's still secretly in love with Julian, now a heterosexual hedonist in the swinging Sixties. At twenty-eight, deeply conflicted, he marries Julian's sister Alice, and abandons her hours after the wedding. Of all the stupid things that Cyril does, this is the worst. By thirty-five (1980), he's ended up in Amsterdam, where gay people can live openly, and he's in a relationship with Bastiaan, a Dutch doctor. In 1987, Bastiaan and Cyril are in New York, working with dying AIDS patients. Fear of AIDS is rampant and many of them have been rejected by their families. One of these patients is Julian, who is determined that no-one in Ireland should know that he's dying of such a shameful disease. From Julian, Cyril belatedly learns that Alice has a teenaged son, Liam. Julian insists that Cyril should have told him long ago, that he would have understood. Cyril knows better. Cyril couldn't be honest with the world for a long time, because the world—and Ireland—wouldn't let him be honest about himself. The episode ends with Bastiaan being killed by fagbashers who leave Cyril in a coma.</p> <p>After that, Cyril returns to Dublin, where he lives openly if quietly as a gay man. In time, he is reunited with his mother Catherine and makes peace with Alice and Liam. The final episode takes place in 2015, months after Ireland voted to <a class="reference external" href="https://www.theguardian.com/world/2015/may/23/gay-marriage-ireland-yes-vote">legalize same-sex marriage</a>. Cyril, now seventy, wonders at the New Ireland and he marvels at his own 17-year-old grandson openly expressing affection for his boyfriend. It's a bittersweet moment for Cyril. He thinks about Senator David Norris, Ireland's most famous gay rights campaigner for forty years, saying, when asked if he would take advantage of the new law, “It’s a little bit late for me ... I’ve spent so much time pushing the boat out that I forgot to jump on and now it’s out beyond the harbour on the high seas, but it’s very nice to look at.”</p> <p>In a <a class="reference external" href="https://www.powells.com/post/original-essays/the-most-truthful-paragraph-ive-written">somewhat bitter afterword</a>, the author John Boyne describes writing the book. He knew from a young age that he was gay, long before he knew what “gay” meant. Boyne drew upon his own early experiences in creating Cyril and regrets the potential that is lost to our generation. Boyne was born in Dublin in 1971, six years after me. I realized in my mid-teens that I am bisexual, something that I kept entirely buried for a decade. I found Ireland no more accepting than John Boyne did in the 1980s. If I were thirteen years old now in Dublin, instead of being thirteen in 1978, I wonder how things would be turning out for me. Would coming out as a bisexual teen be a near non-event, instead of the long-deferred and fearful-but-exhilarating landmark of my twenties? My mother gave me this book, after reading it herself, so I think it would be easier this time around.</p> Election Day 2018 tag:www.georgevreilly.com,2018-11-06:/blog/2018/11/06/ElectionDay2018.html 2018-11-06T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <img alt="Grab 'Em by the Midterms" class="right-float" src="https://www.georgevreilly.com/content/binary/grab-them-by-the-midterms.jpg" style="width: 400px;"/> <!-- source: http://chicagowomentakeaction.com/event/vote-in-2018/ --> <p>In 2016, I threw an Election Night victory party for Hillary Clinton. It turned into a wake.</p> <p>In 2016, it was obvious to me and to millions of others that Trump was unfit to be President. There were weak excuses that he might turn out to be more presidential after the campaign was over. Instead, we got the worst president we've ever seen in the US. Morally unfit. Temperamentally unfit. Ethically unfit. A pathological liar. A shatterer of alliances. A demagogue, stoking the flames of right-wing violent extremism.</p> <p>In 2018, there are no excuses for not seeing how dangerous Trump is. Trump himself is not on the ballot, but this is nonetheless a referendum on Trump. The Republican party have done nothing to rein in his excesses. They will continue to do nothing if they retain the House of Representatives.</p> <p>The Republicans must be voted out of power in the House. The Senate too, if that's possible. Trump is deeply unpopular outside his base, but structural inequalities, gerrymandering, and voter suppression all make it far from certain that the Democrats can win tonight.</p> <p>Brace yourselves for a backlash.</p> <p>If the Democrats win a narrow victory, expect rage and violence from the right-wing that dwarfs that of the last few weeks. Trump has been laying down a narrative of a stolen election again, just as he did before the 2016 election.</p> <p>If the Republicans retain control, what little restraint that was still in place will go. They will not be magnanimous in victory.</p> <p>We may not know the final results for weeks, as tight races are counted and recounted.</p> <p>The best outcome is for the Democrats to win a resounding victory and sweep the House. I don't know if we can pull that off. But the best way to be sure is for everyone to <em>vote Democrat</em> today.</p> <p>Please vote. It matters.</p> Brilliant Jerks in Engineering tag:www.georgevreilly.com,2017-12-30:/blog/2017/12/30/BrilliantJerksInEngineering.html 2017-12-30T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <a class="reference external image-reference" href="http://www.brendangregg.com/blog/2017-11-13/brilliant-jerks.html"><img alt="&quot;Do not tolerate brilliant jerks. The cost to teamwork is too high&quot; — Reed Hastings, CEO Netflix" class="right-float" src="https://www.brendangregg.com/blog/images/2017/brilliantjerks.jpg" style="width: 400px;"/></a> <p>Brendan Gregg's <a class="reference external" href="http://www.brendangregg.com/blog/2017-11-13/brilliant-jerks.html">Brilliant Jerks in Engineering</a> is an excellent discussion of the &quot;No Asshole Rule&quot; applied to software engineers.</p> <p>He posits two kinds of brilliant jerks, the selfless and the selfish. You might call them unempathic and sociopathic, respectively. The former, if they develop some emotional intelligence, are worth saving. The latter are simply toxic and probably need to be fired.</p> How to talk to people you disagree with tag:www.georgevreilly.com,2017-12-27:/blog/2017/12/27/HowToTalkToPeopleYouDisagreeWith.html 2017-12-27T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <p>I came across an interesting post on Medium earlier tonight, <a class="reference external" href="https://medium.com/@jeremycaney/how-to-talk-to-people-you-disagree-with-420e12f695bc">How to talk to people you disagree with</a>.</p> <p>It can be hard to have a fruitful conversation with people you're at odds with, especially online.</p> <p>Jeremy Caney has 10 suggestions:</p> <ol class="arabic simple"> <li>Leave the insults at the door</li> <li>Understand what’s driving their views</li> <li>Speak to their values</li> <li>Know what you’re talking about</li> <li>Acknowledge when you’re wrong</li> <li>Stay focused on the issue at hand</li> <li>Be prepared to take heat from your team</li> <li>Don’t expect capitulation</li> <li>Know when to walk away</li> <li>Be mindful of the onlookers</li> </ol> Git Diff Tips tag:www.georgevreilly.com,2017-12-26:/blog/2017/12/26/GitDiffTips.html 2017-12-26T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <p>The <a class="reference external" href="https://git-scm.com/docs/git-diff">Git Diff</a> utility is much more functional than the standard command-line <tt class="docutils literal">diff</tt>.</p> <p>To see changes relative to the staging area (aka the index), use <tt class="docutils literal">git diff</tt>.</p> <p>To see <em>staged</em> changes, use <tt class="docutils literal">git diff <span class="pre">--staged</span></tt> (or <tt class="docutils literal"><span class="pre">--cached</span></tt>).</p> <p>To see changes side by side on a line (where it makes sense), use the <tt class="docutils literal"><span class="pre">--color-word</span></tt> option.</p> <p>To compare two arbitrary files in the file system, use <tt class="docutils literal">git diff <span class="pre">--no-index</span></tt>.</p> <p>To try some other <a class="reference external" href="https://stackoverflow.com/questions/4045017/what-is-git-diff-patience-for">diff algorithms</a>, use the <tt class="docutils literal"><span class="pre">--patience</span></tt>, <tt class="docutils literal"><span class="pre">--histogram</span></tt>, or <tt class="docutils literal"><span class="pre">--minimal</span></tt> options. The default diff algorithm is <tt class="docutils literal"><span class="pre">--myers</span></tt>.</p> <p>Lots more at the <a class="reference external" href="https://git-scm.com/docs/git-diff">docs</a>.</p> Review: Coco tag:www.georgevreilly.com,2017-12-25:/blog/2017/12/25/ReviewCoco.html 2017-12-25T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <a class="reference external image-reference" href="https://en.wikipedia.org/wiki/Coco_(2017_film)"><img alt="Coco" class="right-float" src="https://upload.wikimedia.org/wikipedia/en/9/98/Coco_%282017_film%29_poster.jpg"/></a> <div class="line-block"> <div class="line">Title: Coco</div> <div class="line">Director: Lee Unkrich</div> <div class="line">Rating: ★ ★ ★ ★ ½</div> <div class="line">Released: 2017</div> <div class="line">Keywords: animation</div> <div class="line">Country: USA</div> <div class="line">Watched: 25 December, 2017</div> </div> <p><a class="reference external" href="https://en.wikipedia.org/wiki/Coco_(2017_film)">Coco</a> is another delightful movie from Pixar: It's a magical tale of a Mexican boy who passionately wants to play music, even though his shoemaking family has rejected music ever since his great-great-grandfather pursued his own musical ambitions and abandoned his wife and child—the eponymous Coco, who is now ancient. Miguel discovers that his despised ancestor is none other than Ernesto de la Cruz, the most famous musician of his time. In order to enter a talent competition on Día de los Muertos, he steals Ernesto's guitar from his mausoleum, whereupon he is transported to the Land of the Dead. Surrounded by lively skeletons, he must find his way home before the night is out.</p> <p>Coco is visually beautiful, musically satisfying, entertaining, and uplifting. While amusing, it manages to avoid the pitfalls of inanity that afflict so many animated films targeted at children, and it also deals sensitively with topics such as death, bereavement, and abandonment, as well as the difficulties of being in a family and the costs of following one's dreams.</p> <p>Highly recommended.</p> Seattle: Overturned Truck tag:www.georgevreilly.com,2017-02-27:/blog/2017/02/27/SeattleOverturnedTruck.html 2017-02-27T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <p>Seattle traffic ground to an eight-hour standstill today after a propane truck <a class="reference external" href="http://www.seattletimes.com/seattle-news/transportation/semi-rollover-on-i-5-near-west-seattle-bridge-blocking-lanes/">overturned</a> on southbound I-5. Both directions of I-5 were closed until the truck had been emptied enough so that it could be righted. The wet winter weather didn't help, as snow fell intermittently.</p> <p>Given the danger of a massive explosion, this was the right thing to do. Had the truck gone up, the outcome would have been much, much worse.</p> <p>Still, it drives home (heh) the fragility of our infrastructure. The closure's effect rippled throughout the region's roads, causing a systemic failure of the road network. Snarled traffic and chaos everywhere for miles.</p> <p>We've had two similar incidents in recent years, both involving spillages of seafood that caused multi-hour freeway closures. When the system is running hot most of the time, close to its capacity, it only takes one bad accident to trigger cascading traffic jams everywhere.</p> Stitches in my Finger tag:www.georgevreilly.com,2017-02-23:/blog/2017/02/23/StitchesInMyFinger.html 2017-02-23T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <img alt="Five Stitches in Left Index Finger" class="right-float" src="https://www.georgevreilly.com/content/binary/finger-stitches.jpg"/> <p>I've had stitches before, but never from a self-inflicted cut. Five stitches this morning from a bread knife slipping on a stale loaf that I was going to toast. I never did get that snack.</p> <p>Fortunately, it hasn't been painful. The stitches should come out in 7–9 days. Meanwhile, typing is awkward and slow.</p> <p>My brother David when he was eight or nine nearly ruined his thumb with a chisel. He was using it to make a wooden sign for his cub scout troop. The chisel slipped and jammed into the back of his left thumb. He didn't sever the tendon and he got the full use of his thumb back, but he still has an ugly scar.</p> <p>Within a year or two of David's accident, my father lost the tip of his index finger. He was making a deposit into a night safe when some keys got in the way as the door swung closed. He flipped them out of the way, but his fingertip didn't clear the gap in time. Fortunately his own father was present and took him for medical treatment. He has a divot missing from the tip of his right index finger. Vincent, unlike his children, is left-handed so it wasn't quite as much of a loss as it sounds.</p> <p>Life! No-one gets out without scars.</p> I Haz Shoes tag:www.georgevreilly.com,2017-02-22:/blog/2017/02/22/IHazShoes.html 2017-02-22T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <p>I've been wearing a surgical boot since my <a class="reference external" href="https://www.georgevreilly.com/blog/2017/01/17/KneeWalker.html">heel surgery</a> six weeks ago. The first month I spent getting around on a knee walker or crutches. For the last couple of weeks, I've been walking more and more, but always wearing the boot. The doctor told me today that I could start weaning myself off the boot, but not to rush it.</p> <p>I wore shoes on my right foot tonight for the first time. I tried on a Doc Marten shoe but found it too stiff. It rubbed against the scar at the back of my heel. A pair of sneakers were more comfortable. I'll bring the right sneaker to work with me tomorrow and wear it for a while.</p> <p>It's such a relief to let my ankle flex forward. As soon as I step off my right foot while wearing the surgical boot, the inability to lean forward crimps my stride. And the thick raised sole pushes me off-kilter and does my hips and lower back no good.</p> <p>Baby steps!</p> OrderedDict Initialization tag:www.georgevreilly.com,2017-02-21:/blog/2017/02/21/OrderedDictInitialization.html 2017-02-21T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <p>An <a class="reference external" href="https://docs.python.org/2/library/collections.html#collections.OrderedDict">OrderedDict</a> is a Python <tt class="docutils literal">dict</tt> which remembers insertion order. When iterating over an <tt class="docutils literal">OrderedDict</tt>, items are returned in that order. Ordinary <tt class="docutils literal">dicts</tt> return their items in an unspecified order.</p> <p>Ironically, most of the ways of constructing an initialized <tt class="docutils literal">OrderedDict</tt> end up breaking the ordering in Python 2.x and in Python 3.5 and below. Specifically, using keyword arguments or passing a <tt class="docutils literal">dict</tt> (mapping) will not retain the insertion order of the source code.</p> <pre class="code pycon literal-block"> <span class="go">Python 2.7.13 (default, Dec 18 2016, 07:03:39) </span><span class="kn"></span><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">OrderedDict</span> <span class="go"> </span><span class="n"></span><span class="gp">&gt;&gt;&gt; </span><span class="n">odict</span> <span class="o">=</span> <span class="n">OrderedDict</span><span class="p">()</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">odict</span><span class="p">[</span><span class="s1">'one'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">odict</span><span class="p">[</span><span class="s1">'two'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">2</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">odict</span><span class="p">[</span><span class="s1">'three'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">3</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">odict</span><span class="p">[</span><span class="s1">'four'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">4</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">odict</span><span class="p">[</span><span class="s1">'five'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">5</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">odict</span><span class="o">.</span><span class="n">items</span><span class="p">()</span> <span class="go">[('one', 1), ('two', 2), ('three', 3), ('four', 4), ('five', 5)] </span><span class="n"></span><span class="gp">&gt;&gt;&gt; </span><span class="n">OrderedDict</span><span class="p">(</span><span class="n">one</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">two</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">three</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span> <span class="n">four</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span> <span class="n">five</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span><span class="o">.</span><span class="n">items</span><span class="p">()</span> <span class="go">[('four', 4), ('one', 1), ('five', 5), ('three', 3), ('two', 2)] </span><span class="n"></span><span class="gp">&gt;&gt;&gt; </span><span class="n">OrderedDict</span><span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="n">one</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">two</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">three</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span> <span class="n">four</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span> <span class="n">five</span><span class="o">=</span><span class="mi">5</span><span class="p">))</span><span class="o">.</span><span class="n">items</span><span class="p">()</span> <span class="go">[('four', 4), ('five', 5), ('three', 3), ('two', 2), ('one', 1)] </span><span class="n"></span><span class="gp">&gt;&gt;&gt; </span><span class="n">OrderedDict</span><span class="p">({</span><span class="s2">&quot;one&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s2">&quot;two&quot;</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span> <span class="s2">&quot;three&quot;</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span> <span class="s2">&quot;four&quot;</span><span class="p">:</span> <span class="mi">4</span><span class="p">,</span><span class="err">↩</span> <span class="go"> &quot;five&quot;: 5}).items() [('four', 4), ('three', 3), ('five', 5), ('two', 2), ('one', 1)]</span> </pre> <p>Only passing an <em>iterable</em> of key-value pairs will retain the order.</p> <pre class="code pycon literal-block"> <span class="n"></span><span class="gp">&gt;&gt;&gt; </span><span class="n">OrderedDict</span><span class="p">([(</span><span class="s2">&quot;one&quot;</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span> <span class="p">(</span><span class="s2">&quot;two&quot;</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="p">(</span><span class="s2">&quot;three&quot;</span><span class="p">,</span> <span class="mi">3</span><span class="p">),</span><span class="err">↩</span> <span class="go"> (&quot;four&quot;, 4), (&quot;five&quot;, 5)]).items() [('one', 1), ('two', 2), ('three', 3), ('four', 4), ('five', 5)]</span> </pre> <p>Note that <tt class="docutils literal">OrderedDict</tt> is noticeably slower than <tt class="docutils literal">dict</tt> in Python 2.7, so only use <tt class="docutils literal">OrderedDict</tt> when insertion order matters.</p> <p>In Python 3.6, the <a class="reference external" href="https://www.python.org/dev/peps/pep-0468/">order of kwargs is preserved</a>, thanks to the <a class="reference external" href="https://mail.python.org/pipermail/python-dev/2016-September/146327.html">new compact implementation of dict</a>. <tt class="docutils literal">OrderedDict</tt> is also implemented in C and its performance is on par with that of <tt class="docutils literal">dict</tt>.</p> <p>Python 3.6 <tt class="docutils literal">dict</tt>s are now ordered too, but if you care about portable code (earlier CPython or other Python implementations such as Jython), use <tt class="docutils literal">OrderedDict</tt> rather than relying on this implementation detail.</p> <pre class="code python literal-block"> <span class="n">Python</span> <span class="mf">3.6</span><span class="o">.</span><span class="mi">0</span> <span class="p">(</span><span class="n">default</span><span class="p">,</span> <span class="n">Dec</span> <span class="mi">24</span> <span class="mi">2016</span><span class="p">,</span> <span class="mi">08</span><span class="p">:</span><span class="mi">01</span><span class="p">:</span><span class="mi">42</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">OrderedDict</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">OrderedDict</span><span class="p">(</span><span class="n">one</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">two</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">three</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span> <span class="n">four</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span> <span class="n">five</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span><span class="o">.</span><span class="n">items</span><span class="p">()</span> <span class="n">odict_items</span><span class="p">([(</span><span class="s1">'one'</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span> <span class="p">(</span><span class="s1">'two'</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="p">(</span><span class="s1">'three'</span><span class="p">,</span> <span class="mi">3</span><span class="p">),</span> <span class="p">(</span><span class="s1">'four'</span><span class="p">,</span> <span class="mi">4</span><span class="p">),</span> <span class="err">…</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">OrderedDict</span><span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="n">one</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">two</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">three</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span> <span class="n">four</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span> <span class="n">five</span><span class="o">=</span><span class="mi">5</span><span class="p">))</span><span class="o">.</span><span class="n">items</span><span class="p">()</span> <span class="n">odict_items</span><span class="p">([(</span><span class="s1">'one'</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span> <span class="p">(</span><span class="s1">'two'</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="p">(</span><span class="s1">'three'</span><span class="p">,</span> <span class="mi">3</span><span class="p">),</span> <span class="p">(</span><span class="s1">'four'</span><span class="p">,</span> <span class="mi">4</span><span class="p">),</span> <span class="err">…</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">OrderedDict</span><span class="p">({</span><span class="s2">&quot;one&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s2">&quot;two&quot;</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span> <span class="s2">&quot;three&quot;</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span> <span class="s2">&quot;four&quot;</span><span class="p">:</span> <span class="mi">4</span><span class="p">,</span><span class="err">↩</span> <span class="s2">&quot;five&quot;</span><span class="p">:</span> <span class="mi">5</span><span class="p">})</span><span class="o">.</span><span class="n">items</span><span class="p">()</span> <span class="n">odict_items</span><span class="p">([(</span><span class="s1">'one'</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span> <span class="p">(</span><span class="s1">'two'</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="p">(</span><span class="s1">'three'</span><span class="p">,</span> <span class="mi">3</span><span class="p">),</span> <span class="p">(</span><span class="s1">'four'</span><span class="p">,</span> <span class="mi">4</span><span class="p">),</span> <span class="err">…</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">OrderedDict</span><span class="p">([(</span><span class="s2">&quot;one&quot;</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span> <span class="p">(</span><span class="s2">&quot;two&quot;</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="p">(</span><span class="s2">&quot;three&quot;</span><span class="p">,</span> <span class="mi">3</span><span class="p">),</span><span class="err">↩</span> <span class="p">(</span><span class="s2">&quot;four&quot;</span><span class="p">,</span> <span class="mi">4</span><span class="p">),</span> <span class="p">(</span><span class="s2">&quot;five&quot;</span><span class="p">,</span> <span class="mi">5</span><span class="p">)])</span><span class="o">.</span><span class="n">items</span><span class="p">()</span> <span class="n">odict_items</span><span class="p">([(</span><span class="s1">'one'</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span> <span class="p">(</span><span class="s1">'two'</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="p">(</span><span class="s1">'three'</span><span class="p">,</span> <span class="mi">3</span><span class="p">),</span> <span class="p">(</span><span class="s1">'four'</span><span class="p">,</span> <span class="mi">4</span><span class="p">),</span> <span class="err">…</span> </pre> Review: Skinny Dip tag:www.georgevreilly.com,2017-02-19:/blog/2017/02/19/ReviewSkinnyDip.html 2017-02-19T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <a class="reference external image-reference" href="https://www.amazon.com/dp/0446615129/?tag=georgvreill-20"><img alt="Skinny Dip" class="right-float" src="https://images-na.ssl-images-amazon.com/images/P/0446615129.01.MZZZZZZZ.jpg"/></a> <div class="line-block"> <div class="line">Title: Skinny Dip</div> <div class="line">Author: Carl Hiaasen</div> <div class="line">Rating: ★ ★ ★ ★</div> <div class="line">Publisher: Warner</div> <div class="line">Copyright: 2004</div> <div class="line">ISBN: <a class="reference external" href="https://www.amazon.com/dp/0446615129/?tag=georgvreill-20">978-0446-615129</a></div> <div class="line">Pages: 496</div> <div class="line">Keywords: humor, crime</div> <div class="line">Reading period: 18–19 February, 2017</div> </div> <p>Joey Perrone is very surprised to find herself thrown off a cruise ship on her second wedding anniversary. After a night of swimming, she washes up on a small Florida island in the company of a prematurely retired investigator. Joey persuades Mick Stranahan not to report the attempted murder, but instead to investigate and torment her worthless husband, Chaz, who turns out to be a <a class="reference external" href="http://www.urbandictionary.com/define.php?term=biostitute">biostitute</a> for a major polluter of the Everglades, as well as a relentless pussyhound, an inept killer, and an all-round shitheel.</p> <p>Hiaasen has a lot of fun at the expense of some of his characters, but his outrage at the despoliation of the Everglades is quite real. And once again, the grizzled cynical middleaged guy attracts the pretty younger woman.</p> HTML5 tables require tr inside thead tag:www.georgevreilly.com,2017-02-18:/blog/2017/02/18/HTML5TablesRequireTrInsideThead.html 2017-02-18T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <p>When I learned HTML tables back in the 90s, at some point I discovered the <tt class="docutils literal">&lt;thead&gt;</tt> element for grouping the <tt class="docutils literal">&lt;th&gt;</tt> column headers. What I missed was there should be a <tt class="docutils literal">&lt;tr&gt;</tt> element between the two. In other words, a well-formed HTML table with a header looks like this:</p> <pre class="code html literal-block"> <span class="p">&lt;</span><span class="nt">table</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">thead</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">tr</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">th</span><span class="p">&gt;</span>Name<span class="p">&lt;/</span><span class="nt">th</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">th</span><span class="p">&gt;</span>Value<span class="p">&lt;/</span><span class="nt">th</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">th</span><span class="p">&gt;</span>Date<span class="p">&lt;/</span><span class="nt">th</span><span class="p">&gt;</span> <span class="p">&lt;/</span><span class="nt">tr</span><span class="p">&gt;</span> <span class="p">&lt;/</span><span class="nt">thead</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">tbody</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">tr</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">td</span><span class="p">&gt;</span>USERNAME<span class="p">&lt;/</span><span class="nt">td</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">td</span><span class="p">&gt;</span>John.Smith<span class="p">&lt;/</span><span class="nt">td</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">td</span><span class="p">&gt;</span>2017-02-18T23:47<span class="p">&lt;/</span><span class="nt">td</span><span class="p">&gt;</span> <span class="p">&lt;/</span><span class="nt">tr</span><span class="p">&gt;</span> <span class="p">&lt;/</span><span class="nt">tbody</span><span class="p">&gt;</span> <span class="p">&lt;/</span><span class="nt">table</span><span class="p">&gt;</span> </pre> <p>and not:</p> <pre class="code html literal-block"> <span class="p">&lt;</span><span class="nt">table</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">thead</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">th</span><span class="p">&gt;</span>Name<span class="p">&lt;/</span><span class="nt">th</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nt">th</span><span class="p">&gt;</span>Value<span class="p">&lt;/</span><span class="nt">th</span><span class="p">&gt;</span> … </pre> <p>The latter form—<tt class="docutils literal">&lt;thead&gt;</tt> directly enclosing <tt class="docutils literal">&lt;th&gt;</tt>s—had always worked for me. Until yesterday when I ran afoul of an HTML5 validator on a remote API, which simply would not let me proceed until I wrapped my <tt class="docutils literal">&lt;th&gt;</tt> cells with a <tt class="docutils literal">&lt;tr&gt;</tt>.</p> <p>Who knew?</p> Review: The Italian Job tag:www.georgevreilly.com,2017-02-17:/blog/2017/02/17/ReviewTheItalianJob.html 2017-02-17T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <a class="reference external image-reference" href="https://en.wikipedia.org/wiki/The_Italian_Job"><img alt="The Italian Job" class="right-float" src="https://upload.wikimedia.org/wikipedia/en/b/b3/The_Italian_Job_1969_poster.jpg"/></a> <div class="line-block"> <div class="line">Title: The Italian Job</div> <div class="line">Director: Peter Collinson</div> <div class="line">Rating: ★ ★ ★ ★</div> <div class="line">Released: 1969</div> <div class="line">Keywords: heist, comedy, cars</div> <div class="line">Country: UK</div> <div class="line">Watched: 17 Febuary, 2017</div> </div> <p><a class="reference external" href="https://en.wikipedia.org/wiki/The_Italian_Job">The Italian Job</a> movie is worth your time. One of the quintessential movies of the Swinging Sixties, its British sensibility wears well, almost 50 years on. The humour still works. And it's probably the best advertisement that the <a class="reference external" href="https://en.wikipedia.org/wiki/Mini#Mini_Cooper_and_Cooper_S:_1961.E2.80.932000">Mini</a> ever had.</p> <p>Charlie Croker (Michael Caine) has inherited a plan to rip off $4 million in gold bullion from Fiat in Turin. He and the lads are going to help the balance of payments by bringing the loot back from the Common Market. (They're proto-Euroskeptics.) And they're going to do it by causing the mother of all traffic jams and making their getaway in three little Mini Coopers.</p> <p>Don't expect too much coherence from the plot. There's at least 20 minutes of gloriously improbable chase scene set pieces where the three Minis perform balletic moves and lose another police Alfa Romeo or two. The literal cliffhanger ending is a classic. As with most comedies of the time, the characterization is weak—we learn a bit about what makes Croker and crime boss Bridger (Noël Coward) tick, but the rest are ciphers. No matter. The plot ticks along and it delivers.</p> Review: Kill Me Three Times tag:www.georgevreilly.com,2017-02-10:/blog/2017/02/10/ReviewKillMeThreeTimes.html 2017-02-10T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <a class="reference external image-reference" href="https://en.wikipedia.org/wiki/Kill_Me_Three_Times"><img alt="Kill Me Three Times" class="right-float" src="https://upload.wikimedia.org/wikipedia/en/2/2e/Kill_Me_Three_Times_poster.jpg"/></a> <div class="line-block"> <div class="line">Title: Kill Me Three Times</div> <div class="line">Director: Kriv Stenders</div> <div class="line">Rating: ★ ★ ★</div> <div class="line">Released: 2014</div> <div class="line">Keywords: black comedy thriller</div> <div class="line">Country: Australia</div> <div class="line">Watched: 10 February, 2017</div> </div> <p>A jealous husband engages a private detective-cum-killer for hire (Simon Pegg) to follow his wife. Upon proof of her infidelity, he orders a hit, which triggers a comedy of errors and double crosses, which ultimately leaves most of the cast dead at each other's hands.</p> <p>There's not much to like about this Australian noirish comedy. It's bloody but not that funny. The characters are thinly drawn and unengaging. They're a far cry from Tarantino's gonzo motormouths or the Coen Brother's quirky killers.</p> Jenkins #6: Miscellenea tag:www.georgevreilly.com,2017-02-09:/blog/2017/02/09/Jenkins06-Miscellenea.html 2017-02-09T07:00:00Z George V. Reilly https://www.georgevreilly.com/ george@reilly.org <p></p> <blockquote> [Previously published at the now defunct <a class="reference external" href="https://web.archive.org/web/20171001220321/http://devblog.metabrite.com/">MetaBrite Dev Blog</a>.]</blockquote> <p>A collection of miscellaneous tips on using Pipelines in Jenkins 2.0.</p> <p><a class="reference external" href="https://www.georgevreilly.com/blog/2017/02/04/Jenkins01-MigratingToPipelines.html">#6 in a series on Jenkins Pipelines</a></p> <div class="section" id="environment-variables"> <h3>Environment Variables</h3> <p>Use the <tt class="docutils literal">withEnv</tt> step to set environment variables. Don't manipulate the <tt class="docutils literal">env</tt> global variable.</p> <p>The confusing example that you see in the documents, <tt class="docutils literal"><span class="pre">PATH+WHATEVER=/something</span></tt>, simply means to <em>prepend</em> <tt class="docutils literal">/something</tt> to <tt class="docutils literal">$PATH</tt>. The <tt class="docutils literal">+WHATEVER</tt> has no other effect.</p> </div> <div class="section" id="credentials"> <h3>Credentials</h3> <p>The <tt class="docutils literal">withEnv</tt> step should not be used to introduce secrets into the build environment. Use the <a class="reference external" href="https://wiki.jenkins-ci.org/display/JENKINS/Credentials+Binding+Plugin">withCredentials</a> plugin instead.</p> <pre class="code groovy literal-block"> <span class="n">withCredentials</span><span class="o">([</span> <span class="o">[</span><span class="n">$class</span><span class="o">:</span> <span class="s1">'StringBinding'</span><span class="o">,</span> <span class="nl">credentialsId:</span> <span class="s1">'GPG_SECRET'</span><span class="o">,</span> <span class="nl">variable:</span> <span class="s1">'GPG_SECRET'</span><span class="o">],</span> <span class="o">[</span><span class="n">$class</span><span class="o">:</span> <span class="s1">'AmazonWebServicesCredentialsBinding'</span><span class="o">,</span> <span class="nl">credentialsId:</span> <span class="s1">'0defaced-cafe-f00d-badd-0000000ff1ce'</span><span class="o">,</span> <span class="nl">accessKeyVariable:</span> <span class="s1">'AWS_ACCESS_KEY_ID'</span><span class="o">,</span> <span class="nl">secretKeyVariable:</span> <span class="s1">'AWS_SECRET_ACCESS_KEY'</span><span class="o">]</span> <span class="o">])</span> <span class="o">{</span> </pre> <p>Tip: Use the Pipeline Syntax Snippet Generator from <em>your</em> Jenkins project to figure out the exact invocation for your credentials.</p> <p>If you need to display environment variables for debugging purposes, you can use something like this Python script instead of <tt class="docutils literal">sh 'env | sort'</tt>, which can leave secrets in cleartext in your logs. Note that <tt class="docutils literal">withCredentials</tt> will mask values that it knows about.</p> <pre class="code python literal-block"> <span class="ch">#!/usr/bin/env python</span> <span class="kn">import</span> <span class="nn">os</span><span class="o">,</span> <span class="nn">re</span> <span class="n">BLOCKLIST_RE</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="sa">r</span><span class="s1">'^(.*_)?(PASSWORD|AUTH|TOKEN|SECRET|KEY)(_.*)?$'</span><span class="p">)</span> <span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">keys</span><span class="p">()):</span> <span class="n">v</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="n">k</span><span class="p">]</span> <span class="k">if</span> <span class="n">BLOCKLIST_RE</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="n">k</span><span class="p">)</span> <span class="ow">and</span> <span class="nb">len</span><span class="p">(</span><span class="n">v</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">2</span><span class="p">:</span> <span class="n">v</span> <span class="o">=</span> <span class="s1">'*</span><span class="si">{}{}{}</span><span class="s1">*'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">v</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="nb">len</span><span class="p">(</span><span class="n">v</span><span class="p">)</span><span class="o">-</span><span class="mi">2</span><span class="p">,</span> <span class="n">v</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span> <span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="si">{}</span><span class="s1">=</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">))</span> </pre> </div> <div class="section" id="cleaning-up-workspaces"> <h3>Cleaning Up Workspaces</h3> <p>You may need to clean up your workspace. <a class="reference external" href="https://www.georgevreilly.com/blog/2017/02/09/jenkins-03-github-integration/index.html">Don't remove the .git directory</a>.</p> <pre class="code groovy literal-block"> <span class="n">sh</span> <span class="s2">&quot;ls -A1 | grep -v &quot;</span><span class="o">^</span><span class="err">\\</span><span class="o">.</span><span class="na">git</span><span class="err">\</span><span class="n">$</span><span class="s2">&quot; | xargs rm -rf&quot;</span> </pre> <ul class="simple"> <li><tt class="docutils literal">ls <span class="pre">-A1</span></tt> lists all the files and directories in the current directory, except <tt class="docutils literal">.</tt> and <tt class="docutils literal">..</tt>, one per line.</li> <li>The <tt class="docutils literal">grep <span class="pre">-v</span></tt> excludes <tt class="docutils literal"><span class="pre">^\.git$</span></tt>. Note that the <tt class="docutils literal">\</tt> and the <tt class="docutils literal">$</tt> are escaped in the <tt class="docutils literal">sh</tt> step to prevent <a class="reference external" href="https://www.georgevreilly.com/blog/2017/02/09/jenkins-04-sh-step/index.html">GString interpolation</a>.</li> <li>The <tt class="docutils literal">xargs rm <span class="pre">-rf</span></tt> deletes the remaining files and directories.</li> </ul> </div> <div class="section" id="editing-jenkinsfile"> <h3>Editing Jenkinsfile</h3> <p>Be sure to put <tt class="docutils literal">#!groovy</tt> as the first line in your <tt class="docutils literal">Jenkinsfile</tt>, as a hint to your editor to use Groovy syntax highlighting.</p> <p>We use <a class="reference external" href="https://www.jetbrains.com/pycharm/">PyCharm</a> a lot, since we're a Python shop. However, despite being written in Java, PyCharm's support for Groovy is mediocre. You're expected to buy JetBrains' IntelliJ IDE.</p> <p><a class="reference external" href="https://www.vim.org/">Vim</a> has built-in support for Groovy. <a class="reference external" href="https://atom.io/">Atom</a> with the <a class="reference external" href="https://atom.io/packages/language-groovy">language-groovy</a> package works great. Many other editors also support Groovy.</p> </div> <div class="section" id="stashes-and-artifacts"> <h3>Stashes and Artifacts</h3> <p>Use <tt class="docutils literal">stash</tt> and <tt class="docutils literal">unstash</tt> to get artifacts across nodes within a pipeline. If you're using the <tt class="docutils literal">parallel</tt> step, some of your branches will start out with an empty workspace, and <tt class="docutils literal">unstash</tt> is the easiest way to prime the workspace.</p> <p>If you need to transfer artifacts between different builds, you'll need to use <tt class="docutils literal">archiveArtifacts</tt> and <tt class="docutils literal">CopyArtifact</tt>.</p> <p>In project <tt class="docutils literal"><span class="pre">build-1</span></tt>:</p> <pre class="code groovy literal-block"> <span class="n">archiveArtifacts</span> <span class="nl">artifacts:</span> <span class="s1">'my_artifact.zip'</span><span class="o">,</span> <span class="nl">fingerprint:</span> <span class="kc">true</span> </pre> <p>In project <tt class="docutils literal"><span class="pre">build-2</span></tt>:</p> <pre class="code groovy literal-block"> <span class="n">step</span> <span class="o">([</span><span class="n">$class</span><span class="o">:</span> <span class="s1">'CopyArtifact'</span><span class="o">,</span> <span class="nl">projectName:</span> <span class="s1">'build-1'</span><span class="o">,</span> <span class="nl">filter:</span> <span class="s1">'my_artifact.zip'</span><span class="o">,</span> <span class="nl">selector:</span> <span class="o">[</span><span class="n">$class</span><span class="o">:</span> <span class="s1">'StatusBuildSelector'</span><span class="o">]</span> <span class="o">]);</span> </pre> <p>Note that the <tt class="docutils literal">StatusBuildSelector</tt> selector is picking the artifact from the last successful build of <tt class="docutils literal"><span class="pre">build-1</span></tt>.</p> <p>You can find the various selectors in the <a class="reference external" href="https://github.com/jenkinsci/copyartifact-plugin/tree/master/src/main/java/hudson/plugins/copyartifact">CopyArtifact source</a>. If you dig into the tests, you can probably figure out how to use the other selectors.</p> <p>In <tt class="docutils literal"><span class="pre">build-1</span></tt>, you can use <tt class="docutils literal">build job: <span class="pre">'build-2',</span> wait: false</tt> to kick off <tt class="docutils literal"><span class="pre">build-2</span></tt>.</p> </div> <div class="section" id="debugging"> <h3>Debugging</h3> <p>Here are some tricks for debugging.</p> <div class="section" id="local-groovy"> <h4>Local Groovy</h4> <p>Run <a class="reference external" href="https://www.georgevreilly.com/blog/2017/02/08/Jenkins05-Groovy.html">Groovy locally</a> to debug syntax issues and Groovy functions.</p> </div> <div class="section" id="timestamps-and-ansicolor"> <h4>timestamps and ansiColor</h4> <p>Use the <tt class="docutils literal">timestamps</tt> plugin to get timestamps to appear in the Jenkins log. This is so useful that it should be the default.</p> <p>The <tt class="docutils literal">ansiColor</tt> plugin will generate transform colored console output into colored log output.</p> <pre class="code groovy literal-block"> <span class="n">node</span><span class="o">(</span><span class="s1">'ubuntu'</span><span class="o">)</span> <span class="o">{</span> <span class="n">timestamps</span> <span class="o">{</span> <span class="n">ansiColor</span><span class="o">(</span><span class="s1">'xterm'</span><span class="o">)</span> <span class="o">{</span> <span class="n">stage</span><span class="o">(</span><span class="s2">&quot;Source Checkout&quot;</span><span class="o">)</span> <span class="o">{</span> <span class="n">checkout</span> <span class="n">scm</span> </pre> </div> <div class="section" id="job-debugging"> <h4>Job Debugging</h4> <p>Instead of troubleshooting build problems in a slow heavyweight job, create a new lightweight job that isolates the problem. You may get your iteration time down to a minute or so, instead of many minutes.</p> <p>Temporarily use &quot;Pipeline script&quot; instead of &quot;Pipeline script from SCM&quot;, so that you can try out changes more quickly. Note that <tt class="docutils literal">checkout scm</tt> won't work unless the Pipeline script comes from SCM.</p> <p>The <tt class="docutils literal">Replay</tt> command in the left menu lets you edit the Pipeline script and rerun it.</p> </div> <div class="section" id="sh-errors"> <h4>sh Errors</h4> <p>You can use <a class="reference external" href="https://www.georgevreilly.com/blog/2017/02/07/Jenkins04-shStep.html">cat -n $0</a> to echo the interpolated <tt class="docutils literal">sh</tt> script to the log.</p> </div> </div>