Weapons of Math Destruction by Cathy O’Neil

Weapons of Math Destruction: How Big Data Increases Inequality and Threatens DemocracyWeapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy by Cathy O’Neil
My rating: 5 of 5 stars

This fascinating and well-written non-fiction book explores how the brokers and manipulators of “big data” affect us all, often in harmful ways. The author is a former math professor, Wall Street quant, and now is a full-time “Data Scientist,” a title she gave herself. She definitely has a great deal of inside knowledge about the users of big data and the algorithms they use to churn through the data and direct their activities. She calls them Weapons of Math Destruction or WMDs. Among the users are banks, credit companies, employers, government agencies, universities, advertisers, search engines, police departments, payday lenders, criminal sentencing courts, and insurance agencies.

She gives an example of a teacher in Washington, D.C. who was fired because a WMD identified her as being in the worst 10% of teachers despite having had glowing reviews in the current year and top scores in previous years. It turns out that the teacher who had taught most of her students the previous year had corrected the standardized tests of her students to make it look like they were performing better than they actually were. The result was that their scores on the test dropped during the next year even though they actually made good progress. The cheater kept her job, while the honest teacher lost hers.

Another surprising area to me was how the U.S. News rankings of colleges and universities has driven up tuition, lowered the quality of faculties, and actually made it harder for some top students to get into a “safety” school. You’ll have to read the book to understand how this happens. Many such tidbits are set forth throughout the book.

The book has a definite political slant to it. The author decries unfairness in general, which I consider apolitical, but then tends to harangue on anything she sees as racial inequality or “targeting the poor.” Many WMDs take into account such things as Zip codes or credit scores, things she considers “proxies” for race. From the viewpoint of a civil libertarian, this is a valid approach, but, as she herself admits, from a business standpoint, some of these WMDs are effective at reducing inefficiencies and increasing profits. Corporations, banks, and even many government agencies are not in the business of fairness or eliminating racial inequality; they’re in the business of business, i.e. making money, or in the case of the government agencies, accomplishing an important task like public safety or building infrastructure at a reasonable cost. One could argue that they have a legal and moral duty, a fiduciary responsibility toward stockholders or taxpayers, to increase those profits or efficiencies. As she also admits, using traditional human judgment alone, without the WMDs, has its own history of unfairness and racial prejudice.

A central theme throughout the book that was not explicitly stated is the failure of nearly all the WMDs to take into account the effect they themselves have on human behavior. Take the mortgage crisis of 2007-2008, for example. The math behind the collateralized debt obligations (CDOs) that packaged sub-prime mortgages with other debts was valid. If borrowers had continued to behave as they had statistically in the past, defaulting at the same rate, the CDOs would have been sound investments. What the lenders and brokers did not factor in was that once the market was created for these securities, lenders and borrowers would both change their behavior, increasing the number of mortgages granted to people who obviously had no way to repay them, thus changing the long-standing statistics on which the WMD was based.

If you want to know how to increase your chances of getting hired or how to get a better college education at a lower cost, this book is worth your time studying. I don’t have any skin in the game, but I found it a very interesting read even so.

View all my reviews

Inside the FBI: New York – a review

I don’t usually comment on television, but regular readers of this blog know I’m a retired FBI agent and may have some special insight on the new series “Inside the FBI – New York.” I watched the premiere Thursday on USA Network. My overall impression: there’s good news and bad news.

The series is an unscripted documentary-style show made with the full cooperation of the FBI, mandated from Director Comey. There is plenty of footage of real FBI agents inside the office, on the streets, and at home. The good news is that the show is realistic. It rang true to me, reflecting what the FBI is really like. I served in the New York office for a year early in my career. The ridiculous portrayal of FBI agents in drama shows is put to rest here. Instead, it showed men in suits sitting around conference tables discussing threat reports or out on the street looking for a terrorist suspect (only a suspect – don’t mistake that for a terrorist) who may be in New York during a major event like the Thanksgiving Parade or New Year’s Eve in Times Square. Unlike fake FBI TV, this show depicted the confusing information that comes in – an email trail that proved the suspect was trying to acquire a weapon, but nothing that showed he had succeeded in getting one, the lack of information on his current whereabouts. I think the dedication and stress of the FBI agents came through accurately.

The bad news is that it was rather boring. it showed men in suits sitting around conference tables discussing threat reports. Hey, didn’t I just say that? Yes – and that’s the good and bad of it. That doesn’t make riveting TV, although that is often the real life of the FBI. I thought the scenes humanizing the agents by showing their families and personal interactions were rather interesting, especially the anecdote about the suspect who stabbed an agent with a butcher knife during an arrest – and he was someone not expected to be violent. Both the knife and the protective vest worn by the agent that saved his life were shown to the TV audience and they could imagine, almost feel, having that 8-inch blade thrust at their abdomen. It highlighted the danger that even a “routine” case can present to an agent. Unfortunately, that bit was buried well into the episode, after fifteen or twenty minutes of men in suits around tables. That was a major editing/directing flaw in my opinion, especially for a premiere episode.

The producers no doubt thought that the tension would be ratcheted up by featuring the terrorist task force right after the attacks in Paris and San Bernardino, which took place shortly before the holiday season in New York. To some extent, perhaps, it was, but really, viewers already knew that nothing happened. Nothing, that is, in 2013 when this was filmed, but remember Faisal Shahzad, a would-be ISIS sympathizer, parked a large SUV bomb in Times Square in 2010, not to mention the 9-11 attacks. The danger was real, but the suspense for the viewer was not.

I will continue to watch the series, at least for a while, but I predict it will not be a commercial success.

Electric vehicle adoption

The map on the left has been circulated quite a bit recently, showing that electric vehicles (EVs) have been doing well in certain states like California, Texas, and New York. That map is misleading, however. Of course states with large populations and large numbers of cars registered are going to have more EVs. The one on the right shows which states have the highest rate of EV adoption based on the percentage of the vehicles in the state that are EVs, a better indication of how EVs have caught on. While many similarities exist between the two maps, it becomes evident from the second chart that states like Hawaii, Vermont, and Nevada are actually among leaders in adopting EVs. Georgia, with its strong state incentive program, now leads the nation in current sales of EVs as a percent, although California still leads in overall percentage registered.

Sources: Energy.gov, CleanTechnica.gov, USEIA. Data is from 2014, the most recent I could find.


Trump vote vs. Average SAT score

This graphic illustrates the percent of the vote total Donald Trump received from each state (top map) and the average SAT score of entering freshman for each state’s flagship public university. Both were obtained from data published by the responsible state officials (for the vote totals) and by the universities (normalized for the new (2016) SAT scoring system). The darkest red in the top map represents the highest vote percent for Donald Trump and in the lower map, it means the lowest average SAT score.

Note that blue (top) represents all non-Trump votes, not necessarily a vote for Clinton, since in some states (e.g. Utah)  third party candidates received significant votes. The universities selected were the premier public university in the state, excluding specialty schools such as engineering or arts. The District of Columbia had no qualifying universities since all the public universities were specialty schools (e.g. military/intelligence).

The Poison Artist by Jonathan Moore

The Poison ArtistThe Poison Artist by Jonathan Moore
My rating: 5 of 5 stars

Caleb is living the life of a Gen X at the pinnacle of success. He owns a house in a tony district of San Francisco and has his own biochemistry research lab where he is a respected expert on poisons and pain-inducing compounds. No Budweiser on the sofa for him. He drinks Jameson and Guinness in high-end bars, until, that is, a mysterious, captivating woman sits next to him and orders a Berthe de Joux, French pour. Caleb has just had a fight, a serious one, with his girlfriend Bridget, and it has sent him on this bender, but this new woman enthralls him. He watches as chilled water is poured over sugar cubes through a slotted silver spoon into the drink. It’s French, the notorious absinthe which was banned for decades because of its mythic poisonous qualities. The woman downs the drink and disappears as quickly as she appeared. He must try the same drink and he must find her again somehow.

When he sobers up and returns to work, he is confronted with the possibility of his grant money not being renewed. For his funding he needs more data on pain, on the chemicals that are produced during extreme pain, in order to help researchers and drug companies develop better anesthetics and manage chronic pain better. While he is straining to obtain this funding, he is also helping his best friend Henry, the chief medical examiner, determine the cause of death of some recent victims who appeared to die of natural causes, but whom Henry suspects were poisoned. Caleb confirms the poisoning. Soon he is obsessed with tracking the serial killer and trying to find the ephemeral woman, a woman we later learn is named Emmeline. While this is going on, we find allusions to a dark and troubling past of Caleb’s, something Henry knows about.

I won’t spoil with any further plot details. This book is all atmospherics and they very dark indeed. It’s so noir, it’s ebony. The style is something of an acquired taste, but I acquired it quickly and became fully absorbed in the depth of the mood. Be warned: it is not for the faint of heart or queasy of stomach. Stephen King fans will soak it up. Others may find it hard to take toward the end, but if you can get yourself around a large dose of creepiness, this is the book for you.

View all my reviews

Delphi polygons

I just discovered the Polygons method in Delphi yesterday, so now I have a new toy to play with. I’m working on creating my own USA state-by-state map. You can expect to be inundated with maps showing all sorts of possibly meaningless correlations. There’s probably some freeware out there that does the same thing, but that wouldn’t be as much fun and there’s always a hangup of some kind with that stuff. Here’s my work in progress.

The Snowden Files by Luke Harding

The Snowden Files: The Inside Story of the World's Most Wanted ManThe Snowden Files: The Inside Story of the World’s Most Wanted Man by Luke Harding
My rating: 3 of 5 stars

I read this book because my book club chose it. I had expected to hate it largely because I expected it to lionize a man I considered a traitor, yet I harbored a secret fear that I might be persuaded to find Snowden to be a true hero, proving my own instincts wrong. In the end I was, surprisingly, rather bored with the whole thing.

The book is written well enough, but for a purported piece of investigative journalism it sure didn’t say much. It gave a bit of Snowden’s background, and at the end a short epilogue about his unintended self-exile in Russia. The big “revelations” in the book consist of a general description of the NSA’s major programs, such as listening to cell phone traffic, buffering internet data, and so forth, and listed their rather fanciful code names. My reaction to that was much the same as one British politician quoted near the end: “Spies spy.” Well, duh! Other than that, the remaining 90% of the book was pretty much a puff piece for The Guardian, the British tabloid that Snowden chose as his outlet for the stolen documents, or some of them at least. The author, a writer for that paper, seems to have an inferiority complex and tried mightily to use this platform to cast his employer as a major player and knight in shining armor for civil liberties everywhere. Ho hum (although I do like their cryptic crosswords).

What the book didn’t do is provide a single instance of anyone who was ever harmed by the NSA’s surveillance actions. Balance this with the fact the NSA did provide a few examples of terrorist plots that had been disrupted thanks to their monitoring efforts. To be fair, it also didn’t provide any examples of how Snowden’s action resulted in any harm to the U.S. or its Anglophone Five. As a former FBI agent I know how public foofaraw can be disruptive to an agency, but soon enough such revelations fade into irrelevance like a mosquito bite on an elephant.

Perhaps the same homily can be applied to both the NSA and Snowden: no harm, no foul. I have no doubt the NSA continues to intercept almost everything. Spies spy. I’m happy to have them record, read, watch, listen to, or parse everything I say or do. I’m not a criminal or terrorist. I will never understand those people who are outraged at the idea their communications are monitored by the government. Whenever I hear someone say that they are, I wonder what crime they’re worried about being caught committing. I’m afraid of criminals (of whom terrorists are merely a subset, and rather a small one at that – drivers with cell phones are much more of a threat), not the government. Criminals actually hurt people. The NSA doesn’t. I’ve seen many innocent people’s lives ruined by criminals but never once by the FBI or NSA. Even Snowden is quoted only as saying that the massive collection of data has the potential to be abused and result in an innocent person being accused. True. Letting a doctor give you general anesthesia and cut into your body has the potential for abuse, and so does giving police guns, but there’s such a thing as necessary risk.

Unlike many former FBI agents, I don’t see Snowden as one of the worst breaches of U.S. national security, and I don’t see getting him back for prosecution as all that important. He’s languishing in his own prison of sorts living with no job in Russia. The irony is delicious. I still think he’s a traitor and should go to prison, but his current situation is close to that.

View all my reviews

Prices lowered on all Cliff Knowles novels

I’ve lowered the Kindle price on all the Cliff Knowles novels, including all the overseas markets (except where it was already the lowest permitted price). Amazon does not permit a lower price except for very limited promotional days. The U.S. price for all of them is now $2.99 and if you have Amazon Prime, which I’ve read is now subscribed to by more than 50% of American households, you can borrow it for free. You get a one free book download a month. If you have Kindle Unlimited, you also have all the free downloads you want. All the books are available there, too. You don’t need a Kindle device – there are free apps for all the desktop and mobile devices in common use, I believe.

See my Cliff Knowles Mysteries page for descriptions and links.

Solving a 6×6 Tri-Square Cipher

Recently I had occasion to tackle a 6×6 Tri-Square cipher published on a puzzle geocache.  I had some misadventures but eventually solved it. I thought the process might be both amusing and instructive to some, so I am writing up my experience. First, here’s the ciphertext in case you want to try it.

JEX PQD YHN 979 L00 ALT Q91 BKZ Q0B 990 SEX 8LW KTD 5RE 2RT PGW OWH 962 SZQ P4V CEI BRA KSN L0C JLD O9A EKS P6G CTO HIA 3T4 ZIP 2CY 0M8 3SQ 1U6 990 IEX O17 CSL T0A 7TO 6NA L1E S9J ALT O6R S0E 2R0 Z7G 9VT LUP 5RE P5Z YH3 8M0 Q11 6LW N9J MYP XVW TEP RBQ JUF 5HO PQD 7L4 G3D 2RJ 8PZ QGT 9VT ZCZ 4K7 1TQ S4H ZIA crib: characters sharing the same common environments

With the crib, it’s not very hard to solve the plaintext. That was accomplished quickly. But to get the coordinates I needed, one must enter the key(s) into an online checker. It is thus the process of recovering the keys that was challenging and is set forth here.

The first obstacle was my lack of computer tools for a 6×6 Tri-Square. I have one for the 5×5, but not the 6×6. This meant I either had to rewrite the one I had, or solve with paper and pencil. I chose the latter option. After solving the plaintext I had many equivalencies established in all three cipher squares, that is, I knew various letters had to be in the same row or column as other letters. For the leftmost square (Sq. 1) I saw an alphabetic series that told me the probable route used. I also saw what I thought was the beginning of the key. Bear in mind that with a 6×6 key, under American Cryptogram Association (ACA) rules the letters A-J are followed by the numbers 1 – 0. Thus BAD would be written B2A1D4 and an entire row or column may be filled with only three letters.

I decided to try finding the key to that square using a tool I had: a 6×6 Polybius square solver. The solver uses word lists. I ran it and was unable to find a one-word key that met all the known letter relationships in any of the ACA routes. That told me the key was not a single word, or, possibly, was not a common word. I reran the program using some very complete word lists still without results. I decided to move on to the other keysquares. I should mention that I have always positioned the squares differently from the way they are shown in the linked ACA page. I always put the square marked 2 (the one on top) underneath the corner square. I always thought of them in a different order for that reason, with the numbering of squares 2 and 3 reversed from the diagram. I will use the standard diagram numbering for reference here, but I still think of the corner square as being in the middle and thus number 2, with the others being 1 and 3.

With square 2 I worked in a similar way and was able to reconstruct a large part of the square. Once again I tried my polybius square solver and confirmed that no one-word key worked completely, but I got some keys that had almost all the right equivalencies. I was confident enough that I knew the first word of the phrase, that I modified my program to run through the word list again tacking the first word of the phrase in front of every word to make a two-word key. I got several good-looking keys this way that were almost perfect, but not quite. There were still some conflicts. I was able to produce a list of keys and select only the letters that were the same in all of them. Then I went back to square 1 and working with paper and pencil again, I was able to fill in more of that keysquare. The numerals actually helped finish off that square since if you know the route and can place a numeral, you know the letter that comes before it, and vice versa. Thus I completed square 1 first. I could tell it was a phrase, and while I recognized all the words in the phrase, the phrase itself made no sense to me. I had never seen those words in that combination. I searched the complete phrase online and got no hits on Google.

Going  back to sq. 2 again, I was able to complete the first, third, and fourth rows of the square, but still had gaps in the others. Still, working with the letters I was sure of in squares 1 and 2, I was able to fill in sq. 3 enough that I could tell the route and much of the alphabetic sequence. Eventually I was able to completely fill sq. 3. Again, I recognized the first word, but could not tell what the complete phrase was for sq. 3. After completing that square, I was then able to go back to sq. 2 and complete it. Like the others, I recognized the first word, but could not tell what the full phrase was for sqs. 2 or 3. Bear in mind that polybius square keys use condensed forms. That is, repeated letters are removed, so Banana Rebel will condense to BANREL which could equally be Ban Barbell or various other things. Since I was confident of the three initial words, and they were vaguely related, I tried searching them together online to find a common thread. I did not succeed. As it turned out, this was because I had two of the words wrong.

So there I sat with the three key squares filled in completely but did not know two of the phrases, and had no confidence in the one I thought I did have (sq. 1). Here, another factor came into play. The online checker for the geocache page shows how many successful and unsuccessful attempts had been made. It showed 5 successful attempts and no unsuccessful ones. This meant the solvers had known exactly what to enter and didn’t have to guess. This made me nervous, because even if I figured out the complete phrases, it seemed to me that there were six possible ways to enter the keys, at least if one were to enter all three phrases. I began entering the keys in their condensed forms one by one and kept getting rejected. Obviously I need to figure out the complete phrases. I felt somewhat bad spoiling the perfect 5-0 record on the checker, racking up multiple wrong guesses. It began to look like I was guessing randomly. A true solver, I thought, shouldn’t enter a key until he was certain of the answer, so I stopped. I felt very inadequate.

I don’t know how long I sat there staring at the keys before I finally realized that the square 2 key first word could be another word, one not in my word list. Using that as the first word, the remaining letters made a logical phrase. I searched that phrase online and something popped up immediately, something that made sense. I had to do a bit of research since I was unfamiliar with the subject matter, but once I did, I quickly knew what all three phrases were. It turns out that previously  I had had only the first word right in sq. 1, but none of the words right in sq. 2, or sq. 3. That is to say, I had the keys right in condensed form, but had not deduced the full form correctly, even a single word. The checker, I thought, required the full phrases spelled out. I zipped back to the checker and entered them in using what I thought was the most logical order. I got rejected once again. Aarghh! It must not be the three keys, after all, I thought. Instead, I became sure it was the common subject matter that connected the three keys. That was a much shorter phrase that was very recognizable. I entered that into the checker and still got rejected. I tried using variations of it and still no luck.

I gave up and began writing an email to the cache owner for a hint. As I was composing it I started to say I had tried entering all the keys in without luck as well as the connecting phrase in all its variations when I realized that I actually had not entered in the three keys in every possible order, only in the most logical order. I went back to the checker and entered the keys in using a different order, then another, and so on until I got to the only remaining possible order. I was sure that wasn’t going to work, either, and was composing my email in my head when I hit enter and saw the checker return with a thumbs up and the coordinates to the cache! It had taken me thirteen wrong guesses before I got the correct key despite having completely solved the cipher and the three keysquares. My total unfamiliarity with the subject matter connecting the keys was part of the problem, but not the only problem. I understand now how others could enter the full correct keys in the correct order without having to guess. If you have read this whole post carefully, you can probably figure out why, too.


A Puzzle to be Named Later by Parnell Hall

A Puzzle to be Named Later (Puzzle Lady #18)A Puzzle to be Named Later by Parnell Hall
My rating: 2 of 5 stars

I like the concept of this cozy mystery since I am a crossword fan and constructor as well as big mystery fan and writer, but I think it could be done better. Solving the crossword or the Sudoku won’t help solve the mystery, which was a disappointment. I found the main character, the so-called Puzzle Lady, to be irritating and unlikable. The main appeal for fans, I suppose, is the dialogue, but I found it forced and distracting from what I hoped would be an actual mystery plot. It turned out there was no actual plot. The Puzzle Lady’s outrageous personality and (snappy?) dialogue is the whole thing. You either like it or you don’t.

View all my reviews

Conspiracy nuts

I recently received an email from Audible.com with a graph showing where various genres of books are most popular. I thought the pattern looked familiar for one of them. The top graph shows the percentage vote for Trump; the bottom shows the relative popularity of conspiracy books. Please share, retweet and all that.

Computer cipher solving – Lesson 5½ : cribs revisited

I thought it might be useful to expand a bit on the use of cribs. In particular, I’d like to go into more detail on what I called Length scoring back in Lesson 5. Hence the captioned 5½ on this post. Here’s the original paragraph on that reposted for convenience:

Length scoring: I’ve found this to be a quite effective improvement to tetragram scoring, although they can be used together. Like tetragram scoring it has the advantage of not requiring any additional programming on individual ciphertexts, but unlike tetragram scoring, it does use up a bit of extra run time. It solves the problem I just mentioned in the previous paragraph. What I do is run the crib down the decryption and in each spot count the number of letters that are in the same place in both crib and decrypt. In the example above hisbeard and hixbeaqd have six letters in common. I then take the highest-scoring instance for the length of a decryption, 6 in this example. I typically take that number, subtract 3 (assuming it is at least 3),  and square the result, then add that to my score. In this example it would add 9 points (6-3 squared) to the score, the equivalent of a high-scoring tetragram. I use this method mostly on cipher types that have longer cribs. It has a good ability to hold hillclimbers close when they get close. It works well with a wide variety of cipher types, but not as well on transposition types or combination tramp/sub types like Bazeries or Myszkowskis. Those types may have the crib letters in close proximity to each other, but not in the right order, or with an extra letter or two between. I’ve considered writing something that will give extra points for those situations, but I haven’t been industrious enough to do that yet.

I think it’s worthwhile to follow a more typical example than what I used above. Let’s take AC-1159 in the MA2017 issue, a 6×6 Seriated Playfair. The crib is SELSEWHERETOESTAB. Clearly we can safely extend that to SELSEWHERETOESTABLISH, a crib of length 21. This method works better with longer cribs. Seriated Playfairs are not ideal types to use. As long as you have the correct seriation period, a trial decrypt that is getting close to the correct  solution will usually have some crib crib letters in their correct relative positions, however, this cipher type inserts extra X’s to avoid doubles so the crib and correct decryption may not match. Since I happen to know the crib does fit exactly in this case, I will use it. Now the point of this crib method is to identify a trial solution that has a section that “looks like” the crib, i.e. is more like the crib than random chance would dictate, and then boost the score of that trial decrypt in an amount relative to the degree it departs from random chance (and is thus likely to be generated by the crib) .

First we need to establish what random chance would dictate, since we don’t want to boost the score of a trial decrypt that shows some similarity to the crib here and there by chance. Since the index of coincidence in English is around 7%, random chance would dictate that if you compare the crib to any trial decrypt that is close to English in its index of coincidence and letter frequencies, 7% of the crib letters are going to match the decrypt letters. For this 21-letter crib, that’s about two letters. Of course this is only an average. Some will hit three, four or even more letters by random chance while in other cases there will be no matches. Bear in mind that, assuming we haven’t placed the crib by other means, we are not testing the crib in just one spot. We are running the crib through the entire trial decrypt and using the highest scoring spot. We don’t care how well the crib fits lots of different spots, but whether there is one spot where it really strongly seems to fit. Since the length of this con ct is 190 characters, that’s 190-21 or 169 comparisons. The question thus arises, given random variation, what can we expect the maximum number of letter matches to be by random chance in 163 trials? We need that to establish a baseline number.

There’s no doubt a way to do this using the index of coincidence, lengths, and known probability formulas, but for me it’s easier just to write a program that tests this. My somewhat limited testing indicates that for a ct and crib of this length, random chance will produce a best fit for the crib of 5 or 6 letters even if the crib is totally unrelated to the correct plaintext. So a positive result is really only indicated if your test shows seven or more letters that match the crib in the best spot, and even seven is within the range of normal. The shorter the crib and shorter the trial decrypt being tested, the smaller that number will be. Since most cribs and ACA cons are shorter than this example, I use normally 3 as my baseline since I don’t have a chart or formula that applies to all crib and ct lengths. Even though testing shows 6 is probably a better number to use for this con, let’s examine it using my normal 3.

The way I use this to score a trial decrypt is with a routine called CribFit that runs the crib along the trial decrypt and in each possible spot measures the number of letters that match crib with decrypt. I find the maximum number for that decrypt, let’s assume this placement: “qelmnuharptorrtingise”, which produces 9 letter matches with the crib.


Subtract the baseline number of 3, and square the difference. Here 9-3=6 so that would add 6×6 or 36 points to my score, a significant enough number to influence the hill-climbing function. So even though “qelmnuharptorrtingise” does not look to the eye like a good crib fit, the computer recognizes it as one. If 10 letters matched the score would increase by 49 points, and 11 would produce 64. As you can see, the change in decrypt score really starts changing a lot as a long crib appears in the decrypt.

E- book borrowing is increasing

As an author I have noticed a trend in book sales. Not only are readers moving toward more digital content and less paper, which is unsurprising, but they are also moving toward more e-book borrowing and less purchasing. I decided to graph the royalties I receive on my best selling book, Cached Out, comparing on a percentage basis the royalties I receive from those who borrow the book (either through Amazon Prime or Kindle Unlimited)  and those from sales of Kindle ebooks. This chart excludes the paperback and audiobook royalties.

As you can see, I now get more than a third of my Kindle royalties from borrowing. Not all of the increase is due to readers’ changing preferences. Amazon changed its formula for compensating authors for borrowed books in mid-2015. Before that time the authors’ “pot” was split up on a per book basis, i.e. someone whose 15-page children’s book was borrowed once would get the same amount as someone whose 800-page tome was borrowed once. Yielding to complaints, Amazon changed the formula to make the compensation proportional to pages read. (Yes, Amazon can tell how many pages of a book you’ve read with your Kindle/Kindle app, at least if you borrow it). It may be leveling off now, but the 2017 number only shows the average for the first two months. The March numbers aren’t out yet. By the end of the year it may be more.

This trend is consistent with what I read about other digital media. I hear that movie providers are going to earlier streaming because viewers don’t want to buy or borrow the DVDs/Blu-rays when they can stream. Even Snapchat reflects this. Younger people just don’t want to own or even handle digital media; they just want to view it and have it disappear when done. Why fill up your phone or reader with a bunch of books you’ve already read? They aren’t like music that you’re going to listen to again and again.

Aurora by Kim Stanley Robinson

AuroraAurora by Kim Stanley Robinson
My rating: 3 of 5 stars

Robinson gets good reviews from Scientific American, Science, and other hard science sources. His educational background, however, is in writing, not science. Perhaps this is why I was surprised that this novel seemed too heavy on the science jargon and too light on the storytelling. It is the story of the first interstellar flight of settlers, destined for a multi-generational trip to the Tau Ceti star system where they are to terraform and settle Aurora, an Earth-like planet there.

The author has conjured up some interesting and mostly credible characters that we follow throughout the book, despite the hurdles of time passage. The narrative is created by the AI that controls the ship and even calls itself ship. The setting is hundreds of years in the future. The central character is Theya, a child at the beginning of the book, much older at the end. For reasons unknown, the author chose to make her taller than anyone else on the ship, over two meters (at least six foot six) but her height became irrelevant and wasn’t mentioned in the second half of the book. She was also a slow learner, not very bright, scientifically ignorant, and somehow became the leader of the mission. (Remind you of anyone?)

The plot was interesting enough, but slow to develop. The author must have been paid by the word, like Dickens or Trollope. He was rather pedantic, too, choosing never to use a simple word like rut or gulch if there was a technical or scientific term for it. Use of robotic arms was waldo work. Every disease had to have the precise medical term for it, somebody’s syndrome, etc., rather than the common one. It made the reading rather tedious. There was an excess of every kind of babble – psycho-, techno-, medico-, and socio-.

It’s clear the author is promoting a certain pro-environmentalist world view, and part of the message is the harm we may do if we don’t straighten up our act here on earth. The book will appeal to hard SF fans with patience.

View all my reviews

Mother’s Cookies – no more Macaroons

I used to love Mother’s Macaroons, but I noticed my wife hadn’t bought them in ages. I put them on the grocery list but she came back saying she couldn’t find them. I looked online, and sure enough, they are no longer made. Mother’s Cookies, which was primarily a west coast brand, is now owned by Kellogg. Mother’s went bankrupt in 2008 during the financial crisis and amid an accounting scandal and sold its recipes and brand to Kellogg’s. Many of the varieties have been brought back, at least to the west coast, including English Tea Cookies and Oatmeal, but I haven’t seen macaroons in a long time. There are some indications online that they were available in 2015, but I can’t find any references newer than that. This wouldn’t be so bad if Kellogg’s replaced them with some other brand of macaroons, but my wife has not been able to find any in the four or five markets she frequents.

Kellogg’s, I’m mad at you! Bring back macaroons!

Retiree free time vs. Age

I’m still analyzing the data on retiree free time from my survey. The chart below shows a generally upward trend in the amount of free time as one ages. This is unsurprising, but what may be surprising is the degree of variation. The numbers on the left show the reported number of hours of free time per week of the respondents, or, more precisely, hours that fit into the nine broad categories I listed. I still think there are substantial free time activities not reported in the survey.

Between the World and Me by Ta-Nehisi Coates

Between the World and MeBetween the World and Me by Ta-Nehisi Coates
My rating: 1 of 5 stars

My book club chose this book, so I dutifully read it. The author has chosen to always be angry, which seems like an unpleasant way to live. I’d rather be happy, so I choose that approach. Still, everyone has to make a living and Coates has this professional victim shtick working for him. More power to him, but I wouldn’t recommend this to anyone who wasn’t a masochist.

View all my reviews

Our hilarious newsies – KQED Newsroom edition

I’d never thought of KQED Newsroom as a comedy show but last night there was some unintentional levity. For those of you not familiar with the show, it is a San Francisco-based news interview and analysis show, concentrating on local and California issues. One of the panelists was with the Asian Law Caucus discussing the effect of the new Executive Order on travel. First she said she and her colleagues went to the airport to work “on the ground” (which seemed amusingly counterintuitive). Once there it was “back and forth, back and forth, a real roller coaster.” I paused the recording and tried to envision a roller coaster that went back and forth while I suppressed a smile. The poor girl needs some work on her metaphors. I commented about it to my wife and she had a chuckle, too. Then when I hit the play button again the next thing the woman said was that a man had come up to her and said “I’m a U.S. citizen and my wife is abroad.” My wife, who is definitely not a broad, and I cracked up and couldn’t stop laughing. The panelist seemed totally unaware of what she had just said. As if that weren’t enough, her first name was Elica and she worked for the Asian Law Caucus. I told my wife her real name was probably Erica and she spelled it that way so her clients could pronounce it. (Okay, that last one was a cheap shot, but having done a year in Japan and studied Mandarin for a year, too, I appreciate Asian culture and people as much as anyone. The L and R thing is real, though).

Retiree activity analysis

A few days ago I posted a survey asking retirees to answer a few questions about how they spend their free time. If you are retired and haven’t yet taken the survey, you can do so here:

Retiree activity survey

Twenty-eight responses have come in to date. If you want to see the individual question results so far, you can do so here:  https://www.surveymonkey.com/results/SM-QZWP9YJG/ . There was a rather wide range of responses on most of the questions.

Here is my own summary chart showing the overall averages for all respondents over the age of 50:

Several things struck me about the results. One number not shown here is the average number of hours of free time per week for respondents. In this survey, the average for everyone over 50 (I excluded two below that age on the assumption they weren’t actually retired) was 44.29 hours. That’s about 6 hours per day. That seems really low to me. I sleep about seven and a half hours a night. I spend perhaps an hour a day eating, another half hour on changing clothes, showering, brushing teeth, etc. Chores like feeding the cat, doing dishes, mowing lawns, paying bills, running errands, etc. take maybe another hour a day on average. Throw in another hour a day to account for things that are for me rare but can take substantial time, like illness, major repair projects, travel, etc. So that’s 11 hours a day taken up with necessary stuff. That’s still 13 hours a day of free time, more than double what other respondents show. It makes me wonder why the discrepancy.

There are several possible explanations. I don’t have any grandchildren (yet- get going, kids!) and I know many retirees babysit grandchildren. I consider that a legitimate activity to put in the socializing category, assuming it’s voluntary and not done solely out of financial necessity or duty toward children, but it can take up large amounts of time and some respondents no doubt omitted that from their responses. Some retirees may still be employed at least part-time or have extensive business matters to handle. Some of that may be at least in part recreational in nature, not entirely out of financial necessity. I count all my novel-writing as hobby activity. As a business it’s a failure but I enjoy it. Some travel a lot – a category I omitted because I am trying to identify things I can do in my free time when I’m home. Please comment on this post if you have a big time commitment not included in the survey. Also, my wife does nearly all the shopping and cooking and shares other chores, so someone living alone might have more stuff to do of that nature. However, I suspect the single largest reason for the disparity is underreporting. I’m guessing that most people don’t realize how much time they spend on many things. I notice my own responses total 76 hours, less than 11 hours a day, 2 hours less than what I just estimated as my actual free time. Even though that’s one of the highest responses in the survey, it is still probably underreported. I must be guilty of this. I am probably underestimating the time I spend on the computer and TV.

This causes me to wonder about the accuracy of the results. One respondent accounted for 129 hrs/wk of free time. That’s more than 18 hours a day. That sounds like someone who is sleep-deprived. Perhaps some of that is accounted for by double counting. For example, Socializing can be a main reason for volunteer work, and people can watch TV and do crafts at the same time (my wife crochets constantly while we watch), and so forth. The person may have counted those hours in both categories. On the low end, there was someone with only 15 hours a week of free time. Ultimately,something is taking a great deal of most respondents’ time that doesn’t apply to me, and I am hoping to find out what that is. Maybe I’m missing a rewarding activity I just haven’t thought of. Please do leave comments to help me understand what that is.


Sinner Man by Lawrence Block

Sinner ManSinner Man by Lawrence Block
My rating: 1 of 5 stars

I only read about a hundred pages of this one, but it was enough to know I didn’t like it. I don’t believe in continuing to do something I don’t enjoy so I stopped reading. I’ve read some of Block’s Bernie Rhodenbarr series (e.g., The Burglar Who Counted the Spoons) and enjoyed the wit and cleverness of plot, so I had high hopes for this one, but I found it to be crude and rather offensive. Block at least avoids the all-too-prevalent four-letter words of other novels with indirect language, but still manages to descend into tackiness. (I pushed her head down below my waist and told her to pretend she’s a French girl). The fact that the main character (“hero?”) beat and then murdered his wife didn’t help. As a former FBI agent I know something about New York organized crime and the whole plot was preposterous. The notion that a new guy could just blow into town and be recognized as a “hotster” (whatever that is) and be given a mob assignment just doesn’t fly with my plausibility meter. From now on I’ll stick to The Burglar Who series with Block.

View all my reviews