Monthly Archives: September 2016

Anagrams on the news – Swing States edition

Pennsylvania Savvy? No
Ohio Ha ha, I win
Florida Hi, Israeli clan
North Carolina Loo law
Iowa Corn-vocation
Nevada In cardroom
Colorado Dope Inn
Virginia SOG II
Wisconsin Arian
Michigan Fading

I probably don’t have to say this, but for those unclear on the concept, no, the individual lines don’t anagram, but if you take all the letters in Column A (States) and rearrange (anagram) them, they make Column B

Marketing a self-published book

I get quite a few questions from friends and acquaintances about how I sell my books, often in an ill-disguised form of the real question: if they were to write one, how could they get it sold or at least read? It’s a legitimate question and I’ve learned a lot about marketing my books, but I certainly have no magic bullet.

For starters, I’m a big fan of Amazon and its marketing tools. I wrote my first book, Held for Ransom, in 2001 and was lucky enough to get an agent to take it. It got read by 14 different publishers over the next 18 months. I got all sorts of feedback, both positive and negative, but much of it contradictory. In the end no one wanted it. Fast forward to 2011. I’d written about half of my second novel but given up on it largely due to my bad experience with the first one. Then my wife mentioned self-publishing to me. I did some research online and saw how easy and cheap creating and uploading a digital book was to both CreateSpace and Amazon. I cleaned up my first book, created a cover using CreateSpace’s online tool, and uploaded it to both places. It sold pretty well at first, but mainly to friends, former co-workers, and so forth. I was hooked.

I’ve now written a total of six novels and I’m working on my seventh. Marketing them hasn’t been easy. I’ve tried using Smashwords. It’s tricky getting your book in the premium catalog and when I have succeeded, I’ve found that it doesn’t really sell well in any other market besides Amazon. If you make your book available there, you can still sell on Amazon, but you cannot join KDP, which requires exclusivity. Amazon’s Kindle Direct Marketing (KDP) is where the marketing tools are. KDP grants you 5 days every six months to make your Kindle book free. You won’t make royalties during that time, but your book will get exposure from Amazon and may receive hundreds or even thousands of downloads, especially if you list it with various marketing websites as a free or bargain book. I’ve found that only the first two days are worthwhile. You get onto various lists then and downloads max out by the end of the first day, although they usually continue well into the second. I’ve always seen my sales go up when I use this tool, especially among the other books in the series, but even right after the free promo with the free book. Listing it with those marketing websites can be done cheaply or for free if you shop carefully, but most have very little effect on sales or downloads. You can also waste a lot of money with them. Readers who want freebies know how to go Amazon’s own free books site. BookBub is probably the only really effective one and it’s both expensive and very hard to get on unless your book is already a big success.

Amazon gives you a great deal of control over the product page, too. You can add reviews, create a detailed Author page with a photo and link to your blog. Amazon also has paid advertising options through Amazon Marketing Services. I’ve used AMS, Facebook, and Google AdWords. The only one that ever produced increased sales was AMS. The other two are subject to abuse, overcharging you and placing your ads on unsuitable pages. The reality is that people don’t search Google or Facebook for a good book to read. They search Amazon. It’s the 800-lb. gorilla in this space. Its tool is much easier to use than the other two sites, too. You have to be careful not to bid too high, though, or you will not recoup in sales what you spend. The main advantage is that your book will get exposure in places you don’t normally get any and with readers (i.e. book buyers!) who like your genre. A similar promotion at 99 cents is also available. Yet another great Amazon tool is the Kindle Online Lending Library (KOLL). More about that in a future post.

You can’t rely on Amazon alone. You have to use social media, especially free media. It may be distasteful to hype yourself, but someone has to do it if you want anyone to buy, or even hear about, your book. Join Facebook and Google+ groups, real groups of actual people who discuss something, not just book marketing groups, and post enough there to have credibility on the subject matter. Then when your book comes out, find a way to mention it without sounding too much like an advertisement. For me, that has meant geocaching and law enforcement groups. On occasion someone objects to commercializing the group, but if you are a regular contributor and don’t overdo it, you can get by with it. If you’ve written a cooking/recipe book, join a recipe-sharing or foodie group – preferably lots of them.  You get the idea. I have a Twitter name but I doubt I’ve ever sold a book through Twitter. I do get lots of marketing spam from book promotion sites, proofreaders/editors, audiobook voice actors, and so on from Twitter, but from time to time some of it is useful in identifying another free and easy promotion route.

Ultimately, what you need to get people to buy your books is good content, a good cover (more important for fiction than non-fiction), and lots of good reviews, especially on Amazon. Once your book’s review total creeps near triple digits and maintains a 4-star plus rating, the other tools, like the free and 99 cent promotions, work much better.  How do you get lots of reviews, you may ask? That’s a subject for another post. It’s a topic that is very hot among self-published authors.

Lastly, develop a mailing list. I now have over 200 fans on mine. That is the cheapest and easiest way to get sales on a new book and can be used for other promotions. It’s slow going at first, but you can get email addresses several ways. First make your email address available in your book (e.g. about the author page), through social media, etc. Collect the email address of every fan who contacts you. If you have a blog, put a contact form or email address on it. If you sell through your own website, as I do for my paperbacks, accept PayPal because you’ll get the buyer’s email address. Hold a contest with a book giveaway, preferably several copies, and spread the word through social media and require them to fill in a form. People are much more willing to give up their email address if they can get something they like free.

Slow Burn by Ace Atkins

Robert B. Parker's Slow Burn (Spenser, #44)Robert B. Parker’s Slow Burn by Ace Atkins
My rating: 3 of 5 stars

If you’re a long-time Spenser fan, ignore this review; you know what you’re getting and no doubt that’s what you want. I’d never read a Spenser mystery nor seen the TV show based on the books, so this was new to me. But not very new, since it followed a familiar formula. Spenser is a tough guy Boston P.I. with a rockin’ bod, hot girlfriend (Susan), a loyal dog (Pearl), and two tough guy sidekicks (Hawk and Z) with equally hard bodies and the skill sets to match. Together they take on a trio of wannabe firefighters who resort to arson to get their flame fix. Spenser and gang win every fight with ease, fearlessly rip off the local Mafioso and his entire gang, and donate his drug money to an orphanage. Unsurprisingly, he identifies the arsonists when the entire Boston Arson Squad and Police Homicide Unit can’t.

Ace Atkins, the author, was chosen by the Parker family to continue the long-running Spenser series originated by the late Robert B. Parker. I have no quibble with the formula. It obviously works for many, but I can’t say it made me want to read more. Spenser has a certain style, with plenty of wisecracks and constant restaurant hopping. He must be made of money since no one ever pays him and he lives a luxurious lifestyle. Ah the glorious life of a P.I.! Right – I was one briefly after I retired from the FBI.

One tidbit I found interesting: the plot mentions real-life reporter and author Hank Philippi Ryan. I read and reviewed her book The Wrong Girl (2 stars) a couple of years ago. Atkins and Ryan share one rather obnoxious stylistic trait: they constantly describe every character’s attire, with special emphasis on designer names. Now I’ve seen that in several chick lit mysteries written by women, but this is the first time I’ve seen it from a male writer. Yet here it is not only from a male author but with a testosterone-laden male protagonist. It wouldn’t have surprised me if it had come from a gay author with a gay leading man, but what the hey, I think the Spenser series is intended to appeal to women, not men. Baryshnikov had a rockin’ bod, too, and women loved him. If only he could have reciprocated.

The book had enough style to keep me reading to the end, but no suspense and nothing to make me pick up another Spenser. By the way, the full title is Robert B. Parker’s Slow Burn, a Spenser Novel in case you have any trouble finding it under the name Slow Burn. My library had it alphabetized under Atkins, not Parker.

View all my reviews

Where Cliff Knowles fans are located

Click on map to see full screen version. Colored countries show sales of Cliff Knowles mysteries.

This map shows all the countries where I can verify that Cliff Knowles Mysteries have been sold or downloaded. There are probably several others, but I do not get country-specific reports from Amazon for most countries. This chart was constructed partly from Amazon sales reports but also from personal emails, Facebook or other forum comments from readers, and so on. I have not been tracking this over the years, so I may have missed some. I’m pretty sure I heard from someone in Mexico who bought it, but I couldn’t find that email if there was one.

The individual colors have no particular significance other than to distinguish a country from its neighbors. If your country is not on the map and you want to get it there, you can fill in the comment form below, or (faster) use the contact form on the About the Author/Contact page (see top menu) and be sure to tell me what country you’re in. Of course, you can also buy the book. In general Amazon only reports book sales by country where they have established a specific domain for sales there. Many countries buy from (i.e. the U.S. domain) or the domain of a neighboring country. Sweden and Norway, for example do not show up in my sales reports, but I know several readers there downloaded some of my books from the UK or US sites based on email contacts. It’s gratifying to see that every continent except Antarctica is represented.

The Flight of the Phoenix by Elleston Trevor

The Flight of the PhoenixThe Flight of the Phoenix by Elleston Trevor
My rating: 5 of 5 stars

A storm … a bleak, dry landscape inhospitable to human life … the only things between life and certain death: engineering knowledge and teamwork. Sound like The Martian? The similarities are remarkable, but Elleston Trevor’s masterpiece of 1964 recounts the tale of an airplane crash in the Libyan desert, not a spacecraft on Mars. I’ve given several five star reviews lately, but this one tops them all without question.

The personnel aboard the plane must coalesce into a team if they are to escape. The pilot Towns has a crisis of confidence, knowing that his bad judgment in flying into the sandstorm instead of turning is cause of several deaths already and could doom the rest of them. Harris and Watson, a captain and sergeant in the British army, contrast starkly, one haughty and condescending, the other full of simmering resentment of the upper classes. Stringer, an arrogant young aircraft designer, is determined to fashion a working craft out of the remaining parts of the wrecked plane so they can fly out, but his petty superiority enrages Towns who is used to being in charge. Moran, the navigator, has the people skills to serve as moderator, counselor, therapist, and negotiator. The rest of the survivors have their own issues and personalities which are so believable as to resonate with the reader. We all know someone like each one of them.

This is a book about the human condition, the difference between survivors and defeatists, between technical brilliance and wisdom. The author is clearly very knowledgeable about aircraft and their design. Engineers and vintage aircraft buffs alike will appreciate the detail with which the author relates the technical and scientific challenges that are dealt with.

The survivors encounter thirst, starvation, hostile Bedouins, sandstorms, and their own human frailties. The plot is expertly filled in equal measure with riveting suspense and keen observations about what makes humans both wonderful and horrible creatures.

I listened to the audiobook. The reader is outstanding. Each character comes alive as a unique individual. I highly recommend this book.

View all my reviews

Guilty Pleasures – snack food survey

I like to run a survey now and then just for fun. Try this one. I’ll analyze the responses eventually.

To leave a comment or get on my Cliff Knowles Mysteries mailing list fill in the form below

Born to Run by Christopher McDougall

Born to Run: A Hidden Tribe, Superathletes, and the Greatest Race the World Has Never SeenBorn to Run: A Hidden Tribe, Superathletes, and the Greatest Race the World Has Never Seen by Christopher McDougall
My rating: 2 of 5 stars

The term purple prose is usually reserved for works of fiction but I find it apropos here. This disjointed work is more a collection of anecdotes gussied up by hyperbole than an actual story. Here is how the author describes the Barrancas (Copper Canyons) of Mexico:
“The deeper you penetrate into the Barrancas the more it feels like a crypt closing around you.”
“…it looked like a world-ending explosion frozen in stone.”
And he described a runner known as the White Horse this way:
“a dusty cadaver”
“his legs shifted and reformed like molten metal”
“he’s so long-limbed and lean-muscled he looks like an endoskeleton of a bulkier beast”
Leadville, Colorado is “the wildest city in the Wild West,” “an absolute death trap.”
I realize ultrarunning, running distances of 50 miles or more, is a sport of excesses and extremes, but I got sick of finding the absolute ultimate this and that on every page. I made it about halfway through before giving up. If you’re obsessed with running you might make it through happily enough, but I guess I’m more of a half-marathon kind of reader.

View all my reviews

Savage Run by C.J. Box

Savage Run (Joe Pickett, #2)Savage Run by C.J. Box
My rating: 3.5 of 5 stars

In this second book by Box in the Joe Pickett series, he continues the theme of environmentalists vs. ranchers and loggers. Box appears to be trying to stay the middle course since he portrays both sides as extremists, willing to kill or maim the others. Joe is the same straight arrow game warden just trying to due his duty while finding a way to support his family on his meager salary. Joe has been called a “flawed character” by reviewers but that term is not applied in the normal way. He’s not a rule-breaker, an alcoholic, a cocaine addict, or a rogue cop. His flaws are of a milder, more benign sort. He lacks self-confidence. He’s not a good shot, at least not with a pistol. He fails to grasp evidence in plain sight when he should and thus makes it impossible to prosecute a trophy-hunting poacher. He’s almost wimpy for an action hero.

The best part of this book, like the first one, is its eloquent description of the gorgeous setting of the Bighorn Mountains of Wyoming. The author’s personal knowledge makes the telling credible and authentic while his skill as a writer makes the prose almost lyrical when describing the land. The plot is less believable than his first book and the gore is both more plentiful and more unnecessary. An exploding cow? Really? Huge trees flying through the air? I have a whole list of improbabilities, but they would be spoilers, so I’ll skip them. I liked the book, but I thought Box was trying too hard to be edgy and action-oriented, which are just euphemisms for violent and gory (probably at his publisher’s urging). His protagonist’s appeal is his sweet, semi-naive honesty and peaceable nature. He should have stuck with that strength. I have to drop the rating on this book from my first one on Open Season.
Open Season (Joe Pickett, #1) by C.J. Box

View all my reviews

Computer cipher solving – Lesson 6: interactive solvers

I’ve had a fun three days at the ACA convention in Dallas but I’m back and ready to finish up this topic. There is way too much involved in computer solving to cover it all in my blog, so this is going to be my last post on the subject for now. I just hope that I’ve given you enough inspiration and maybe some clear understanding of how you can solve ciphers with computers.

Perhaps the most valuable and easiest computer solving program you can write is an interactive solver that mimics what you would do to solve it on paper. For example, with a simple cryptogram, a good interactive solver will allow you to input the ciphertext, display it in a handy way and have a way to input a letter you want to substitute for another, such as typing the letters in text fields or by clicking a mouse on an alphabet, etc. When you tell it that G=A it will go through the program and display above (or below if you prefer) an A over every G. It will also display a key alphabet equivalence. The advantage of using a computer program is that it makes all the substitutions or transpositions immediately and error-free . It should allow you to delete or undo some action you’ve tried. Such a program avoids all the transcription errors and messy erasures and can provide some useful statistics such as the frequency count. Even if you have an automatic solver (like a hillclimber) for a cipher type, you should probably also have an interactive solver since the automatic ones aren’t always so automatic.

I include in this class programs that perform a single useful task. A good example would be a crib placer. Many puzzlers provide a crib (short plaintext segment) along with the cipher, but usually the location is not known. A crib placer can tell you all the places it can fit. Another example would be a program that computes useful statistics such as letter frequency count, index of coincidence, Normor score, and so forth.

Many serious programmers are used to working in a command-line environment. That’s fine if that’s what you like, but I really like graphic user interfaces (GUIs). A decent GUI will allow you to see plaintext in one color and ciphertext in another. It will allow a large or small font. It provides nicely labeled input and output boxes. It will allow you to interrupt or pause execution when things look promising. It will display everything in nice neat rows or columns. It can highlight things of interest such as a possible placement for a crib, or a count of the number of trial decrypts it has done. As I mentioned earlier, I’m not much of a programmer. I’ve never worked as one and I don’t know the languages that are now popular, but I know that I solve ciphers better when I can see things displayed in the way I want them displayed. Cut, paste, click the button, and watch it go. I’ve learned enough to do that.

So how do you write one? For each cipher it will be different, but most programmers will not have trouble figuring out how to do that. For non-programmers, though, you can get much of this functionality with programs you already have available such as spreadsheets, word processors, and, of course, an Internet full of interactive solvers.

Computer cipher solving – Lesson 5: Cribs

I use cribs in cipher solving at least four ways: 1) Research; 2) Tetragram scoring; 3) Length scoring; 4) Restrictive coding.

Research: I use the crib to determine the subject matter of the plaintext. That allows me to guess other words in the text or recognize likely letter sequences or keys. For example, if the crib is “his beard,” I might be inclined to look for or at least recognize the words Lincoln or Hemingway in the plaintext or key. I can use the crib content to try to extend the crib. Google Ngrams, for example, will tell you what words most often follow a sequence of other words. Sometimes I find the full plaintext online, although I rarely do this except for Xenocrypts since I don’t want to spoil the fun of solving. For some of the toughest ones, though, it may be the only way. Pencil and paper solvers use cribs the same way.

Tetragram scoring: I mentioned this in an earlier lesson. At the beginning of the program I load the tetragram frequency data into an array. After doing that I add points for each tetragram that appears in the crib. In addition to making my hillclimber or other program recognize a better decryption, it has the advantage of not requiring significant additional run time. The extra points don’t have to be added in during each tetragram lookup, only once at the very start. This method has a minor drawback. Sometimes the program may tend to lock in to a false solution that happens to produce the crib, or some portion of the crib, but this is rare and is usually short-lived. You can always rerun the program without a crib being entered. It has another disadvantage: it will not recognize a close match if there are no full identical tetragrams in the matching section. For example, if the crib is “hisbeard” and a trial decryption produces “hixbeaqd” the tetragram scores will be the default ones and not recognize this as coming close to the crib.

Length scoring: I’ve found this to be a quite effective improvement to tetragram scoring, although they can be used together. Like tetragram scoring it has the advantage of not requiring any additional programming on individual ciphertexts, but unlike tetragram scoring, it does use up a bit of extra run time. It solves the problem I just mentioned in the previous paragraph. What I do is run the crib down the decryption and in each spot count the number of letters that are in the same place in both crib and decrypt. In the example above hisbeard and hixbeaqd have six letters in common. I then take the highest-scoring instance for the length of a decryption, 6 in this example. I typically take that number, subtract 3 (assuming it is at least 3),  and square the result, then add that to my score. In this example it would add 9 points (6-3 squared) to the score, the equivalent of a high-scoring tetragram. I use this method mostly on cipher types that have longer cribs. It has a good ability to hold hillclimbers close when they get close. It works well with a wide variety of cipher types, but not as well on transposition types or combination tramp/sub types like Bazeries or Myszkowskis. Those types may have the crib letters in close proximity to each other, but not in the right order, or with an extra letter or two between. I’ve considered writing something that will give extra points for those situations, but I haven’t been industrious enough to do that yet.

Restrictive coding: This term refers to the use of information from the crib to restrict the search space or execution time on a solving program. It can take many forms. For example, if you know the crib and its placement, you can write code into your solver that ignores trial decryptions that do not have that crib in that location, thus saving the time of  scoring them and the problem of high-scoring false solutions crowding out the correct solution. I have a polybius square program I use to produce possible keys for many cipher types like Bifids, Two-squares, Playfairs, etc. I have a section in the source code where I program in the various letter relationships that I learned from the crib placement, such as requiring specific letters to be in the same row or column, etc. Thus it can be used to search for keys, not just on trial decryptions. The obvious disadvantage is that it requires programming for each individual cipher that is attacked, which can be fun but also is subject to the usual frustrations of bugs in the code and the time it takes to get to a solution. It helps if you can restrict the area of the code that is modified to a single compact module so that you don’t have to find (and later find to undo) all the scattered modifications.

Computer cipher solving – Lesson 4: Simulated Annealing

Hillclimbing has an inherent problem: the local maximum. I’ve described this in previous lessons. One way to deal with it is is simply to start over again with a new scrambled key repeatedly. Scryer calls that Shotgun Hillclimbing, I think because a shotgun scatters its shot randomly over a target surface.

Another approach is Simulated Annealing. If that Wikipedia description looks intimidating or unclear, I’ll simplify it as best I can. For our purposes in hillclimbing the idea is really pretty simple. When you compare a new key to the previous one, instead of swapping keys only when the new one is better, swap sometimes when it is the same or even a bit worse. At first you can do this a significant number of times, but as your loop progresses, you must gradually reduce the number of times this happens and the size of the downward gap you are willing to accept. For example you might start by randomly accepting 20% of the time swaps that score 5% worse. After a few hundred or a few thousand iterations, though, those numbers might be 3% worse 10% of the time, and so on. Eventually you will end up not accepting worse-scoring swaps at all. In other words, you will be doing straight hillclimbing at the end. The progression from many to few “bad” swaps is made by reference to a table and a variable called the Temperature (T). When T is high, many relatively large “bad” swaps are accepted, but as T is lowered, fewer such swaps are accepted. The table contains values used to specify the random breakpoint numbers for the scoring gap. Every  X number of iterations T is lowered until eventually you are just hillclimbing. You will be required to experiment to see the range from high to low of T based on your cipher type and scoring method.

This approach allows you to jump around the landscape a lot more at the beginning trying to get onto the right mountain before settling in on a steady climb up. The improvement is quite marked for some ciphers. After a while, though, if no solution comes, you will probably need to shoot that shotgun again and start over with a newly scrambled key and T boosted back to a high number.

My description of simulated annealing may not comport with a strict mathematical description. There are other, similar methods that use other terms for technical reasons but I lump them all together as simulated annealing. The concept is the same: allow your program to make some key changes that appear bad at first, but as you improve your result, limit those allowances.

Computer cipher solving – Lesson 3: scoring decryptions

A hillclimbing program needs to know when a decryption is better than another. So do other types of attacks. So your program swaps two letters of a key and decrypts, but the decryption looks like gibberish to the naked eye, just as the previous one did using the old key. Which key do you save? There are several methods. I mentioned locating the crib in the previous lesson. But the crib and other normal text is likely to appear only at the end of a successful hill climb, i.e. as you near the top. A better approach to start with is to use statistics.

There are various statistical measures that can be used. They include the Index of Coincidence, the Digraphic Index of Coincidence, the Normor score, word list scoring (i.e. counting how many words appear in the trial decryption, or what percentage of the letters are contained in the words that appear), and others. But the most useful measure I have found is n-gram (or n-graph) frequency scoring. An n-gram is a sequence of n adjacent characters where n is an integer. I have successfully used digram and trigram scoring in the past, especially when I was using Turbo Pascal on a 16-bit machine and the language was not capable of a full tetragram array structure. There were workarounds, but the big breakthrough in efficiency and effectiveness came with the advent of 32-bit (and now 64-bit) machines capable of holding frequency data for tetragrams, i.e. a 27x27x27x27 data structure. I include spaces as well as the 26-letter alphabet in my data, although most others I know in the ACA do not, so their arrays are a bit smaller. I have not found that using 5- or 6-gram frequency data is any better than tetragram, so that seems to be the sweet spot.

The basic idea is to examine the decryption taking it in overlapping 4-letter (or 4-character) sections and adding points to the score based on the frequency of the tetragram in normal English. For example, if your trial decryption were ‘frumqxing…’ you would look up the score for “frum” then add the score of “rumq”, “umqx”, etc. until done. Although these are not words, some of them will have a non-zero frequency in English and thus some score. For example “frum” appears in the phrase “cupofrum” and the word “frumpy.” So how do you know what score to give each tetragram? There are tables of data out there on the Internet if you look hard enough, but I recommend  collecting your own data. To translate raw frequency data into points you will have to decide your own method. I know some people add the logarithms of the frequencies. (Adding the actual frequencies tends to overweight the most frequent tetragrams). Since I do my cipher solving almost exclusively on ACA ciphers, I use my own collection of solved ACA ciphers to collect the data and I give each observed tetragram a score of 1 to 9. I don’t have a fixed algorithm for where the breakpoints are; I just picked ones that seemed to divide up the set into useful-sized chunks. Such data is available for other languages, too. Obviously the source of the data should match the type of text you expect, not only in the language used, but whether it is technical, military, dialogue, etc.

These methods are not mutually exclusive. When scoring a decryption you can use tetragram frequencies but also add some points if the crib appears, or you can boost the score of the tetragrams that appear in the crib. I’ll discuss use of the crib in scoring more in a later lesson. Use of the IC or DIC in combination with n-gram scoring sometimes is helpful, too.

Computer cipher solving – Lesson 2: Hillclimbing continued

Hillclimbing sounds fairly straightforward, but there are issues to be aware of. For example, step 1 is to pick a random key. Why not use a standard key, such as the regular alphabet A-Z for a cryptogram? After all, you end up swapping letters around randomly anyway. Actually, you can do this but it will usually take longer for statistical reasons too complicated to explain, and you may well get stuck on the same local maximum every time. See the last paragraph for more on this. More important is that you modify the key (step 4) randomly. For example, you cannot just swap every letter with every other letter in an orderly fashion, i.e. first with second, then first with third, fourth, etc. That first comparison you made will only be a valid test for that one arrangement of all the other letters. If you were to compare those same two letters in the same two places after many of the other letters have been swapped around, they may not produce the same result. If you don’t want to take my word for it, just try it yourself and you’ll see that choosing two letters randomly works better.

Another issue is how to modify the key. I’ve used swapping two letters in the key as an example, and that works fine for simple substitution ciphers because we know what the key is – 26 different letters, A-Z, in some order. But a different type of modification may be necessary for other cipher types. For a Playfair or other polybius-square based key you may need to swap entire columns or rows sometimes instead of letter pairs or use a mix of methods. Sometimes making a three-letter switch works better. For a Myszkowski, swapping doesn’t work because the key contains an unknown number of repeated letters. You might need to pick a random letter in the right range (but not necessarily another key letter) and replace one of the key letters for step 4. You need to experiment and adapt to the specific cipher type.

A very important issue is how to tell if you’re program is producing better keys. Metaphorically speaking, what do you use for your altimeter as you climb the hill? For that matter, how do you know that you’ve solved the cipher? This method produces thousands or even millions of decryptions in a matter of minutes. You can’t examine them all by eye and make a personal judgment call. There are several useful methods. One simple method applies when you have a crib, i.e. bit of known plaintext. Suppose you know the word “nuclear” appears in the text of the message, but you don’t know where. That’s called a crib. You can test your decryptions by checking to see if the word nuclear appears in the decryption and either save or display those decryptions and the keys that produced them. That might work pretty well with a short simple cipher like a Baconian, since if the crib is produced, usually the rest of the decryption will also be correct or close enough that you can figure it all out. But for other types this isn’t enough. You can produce that one word many ways and the rest of the decryption may be useless gibberish. Not only that, you may have the entire solution almost perfect but for the word “nuclear” misspelled as “unclear.” Your program will skip over that and never show it to you since the crib isn’t there. What you need for hillclimbing and most other forms of computer attack is a scoring method. You need to give points for improvements. For step 6 you will save a key if it produces a decryption with a higher score and reject it if it doesn’t. I’ll discuss in another lesson how to do this.

Before we leave hillclimbing, though, there is another big issue to deal with: local maxima, i.e. those incorrect decryptions that can’t be improved further by the incremental changes to the key. Remember how I mentioned that the paratrooper can end up on the top of the wrong hill or rise and not be able to climb any further? That happens constantly in hillclimbing, which is why I put in step 8. You can simply give up and go back and scramble your key completely and start over. If you always start with a straight A-Z alphabet or other “standard” key, that’s like always dropping onto the same exact starting point, the small hill  next to the correct mountain and you will never get to the radar station by climbing the hill. Start over with a random key. Computers can repeat a process millions of times very quickly, so this method generally works pretty well, but there are more sophisticated, more effective methods of getting off, or avoiding altogether, the hill. That, too, is a topic for a future lesson.

Computer cipher solving – Lesson 1: Hillclimbing

Since I’m going to the ACA convention in a few days, cryptography is on my mind. I thought I’d depart from my regular book reviews and give a few brief lessons on computer solving of ciphers. I only apply these methods to simple ACA ciphers like cryptograms, Playfair, etc., but the principles and general techniques apply to many sophisticated encryption methods. I’ll only use pseudocode since programmers use many different languages, and I only know a bit of some older types (BASIC, Pascal, Delphi). First, I’ll discuss a common method of attack: hillclimbing.

Before I describe how to do it, you’ll understand it better if you know why it’s called hillclimbing. Imagine you’re a paratrooper who is dropping onto an enemy’s mountain range. Your mission is to get to the top of the mountain to disable his radar station and anti-aircraft guns. You parachute down in the dead of night. It’s foggy and pitch black. You can’t turn on a light and can’t see exactly where you are or which direction the mountaintop is. Your only tool is a very sensitive altimeter with a lighted screen. You decide just to go up. You measure your altitude and then take a step, then look at it again. If you’re higher, you repeat the process from there. If you’re lower or the same altitude, you step back to where you were and step a different direction. By always going higher you will eventually reach the top. But the top of what? You may be on a foothill or ridge or side bluff or even the wrong mountain. After all, the mountain isn’t a smooth cone rising from a smooth plain. However, you aren’t the only paratrooper. If hundreds of your fellow paratroopers land all over the area, scattered widely, one or more is bound to land close enough to the top so that this method of always climbing will get him to the radar station. That’s the principle behind hillclimbing. Take a random shot at a solution, and improve it gradually until you can’t improve it any more. Do that many times until one of the attempts produces a solution.

I’m assuming you know what type of cipher you are attacking, or at least think you know, have a cipher to decrypt, and have a decryption engine for it. All you need is the right key. Hillclimbing code will:

  1. Pick a random key
  2. Decrypt using that key
  3. Measure how good the decryption is ( a future lesson will discuss how to do this)
  4. Make a small change in the key (e.g. swap two letters)
  5. Repeat steps 2 and 3
  6. If the new key produces a better decryption, keep it, otherwise keep the first one
  7. Repeat from step 4 until you get no more improvement
  8. Go back to step 1 and repeat the whole process

The process is a looping procedure that ends only when you stop it or it meets some test or limit you have programmed into it. If you’re lucky, or good, you stop because you have found a solution. That’s it for today. You now understand the idea behind hillclimbing.

New American Cryptogram Association (ACA) website

The ACA just launched a redesigned website. The main domain name hasn’t changed but the files are in new directories and many have been renamed, so you cipher fans may encounter broken links. Please update your own if you have any on your puzzle caches, etc. Here’s the URL for the new Cipher Types page:

Speaking of the ACA, I will be attending the convention in Irving, Texas next weekend and presenting a talk on a method for solving Foursquare ciphers. I hope to see some of my long-time ACA friends and make some new ones.