What kind of cipher is this? How do I find out? I’ve avoided this topic up to now because it is both the most sought after question and the most difficult one to tackle. Determining what kind of cipher you are dealing with is certainly the necessary starting point and often the most frustrating. I wrote a series of columns for The Cryptogram, the magazine published by the American Cryptogram Association (ACA), called Tackling the Unknown. I believe I stopped after 19 articles. If you are an ACA member you can access those. I am not going to republish or even summarize all of those here. They were specific to ACA cipher types and not necessarily all that useful in other contexts, anyway.
The fact is, there is no magic bullet for this. Depending on the source, there is an infinite number of possible cipher types (since some people make up their own system which has never been seen before or since). For most modern computer or communication applications, the ciphers used are unbreakable by us mere mortals so I am restricting this post to the types you are likely to find in various puzzle venues like the ACA or geocaching unknown types or that weird message you found in your great-grandfather’s letters.
Step one is to determine the general class as either code or cipher, and if it’s a cipher, whether it’s substitution or transposition. With a code plaintext words or phrases are substituted by another word, symbol, numbers, or a collection of characters, usually much shorter in length than the original. “Washington, D.C.” might be encoded TSOPN. I’m not going to deal with codes. Ciphers work on individual characters or small groups (e.g. pairs) of characters. They may be substituted by other characters, or just moved around (transposed) or both. That’s what I will deal with here.
A pure transposition cipher is one that rearranges the letters (or other characters such as spaces, numbers, or punctuation). There are many well-known types such as the Route, Railfence, Amsco, Columnar, Double Columnar, Knight’s Tour, Scytale, and Turning Grille, ciphers. Some are (or once were) useful in the real world, but most were not. As long as you know the language and the plaintext is normal, it is usually fairly easy to spot such ciphers because you can see lots of frequent letters like ETAOINS in English. However, I have devised a test for measuring how normal the letter frequencies are which I call the Normor test. If you have text you’d like to test, you can paste it into my cipher test page. Generally transposition types will score less than 100 on that test. There is a link to an article on that page explaining in more detail how to use the test and interpret the results.
If the test indicates it is a transposition type, how do you know which one? I don’t have the time or patience to go into detail, but there are several things to look at, such as the number of letters (a square of an even number, e.g. 64 or 100, might indicate a Grille, a prime number might eliminate a Route, etc.) Look at the history of the source for clues as to types used before, types probably known to the author, etc. If it’s a puzzle, clues as to type are probably there. The best way to diagnose which transposition type it is in my experience is to run it through autosolvers of the various types to see if plaintext emerges. Most transposition types yield rather easily to brute force autosolvers.
For simple substitution types like cryptograms or patristocrats (cryptograms without spaces or punctuation), the Index of Coincidence (IC) will usually identify it as such because the IC will be the same as the original text. My test page also measures the IC. For other types, including ones that mix substitution and transposition, there are various tests, some specific to a particular type. I am not going to go through them all here, but there are resources available. A good website for diagnosing the type is BION’s gadget page. See his various ID tests.
Beyond that, it boils down to using your brain and reading up on the different kinds of ciphers. Some so-called ciphers are actually hoaxes. Others may be too short or too convoluted and unique so as never to be solvable. All you can really do is to keep trying different things and hope you can at least identify the type so you can start solving.