Ever wonder how they know?

You've probably been creeped out by this. You Google something, maybe thinking about a gift for your sweetie. Then you start noticing ads for that showing up in your Facebook feed. Or wondered how the damnyouautocorrect works. Or how Neflix knew that House of Cards was going to be a hit. Or why England had a monetary crisis in the late 1600's. Well, maybe not that last one.

AIQ by Nick Polson and James Scott will tell you why in clear readable language, plus a little tiny wee almost minuscule bit of math, in an easy, reader friendly manner. Really, it's all good.

So for example, where they use math and I won't. The monetary crisis. Most people know the problem with silver coins back in the day is they were subtly different sizes and weights. People would shave off a sliver and pass the coin. Again and again. Better manufacturing and milled edges made that more difficult.

Then there's the possibility that if the coins are supposed to be a certain weight, within a certain tolerance over or under, what's to stop the people in the mint from issuing coins that were ever so slightly on the under side, and pocketing the difference?

It turns out they were using the wrong formula in the Trial of the Pyx to figure out if there was a problem. They had one of the smartest guys in history in charge, Isaac Newton, and he missed it. They didn't know that the size of your sample is important. (I'm sparing you the math, even though it's easy, but because I don't want to play with the Greek letters on this keyboard. After all, I've only had one coffee as I'm writing this.)

And autonomous cars, how do they know where they are? This is critically important. In order to navigate to some other place, a car needs to know exactly where it is now, and now again, and again, and again, many times every second. Humans do this too,  some better than others, making tiny corrections to the controls to stay in the driving lane, and a safe distance from other vehicles. Humans do it much slower, of course.

It isn't just GPS. That isn't accurate enough. It isn't just LIDAR either. There's lots of math and something called Bayes's Rule or Bayesian search. It's how they found the USS Scorpion, and part of how autonomous cars work, and medical testing, and so much more. It's a form of updating your prior knowledge in light of new evidence.

As a side note, you should know it's taking everything I have to avoid springing into a full blown rant about how many people go about their daily life, oblivious to the relevant facts. It's my personal opinion this is how people vote for politicians who are a menace. I can get why rich white men vote for rich white male politicians who will enact policies favourable to rich white men. I hate it, but I understand it.

But why do other people vote for them? Why do women vote for men who restrict family planning choices and limit their professional lives, who demean them in public? Why do poor people vote for rich people who make things harder for the poor?

I get that you could be fooled by someone saying they are going to enact legislation to do x, y, and z, and they actually does the opposite. Once. But many politicians come right out and say what they are going to do. Kenney, for example. Nobody should be expressing surprise at what his political party is up to. There's a joke. A rich banker, a middle class white collar worker, and a blue collar union worker meet up to discuss issues. While the white and blue collar workers are off getting a coffee and discussing how to gang up on the banker, the banker is stuffing doughnuts down his throat and into his pockets. The two others come back to find the box of a dozen doughnuts has only one left, and the banker tells the white collar worker, "You'd better watch that union guy, he's going to eat your doughnut. In this joke, Kenney is the meeting moderator, and he helped snarf the doughnuts and pass them to the banker, and distract the workers.

(Thinks calming thoughts.) Back to the book. Math, constantly updated predictions based on current evidence, done by computers who can deal with a statistical universe far larger than humans can comprehend, is how Amazon decides what to show you as suggested purchases. Based on what you buy, it compares you to other people. So if you like a certain set of movies, for example, it looks at all the other people who like that set of movies, and it compares what they buy that you have not bought. Voila. Of course, it's much more detailed and complicated than that.

Humans make their judgments on a very limited data set. It's basically what we see and hear. There's a limit how much input we have available to us, and how much we can actually absorb. It's easy to make a false conclusion based on data that is not representative of the larger set, and that's before deception, and fraud, and manipulation, and theft.

That's why these big data sets are so important. Consider images of skin cancer. What does that look like? How do you compare a deadly melanoma from a benign mole? If caught early by a trained dermatologist, the cancers are often treatable, so the stakes are high. If your comparison database is small, or doesn't contain good examples, you are likely to make mistakes. But what if there were many images, from many different kinds of cameras, under many different lighting conditions? Such an AI system can distinguish the two most common types of skin cancer from each other, and from a benign mole, and what's more, do so as accurately as a panel of dermatologists. Even more important, do it much faster. It may soon be possible to send in an iPhone photo of a suspect patch of skin, and get an answer back in real time.

Right now a Formula 1 race car generates enormous amounts of data every lap of a race. That's analyzed to determine what tiny changes need to be made to make the car go faster. Tiny fractions of a second add up. Lots of money is at stake.

So why aren't we getting the same sort of analysis done our health? It's literally life and death to us to get a grip on the patterns of our changing health results. Seeing a trend early can get treatment that is likely to be cheaper and more effective, leading to lower medical costs and longer better lives for us.

So, why don't we? Privacy concerns is one reason. The other is that the data is not organized well, and nobody is really looking at any of it. You know who Florence Nightingale is, right? She didn't get the results she did by asking nice and smiling. She had data because she loved math, and in particular, statistics. That data eventually brought about the changes.

Right now you are lucky if you see your family doctor once a year for less than a half hour. They rely on your oral history about health in your family. But really, what do you know of your parent's health? Your grandparents? Maybe you have a genetic predisposition to something, and the people you inherited it from died in farm accidents before that could kick in. You are much less likely to die in a farm accident, and much more likely to die from a cardiovascular disease, or a cancer, that is, if you survive driving.

There is great promise in genetic analysis to aid diagnoses and treatment of many conditions. Knowing your parents genetic history makes that analysis even more accurate. Knowing what similarities are important or not, it becomes possible for the Netflix of medical systems to happen. Your doctor would get told, there is a chance your patient has this condition or is at risk for these, we suggest these particular tests.

Some of you are going, Hell yes! Others are horrified. Fair enough. There would need to be lots of controls on that sort of thing. I'd worry about the insurance industry seeing this data about us and manipulating the industry to make more money, or deny coverage. But mainly the making more money part. The Orwellian implications are obvious.

What's important to realize is that this isn't going away. If we don't want that Orwellian insurance world, we're going to have to make the politicians enact legislation to prevent it, and a system that can enforce those regulations, and do it in the face of lobbying by deep-pocketed industries. We goofed on cell phones, and Canadians pay some of the highest rates in the world. Do you want to get it wrong on your health?

As a reward for reading through the giant wall of text, here's the real stars of the blog, hard at it.

Deadwood of the Day