Saturday, February 24, 2018

Probability Computations

Training Data

I'm impressed how scikit-learn lets us compare models, such as classifiers, using cross-check validation.  Having scikit-learn is like having a staff of professional statisticians, which may be dangerous if one knows nothing of the pitfalls in that discipline.  So lets make that a responsibility, now that we have these free and open tools.

Sometimes, to give one muscle group a rest, one works out with another.  I use this technique when mining Youtubes.  This morning, without any premeditated plan, I found myself back to the JFK conspiracy theorists (researchers) and decided to snarf down some lengthy testimonials.

The idea of two Oswalds having their identities conflated in the rush of the initial investigation, with a justice system bearing down, doesn't seem that far fetched.  The ostensibly guilty party, a one time defector, now sympathetic to Castro, was close to a comic book caricature of the type needed, if "lone gunman" (LG) was the true hypothesis p(LG | BS) = p(BS | LG) p(LG) / p(BS) where BS = backstory (its probability) and our prior.  Monte Carlo + Markov Chain = the Metropolis algorithm.

Regarding public opinion, I find pollsters too focused on a "general public" whereas the more interesting surveys are of those who've had the time to look in to some matter.  Of those who've spent at least forty hours studying conspiracy X, how many believe Y?  P(Y | X) in other words.  I'm not saying "general public" surveys are unimportant, but lets remember sub-populations who've spent a lifetime driving cars, are the ones we like to write our car reviews. We listen to experts, by definition (I might say "trusted experts" but then "listen" doesn't necessarily mean "believe").

Those in the know are seeing Bayesian thinking flying by, which might be considered "trolling" ("trawling"?) among the frequentists.  Am I trying to piss off some segment in the peanut gallery?  Some might be preparing to feel offended.

By the way, I do think "trolling" and "trawling", though not at all etymologically linked, are conceptually connected.  To "trawl" is to drag a net (dragnet), sometimes across a bottom (bottom feeding) in hopes of turning up some interesting low frequency events, such as two-headed fish in the Columbia Gorge (another specimen for the tribal museum perhaps).

Yes, these same statistical techniques (some call it deep learning) might be used by 911TMer types to study relative probabilities of various scenarios, though in this case we start with the low frequency anomalies, of reinforced buildings melting after bee stings (not to minimize so much as to emphasize their narrow locality or non-existence in the case of WTC7).  I'm not the big expert in this area as I've mentioned repeatedly.  You will find I revisit this thread repeatedly.

Finally, lets turn to Russiagate and the probabilities there.  The "troll farm" or "trawl farm" has been dismissed by Rush Limbaugh as an insignificant provocation.  This might be seen as a left / right flip, as in Castros day, the right wingers were keen to push Castro as the culprit behind JFK's shooting, as to them, the whole point was to gain back their gambling valhalla.  Oswald was looking like the right narrative, given his overt Cuban ties.  One would think a loyal Limbaugh, ostensibly "conservative" would take the radical view that the FBI was going up the wrong tree.

The explanation is fairly simple:  the current president is being cast as high up in the KGB (that's science fiction language, my excuse being Rush uses it), or as being case handled by the Kremlin, so letting the FBI's trolling turn the Internet Research Agency into a low frequency unveiling, of a true spy scandal, is against the White House's best interests.  No one said the executive branch is forbidden to act in its own defense after all; that's why the designing engineers put in at least three government branches.  If the FBI is the weapon of the DNC, as some allege, then clearly the RNC (Rush an ally) is going to back the White House, a highly prized set of offices.

I've been critical of 911TMers for denying they're conspiracy theorists, as there's nothing wrong with theorizing about conspiracies, and a lot of people get paid to do just that, plus get a government pension.  No one should have to apologize about the bare fact of wanting to investigate, do detective work.  That a real investigative journalist would use "conspiracy theorist" in the pejorative is the height of irony.  The quality controls come at a deeper level:  what is the actual quality of the research?  Is this professional grade stuff, like much of the JFK stuff is?  Or is this from an amateur newbie just getting in to the research business, maybe cutting teeth on Pizzagate or on a scikit-learn toy data set?