Root Cause Analysis

Let’s talk about America and the gutting of its civic institutions for power and profit in the name of patriotism.

The overarching theme in my sparse blogging history since I left the military has been the exploration of what it means to be a patriotic American, gradually stripping away and discarding potential partial definitions in search of a distilled essence:

Basically, after PCSing out of the military, I wanted to reinforce in my own mind the integrity of the term “patriotism” in the face of a commandeered chickenhawk interpretation of American patriotism where he who waves the flag hardest is the most patriotic, regardless of any of his other actions.

Sacred cows have been slain in the last decade because of never-ending war, polarization in politics, and the battle to own the term “patriotic”; July 4th has become a vaguely gauche holiday, standing or kneeling for the anthem is a politically divisive act, and even hearing the phrases “thank you for your service” or “respect the troops” have veterans and servicemembers hearing red flags.  Good people have abandoned ritual, leaving it to scoundrels and opportunists.

What could I strip the definition of American patriotism down to, such that even the least principled, most opportunistic chickenhawk or troll couldn’t pollute it?

Meanwhile, professionally, I spent some years unifying my experience being an internet-American comfortable with social media with my career in analysis/intelligence.  For a couple years, I was full-time watching social media for emergency management.  In that position, part of our advantage was being able to suss out what was old news, what was poor eyewitness reporting, and what was truly new news.  In that time, the spread of social media into the daily consciousness was a benign thing in all ways except the distraction of checking one’s cellphone.  Fake news was not weaponized yet; fake news was unintentional, such as poor media literacy or poor eyewitness memories and testimony.  Our job at that time was to assess mostly reliability, whereas someone in the same role now might also have to assess malicious intent at a troll level or even at a state actor level.

After what I perceived to be not only a disastrous election result but also a deeply confusing one that I did not see coming, I was thrown for a loop.  What was wrong with how I perceived reality?  I am fairly skeptical in my prediction-making and assessments, but I misjudged this one pretty bad.  Why?  And how?  This had occurred even though I was deeply troubled after Obama won re-election, stating this on Facebook amidst a taunting, mostly liberal friend feed.

My friend feeds reveling in Republican loss but we're still a nation divided on core issues (how to provide economic…

Posted by Ben Turner on Tuesday, November 6, 2012

As a result of getting married, I gave up my Texas citizenship and changed my residency to New York.  I also enrolled in the Democratic Party.  Running up to the primaries and general election, I received several of the same survey in the mail, even after I submitted my answers.  The questions were all focused on the Republican party, and did not question Hillary’s nomination.  Most importantly, the options for which issues the Party should run on (the Democratic party!) did not include anything on education or single-payer/universal health care, despite evidence that there is hunger and precedent for it amongst Democrats.  Why would the Democratic Party leadership make such stupendously foolish decisions?

Now, months after the election, the news cycle is converging around a narrative where much of the confusion can be blamed on dezinformatsiya, a substantial and tremendously successful, yet low-cost disinformation campaign by Russia in an attempt to destabilize the United States and a world order which threatens Russian security, e.g. “How Russia Created the Most Popular Texas Secession Page on Facebook”.  While the scope of the campaign is not yet known, it involves Russia’s push for RT in American, Twitter bots, astroturfed protests, fake Americans, Facebook groups, etc. Essentially focusing American social media’s energies against itself. Weaponizing it. Sowing discord and discontent.

Fortunately, it appears that there are well-reasoned, stable, legitimate investigations into the degree to which these campaigns affected US elections. Personally I want to have a betting pool on which day we’ll inevitably label as Mueller Day, an annual day where we celebrate Mueller’s principled investigation toppling the Trumpist movement.  But that may never happen.  In the meantime, the US intel community provided some background on how to assess the impact including these crucial judgments:

Russian efforts to influence the 2016 US presidential election represent the most recent expression of Moscow’s longstanding desire to undermine the US-led liberal democratic order, but these activities demonstrated a significant escalation in directness, level of activity, and scope of effort compared to previous operations. We assess Russian President Vladimir Putin ordered an influence campaign in 2016 aimed at the US presidential election. Russia’s goals were to undermine public faith in the US democratic process, denigrate Secretary Clinton, and harm her electability and potential presidency. We further assess Putin and the Russian Government developed a clear preference for President-elect Trump. We have high confidence in these judgments.

We also assess Putin and the Russian Government aspired to help President-elect Trump’s election chances when possible by discrediting Secretary Clinton and publicly contrasting her unfavorably to him. All three agencies agree with this judgment. CIA and FBI have high confidence in this judgment; NSA has moderate confidence.

Moscow’s approach evolved over the course of the campaign based on Russia’s understanding of the electoral prospects of the two main candidates. When it appeared to Moscow that Secretary Clinton was likely to win the election, the Russian influence campaign began to focus more on undermining her future presidency.

Further information has come to light since Election Day that, when combined with Russian behavior since early November 2016, increases our confidence in our assessments of Russian motivations and goals. Moscow’s influence campaign followed a Russian messaging strategy that blends covert intelligence operations—such as cyber activity—with overt efforts by Russian Government agencies, state-funded media, third-party intermediaries, and paid social media users or “trolls.” Russia, like its Soviet predecessor, has a history of conducting covert influence campaigns focused on US presidential elections that have used intelligence officers and agents and press placements to disparage candidates perceived as hostile to the Kremlin.

Russia’s intelligence services conducted cyber operations against targets associated with the 2016 US presidential election, including targets associated with both major US political parties.

We assess with high confidence that Russian military intelligence (General Staff Main Intelligence Directorate or GRU) used the Guccifer 2.0 persona and DCLeaks.com to release US victim data This report is a declassified version of a highly classified assessment; its conclusions are identical to those in the highly classified assessment but this version does not include the full supporting information on key elements of the influence campaign. iii obtained in cyber operations publicly and in exclusives to media outlets and relayed material to WikiLeaks.

Russian intelligence obtained and maintained access to elements of multiple US state or local electoral boards. DHS assesses that the types of systems Russian actors targeted or compromised were not involved in vote tallying.

Russia’s state-run propaganda machine contributed to the influence campaign by serving as a platform for Kremlin messaging to Russian and international audiences. We assess Moscow will apply lessons learned from its Putin-ordered campaign aimed at the US presidential election to future influence efforts worldwide, including against US allies and their election processes.

It is tempting to lean wholly on this narrative of Russian complicity with Trump and the GOP — it explains a Trump victory, it explains what Putin has been up to to counter American hegemony, it explains the increase in vitriol online, where most normal people don’t have time to engage so much.

It is tempting to write off all blame on Russian interference, instead of questioning gerrymandering, or the internal divide in our politics, or the pervasive sense of injustice, or why our system was weak enough to be affected, and why we were not on alert enough to know what to look for, and why we did not protect the sanctity of our institutions . .

For the Democrats, they were blinded by a sense of entitlement; Hillary is by far the most qualified candidate, so QED, she will win. But she had a high unfavorability among even loyal Democrats.  Bernie was attacked not only by core Democrats but also identified as a threat to Hillary’s nomination where it was more certain she would lose to a Republican candidate.  Debbie Wasserman Schultz inconceivably screwed up not only handling the email scandals, but hiring sketchy IT people, ignoring her constituency, and overall just perpetuating the illegitimacy of Florida politics. Her removal was a reactionary move and not a positive outcome.  She had burned all her capital.

A Historical Aside

Let’s take a step back and think about how far the internet’s come.

I got my start on online networks using BBSs, Prodigy*, The Sierra Network/ImagiNation, and eventually MUDs via university telnet. Eventually web browsers unlocked a world not only of web pages but also of the ability to create your own pages. But there was no commercial incentive, the graphics still sucked, and everything was still primarily text-based. The dotcom era was a perfect storm of more interactive web sites, more consumer bandwidth, and unbridled optimism for social networking and the promise of profit.

In 2007-2009 or so, social media had advanced past the dotcom and bust stages of Gold Rush open API euphoria to a more stable, yet more walled garden-ish ecosystem of services which, at least in my career, was organized and standardized enough to be used to respond to emergencies and crises faster.  The main concern for my job was ignoring innocent bystanders who posted bad information because they didn’t know what they were talking about, and assessing who would be in unique positions to have a direct view of an emergency as it unfolded.

Another concern was having to manually link together disparate spheres of knowledge.  Jihadis were prolific in using social media because it helped to tie together their community across multiple countries without official channels.  But journalists were not yet on Twitter the way they are now.  Academics who may have known a lot about jihadi or criminal cultures certainly didn’t use computers or the internet any more than they had to.

The post-dotcom boom Internet up until 2014 or so represented the normalization of the online world amongst the broader population.  Dating online became more accepted. Buying food online to have it delivered became something you would sensibly do to save time. Amazon and Target deliveries changed regular spending and shopping habits. Obama-era campaign parties provided a left-wing answer to the more focused grassroots single-issue communities on the right.

Meanwhile, internet architecture had improved such that more data could be organized online, and processed wherever.  More people relied on data being available to them online, too.  Hackers still have a field day to this day with unsecured systems.  Mostly all it took was will in order to get access to secret data.

Vulnerable Surface Area

For hackers, this meant easy money.  For Russia, this meant a sensible, low-risk attack vector against a trusting, open internet and an American population conducive to sharing their opinions on it.  In 2008, a Russian KGB/foreign affairs/information warfare expert, Igor Panarin, got a lot of press for his theory that there would be a pending breakup of the United States into several different regions.

His background:

Panarin graduated from the Higher Military Command School of Telecommunications of the KGB (now the Academy of Federal Security Guard Service of the Russian Federation) in Oryol and the Division of Psychology of the Lenin Military-Political Academy (with a gold medal). In 1993 he defended his thesis for Candidate of Psychological Sciences, titled “Psychological Factors of the Officer’s Activity in Conditions of Innovations”. His Doctorate in Political Sciences was awarded by the Russian Academy of Public Administration in 1997 for a thesis titled “Informational-Psychological Support of the National Security of Russia”.

Panarin began his career in the KGB of the Soviet Union in 1976. After 1991, he worked in the FAPSI, then the Russian equivalent of the U.S. National Security Agency, reaching the Colonel rank. His field of activity was strategic analysis and integration of closed and open information streams, information stream management in crisis situations, and situation modelling of global processes. He did strategic forecasts for the then President Boris Yeltsin. From 1999 to 2003, he worked as the Head of the Analytical Division of the Central Election Commission of Russia. From 2006 to 2007, Panarin was the Press Secretary of the Russian Federal Space Agency (Roskosmos), the Russian analogue of the U.S. NASA.

Prof. Panarin started his teaching career in 1989 and has taught in the Moscow State University (MGU), the Moscow State Institute of International Relations (MGIMO-University), the Russian Academy of Public Administration, and the Diplomatic Academy of the Ministry of Foreign Affairs of Russia, where he has worked since 1999. He also carries out activities on his own. In 2004, he launched his official website.[citation needed] In April 2008, his first training seminar called “Information warfare – technologies for success” was held. It was targeted at top managers of state and business structures, press service managers of authorities and large corporations, anti-crisis management experts, and decision makers in time-deficit situations. On 20 May 2009, Panarin started World politics – his own weekly radio programme on the Voice of Russia radio.

Panarin is currently the dean of the Russian Foreign Ministry’s school for future diplomats and appears regularly on Russian television channels.

Legit af.

It would be interesting to me to have more access to his (mostly Russian-language) works.  I would wonder how much his ideas have permeated Russia’s operational plans, or if we are falsely attributing him to current events.

Post-Frontier West

So here we are. The internet has already lost its innocence after toxic corporatization of public spaces online, but now it has been violated by directed state-sponsored attempts to undermine the country under which much of the internet’s companies operate out of.

We have Mark Zuckerberg addressing the nation in a close-shot video that is akin to an apology video meme, while his company laid out a Root Cause Analysis and Performance Improvement Plan for itself in a classic engineering-focused style.

The Gold Rush of the dotcom era gave way to where we are now: a Wild West frontier where bandits pillage defenseless villagers and groups with ambition seek to wipe out all those who could stop them from taking over.  Social media companies have been exposed for having put off their social responsibilities for as long as possible under the guise of freedom of speech and non-responsibility for what content is posted on their neutral networks.

What happens next?

If you presume we do not have a government hostile enough to the fact that social media may have been corrupted to subvert American interests, then you wouldn’t expect the federal government to flex its muscles the same way it might have with the famed US Marshal Service, whose Marshals provided the only law some towns had in the Frontier West.  You might hope that social media companies do more to combat active state-actor disinformation and subversion efforts.  You would presume social media companies and governments would begin to officially share more information, as it appears they’re beginning to do now.

I enjoyed reading Nabiha Syed’s framing of the issue of how to confront fake news and promote free speech, particularly this (incomplete, as she points out) theory:

Third, and far less fashionable, is the idea that the First Amendment exists to promote a system of political engagement. This “collectivist,” or republican, vision of the First Amendment considers more fully the rights of citizens to receive information as well as the rights of speakers to express themselves. Practically and historically, this has meant a focus on improving democratic deliberation: for example, requiring that broadcasters present controversial issues of public importance in a balanced way, or targeting media oligopolies that could bias the populace. This theory devotes proactive attention to the full system of speech.

The republican theory, which accounts for both listeners and speakers, offers an appealingly complete approach. The decreased costs of creating, sharing, and broadcasting information online means that everyone can be both a listener and a speaker, often simultaneously, and so a system-oriented focus seems appropriate. But the collectivist vision, like the marketplace and autonomy approaches, is still cramped in its own way. The internet—replete with scatological jokes and Prince cover songs—involves much more than political deliberationAnd so any theory of speech that focuses only on political outcomes will fail because it cannot fully capture what actually happens on the internet.

It’s not clear what actions will be taken by these large entities, but a few things seem crystal clear to me:

One: the US as of 2017 has no plan to actively combat disinformation attacks and voting integrity hacks for any near-term elections, leaving us at risk for larger campaigns seeking to destabilize American interests. These campaigns could also be extended to other countries, since it worked so well in the US and perhaps for Brexit.

Two: the disillusionment with Silicon Valley has teeth. Before, the sentiment was constrained to people who wanted to cut the cord for cable, or who hate cellphones at dinner, or who want you to get real friends instead of talk online all day. Now, Silicon Valley will be perceived in a similar vein as Big Business, enabling the worst behaviors of monied interests who seek to take more and more away every year.  Companies will hide behind their algorithms, and poor decisions made now by humans will eventually be poor decisions made by AI, removing humans from feeling responsibility for their actions. How long will it be until engineers and developers seriously propose something like the Hippocratic Oath?

Three: the loss of American identity. American identities can be recreated for pennies by Russian intelligence operations. Valid American identities are sold on the deep and dark web for dollars. Because of things like Citizens United, now more money than ever can be represented as individual Americans just hoping to get you to change your view on something. If voting machine hacks actually happen (and we certainly know they can, given how easily and ubiquitously they have been hacked in the past), oops, all the sudden your legitimate vote was just used to vote for someone else.  Where are the protections for the atomic unit of American democracy, the American citizen?

Am I who I say I am, or am I really a Kevin Durant sockpuppet saying you suck for criticizing Kevin Durant?

Or am I the NFL commissioner’s wife attacking her husband’s detractors with my own sockpuppet account?

The tenor of online culture has changed, and it’s uglier and colder; read Mike Monteiro’s history of Twitter:

Twitter would have you believe that it’s a beacon of free speech. Biz Stone would have you believe that inaction is principle. I would ask you to consider the voices that have been silenced. The voices that have disappeared from Twitter because of the hatred and the abuse. Those voices aren’t free. Those voices have been caged. Twitter has become an engine for further marginalizing the marginalized. A pretty hate machine.

Biz Stone would also believe that Twitter is being objective in its principled stance. To which I’d ask how objective it is that it constantly moves the goal posts of permissibility for its cash cow of hate. Trump’s tweets are the methane that powers the pretty hate machine. But they’re also the fuel for the bomb Twitter doesn’t yet, even now, realize it is sitting on. There’s a hell of a difference between giving Robert Pattinson dating advice and threatening a nuclear power with war.

American Patriotism

Okay, so back to defining American patriotism.

What if I referred to the Army values: LDRSHIP, or Loyalty, Duty, Respect, Self-less service, Honor, Integrity, and Personal courage.

Is loyalty patriotic?  Not particularly.  In fact probably everyone values loyalty at some level, or at least justifies it to himself as being loyalty.

Duty? Respect? Honor? Personal courage? I think these values are easy to co-opt for your own interests and to look good when you want people to look at you. Think about Congressmen saying they are fulfilling their duty by “serving” in the House or Senate. Think of the “honor” of scumbag servicemembers or the personal courage of always voicing your opinion against a corrupt mainstream media or government.  These definitions can easily be twisted.

Well, what then about self-less service? This one is interesting to me. The removal of self from the equation, along with service (to others), means the calculus changes. I guess you could enhance your brand by volunteering, or you could be assuaging your own guilt about something, but the time element with no monetary reward means that you are sacrificing potential profit for helping someone else.  This is less easy to fudge or to fake.  You have to put in the time.  You’re not writing off profits from your taxes by donating.  You’re not a “not-for-profit” income-tax-protected-class church raising money but then squirming through explanations of why your church didn’t open its ginormous doors to displaced families like Joel Olsteen’s infamous interview (note the apologetic style similar to Zuck’s).

But would you define American patriotism as selfless service?

Making Sense of It All

In light of the evidence presented on a daily basis, and assuming Trump is a rational actor (I feel like this has to be stated; somewhat similarly most discussion about North Korea is most productive when it assumes North Korea is a rational actor and not just some crazy fat man-child with one finger on the button), the only theory that makes sense to me (and I have to make it clear that I don’t really think this could be possible) is that Trump, clearly a type who wants to make money at all costs and at everyone else’s expense, also is particularly vulnerable to flattery and machismo. Vladimir Putin’s overtures to Trump early on were highly successful, and Trump became open to suggestion. Putin, who is probably left with a lifelong scar of bitterness for the breakup of the Soviet Union, still profited off of the USSR’s breakup handsomely along with the other infamous oligopolists who took shattered government assets and assumed them for nothing, consolidated properties, and became massively rich.

To me, Trump’s willful animus towards American tradition and history does not come so much from deep Republican distrust of government spending, but because Trump has been convinced at some level by Putin that what happened in the USSR could stand to handsomely enrich him if it happened in the US.  Trump is certainly a member of the cosmopolitan class and so destabilization at the national level would hardly affect him — like most cosmopolitans, they could live happily and with identical lives in just about any major city in the world these days. His disdain for minorities would not be challenged, his lust for money would be sated, and his statements that are divisive with no seeming logic or theory behind them would make more sense.

Again, I don’t really believe this has happened, but it’s the only narrative I can think of that makes the most sense given all the evidence.  I just wanted to lay it out here because, well, what the fuck else am I going to do except worry about the country I love?

American patriotism, I think, partially involves optimism for the future.  Obama correctly and intuitively perceived this to be Hope.  Other countries do not necessarily have this vast wellspring of belief in optimism, because their geopolitics and culture do not allow it as well as America, with its relatively peaceful continent and bountiful resources, does.

But let’s go a step further and say that regions like the EU believe in a somewhat different form of hope.  They do see sense in policies that target poverty reduction, increase in education levels, etc.  This certainly qualifies as optimism for the future for all people, not just oneself, though it comes with a tinge of “this solution comes as a result of suffering and of trial and error”.  American optimism, I think, is embodied in Neil Armstrong’s partial quote, “one small step for [a] man, one giant leap for mankind”. How ever he actually intended that quote, the US seems to undertake grand projects that help mankind, but through ambition and wanting to leave one’s mark on the world, and less of a humanitarian purpose.

Would we ever see a grand project for eliminating poverty?  Perhaps if there were a technological solution for it.  But otherwise there is no compassion for the poor, no safety net for the unlucky, no sense of moral purpose to protect the innocent.  Being poor is being guilty before being proven innocent in the US, and that is unlikely to change anytime soon.

American optimism as a high-level concept is personified in leaving a grand mark upon history, but optimism for many of its immigrants, illegal and legal, is for a better future for one’s children.  What does that disconnect mean for us?

Russia and China obviously identified American optimism as a critical blindspot long ago. China saw the internet as such a threat that it constructed the Great Firewall to close off its internet from the rest of the world. It has since focused on shutting down key nodes in the resistance’s social graph, preventing influential protesters from organizing in realspace but allowing other meandering complaints to occur with freedom. Russia, stereotypically paranoid about its own insecure borders and withholding as much information as possible, sought to exploit America’s — and its citizens’ — free and somewhat irresponsible handling of information.

This is akin to projection psychology, and, if you subscribe to that theory, it makes it fairly easy to predict Chinese and Russian moves, particularly since it suggests they are actually doing the hard work of identifying their vulnerabilities (and yours) and then mobilizing to protect themselves while attacking you at it.  If China or Russia make a move against their people, then they have identified the exploit for it as being valuable enough to use on you.  It applies even here at home; how many times do we need to listen to calls for increased legislation around marriage, sex, the female body, sexual affiliation, etc. while seeing the people in power who propose it be convicted of offenses in the similar area? They are legislating in an attempt to contain their hardly controlled desires and impulses. Pray the gay away. Make women wear head scarves because you can’t control your dick. Censor the internet because your policies are unpopular. Repress guilt for your adultery by preventing even the most benign divorces.

This doesn’t come out of a valid, rational, well-evidenced alternative, this comes out of fear and insecurity about one’s own impulses.

I grew up in Dallas believing that the US was post-racial divide, sure, and as I’ve gotten older I’ve learned that certainly is not the case.  Police brutality, and the kneeling for it, along with endless reports of sexual harassment of women, along with having lived in the southwest and in the northeast, have opened my eyes, even if I did not always want to see.

How much did it take for football players to lose their memories, or to lose their credibility as deeply-thinking human beings, in order for people to care about CTE and Colin Kaepernick‘s and Eric Reid‘s abilities to have informed opinions? It took a Trump election to change it for liberals. Would Ta-Nehisi Coates’s profound book be as powerful as it is now instead of yet another protest similar to reading a Greenpeace or PETA campaign? Would people still have responded as desperately to many of today’s issues if Hillary had won, and we could rest more assured that the arc of moral justice bends towards progress?  Now, are you as certain that that bending is true?

Imagine how other peoples’ perceptions of the above photo has changed since it first happened, after the repeated fatilities from police brutality, the Trump election, the “disrespect of the flag” debate, the dragging of veterans for some reason into this discussion.

Are we really certain of our principles and opinions on issues?  If we just got rid of the National Anthem at sporting events, perhaps this is a logical move since, well, why the fuck is it there to begin with? But have we enriched our culture by taking it away, or just leave an empty hole? Or is this like removing Confederate statues where some would say we should remember the past and others would say we should not celebrate the past. Would we have opinions on these matters that would shift in different contexts?

Proposal

I propose that we as a culture reaffirm the power of the individual vote. Just focus on that. Some (I don’t think I do, given studies that don’t necessarily show it helps) believe we should get a day off to vote. Enact measures that increase the percentage of eligible Americans who can vote. Enact policies that encourage non-voting blocks who long gave up on the system to re-engage. Put research, people, money, and innovation into secure voting methods and machines (or paper ballots if those make more sense) that implement modern-day advances in social media (the complaint about American Idol being more reliable than our voting systems). I would imagine the Electoral College is a disincentive too towards any reaffirmation of the power of the vote. Halt Citizens United and require transparency in campaign funds, even if just temporarily to solidify the mission around individual votes.

I know that is an unrealistic proposal and it probably weakens the rest of this essay for some readers, but to me it is the most actionable, most grassroots political, most confidence-inspiring thing that could be done, rather than high-up deals made behind closed doors between groups that have no connection with the American people anymore.

As for foreign corruption of our institutions, well that seems like it hardly needs to be mentioned. And we are still dealing with the shock that it even needs to be addressed. Can we not conduct politics in such a way that we need to collude with hostile foreign nations in order to succeed? That seems like pretty low-hanging fruit to me.

Newslint: My Project to Supplement Media Literacy

I just released a new site that I built, Newslint.  Check it out!

A while ago I was reading Hackernews and I saw someone posted a link to the Javascript source for joblint, an app that you copy and paste text from a job listing into, returning results on whether the job sounds fluffy, unrealistic, bro-ish, tech buzzword-y, etc.

I thought it was an awesome idea, and the implementation was such that it was easy to add rules, keywords, and phrases to check against.  Having just gone through a job hunt before being hired by a very professional group of people as a developer at The Barbarian Group, I was sensitive to junk in 90% of job listings.  I was also considering my love for news curation, media literacy, and good journalism, after having been an Army intelligence collector and analyst and then a social media operations analyst for a DHS contractor.  So I started thinking about making a port of joblint so that I could lint news articles in a similar way in an effort to explore media literacy.

pew

Fast-forward a little while and I was preparing to work on a Django project for The Barbarian Group.  I had already put in a lot of time learning Python while at ChatID as an engineering intern, but now I had to learn the Django framework.  Since Python and JavaScript are similar and flexible enough that I didn’t anticipate many problems, I decided I could try to port JavaScript joblint to Python newslint and then incorporate newslint into a Django server and experimental test-bed.

How It Works

During my time doing social media analysis, we hired for and trained for our analysts to be able to quickly assess whether news information was valid, credible, interesting to our client, potentially dangerous, environmentally relevant, etc.  More of an art than a science, this involved knowing which sources tended to put out good info, knowing the current situation and deciphering which new information would most affect the status quo, where the best sources of information in different spheres of influence could be found on the public internet.  This is actually a pretty difficult skill to acquire and that alone has the largest influence on the quality of analysis output.  That is, if only 5% of the info out there is actually game-changing, then less time has to be spent on the other 95% so that more direct analysis can be done on the 5% — but at the same time, the 95% of noise is still relevant as an environmental check.

Media literacy is crucial even if it’s not your job.  A lot of my Army friends are more conservative and they’ll post articles from certain biased sources that end up not being true.  And a lot of my liberal NYC friends will post stuff from advocacy blogs about the NSA and eavesdropping which are demonstrably false or short-sighted.  For others who don’t really consume the news, the tangential connections they have with the news are even more important.  Those decontextualized sound bites from the news are all those people will hear about an issue and so it will largely shape their opinion on the matter without more study.  FOXNews used to be on every TV all day, and now it’s likely you’ll see CNN instead.  Some people only watch The Colbert Report and The Daily Show.  Others watch the worst hours of cable news television, the afternoon lineups on FOX News and MSNBC.

It is crucial to understand how businesses buy people to write in newspapers or make TV ads or form political action groups to shape public opinion through blanketing the air with a specific message.  Non-profits, advocacy groups, and different areas of the government do it as well.  Whenever you see a poster advocating for or against a bill, you should always look up the group named in small print at the bottom and see who’s behind it.  It’s probably not a grassroots campaign — it’s probably astro-turfing.

In short, like any good intel, you should be suspicious of any information that finds its way to you because it most likely was intended to reach you, and wasn’t a happy accident or a sign of unstoppable progress towards that position.  Media literacy helps people decipher incoming input for true intent and agenda.

So that’s what newslint can help you do.  It takes raw text and looks up key words and phrases that indicate credibility, non-partisanship, and professionalism.  Do you read solid sources from solid journalists in solid publications.  Are you learning partisan phraseology that slants your opinion?  How objective and experienced are the people you read?

Here are all the rules for newslint.  I would definitely appreciate an email, or even better, a pull request, if you want to add more rules.

swears = [
 'bloody',
 'bugger',
 'cunt',
 'cock',
 'pussy',
 'dick',
 'douche',
 'jackass',
 'asshole',
 re.compile('fuck(?:er|ing)?'),
 re.compile('piss(?:ing)?'),
 'shit'
]
partisan_words = [
 re.compile('obama[ -]?care'),
 'libtard',
 'nobama',
 'death panel',
 'leftist',
 'communist',
 'malkin',
 'coulter',
 'sharpton',
 'sarah palin',
 'pinhead',
 'limbaugh',
 'o\'reilly',
 'krauthammer',
 'bachmann',
 'ron paul',
 'rand paul',
 'john bolton',
 'alex jones',
 'martin bashir',
 'perino',
 'karl rove',
 'stasi',
 re.compile('police[ -]?state'),
 re.compile('hipp(?:ies|y)'),
 re.compile('fly-?over state'),
 'wingnut'
]
pundit_words = [
 'krauthammer',
 re.compile('(?:thomas|tom) friedman'),
 'ayn rand',
 'john galt',
 'jesse jackson',
 'gold standard',
 'rand paul',
 'ron paul',
 'bachmann',
 'limbaugh',
 'o\'reilly',
 'gingrich',
 'slate',
 'glenn beck',
 'msnbc',
 'huffpo',
 'huffington[ ]*post',
 'fox[ ]*news',
 'coulter',
 'david brooks',
 'john bolton',
 'michael hayden',
 'alex jones',
 'scarborough',
 'chris matthews',
 'sharpton',
 'martin bashir',
 'sarah palin',
 'ezra klein',
 'perino',
 'malkin',
 'evgeny morozov',
 'karl rove',
 'drudge',
 'dowd',
 re.compile('chris(?:topher|) hayes')
]
rag_words = [
 'gawker',
 'tmz',
 'slate',
 'infowars',
 'buzzfeed',
 re.compile('the blaze'),
 re.compile('huffington[ ]?post'),
 re.compile('fox[ ]?news'),
 re.compile('drudge[ ]?report')
]
sensationalism_words = [
 re.compile('tears? apart'),
 'screed',
 'demolish',
 'crush',
 re.compile('brown[- ]?shirt'),
 'hitler',
 'gestapo',
 'snitch',
 'stooge',
 re.compile('game[- ]?chang(?:e|ing)'),
 re.compile('cutting[- ]?edge'),
 re.compile('bleeding[- ]?edge'),
 re.compile('marxis[tm]'),
 re.compile('cron(?:y|ie)'),
 re.compile('fema[ ]?trailer'),
 re.compile('chem[ ]?trails'),
 'delusion',
 'false flag',
 re.compile('racis[tm]'),
 re.compile('meme[- ]?wrangl(?:ing|e)'),
 'flagrant',
 'cult',
 'the establishment',
 'police state',
 re.compile('solutionis[mt]'),
 re.compile('shock(?:ing|er|ed)')
]

And Now, the Tech Details

I ported over the code (it’s not very large) in a day or two, then debugged it for a while.  It worked — I made some additions, and, like joblint, it can be run independently via the command line.  Then I forked joblint and turned it into newslint in a separate git repo.

console

Next I figured I could not only use the opportunity to learn Django but also learn other stuff.  I really hadn’t needed to use LESS or SASS up till now because I was working on code I could just throw classes into, but since then at ChatID and at The Barbarian Group I ran into projects where I wouldn’t be able to modify HTML markup and would have to traverse the DOM or find other ways to hook into the code.  So I set up for SASS and Compass and installed Grunt so I could have all my different tasks (uglification, concatenation, code linting, copying to static directory, etc.) automated.  Along the way I found autoprefixer, which you can run as a Grunt task to take any CSS3 stylings and automatically add support for browser extensions.  I also decided to try Django-Bower (based on Bower) for making updating to the latest version of my JavaScript dependencies easier, and within Django’s environment bubble no less.

Django is incredibly easy to use and you get a lot of control over it, which is something I like about express.js.  But then I’m also coming off learning Drupal (PHP) for a project, which seems like a black box most of the time.

I got a simple version of newslint running on a local Django server and then things snowballed; I fleshed out some JSON endpoints for an API, I enabled form submission for saving news clips, and I wrote some tests in Django’s TestCases and Python’s unittests.  Super-easy, especially after dealing with a somewhat problematic time spent figuring out correct resources and syntax for Angular.js tests with mocha for my project Momentous.

front

And then I figured I would try deploying this Django app to Amazon’s Elastic Beanstalk, because I’d never tried that before!  I ran in to some issues there; my static files directory was split up and not in a standard directory, I had newslint loaded as a git submodule under the newslint-server and automatic deployment services like EB and Heroku don’t like submodules.  I also would have had trouble getting underneath the EB abstraction to make edits directly to server settings.

I decided to tear that app down and just get an EC2 instance (m1-small).  It costs a bit, but not really that much, and I’ll probably take down the instance once there’s no traffic on it.

My small test app turned into a full day deploying the app underneath Varnish and Apache to a new ubuntu instance.  I plugged in memcached and set up mysql and added appropriate Django middleware to help get my pagespeed score up and remove warnings and errors.  The full control of an EC2 instance made this all super easy whereas I’m not sure how I would’ve managed dealing with the EB thicket.

I had some problems making sure my headers were set up correctly so that stuff would get cached okay but tweaks to Apache and Varnish settings, along with Django, helped to mitigate those problems.  Updates to code were as easy as a git push on my local computer and a git pull to the instance.

I ran some apache bench tests on the server and it seemed okay; one thing I think I ran into was that having a form on the front page slows down the response slightly because it’s not caching the page (CSRF token?).  ab tests to a non-calling API endpoint on the other hand were super fast.  Most of the time, pageloads are under 400ms, which is pretty sweet!  Thank you to the god of page loadability, Ilya Grigorik!

So then it was rather late and I was thinking, hey, how hard would it be to get a domain for this instead of the long EC2 address?  Well, whois’ing newslint.com actually showed that it wasn’t owned!  And namecheap sells domains for $10-13 usually so I picked it up and pointed it from namecheap to Route 53 and all the sudden very early in the morning I had a working newslint.com!

result

I found some more bugs the next day, for which I’m writing regression tests, but otherwise this has been a really successful learning experiment and confidence booster for my developer chops.  Really glad this worked out so well, and thank you to Rowan Manning for his joblint work and to The Barbarian Group for letting me be a developer.