Experimenting with React

resume_mobileWorking on Galapag.us allows me to experiment with new frameworks and to try out new design patterns on mini-apps.

React is one of the newer front-end frameworks and rivals Angular in popularity in new JavaScript apps these days.  Created at Facebook, React has done well to allow a developer to make isolated components which have readily-understandable state at any given point in time.  Having been spoiled at Vimeo by their breezily easy codebase, I was excited to try React to see if it would resemble working with Vimeo’s code.

It’s pretty good!  Components aren’t entirely isolated — you still have to find some clever way to inject style, or just settle on styling globally if you don’t want to do it inline (which React allows for).

Anyway, I built 2 mini-apps.  They’re not quite ready for production yet, so I’m not really going to link to them, but…

Resume

I’d been looking around the web and found JSON Resume, a schema proposal that allows for saving one’s résumé details in a JSON format, so it’s easily portable and separated from style and presentation concerns.  Perhaps an even more important goal of the project is to provide an alternative to the horrendous process of uploading your details via PDF or DOC to some job application site, only to have it either mangle the parsing of your file or, even worse, to force you to re-enter all the info into some shitty 5 page web form that looks like it was constructed a decade ago.  And God forbid you should ever have to update your info when you come back to your profile years later.

I figured it would be nice to have this app built-in for free — perhaps entering in all the pertinent data would become a quest to complete.

So I mapped a lot of the pre-existing modeling I already had in Galapag.us into a call to the API server that serves up your résumé data in JSON format compatibile with JSON Resume 0.0.0.

json

This app didn’t have that much complication in terms of interaction or dynamic changes — a straight-forward app making an AJAX call to populate the front-end.

But it’s fairly clean and extensible — and it’s mobile-friendly.

butterfly

Butterfly

The other mini-app I made is called Butterfly, as in a social butterfly.  The goal of Butterfly is simply to provide an easy tool for you to jot down someone’s name whom you just met at your local deli, or at a party, or whatever, so you know how to find his/her name later once you’ve long forgotten.

After all, we remember close friends but often meet so many random people who are still remarkable in their own way in your life, that we can’t remember their names.

This app is mobile-friendly too, and should make it easy for you to load the app on your phone and add the details of someone you just met.

The Bigger Picture

The résumé data signifies that my data modeling is at least getting to an intermediate point where it’s beginning to provide value in having a large datastore but also diverse enough to support applications which need broad access to different datapoints.

And the butterfly data model helps to map the social graph.  I’ve witnessed the torch and pitchfork brigades that attack services which attempt to allow people to post content about other people without their express permission.  While I disagree with this as a supporter of radical transparency, I understand how it’s just not tenable in today’s society.  But at least for those who opt-in and for certain datapoints, those who volunteer should reap all the benefits of a more interconnected virtual world.  So I do plan on rewarding those who share more — with things like the genetic crossing mini-app.

Galapag.us still isn’t useable in the sense that it replaces my daily viewing habits, and it’s not my first choice for a datastore yet.  I know when the site will be getting approachable for others when I start to dogfood it myself.  I keep adding more and more over the years, and refining that which is already there.  Eventually it will become useful, and, hopefully, necessary in terms of providing competitive reputation.

gc

A Liminal Event

At 36 years of age, as of March 7th, I have just become gloriously married.

You know those times at school when the teacher or a counselor would have you write down your 1-year, 5-year, and 10-year goals?  For as long as I’ve been an adult, marriage has been one of my key goals, since I value stability and a strong base as extremely helpful building blocks towards a better, unique, independent, meaningful future.  This has been a wondrous achievement, to become married to my wife!

It took me a while, and I was woefully ill-prepared for any form of marriage up until just recently despite wanting it, having incurred some hefty life experiences that forged my Ben-ness: training, war, official mistakes, transitions between cities and schools, finding a fulfilling career.  Fortunately, I also hadn’t found the right woman yet.  In retrospect I think I’ve dodged some bullets in life and I safely made it to this happy point; healthy and happy and intact to be able to recognize it in another and now I’m very lucky to have met but also deserving of my wife.  My Best Man wanted it clear to me that I was fortunate to have met her, whereas I countered that I also put in my time and effort to earn that chance.

And now we’re here.  I must confess; I’m a pretty even-keeled guy but our wedding was amazing.  Even better than we could have expected, better than all the fantasies and cliches.  Our friends and family, all holding court in their own ways, keeping the party on lock, the wedding vows themselves touching, the pacing and pre-wedding rituals, the smiles on family who have given us so much but we have not quite given much back to yet.

The honeymoon as well was what I would call a true honeymoon.  While I am loathe to engage in goofy grinny sugar, the honeymoon in Costa Rica (a bit in a secluded rainforest resort where we could sleep and eat the best food, a bit in a beach resort full of sun and surf) was almost fairy-tale like.  The best part was to share it all with a true equal, someone who is equally dedicated and invested in us — perhaps the most valuable thing one can find in life as well as one of the hardest and rarest things to find.

Prediction: Anniversary Weddings

As a side note, after having seen what the wedding culture is like, how the TV shows portray weddings, how our own planning went, I think if you wanted to create an extremely effective long-term marketing campaign, you would push for anniversary weddings as a major new event.  Think about what couples go through: their first wedding is probably going to be at least somewhat on the cheap since they’re either on the hook for it and/or they have no disposable income.  So a bride is not necessarily going to get her dream wedding when she’s so young, unless someone’s parents are loaded.

But imagine: after 5 years or so, the couple will probably have more money and will want to throw a more lavish wedding.  Perfect opportunity to market a second wedding.  Then the longevity wedding at 10 years or whatever, before a couple starts to get old and the glamour of the wedding’s superficialities fade away.  This would create several more opportunities for wedding industry folks to cater to couples and to create narratives out of multiple weddings.  It would also create more excuses for families to gather (which in the grand scheme of things is immensely important, overlooking the pettiness and jokes that surround having to deal with the in-laws and crazy uncles and whatnot).

Anyway.

Introspection

This is one of those moments in life where I take a knee at the top of the hill, look back at the path I’ve come from, and sigh and reflect on all that’s happened.

And then look further up the ridges and ranges to see what’s coming next.

I can definitely feel the strong pull of crossing over from one stage of my life into another.  For the first time, that journey is not alone, my decisions are not solely my own, and instead of destructive creation, there is now only creation, at least for the fortunate time being.  The teens and 20s are pretty destructive years but they chip away at your character and you end up in your 30s and 40s as the type of person you have chosen to be.

We love our little place in the East Village, but we have our story to write and we wonder where we go next.  For once it’s nice to share that decision, that conscious choice.

A developer’s job as a crash course in multilingualism for coding languages, organizational principles and methodologies for projects, and building applications rapidly — I feel this work has deepened my appreciation and respect for the creator and the builder and the artist, a continuation of my studies in art/tech school.  Writing code has in some ways taken the place of writing for me; I no long blog so much unless some large sweeping societal issue has me particularly wound up to write.

I was writing finger .plans in high school and college because I saw John Carmack doing it.  I wrote blog posts through the rest of most of my life thus far, even while in the Army.  Nowadays I think my conversation with the internet, and therefore with you, the reader, has contracted into more private communities.  In fact I think the most interesting communities will become smaller and more niche — it’s already happened on reddit where the subreddits are still positive, informative, and overly open to each other.  I would love to see local intranets for residential buildings or mesh networks that require some form of test in order to gain entry.  When I want to converse, I want to talk about what can be built, not why things shouldn’t, which has become the norm for discourse in most public forums.

Negativity

If I had to say what I hate most, it’s probably negativity.  I’m a pretty conservative person in my own assessments, probably a product of both my Chinese mother’s caution and my sergeant’s training to hope for the best but prepare for the worst, but this should not be confused with negativity.  Negativity is debilitating, demotivating, draining.  Those who find the inner strength to build and create and love need to be supported by the rest of us, not brought down.  And when it comes to examining what is possible, I’m usually overly optimistic.

LOOK AT THIS SHIT.  I PULLED THIS CLASSIC SLATE EXAMPLE UP ON THE FIRST TRY. LOOK AT THIS SHIT. I PULLED THIS CLASSIC SLATE EXAMPLE UP ON THE FIRST TRY.

Cranks and haters have always been a part of online and intellectual discourse, but it’s run rampant now that we have what is somewhat of a critic’s internet buffet. The Twitter community I used to enjoy, journalists and reporters and analysts and policy folks, once encouraging of a naive eagerness for new information, has become TMZ-like (in that it loves gossip, the less true the better) and pompous in its derision for anything and anyone.  No one is safe, in the end, from being torn apart for whatever reason: the idea won’t scale, this or that person is a fraud, etc.  The Atlantic, Slate, these are publications that your liberal news junkie loves to read.  And the writers they hire at those places are absolutely TERRIBLE in their negativity or insinuations of impending failure.

Facebook buying Oculus Rift, Amazon studying the use of drones for delivery, Google making a HUD interface in Google Glass.  What I want to be a part of is a community where those achievements are exalted, not ripped to shreds, as they have been by people I might have respected at some point.  I grew up in an idealistic proto-internet time of downloading MP3s and playing Quake online and Napster was a giant; anything was possible even if it was all insane.  Unless you tune all the negativity stuff out, you wouldn’t know.  You wouldn’t marvel at all the amazing work, such as the people who have contributed research and proofs of work towards, say, NP-Complete math problems.  Louis CK would say “everything is great and no one is happy”; I mean the research and consideration going into those Facebook and Amazon drones and balloons is just a beautiful thing to behold.

As the internet has passed through most of its adolescence and begins its maturity, the rest of everyone who plugged in were stuck in a status quo where it seems none of those critics and cranks have really advanced or improved at their own crafts.  To see the amazing achievements announced daily, and then see them shit on, well, it’s frankly infuriating.

Jason Collins coming out as a gay NBA player.  The quickest response: “But he’s a horrible player.”  Sports, one of the worst refuges of the shitbag critic.  A dude makes the NBA and instead of people showing him respect, they’re quick to point out he’s not a perennial all-star.  I’m sure even in Jackie Robinson’s day, people were slagging on him for something.  Some petty something.  As another sports example, what could be worse than this upvoted rant against watching sports?

The Snowden affair has made everything absolutely toxic.  The government lurks behind every piece of electronics now, in the minds of Snowden’s supporters.  It’s the same kind of paranoia I’m sure most of those people would make fun of the Republicans for in their loathing of Obama, or of conspiracy theorists for their suspicion of anti-gun, anti-religion, etc. liberals.  When John Schindler is pointing out how Glenn Greenwald is one bad day away from appearing on Alex Jones’ show, everything seems absurd.  My liberal peers now make blase jokes about the NSA but barely bat an eyelash at unprecedented expansion of corporate personhood.

SXSW just happened and I’ve never been, but I know people who seem to go regularly.  How do they afford it?  It’s expensive as fuck.  Do their employers pay out for it?  How much are people really getting out of SXSW?  So many people who go aren’t even creatives or builders.  I’m definitely not against a flashy party, but I do wonder when it seems like the B-players are being sent.  E3, SXSW, and other conferences are now meetups for the elite and rich, those who can either afford to go or who are paid to go.  And what are they contributing back as a result?  This is why conferences lose their magic.  This is why, most notably, TED tanked.

Wonder why communities stagnate?  Because they have more cranks than dreamers and helpers.

As I’ve gotten older, I’ve come to love watching the Grammy’s and Oscars more.  For me it’s bewilderment at the complexity and sheer market size of their industries.  The amount of resources, attention, and output generated by dresses, producers, talented artists and directors, the entertainment and happiness that they bring.  I love the process.  Studying how films are cast, the bonding the cast goes through for the film, the pushing of boundaries in the technical areas of the film, the representation of songs off an album into powerful videos, live tours, and performances for award shows, the biographies written up in Rolling Stone and whatnot.

20130813-1190-macklemore-cover-306x-1376429053

But the pettiness of the crowd drags it all down.  All the slagging on Macklemore even though him and Ryan Lewis were by far the most interesting artists of the year in terms of touching all aspects of the industry and affecting culture.  I actually thought Lorde’s album was incredibly powerful for such a young person but I’m happy with all the results.  The same goes with Oscars.  I try to make a point of it to see as many nominees in as many categories as I can, because those picks represent a curation by the industry.  Hence it drives me nuts to hear people completely destroy films in their critiques — the labor going into a project is admirable, even for lesser quality films.

Negativity.  I can’t stand being around it.

Elite Self-Loathing

One story I really hate seeing is the condescension towards Silicon Valley startups who seem to be creating fluff products when, you know, there are people dying in Africa and stuff!  Why can’t those engineers work on actual important problems?

Of course, who’s writing those stories?  Well in the case of the latest NYTimes story, it is written by none other than a Columbia computer science grad student who lives among the gilded engineer community being called out.

Man, where to start.  The insanely awesome advances in open source software, particularly in the promiscuity of excellent language features being shared among the major coding languages such that they’re all beginning to share a lot of commonalities, these are created by developers and coders and hackers and engineers, people who actually do enjoy spending a whole weekend by themselves indoors in the dark pushing characters around on the screen.

0199210829.02.LZZZZZZZ

You don’t have to wait around too long before you hear people say, “We must teach the kids how to code!”  Chances are, the people saying it can’t code.  And they won’t go learn how to code beyond tutorials.  This is fine — I certainly don’t think everyone needs to code — but I am definitely a believer that if you tell others they should do something, you should have done it yourself first.  Now, code as a gateway towards solving problems that kids might be interested in, this is a great addition to a school syllabus. (having them design a database to catalog what they own, or a double-booking ledger system to keep track of what little money they make, etc.)

Anyway, my point was that engineers are good at primarily one thing, which is writing very specific code that can test the limits of what was previously possible.  Criticizing Facebook for focusing on creating end-of-year videos?  That’s an immense computational task and it had to be done using tons of peoples’ disparate datapoints.  But the knowledge of how to pull that off is now going to be passed along to successors, to schools and research facilities, and eventually to grad student theses and private products.  As a former student of international development practices, that shit is hard enough as it is and most everyone in that world barely has the statistician’s background enough to be able to make informed decisions about whether this or that program or measurement is actually helping the people who need it.

So it’s the height of arrogance for people who don’t know how to build or measure things to tell others who can do one or the other that they should not only do that task but also design, fund, scale out, and deliver onto the ground solutions to improve society.  Some mercenary blog-quality writer for The Atlantic decides that this or that project is worth slagging because he was clever enough to think of holes in the project’s design while he sips his morning coffee.

I saw this at ITP — while the community itself is immensely supportive and open towards nonsensical, wildly over ambitious ideas, when it comes to critiquing others’ work, the feedback often comes back to blowing huge holes through someone’s work just to sound smart about it.

Getting back to those life goal write-ups.  I never really did them very seriously but when I think back to them now, they were immensely informative and helpful.  When I was in college, I think my short-term goal was to be a full-time daytrader, since I was doing fairly well trading the dotcom boom.  I didn’t have the perception and experience to realize that market volatility would dry up, that the system’s regulations would be gamed so that small-time daytraders (less than $25k in their account) would be locked out of the practice because of restrictions on the number of trades one can make.  If you want to know how Wall Street has changed, beyond the infamous “quants”, then you should look at how daytrading restrictions have locked out regular folks from trading actively, and also look at how IPOs, buyouts, and other mostly stock market-related business deals are all about cashing out and not adding value.  The stock market’s investment and negotiation deals handled by the big finance firms are not too far off from nobles trading title and land for reputation and power.  Their employees get little.  The only way normal folks can win in the market anymore is to just play for the long-term.

My Flaws and Strengths

Flaws
  • not “great”
  • too stoic
  • too serious
  • not close with enough friends and family
  • no natural ability
Strengths
  • relentless
  • high tolerance
  • courageous
  • top caliber friends
  • respectful
  • reverent of the process
That’s my list.  I might also add that the part I probably think the most about is that I know that positivity and being enthusiastic and empowering others to be happy with things is the most fulfilling thing one can do day in and day out, but for me what I lament is that I am by my nature just not that person.  Having met some powerful leaders, wonderful motivators, and just all-around badasses in my time (many of whom were at the wedding, if only for a few hours before they had to return home!), I have witnessed what they can do for others, and that’s just their natural state.  In the world I’m more of a defensive-minded person, more of a guardian, more of a sheepdog.  It’s just the way I am, and it’s what I’m best at.  How will I integrate that into being successful and truly great?

Taste

People fight so strongly for their opinions.  What for?  How about this quote?
“Good taste is a myth. A story our rider creates to serve the needs of the elephant. And the sooner you kill your good taste idol, the sooner you’re going to give yourself a chance to be a better designer. It frees you up to add taste as another tool in your designer’s toolbox. Consequently, instead of focusing on good taste, your focus becomes the right taste for the problem at hand. There’s a subtle but profound difference.”

The magic moments of online forums or of hobby-based communities are that in most of the growth stage, the conversation is dominated by doers and builders, people who are curious about the implications and who love to share positive aspects of it all.

And then I’m sorry to say but the “intellectuals” then bandwagon on, having read about it all in this or that magazine or on a prominent blog.  This is around the time when communities start becoming caustic, full of cynical comments enforcing behavior of entrenched, mature communities elsewhere.  The doers mostly get drowned out.  I saw it happen on HackerNews, and to some degree on reddit.  Those amazing comments you go to read when you first joined are replaced more and more by negative comments (and by memes, but I love memes).

These folks are full-time consumers: of culture (which they did not live in), of art (which they could not possibly create on their own), of popular culture (which they foster animosity towards but can’t help but indulge in).  A community of fast-food binge watchers.  I am an avid consumer of pop culture myself but I do feel a certain pressure to need to contribute something back: to try to find artistic outlets for my more pragmatic views, or to attempt things that are somewhat subversive.

The full-time consumers are up against the people who believe in doing, as referenced in this hackernews comment section on ageism in tech:

“I have a ton of cultural criticisms of the current tech industry, but the one thing that it is absolutely getting right is valuing creation. The damage that could have been caused by the Google/Apple cartel has been limited largely by the startup industry that, for all its faults, has as a crucial belief that the act of building things is valuable, and that people who make things are the ones who create the most value.”

Strengthening a Community

To me it seems like it would take very little actual effort to maintain the positive tone of a community and avoid the police that have made Wikipedia and StackOverflow so abhorrent to contribute to.  Active enforcement of positive messages would be a given, but bigger than that, I’d prefer a whitelist system where your reputation precedes you.  Do you have a good online history of being helpful, of encouraging others and of sharing lots of content?  Then you’re in.   Admittance is earned.

One major piece of Galapag.us that I hope to implement eventually goes along with the theme of different islands.  The islands have their own cultures, wildlife, climates — and their populations should have culture and personality that reflects it.  So each island would have its own rules for communication and culture.  For example, a welcome island would have strict rules against hurting new players, whereas another island may be a free-for-all.  But I think most islands would be somewhere in between, capped mostly by having formula requirements to gain admittance, based on a person’s stats.  Perhaps an island requires a very high reputation with very low negativity and an innate nature for helping others, so one would need to have very useful posts as well as have an “angelic” alignment.

I like anonymity, pseudonymity, and verified identity all at once.  I think they all have their place.  But I also think that one’s reputation should be compatible with a community.

So here I am.  Here we are.  Starting a new family, a new chapter in life.  I think I’ve definitely become more of a homebody as I’ve gotten older; saving money, disliking the quirks of others more, understanding which types of events are never that entertaining yet knowing which ones are really key.  I’d like to think I’m stressing meeting up with friends more now.  I definitely cherish that time.

And, to conclude, my goals:

1 Year:

  • get a substantial raise to pay off more school debt and begin having children
  • plan on where my wife and I might like to move to and live
  • crash-course on C/C++ and algorithms

2015 (37 years old):

  • fluency with computer science algorithms, data structures, searches, and sorts
  • west coast driving trip
  • scouting trips for places to live

5 Year (41 years old):

  • children
  • Galapag.us as a command-line/search box-driven game

10 Year (46 years old):

  • at the age when I should expect to create a life’s work or project that is “great” and “glorious”

 

Note: I’ll probably be adding to this, as I had trouble expressing what I wanted to say in a cohesive manner and worked on this over several nights.  I know I’m leaving tons out, and need to think more about future plans.

But hell, it’s so good to be in a state of strengthening and building now.

Newslint: My Project to Supplement Media Literacy

I just released a new site that I built, Newslint.  Check it out!

A while ago I was reading Hackernews and I saw someone posted a link to the Javascript source for joblint, an app that you copy and paste text from a job listing into, returning results on whether the job sounds fluffy, unrealistic, bro-ish, tech buzzword-y, etc.

I thought it was an awesome idea, and the implementation was such that it was easy to add rules, keywords, and phrases to check against.  Having just gone through a job hunt before being hired by a very professional group of people as a developer at The Barbarian Group, I was sensitive to junk in 90% of job listings.  I was also considering my love for news curation, media literacy, and good journalism, after having been an Army intelligence collector and analyst and then a social media operations analyst for a DHS contractor.  So I started thinking about making a port of joblint so that I could lint news articles in a similar way in an effort to explore media literacy.

pew

Fast-forward a little while and I was preparing to work on a Django project for The Barbarian Group.  I had already put in a lot of time learning Python while at ChatID as an engineering intern, but now I had to learn the Django framework.  Since Python and JavaScript are similar and flexible enough that I didn’t anticipate many problems, I decided I could try to port JavaScript joblint to Python newslint and then incorporate newslint into a Django server and experimental test-bed.

How It Works

During my time doing social media analysis, we hired for and trained for our analysts to be able to quickly assess whether news information was valid, credible, interesting to our client, potentially dangerous, environmentally relevant, etc.  More of an art than a science, this involved knowing which sources tended to put out good info, knowing the current situation and deciphering which new information would most affect the status quo, where the best sources of information in different spheres of influence could be found on the public internet.  This is actually a pretty difficult skill to acquire and that alone has the largest influence on the quality of analysis output.  That is, if only 5% of the info out there is actually game-changing, then less time has to be spent on the other 95% so that more direct analysis can be done on the 5% — but at the same time, the 95% of noise is still relevant as an environmental check.

Media literacy is crucial even if it’s not your job.  A lot of my Army friends are more conservative and they’ll post articles from certain biased sources that end up not being true.  And a lot of my liberal NYC friends will post stuff from advocacy blogs about the NSA and eavesdropping which are demonstrably false or short-sighted.  For others who don’t really consume the news, the tangential connections they have with the news are even more important.  Those decontextualized sound bites from the news are all those people will hear about an issue and so it will largely shape their opinion on the matter without more study.  FOXNews used to be on every TV all day, and now it’s likely you’ll see CNN instead.  Some people only watch The Colbert Report and The Daily Show.  Others watch the worst hours of cable news television, the afternoon lineups on FOX News and MSNBC.

It is crucial to understand how businesses buy people to write in newspapers or make TV ads or form political action groups to shape public opinion through blanketing the air with a specific message.  Non-profits, advocacy groups, and different areas of the government do it as well.  Whenever you see a poster advocating for or against a bill, you should always look up the group named in small print at the bottom and see who’s behind it.  It’s probably not a grassroots campaign — it’s probably astro-turfing.

In short, like any good intel, you should be suspicious of any information that finds its way to you because it most likely was intended to reach you, and wasn’t a happy accident or a sign of unstoppable progress towards that position.  Media literacy helps people decipher incoming input for true intent and agenda.

So that’s what newslint can help you do.  It takes raw text and looks up key words and phrases that indicate credibility, non-partisanship, and professionalism.  Do you read solid sources from solid journalists in solid publications.  Are you learning partisan phraseology that slants your opinion?  How objective and experienced are the people you read?

Here are all the rules for newslint.  I would definitely appreciate an email, or even better, a pull request, if you want to add more rules.

swears = [
 'bloody',
 'bugger',
 'cunt',
 'cock',
 'pussy',
 'dick',
 'douche',
 'jackass',
 'asshole',
 re.compile('fuck(?:er|ing)?'),
 re.compile('piss(?:ing)?'),
 'shit'
]
partisan_words = [
 re.compile('obama[ -]?care'),
 'libtard',
 'nobama',
 'death panel',
 'leftist',
 'communist',
 'malkin',
 'coulter',
 'sharpton',
 'sarah palin',
 'pinhead',
 'limbaugh',
 'o\'reilly',
 'krauthammer',
 'bachmann',
 'ron paul',
 'rand paul',
 'john bolton',
 'alex jones',
 'martin bashir',
 'perino',
 'karl rove',
 'stasi',
 re.compile('police[ -]?state'),
 re.compile('hipp(?:ies|y)'),
 re.compile('fly-?over state'),
 'wingnut'
]
pundit_words = [
 'krauthammer',
 re.compile('(?:thomas|tom) friedman'),
 'ayn rand',
 'john galt',
 'jesse jackson',
 'gold standard',
 'rand paul',
 'ron paul',
 'bachmann',
 'limbaugh',
 'o\'reilly',
 'gingrich',
 'slate',
 'glenn beck',
 'msnbc',
 'huffpo',
 'huffington[ ]*post',
 'fox[ ]*news',
 'coulter',
 'david brooks',
 'john bolton',
 'michael hayden',
 'alex jones',
 'scarborough',
 'chris matthews',
 'sharpton',
 'martin bashir',
 'sarah palin',
 'ezra klein',
 'perino',
 'malkin',
 'evgeny morozov',
 'karl rove',
 'drudge',
 'dowd',
 re.compile('chris(?:topher|) hayes')
]
rag_words = [
 'gawker',
 'tmz',
 'slate',
 'infowars',
 'buzzfeed',
 re.compile('the blaze'),
 re.compile('huffington[ ]?post'),
 re.compile('fox[ ]?news'),
 re.compile('drudge[ ]?report')
]
sensationalism_words = [
 re.compile('tears? apart'),
 'screed',
 'demolish',
 'crush',
 re.compile('brown[- ]?shirt'),
 'hitler',
 'gestapo',
 'snitch',
 'stooge',
 re.compile('game[- ]?chang(?:e|ing)'),
 re.compile('cutting[- ]?edge'),
 re.compile('bleeding[- ]?edge'),
 re.compile('marxis[tm]'),
 re.compile('cron(?:y|ie)'),
 re.compile('fema[ ]?trailer'),
 re.compile('chem[ ]?trails'),
 'delusion',
 'false flag',
 re.compile('racis[tm]'),
 re.compile('meme[- ]?wrangl(?:ing|e)'),
 'flagrant',
 'cult',
 'the establishment',
 'police state',
 re.compile('solutionis[mt]'),
 re.compile('shock(?:ing|er|ed)')
]

And Now, the Tech Details

I ported over the code (it’s not very large) in a day or two, then debugged it for a while.  It worked — I made some additions, and, like joblint, it can be run independently via the command line.  Then I forked joblint and turned it into newslint in a separate git repo.

console

Next I figured I could not only use the opportunity to learn Django but also learn other stuff.  I really hadn’t needed to use LESS or SASS up till now because I was working on code I could just throw classes into, but since then at ChatID and at The Barbarian Group I ran into projects where I wouldn’t be able to modify HTML markup and would have to traverse the DOM or find other ways to hook into the code.  So I set up for SASS and Compass and installed Grunt so I could have all my different tasks (uglification, concatenation, code linting, copying to static directory, etc.) automated.  Along the way I found autoprefixer, which you can run as a Grunt task to take any CSS3 stylings and automatically add support for browser extensions.  I also decided to try Django-Bower (based on Bower) for making updating to the latest version of my JavaScript dependencies easier, and within Django’s environment bubble no less.

Django is incredibly easy to use and you get a lot of control over it, which is something I like about express.js.  But then I’m also coming off learning Drupal (PHP) for a project, which seems like a black box most of the time.

I got a simple version of newslint running on a local Django server and then things snowballed; I fleshed out some JSON endpoints for an API, I enabled form submission for saving news clips, and I wrote some tests in Django’s TestCases and Python’s unittests.  Super-easy, especially after dealing with a somewhat problematic time spent figuring out correct resources and syntax for Angular.js tests with mocha for my project Momentous.

front

And then I figured I would try deploying this Django app to Amazon’s Elastic Beanstalk, because I’d never tried that before!  I ran in to some issues there; my static files directory was split up and not in a standard directory, I had newslint loaded as a git submodule under the newslint-server and automatic deployment services like EB and Heroku don’t like submodules.  I also would have had trouble getting underneath the EB abstraction to make edits directly to server settings.

I decided to tear that app down and just get an EC2 instance (m1-small).  It costs a bit, but not really that much, and I’ll probably take down the instance once there’s no traffic on it.

My small test app turned into a full day deploying the app underneath Varnish and Apache to a new ubuntu instance.  I plugged in memcached and set up mysql and added appropriate Django middleware to help get my pagespeed score up and remove warnings and errors.  The full control of an EC2 instance made this all super easy whereas I’m not sure how I would’ve managed dealing with the EB thicket.

I had some problems making sure my headers were set up correctly so that stuff would get cached okay but tweaks to Apache and Varnish settings, along with Django, helped to mitigate those problems.  Updates to code were as easy as a git push on my local computer and a git pull to the instance.

I ran some apache bench tests on the server and it seemed okay; one thing I think I ran into was that having a form on the front page slows down the response slightly because it’s not caching the page (CSRF token?).  ab tests to a non-calling API endpoint on the other hand were super fast.  Most of the time, pageloads are under 400ms, which is pretty sweet!  Thank you to the god of page loadability, Ilya Grigorik!

So then it was rather late and I was thinking, hey, how hard would it be to get a domain for this instead of the long EC2 address?  Well, whois’ing newslint.com actually showed that it wasn’t owned!  And namecheap sells domains for $10-13 usually so I picked it up and pointed it from namecheap to Route 53 and all the sudden very early in the morning I had a working newslint.com!

result

I found some more bugs the next day, for which I’m writing regression tests, but otherwise this has been a really successful learning experiment and confidence booster for my developer chops.  Really glad this worked out so well, and thank you to Rowan Manning for his joblint work and to The Barbarian Group for letting me be a developer.

Introducing Cryptstagram

[This is the long version of a post I wrote for The Barbarian Group’s tumblr.  We also got written up in Wired’s design blog, Laughing Squid, and PSFK, my first time getting such large coverage!]

Periodically at The Barbarian Group, a small team of new employees and veterans with varying skill-sets are brought together to work on Project Popcorn, a side project that embodies the inventiveness, technical skills, and values of the agency.

Within my first week as a developer for the agency, I was assigned to Project Popcorn, along with a senior art director, a senior copywriter, a creative director, and a technical director.

Cryptstagram is the resultant product of our teamwork. Cryptstagram is a web site that lets you steganographically encrypt a hidden message into any image, unlockable only with a specific password, and then apply glitchy Instagram-like filters to that image.

The National Security Agency whistleblowing story had just broken, and as a result we quickly became fixated on doing something involving cryptography and, to fulfill our artistic ambitions, on doing something involving the glitch aesthetic.

Cryptstagram-5

The glitch aesthetic is popular amongst our creatives and developers because it requires technical knowledge of formats and methods underpinning systems in order to then manipulate, pixel-push, and distort those systems towards the artist’s ambitions.

As a team we spent a lot of time discussing what tone we wanted to convey with Cryptstagram. How secure should it be end-to-end to pay respect to citizens’ cryptographic requirements and expectations of privacy? Should Cryptstagram’s filters create aesthetically-pleasing images or should they reflect entropy of input and interaction from the user? To what degree did we want Cryptstagram to be a statement about the NSA and privacy in general or just a reaction to the state of our relationship with technology?

In the end we decided on a reflective tone for Cryptstagram, emphasizing a common public desire to share stylized parts of ourselves in public but other parts of ourselves in private or semi-private. We should empower ourselves with stealth and style.

Cryptstagram-2

Cryptstagram-4

The Project Popcorn team also wanted to keep Cryptstagram open-ended, taking advantage of the fact that JavaScript is pervasive in users’ browsers and that most computation, SHA-1 encryption and decryption, and glitching can now be done with open source libraries, entirely on the client’s side, and using HTML5 Canvas. A JavaScript stack (Node.js/Express, MongoDB, and good ol’ jQuery) cached and served up via cloud services allows for rapid app development and easy deployment for a small creative team.

So, easily, a citizen could choose to encrypt her image with a message and save it without it ever touching an external server, then send that image via private means to intended recipients with a password that could be distributed in any number of ways. Or the citizen could share the image on Cryptstagram’s wall for others to see and even try to decrypt!

A perk of this open-ended design is that we can extend its utility: we (or you, via jsfiddle sandbox!) can add seasonally- or topically- themed filters for specific clients, a celebrity can release images with a password known only to fans, or people can use the Cryptstagram wall to create puzzles or scavenger hunts or just post encrypted messages publicly without drawing suspicion.

At The Barbarian Group, we’re invested in making things that are gonna be awesome, and in order to do so, we need to be just as able to wade through the technical limitations of CORS and CSRF as to create something people love to use and share online.

So, what creative uses can you come up with for Cryptstagram?

Momentous

After thesis and graduation, I took a break from Galapag.us to focus on getting a job.  I ended up getting hired at the wonderful The Barbarian Group as a developer (!!), about which I’m extremely happy, for that career change was exactly what I left DC and came to NYC to accomplish.

In the time since, I’ve been lurking in the nether regions of HTML5 canvas, HTTP headers, XSRF protections, and Drupal.  And now that I’m a little more comfortable with all the life changes around me, I have started coding for fun again.

I gravitate towards the MEAN stack for most any new fun project now: node.js/express.js being extremely easy to set up and expand, plus mongo which makes it super easy to drop in and manipulate with mongoose as an ORM, and then learning the intricacies of angular.js, which has made front-end state super easy.

And these projects I like to work on tend to relate to quant self and meta data about the self, which means they’re natural fits as an app on Galapag.us.  So with that in mind, I fleshed out a new interface for collecting moments at Galapag.us.

Momentous

There are many categories of data on Galapag.us, like exercise, school, and work, but there are also more nebulous forms like moments.  What discrete events did you attend today?  What special interactions did you have with random strangers, or with your loved ones?  What was the quality of those experiences?  That’s all Momentous does — it’s an excuse for me to build a quick angular.js app that collects the random moments of my day.

momentous

You can give it a shot at https://galapag.us/momentous/ (you’ll have to login/register) or just use the main moments page at https://galapag.us/c/moments/ if you like.

These sorts of mini-apps allow me to work on Galapag.us, experiment with new frameworks, and build nicer front-ends than the messy base template I’m currently using.  It’s still a work in progress.

The Future

I’m loving angular.js for building nice interactive apps that react quickly to the user and to the backend.  I’m thinking it’s a huge leap forward for writing HTML and JavaScript that expresses its meaning and functionality within the HTML source (by way of directives and filters) instead of having events bound to HTML states but stored in separate JS files.

From there, I’m thinking there’ll soon be code editors for JavaScript that build server-side API routes to receive and respond to the requests written by the user in client-facing scripts.  So you’d write some angular.js or jquery front-end stuff to hit an API route, and the IDE would automatically build the route to receive it and pull it from the database.  There’s still some disconnection between development for the client and for the server, but I think that’s what the next major leap forward will be for writing apps for the web.

Opening Up Galapag.us for Alpha

My thesis project for NYU-ITP has been Galapag.us, a tribe and ecosystem for promoting the idea that we should be radically open and transparent with our data so that we can form and share metrics to measure our progress and success in different areas of our lives.  More info at the front page of Galapag.us.

User Zero

I came up with the idea in 2006.  An email I sent to my Army buddy in April, 2006:

I sort of had an idea but it seems like it’ll be difficult to build out.  My idea would be for something similar to Xbox Live’s ranking system.  Except it’s for your life.  Privacy issues aside, people would volunteer to put in as much personal info as they want.  At first it might seem cumbersome putting in so much info but I think as myspace and other services have shown, people are willing to do it if it means it cultivates their identity.

So for instance you put in your income and number of kids and connect your accounts for online game rankings (like in Halo or Battlefield 2) and your exercise plan and your birthdate and your finances and investments and how many web sites you’re on (like myspace, digg, yahoo, etc.) and from all that data, the company would generate statistics that break down your life and give you info about how much time you spend on certain tasks, how efficient you are with your money, what your online reputation is.  Stuff like that.  The core would be statistics…anonymous statistics I think so people won’t have any incentive to forge their results.  The point would be to turn peoples’ lives into a numeric game where they can see how they rate in certain aspects of their lives.  Think of all those online quizzes people take about what kind of lover they are or what their personality is.  That could be tabulated into the statistics, which could be searchable or broke down any way the person wanted.

At the end of a year, we could look internally at our statistics and go visit the top overall people in person to go verify their data and videotape their lives, interview them.  Then a winner would be announced…like the best person award.  Heh…there’d be so much controversy and whining and competition if it caught on.  Then we could write a book about our experiences going out and discovering what makes someone “the best” compared to everyone else.

So…that’s my idea so far.  Sort of like a real-life RPG.  Perhaps we could offer points for real-world scavenger hunts or traveling to different countries around the globe.  What about having life coaches for certain segments, if someone was weak in an area like professional development?  I was thinking we could also offer points for accomplishing certain tasks like humanitarian work.

A lot of stuff happened in the meantime: I got out of the Army, went to study foreign policy in DC, worked for Homeland Security, moved to NYC for school.  And so now I’m wrapping up the thesis, which allowed me more than a semester to work just about full-time (including any waking moment) on trying to make Galapag.us a reality before I can either A) get funding or B) get an engineer/developer job after school.

So I present Galapag.us for thesis on May 15 at NYU.  I have two weeks still to work on it before then.  I think I’ve gotten it to a point where I can start letting alpha testers in to explore, and think about it.  My work log has been tracked on the thesis blog.

Beginning Alpha Testing with Thesis

Want to help alpha test or just look around?  Give it a try at https://galapag.us/login and see what you think!

Here’s the slide deck I’ll be presenting at thesis:

Stack

  • Amazon EC2 small instance with ubuntu
  • node.js/express.js: So easy to build a site using this framework.
  • varnish/nginx+ngx_pagespeed: Caching, run-time optimizations for faster page loads/downloads.  Routes to https and socket.io server too.
  • python scripts for maintaining server default state
  • celery for queuing
  • redis for temporary data dumps and lookups
  • mongodb for permanent data storage
  • angularjs for the comment system
  • mocha, unittest, qunit for unit testing in python and javascript

 

I know the site’s confusing — like an airplane pilot dashboard.  It’ll become more cohesive over time.  A lot of things aren’t quite working yet, or they have filler data to get them going.  Apologies for that.  For more familiarization, try the welcome demo.

But here are some features that are worth checking out:

Comment System

Comments will be available for tribe forums, formula critiques, peoples’ profiles.  I decided to use angularjs so I could learn how to build SPAs with it!

ss_comments

 

Tribes

By tracking individual data, one can also track internal company metrics and state-level happiness metrics too!

ss_graph

 

The Islands

Each island has its own weather, environment, and bonuses/penalties for certain user behavior, so it benefits you to live on the island that incorporates your style best.

ss_islands

 

Professions and Skills

What does it mean to be “good” at something? Are companies hiring the most qualified candidates? How do we standardize that?

ss_professions

 

Your Genome

A profile for your data.  You get reputation scores in different areas.  Those scores are determined by which formulas you choose to use.  You can also see your internet of things (devices, pets, objects) is on the bottom right, while you’ll also be able to create gaming characters using your own data.

ss_profile

 

Quests

You can complete quests within Galapag.us to gain experience.  Some tasks will be data-gardening for other people, some will be to introduce gaming elements, others will be to visit lesser-seen parts of the site.  But mostly the quests should be geared towards helping others.

ss_quests

 

Status Bar

I love github’s command bar. I want users to be able to do most everything through the search bar.

ss_searchbar

 

Tribes and Their Genomes

Tribes are important to our identities. formulas serve as their DNA.

ss_tribes

 

Universal Reputation Lookup

The left-side widget can be opened on most pages to see what reputations the people named on them have.  I intend to allow people to look up reputations from just about anywhere.

ss_widget

 

 

API/Sandbox

Galapag.us will have an API to access one’s data, as well as common stats such as state population census results, zodiac signs, and global stats.  Plus a place to test the routes, within the sandbox:

ss_api

 

So with all that, please go help alpha test at https://galapag.us/login to begin your exploration of identity and reputation.  Thanks, and come to the ITP Spring Show if you can!

ITP Glitch: Editing Civilization Save Files

So far in my Glitch class, we’ve experimented with databending and transcoding images, audio, and video, using a variety of different techniques, but mainly trying to de-abstract how software handles file formats so that we can edit the actual bytes of files to alter the expected output — e.g. shifting the bytes by one in order to cause the image’s dimensions to warp or change color.

Here is our homework assignment for this week:

  • Find and write about a file format / specification. For example, ISO/IEC 10918 (JPEG),  ISO/IEC 1172-3 (MP3), ISO/IEC 14496-3 (AAC) GIF89a, ZIP, MIDI, etc. See List of Codecs for more. You should provide an analysis of any features of the format that may be interesting to glitch artists working with it, or provide a history that explains the various biases that are reified by the format, and the advantages and disadvantages of those biases in actual use.
  • Create your own file format / specification. A complete specification will allow someone else to both encode media into an intermediate representation, and then decode it back to an approximation (lossy) or exact replica (lossless) of the original. As far as “types” of media: audio, images and video are popular; dance, haptic sensations, taste, legal requirements, written language, etc. all count as media. You must run at least one piece of media through your encoding / decoding process, but you may do this by hand — it does not need to be executed by a computer.

 

The second assignment seems pretty difficult at this point.  I looked into making “extrafiles” and found this OS X app, extrafile, which introduces a series of unique file formats such as one called “Block Ascii” which is only 1bit color:

So you can save a file into one of those formats using the app.

Coming up with my own file format specification would take a while, and I’m not sure how I would transpose data through media processing without losing everything in a jumble of noise.  However, I did do an assignment last semester dealing with genetic inheritance which involved passing a database record of peoples’ stats (1-10 sliding scales of peoples’ stats, like creativity or strength or determination) into a Wolfram cellular automata algorithm, via a node.js server passing JSON, so that the output looked like this:

But that’s not really a file format per se.  It might be interesting to be able to encode someone’s physical and life characteristics into a photo file of them, so that in, say, 50 years, someone could open up the photo image and see what the person’s biometrics and other statistics were at the time the photo was taken.  Hmm, possible idea!

So in terms of describing another existing file format, it made me think about when I first played Civilization as a kid.  Back then, there was no real internet to speak of, and I was using Prodigy* via a dialup modem so that I could (slowly) browse the Civ forums there.  Some people who knew their shit were hex editing their save files so that they could test the limits of the game (e.g. I think cities couldn’t get any higher than 99 population or something, because of memory issues from going to 3 digits) and cheat to win.  That stuff was over my head so I never played with it, but I remember it to my day as a way that some people were able to change settings for games, registration verification, etc.

Anyway, even though it’s hard to look up info on the first couple Civ games (because most Google results now are for Civ 4 and 5), I did find pages with a pretty good write-up of the Civilization 2 savefile format and of the Civilization 3 BIC file format.

On data types:

"...there are three data types: The null-terminated string is a variable length type for text. It always ends with a byte value of zero, meaning a string of 9 bytes only has place for at most 8 characters. The signedshort integer is a numeric data type taking up 2 bytes. It can take values from -32768 to +32767. The unsigned byteshould be self-explanatory. It takes values from 0 to 255."

For handling states of engineer units:

An engineer in CIV II generates 2 points per turn. A settler generates 1 point. These numbers remain constant when the “Base time for engineers to transform terrain (x2)” is changed in the Rules file. With the “Base time…….” set to 20 (vanilla CIV II), twenty points are needed to transform Grassland to Hills.
Consequently, if the Grassland to Hills transformation command is entered on turn 1, a single Engineer will complete the transformation on turn 10. Its byte 15 is changed to 2 on turn 1, is incremented by 2 on each of the following 9 turns, and reaches 20 on turn 10.
If 2 engineers (E1 and E2) are assigned to the task, their byte 15’s change as follows:
UNIT -- E1 E2
 TURN 1 04 00
 TURN 2 08 00
 TURN 3 12 00
 TURN 4 16 00
 TURN 5 00 00 Task completed

Some culture variables in the savefile:

CULT SECTION (Culture)
  4	char		"CULT"
  4	long		??? (33)
  4	long		??? (5)
  4	long		number of culture opinions
  For each culture opinion:
    4	long		length of culture data (88)
    64	string		culture opinion name
    4	long		chance of successful propaganda
    4	long		culture ratio percentage (3:1 = 300, 3:4 = 75)
    4	long		culture ratio denominator (e.g. the 1 in 3:1)
    4	long		culture ratio numerator (e.g. the 3 in 3:1)
    4	long		initial resistance chance
    4	long		continued resistance chance

It might be interesting, given the exact formats and memory sizes of the variables, to make a random savefile generator, and then open it up within the game to see what you get.  Would the game be playable, or does it require a certain logic within the savefile where, say, you couldn’t have some variables if other variables were of a certain value?

Tool-assisted speedruns have become big as a result of savefiles.  Basically, if you can save the exact state of a game in a file, you can restore from that point without limit.  I played through Zelda: Ocarina of Time via an emulator on my computer back when I was in college, so it was easy to quickly restore at difficult points in the game.  Apparently all this is called save scumming.  This is how you get those weird videos every once in a while that show some Japanese player doing a perfect speedrun through a custom-made annoying-as-holy-fuck Mario Brothers-family game, e.g.:

That video introduces some followup questions.  Why do people subject themselves to such torture?  How did that person not kill everyone in his building?  Is there a good reason he never uploaded past part 3 since 2007?  Did he go on a murderous rampage?

In terms of savefile formats, I know that later Civ games used XML for settings files and maybe savefile formats?  What if you could translate the format of one game’s savefiles to that of another, so you could play your character in a different world?  More importantly, what if there were a standardized player file format that you could re-use between different games through a secure authenticator? (say, you used it on Steam)  This has been done with some games and some networks (such as Mass Effect on Xbox Live), but how about an open player file format?  And how about if it included your real-world Galapag.us characteristics and history?

Understanding Genomes: Extracting DNA from Strawberries

This semester I’m taking Yasser Ansari’s Understanding Genomes class.  Professor Ansari is a co-founder of Project Noah, which lets mobile users become urban citizen scientists, taking photos of flora and fauna to post online for identification, unlocking badges, helping kids learn in class, etc.

We’re already reading the biopunk manifesto and James D. Watson’s book The Double Helix, about his and Crick’s Nobel-winning discovery of DNA’s helical pattern.  We’re also getting to use Lore, which has emerged as a more ajaxy user-friendly, minimalistic-looking alternative to Blackboard and its competitors.

Last week we made online dating profiles using different model organisms to learn about why they’re used, how many base pairs they have, etc.  My model organism was the fruit fly (Drosophila melanogaster); here’s the Okcupid profile.

So this week, we had to use a recipe for extracting DNA (nucleic acid extraction, includes RNA as well as DNA) from strawberries.  Here are the instructions:

DNA Extraction at Home

In this exercise, you will extract DNA from strawberries. Below is a list of materials along with a protocol courtesy of Genspace. Document the process and pay attention to the procedure so we can discuss your experience in class. Feel free to work in groups and share materials. Read through the materials and protocol before you start the experiment. Contact me if you have any questions or concerns.

Materials:

  • Strawberries (at least 2 juicy ones)
  • Quarter teaspoon of Table salt/NaCl (don’t use too much of this)
  • A teaspoon of dishwashing detergent (any brand will do)
  • Meat Tenderizer (any brand should work)
  • Half a glass of water
  • Ziplock baggie (1 quart volume)
  • A disposable coffee filter
  • A plastic funnel (to hold the coffee filter in place. Feel free to ditch it if you have a better method)
  • Rubbing alcohol (91% isopropanol works better than the 70% strength)
  • Ice or a fridge to cool the rubbing alcohol prior to using
  • A tall glass (Note: don’t drink this stuff.)
  • A thin wooden stick or skewer (to pull out the DNA)

Protocol:

  • Step 1: Place the strawberries into a ziplock baggie.
  • Step 2: Add the detergent, meat tenderizer and salt (again, don’t use too much salt).
  • Step 3: Seal the baggie and “mechanically disrupt the cells”. This “science speak” for you to start squishing the strawberries in the baggie with your hands. Squish and mix them really well with all of the ingredients. Knead the bag for about 5 minutes until you have the consistency of a smoothie. (Note: again, don’t drink this crap)
  • Step 4: Open the baggie and add a little bit of water. Shake it up a little.
  • Step 5: Place your filter in the funnel and place this in the glass. Now pour the contents of the baggie (the strawberry mixture) into the filter.
  • Step 6: Let the contents drip through the filter into the glass. It should look milky pink. Be patient! If the filter is working too slowly, you can gently squeeze it, but not too hard or it may break.
  • Step 7: Holding the glass at an angle, very slowly pour the chilled rubbing alcohol down the side into the strawberry mixture. You want to pour it slow enough so that a layer of the clear alcohol forms on top of the pink layer underneath, since the alcohol is less dense than the water layer containing the strawberries. If you pour too quickly you won’t see this layer form, so remember to do it slowly. Pour enough alcohol so that you have at least a 2 inch layer on top.
  • Step 8. Now set your glass down. You should start to see a wispy white layer start to form between the bottom pink layer and the clear alcohol on top (like the photo shown in class). It should take about a minute or more to develop. It will start to look like a small ball of cotton. Sometimes it looks like many small pieces of cotton. Give it some more time and it will gently rise to the surface. This is the strawberry’s DNA starting to precipitate!
  • Step 9: Using your thin wooden stick, you can now fish out the DNA. It will be rather “stringy” since DNA molecules form long chains. These types of molecules are known as “polymers”.

Some questions to think about:

Why use strawberries?

Strawberries have far more DNA than other readily-available materials, and their cell membranes are delicate, which means they’re easy to remove for access to the DNA. [possible citation]

What’s the table salt for?

Salt helps the DNA precipitate when the rubbing alcohol is added. It ensures that proteins in the cell are not released from the rest of the solution with the DNA. [citation #1, #2]

What about the detergent?

The detergent breaks apart the cell membranes in the strawberries. [citation #1#2]

Meat tenderizer, what’s that for?

Meat tenderizer contains bromelain and papein, which break apart proteins, allowing us to get at the DNA. [citation #1#2]

And the rubbing alcohol?

DNA is insoluble in alcohol, so it will precipitate for us to see with the naked eye, and for us to extract it in clumps. [citation #1#2]

Documentation

Phil, who dropped the class, wanted to do this homework with me anyway (we’re such a good pair) so we worked on it Monday night.

Me, as Phil tests the camera

Prepping the strawberries with salt, meat tenderizer, and detergent.

Mashing up the materials.

The materials turn dark purplish

Phil crushing the materials like a pro

Pouring the contents into the filter

Filtering the materials through a coffee filter plus funnel

Phil macgyvers together a pretty awesome pipette technique using a straw to drop in rubbing alcohol slowly

Rubbing alcohol separating the DNA

Separated DNA

DNA tendrils

Redial Final Project: Hermes Ordering System

In my Redial class, we learned about the dark arts of telephony in an age of digital, packets, and VOIP.  Once we learned the basics of operating in an Asterisk telephony server environment, we moved on to how to write scripts in Ruby, bash, whatever to interact with Asterisk during a phonecall.  We set up our own Rackspace servers loaded up with Asterisk and used our professor’s script to install needed software on its Ubuntu instance.  I’ve saved that script to add extra stuff when I’m setting up a new Ubuntu server from scratch.  Ubuntu basic literacy is now attained.

That part was fun enough, but it started to get interesting when our badass professor, Chris Kairalla, published some code to Github for interacting with a web front-end in JavaScript by using socket.io, node.js, and a Ruby-Asterisk Gateway Interface script interfacing with calls placed to Asterisk. That’s when the realm of ideas greatly opened up.  Real-time interaction with the phone dialpad to control a screen, an Arduino, robots, electric switches, whatever.  And with the explosion of JavaScript and Node.js, it couldn’t have come at a better time.

Here are the class notes:  http://www.itp-redial.com/class/

And the Github repo: https://github.com/itp-redial/

I set out with confidence, knowing I could use Professor Kairalla’s brilliant tinyphone code, which provided the framework to facilitate an incoming call into a caller object that one could manipulate in the web page DOM.

My idea was to build a bar or restaurant menu displayed on screens around a bar which would let customers place orders by just pressing digits on their phone, instead of waiting on waitstaff or having a gimmicky touchscreen or table screen.  The orders would be tied to someone’s phone number so paying for bills wouldn’t require the awkward splitting up of the bill, and customers would be able to use a very personal device to them, their own cellphone, to control interaction in the bar/restaurant.  Orders could be processed quickly, verified afterwards, and immediately incorporated into a database which could provide business intelligence in the form of statistics on food orders, frequencies, favorites for each customer, etc. All in a scalable, modular framework such as Node.js + Express.

Previous blog documentation: post #1.

Demo video:

Screenshots of some functionality:

Initially I figured I’d follow advice and build out the API first.  For that, I’d need an admin dashboard to perform CRUD operations to build up the menu.  I’m pretty tired of having to build out AJAX and menus to add, remove, edit, etc. so I’ve started building a Node.js app with all that stuff built-in, including JSON operations and other common usage events so I don’t have to keep testing it.  Anyway, I got my menu add functions working so I quickly started adding all my favorite Stellas, Dark & Stormies, Mojitos, etc.

Then I spent most of my development time building an interface that could only use the dialpad numbers 1-9, 0, *, and #.  Because I was fairly ill-versed with JavaScript still, I actually ended up refactoring my interface about three or four times, each time becoming more and more modular, using JavaScript objects which contained each “menu” (i.e. a beer menu that would point to each individual beer’s screen, which would pass to another object that would fill out a screen to let you order the beer) pushed into an interface that had no more than 8-9 choices.  Then the screen would have to have text displayed large enough to read, yet still accomodate perhaps 3-4 simultaneous callers at once.

var menuFoodObj = {
	name : "Food/Drinks Menu",
	upMenu : "Main",
	currentMenu : "Food",
	id : "",
	p : ["", "Beer", "Wine", "Liquor", "Non-Alcoholic Drinks", "Water", "Burger", "Specials", "Chips & Salsa"],
	pClick : ["", "Beer", "Wine", "Liquor", "Other", "Water", "Burger", "Specials", "Chips"]
}

By the end of the refactoring though, I had a really nice (but still a little soupy) interface of JavaScript objects being pushed into the DOM by jQuery.  I learned OO JavaScript pretty well although I didn’t understand fully how to create new objects and then properly make methods for objects (I was declaring new functions within objects).  Since I did most of the development and testing just using the web browser and mouseclicks, I was happy when all my refactoring allowed me to easily map a phone’s keypresses to the interface interactions using jQuery’s .triggerHandler event.  My code is still quite soupy though, and I think I’m about ready to learn backbone.js for injecting data from models into a static template — it would have been perfect for this type of project where the layout remains mostly the same but the data changes frequently.  I’m happy, though, that it was easy for me to instantiate new user objects (e.g. user[1] = new User();) for the 4 users that Hermes currently supports.

I actually ran into a weird problem after I’d hooked in caller keypresses, which I thought at first was within the DOM.  Keypresses from my softphone (I used Zoiper to connect via SIP) were happening twice and I thought the click event handler was being called twice in the DOM.  This did not occur on normal phone calls over, say, a cellphone.  Further inspection saw two keypresses coming across in Asterisk:

 <SIP/general_sip-000000ae>AGI Rx << WAIT FOR DIGIT -1
 <SIP/general_sip-000000ae>AGI Tx >> 200 result=50
 <SIP/general_sip-000000ae>AGI Tx >> 200 result=50

Prof. Kairalla found that we were actually picking up feedback on my laptop from the call being made over the software.  Once we muted my laptop, the audio wasn’t being sent to the Asterisk server, which had been detecting the extra audio as what it assumed was an extra keypress.

I added an option to check news headlines from Google News’ API, which has been deprecated so will probably disappear soon?  The other bad thing about this was that you can’t really link to the body of an article since 1) Google News just provides the link and the source’s weak abstract and 2) news sources are so proprietary about people stealing their content.  APIs always leave a bad taste in my mouth because they’re hamstrung from the outset.  Anyway, there’s a definite art to creating headlines, as I learned in my previous job covering the news cycle, and most headlines are expertly packed with relevant info, enough for you to get an idea of things just at a glance.

I also added features for sending drinks/orders to another table, to facilitate social interaction within the bar.  Trivia can be played as well, though it wasn’t built into a central system where scores are kept.  Right now it just tests a user’s response against the correct answer and then generates a random question.  Another feature added was to flirt with other tables and send them messages.

I used Flowroute, which was super easy to set up in my sip.conf and extensions.conf for Asterisk, using Flowroute’s web-based cheatsheets for config settings unique to my account.  Very nice set-up screens there.

emailjs and Twilio

To process orders (when someone wishes to buy a beer, but he hasn’t received it yet) and tabs (for delivered food/drinks), I took advantage of node.js’s awesome modules and found emailjs, which lets you package up a JS object and send it as an email via SMTP to, say, gmail.  So when a customer places an order in Hermes, the order is sent via email to the company for later verification.  Easy, piece of cake.  Can be used with any SMTP server you have an account on and that has permissions.

Then Prof. Kairalla mentioned that it would be great if a customer received a text message thanking them after they close their tab.  So I bought a phone number on Twilio ($1/mo) and used the Twilio SMS API to package up another JS object with the tab details, and then have it again email the company but also text msg the caller with his closed tab details.  I actually had some problems with this and didn’t like the available Node.js modules (because they didn’t include examples for SMS), but luckily I found Dustin McQuay’s godsend blog post in which he posts some easy node.js code as a sendSMS() function, which I used to get my Twilio SMSs working.  I then had further problems but I found out I had pasted in the wrong Twilio auth token, so once I re-checked that, it then worked!  I feel as though the auth token changed because I funded my Twilio account, changing it from a free account to a paid one, and changing the token.  I assume?  Anyway, too much time wasted on that…

I also got a fraud alert from my credit card after trying to fund my Twilio.  Apparently they get a lot of fraudulent credit card funds, which makes sense since I’m pretty sure telephony and stuff related to it are havens for people trying to commit fraud, fake identities, and game systems.  But my credit card company cleared my purchase and I was fine within minutes.  Just thought it was worth mentioning.

Conclusion

I demo’d this in class and had 4 people call in.  They quickly broke it, but it was mostly the display of each person’s menu on the page.  That’s an easy UI fix.  I also found that I would need to make each person’s number more clearly visible so he’d know which interface he’d be working on.  But I felt like people were very quickly navigating through the menus and would learn it fairly quickly once they figured out it was just using a phone dialpad.  One thing that broke the demo was that the bottom bar (telling you how to use ‘*’, ‘0’, and ‘#’) got pushed down so people couldn’t see it.  Big problem there!

This app works hand in hand, I think, with my other app, Karaoke Flow.  I would love to create products to make the bar/restaurant/club/concert/dance party/rave/big gathering experience more enjoyable for people looking to have a kickass time and more profitable for the businesses providing it to them.

And I’m blown away with what is possible now with socket.io and Node.js.  All these projects I’ve wanted to do, like a web-based MUD to hook into Galapag.us, real-time chat, real-time flowing data, it’s now possible.  Love it.  And Redial…what a great class.  It’s recaptured the mystique of phones before they went smartphone, the weirdness and coolness of the phonebooth which has become a portal in pop culture (Doctor Who, The Matrix, Danger Mouse), and the fun of messing with somewhat quirky and buggy technology after it’s fallen out of vogue.  And to see how easy it is to set up a Vonage-like system!

Nature of Code Final: Genetic Crossings

Professor Daniel Shiffman’s Nature of Code is outstanding.  Just check out the syllabus. And play with the Nature of Code repo Processing sketches hosted on Github. And be sure to get his book when it comes out! We covered some basic algorithms for simulating inside the Processing environment vectors and forces/repellers, genetical algorithms, Wolfram cellular automata, neural networks, autonomous agents, flocking behavior, particle systems.

Background

For me, the pull towards genetic algorithms, heredity, fitness, evolution, Punnett squares, etc. was great, so my project for my first intro to computational media class turned into my Nature of Code final.

My "EUDAIMONIA" Tattoo

My Genetic Crossings project attempts to create a simulated environment where people exist within a world connected to God, the peoples’ religions, their nationalities, and each other.  They produce offspring based on characteristic attraction rules (for my demo I only used “appearance”, “money”, and “religion”, but only to demonstrate what was possible –I would like to create a more fully-formed algorithm for my personal reputation/identity ecosystem Galapag.us to approximate and adjust to the infinite ways that people become attracted to each other and become married or have children or devote themselves to the other), and they can die.  Their well-being or happiness (what in Galapag.us will be eudaimonia) is dependent on their quality of living within their religions and nations.

Map View

A video from an older version:

See previous documentation on this project:

 

Here are the initial characteristics I created for each person.  There are so many more yet to add! Strength, intelligence, wisdom, charisma, stamina, wit, humor, education, creativity, responsibility, discipline, honesty, religiosity, entrepreneurialism, appearance, money, gracefulness, stress, health, luck, talent_math, talent_art, talent_sports.

Initial nationalities/regions: USA, China, EU, Africa, South America.  Characteristics of security, innovation, job opportunity, immigration policy, life expectancy, education, sanitation, standard of living, pollution, biodiversity, crime, political freedom, and nutrition.  By no means comprehensive.

Initial religions/spiritualities: Christianity, Judaism, Islam, Taoism, Confucianism, Hinduism, Buddhism.  With characteristics of commercialism, morality, hierarchy, portability.  Obviously these need some tightening up/additions/discarding.

Final Project

So in my midterm I managed to add Verlet 2D physics to the sketch so that people, nations, and religions have connections to each other which make them bounce around like they’re on springs, relative to their attractions and strength of ties to each other.

In my final proposal, I sought to add the following:

  1. Wolfram cellular automata
  2. Connect to external database
  3. A selection algorithm to choose certain parts from parents based on mutation rate and fitness to pass along to new offspring
  4. Interaction sliders to change variables
  5. Macro events that affect well-being of all objects in the world, such as earthquakes or war
  6. Micro events that affect individual well-being, like rites of passage
  7. Discrete clusters of people, mostly based on familial strength of ties, instead of one big clump of people in the middle like in my midterm

 

I managed to get most of this done.

I enjoyed adding Wolfram CA.  I wrote a node.js app for express which would act as a JSON middleman between my main genetic crossings sketch and a wolfram sketch.  Basically, when viewing the chromosomal stainings (genotypes), you can click on the CA button to the left of a person’s staining, and this will pass a JSON object (using Prof. John Schimmel’s Processing-Nodejs code) to the node server, which is detected within 5 seconds by a polling timer within the wolfram sketch.  The wolfram sketch then uses the JSON object as its data to apply rules to to construct a pattern unique to the selected person’s genetic characteristic code.

1) I added a third parameter to the rules which would display as either black or aqua, depending on the CA rules.  Prof. Shiffman’s code used base-2 groups of 3, which had 2^3, or 8 total possible combinations (using only the digits 0 and 1), but I used base-3 (0 = white, 1 = black, 2 = aqua), so it became 3^3, or 27 possible combinations.  For this ruleset, I duplicated Wolfram rule 90 three times, then added a few extra codes.  The triangular look seemed the most visually interesting for what I was doing.  Anyway, what was cool about Prof. Shiffman’s code was that it keeps streaming the pattern from bottom to top.  So when a new person is clicked on, that person’s signature is integrated into the flow seamlessly once the sketch loads the JSON off the node server.  I don’t know that any of this actually is useful except that it looks cool and uses principles from class and maybe, just maybe, shows someone’s digital characteristic “signature”.

Streaming Wolfram CA on top of Processing sketch code. Console of Processing shows a passed JSON object.

2) I’m happy I was able to set up a MongoDB to be accessed via node as well as by my Processing sketches.  When I build out Galapag.us, I’ll be able to pump out JSON objects of actual users into these sketches for data visualization.  I’d been wanting to do this since my first semester and now it’s done.  Most credit goes to John Schimmel though for writing the hook into node though!

node.js instance output including passed JSON object

3) I realized that I already had some mutation within my matingDance.sex() function, once the two parents’ characteristics were passed to a matingDance.punnettSquare() function.  Before, the function would just average the two parents’ characteristic values together and then add or subtract a random amount from them for mutation.  What I changed was making the function choose randomly from either parent’s base characteristic.  So if one parent had 10, and the other had 1, the result would not be 5 (rough average) but either 1 or 10.  Then I would offset a random amount (hardcoded as 4) if mutation kicked in (if a random number between 0 and 50 was less than 2, for approximately a 1 in 25 chance of mutation per characteristic).  What this ended up doing was increase the diversity within the genepool and more accurately reflect reproduction.  I still need to tweak these numbers to get more consistent levels of variety but the algorithm is mostly there.

Here’s a view of the chromosomal stainings before I changed the algorithm — here it averaged the parents’ traits, which, in the case of this sketch’s iteration’s octomom, created many extremely similar offspring:

Before

And here’s the view after picking from either parent and allowing for a slim chance of mutation:

After

I feel as though the end result has a more diversified population with more variability between generations and individuals.  I need to tweak this so that if the value from a parent is at 1 or 10, it may mutate in only one direction, but here’s the matingDance.punnettSquare() function:

int punnettSquare(int comp1, int comp2) {
    int mutation = (int)random(0, 50);
    int dominance = floor(random(0, 1.99));
    int crossover = 0;
    if (dominance == 0) {
      crossover = comp1;
    }
    else {
      crossover = comp2;
    }
    if (mutation < 2) {
      crossover += (int)random(-4, 4);
    }
    // don't want it to be out of bounds
    // TODO: fix so it can mutate only one way if parent is 1 or 10
    if (crossover < 1) { crossover = 1; } else if (crossover > 10) { crossover = 10; }
    return crossover;
  }

4) I didn’t add sliders to change variables mid-sketch — at this point I can use variables pre-set in the main class but I’d like to make the interface more user-friendly and interactive later.

5 and 6) I didn’t do macro and micro events because I figured they’d just require making a button that, when pressed, would cause particles’ values to change.  What would be interesting would be to have random events happen based on their likelihood to occur and then some events would have permanent effects (damage to peoples’ personalities) or temporary effects (nationalities’ well-being that would later recover).  This kind of introduces the possibility for individual peoples’ health and whether they have injuries/disabilities/diseases/gifts/talents.

I did add a Ritual class though, which only includes right now a funeral function.  When funerals are recognized by a culture (by pressing ‘f’), the dead are removed from view on the map and their attraction springs are removed as well.  What this is supposed to represent is that funerals are a way for the living to remember the dead and then put them to rest so that the living can move on and create new ties with the living.  I do like the idea that we retain our ties to the past, which can sometimes become weaker in death and sometimes become even stronger.  I didn’t model that yet.

public class Rituals {

  Rituals() {
  }

  void funeral(boolean funeralsRecognized) {
    println(funeralsRecognized);
    for (int i=0; i<numPeople; i++) {
      if (person[i].parent1 != -1 && person[i].parent2 != -1 && person[i].alive == false) {
        if (funeralsRecognized == true) {
          physics.removeSpring(parentSpringArray.get(person[i].parent1Spring));
          physics.removeSpring(parentSpringArray.get(person[i].parent2Spring));
          physics.removeSpring(parentMinDistanceSpringArray.get(person[i].parent1MinDistanceSpring));
          physics.removeSpring(parentMinDistanceSpringArray.get(person[i].parent2MinDistanceSpring));
          person[i].lock();
          person[i].display(0);
        }
        else { // TODO: re-reference spring after it's recreated?
          parentSpringArray.add(new VerletConstrainedSpring2D(person[i], person[person[i].parent1], person[i].parent1RL, random(parentGravity1, parentGravity2)));
          person[i].parent1Spring = parentSpringArray.size()-1;
          physics.addSpring(parentSpringArray.get(parentSpringArray.size()-1));
          parentSpringArray.add(new VerletConstrainedSpring2D(person[i], person[person[i].parent2], person[i].parent2RL, random(parentGravity1, parentGravity2)));
          person[i].parent2Spring = parentSpringArray.size()-1;
          physics.addSpring(parentSpringArray.get(parentSpringArray.size()-1));
          parentMinDistanceSpringArray.add(new VerletMinDistanceSpring2D(person[i], person[person[i].parent1], random(parentMinDistanceRL1, parentMinDistanceRL2), random(parentGravity1, parentGravity2)));
          physics.addSpring(parentMinDistanceSpringArray.get(parentMinDistanceSpringArray.size()-1));
          parentMinDistanceSpringArray.add(new VerletMinDistanceSpring2D(person[i], person[person[i].parent2], random(parentMinDistanceRL1, parentMinDistanceRL2), random(parentGravity1, parentGravity2)));
          physics.addSpring(parentMinDistanceSpringArray.get(parentMinDistanceSpringArray.size()-1));
          person[i].unlock();
          person[i].display(1);
        }
      }
    }
  }
}

7) Discrete clusters.  I ended up adding relationships between people and their parents from just constrained springs to a combination of constrained springs and minimum distance springs.  What this would change is that a person’s distance from his parents is both constrained to no more than a certain length but also more than a minimum length, so that they can both be more visible instead of overlapping visually, and also be more clustered together.  I found that this makes certain groups on the map appear more clustered instead of forming a big ball in the middle.  I still need to do more work on this though because as there are more people in the sketch, the big clusterfuck returns (because there are too many connections between everything and I can’t zoom in closer to see the gaps and relative spacing between different networks).

I converted a lot of my arrays of spring connections over to one large ArrayList, which I think was easier to deal with in the end in terms of manipulating them after they were initiated into the environment.  I did find, however, that I had to pass a reference to the spring’s number (since it was just an ArrayList entry) to the person’s class instance so it could refer to it later.  A problem with this though, as I realize just now, is that if I remove springs (as I do in the funeral ritual), I’ll lose the correct references.  So I have to make sure that when the springs are added again, when funerals are disabled, that a pass a new reference to the ArrayList.

I also found that there tends to be super-breeders every time I run the sketch, with certain people tending to produce tons of offspring while others produce none.  I’m talking like 1 or 2 people will produce 10 kids, which tends to make the sketch appear too tightly clustered because everyone is closely linked.  Perhaps this is a feature, rather than a bug, of reproduction?

Code

You can download the code from Github.  You’ll probably want to start up a node instance and then start the genetic crossings sketch, then finally the wolfram sketch.  Instructions are in the README.md.

Github: https://github.com/Xeus/Genetic-Crossing

Conclusion

And this leads me to some closing notes. I shied away from adding fitness yet again to my reproduction algorithms because I felt like “fitness” in the short-term was too much like large-scale evolution theory and autonomous agent simulation.  In my sketch there wasn’t really an ideal fitness state, with no limitations or rules imposed on the larger scale.  What I wanted was to break into modeling some culture into the simulation, so that choices were made between sexual partners based on cultural norms and not as much on randomized reproduction. Obviously modeling culture would work best if it were overlaid on top of basic biological reproductive theory such as choosing the fittest partner and whatnot, but I felt that was too much for the scope of this simulation, which I wanted mainly to focus on social networks.

JavaScript has come a long way.  It’s now the same on the backend and the frontend.  Processing can be exported to JavaScript in some capacity, and dataviz libraries such as D3 are taking off.  Soon we will be able to introduce more fluid, data, physics, and particle system simulations within a browser.  It’s too early for my sketch yet (ToxicLibs takes some finagling) but this is a glimpse of the web to come.

As I begin to do more serious work on the internal mechanics of Galapag.us, it’s stuff like this Processing project that makes me appreciate how careful I’ll have to be with positioning different factors against each other so that people can create their own formulae/evolutions to weight different priorities how they deem fit.

Looks like this book out of the Harvard Berkman Center, “Interop: The Promise and Perils of Highly Interconnected Systems”, by John Palfrey and Urs Gasser, is a must-read.

What I do feel is that current online social networks have not really tried to map out the complex interweaving, competing, variable connections and attractions we have between ourselves and others, between the different identities we all have, between the things we care about more or less at different times in our lives, etc.  To facilitate something like this, I can’t help but feel there needs to be a massive API that allows people to access all this data (if privacy settings allow it) so that we can take advantage of the multi-dimensional nature of our species.

You can think of someone’s identity as a meshed web that is being pulled apart by the external world and people and ideas and being pulled together by muscle and ligament and cartilage and sense of self and personality and such.  You can think of a community as a bunch of these springy people pulling on and apart from each other constantly, but at a stronger tension than from other communities.  Communities form religions and nations and cultures, again with that same network of relationships and competing identities. I hope that’s the dynamic I was able to capture in doing this project.

Credit

Special thanks to:

  • Prof. Dan Shiffman for all his documentation and code from Nature of Code, particularly his chapters on forces, genetic algorithms, ToxicLibs, and cellular automata
  • Prof. John Schimmel for his Processing-Nodejs code