Special secret stuff with s3cmd

According to my coworker, "Amazon's S3 is the best thing since sliced bread." For working with S3, s3cmd is a handy little tool. It's documentation is a bit on the sparse side, but, what do you expect for free?

One gotcha with S3 is that buckets and the files in them have entirely distinct ACLs. This can lead to scenarios where the owner of a bucket can't work with the files in it. An easy way for this to come about is to log into the S3 console and create a bucket with one set of credentials, then upload files with a tool like s3cmd under another set of credentials.

cd /path/to/files/
s3cmd -v sync . s3://damnbucket.bucketowners.org

You can give permissions to the bucket owner like so:

s3cmd setacl --acl-grant=full_control:i_own_the_damn_bucket@bucketowners.org --recursive s3://damnbucket.bucketowners.org

You might also want to make the files public, so they can be served as a static website.

s3cmd setacl --acl-public --recursive s3://damnbucket.bucketowners.org

AWS Command Line Interface

I've been using s3cmd for a while, out of habit, but maybe it's time to try to Amazon's AWS Command Line Interface, which just had their version 1.0 release.

From a brief look, AWS CLI looks nice. You can do the same sync operation as above and make files public in one command:

aws s3 sync . s3://damnbucket.bucketowners.org --acl public-read

Amazon is very mysterious about how to specify the target of a grant of permissions, aka the grantee. I tried to give permission to the owner of a bucket, but kept getting an error. Some more examples in the docs would help! I also get "Invalid Id" for no apparent reason in the permissions section of the web UI for S3, so maybe I'm just clueless.

aws s3api put-object-acl --bucket damnbucket.bucketowners.org --grant-full-control i_own_the_damn_bucket@bucketowners.org --key genindex.html
#> A client error (InvalidArgument) occurred: Argument format not recognized.

As far as I could tell, the AWS CLI tool seems to be missing the --recursive option that we used with s3cmd. That seems like a fairly essential thing.

Also, I couldn't get the profile feature to work:

aws s3 ls s3://damnbucket.bucketowners.org --profile docgenerator
#>The config profile (docgenerator) could not be found

NOTE: Many thanks to Mitch Garnaat, I now know how the --profile switch works. Contrary to the documentation, you need a heading in your config file like this: [profile docgenerator] rather than like this: [docgenerator].

I'm glad Amazon is taking the lead in developing this tool, and I'm sure they'll keep making it better. And, there's a Github repo, so I'm guessing that means they take pull requests.

Who will prosper in the age of smart machines?

What if Siri really worked? ...worked so well that those that mastered co-operation with a digital assistant had a serious competitive advantage over those relying on their own cognitive powers alone?

That question is considered by economist Tyler Cowen in a new book called Average is Over. Previously, Cowen wrote The Great Stagnation and maintains the Marginal Revolution blog and teaches online classes at Marginal Revolution University.

"Increasingly, machines are providing not only the brawn but the brains, too, and that raises the question of where humans fit into this picture — who will prosper and who won’t in this new kind of machine economy?"

Tyler Cowen's answers

Who will gain: People who can collaborate with smart machines; life long learners; people with marketing skills; motivators.

"Sheer technical skill can be done by the machines, but integrating the tech side with an attention-grabbing innovation is a lot harder."

The psychological aspect is interesting. The traditional techy nerd (ahem... hello, self) has a psychology adapted to machines. But, machines are gaining the capacity to interface on a level adapted human psychology.

Who will lose: People who compete with smart machines; people who are squeamish about being tracked, evaluated and rated; the sick; bohemians; political radicals.

On being quantified

"Computing and software will make it easier to measure performance and productivity. [...] In essence everyone will suffer the fate of professional chess players, who always know when they have lost a game, have an exact numerical rating for their overall performance, and find excuses for failure hard to come by."

On hipsters

"These urban areas [he doesn't mention Portland by name] are full of people who are bright, culturally literate, Internet-savvy and far from committed to the idea of hard work directed toward earning a good middle-class living. We’ll need a new name for the group of people who have the incomes of the lower middle class and the cultural habits of the wealthy or upper middle class. They will spread a libertarian worldview that working for other people full time is an abominable way to get by."

How many will prosper

The current trend of unequal wealth distribution will only continue as technological literacy takes on new dimensions. "Big data" makes it easier to measure and grade our skills and failings. Apex skills are the ability to grab human attention, to motivate, to manage humans and machines in collaboration.

Another, not quite contrasting view comes from Northwestern University economist Robert Gordon. His paper "Is U.S. Economic Growth Over? Faltering Innovation Confronts the Six Headwinds", suggests that "the rapid progress made over the past 250 years could well turn out to be a unique episode in human history." In particular, he raises the possibility that the internet and digital and mobile technology may contribute less to productivity than previous industrial revolutions.

Technology can create winner-take-all situations, where a few capture enormous gains leaving the also-rans with little. A lot depends on the distribution of technology's benefits.

More

• Who will prosper in the new world? in the New York Times Opinionator section
• If you don't like to read, here's the video version: Smart Machines: The Next Big Thing For Smart Human Beings
• Average is Over, Tyler Cowen's latest book
• Thomas Friedman on Average is Over and Average is over, part 2.
• Of course, I was totally hip years ago to the potential for algorithms to know their users, when I asked: Does Google know if you're honest?
• The prospects of integration between digital tools and human psychology appears in Christopher Steiner's Automate This.
• Hacking education
• The online education tidal wave