jm + athena   3

Querying OpenStreetMap Buildings with AWS Athena – door2door Engineering
Well this is damn cool:
AWS not only made OpenStreetMap planet data available on S3, but it also made it query-able with Athena. Pretty cool, no? Now, in theory, you can just construct an SQL query, send it to Athena, and then do whatever you want with the results. No more:

Updating OSM planet data yourself; it gets updated on AWS whenever OSM publishes it, once a week.

Transforming the data into a query-able format; Athena handles that for you.

Query/request frequency limits (it’s still AWS though, so other limitations might apply 💸 💸)

At door2door, we had a pretty straight-forward use-case for this: we needed to get buildings in specific regions based only on where they were, and transform those buildings into GeoJSON that we can attach our data to, and visualize on the front-end on top of our base map.
athena  osm  buildings  aws  geodata  mapping  maps  door2door  cool  hacks 
4 weeks ago by jm
Mythology about security…
A valuable history lesson from Jim Gettys:
Government export controls crippled Internet security and the design of Internet protocols from the very beginning: we continue to pay the price to this day.  Getting security right is really, really hard, and current efforts towards “back doors”, or other access is misguided. We haven’t even recovered from the previous rounds of government regulations, which has caused excessive complexity in an already difficult problem and many serious security problems. Let us not repeat this mistake…

I remember the complexity of navigating crypto export controls. As noted here, it was generally easier just not to incorporate security features.
security  crypto  export-control  jim-gettys  x11  history  x-windows  mit  athena  kerberos 
april 2018 by jm
S3 Inventory Adds Apache ORC output format and Amazon Athena Integration
Interesting to see Amazon are kind of putting their money behind ORC as a new public data interchange format with this.

Update: the Amazon senior PM for Athena and EMR says: 'Actually, we like both ORC and Parquet. Athena can process both ORC and Parquet, and teams can choose if they want to use either.' --
orc  formats  data  interchange  s3  athena  output 
november 2017 by jm

Copy this bookmark: