jm + aws   266

House Democrats Sent A New Letter To Jeff Bezos About Amazon’s Facial Recognition Tool
The House Democrats’ questions focus largely on possible embedded bias in Amazon Rekognition, including how the tool’s accuracy breaks down by race, gender, ethnicity, and age. Also of particular concern is whether Amazon will build privacy protections into its facial recognition system and how it will ensure it is not abused for secret government surveillance. [....]

Meanwhile, Jeff Bezos has yet to address mounting criticism of Amazon’s Rekognition technology by Amazon employees, shareholders, and civil rights groups. In November, Amazon executives defended the company’s controversial facial recognition technology at an all-hands staff meeting after employees raised civil rights concerns about the tech’s potential misuse.

“It’s hard to trust that harm and abuse can be prevented if it is only post-mortem and through the Terms of Service,” an Amazon employee who requested anonymity told BuzzFeed News at the time.

rekognition  aws  privacy  data-protection  surveillance  amazon  us-politics  civil-rights 
18 days ago by jm
Introducing Predictive Scaling for Amazon EC2 in AWS Auto Scaling
Predictive Scaling predicts future traffic based on daily and weekly trends, including regularly-occurring spikes, and provisions the right number of EC2 instances in advance of anticipated changes. Provisioning the capacity just in time for an impending load change makes Auto Scaling faster than ever before. Predictive Scaling’s machine learning algorithms detect changes in daily and weekly patterns, automatically adjusting their forecasts. This removes the need for manual adjustment of Auto Scaling parameters over time, making Auto Scaling simpler to configure and consume. Auto Scaling enhanced with Predictive Scaling delivers faster, simpler, and more accurate capacity provisioning to our customers.


Fantastic! More heavy lifting taken care of.
aws  amazon  scaling  autoscaling  predictive-scaling  ml  ec2  asg 
27 days ago by jm
Productionproofing EKS
'We recently migrated SaleMove infrastructure from self-managed Kubernetes clusters running on AWS to using Amazon Elastic Container Service for Kubernetes (EKS). There were many surprises along the way to getting our EKS setup ready for production. This post covers some of these gotchas (others may already be fixed or are not likely to be relevant for a larger crowd) and is meant to be used as a reference when thinking of running EKS in production.'
eks  aws  docker  kubernetes  k8s  ops  prod 
6 weeks ago by jm
pusher/k8s-spot-rescheduler
'Tries to move K8s Pods from on-demand to spot instances':

K8s Spot rescheduler is a tool that tries to reduce load on a set of Kubernetes nodes. It was designed with the purpose of moving Pods scheduled on AWS on-demand instances to AWS spot instances to allow the on-demand instances to be safely scaled down (By the Cluster Autoscaler).

In reality the rescheduler can be used to remove load from any group of nodes onto a different group of nodes. They just need to be labelled appropriately.

For example, it could also be used to allow controller nodes to take up slack while new nodes are being scaled up, and then rescheduling those pods when the new capacity becomes available, thus reducing the load on the controllers once again.
k8s  kubernetes  aws  scaling  spot-instances  ops 
8 weeks ago by jm
Querying OpenStreetMap Buildings with AWS Athena – door2door Engineering
Well this is damn cool:
AWS not only made OpenStreetMap planet data available on S3, but it also made it query-able with Athena. Pretty cool, no? Now, in theory, you can just construct an SQL query, send it to Athena, and then do whatever you want with the results. No more:

Updating OSM planet data yourself; it gets updated on AWS whenever OSM publishes it, once a week.

Transforming the data into a query-able format; Athena handles that for you.

Query/request frequency limits (it’s still AWS though, so other limitations might apply 💸 💸)

At door2door, we had a pretty straight-forward use-case for this: we needed to get buildings in specific regions based only on where they were, and transform those buildings into GeoJSON that we can attach our data to, and visualize on the front-end on top of our base map.
athena  osm  buildings  aws  geodata  mapping  maps  door2door  cool  hacks 
9 weeks ago by jm
Running high-scale web applications on Amazon EC2 Spot Instances
AppNext's setup looks like quite good practice for a CPU-bound fleet
appnext  spot-instances  ec2  scalability  aws  ops  architecture 
10 weeks ago by jm
Amazon S3 Announces Selective Cross-Region Replication Based on Object Tags

Amazon S3 Cross-Region Replication (CRR) now supports object filtering based on S3 object tags. This allows you to identify individual objects using S3 object tags for automatic replication across AWS Regions for compliance and/or data protection.

CRR is an Amazon S3 feature that automatically replicates every object uploaded to an S3 bucket to a destination bucket in a different AWS Region that you choose. S3 object tags are key-value pairs applied to S3 objects that allow you to better organize, secure, and manage your data stored in S3. By using S3 object tags to determine which objects to replicate using CRR, you now have fine grained control to selectively replicate your storage to another AWS Region to backup critical data for compliance and disaster recovery.
aws  s3 
11 weeks ago by jm
randomised prefixes in S3 are no longer necessary to improve performance
This S3 request rate performance increase removes any previous guidance to randomize object prefixes to achieve faster performance. That means you can now use logical or sequential naming patterns in S3 object naming without any performance implications.


Having said that, it sounds like they may still help to a degree anyway.
s3  coding  architecture  aws  tips 
11 weeks ago by jm
ncw/rclone
"rsync for cloud storage" - Google Drive, Amazon Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Cloudfiles, Google Cloud Storage, Yandex Files
backup  github  sync  cloud  s3  storage  rsync  rclone  google  aws  dropbox  backblaze  yandex  onedrive 
august 2018 by jm
Humanitarian OpenStreetMap Team | Fast, Powerful, and Practical: New Technology for Aerial Imagery in Disaster Response
Aerial imagery can play an important role in disaster response operations, enabling response teams to identify and prioritize hardest-hit areas, conduct damage assessments, and plan response activities. Existing tools make this relatively easy in connected environments; users can browse high-resolution satellite imagery catalogs and download the relevant imagery, and can process drone imagery using online tools. Current solutions don’t work well in disconnected environments, however. Even offline tools lack the storage space and processing power to be effective for addressing large areas.

This blog post shows how rugged, portable Amazon Web Services (AWS) servers can be turned into a kit that’s mega-powerful, deployable, and purpose-built for post-disaster imagery operations. This can help humanitarians and government agencies to more accurately and efficiently conduct damage assessments and identify hardest-hit areas, potentially making a real difference in the aftermath of a natural disaster.
snowball  aws  humanitarian  emergency-response  osm  openstreetmap  mapping  aid  disasters 
july 2018 by jm
15 Key Takeaways from the Serverless Talk at AWS Startup Day
Best current practices for AWS Lambda usage. (still pretty messy/hacky/Rube-Goldberg-y from the looks of it tbh)
aws  lambda  serverless  ops  hacks  amazon 
july 2018 by jm
The problems with DynamoDB Auto Scaling and how it might be improved
'Based on these observations, we hypothesize that you can make two modifications to the system to improve its effectiveness:

trigger scaling up after 1 threshold breach instead of 5, which is in-line with the mantra of “scale up early, scale down slowly”;
trigger scaling activity based on actual request count instead of consumed capacity units, and calculate the new provisioned capacity units using actual request count as well.

As part of this experiment, we also prototyped these changes (by hijacking the CloudWatch alarms) to demonstrate their improvement.'
dynamodb  autoscaling  ops  scalability  aws  scaling  capacity 
july 2018 by jm
Scarr
S3 + Cloudfront + ACM + Route53, automated.
There are a bunch of free/cheap options for hosting static sites (just html/css/js) out there: github pages, netlify, firebase hosting - but when I want to build a bulletproof static site "for real", my go-to toolset is S3 for hosting with Cloudfront caching in front of it. I figured that after a few times doing this, I'd automate it. There are a few pre-existing tools for parts of this, but none I could find that did the whole thing from registration through uploading and Cloudfront invalidation.
cli  acm  aws  s3  cloudfront  route53  static-sites  web  html  hosting 
july 2018 by jm
Save on your AWS bill with Kubernetes Ingress
decent into to Kubernetes Ingress and the Ambassador microservices API gateway built on Envoy Proxy
envoy  proxying  kubernetes  aws  elb  load-balancing  ingress  ambassador  ops 
june 2018 by jm
AWS Region Table
what products are available where
amazon  aws  regions  azs  services  architecture  ops 
june 2018 by jm
schibsted/strongbox: A secret manager for AWS
Strongbox is a CLI/GUI and SDK to manage, store, and retrieve secrets (access tokens, encryption keys, private certificates, etc). Strongbox is a client-side convenience layer on top of AWS KMS, DynamoDB and IAM. It manages the AWS resources for you and configure them in a secure way. Strongbox has been used in production since mid-2016 and is now used extensively within Schibsted.
schibsted  strongbox  kms  aws  dynamodb  storage  secrets  credentials  passwords  ops 
may 2018 by jm
EC2 Instance Update – C5 Instances with Local NVMe Storage (C5d)
With a 25% to 50% improvement in price-performance over the C4 instances, the C5 instances are designed for applications like batch and log processing, distributed and or real-time analytics, high-performance computing (HPC), ad serving, highly scalable multiplayer gaming, and video encoding. Some of these applications can benefit from access to high-speed, ultra-low latency local storage. For example, video encoding, image manipulation, and other forms of media processing often necessitates large amounts of I/O to temporary storage. While the input and output files are valuable assets and are typically stored as Amazon Simple Storage Service (S3) objects, the intermediate files are expendable. Similarly, batch and log processing runs in a race-to-idle model, flushing volatile data to disk as fast as possible in order to make full use of compute resources.


Very nice!
ec2  instance-types  ops  storage  hardware  aws 
may 2018 by jm
Parallelizing S3 Workloads with s5cmd
nice parallel download/upload tool for S3, developed by Peak Games, open source, in Go
golang  go  s5cmd  open-source  tools  cli  s3  aws 
april 2018 by jm
Best Practices for DynamoDB
Amazon have updated this architectural tips doc for DynamoDB users -- good reference
dynamodb  nosql  aws  storage  databases  design  coding 
april 2018 by jm
Securing Docker Containers on AWS | nearForm
'On most projects at nearForm we are deploying our solutions within Docker containers. There are tasks that are repeated on each project to secure and harden off those deployments and we built this packer template to produce a quick and easy way for you to spin up an AWS AMI that passes the Docker-Bench-Security script. The Docker-Bench-Security repo is a work product of the above mentioned consolidation efforts by the Docker team.'
docker  aws  security  nearform  containers  linux  packer 
january 2018 by jm
auto53
'The missing link between AWS AutoScaling Groups and Route53 [...] solves the issue of keeping a route53 zone up to date with the changes that an autoscaling group might face.'
auto53  route-53  dns  aws  amazon  ops  hostnames  asg  autoscaling 
december 2017 by jm
AWS CodeBuild Plugin - Jenkins - Jenkins Wiki
Trigger AWS CodeBuild jobs as build steps for a Jenkins project. :thinking_face_emoji:
jenkins  hacks  aws  codebuild  build  coding  ci 
december 2017 by jm
Using AWS Batch to Generate Mapzen Terrain Tiles · Mapzen
Using this setup on AWS Batch, we are able to generate more than 3.75 million tiles per minute and render the entire world in less than a week! These pre-rendered tiles get stored in S3 and are ready to use by anyone through the AWS Public Dataset or through Mapzen’s Terrain Tiles API.
mapzen  mapping  tiles  batch  aws  s3  lambda  docker 
december 2017 by jm
AWS re:invent 2017: Container Networking Deep Dive with Amazon ECS (CON401) // Practical Applications
Another re:Invent highlight to watch -- ECS' new native container networking model explained
reinvent  aws  containers  docker  ecs  networking  sdn  ops 
december 2017 by jm
Introducing the Amazon Time Sync Service
Well overdue; includes Google-style leap smearing
time-sync  time  aws  services  ntp  ops 
november 2017 by jm
Introducing AWS Fargate – Run Containers without Managing Infrastructure
now that's a good announcement. Available right away running atop ECS; EKS in 2018
eks  ecs  fargate  aws  services  ops  containers  docker 
november 2017 by jm
'Lambda and serverless is one of the worst forms of proprietary lock-in we've ever seen in the history of humanity' • The Register
That doesn't mean Polvi is a fan. "Lambda and serverless is one of the worst forms of proprietary lock-in that we've ever seen in the history of humanity," said Polvi, only partly in jest, referring to the most widely used serverless offering, AWS Lambda. "It's seriously as bad as it gets."

He elaborated: "It's code that tied not just to hardware – which we've seen before – but to a data center, you can't even get the hardware yourself. And that hardware is now custom fabbed for the cloud providers with dark fiber that runs all around the world, just for them. So literally the application you write will never get the performance or responsiveness or the ability to be ported somewhere else without having the deployment footprint of Amazon."


Absolutely agreed...
lambda  amazon  aws  containers  coreos  deployment  lockin  proprietary  serverless  alex-polvi  kubernetes 
november 2017 by jm
AWS Service Terms
57.10 Acceptable Use; Safety-Critical Systems. Your use of the Lumberyard Materials must comply with the AWS Acceptable Use Policy. The Lumberyard Materials are not intended for use with life-critical or safety-critical systems, such as use in operation of medical equipment, automated transportation systems, autonomous vehicles, aircraft or air traffic control, nuclear facilities, manned spacecraft, or military use in connection with live combat. However, this restriction will not apply in the event of the occurrence (certified by the United States Centers for Disease Control or successor body) of a widespread viral infection transmitted via bites or contact with bodily fluids that causes human corpses to reanimate and seek to consume living human flesh, blood, brain or nerve tissue and is likely to result in the fall of organized civilization.


Seems fair enough.
aws  zombies  funny  t-and-cs  legal  civilization  just-in-case 
november 2017 by jm
aws-vault
'A vault for securely storing and accessing AWS credentials in development environments'.

Scott Piper says: 'You should not use the AWS CLI with MFA without aws-vault, and probably should not use the CLI at all without aws-vault, because of it's benefit of storing your keys outside of ~/.aws/credentials (since every once in a while a developer will decide to upload all their dot-files in their home directory to github so they can use the same .vimrc and .bashrc aliases everywhere, and will end up uploading their AWS creds).'
aws  vault  security  cli  development  coding  dotfiles  credentials  mfa 
november 2017 by jm
srcecde/aws-lambda-cheatsheet
'AWS Lambda cheatsheet' -- a quick ref card for Lambda users
aws  lambda  ops  serverless  reference  quick-references 
october 2017 by jm
How to operate reliable AWS Lambda applications in production
running a reliable Lambda application in production requires you to still follow operational best practices. In this article I am including some recommendations, based on my experience with operations in general as well as working with AWS Lambda.
aws  cloud  lambda  ops  amazon 
october 2017 by jm
HN thread on the new Network Load Balancer AWS product
looks like @colmmacc works on it. Lots and lots of good details here
nlb  aws  load-balancing  ops  architecture  lbs  tcp  ip 
september 2017 by jm
Going Multi-Cloud with AWS and GCP: Lessons Learned at Scale
Metamarkets splits across AWS and GCP, going into heavy detail here
aws  gcp  google  ops  hosting  multi-cloud 
august 2017 by jm
cristim/autospotting
'Easy to use tool that automatically replaces some or even all on-demand AutoScaling group members with similar or larger identically configured spot instances in order to generate significant cost savings on AWS EC2, behaving much like an AutoScaling-backed spot fleet.'
asg  autoscaling  ec2  aws  spot-fleet  spot-instances  cost-saving  scaling 
august 2017 by jm
Working with multiple AWS accounts at Ticketea
AWS STS/multiple account best practice described
sts  aws  authz  ops  ticketea  dev 
august 2017 by jm
GitHub - jorgebastida/awslogs: AWS CloudWatch logs for Humans™
This feature alone is a bit of a killer app:
$ awslogs get /var/log/syslog ip-10-1.* --start='2h ago' | grep ERROR


Nice.
cli  logging  aws  cloudwatch  logs  awslogs  ec2 
august 2017 by jm
AWS Lambda Deployment using Terraform – Build ACL – Medium
Fairly persuasive that production usage of Lambda is much easier if you go full Terraform to manage and deploy.
A complete picture of what it takes to deploy your Lambda function to production with the same diligence you apply to any other codebase using Terraform. [...] There are many cases where frameworks such as SAM or Serverless are not enough. You need more than that for a highly integrated Lambda function. In such cases, it’s easier to simply use Terraform.
infrastructure  aws  lambda  serverless  ops  terraform  sam 
august 2017 by jm
BinaryAlert: Serverless, Real-time & Retroactive Malware Detection
This is a serverless stack built on AWS, deployed with Terraform. Not sure what to think about this -- it still makes me shudder a little
aws  serverless  lambda  airbnb  malware  yara  binaryalert  architecture 
august 2017 by jm
EBS gp2 I/O BurstBalance exhaustion
when EBS volumes in EC2 exhaust their "burst" allocation, things go awry very quickly
performance  aws  ebs  ec2  burst-balance  ops  debugging 
july 2017 by jm
awslabs/aws-ec2rescue-linux
Amazon Web Services Elastic Compute Cloud (EC2) Rescue for Linux is a python-based tool that allows for the automatic diagnosis of common problems found on EC2 Linux instances.


Most of the modules appear to be log-greppers looking for common kernel issues.
ec2  aws  kernel  linux  ec2rl  ops 
july 2017 by jm
Fastest syncing of S3 buckets
good tip for "aws s3 sync" performance
performance  aws  s3  copy  ops  tips 
july 2017 by jm
Scheduled Tasks (cron) - Amazon EC2 Container Service
ECS now does cron jobs. But where does AWS Batch fit in? confusing
aws  batch  ecs  cron  scheduling  recurrence  ops 
july 2017 by jm
SQS performance and latency
Some decent benchmark data on SQS:
We were looking at four values in the tests:
total number of messages sent per second (by all nodes)
total number of messages received per second
95th percentile of message send latency (how fast a message send call completes)
95th percentile of message processing latency (how long it takes between sending and receiving a message)
sqs  benchmarking  measurement  aws  latency 
july 2017 by jm
Use AWS WAF to Mitigate OWASP’s Top 10 Web Application Vulnerabilities
'describes how you can use AWS WAF, a web application firewall, to address the top application security flaws as named by the Open Web Application Security Project (OWASP). Using AWS WAF, you can write rules to match patterns of exploitation attempts in HTTP requests and block requests from reaching your web servers. This whitepaper discusses manifestations of these security vulnerabilities, AWS WAF–based mitigation strategies, and other AWS services or solutions that can help address these threats.'
security  waf  aws  http  owasp  filtering 
july 2017 by jm
Top 5 ways to improve your AWS EC2 performance
A couple of bits of excellent advice from Datadog (although this may be a slightly old post, from Oct 2016):

1. Unpredictable EBS disk I/O performance. Note that gp2 volumes do not appear to need as much warmup or priming as before.

2. EC2 Instance ECU Mismatch and Stolen CPU. advice: use bigger instances

The other 3 ways are a little obvious by comparison, but worth bookmarking for those two anyway.
ops  ec2  performance  datadog  aws  ebs  stolen-cpu  virtualization  metrics  tips 
july 2017 by jm
How Did I “Hack” AWS Lambda to Run Docker Containers?
Running Docker containers in Lambda using a usermode-docker hack -- hacky as hell but fun ;) Lambda should really support native Docker though
docker  lambda  aws  serverless  ops  hacks  udocker 
june 2017 by jm
lambci/docker-lambda
A sandboxed local environment that replicates the live AWS Lambda environment almost identically – including installed software and libraries, file structure and permissions, environment variables, context objects and behaviors – even the user and running process are the same.


(via og-aws)
docker  lambda  images  testing  aws  serverless 
june 2017 by jm
Open Guide to Amazon Web Services
'A lot of information on AWS is already written. Most people learn AWS by reading a blog or a “getting started guide” and referring to the standard AWS references. Nonetheless, trustworthy and practical information and recommendations aren’t easy to come by. AWS’s own documentation is a great but sprawling resource few have time to read fully, and it doesn’t include anything but official facts, so omits experiences of engineers. The information in blogs or Stack Overflow is also not consistently up to date. This guide is by and for engineers who use AWS. It aims to be a useful, living reference that consolidates links, tips, gotchas, and best practices. It arose from discussion and editing over beers by several engineers who have used AWS extensively.'
amazon  aws  guides  documentation  ops  architecture 
june 2017 by jm
Scaling Amazon Aurora at ticketea
Ticketing is a business in which extreme traffic spikes are the norm, rather than the exception. For Ticketea, this means that our traffic can increase by a factor of 60x in a matter of seconds. This usually happens when big events (which have a fixed, pre-announced 'sale start time') go on sale.
scaling  scalability  ops  aws  aurora  autoscaling  asg 
may 2017 by jm
Spotting a million dollars in your AWS account · Segment Blog
You can easily split your spend by AWS service per month and call it a day. Ten thousand dollars of EC2, one thousand to S3, five hundred dollars to network traffic, etc. But what’s still missing is a synthesis of which products and engineering teams are dominating your costs. 

Then, add in the fact that you may have hundreds of instances and millions of containers that come and go. Soon, what started as simple analysis problem has quickly become unimaginably complex. 

In this follow-up post, we’d like to share details on the toolkit we used. Our hope is to offer up a few ideas to help you analyze your AWS spend, no matter whether you’re running only a handful of instances, or tens of thousands.

segment  money  costs  billing  aws  ec2  ecs  ops 
may 2017 by jm
jantman/awslimitchecker

A script and python module to check your AWS service limits and usage, and warn when usage approaches limits.

Users building out scalable services in Amazon AWS often run into AWS' service limits - often at the least convenient time (i.e. mid-deploy or when autoscaling fails). Amazon's Trusted Advisor can help this, but even the version that comes with Business and Enterprise support only monitors a small subset of AWS limits and only alerts weekly. awslimitchecker provides a command line script and reusable package that queries your current usage of AWS resources and compares it to limits (hard-coded AWS defaults that you can override, API-based limits where available, or data from Trusted Advisor where available), notifying you when you are approaching or at your limits.


(via This Week in AWS)
aws  amazon  limits  scripts  ops 
may 2017 by jm
_Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases_
'Amazon Aurora is a relational database service for OLTP workloads offered as part of Amazon Web Services (AWS). In this paper, we describe the architecture of Aurora and the design considerations leading to that architecture. We believe the central constraint in high throughput data processing has moved from compute and storage to the network. Aurora brings a novel architecture to the relational database to address this constraint, most notably by pushing redo processing to a multi-tenant scale-out storage service, purpose-built for Aurora. We describe how doing so not only reduces network traffic, but also allows for fast crash recovery, failovers to replicas without loss of data, and fault-tolerant, self-healing storage. We then describe how Aurora achieves consensus on durable state across numerous storage nodes using an efficient asynchronous scheme, avoiding expensive and chatty recovery protocols. Finally, having operated Aurora as a production service for over 18 months, we share the lessons we have learnt from our customers on what modern cloud applications expect from databases.'
via:rbranson  aurora  aws  amazon  databases  storage  papers  architecture 
may 2017 by jm
Backdooring an AWS account
eek. Things to look out for on your AWS setup:
So you’ve pwned an AWS account — congratulations — now what? You’re eager to get to the data theft, amirite? Not so fast whipper snapper, have you disrupted logging? Do you know what you have? Sweet! Time to get settled in. Maintaining persistence in AWS is only limited by your imagination but there are few obvious and oft used techniques everyone should know and watch for.
aws  security  hacks  iam  sts 
may 2017 by jm
cristim/autospotting: Pay up to 10 times less on EC2 by automatically replacing on-demand AutoScaling group members with similar or larger identically configured spot instances.
A simple and easy to use tool designed to significantly lower your Amazon AWS costs by automating the use of the spot market.

Once enabled on an existing on-demand AutoScaling group, it launches an EC2 spot instance that is cheaper, at least as large and configured identically to your current on-demand instances. As soon as the new instance is ready, it is added to the group and an on-demand instance is detached from the group and terminated.

It continuously applies this process, gradually replacing any on-demand instances with spot instances until the group only consists of spot instances, but it can also be configured to keep some on-demand instances running.
aws  golang  ec2  autoscaling  asg  spot-instances  ops 
may 2017 by jm
acksin/seespot: AWS Spot instance health check with termination and clean up support
When a Spot Instance is about to terminate there is a 2 minute window before the termination actually happens. SeeSpot is a utility for AWS Spot instances that handles the health check. If used with an AWS ELB it also handles cleanup of the instance when a Spot Termination notice is sent.
aws  elb  spot-instances  health-checks  golang  lifecycle  ops 
may 2017 by jm
AWS Greengrass
AWS Greengrass is software that lets you run local compute, messaging & data caching for connected devices in a secure way. With AWS Greengrass, connected devices can run AWS Lambda functions, keep device data in sync, and communicate with other devices securely – even when not connected to the Internet. Using AWS Lambda, Greengrass ensures your IoT devices can respond quickly to local events, operate with intermittent connections, and minimize the cost of transmitting IoT data to the cloud.

AWS Greengrass seamlessly extends AWS to devices so they can act locally on the data they generate, while still using the cloud for management, analytics, and durable storage. With Greengrass, you can use familiar languages and programming models to create and test your device software in the cloud, and then deploy it to your devices. AWS Greengrass can be programmed to filter device data and only transmit necessary information back to the cloud. AWS Greengrass authenticates and encrypts device data at all points of connection using AWS IoT’s security and access management capabilities. This way data is never exchanged between devices when they communicate with each other and the cloud without proven identity.
aws  cloud  iot  lambda  devices  offline  synchronization  architecture 
april 2017 by jm
Amazon DynamoDB Accelerator (DAX)
Amazon DynamoDB Accelerator (DAX) is a fully managed, highly available, in-memory cache for DynamoDB that delivers up to a 10x performance improvement – from milliseconds to microseconds – even at millions of requests per second. DAX does all the heavy lifting required to add in-memory acceleration to your DynamoDB tables, without requiring developers to manage cache invalidation, data population, or cluster management.


No latency percentile figures, unfortunately. Also still in preview.
amazon  dynamodb  aws  dax  performance  storage  databases  latency  low-latency 
april 2017 by jm
Ubuntu on AWS gets serious performance boost with AWS-tuned kernel
interesting -- faster boots, CPU throttling resolved on t2.micros, other nice stuff
aws  ubuntu  ec2  kernel  linux  ops 
april 2017 by jm
Lessons Learned in Lambda
In case you were thinking Lambda was potentially usable yet
lambda  aws  shitshow  architecture  serverless 
april 2017 by jm
Deep Dive on Amazon EBS Elastic Volumes
'March 2017 AWS Online Tech Talks' -- lots about the new volume types
aws  ebs  storage  architecture  ops  slides 
march 2017 by jm
atlassian/localstack: A fully functional local AWS cloud stack. Develop and test your cloud apps offline!
LocalStack provides an easy-to-use test/mocking framework for developing Cloud applications. Currently, the focus is primarily on supporting the AWS cloud stack.

LocalStack spins up the following core Cloud APIs on your local machine:

API Gateway at http://localhost:4567;
Kinesis at http://localhost:4568;
DynamoDB at http://localhost:4569;
DynamoDB Streams at http://localhost:4570;
Elasticsearch at http://localhost:4571;
S3 at http://localhost:4572;
Firehose at http://localhost:4573;
Lambda at http://localhost:4574;
SNS at http://localhost:4575;
SQS at http://localhost:4576

Additionally, LocalStack provides a powerful set of tools to interact with the cloud services, including a fully featured KCL Kinesis client with Python binding, simple setup/teardown integration for nosetests, as well as an Environment abstraction that allows to easily switch between local and remote Cloud execution.
aws  emulation  mocking  services  testing  dynamodb  s3 
march 2017 by jm
Segment.com on cost savings using DynamoDB, autoscaling and ECS
great post.

1. DynamoDB hot shards were a big problem -- and it is terrible that diagnosing this requires a ticket to AWS support! This heat map should be a built-in feature.

2. ECS auto-scaling gets a solid thumbs-up.

3. Switching from ELB to ALB lets them set ports dynamically for individual ECS Docker containers, and then pack as many containers as will fit on a giant EC2 instance.

4. Terraform modules to automate setup and maintainance of ECS, autoscaling groups, and ALBs
terraform  segment  architecture  aws  dynamodb  alb  elb  asg  ecs  docker 
march 2017 by jm
The Occasional Chaos of AWS Lambda Runtime Performance
If our code has modest resource requirements, and can tolerate large changes in performance, then it makes sense to start with the least amount of memory necessary. On the other hand, if consistency is important, the best way to achieve that is by cranking the memory setting all the way up to 1536MB.
It’s also worth noting here that CPU-bound Lambdas may be cheaper to run over time with a higher memory setting, as Jim Conning describes in his article, “AWS Lambda: Faster is Cheaper”. In our tests, we haven’t seen conclusive evidence of that behavior, but much more data is required to draw any strong conclusions.
The other lesson learned is that Lambda benchmarks should be gathered over the course of days, not hours or minutes, in order to provide actionable information. Otherwise, it’s possible to see very impressive performance from a Lambda that might later dramatically change for the worse, and any decisions made based on that information will be rendered useless.
aws  lambda  amazon  performance  architecture  ops  benchmarks 
march 2017 by jm
« earlier      
per page:    204080120160

related tags

10/8  aas  accounts  acm  acm-queue  adrian-cockcroft  adrian-cockroft  adroll  advent  advice  aid  airbnb  alarming  alb  alerting  alestic  alex-polvi  alibaba  aliyun  allocation  ama  amazon  ambassador  ami  analytics  andrew-spyker  animation  anti-fraud  ap  aphyr  api  api-gateway  apis  appnext  architecture  asg  asgard  athena  atlas  auc  aurora  authentication  authorization  authz  auto-scaling  auto53  autoscaling  availability  aws  aws-cli  awscli  awslogs  awsume  az  azs  azul  azure  b2b  b2c  backblaze  backup  backups  batch  batching  beanstalk  benchmarking  benchmarks  billing  binaryalert  blake2  blast-radius  blogs  blue-green-deployments  blue-green-deploys  boto  bots  bugs  build  buildings  burst  burst-balance  c5  cacerts  campaigns  cap  capacity  cassandra  cd  cdn  cep  certificates  certs  chaos-kong  chaos-monkey  chatops  chef  china  chris-newcombe  ci  circuit-breakers  civil-rights  civilization  cli  clients  clock  cloud  cloud-connect  cloudflare  cloudformation  cloudfront  cloudnative  cloudsearch  cloudsmith  cloudsploit  cloudwatch  cluster  clustering  clusters  cms  code-spaces  codebuild  codedeploy  coding  cold-start  colmmacc  comet  command-line  comparison  conferences  consistency  constant-load  containers  continuous-delivery  cool  copy  coreos  corporate-culture  corruption  cost-saving  costs  counters  coursera  cp  cpu  credentials  credstash  cron  cross-region  crypto  culture  curl  danilop  data  data-pipelines  data-protection  data-science  data-structures  databases  datadog  dataman  datawire  dax  daylight-savings  ddos  debugging  dedupe  delete  delivery  delta  demo  deploy  deployment  design  dev  development  devices  devops  dht  disasters  disk  disks  distcomp  distributed-cron  distsys  dns  docker  documentation  documents  door2door  dos  dotfiles  dropbox  dst  duplicity  duply  dynalite  dynamodb  dynect  ebs  ec2  ec2rl  ecommerce  ecs  eks  elasticache  elb  email  embedded  emergency-response  emr  emrfs  emulation  environments  envoy  eric-brandwine  eric-hammond  etcd  etl  eu  eu-central-1  eureka  event-processing  event-streaming  events  eventual-consistency  examples  expiry  export  extortion  fail  failover  failures  fargate  fault-domains  fault-tolerance  ffmpeg  fifo  figures  filesystems  filtering  firewalls  five-eyes  flow-logs  fluentd  formal-methods  fpga  fraud  freebsd  funny  fuse  ga  games  gaming  gce  gchq  gcp  gdpr  geodata  germany  gifs  gilt  git  github  go  golang  google  google-storage  gossip  graphics  grey-failures  guides  gulp  h264  ha  hacks  hadoop  hailo  haproxy  hardware  health-checks  history  hls  hosting  hostnames  html  http  https  hugo  humanitarian  hvm  hystrix  iam  images  incident-response  infrastructure  ingress  instance-types  instances  instapaper  integration-testing  inter-region  internet  io  iops  iostat  iot  ip  ip-addresses  ipc  ireland  ironfan  james-hamilton  java  javascript  jenkins  jepsen  jmespath  jobs  json  just-in-case  k8s  kafka  kappa  kcl  kernel  key-length  key-management  key-rotation  keys  kinesis  kms  knife  kpl  kubernetes  kvm  lambda  languages  latency  law  lbs  leap-seconds  leap-smearing  legacy  legal  leveldb  libraries  lifecycle  limits  linux  load  load-balancing  load-testing  loaders  loading  lockin  logging  logs  low-latency  lucene  m5  machine-learning  malware  management  mapping  maps  mapzen  marc-brooker  measurement  memcached  memory  mesos  messaging  metrics  mfa  microservices  microsoft  mit  mitch-garnaat  ml  mocha  mocking  mocks  model-checking  money  mongodb  monitoring  mp4  multi-az  multi-cloud  multi-region  mysql  mytaxi  nat  nearform  netflix  network  network-partitions  networking  nginx  nlb  node  node.js  nosql  notifications  nsa  ntp  ntpd  obama  object-model  offline  omgwtfbbq  onedrive  onlive  oo  open-source  openstack  openstreetmap  ops  opsgenie  optimization  osm  osx  outages  overload  owasp  packaging  packer  packet-capture  packets  papers  partitions  passwords  pbailis  pdf  percentiles  perf  perfect-forward-secrecy  performance  periodic-tasks  ping  pinterest  piops  pipeline  pipelines  pki  plugins  pluscal  post-mortem  post-mortems  postmortem  prediction  predictive-scaling  presentations  presto  pricing  privacy  prod  production  programming  proprietary  proving  provisioning  proxying  puppet  pv  pylons  python  qcon  qos  qubole  querying  queueing  queues  quick-references  quora  quorum  r3  rabbitmq  raid  rclone  rdbms  rds  read-after-writes  recovery  recurrence  reddit  redis  redshift  reference  regions  registry  reinvent  rekognition  reliability  reliabilty  replication  resilience  resource-limits  ribbon  rightscale  risks  roles  round-trip  route-53  route53  rrdns  rsa  rsync  ruby  runjop  s3  s3fs  s3funnel  s3ql  s5cmd  sam  sandboxing  scala  scalability  scale  scaling  scalr  schedulers  scheduling  sched_batch  schibsted  scraping  scripts  scryer  sdk  sdn  search  secrets  security  segment  serverless  servers  service-discovery  service-registry  services  ses  sha1  sharding  shitshow  sift-science  signalfx  simulation  skyliner  slack  slas  slides  smartstack  smtp  smugmug  snapshots  snooping  snowball  sns  sockets  solr  spark  speculative-execution  spikes  spinnaker  spot  spot-fleet  spot-instances  sql  sqs  ssl  stack-hammer  stackdriver  stacks  startups  static-sites  steam  stolen-cpu  storage  streaming  streams  strongbox  sts  surveillance  survey  swf  sync  synchronization  sysadmin  system-tests  systemtap  t-and-cs  tail  talks  tcp  tcpdump  tellybug  terraform  testing  tests  thrift  ticketea  tiles  time  time-sync  timezones  tips  tla  tla+  tlc  tls  tools  tornado  tracing  transactions  ttl  tunables  tuning  tutorials  two-factor-authentication  ubuntu  udocker  ultradns  unit-tests  unix  uploads  us-east  us-politics  user-submitted-code  utc  vault  vegas-rule  version-control  versioning  via:brianscanlan  via:chorn  via:highscalability  via:jgilbert  via:marc  via:marc-brooker  via:matt-sergeant  via:nelson  via:pdolan  via:rbranson  video  videos  virtualization  vlans  vpc  waf  web  web-services  website-generators  white-papers  whitepapers  windows  wishlist  workarounds  worker-pools  workflows  xen  xpath  yandex  yara  yas3fs  yelp  yeobot  zing  zombies  zonify  zookeeper 

Copy this bookmark:



description:


tags: