The million dollar engineering problem


94 bookmarks. First posted by chris.heisel 6 weeks ago.


# ALB > ELB
Application load balancing is better than elastic load balancing because it allows you to fill up an instance with as many containers as it can handle. The reason elastic load balancing cannot do this is because it requires that each instance has a static port dedicated to each service, meaning that only one container for a particular service can reside on a given instance.

# Large instances > small instances
Filling up large instances is more cost efficient than having many small instances, partly for the reason above but also because some things such as monitoring are charged on a per instance basis.

# Build in cost efficiency early
Doing a one off, cost cutting exercise will only have a temporary effect. Build it in to your process so that it never happens in the first place.
ifttt  pocket  alb  aws  container  cost-cutting  ecs  efficiency  elb  terraform 
18 days ago by d-metcalfe
The million dollar engineering problem by and why they use
Terraform  from twitter_favs
5 weeks ago by kennygatdula
hidden costs scaling amazon services
AWS  lv 
5 weeks ago by diasyrmus
The Terrible Secret of AWS: sub-linear scaling, aka economies of scale, is what powers growth ever since the Industrial Revolution. But the delightful linear scaling of AWS pricing denies YOU precisely that, and the wedge between the two is why Jeff Bezos is rich.
aws  architecture  cloud  scalability  ovum 
5 weeks ago by yorksranter
RT : How do you cut a million dollars from your annual bill?
AWS  from twitter
5 weeks ago by gatestone
Segment is the analytics API you've always wanted. It's the easiest way to install all of your favorite analytics tools at once!
architecture  scalability 
5 weeks ago by derekharmel
Smart techniques for reducing AWS Dynamo and ECS bill
architecture  cloud  cost  theskinny 
5 weeks ago by coldclimate
How do you cut a million dollars from your annual bill?
from twitter
5 weeks ago by peba
Cut AWS costs
aws  business  cloud 
5 weeks ago by sdellis
The Segment Stack
aws  costs  architecture  business  mezu 
5 weeks ago by melo
great post.

1. DynamoDB hot shards were a big problem -- and it is terrible that diagnosing this requires a ticket to AWS support! This heat map should be a built-in feature.

2. ECS auto-scaling gets a solid thumbs-up.

3. Switching from ELB to ALB lets them set ports dynamically for individual ECS Docker containers, and then pack as many containers as will fit on a giant EC2 instance.

4. Terraform modules to automate setup and maintainance of ECS, autoscaling groups, and ALBs
terraform  segment  architecture  aws  dynamodb  alb  elb  asg  ecs  docker 
5 weeks ago by jm
The implication here is that you aren’t paying for total throughput, but rather partition count. And if you happen to have a few keys which saturate the same individual partitions, you have to double capacity to split a single hot partition onto their own partitions rather than scale the capacity linearly. And even there you are limited to the throughput for a single partition.
architecture  business  aws 
5 weeks ago by oisin
RT : "there's (still) no magic bullet": a deep dive into gory bits of AWS cost and resource optimization by :
from twitter
5 weeks ago by bob
RT : "there's (still) no magic bullet": a deep dive into gory bits of AWS cost and resource optimization by :
from twitter_favs
5 weeks ago by roens
“make the easy way, the right way”
from twitter
5 weeks ago by WibbleFarmer
RT : "there's (still) no magic bullet": a deep dive into gory bits of AWS cost and resource optimization by :
from twitter
5 weeks ago by pronoiac
# [The million dollar engineering problem](https://segment.com/blog/the-million-dollar-eng-problem/)

> it’s easy to fall into a cycle where the first response to any problem is to spend more money



> we managed to cut our AWS bill by over one million dollars annually



> two thirds of our cost of goods sold (COGS) was the bill from AWS



> hundreds of large EBS drives, over-provisioned cache and RDS instances…Relics left over from incidents of increased load that had not been sized back down



> While we can fix bugs in our own code, we can’t control our customers.



> Because we only have 60% of peak traffic at nights and on weekends, we’ve been able to save substantially by adding auto-scaling, and not have to worry about sudden traffic spikes.



> The general argument is that you can get less steal from noisy neighbors if you are the only one on a physical machine. And there’s more likelihood that you will be the sole VM on a physical machine if you are running at the largest possible instance size.



> From our best estimation, ELBs are essentially built atop an army of small, auto-scaling instances running HAProxy.



> The ALB (application load balancer) allows ECS to set ports dynamically for individual containers, and then pack as many containers as can fit onto a given instance.



> Aurora automatically scales disk capacity in 10GB increments

sysadmin  aws  budget  pricing 
5 weeks ago by jacklittleton
Favorite tweet: assaf

The million dollar engineering problem https://t.co/k2zjaiF4C9

— Assaf (@assaf) March 19, 2017

http://twitter.com/assaf/status/843496579952754688
IFTTT  twitter  favorite 
5 weeks ago by tswaterman
"there's (still) no magic bullet": a deep dive into gory bits of AWS cost and resource optimization by :
from twitter_favs
5 weeks ago by kas
The million dollar engineering problem
from twitter_favs
5 weeks ago by andydavies
RT : This is the kind of AWS blog post people need

o7 to &co
from twitter
6 weeks ago by stilkov
Segment is the analytics API you've always wanted. It's the easiest way to install all of your favorite analytics tools at once!
aws 
6 weeks ago by strzalek
For an early startup, using the cloud isn’t even a question these days. No RFPs, provisioning orders, or physical shipments of servers. Just the promise of…
from instapaper
6 weeks ago by tmcw
We managed to cut our AWS bill by over one million dollars annually
sysadmin  business 
6 weeks ago by verbila
The million dollar engineering problem http://bit.ly/2mRwUEb
IFTTT  Buffer 
6 weeks ago by soofaloofa
The million dollar engineering problem Wise words from on learning to managing infrastructure costs.
from twitter_favs
6 weeks ago by bf4
The million dollar engineering problem: For an early startup, using the cloud isn’t even a question these days. No RFPs, provisioning orders, or physical shipments of servers. Just the promise of getting up and running on “infinitely scalable” compute power within minutes. via Pocket, added at:March 17, 2017 at 12:14PM
IFTTT  Pocket 
6 weeks ago by LordEnzo
After a three months of focused work, we managed to cut our AWS bill by over one million dollars annually. Here is the story of how we did it.
aws  scaling  cost  segment 
6 weeks ago by fjordaan
For an early startup, using the cloud isn’t even a question these days. No RFPs, provisioning orders, or physical shipments of servers. Just the promise of getting up and running on “infinitely scalable” compute power within minutes. via Pocket
business  pinboard  tech 
6 weeks ago by lfcipriani
"At Segment, we found ourselves in a similar situation near the end of last year. We were hitting the classic startup scaling problems, and our costs were starting to grow a bit too quickly. So we decided to focus on reducing the primary contributor: our AWS bill."
aws  architecture  experience-report  cloud-computing  cost-optimization 
6 weeks ago by bkittelmann
Segment is the analytics API you've always wanted. It's the easiest way to install all of your favorite analytics tools at once!
6 weeks ago by akarpo
How a company saved a million dollars a year by looking in depth into their usage of AWS
aws  architecture  cloud 
6 weeks ago by beebware
RT : Drop everything and carefully read this:
from twitter
6 weeks ago by rjw1
After being in the weeds for three months, we managed to hit our goal. We eliminated over $1m dollars in annual spend off our AWS bill. And managed to increase our average utilization by 20%.
architecture  aws  cost  performance 
6 weeks ago by euler