straup + sysadmin   59

Chef Solo tutorial: Managing a single server with Chef
“The Chef documentation assumes you have an entire server farm to manage, so it hits you with a lot of complexity. If all you want is to set up and maintain a single VM, this tutorial will help.

We will be creating “throwaway” VMs that we can recreate with a single call. If you need to have a single server system persist for years across all changes, you should perhaps check out Puppet for a less cloudy solution.

Using Amazon EC2?

Any fresh instance of an Ubuntu AMI will do, for example ami-06ad526f (11.04, 32-bit EBS in us-east-1).

Hint: The EC2 API tools are (by nature) complex and error-prone, so I recommend you stay with the point-and-click web interface for as long as possible, instead of trying to automate launching new instances.

Hint 2: Create a separate EBS volume as a data partition to hold your databases, etc.

Hint 3: Use Elastic IPs for all your instances.

All you need is

a laptop with Bash and SSH (no need to even install Chef), and

any vanilla Linux server that you can SSH into.

Why do we run against vanilla servers? In my opinion, the system your deployment scripts run against should be as minimal possible, essentially straight from the vendor. So instead of manually installing packages like Chef and then creating a snapshot to instantiate our VMs from, we will use a freshly installed system and make the entire process scripted. On the downside, we need some bootstrapping code (~30 LOC), but on the upside we get more repeatability and less maintenance overhead, and we stay independent from any specific cloud or virtualization provider.”
chef  tutorial  devops  linux  sysadmin  aws  ec2  via:migurski 
october 2014 by straup – docker misconceptions #f7228d0fdbfa261f2a6cbf2c7a2a34ad
Service discovery is very useful when you get to very large scales and are using auto-scaling. In those cases it becomes too costly and problematic to manage hard-coded service locations. However, service discovery apps introduce more complexity, magic, and point of failures, so don't use them unless you absolutely need to. Instead, explicitly define your servers in your configurations for as long as you can
sysadmin  highlights  from:dogeared  url:f1f9c5790be02aede60f2713e1095333  dt:year=2014  dt:month=06  dt:day=13  dt:timestamp=1402668864 
june 2014 by straup
Fucking Shell Scripts | The easiest, most common sense server configuration management tool
Completely confused by Chef? Blowing your brains out over Ansible? Wanna just use fucking shell scripts to configure a server? Read on!
bash  deployment  sysadmin  via:mojodna 
march 2014 by straup
How the NSA Plans to Infect 'Millions' of Computers with Malware - The Intercept #c2cb7b6f8b265db224ea32b62479f6c6
In one secret post on an internal message board, an operative from the NSA’s Signals Intelligence Directorate describes using malware attacks against systems administrators who work at foreign phone and Internet service providers. By hacking an administrator’s computer, the agency can gain covert access to communications that are processed by his company. “Sys admins are a means to an end,” the NSA operative writes.

The internal post – titled “I hunt sys admins” – makes clear that terrorists aren’t the only targets of such NSA attacks. Compromising a systems administrator, the operative notes, makes it easier to get to other targets of interest, including any “government official that happens to be using the network some admin takes care of.”
nsa  security  surveillance  sysadmin  highlights  from:instapaper  dt:year=2014  dt:month=03  dt:day=13  url:33439e0f701b9a309c90f28950ea2409 
march 2014 by straup
Simple init.d script template « Werx Limited
this is good especially because upstart seems so easy ... but isn't
sysadmin  unix 
march 2013 by straup
alandipert/fswatch · GitHub
"This is a small program using the Mac OS X FSEvents API to monitor a directory. When an event about any change to that directory is received, the specified shell command is executed by /bin/bash."

-- I was not able to make this work, FWIW
osx  sysadmin  notifications 
january 2013 by straup
etsy/mctop · GitHub
"Inspired by "top", mctop passively sniffs the network traffic passing in and out of a server's network interface and tracks the keys responding to memcache get commands. The output is presented on the terminal and allows sorting by total calls, requests/sec and bandwidth."
memcache  sysadmin 
december 2012 by straup
"Python API library and shell utilities to monitor file system events."
python  sysadmin  notifications  via:karlcow 
december 2012 by straup
Mac Process Information — MacInside (Beta)
"MacInside is a library of information about processes and application running on your Mac. You can use it to learn about the hidden processes and applications running on your system, get recommendations for similar applications using our tagging system and make sure that your system is safe and secure."
osx  sysadmin  security 
june 2012 by straup
"Zipkin is a distributed tracing system that helps us gather timing data for all the disparate services at Twitter. It manages both the collection and lookup of this data through a Collector and a Query service. We closely modelled Zipkin after the Google Dapper paper."
logging  monitoring  sysadmin  twitter 
june 2012 by straup
A Cloud of My Own (Pinboard Blog)
"These pitfalls were well-documented in the manual, but navigating them required Talmudic levels of attention to the text and left me feeling paranoid and alone."
hardware  pinboard  sysadmin 
may 2012 by straup – the Blog ·
"balance was actually just what we needed. The only thing it was missing was support for monitoring all back end connections and dynamically disabling and enabling them as they fail or pass the monitoring checks.

So eventually I started looking into adding exactly that functionality to balance."
sysadmin  network  backup  monitoring 
march 2012 by straup
iamcal/logSmush - GitHub
"Aggregate logs from multiple servers in real time"
nodejs  logging  sysadmin 
november 2011 by straup
logtogss 0.1 : Python Package Index
"This is a small tool to log data to a Google Spreadsheet. Google Spreadsheets are very handy for manipulating data, but it's less easy to insert data programmatically."
google  sysadmin  logging  notifications  spreadsheets  cloudcastles 
november 2011 by straup
linux-wizard/timegrep - GitHub
"Perform a binary search through a log file to find a range of times and print the corresponding lines" ... also, it's called "TIMEGREP"
spacetime  grep  cli  sysadmin 
october 2011 by straup
Here be Dragons - OpenGeoData
"In 2008, the OpenStreetMap community decided to use dragon names as
the theme for OSM server names. Dragon names were chosen as a tribute
to the "Here be Dragons" marked on unexplored portions of maps and globes."
osm  dragons  sysadmin 
september 2011 by straup
Find out what is using your swap | All things Sysadmin
"...starting from kernel 2.6.16, we can find out using smaps which can be found in the proc filesystem. I’ve written a simple bash script which prints out all running processes and their swap usage." –– yay!
linux  sysadmin  swap  via:crowley 
august 2011 by straup
File::UStore -
"File::UStore is a perl library based on File::HStore to store files on a filesystem using a UUID based randomised storage with folder depth control over storage."
perl  sysadmin  storage  uuid 
august 2011 by straup
logstash - open source log management
"logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs."
logging  sysadmin  investgate 
july 2011 by straup
Tracking stats in your application | Allan Feid
"Inspired by Etsy's post on keeping track of everything, I set out to achieve similar functionality on a Rails application. Essentially, you run Graphite somewhere and send UDP packets from your application to StatsD. When you have Graphite and StatsD running, you can really get some awesome stats from your application and at the same time, since it's using UDP, the overhead is very minimal."
graphite  statsd  capacityplanning  sysadmin 
april 2011 by straup
rsms/ec2-webapp - GitHub
"This is a template I use to quickly set up Node.js-backed web apps on Amazon EC2"
aws  ec2  nodejs  sysadmin 
march 2011 by straup
Two tools that I would love more than anything
"I don’t think that these have been built yet by anyone, but I want these" -- what john said... in 2007
sysadmin  allspaw 
march 2011 by straup
iamcal's service.js at master - GitHub
"This module allows you to turn your node app into a linux init.d daemon with a minimum of hassle. It requires daemon.node."
unix  sysadmin  nodejs  iamcal  from delicious
november 2010 by straup
Start page – collectd – The system statistics collection daemon
"collectd is a daemon which collects system performance statistics periodically and provides mechanisms to store the values in a variety of ways, for example in RRD files."
collectd  capacityplanning  monitoring  sysadmin  from delicious
november 2010 by straup
Bees with machine guns! Low-cost, distributed load-testing using EC2. « News Apps Blog
"Bees with machines guns is a fabric script that I put together to aggressively load-test our sites. It provides methods to spin up an army of EC2 instances (the bees), assault a target server with a configurable amount of traffic, and then spin those instances back down."
aws  capacityplanning  devel  sysadmin  from delicious
november 2010 by straup
5 Minutes to Binary Replication (in Postgres)
"The official documentation on how to set up binary replication (or "hot standby") for version 9.0 are kind of daunting. So I thought I'd give you the 5 minute replication setup version: if you have a very simple setup and you can afford to shut down the master database, getting replication running can be simple indeed."
postgres  database  sysadmin  from delicious
october 2010 by straup
Foursquare outage post mortem - mongodb-user | Google Groups
"The main thing to remember here is that once you’re at max capacity, it’s difficult to add more capacity without some downtime when objects are small."
mongodb  foursquare  nosql  sysadmin  warstories  capacityplanning  from delicious
october 2010 by straup
Code as Craft » Frank Talk About Site Outages
"I was hoping this post would include a photo of you standing up on a desk, yelling at everyone to shut the fuck up if they’re not working on the outage"
etsy  capacityplanning  allspaw  sysadmin  community  from delicious
october 2010 by straup
Getting Good IO from Amazon's EBS
"Performance testing on EBS is very hard. The disks speed up and slow down on their own. A lot. Telling when your tweak is helping vs it just being luck is not easy. It feels a bit like trying to clock the speed of passing cars with a radar gun from the back of a rampaging bull."
aws  ebs  sysadmin  io  disk  from delicious
september 2010 by straup
Shine » New OpenCycleMap tileserver :: GravityStorm
"New hosting, for a start, on a dedicated server with both huge SATA disks (mirrored 1.5Tb) and also a pair of blindingly fast 128Gb striped SSD drives. The postgis database is on the fast disks, everything else being on the SATAs. The speed of the SSD drives really comes into its own while rebuilding the indexes during hourly updates, rather than being directly related to rendering."
osm  adnyallan  tilefarm  maps  geo  sysadmin  from delicious
september 2010 by straup
devstructure's doubledown at master - GitHub
"doubledown was written to make it easier for DevStructure users to use Textmate and other IDEs but it's far from the only way to skin the cat."
crowley  sync  fsevents  sysadmin  unix  from delicious
september 2010 by straup
"Simple and automated development environments. Experiment, collaborate and deploy with confidence." -- crowley's new thing!
crowley  sysadmin  devel  from delicious
august 2010 by straup
maplebed / ganglia-logtailer / overview —
"ganglia-logtailer is a python framework to crunch data from log files and submit it to ganglia via gmetric."
apache  ganglia  python  sysadmin  from delicious
july 2010 by straup
Python Package Index : gunicorn-console 0.1
"A curses application for managing gunicorn processes. The ps command is polled for gunicorn processes named with the setproctitle package and its output is parsed to display a list of gunicorn processes showing the process ID, port, name, rough memory used and number of workers for each gunicorn."
sysadmin  python  httpony  gunicorn  from delicious
may 2010 by straup
Appstats for Python - Google App Engine - Google Code
"The Python SDK includes a suite of tools for measuring the performance of your application, called Appstats. Appstats integrates with your web application framework to record events, and provides a web-based administrative interface for browsing statistics."
appengine  python  sysadmin  metrics  from delicious
may 2010 by straup

related tags

aa:ima=link  aa:post=airportcity  aa:year=2011  adnyallan  alert-design  allspaw  apache  api  appengine  automation  aws  backup  bash  capacityplanning  chef  cli  cloudcastles  collectd  community  containers  crowley  daemon  database  deployment  design  devel  devops  disk  dragons  dt:day=13  dt:month=03  dt:month=06  dt:timestamp=1402668864  dt:year=2014  ebs  ec2  etsy  foursquare  from:dogeared  from:instapaper  fsevents  ganglia  gdal  geo  git  golang  google  graphite  grep  gunicorn  hardware  highlights  http  httpony  iamcal  investgate  io  library  libtiff  linux  logging  mapnik  maps  memcache  metrics  migurski  mongodb  monitoring  netcat  network  newrelic  nginx  nodejs  nosql  notifications  nsa  osm  osx  perl  pinboard  postgres  proxy  python  samba  security  solr  spacetime  spreadsheets  statsd  storage  surveillance  swap  sync  sysadmin  tcp  tilefarm  tutorial  twitter  ubuntu  unix  url:33439e0f701b9a309c90f28950ea2409  url:f1f9c5790be02aede60f2713e1095333  uuid  via:crowley  via:karlcow  via:migurski  via:mojodna  warstories 

Copy this bookmark: