will.brien + cli   141

Getting curl to output HTTP status code? - Super User
NB: curl -I does a HEAD HTTP request, which can be problematic for testing the HTTP status code for some web application servers and services – Jay Taylor Sep 6 '12 at 17:32

And to get just the status number, pipe it to head -n 1|cut -d$' ' -f2 – Benubird Jul 17 '13 at 11:33

Don't forget to redirect curl's stderr: curl -I http://www.example.org 2>/dev/null | head -n 1 | cut -d$' ' -f2. Add -L to curl if you need the final status after redirects. – Aaron Blenkush Jul 24 '14 at 21:16
Following the redirect after only doing a HEAD request may cause interesting behavior, depending on how the app is programmed. – Scott McIntyre Sep 21 '15 at 21:16

curl -I -X GET will send a GET request, but give the same output. – jiggy Nov 30 '15 at 19:20
http  curl  linux  cli  reference 
18 days ago by will.brien
find - Finding all large files in the root filesystem - Unix & Linux Stack Exchange
The following command not only find you the top 50 largest files (>100M) on your filesystem, but also sort (GNU sort) by the biggest:

find / -xdev -type f -size +100M -exec du -sh {} ';' | sort -rh | head -n50

-xdev Don't descend directories on other filesystems.

On BSD find use -x which is equivalent to the deprecated -xdev primary.

For all files and directories, it's even easier:

du -ahx / | sort -rh | head -20

(the -x flag is what's required to constrain du to a single filesystem)

If you're not using GNU sort (from coreutils), use it without -h:

du -ax / | sort -rn | head -20

For currently directory only (for quicker results), replace / with ..
cli  linux  sysadmin  reference 
december 2017 by will.brien
postfix: aliases will be ignored - Server Fault
Instead, you probably want to use the virtual maps.

As root (or sudo)

In /etc/postfix/virtual (or where virtual is)

root hostmaster@my.domain

In main.cf

virtual_maps = hash:/etc/postfix/virtual

or (modern versions of postfix)

virtual_alias_maps = hash:/etc/postfix/virtual

after the virtual map has been modified

# postmap /etc/postfix/virtual
# postfix reload

Beware that all mail for "root" will be redirected to "hostmaster@my.domain".
postfix  mail  cli  linux  debian  sysadmin  reference 
october 2017 by will.brien
Follow-Up: Bash script locking with flock – tobrunet.ch Techblog
Example:

# stop on errors
set -e

scriptname=$(basename $0)
pidfile="/var/run/${scriptname}"

# lock it
exec 200>$pidfile
flock -n 200 || exit 1
pid=$$
echo $pid 1>&200

## Your code:

This needs some explanation:

Line 8 opens a file handle with ID 200 on $pidfile
Line 9 uses flock to lock file handle with ID 200 with exclusive access. The parameter -n means “Fail (with an exit code of 1) rather than wait if the lock cannot be immediately acquired”. This is catched by the two pipes (or operation) and will exit if the lock fails, f.e. if the handle is already locked
Line 10 writes the PID to the file handle

Everything after Your code is only run if the file has the exclusive lock on the file handle 200. This ensures that it only runs once at a time.
linux  cli  scripts  bash 
september 2017 by will.brien
python - Upgrading all packages with pip - Stack Overflow
To upgrade all local packages; you could use pip-review:

$ pip install pip-review
$ pip-review --local --interactive

pip-review is a fork of pip-tools. See pip-tools issue mentioned by @knedlsepp. pip-review package works but pip-tools package no longer works.
python  pip  sysadmin  cli  linux 
september 2017 by will.brien
ssh - Converting keys between openssl and openssh - Information Security Stack Exchange
If you just want to share the private key, the OpenSSL key generated by your example command is stored in private.pem, and it should already be in PEM format compatible with (recent) OpenSSH. To extract an OpenSSH compatible public key from it, you can just run:

ssh-keygen -f private.pem -y > private.pub
ssh  cli  sysadmin  aws  linux 
september 2017 by will.brien
youtube-dl
youtube-dl is a command-line program to download videos from YouTube.com and a few more sites. It requires the Python interpreter (2.6, 2.7, or 3.2+), and it is not platform specific. We also provide a Windows executable that includes Python. youtube-dl should work in your Unix box, in Windows or in Mac OS X. It is released to the public domain, which means you can modify it, redistribute it or use it however you like.
linux  youtube  python  video  archives  music  cli  scripts  baseball 
august 2017 by will.brien
Nyan Cat Telnet Server
This is an animated, color, ANSI-text telnet server that renders a loop of the Nyan Cat / Poptart Cat animation.
telnet  linux  cli  awesometown 
july 2017 by will.brien
Running commands - Ansible Tips and Tricks
Limit to one or more hosts

This is required when one wants to run a playbook against a host group, but only against one or more members of that group.

Limit to one host

ansible-playbook playbooks/PLAYBOOK_NAME.yml --limit "host1"

Limit to multiple hosts

ansible-playbook playbooks/PLAYBOOK_NAME.yml --limit "host1,host2"

Negated limit. NOTE: Single quotes MUST be used to prevent bash interpolation.

ansible-playbook playbooks/PLAYBOOK_NAME.yml --limit 'all:!host1'

Limit to host group

ansible-playbook playbooks/PLAYBOOK_NAME.yml --limit 'group1'

Limiting Tasks with Tags

Limit to all tags matching install

ansible-playbook playbooks/PLAYBOOK_NAME.yml --tags 'install'
ansible  documentation  reference  cli  sysadmin  python 
june 2017 by will.brien
Duply (simple duplicity) - duply
duply is a frontend for the mighty duplicity magic. duplicity is a python based shell application that makes encrypted incremental backups to remote storages. Different backends like ftp, sftp, imap, s3 and others are supported. See duplicity manpage for a complete list of backends and features.

duply simplifies running duplicity with cron or on command line by:

keeping recurring settings in profiles per backup job
automated import/export of keys between profile and keyring
enabling batch operations eg. backup_verify_purge
executing pre/post scripts
precondition checking for flawless duplicity operation

Since version 1.5.0 all duplicity backends are supported. Hence the name changed from ftplicity to duply.
duplicity  backup  cli  linux  aws  sysadmin  python 
june 2017 by will.brien
GitHub - petemcw/ansible-role-logwatch: Logwatch Role for Ansible
This role installs Logwatch which is an application that helps with simple log management by daily analyzing and reporting a short digest from activities taking place on your server.
ansible  cli  debian  linux  ssh  sysadmin  logs  logwatch 
june 2017 by will.brien
Securing a Server with Ansible
A while back, Bryan Kennedy wrote a post describing how he spends the first 5 minutes configuring and securing a new linux server. He runs through the list of commands and configuration settings that address things like:

secure passwords
automatic updates
basic intrusion detection
public key authentication
firewall settings
log monitoring

There were a couple of blog posts in response that took this one step further and demonstrated how to accomplish the same things in a more automated fashion using Ansible. Things move pretty fast and I found both posts were a little outdated. So this post continues the tradition and automates the process using an Ansible playbook. It takes care of the basic things described in these posts with a couple of additions and enhancements.
ansible  cli  sysadmin  linux  security  firewall  logwatch 
june 2017 by will.brien
GitHub - berzerk0/Probable-Wordlists: Wordlists sorted by probability originally created for password generation and testing
While I was able to locate a few Password Wordlists that were sorted by popularity, the vast majority of lists, especially the larger lists, were sorted alphabetically. This seems like a major practicality flaw! If we assume that the most common password is password, (which is actually the 2nd most common, after 123456) and we are performing a dictionary attack using an English dictionary, we are going to have to slog from aardvark through passover to get to password. I don't know off the top of my head just how common "aardvark" is as a password - but we could be wasting a lot of time by not starting with the most common password on our list!

I went to SecLists, Weakpass, and Hashes.org and downloaded nearly every single Wordlist containing real passwords I could find. These lists were huge, and I ended up with over 80 GB actual, human-generated and used passwords. These were split up among over 350 files of varying length, sorting scheme, character encoding, origin and other properties. I sorted these files, removed duplicates from within the files themselves, and prepared to join them all together.

Some of these lists were composed of the other lists, and some were exact duplicates. I took care to remove any exact duplicate files - we didn't need to have any avoidable false positives. If a password was found across multiple files, I considered this to be an approximation of its popularity. If an entry was found in 5 files, it wasn't too popular. If an entry could be found in 300 files, it was very popular. Using Unix commands, I concatenated all the files into one giant file representing keys to over 4 billion secret areas on the web, and sorted them by number of appearances in the single file. From this, I was able to create a large wordlist sorted by popularity, not the alphabet.
security  sysadmin  cli  wordlists 
june 2017 by will.brien
Tutorial: Create your own automated backup scripts in Linux with S3
As a software development company, we are often tasked to create backup scripts to ensure data is recoverable in case of catastrophic failure. I’m sharing below the basic script that we use to some clients that require automated daily backup script in Linux and Amazon S3. This script supports backup of files and database into a local storage and transfer the backup files to Amazon S3.
aws  backup  cli  linux  sysadmin 
june 2017 by will.brien
How to convert existing non-empty directory into a Git working directory and push files to a remote repository - Stack Overflow
Given you've set up a git daemon on <url> and an empty repository:

cd <localdir>
git init
git add .
git commit -m 'message'
git remote add origin <url>
git push -u origin master
git  cli  linux  reference  sysadmin  work 
june 2017 by will.brien
How to Secure Postfix Using Let's Encrypt - UpCloud
Once you have finished the process, the certificates will be stored under /etc/letsencrypt/live/<your.domain>/. You can add your new certificates to the Postfix configuration using the two commands below. Replace the <your.domain> with your email server’s domain name.

sudo postconf -e 'smtpd_tls_cert_file = /etc/letsencrypt/live/<your.domain>/fullchain.pem'
sudo postconf -e 'smtpd_tls_key_file = /etc/letsencrypt/live/<your.domain>/privkey.pem'

With the certificate installed, you can configure the rest of the email server.
letsencrypt  postfix  cli  sysadmin  email  encryption  <github> 
june 2017 by will.brien
Install Dropbox In An Entirely Text-Based Linux Environment - The Unofficial Dropbox Wiki
dropboxd will create a ~/Dropbox folder and start synchronizing it after this step! Go to the URL given; you should see a success message at the top of your screen.

NOTE: If you want to change the account it is linked to, unlink it from the first account, then kill the running dropbox process, start it up again (with “~/.dropbox-dist/dropboxd &”) and obtain the new host_id with dbreadconfig.py . If you don’t restart the dropbox client, it will give the same host_id (which for some reason cause me to be unable to change the account it is linked to).
linux  dropbox  cli  documentation  sysadmin 
june 2017 by will.brien
ftp - lftp timeout not working - Stack Overflow
I had the same problem with timeout not working when I used:

lftp -e "set net:timeout 5" -u user,pwd server

I have removed the -e "set..." part and use only lftp -u user,pwd server and the commands insert it into ~/.lftp/rc file:

cat ~/.lftp/rc
set net:timeout 5
set net:max-retries 2
set net:reconnect-interval-base 5

This works fine for me now.
bash  cli  ftp  linux  reference  tutorials  lftp 
june 2017 by will.brien
How to use grep and cut in script to obtain website URLs from an HTML file - Unix & Linux Stack Exchange
Not sure if you are limited on tools:

But regex might not be the best way to go as mentioned, but here is an example that I put together:

cat urls.html | grep -Eo "(http|https)://[a-zA-Z0-9./?=_-]*" | sort | uniq

grep -E : is the same as egrep
grep -o : only outputs what has been grepped
(http|https) : is an either / or
a-z : is all lower case
A-Z : is all uper case
. : is dot
\?: is ?
*: is repeat the [...] group
uniq : will remove any duplicates
html  bash  linux  cli  reference  tutorials 
june 2017 by will.brien
jq
jq is like sed for JSON data - you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text.
json  sed  cli  linux  software 
june 2017 by will.brien
GitHub - RoverWire/virtualhost: Bash Script to easy create or delete apache virtual hosts on ubuntu
Bash Script to allow create or delete apache/nginx virtual hosts on Ubuntu on a quick way.
apache  linux  cli  bash  scripts  nginx  <github> 
june 2017 by will.brien
Synchronizing folders with rsync
Synchronizing two folders with rsync

To keep two folders in synchrony, not only do we need to add the new files in the source folder to the destination folder, as in the past topics, we also need to remove the files that are deleted in the source folder from the destination folder. rsync allow us to do this with the parameter --delete, this used in conjunction with the previously explained parameter -u that updates modified files allow us to keep two directories in synchrony while saving bandwidth.

rsync -rtvu --delete source_folder/ destination_folder/

The deletion process can take place in different moments of the transfer by adding some additional parameters:

rsync can look for missing files and delete them before it does the transfer process, this is the default behavior and can be set with --delete-before
rsync can look for missing files after the transfer is completed, with the parameter --delete-after
rsync can delete the files done during the transfer, when a file is found to be missing, it is deleted at that moment, we enable this behavior with --delete-during
rsync can do the transfer and find the missing files during this process, but instead of delete the files during this process, waits until it is finished and delete the files it found afterwards, this can be accomplished with --delete-delay

e.g.:

rsync -rtvu --delete-delay source_folder/ destination_folder/
rsync  linux  cli  backup  library 
june 2017 by will.brien
Ansible Galaxy | geerlingguy.certbot
Installs and configures Certbot (for Let's Encrypt).
Requirements

If installing from source, Git is required. You can install Git using the geerlingguy.git role.
ansible  cli  debian  linux  letsencrypt  encryption  ssh  sysadmin 
may 2017 by will.brien
How do I install cygwin components from the command line? - Stack Overflow
Cygwin’s setup.exe

It also has a command line mode. Moreover, it allows you to upgrade all installed packages at once (as apt-get upgrade does on Debian based Linux).

Example use:

setup-x86_64.exe -q --packages=bash,vim

You can create an alias for easier use, for example:

alias cyg-get="/cygdrive/d/path/to/cygwin/setup-x86_64.exe -q -P"

Then you can, for example, install Vim package with:

cyg-get vim
windows  cli  cygwin  sysadmin 
may 2017 by will.brien
Ansible Galaxy | tersmitten.postfix
Set up a postfix server in Debian-like systems.
ansible  postfix  cli  debian  linux  ssh  sysadmin 
may 2017 by will.brien
Ansible Galaxy | geerlingguy.php
Installs PHP on RedHat/CentOS and Debian/Ubuntu servers.
ansible  linux  debian  sysadmin  cli  ssh  php 
may 2017 by will.brien
Ansible Galaxy | geerlingguy.mysql
Installs and configures MySQL or MariaDB server on RHEL/CentOS or Debian/Ubuntu servers.
ansible  linux  debian  sysadmin  cli  ssh  mysql 
may 2017 by will.brien
GitHub - geerlingguy/ansible-role-apache: Ansible Role - Apache 2.x.
An Ansible Role that installs Apache 2.x on RHEL/CentOS, Debian/Ubuntu, SLES and Solaris.
ansible  linux  debian  sysadmin  cli  ssh  apache 
may 2017 by will.brien
GitHub - debops/ansible-fail2ban: Install and configure fail2ban service
fail2ban is a service which parses specified log files and can perform configured actions when a given regexp is found. It's usually used to ban offending IP addresses using iptables rules (only IPv4 connections are supported at the moment).
ansible  linux  debian  sysadmin  cli  ssh  security 
may 2017 by will.brien
Your Debian-based data center in a box
Your Debian-based data center in a box

A collection of Ansible playbooks, scalable from one container to an entire data center.
ansible  linux  debian  sysadmin  cli  ssh 
may 2017 by will.brien
Ansible Galaxy | geerlingguy.firewall
Installs an iptables-based firewall for Linux. Supports both IPv4 (iptables) and IPv6 (ip6tables).

This firewall aims for simplicity over complexity, and only opens a few specific ports for incoming traffic (configurable through Ansible variables). If you have a rudimentary knowledge of iptables and/or firewalls in general, this role should be a good starting point for a secure system firewall.

After the role is run, a firewall init service will be available on the server. You can use service firewall [start|stop|restart|status] to control the firewall.
ansible  linux  debian  sysadmin  cli  ssh  security  iptables 
may 2017 by will.brien
Ansible Galaxy | geerlingguy.repo-dotdeb
ansible-galaxy install geerlingguy.repo-dotdeb
ansible  cli  debian  linux  php  ssh  sysadmin  apt 
may 2017 by will.brien
How to undo (almost) anything with Git · GitHub
One of the most useful features of any version control system is the ability to "undo" your mistakes. In Git, "undo" can mean many slightly different things.

When you make a new commit, Git stores a snapshot of your repository at that specific moment in time; later, you can use Git to go back to an earlier version of your project.

In this post, I'm going to take a look at some common scenarios where you might want to "undo" a change you've made and the best way to do it using Git.
git  reference  cli  linux  code 
september 2016 by will.brien
GitHub - trailofbits/algo: 1-click IPSEC VPN in the Cloud
Algo (short for "Al Gore", the Vice President of Networks everywhere for inventing the Internet) is a set of Ansible scripts that simplifies the setup of an IPSEC VPN. It contains the most secure defaults available, works with common cloud providers, and does not require client software on most devices.
vpn  security  linux  tools  cli  ansible  software 
august 2016 by will.brien
Penetration Testing Tools Cheat Sheet
enum4linux -a target-ip

Do Everything, runs all options (find windows client domain / workgroup) apart from dictionary based share name guessing
<todo>  security  tools  reference  cli  linux  windows  samba  sysadmin  work 
july 2016 by will.brien
GitHub - sferik/t: A command-line power tool for Twitter.
The CLI takes syntactic cues from the Twitter SMS commands, but it offers vastly more commands and capabilities than are available via SMS.
twitter  cli  ruby  debian 
april 2016 by will.brien
urlwatch - a tool for monitoring webpages for updates
This script is intended to help you watch URLs and get notified (via email or in your terminal) of any changes. The change notification will include the URL that has changed and a unified diff of what has changed.

The script supports the use of a filtering hook function to strip trivially-varying elements of a webpage.

Basic features

Simple configuration (text file, one URL per line)
Easily hackable (clean Python implementation)
Can run as a cronjob and mail changes to you
Always outputs only plaintext - no HTML mails :)
Supports removing noise (always-changing website parts)
Example hooks to filter content in Python
Uses If-Modified-Since header to save bandwidth (new in 1.9)
Convert non-UTF8 web pages to UTF-8 for mail (new in 1.10)
Handle non-zero shell exit codes as error (new in 1.11)
Support for concurrent/parallel downloads (new in 1.13)
Support for handling UTF-8 in Lynx and html2text (new in 1.15)
linux  cli  python  scripts  bash  archives 
december 2014 by will.brien
Remove old versions of Java via msiexec script
JAVA Uninstall and cleanup script for through Java 1.7.u11 with install of Java 1.7.u11 x86 and X64 in Mixed Environment
Calls removal of Java Autoupdate after install
Skips Removal of versions of Java 1.6.x if SAS is detected
These can be removed if your environment does not use SAS or uses built in Java for SAS
Requires x86 and x64 JRE Executables and x86 MSI files to install x86 Java on x64
MSI files can be located in
C:\Documents and Settings\<install user>\Application Data\Sun\Java\<version> or
C:\Users\<install user>\Appdata\locallow\Sun\Java\<Version>
after install to a workstation preferably 32-bit
written for use in SCCM 2007, but will likely work elsewhere
Written by David Nelson, Computer Professional, CSBS Computing, University of Utah
2012-2013
sysadmin  scripts  cli  windows  java 
april 2014 by will.brien
Detox - cleans up filenames
Detox is a utility designed to clean up filenames. It replaces difficult to work with characters, such as spaces, with standard equivalents. It will also clean up filenames with UTF-8 or Latin-1 (or CP-1252) characters in them.
software  linux  opensource  cli 
january 2014 by will.brien
IPTraf - An IP Network Monitor
IPTraf is a console-based network statistics utility for Linux. It gathers a variety of figures such as TCP connection packet and byte counts, interface statistics and activity indicators, TCP/UDP traffic breakdowns, and LAN station packet and byte counts.
linux  cli  sysadmin  networking  tools  work 
december 2012 by will.brien
Siege Home Page
Siege is an http load testing and benchmarking utility. It was designed to let web developers measure their code under duress, to see how it will stand up to load on the internet. Siege supports basic authentication, cookies, HTTP and HTTPS protocols. It lets its user hit a web server with a configurable number of simulated web browsers. Those browsers place the server “under siege.”
linux  cli  testing  apache  http  web  work 
november 2012 by will.brien
How to configure versioning of your /etc directory with Etckeeper
Keeping a version history of your configuration files is every administrator’s dream. Knowing that you have a complete history of all of your configuration files makes it really easy for system administrators to sleep well at night knowing that if anything goes wrong, they can simply roll back their configuration to an earlier date.

This is all possible with a program called EtcKeeper. EtcKeeper is a revision control system for your /etc directory using bzr, git, hf, or darcs as a back-end. EtcKeeper will allow you to make commits, like any other revision system, that will keep a version history of all your changes to the /etc directory. If configured correctly, you can also use EtcKeeper to check who made configuration changes and at what time, which can be useful for troubleshooting and auditing purposes.

In this article, I am going to show you how you can install and configure EtcKeeper to put your configuration files under version control.
linux  cli  sysadmin  centos  debian  ubuntu  versioncontrol  work  tutorials 
october 2012 by will.brien
Improving your resolv.conf file - Curator
The resolver can load up to 3 name servers. If your server needs to find the mx record of gmail.com, it will first need to resolve gmail.com locally.

using a resolver from resolv.conf

dig @8.8.8.8 gmail.com -t mx +short

result: alt1.gmail-smtp-in.l.google.com
connect to alt1.gmail-smtp-in.l.google.com

dig @8.8.4.4 alt1.gmail-smtp-in.l.google.com +short

result: 209.85.227.27
connect to 209.85.227.27:25

Optimized resolv.conf

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 4.2.2.2
option rotate
option timeout:1
linux  cli  dns  sysadmin  networking  reference  work 
october 2012 by will.brien
The Homely Mutt
Now that Sparrow is effectively dead many of its users will be looking for a new email client. If you’re not afraid of the terminal you may want to give Mutt a try.

Mutt certainly isn’t the prettiest email client around, and its setup/configuration process is one of the ugliest out there. But once you get it set up it’s got a lot of advantages over many other email clients.

In this post I’ll show you how to set up Mutt on OS X like I do.
email  mutt  google  cli  linux  imap  smtp  tutorials  <todo>  reference 
october 2012 by will.brien
Alpine and Gmail
After getting my Linux system updated and configured, I decided to install the Alpine mail client so that I could check my e-mail using a standalone client rather than having to use a web browser. While there are many text-based e-mail clients available (like mutt), I chose to use Alpine (developed by the University of Washington) since is an updated version of the classic Pine mail client and is offered under an Apache license. The primary benefit to using Alpine is that it is a light-weight, full-featured text-based mail client that is also easy to use and includes its own text editor (called Pico). The fact that it does not render HTML e-mail is an added bonus.

After looking through several websites on configuring Alpine, here is how I got it to work with Gmail via IMAP...
email  linux  cli  tutorials  reference  google  mutt  alpine  <todo> 
august 2012 by will.brien
Open Source Tripwire® | Free System Administration software downloads at SourceForge.net
"Open Source Tripwire® software is a security and data integrity tool useful for monitoring and alerting on specific file change(s) on a range of systems. The project is based on code originally contributed by Tripwire, Inc. in 2000."
linux  cli  security  sysadmin  software  opensource 
august 2012 by will.brien
Alpine (email client) - Wikipedia, the free encyclopedia
Alpine is a rewrite of the Pine Message System that adds support for Unicode and other features. Alpine is meant to be suitable for both inexperienced email users and the most demanding of power users. Alpine is developed at the University of Washington, as was Pine before it. Alpine can be learned by exploration and the use of context-sensitive help. The user interface can be customized.
email  linux  cli  mutt  alpine 
august 2012 by will.brien
Answer : Export/Inport package list from YUM?
List everything installed using rpm so you can copy/paste it into yum on another system:

"rpm -qa --qf %{NAME} "
linux  centos  redhat  cli  sysadmin  reference  work 
july 2012 by will.brien
Atoptool.nl
Atop is an ASCII full-screen performance monitor that is capable of reporting the activity of all processes (even if processes have finished during the interval), daily logging of system and process activity for long-term analysis, highlighting overloaded system resources by using colors, etc. At regular intervals, it shows system-level activity related to the CPU, memory, swap, disks, and network layers, and for every active process it shows the CPU utilization, the memory growth, priority, username, state, and exit code.
linux  sysadmin  tools  cli  networking  work 
june 2012 by will.brien
Backup Google Mail GMail / Home / BaGoMa
BaGoMa backs-up and restores the contents of a GMail account. It can restore all the labels (folder structure), as well as the flags (seen/read, flagged) of a message. It differs from other similar solutions in a few important ways:

It is Open Source Software. You can read the code to make sure your password and email contents remain private.

Some backup solutions force you to use only ASCII characters in your labels. BaGoMa has no such restrictions on label names. Specifically, foreign characters and '/' (the label hierarchy delimiter) are allowed.

BaGoMa is tuned to work specifically with GMail. Each message is only downloaded and saved once. Backup solutions that are designed to work with regular IMAP accounts will download each message multiple times (once per label) when used to back up a GMail account. That can significantly increase the bandwidth requirements and the amount of storage required for backup. Additionally, a faithful restoration of your GMail account might not be possible from such a backup.

BaGoMa requires your GMail account to be IMAP accessible. Check the Mail Settings and make sure "Enable IMAP" is selected under the "Forwarding and POP/IMAP" tab. In the "Folder Size Limits" section on the same tab, make sure the following option is selected: "Do not limit the number of messages in an IMAP folder".

BaGoMa never deletes any email from your account. A restore simply compares the contents of your local backup with the contents of your GMail account, and uploads any messages missing from your GMail account. It never deletes messages from GMail to make it match the local backup.

Google sometimes imposes limits on how much data can be transfered from your account. If you run into such a limitation, just wait a few days and try the backup again. BaGoMa will pick up from where it left off. If the temporary loss of IMAP access is critical to you, think twice before using this script. As far as I'm aware, web access to your email should continue unimpeded even if IMAP access is blocked.
google  email  backup  python  cli  linux  documentation  software  <todo> 
may 2012 by will.brien
Specials
The following is a special collection of freeware tools for system administration written in Perl. They cover advanced topics for the experienced admin, and are a valuable addition to a senior admin's collection or those wishing to be a senior admin. Most are related to performance or fault prevention.
perl  cli  linux  unix  sysadmin  humour 
april 2012 by will.brien
LightSquid Home Site : Home
some word about LightSquid

features list:

fast and simple install
fast log parser generatesmall per user data file
perl based cgi script for dynamic generated report pages
html template for design
no database required
no additional perl module
various reports
user groups support
graphics report (v 1.6+)
real name (v 1.6+)
multilangual interface
squid  proxy  logs  search  reference  privacy  linux  cli  work 
march 2012 by will.brien
Wbox HTTP testing tool
Wbox aims to help you having fun while testing HTTP related stuff. You can use it to perform many tasks, including the following.

Benchmarking how much time it takes to generate content for your web application.
Web server and web application stressing.
Testing virtual domains configuration without the need to alter your local resolver.
Check if your redirects are working correctly emitting the right HTTP code.
Test if the HTTP compression is working and if it is actually serving pages faster.
Use it as a configuration-less HTTP server to share files! (see the server mode documentation at the end of the Wbox tutorial in this page, but it's as simple as % wbox servermode webroot /tmp)

Wbox is free software under the BSD license and was written in ANSI C (POSIX runtime required) by Salvatore 'antirez' Sanfilippo.
apache  html  cli  linux  sysadmin  work 
march 2012 by will.brien
List of resources: Article text extraction from HTML documents
Following up to my overview of article text extractors, I’ll try to compile a list of research papers, articles, web APIs, libraries and other software that I encountered during my research.
datamining  python  html  rss  cli  linux 
march 2012 by will.brien
Add Public SSH Key to Remote Server in a Single Command
If you want to setup SSH keys to allow logging in without a password, you can do so with a single command.

The first thing you'll need to do is make sure you've run the keygen command to generate the keys:

ssh-keygen -t rsa

Then use this command to push the key to the remote server, modifying it to match your server name.

cat ~/.ssh/id_rsa.pub | ssh user@hostname 'cat >> .ssh/authorized_keys'
ssh  linux  cli  sysadmin  faq 
march 2012 by will.brien
Sculpting text with regex, grep, sed and awk
In this article, I'll provide a functional introduction to four important concepts and tools for sculpting text: regex, grep, sed and awk.

In short:

regex is a language for describing patterns in strings;
grep filters its input against a pattern;
sed applies transformation rules to each line; and
awk manipulates an ad hoc database stored as text, e.g. CSV files.

With this functional introduction, my goal is to introduce enough of each tool to cover 80%-90% of their niche uses cases.
unix  linux  cli  tutorials  reference  education  <todo> 
january 2012 by will.brien
BashPitfalls - Greg's Wiki
This page shows common errors that Bash programmers make. The following examples are all flawed in some way:
reference  linux  cli  education  lists 
january 2012 by will.brien
Deprecated Linux networking commands and their replacements
Specifically, the deprecated Linux networking commands in question are: arp, ifconfig, iptunnel, iwconfig, nameif, netstat, and route. These programs (except iwconfig) are included in the net-tools package that has been unmaintained for years. The functionality provided by several of these utilities has been reproduced and improved in the new iproute2 suite, primarily by using its new ip command. The iproute2 software code and documentation are available from the Linux Foundation.
linux  networking  cli  sysadmin  work 
january 2012 by will.brien
smtp-cli — command line SMTP client
smtp-cli is a powerful SMTP command line client with a support for advanced features, such as STARTTLS, SMTP-AUTH, or IPv6 and with a scriptable message composition capabilities supporting anything from simple plain-text messages right up to building complex HTML emails with alternative plain-text part, attachments and inline images. The MIME-Type of the attachments can either be guessed automatically or alternatively set on the command line, separately for each attachment if required.
software  linux  cli  smtp  exchange  email  work  utilities 
december 2011 by will.brien
Geek Thoughts: Shared Folders in Ubuntu with setgid and ACL (getfacl, setfacl)
Set Default ACL

The second issue is related to umask, the default mask applied when creating files and directories. By default umask is set to the octal value 0022, as demonstrated if you run the following:

$ umask
0022

This is a negative mask that is applied to the octal permission value of every file or directory created by the user. By default, a file is created with permissions rw-rw-rw-, equivalent to 0666 in octal and a directory is created with permissions rwxrwxrwx, equivalent to 0777 in octal. umask is then subtracted from that default to give the effective permission with which files and directories are created. So for a file, 0666-0022 gives 0644, equivalent to rw-r--r-- and for a directory 0777-0022 gives 0755, equivalent to rwxr-xr-x. This default is sensible for most situations but needs to be overriden for a team directory. The way to do this is to assign specific ACL entries to the team directory. The first thing to do is to install the acl package to obtain the necessary command line tools. Well, in fact, the first thing to do would be to enable acl on the relevant partition but we already did that at the very beginning.
linux  debian  ubuntu  cli  sysadmin  windows  samba  work  security  reference 
december 2011 by will.brien
Using Rsync and SSH
The last step is the cron script. I use something like this:
#!/bin/sh

RSYNC=/usr/bin/rsync
SSH=/usr/bin/ssh
KEY=/home/thisuser/cron/thishost-rsync-key
RUSER=remoteuser
RHOST=remotehost
RPATH=/remote/dir
LPATH=/this/dir/

$RSYNC -az -e "$SSH -i $KEY" $RUSER@$RHOST:$RPATH $LPATH
linux  cli  sysadmin  rsync  ssh  backup  tutorials  work 
november 2011 by will.brien
Mysql automatic backup script - AutoMySQLBackup
A script to take daily, weekly and monthly backups of your MySQL databases using mysqldump. Features - Backup mutiple databases - Single backup file or to a seperate file for each DB - Compress backup files - Backup remote servers - E-mail logs – More
backup  linux  debian  ubuntu  mysql  cli  scripts  sysadmin  work 
november 2011 by will.brien
Chapter 27. Desktop Profile Management
Roaming profiles are feared by some, hated by a few, loved by many, and a godsend for some administrators.

Roaming profiles allow an administrator to make available a consistent user desktop as the user moves from one machine to another. This chapter provides much information regarding how to configure and manage roaming profiles.

While roaming profiles might sound like nirvana to some, they are a real and tangible problem to others. In particular, users of mobile computing tools, where often there may not be a sustained network connection, are often better served by purely local profiles. This chapter provides information to help the Samba administrator deal with those situations.
linux  windows  cli  sysadmin  networking  samba  work 
november 2011 by will.brien
AdditionalResources/Repositories/RPMForge - CentOS Wiki
RPMforge is a collaboration of Dag and other packagers. They provide over 5000 packages for CentOS, including wine, vlc, mplayer, xmms-mp3, and other popular media tools. It is not part of Red Hat or CentOS but is designed to work with those distributions. See also Using RPMforge and Repoforge.

<!> Note: Because this repository is NOT part of CentOS, you should direct support questions to its maintainers at the Repoforge Users mailing list.

Packages are supplied in RPM format and in most cases are ready to use. The default RPMforge repository does not replace official CentOS base packages.
linux  redhat  sysadmin  reference  cli  work 
august 2011 by will.brien
How to back up your Gmail on Linux in four easy steps
I really like Gmail, but I also like having backups of my data just in case. Here’s how to use a simple program called getmail on Unix to backup your Gmail or Google Apps email.
google  backup  email  linux  cli  mutt  diy 
august 2011 by will.brien
'Avoiding domain/subdomain conflicts in blacklists' - MARC
Earlier this week there was a discussion about (what most would call)
squidGuard's unexpected behavior when a domain and one or more of its
subdomains exist in the same squidGuard domains file. As an example,
if your porn domains file contains: <nastypictures.com> and
<really.nastypictures.com>, the result is probably not what you
are expecting.

(Need more background? Read here:
<http://marc.theaimsgroup.com/?l=squidguard&m=108260329925644&w=2>)

This can be a real problem for those of us who merge blacklists from
multiple sources to create our personal lists. Sure, that process
creates a lot of duplicate entries, but those are easily identified
and removed. Identifying domain/subdomains is a bit more
challenging.
linux  cli  squid  filtering  work  reference 
july 2011 by will.brien
URLBlacklist.com
The latest blacklist download can be obtained from:
http://urlblacklist.com/cgi-bin/commercialdownload.pl?type=download&file=bigblacklist

The modified date and md5 information be obtained from:
http://urlblacklist.com/cgi-bin/commercialdownload.pl?type=information&file=bigblacklist

The daily changes contains diff files from the current blacklist to the previous blacklist. A script to use is is shown later. When using the blacklist with DansGuardian the blacklist files are put in a random order as DansGuardian sorts them when it reads them into memory. The Quick Sort algorithm is very slow at sorting a sorted list hence why we do this. The diffs are against a sorted blacklist however.

You are reminded that this is a commercial service that works upon an honour system. If you use the service then you must buy a subscription. We do not password the blacklist download to keep scripts simple for all users. Please do not abuse the service.
linux  cli  firewall  squid  filtering  work  reference 
july 2011 by will.brien
Netcat cheat sheet
It is possible to save the data which is send to you or you can give netcat a file which should be sended when a connection is made.
Here is an example on how to save the data which is being send by the server:

$ nc h.ackack.net 80 > response
# you can now send data but you won't see the output from the server because it is being redirected to a file.
$ nc ackack.net 80 > response
HEAD / HTTP/0.1

$ cat response
HTTP/1.1 200 OK
Date: <timestamp>
Server: Apache
Last-Modified: <timestamp>
Accept-Ranges: bytes
Content-Length: 3383
Connection: close
Content-Type: text/html
networking  sysadmin  firewall  apache  linux  cli  reference  tutorials  work 
july 2011 by will.brien
Regular expression examples - Wikipedia, the free encyclopedia
Despite this variability, and because regular expressions can be difficult to both explain and understand without examples, this article provides a basic description of some of the properties of regular expressions by way of illustration.
code  php  perl  cli  linux  reference  tutorials 
july 2011 by will.brien
Fstab - blag.wiki.aktivix.org
Some authorities suggest putting the User ID and Group ID in the options, though I didn't find that it was nececcsary. It would look like this (obviously using your own numbers from /etc/passwd). You can substitute your username and group name for you ID numbers.

//server/share /home/mountdirectory cifs username=smbusername,password=smbpassword,nocase,noperm,uid=1000,gid=1000,file_mode=0777,dir_mode=0777 0 0

Without the noperm option you will get weird permission errors, at least if the Samba share is on a Linux server or in a Mac share folder with SMB enabled.
linux  windows  cli  sysadmin  networking  samba  work 
april 2011 by will.brien
Pattern | CLiPS
Pattern is a web mining module for the Python programming language.

It bundles tools for data retrieval (Google + Twitter + Wikipedia API, web spider, HTML DOM parser), text analysis (rule-based shallow parser, WordNet interface, syntactical + semantical n-gram search algorithm, tf-idf + cosine similarity + LSA metrics) and data visualization (graph networks).

The module is bundled with 30+ example scripts.
python  linux  cli  database  archives  datamining 
march 2011 by will.brien
Quagga Software Routing Suite
Quagga is a routing software package that provides TCP/IP based routing services with routing protocols support such as RIPv1, RIPv2, RIPng, OSPFv2, OSPFv3, BGP-4, and BGP-4+ (see section Supported RFCs). Quagga also supports special BGP Route Reflector and Route Server behavior. In addition to traditional IPv4 routing protocols, Quagga also supports IPv6 routing protocols. With SNMP daemon which supports SMUX protocol, Quagga provides routing protocol MIBs (see section SNMP Support).

Quagga uses an advanced software architecture to provide you with a high quality, multi server routing engine. Quagga has an interactive user interface for each routing protocol and supports common client commands. Due to this design, you can add new protocol daemons to Quagga easily. You can use Quagga library as your program's client user interface.

Quagga is distributed under the GNU General Public License.
sysadmin  networking  snmp  router  work  linux  cli 
march 2011 by will.brien
PDFMiner
PDFMiner is a tool for extracting information from PDF documents. Unlike other PDF-related tools, it focuses entirely on getting and analyzing text data. PDFMiner allows to obtain the exact location of texts in a page, as well as other information such as fonts or lines. It includes a PDF converter that can transform PDF files into other text formats (such as HTML). It has an extensible PDF parser that can be used for other purposes instead of text analysis.
linux  cli  python  scripts  documentation  government  ireland  html 
february 2011 by will.brien
fdupes - Wikipedia, the free encyclopedia
fdupes is a program written by Adrian Lopez to scan directories for duplicate files, with options to list and delete them. It first compares file sizes and MD5 signatures, and then performs a byte-by-byte check for verification.

fdupes is written in C and is released under the MIT License.
ubuntu  linux  cli  music  sysadmin  work 
february 2011 by will.brien
littleblackbox - Project Hosting on Google Code
LittleBlackBox is a collection of thousands of private SSL and SSH keys extracted from various embedded devices. These private keys are stored in a database where they are correlated with their public certificates as well as the hardware/firmware that are known to use those private keys.

A command line utility is included to aid in the identification of devices or network traffic that use these known private keys. Given a public certificate, the utility will search the database to see if it has a corresponding private key; if so, the private key is displayed and can be used for traffic decryption or MITM attacks. Alternatively, it will also display a table of hardware and firmware that is known to use that private key.

The utility can obtain a public certificate several different ways:

1. You may give it the path to a public SSL certificate file.
2. You may give it the SHA1 hash of a public SSL/SSH certificate.
3. Given a host, it will retrieve the host's public SSL certificate.
4. Given a pcap file, it will parse the file looking for public SSL certificate exchanges.
5. Given a live network interface, it will listen for public SSL certificate exchanges.
security  ssh  sysadmin  cli  linux  unix  database  lists  work 
january 2011 by will.brien
SIP Brute Force Attacks on the Increase | Open Solutions
You can mitigate against these attacks by putting external SIP users into dedicated contexts which limit the kinds of calls they can make (internal only, local and national, etc); ask for a PIN for international calls; limit time and cost; etc.

However, the above might be a lot of work when simply blocking users after a number of failed attempts can be much easier and more effective. Fail2ban is a tool which can scan log files like /var/log/asterisk/full and firewall IP addresses that makes too many failed authentication attempts.

See VoIP-Info.org for generic instructions or below for a quick recipe to get it running on Debian Lenny.
security  sysadmin  asterisk  linux  cli  debian  firewall  voip  telephony  reference 
october 2010 by will.brien
« earlier      
per page:    204080120160

related tags

<github>  <todo>  alpine  ansible  apache  apt  archives  asterisk  audio  awesometown  aws  backup  baseball  bash  bittorrent  blogs  books  centos  cisco  cli  code  curl  cygwin  database  datamining  debian  delicious  dictionary  diy  dns  documentation  dropbox  duplicity  education  email  encoder  encryption  exchange  faq  filtering  firewall  ftp  git  google  government  hardware  html  http  humour  iax  imap  internet  iptables  ireland  java  json  ldap  letsencrypt  lftp  library  linux  lists  logs  logwatch  macintosh  mail  music  mutt  mysql  networking  nginx  opensource  perl  php  pinboard  pip  pix  podcast  postfix  privacy  programming  proxy  python  redhat  reference  router  rss  rsync  ruby  samba  scripts  search  security  sed  sip  sms  smtp  snmp  software  squid  ssh  storage  sysadmin  telephony  television  telnet  testing  tools  tutorials  twitter  ubuntu  unix  utilities  versioncontrol  video  voip  vpn  web  windows  wordlists  wordpress  work  xen  xml  youtube 

Copy this bookmark:



description:


tags: