dusko + commandline   282

Backups - USAIL
Unix System Administration Independent Learning - USAIL
backup  sysadmin  unix  cli  commandline  shell 
march 2018 by dusko
SMTP, testing via Telnet - FreeBSDwiki
When troubleshooting problems with SMTP service - your own, or others - it is frequently very helpful to be able to "speak" to the SMTP server directly, rather than going through a mail client which won't necessarily tell you exactly what the SMTP server is saying. You can easily do this with the telnet client. Note that many ISPs do not allow outbound connections on port 25 to any SMTP server but their own - if you get timeouts when trying to connect to port 25, you should try port 587, which is the standard ESMTP port. (Port 587 connections normally require SMTP AUTH, which is covered below.)
mailserver  sendmail  smtp  cli  commandline  shell 
march 2018 by dusko
Understanding the bin, sbin, usr/bin, usr/sbin split
> I see that busybox spreads it's links over these 4 directories.
> Is there a simple rule which decides which directory each link lives in.....
> For instance I see kill is in /bin and killall in /usr/bin.... I don't
> have a grip on what might be the logic for that.
cli  commandline  terminal  shell  unix 
december 2017 by dusko
pigshell - The missing shell for the web
Pigshell is a pure client-side Javascript app running in the browser, which presents resources on the web as files. These include public web pages as well as private data in Google Drive, Dropbox, Facebook and even your desktop. It provides a command line interface to construct pipelines of simple commands to transform, display and copy data.
cli  commandline  terminal  web 
december 2017 by dusko
HOWTO: Successfully using Mutt in an Exchange 2010 Environment
I recently started using Mutt as my full time mail client at the office and have found (through searches and tinkering) a way to use Mutt in conjunction with Wyrd (ncurses front end for Remind) to effectively manage calendar events sent to me via email. In addition, I also found a way to asynchronously create an aliases file from an Active Directory query. The only thing I've found that I can't do with this setup is view shared Exchange calendars. For that I still have to use OWA. I do not do much of it, though, so I'm OK with that for now. These scripts are far from perfect but they work, which was my goal.

Download the scripts from: https://gist.github.com/5527716 or clone them from here: https://gist.github.com/5527716.git
mutt  plaintext  cli  commandline  shell 
december 2017 by dusko
Backup and Recovery with Native OS Tools
* Linux Complete Backup and Recovery HOWTO - Charles Curley

* Linux Complete Backup and Recovery HOWTO - Charles Curley

* Chapter 8. Backup and Restore Procedures - Charles Curley

* Linux / UNIX Tar Full and Incremental Tape Backup Shell Script

* Full System Backup with rsync - ArchWiki

* FreeBSD - Backup Basics

* FreeBSD UFS/ZFS Snapshot Management Environment

* Howto: Backup and restore your system! (Ubuntu forums)

* Bare Metal Recovery by Charles Curley

* Creating a clone of current server

* Cloning a server in RHEL 4

* Copying config files from one RHEL server to another

* Cloning Hard Drives with GNU/Linux

* DAR -- Disk ARchive

* tar -- Unix Tape Archiver

* Tar options for bare metal recovery

* Easy Automated Snapshot-Style Backups with Linux and Rsync

* How To Backup Your Entire Linux System Using Rsync

* Rsnapshot

* Clonezilla

* rclone - rsync for cloud storage and local filesystem

* Disk and Partition Backup and Cloning

* Cloning systems, disks and partitions

* Cloning parition table on a new disk

* BackInTime

* TimeShift - functionality similar to the System Restore feature in Windows and the Time Machine tool in Mac OS.

* SUSE Backup and Restore

* Usage of Knoppix with Partimage

* Solaris Backup/Restore

* Solaris snapshots

* Usage of Relax-and-Recover on RHEL

* GNU tar: an archiver tool

* cpio command

* Cloning harddrives and partitions using dd

* DD Command

* GNU ddrescue

* Linux LVM-based Snapshots

* Recovery of LVM partitions

* Back Up (And Restore) LVM Partitions With LVM Snapshots

* Perl and Shell Backup Scripts

* luckyBackup - a powerful, fast and reliable backup and sync tool

* rotate-backups

* bash-backup-rotation-script

* Shell script to mirror backups to another server in pull mode using rsync

* Continuous Archiving for Postgres

* Macrium Reflect - Free backup, disk imaging and cloning solution for personal and commercial use

* storeBackup's Top Features
Restore easily - even without storeBackup! The most important aspect of a backup tool is easy restoring from a transparent (native) storage format.
Copies / compresses files to another disk and generates backups with time stamps while saving space by recognizing files with identical contents (even if renamed or copied) and hardlinking them (so each backup is totally complete, independent and autonomous)
Detects identical files in different, independent backups (eg. of different computers)
Splits big image files (from e.g., TrueCrypt, mbox, Xen, KVM, VMware, etc.) or complete devices into small pieces and saves only differences to existing backups, thereby saving space and time
Sophisticated possibilities for including and excluding files and directories.
Support of sparse files.
Supports isolated incremental backups (e.g., when travelling with a laptop) and later integration in master backup.
Supports time shifted replication of backups to additional other disks / locations, even for complex schemes.
Supports checking of backups via md5 sums to eg. recognize bit rot of the hard disk.
Enables checking of (old) files in the source directory with md5 sums in the backup to eg. recognize bit rot of the hard disk.
Fast backups even over slow or high latency network connections.
Contains an entire suite of backup-related tools.

>> storeBackup

>> storeBackup Documentation

✭ Copy hard disk or partition image to another system using a network and netcat (nc)

* Shell Script To Dump All MySQL Databases Every Hour To NAS Storage

* Back Up and Restore MySQL Server (MariaDB) Databases

* MySQL Server (MaridaDB) Backups

* MySQL Backup Bash Shell Script

* Shell script to backup all db's (MySQL Server / MariaDB)

✭ How do I clone an OpenLDAP database - Stack Overflow

* Keep a Backup of Installed Packages
You might prefer to have a clean system on reinstall but sometimes it is nice to reinstall applications from a previous machine/setup. Keeping a backup list of packages will make this a snap. Just give your package manager a list of all the packages you want it to install and let it rip.
- Here are the backup and restore methods for each of the major distros/package managers.










backup  linux  freebsd  sysadmin  tar  rsync  perl  snapshot  lvm  config  zfs  cli  commandline  shell  python  from notes
december 2017 by dusko
oVirt as a VMware replacement? (self.sysadmin)
It took a long time for us to get oVirt working, and we're currently still just testing it. That said, with EL7, it seems usable. I've generally heard that proxmox is one of the easiest to get going, but we are all Red Hat (well Scientific Linux), and were quite wary of using a "black box" for something as important as VMs.

oVirt is backed by Red Hat, and does seem like it and ProxMox are the most like a VMware environment. The other common tool, OpenStack (which I think has the most momentum behind it), isn't really the same sort of thing IMHO, and very focused on private or public cloud. If you're not looking to enable external (to IT) people to spin up and down VMs on a whim, it's not a great fit.

How familiar with Red Hat Enterprise Linux are you? I'm probably intermediate, and it required our dedicated Linux Admin to get Ovirt working at all. However, now that it's working, the WebGui is pretty point and click, but still non-intuitive. There's going to be a lot of manual reading here to figure out what settings have to be set, what the different options even mean, far less why you'd want them one way or another. It is possible to set up a VM by sort of feeling your way through however.

One other point - all of the tools seem to eventually get down to libvirt and KVM (except maybe Xen), so it wouldn't hurt to get a little familiar with those if you're admining it.

I used Ovirt for several years up to version 3.5.

1. It works pretty well. I never had any major problems with it. I did run into a few annoying bugs.

2. You really need to engineer your infrastructure for GlusterFS to make it really shine. It will support shared storage, but most everything I felt was keyed to use Gluster.

3. There are SDKs for it, but when I was using it, the SDKs were not well documented.

4. The Performance was good. I ran about 50 VMs on two servers with another server that acted as a Fibre Channel array.

5. I hated that you had to have an NFS file share to store things like ISO images.

6. Ovirt Node didn't work at all for me. I finally just installed a vanilla Fedora box and then installed Ovirt on there and it worked. Hopefully it's better now.

Do you even need to "spec to the max" ? I get maxing servers with RAM but noticed most workloads don't use nearly as much CPU to RAM ratio. I noticed a lot of admins I have worked with with seem to throw hardware at a problem instead of actually looking at the proper sizing. We were able to increase our consolidation ratios by 40% by just using better capacity planning practices. For example, larger clusters, DRS, etc..
kvm  virtualization  commandline  cli 
november 2017 by dusko
Using ed, the Unix line editor
A line editor is a text editor in which each editing command applies to one or more complete lines of text. Line editors predate screen based editors like vi, and originated in an era where a computer operator interacted with a teleprinter (a printer with a keyboard), with no video display, and no ability to move a cursor within a document.

This explains why ed focus on lines instead of screens. This also explains why ed is not so intuitive and not natural to someone coming from the world of screen based editors. To add insult to injury, ed is from a time where memory was very very precious, so using abbreviated commands and very succinct error messages made a lot of sense, but contributed to the overall confusion.
ed  editor  unix  cli  commandline 
november 2017 by dusko
Mutt notes - Allen Rueter's notes
The From_ (From space) seperator, is a little more picky than you might expect, it has to be in the following format:

From user@server.com Dow Mon Da HH:MM:SS Year tzone

learned this in trying to convert files from webmail to mbox format so they could be bounced to the new server.
mutt  cli  commandline 
october 2017 by dusko
Tips on configuring mutt or pine for use with Exchange
In your .muttrc file you will need to add:

set spoolfile=imaps://LoginId@mail.server.com/INBOX

or use mutt -f imaps://....
for reading meeting request veiwing look at this (https://eitknows.seas.wustl.edu/~allen/meeting.pl)
mutt  cli  commandline 
october 2017 by dusko
Miscellaneous Answers
* wmic errors
* strace httpd
* copy cd/dvd to iso to use on machine with out cd/dvd drive
umout if it is mounted
dd if=/dev/dvd of=somedvd.iso
scp somedvd.iso somehost:
mkdir /mnt/iso
somehost# mount -o loop somedvd.iso /mnt/iso

* make a CentOS boot usb-stick for netinstall
1) lftp or wget to your favorite iso site and get .../centos/5.5/os/x86_64/images/diskboot.img
2) dd if=diskboot.img of=/dev/sdc
3) take the usb-stick and boot it, you will need the above host and path to do the reset of the install it when ask for host and path, give it everything upto but exclude images/diskboot.img

* apache on linux, mod_qos error mutex nospace left on device
try: ipcs -s | grep root | awk ' { print $2 } ' | xargs -n 1 ipcrm -s

* Build OpenCV on CentOS

* So the machine only crashes under load?
MEMTEST86+ runs for over a whole weekend, no problems.
breakin from advanced clustering (http://www.advancedclustering.com/software/breakin.html) runs for a whole weekend without problems.
Try matlab or octave, s=sqrt(ram-in-bytes/16) ; tic;m=rand(s,s); mm=m*m; toc
It will use all of memory and gets every cpu/core running!

* building 64bit php5 with mysql, on centos 5

./configure --prefix=/usr/local/php52 --with-pdo-mysql --with-mysql
checking for mysql_errno in -lmysqlclient... no
configure: error: Try adding --with-zlib-dir=<DIR>. Please check config.log for more information.

adding with-zlib doesn't help, logs show it trying to use 32 bit version, so:
./configure --prefix=/usr/local/php52 --with-pdo-mysql --with-mysql --with-libdir=/usr/lib64
configure: error: Cannot find libmysqlclient under /usr.
Note that the MySQL client library is not bundled anymore!

strange error given the correct incantation is:
./configure --prefix=/usr/local/php52 --with-pdo-mysql --with-mysql --libdir=/usr/lib64 --with-libdir=lib64

* #ffff##: in .Xauthority is not good
the simple thing is to just remove the .Xauthority file.
cli  commandline  shell  linux 
october 2017 by dusko
Five Ways to Keep Remote SSH Sessions and Processes Running After Disconnection
process orphan daemon screen tmux nohup fg disown setsid ssh openssh
shell  ssh  cli  commandline 
october 2017 by dusko
Diff Find & Md5sum ( The difference in 2 Directories )
How to use diff ( we agreed we would forget about cmp ) in combination with the "md5sum" and "find" command.

How to make a list of checksums of all files in a directory and those below. Again we make a "test" directory and placed in it some files to create the example.
cli  commandline  shell 
october 2017 by dusko
Example xfreerdp usage with clipboard plugin - Unix & Linux Stack Exchange
Didn't need the below; now just using this:

xfreerdp -grab-keyboard /audio-mode:1 /size:2540x1045 /u:theskett@rocketsoftware.com /v: +clipboard


$ cd /usr/src
$ git clone git://github.com/FreeRDP/FreeRDP.git
$ cd FreeRDP

$ sudo aptitude install libcunit1-dev libdirectfb-dev xmlto doxygen \
libxtst-dev libavutil-dev libavcodec-dev build-essential git-core \
cmake libssl-dev libx11-dev libxext-dev libxinerama-dev \
libxcursor-dev libxdamage-dev libxv-dev libxkbfile-dev \
libasound2-dev libcups2-dev libxml2 libxml2-dev \
libxrandr-dev libgstreamer0.10-dev \
libgstreamer-plugins-base0.10-dev libxi-dev \

$ sudo cmake -DCMAKE_BUILD_TYPE=Debug -DWITH_SSE2=ON .
$ sudo make
$ sudo apt-get install checkinstall
$ sudo checkinstall make install
$ sudo which xfreerdp
$ xfreerdp --version
cli  commandline  shell 
october 2017 by dusko
X servers use two schemes to copy text between applications. The first one (old and deprecated) is the cutbuffer. It is a simple buffer in which any application can store text. The other scheme is the selection and works differently. There may be many selections in a single server. An application does not copy data in a selection, it "owns" it. When another application wants to retrieve the content of a selection, it asks the owner.

* From

I'm a Linux person at heart, but for me the two equal and separate selections remain an unfortunate usability problem. Luckily for anyone with the same disposition, Autocutsel can help make X's behavior more logical and intuitive. It's a great little program that synchronizes the cut-buffer with CLIPBOARD, or both the cut-buffer and CLIPBOARD with PRIMARY as well.

Install Autocutsel (pacman -S autocutsel on Arch) and put the following two lines into your .xinitrc (or just run them from a terminal to immediately observe the effects):

autocutsel -fork &
autocutsel -selection PRIMARY -fork &

Now, no matter where you copy and paste from, be it Ctrl+C in Chrome, p in Vim, or through text selection in X, your clipboard is consistent across the entire system.
x11  unix  cli  commandline  shell  terminal 
october 2017 by dusko
guard - GitHub
Guard automates various tasks by running custom rules whenever file or directories are modified.

It's frequently used by software developers, web designers, writers and other specialists to avoid mundane, repetitive actions and commands such as "relaunching" tools after changing source files or configurations.

Common use cases include: an IDE replacement, web development tools, designing "smart" and "responsive" build systems/workflows, automating various project tasks and installing/monitoring various system services.

For a full categorized list of known Guard plugins, look here: https://github.com/guard/guard/wiki/Guard-Plugins

If you have any questions about Guard or want to share some information with the Guard community, please go to one of the following places:
cli  commandline  sysadmin  tool 
october 2017 by dusko
SNAP - OpenBSD upgrade tool (OpenBSD, Shell)
SNAP(8) - System Manager's Manual
snap is a ksh(1) script designed upgrade an OpenBSD machine to the latest snapshot available.
openbsd  shell  cli  commandline  sysadmin 
october 2017 by dusko
Pronunciation guide for Unix
Pronunciation guide for UNIX

How do I pronounce vi , or ! , or /* , or ...? You can start a very long and pointless discussion by wondering about this topic on the net. Some people say vye, some say vee-eye (the vi manual suggests this) and some Roman numerologists say six. How you pronounce vi has nothing to do with whether or not you are a true Unix wizard.

Similarly, you'll find that some people pronounce char as care, and that there are lots of ways to say # or /* or ! or tty or /etc. No one pronunciation is correct - enjoy the regional dialects and accents.
shell  cli  commandline  unix 
october 2017 by dusko
doitlive -- doitlive 3.0.0 documentation
doitlive is a tool for live presentations in the terminal. It reads a file of shell commands and replays the commands in a fake terminal session as you type random characters.
commandline  cli  presentation 
october 2017 by dusko
htty - a console application for interacting with web servers
htty is a console application for interacting with web servers. It’s a fun way to explore web APIs and to learn the ins and outs of HTTP.
http  commandline  cli 
september 2017 by dusko
UNIX tips -- Learn 10 good UNIX usage habits
This article suggests 10 UNIX command-line habits worth picking up -- good habits that help you break many common usage foibles and make you more productive at the command line in the process. Each habit is described in more detail following the list of good habits.
unix  cli  commandline 
september 2017 by dusko
What is the exact difference between a 'terminal', a 'shell', a 'tty' and a 'console'?
Lots of good explanations here over the years.
(i added a little bit to the quote to try to make a summary. i might be wrong about all of it.)
-- tmx, 22 aug 2017

terminal = tty = text input/output environment; possibly virtual.
tty = teletypewriter - a teleprinter-style device resembling a typewriter.
console = physical terminal (is also used for virtual terminal).
shell = command line interpreter.
command line - the language used to send commands to the command-line interpreter running in a shell from the terminal/terminal emulator. (also used to mean where you type at the "shell prompt".)

Console, terminal and tty are closely related. Originally, they meant a piece of equipment through which you could interact with a computer: ...=

-- StackExchange
Unix & Linux Stack Exchange | unix.stackexchange.com | 2010-2017
shell  console  tty  commandline  cli 
september 2017 by dusko
whatswrong - A bash script to do some basic checks on the system
A simple bash script to do some basic checks on the system, and report any problems found. The basic concept is that this script should be run as a first step when debugging any issue. Nothing in the script requires root privileges to run.

Please note that there are some situations where this script may cause a system with problems to hang, for example if there are unavailable mounted volumes.

You can run on any system with internet access directly, even if the disks are remounted read-only:

curl https://raw.github.com/BashtonLtd/whatswrong/master/whatswrong | bash

Alternatively, as long as you have SSH access to the remote system you can download via your local machine, and then send and execute over SSH as follows:

curl https://raw.github.com/BashtonLtd/whatswrong/master/whatswrong | ssh $remote_server bash
cli  commandline  sysadmin  server 
september 2017 by dusko
gatherinfo - A script to gather info from a running system
Simple script with dependency on POSIX sh(1) to gather info from the running system. The result of the script is report-like HTML file with outputs of the desired commands.
sh  shell  cli  commandline  sysadmin  server 
september 2017 by dusko
My sysadmin toolbox
With Duplicity you can do encrypted backups of your filesystem to a remote server. I use it to backup my home directory to a server in another town. I do incremental backups daily and full backups monthly. My script looks like this:

----------- duplicity_backup.sh -----------
duplicity -i --include /etc --include /boot/loader.conf --include /root --include /usr/local/etc --exclude '**' / scp://username@backupdomain/BACKUP/backup_system
duplicity -i /home/username scp://username@backupdomain/BACKUP/backup_home

The command duplicity --verify scp://username@backupdomain/BACKUP/backup_home /home/username compares the system with the backed up files to check what has changed since the last backup.

To restore a file as it was four days ago you would run:

duplicity -t 4D --file-to-restore personal/project.c scp://username@backupdomain/BACKUP/backup_home /home/username/personal/project.c


I also use ssync and FUSE to synchronize my workstation with my notebook (in one direction: notebook -> workstation). I mount the remote directory to /tmp with fusefs-sshfs and then I sync it with my home:

mkdir /tmp 2>/dev/null
mount_fusefs auto /tmp/username sshfs username@workstation: && \
ssync -f /home/username -t /tmp/username 1>>backup_fusefs.log 2>&1

Unison (http://www.cis.upenn.edu/~bcpierce/unison/) is a similar application, with which you can mirror two filesystems bidirectionally.
sysadmin  backup  cli  commandline 
august 2017 by dusko
Using Graphviz in your project to create graphs from dot files
This article describes how Graphviz can be incorporated in your project to show graphs describing your state machine / process for example.
cli  commandline  graph 
august 2017 by dusko
inotify - Get your file system supervised
This program is an "inotify cron" system. It consists of a daemon and a table manipulator. You can use it a similar way as the regular cron. The difference is that the inotify cron handles filesystem events rather than time periods.
cron  server  cli  commandline  sysadmin 
august 2017 by dusko
Monitor long-running commands: integrations
March 11, 2017

This is the fourth and last in a series of posts describing the system I built to monitor long-running commands.
cli  commandline  shell 
august 2017 by dusko
Monitor long-running commands: the monitoring script
March 8, 2017

This is the third in a series of posts describing the system I built to monitor long-running commands.
cli  commandline  shell 
august 2017 by dusko
Monitor long-running commands: wait for command to finish
March 5, 2017

This is the second in a series of posts describing the system I built to monitor long-running commands.
cli  commandline  shell 
august 2017 by dusko
Monitor long-running commands: capture return status
March 4, 2017

This is the first in a series of posts describing the system I built to monitor long-running commands.
For me, this started as a way to light a blink 1 LED (https://blink1.thingm.com/) at work when a long build completed, bringing me back to the more important task from whatever may have distracted me. After a recent post by Casey Liss, I’ve added watch notifications to the mix.
cli  commandline  shell 
august 2017 by dusko
gow - Gow (Gnu On Windows) is the lightweight alternative to Cygwin
Unix command line utilities installer for Windows.
Gow (Gnu On Windows) is the lightweight alternative to Cygwin. It uses a convenient Windows installer that installs about 130 extremely useful open source UNIX applications compiled as native win32 binaries. It is designed to be as small as possible, about 10 MB, as opposed to Cygwin which can run well over 100 MB depending upon options.
commandline  cli  windows 
august 2017 by dusko
Text mode diagrams using UTF-8 characters and fancy colors https://pypi.python.org/pypi/diagram
commandline  cli  sysadmin  graph 
july 2017 by dusko
visidata - A console spreadsheet tool for viewing tabular data - VisiData (vd on the command-line) spreadsheet
VisiData (vd on the command-line) is an open-source multitool for interacting with tabular data from within the terminal.
dataanalysis  console  shell  terminal  cli  commandline 
july 2017 by dusko
Colours in Terminal
True Colour (16 million colours) support in various terminal applications and terminals. ---- It's a common confusion about terminal colours... We have this:
Plain ASCII -- ANSI escape codes (16 colour codes with bold/italic and background) -- 256 colour palette (216 colours + 16 ansi + 24 gray) (colors are 24bit) -- 24bit true colour ("888" colours (aka 16 million)).
terminal  shell  cli  commandline 
september 2016 by dusko
apphelper - file viewer and saver
apphelper is a general purpose handler for files downloaded from the web or presented as email attachments. I routinely use it with my mail reader (mutt http://freshmeat.net/projects/mutt/) for most attachments and my web browser for things I wish to box view and keep, such as PDFs.
terminal  shell  mutt  cli  commandline 
july 2016 by dusko
Sam Text Editor by Rob Pike
A minimalist text editor with a graphical user interface, a very powerful command language and remote editing capabilities. Written by Rob Pike (http://genius.cat-v.org/rob-pike/). ---- Also could be considered a graphical version of the venerable ed (http://man.cat-v.org/unix-1st/1/ed) on steroids, lots of steroids.
ed  terminal  shell  editor  texteditor  plaintext  cli  commandline 
july 2016 by dusko
« earlier      
per page:    204080120160

Copy this bookmark: