Big and Ugly Associative Array Loop


    #1/bin/bash

PKG_MANAGER="paludis"
typeset -a OverlayKeys 
typeset -a OverlayVals

OverlayKeys[0]="foobar"
OverlayVals[0]="http://www.someurl.com/file.xml"

OverlayKeys[1]="mpd"
OverlayVals[1]=""


function _overlay_command() {
    key="${1}" && val="${2}"
    
    if [ -z "${val}" ]; then
        case $PKG_MANAGER in
        paludis)
            playman -a ${key}
            paludis -s x-${key}
            ;;
        emerge)
            layman -a ${key}
            layman -s ${key}
            ;;
        esac
    else
        case $PKG_MANAGER in
        paludis)
            playman --layman-url ${val} -a ${key}
            paludis -s x-${key}
            ;;
        emerge)
            layman -f -o ${val} -k -a ${key}
            layman -s ${key}
            ;;
        esac
    fi
}

function _loop_overlay() {
    for (( x=0 ; x < ${#OverlayKeys[@]}; x++ ))
    do
        if [ $key == "${OverlayKeys[$x]}" ]
        then
            _overlay_command "${key}" "${OverlayVals[$x]}"
        fi
    done
}


function setup_overlay() {
    for _key in ${OverlayKeys[@]}
    do
        key="${_key}" _loop_overlay
    done
}

setup_overlay
    Working on a script, took out relevant parts for this. Got a big associative array I'm looping through, and running a few shell commands based on a package manager. It works fine, but she's a bit fugly and I"m not much of a bash guy. Any help prettying up would be appreciated.
    2009-11-14T03:43:23+00:00
    1100
    Bash
    big-and-ugly-associative-array-loop
    3
    Big and Ugly Associative Array Loop
    
    2009-11-14T18:34:18+00:00
    197
    
      197
      http://netzdamon.myopenid.com
      gregf
      0.0
      4


    #!/bin/bash

SRC="/export/shared /export/beancounters/"
DEST_BASE="/export/trash"

for S in $SRC 
do
  DEST=$DEST_BASE/$(basename $S)

  rm -rf ${DEST}/*

  while read F
  do

    DIR=$(dirname "$F"); DIR=${DIR#$S}
    if [[ ! -d $DIR ]]
    then
      mkdir -p \"${DEST}${DIR}\"
    fi

    mv \"$F\" \"${DEST}${DIR}\"

  done < <(find $S -type f -mtime +15 | sort)

  find $S -type d -empty -exec rmdir {} \;

done

    I'm planning to run this snippet in a cron job on the 1st and 16th of each month. The idea is to periodically remove all the old stuff from a shared folder on our LAN. Do you see any issues with the code?
    2009-11-11T15:46:11+00:00
    1095
    Bash
    temp-folder-cleanup
    1
    Temp folder cleanup
    
    2010-01-08T13:15:23+00:00
    652
    
      652
      http://claimid.com/agnul
      agnul
      0.0
      1


    #This is the error command
egrep "smtp" -i ./production.log

#Would like to correct it with !! to have this: (whitout taping it again)
egrep -i "smtp" ./production.log
    I looking for a quick easy way to change this command. I would like to use the !! command with bash to put the -i at the correct place:
    2009-07-27T17:45:14+00:00
    970
    Bash
    how-to-quiclky-change-this-bash-command
    2
    How to quiclky change this bash command
    
    2009-09-09T19:31:27+00:00
    1596
    
      1596
      http://carlpelletier.myopenid.com
      carlpelletier.myopenid.com
      0.0
      1
      http://carlpelletier.ca


    #!/bin/bash

# Global variables
UPS='ups.domain.com'
FS='fs.domain.com'
FS_PORT='2'
PASSWORD='password'
MGR="/opt/bin/mgr"
MS_PORT='42'
CONSUMER_MS='msa1.domain.com msa2.domain.com msa3.domain.com'
BUSINESS_MS='msb1.domain.com msb2.domain.com msb3.domain.com'
count=0

echo "UID:EMAIL:ALIASES:COS:LAST_LOGIN:CREATED:DOMAIN:ENABLED:ANTISPAM:QUOTA_LIMIT:QUOTA_USED:FORWARDING_WITH_COPY\n" > report.txt
for server in $CONSUMER_MS; do
        for uid in `$MGR -s $server -p $MS_PORT -w $PASSWORD USER mbox.domain.com LIST | awk '{ print $3 }'`; do
                # Query MS
                MS_OUT=`$MGR -s $server -p $MS_PORT -w $PASSWORD USER mbox.domain.com SHOW $uid | awk '{ print $2 " " $3 }'`
                used=` awk '/DISKUSAGE/ { print $2 }' <<< "$MS_OUT"`
                limit=`awk '/BASEQUOTA/ { print $2 }' <<< "$MS_OUT"`

                # Query FS
                FS_OUT=`$MGR -s $FS -p $FS_PORT -w $PASSWORD USER SHOW $uid mbox.domain.com`
                mailbox=`   awk '/MAILPATH/              {split($0,a,"="); print a[2]}' <<< "$FS_OUT"`
                forwarding=`awk '/PSFORWARDASATTACHMENT/ {split($0,a,"="); print a[2]}' <<< "$FS_OUT"`
                email=`     awk '/PSACCOUNTEMAILADDRESS/ {split($0,a,"="); print a[2]}' <<< "$FS_OUT"`
                mailhost=`  awk '/MAILHOST/              {split($0,a,"="); print a[2]}' <<< "$FS_OUT"`
                cos=`       awk '/MAILCLASS/             {split($0,a,"="); print a[2]}' <<< "$FS_OUT"`
                enabled=`   awk '/ENABLEDFLAG/           {split($0,a,"="); print a[2]}' <<< "$FS_OUT"`
                last_login=`awk '/LASTSUCCESSFULLOGON/   {split($0,a,"="); print a[2]}' <<< "$FS_OUT"`
			
				# Query mailbox
                if test -n "$mailhost" && -n "$mailbox"
                then
                        echo "created=`ssh $mailhost -i mailbox_info.pub grep $mailbox UID | awk '{ print $2}'`"
                fi
                echo "$uid:$email:$aliases:$cos:$last_login:$created:$domain:$enabled:$antispam:$limit:$used:$forwarding" >> report.txt
                echo $((++count))
        done
done

for server in $BUSINESS_MS; do
        echo $server
done

echo 'Script completed.'
    I'm trying to collect customer statistics with a shell script.Problem is that currently its runtime is about 50h which is too much. As I'm pretty new to scripting so my solution works but obviously it could be better.
My main concern currently is if the refactoring the if statements to use single awk instance would yield some speed gain.

After that I should continue to find some way to reduce those network calls and check if they could be replaced with ldap queries or something as I guess those would be faster to execute.
    2009-06-03T07:14:43+00:00
    894
    Bash
    adding-speed-to-script
    -37
    Adding speed to script
    
    2009-11-19T20:25:43+00:00
    1532
    
      1532
      http://zmyrgel.blogspot.com
      Zmyrgel
      0.0
      2


    gunzip -c database_dump.sql.gz | mysql -u username -h host -ppassword database
    Just wondering if this is the best way to do this when, for example, you need to restore a database dump on a server, but the unzipped file is too large for the disk quota...
    2009-02-11T00:43:52+00:00
    753
    Bash
    unzip-large-database-dump-directly-into-mysql
    4
    Unzip large database dump directly into mysql
    
    2009-06-14T15:30:01+00:00
    1339
    
      1339
      http://bedlam.myopenid.com
      bedlam.myopenid.com
      5.0
      1


    #!/bin/bash
# filename: uniqq

        # uniq replacement. no post sort is needed. the original order persists

        read i
        while read line; do
                i=$i$'\n'$line
        done
        echo "$i" | awk '!x[$0]++'

    I got a text file (about 2000 lines) where i want to remove duplicate lines, but without loosing the orign sort order.

I found out that i can use awk for this. To get this a bit handier i wrote this little script. Unfortunatelly its noticeable slower than using awk directly from the command line. Any idea for improvements?

The whole point is that i want to use "uniqq" instead of "awk '!x[$0]++'" which is harder to remember for me.
    2009-02-07T18:35:25+00:00
    739
    Bash
    sort-uniq-alternative
    6
    sort | uniq alternative
    
    2009-02-14T16:45:59+00:00
    1330
    
      1330
      http://ck01.myopenid.com
      ck01.myopenid.com
      0.0
      3


    #!/bin/bash
echo -e "\n I wish you a merry Christmas and an happy new year! \n"
exit(0);
    Can you rewrite this oneliner into another language of your choice?

Thanks, Eineki
    2008-12-25T08:59:13+00:00
    684
    Bash
    multi-language-refactor
    6
    Multi language refactor
    
    2008-12-30T04:59:19+00:00
    337
    
      337
      http://eineki.wordpress.com
      Eineki
      4.4444
      43
      http://eineki.wordpress.com


    for id in `cat ids|sort -rn|uniq`; do echo `grep -w $id ids|wc -l` :$id; done|sort -nr
    I was processing logs like this:
$ tail -1000000 process.log|grep 'PROCESSING N'|cut -d: -f2 > ids
$ cat ids
1
2
3
4
6
8
26
44
54
55
56
83
87
93
95
121
129
139
143
159
161
165
56
83
159
56
83
159
169
56
83
159
56
83
159
173
56
83
159
56
83
159
175
56
83
159

Then to get stats:
$for id in `cat ids|sort -rn|uniq`; do echo `grep -w $id ids|wc -l` :$id; done|sort -nr
8 :83
8 :56
8 :159
1 :95
1 :93
1 :87
1 :8
1 :6
1 :55
1 :54
1 :44
1 :4
1 :3
1 :26
1 :2
1 :175
1 :173
1 :169
1 :165
1 :161
1 :143
1 :139
1 :129
1 :121
1 :1

Can this be simplified somehow?
    2008-11-01T12:37:55+00:00
    576
    Bash
    groupping-with-stats
    2
    Groupping with stats
    
    2008-11-01T18:29:45+00:00
    1138
    
      1138
      http://gmarik.blogspot.com
      gmarik.blogspot.com
      0.0
      0


    ## mem [shell-unix-generic]
#!/bin/sh

# 2008 Jon Mulder <mulderje@muohio.edu>

AWK=/usr/bin/awk
EGREP=/bin/egrep
FREE=/usr/bin/free
GREP=/bin/grep
HEAD=/usr/bin/head
PERL=/usr/bin/perl
PS=/bin/ps
SORT=/usr/bin/sort

TOTALMEM=`$FREE -m | $GREP Mem | $PERL -pe 's/Mem:\s+([0-9]+).*/$1/'`
RETVAL=0
case "$1" in
    nginx)
          $PS aux | $GREP nginx | $EGREP -v 'grep|/bin/sh' |
          $AWK -v totalmem="$TOTALMEM" 'BEGIN { total=0; printf "%-10s %-10s %-10s %s\n", "USER", "%MEM", "PID", "PROCESS TYPE"
                                   print "---------------------------------------------" }
                                   { printf "%-10s %-10.2f %-10s %s\n", $1, $4, $2, $12; total+=$4; }
                  END { printf "%-10s %-10.2f %-0.0f MB used\n", "TOTAL:", total, (total/100)*totalmem }'
      RETVAL=$?
  ;;
    thin)
          $PS aux | $GREP thin | $EGREP -v 'grep|/bin/sh' |
          $AWK -v totalmem="$TOTALMEM" 'BEGIN { total=0; printf "%-10s %-10s %-10s %s\n", "USER", "%MEM", "PID", "SERVER:PORT"
                                   print "--------------------------------------------" }
                                   { printf "%-10s %-10.2f %-10s %s\n", $1, $4, $2, $13; total+=$4; }
                  END { printf "%-10s %-10.2f %-0.0f MB used\n", "TOTAL:", total, (total/100)*totalmem }'
      RETVAL=$?
  ;;
    php)
        $PS aux | $GREP php5 | $EGREP -v 'grep|/bin/sh' |
        $AWK -v totalmem="$TOTALMEM" 'BEGIN { total=0; printf "%-10s %-10s %-10s %s\n", "USER", "%MEM", "PID", "EXECUTABLE"
                         print "-------------------------------------------" }
                         { printf "%-10s %-10.2f %-10s %s\n", $1, $4, $2, $11; total+=$4; }
                END { printf "%-10s %-10.2f %-0.0f MB used\n", "TOTAL:", total, (total/100)*totalmem }'
    RETVAL=$?
  ;;
        top)
        [ -n $2 ] && [ $2 -eq $2 2> /dev/null ] && NUM=$2 || ( NUM=10 && echo "$2 is not a number - Using 10 by default" )
        $PS aux | $SORT -nr -k 4 | $HEAD -$NUM |
        $AWK -v totalmem="$TOTALMEM" 'BEGIN { total=0; printf "%-10s %-10s %-10s %-10s %s\n", "USER", "%MEM", "MEM", "PID", "EXECUTABLE"
                         print "------------------------------------------------------" }
                         { printf "%-10s %-10s %-10.2f %-10s %s\n", $1, $4, ($4/100)*totalmem, $2, $11; total+=$4; }
                END { printf "%-10s %-10.2f %-0.0f MB used\n", "TOTAL:", total, (total/100)*totalmem }'
    RETVAL=$?
        ;;
    *)
      echo "Usage: mem {nginx|thin|php|top [number]}"
      exit 1
  ;;
esac
exit $RETVAL

    So I use thin to serve my rails projects, php, and nginx as a back end for everything.  I am on slicehost, and have a limited amount of memory to use.  I know there are utilities like top and ps that are already out there, and I use them in this script, but I wanted to make something nice and easy for myself (mostly because I have never used a lot of these things)... I have never really made a shell script, used awk, parsed files with perl, etc.  So bear that in mind while taking a look at this.

I was wondering what I could do to improve on this code, or if there was a better way to go about this in general.  I actually looked a libproc, because I was going to do this in either c or c++, but I couldn't find any documentation on it.

To use the script you run it like: <name of script> {nginx|thin|php|top [number]}

If the first argument is either nginx, thin, or php it prints all of the information for those processes, and tells you how much memory each one of those processes are using, and gives it to you as a percentage based on how much RAM you have in your machine.

If you put in top as the first argument, you get the top ten memory using processes, also with the same information.  But if you put in a number X as the second argument, then you get the top X memory using processes...

It's pretty self explanatory, but I thought I would give a brief description... Anyway let me know what you think.

Thanks,
Jon
    2008-07-30T20:51:00+00:00
    407
    Bash
    memory-usage-script
    2
    Memory usage script
    
    2008-07-31T04:32:42+00:00
    885
    
      885
      http://jonmulder.com/id
      JonM1827
      4.0
      5
      jonmulder.com


    #!/bin/bash
currentTime=$(date -u '+%s')
statement=
until [ -z "$1" ]  # Until all parameters used up . . .
do
  statement="$statement"' '"$1"
  shift
done
 
echo -e $currentTime'\t'$statement >> $HOME/progress/progress.txt
    I wanted to store things I was learning into a simple text file. For example, if the command was called ilearned, you could say 'ilearned working on a bash script' and it would store that info. I'm brand new at this...
    2008-07-08T05:49:54+00:00
    354
    Bash
    store-string-from-terminal-into-file-w-timestamp
    2
    store string from terminal into file w/timestamp
    
    2008-07-08T11:36:20+00:00
    789
    
      789
      http://bporterfield.myopenid.com
      bporterfield
      0.0
      0


    ps ax | grep "nginx: master" | xargs -I % ruby -e 'exec "sudo kill #{"%".split(/\s/).first}"'
    hi,
i am not expert on linux shell command, however i was trying to kill nginx master process. 
i hope you guys can suggest me better simplified version.

best wishes.
    2008-06-27T16:32:14+00:00
    338
    Bash
    kill-nginx-master-proces
    9
    kill nginx master proces
    
    2008-06-29T12:15:31+00:00
    696
    
      696
      http://we4tech.openid.org
      we4tech
      0.0
      0


    #!/bin/bash
for ((i=0;i<=17;i+=1))
         do
                option=`more test.txt|awk -F' ' '{ print $1}'`
                echo $option
        done

    I an working on a script that asks for user to select functions using a kdialog checklist.  The selected options are then processed.
I have pushed all of the output from the kdialog into a temporary file (for testing - later I plan to cram it into a var)
There are 17 items on the list.  I need the code to awk the list 17 times using a different position each time so that each item that is checked can be done.  My biggest challenge is getting the awk { print $x} where x is increased by one each time the for loop runs.  Thanks for your help
    2008-05-08T20:38:43+00:00
    292
    Bash
    for-loop-using-awk
    1
    For Loop using Awk 
    
    2008-05-08T20:50:43+00:00
    579
    
      579
      http://roxannspencerolsen.blogspot.com
      roxannspencerolsen.blogspot.com
      0.0
      1


    SCRIPT="gen_${1}.sql"
echo "--gen_${1}.sql" > $SCRIPT
typeset -i counter
counter=0

echo "Get all lines that start with D|4| (this may take a while)"
LINES=$(grep "D|4|" $1)

echo "Generate insert statements to $SCRIPT"
for LINE in $LINES; do
  REFNO=$(echo $LINE | awk '{split($0,a,"|");print a[8]}')
  echo "insert into CHEQUE (datecomptable, noreference) values (to_date('2007/11/30','YYYY/MM/DD'),'${REFNO}');" >> $SCRIPT
        counter=counter+1
done

echo "commit;" >> $SCRIPT
echo "${counter} records created in $SCRIPT"
    Here's a quick and dirty bash script that takes a file and looks for a particular line, then loops through those lines and splits the contents to get the 8th column and create an sql insert statement with it.  I'm not looking to make it more generic, but the for loop is unbearably slow.  How to improve it?
    2008-02-07T19:57:46+00:00
    229
    Bash
    slow-bash-for-loop
    6
    Slow bash for loop.
    
    2008-02-19T14:44:06+00:00
    13
    
      13
      http://furtive.myopenid.com/
      furtive
      0.0
      2


    #!/bin/bash
# Host checker 0.6
# original idea by no0tic
# modified by motoplux

#Hosts list
HOST[1]=192.168.1.1;
HOSTname[1]=link1;
HOST[2]=192.168.0.1;
HOSTname[2]=link2;
HOST[3]=192.168.0.5;
HOSTname[3]=link3;
HOST[4]=192.168.0.100;
HOSTname[4]=link4;

# number of hosts
NHOST=4;

# email Subject
SUBJECT="Segnalazione problema Host"
# email To
EMAIL="miamail@quelloched'è"

for i in $(seq 1 $NHOST); do
	fail[$i]=0;
done	

while [ 1 ]; do
	#do 1 ping to localhost	
	ping -c 1 localhost > /dev/null ;
	if [[ "$?" -eq 0 ]]
	then
		for i in $(seq 1 $NHOST); do
			#do 1 ping to host		
			ping -c 1 ${HOST[$i]} > /dev/null ;
	   		if ! [[ "$?" -eq 0 ]]
				then 
					if [[ "${fail[$i]}" -eq 0 ]] 
						then 			
							#play alarm.wav;
							#display alarm in terminal
							echo "Problemi all'host ${HOST[$i]} alias ${HOSTname[$i]}";
							# send an email using mail
							echo "Problemi all'host ${HOST[$i]} alias ${HOSTname[$i]}" | mail -s "$SUBJECT" "$EMAIL"
							fail[$i]=1;
					fi
				else
					fail[$i]=0;
			fi
  					
		done			
		
	   #time in seconds before repeating ping		  
	   #sleep 3;
	else
		echo "Problemi di host locale";
		exit -1;
	fi
done
    A simple bash script that send a mail if a set Host is down, simple by pinging it in a time interval.
    2007-11-24T12:18:57+00:00
    162
    Bash
    host-checker
    1
    Host checker
    
    2007-12-14T09:06:19+00:00
    384
    
      384
      http://motoplux.myopenid.com
      motoplux
      0.0
      0


    tab() # new tab from current dir
{
osascript -e "
tell application \"iTerm\"
 tell the first terminal
  launch session \"Default Session\"
  tell the last session
   write text \"cd $(pwd)\"
  end tell
 end tell
end tell"
}
    Here's a small script I wrote to open a new tab in iTerm in the current directory. I place this in my .bash_profile and use it a lot to start Rails server and open a new tab to work in: tab; script/server
Know any better way of doing this ?

Btw, I'm also introducing a new language in here: Bash :) Hope you like it!

Also, I know the server is bloody slow, sorry for that. I'm migrating to a new and absurdly fast server at the moment. Hang on!
    2007-10-04T18:07:03+00:00
    63
    Bash
    open-a-new-tab-in-current-directory-from-iterm
    2
    Open a new tab in current directory from iTerm
    
    2007-10-04T18:07:03+00:00
    1
    
      1
      http://macournoyer.myopenid.com/
      macournoyer
      3.6667
      72
      http://macournoyer.com