Sorting images into folders based on exif year, month and date taken with bash

I need to sort about 18,000 home images taken over four years into a variety of folders on my home computer. We thought: hey, its on the server - mission complete. Three years later it’s an utter mess. Perhaps I can find the words to ask my computer to do this task for me.

Google is weird how it crawls websites...

I recently have been working on a side project hosting dedicated game servers at HostedGameServers.com and have been hitting a lot of snags.

Adaptation to EricBackup.bat

I took the liberty to adapt EricBackup.bat created by Eric.
Eric graciously permitted me to post to his blog aswell.

I have almost completely rewritten the batchfile for it to meet my fancy ;)
Replaced blat with sendmail and using delage32 to cleanup previous backupfiles and logfiles.
Date and time variables are in European format, change this if you want/need to.
The first part declares a bunch of variables, read this carefully.

Hands down to Eric, he laid the foundation for me to create this.

Below the source I have created. Comments are welcome.

MyBackup.bat

@ECHO OFF
:: ====
:: PROGRAMÂ Â Â Â Â Â Â Â Â Â : MYBACKUP.BAT
:: BYÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â : M.T.D. JONKER
:: ADAPTED FROMÂ Â Â Â Â : EricBackup.bat Version 2.8, EricGreer@GMail.com, http://blog.integrii.net/?p=76
:: CREATION DATEÂ Â Â Â : 20-12-2008
:: LATEST REVISIONÂ Â : 22-02-2009
::====
::22-02-2009: REDID REMOVAL OF OLD BACKUPFILES AND LOGFILES THROUGH EXTERNAL TOOL DELAGE32
::20-01-2009: ADDED REMOVAL OF PREVIOUS LOGFILES
::20-01-2009: CREATED VARIABLE "BACKUPFILE" TO REMOVE CLUTTER
::20-01-2009: REWROTE REMOVAL OF PREVIOUS BACKUPFILES
::10-01-2009: ADDED REMOVAL OF PREVIOUS BACKUPFILES
::22-12-2008: REPLACED BLAT WITH SENDMAIL
::20-12-2008: REWROTE BATCHFILE
::====
TITLE ADVANCED NTBACKUP SCRIPT
COLOR 1F
::====SET VARIABLES.HERE====
::ADJUST VARIABLES TO MEET YOUR PREFERENCE

::BACKUP RELATED VARIABLES
::SET USERNAME (AS SHOWN IN C:\DOCUMENTS AND SETTINGS\),
SET USER=
::PATH TO PUT LOGFILES OF THIS BATCHFILE AND NTBACKUP
SET ARCHIVEPATH=C:\Backups\logs
:: ORIGINAL (DEFAULT) LOCATION OF NTBACKUP LOGFILE
SET LOGPATH="%USERPROFILE%\Local Settings\Application Data\Microsoft\Windows NT\NTBackup\data"
::SET THE DESTINATION FOLDER OF YOUR BACKUP
SET DESTINATION=C:\BACKUPS\%COMPUTERNAME%
::SET PATH TO SENDMAIL.EXE
SET MAILPATH=C:\Backups\sendmail.exe
::SET PATH TO delage32.exe
SET DAPATH=C:\Backups\delage32.exe
::SET NAME OR IP OF SMTPSERVER TO BE USED
SET SMTPSERVER=smtp.server.com
::SET THE TO ADDRESS
SET TO=name@domain.com
::SET THE FROM ADDRESS
SET FROM=name@domain.com
::SET THE SOURCE OF YOUR BACKUP (FOR EXAMPLE: D:) NOT USED IN SYSTEMSTATE BACKUP
::SET SOURCE=D:
::SET A NOTE TO DESCRIBE YOUR BACKUP (USED IN FILENAME, SO REFRAIN FROM USING NON COMPATIBLE CHARACTERS)
::SET NOTE=D-DRIVE
::====COMMAND LINE PASSING====
::THE VARIABLES NOTE AND SOURCE CAN BE PASSED FROM THE COMMAND LINE.
::EIHTER SET THEM ABOVE OR USE THE SETTING BELOW
::COMMENT/UNCOMMENT AS NEEDED
::WHEN PASSED FROM THE COMMAND LINE BOTH SOURCE AND NOTE MUST BE PROPERLY FILLED
::SO USE SHORT FILENAMES SUCH AS C:\DOCUME~1\, WITHOUT QUOTES
SET SOURCE=%2
::USED IN FILENAME, SO REFRAIN FROM USING NON COMPATIBLE CHARACTERS
SET NOTE=%3
::====DONE SETTING MANUAL VARIABLES====

::DETERMINE BACKUPMODE
SET BACKUPMODE=NONE
IF "%1" == "systemstate" SET BACKUPMODE=SYSTEMSTATE
IF "%1" == "normal" SET BACKUPMODE=NORMAL
IF "%1" == "differential" SET BACKUPMODE=DIFFERENTIAL
IF %BACKUPMODE% == NONE GOTO NOPARAMS

::SET TIME AND DATE VARIABLES, (EUROPEAN FORMAT)Â DO NOT TOUCH (UNLESS OTHER FORMAT)
FOR /F "TOKENS=1,2 DELIMS=:" %%A IN ('TIME /T') DO (
SET HR=%%A
SET MIN=%%B
SET TM=%%A%%B
SET TME=%%A:%%B
)
FOR /F "TOKENS=2,3,4 DELIMS=- " %%A IN ('DATE /T') DO (
SET D=%%A
SET M=%%B
SET Y=%%C
SET DT=%%A-%%B-%%C
SET DTE=%%C%%B%%A
)
::====!!!OTHER VARIABLES, DO NOT CHANGE!!!====
:: NAME OF SYSTEM TO BE BACKED UP
SET SYS=%COMPUTERNAME%
::NAME OF BACKUPFILE (THIS IS LATER USED IN REMOVING PREVIOUS BACKUPFILES)
SET BACKUPFILE=%SYS%_%BACKUPMODE%_%NOTE%_%DT%.bkf
::LOGFILE OF THIS BATCHFILE
SET BATCHLOG=%ARCHIVEPATH%\%BACKUPMODE%_%NOTE%_%DT%_%TM%.txt
::NAME OF NTBACKUP LOGFILE TO BE ARCHIVED
SET NTBARCLOG=%BACKUPMODE%_%NOTE%_%DT%_%TM%.log
::====DONE SETTING VARIABLES====
::====

::CREATE ARCHIVE AND DESTINATION PATHS IF THEY DO NOT EXIST
IF NOT EXIST "%ARCHIVEPATH%" MD "%ARCHIVEPATH%" & ECHO "ARCHIVE PATH DID NOT EXIST, SO IT WAS CREATED." & ECHO.>> %BATCHLOG%
IF NOT EXIST "%DESTINATION%" MD "%DESTINATION%" & ECHO "DESTINATION PATH DID NOT EXIST, SO IT WAS CREATED." & ECHO.>> %BATCHLOG%

::BACKUP SELECTION, LOG TO CMD WINDOW
ECHO.
ECHO *********************************************************
ECHO * JOB %NOTE% STARTED
ECHO *********************************************************
ECHO *
IF %BACKUPMODE% NEQ SYSTEMSTATE ECHO * JOB SOURCE: %SOURCE%

IF %BACKUPMODE% == SYSTEMSTATE ECHO * %SYS% %BACKUPMODE% BACKUP STARTED AT %TME% ON %DT% & ECHO * TO %DESTINATION%\%BACKUPFILE% & ECHO %SYS% %BACKUPMODE% BACKUP STARTED AT %TME% ON %DT% TO %DESTINATION%\%BACKUPFILE% >> %BATCHLOG%
IF %BACKUPMODE% == DIFFERENTIAL ECHO * %SYS% %BACKUPMODE% BACKUP OF %SOURCE% STARTED AT %TME% ON %DT% & ECHO * TO %DESTINATION%\%BACKUPFILE% & ECHO %SYS% %BACKUPMODE% BACKUP OF %SOURCE% STARTED AT %TME% ON %DT% TO %DESTINATION%\%BACKUPFILE% >> %BATCHLOG%
IF %BACKUPMODE% == NORMAL ECHO * %SYS% %BACKUPMODE% BACKUP OF %SOURCE% STARTED AT %TME% ON %DT% & ECHO * TO %DESTINATION%\%BACKUPFILE% & ECHO %SYS% %BACKUPMODE% BACKUP OF %SOURCE% STARTED AT %TME% ON %DT% TO %DESTINATION%\%BACKUPFILE% >> %BATCHLOG%

::START BACKUP DEPENDING ON BACKUPMODE
IF %BACKUPMODE% == SYSTEMSTATE ntbackup.exe backup systemstate /n "%sys%-Backup" /d "%sys%-Backup" /v:no /r:no /rs:no /hc:off /j "%sys%-Backup" /l:s /f "%destination%\%BACKUPFILE%" >> %BATCHLOG%
IF %BACKUPMODE% == DIFFERENTIAL ntbackup.exe backup "%source%" /n "%sys%-Backup" /d "%sys%-Backup" /v:no /r:no /rs:no /hc:off /m differential /j "%sys%-Backup" /l:s /f "%destination%\%BACKUPFILE%" >> %BATCHLOG%
IF %BACKUPMODE% == NORMAL ntbackup.exe backup "%source%" /n "%sys%-Backup" /d "%sys%-Backup" /v:no /r:no /rs:no /hc:off /m normal /j "%sys%-Backup" /l:s /f "%destination%\%BACKUPFILE%" >> %BATCHLOG%

::ARCHIVE LOGFILES
::CLEAR TIME VARIABLES AND SET ANEW
SET HR=
SET MIN=
SET TM=
SET TME=
FOR /F "TOKENS=1,2 DELIMS=:" %%A IN ('TIME /T') DO (
SET HR=%%A
SET MIN=%%B
SET TM=%%A%%B
SET TME=%%A:%%B
)
::GET NTBACKUP LOGFILE NAME
FOR /F "TOKENS=1,2,4 DELIMS= " %%A IN ('dir %LOGPATH% *.log ^| find "%dt%" ^| find "%tme%"') DO SET NTBLOG=%%C >NUL
ECHO NTBACKUP LOGFILE FOT THIS JOB: %LOGPATH%\%NTBLOG% >> %BATCHLOG%
::COPY AND RENAME THE NTBACKUP LOGFILE
IF EXIST %LOGPATH%\%NTBLOG% copy %LOGPATH%\%NTBLOG% %ARCHIVEPATH% /Y >NUL
IF EXIST %ARCHIVEPATH%\%NTBLOG% ren %ARCHIVEPATH%\%NTBLOG% "%NTBARCLOG%"
::CHECK LOGFILE FOR BACKUP SUCCESS
SET ERRORLEVEL=
find /I "BACKUP COMPLETED" "%ARCHIVEPATH%\%NTBARCLOG%" >NUL
GOTO %ERRORLEVEL%
:0
SET SUCCESS=1
ECHO * BACKUP COMPLETED SUCCESSFULLY & ECHO BACKUP COMPLETED SUCCESSFULLY >> %BATCHLOG%
GOTO SUCCESSBACKUP
:1
SET SUCCESS=0
ECHO * BACKUP FAILED & ECHO BACKUP FAILED >> %BATCHLOG%
GOTO FAILBACKUP

:SUCCESSBACKUP
::SEND EMAIL WITH NTBACKUP LOGFILE ATTACHED
IF %SUCCESS% == 1 %MAILPATH% /smtpserver %SMTPSERVER% /to %TO% /from %FROM% /subject %BACKUPMODE% Backup %NOTE% Successful /body %BACKUPMODE% Backup %NOTE% Successful /attachment %ARCHIVEPATH%\%NTBARCLOG% >> %BATCHLOG%

::CLEANUP PREVIOUS BACKUPFILES
::IF NO PREVIOUS BACKUPFILE EXISTS, SKIP TO CLEANIN LOGFILES
IF NOT EXIST %DESTINATION%\*%BACKUPMODE%_%NOTE%* ECHO NO PREVIOUS BACKUPFILE(S) FOR %BACKUPMODE%_%NOTE% EXIST >> %BATCHLOG% & GOTO CLLOG
::IF PREVIOUS BACKUPFILE(S) EXIST, DELETE ALL PREVIOUS BACKUPFILES. FILTERED BY VARIABLES %BACKUPMODE% & %NOTE%
::THE FOLLOWING STATEMENT IS COMPLETELY DEPENDEND ON THE MAKEUP OF THE VARIABLE %BACKUPFILE%
IF %NOTE% == note %DAPATH% %DESTINATION%\*%BACKUPMODE%_%NOTE%* 60 >> %BATCHLOG% & GOTO CLLOG
IF %BACKUPMODE% == SYSTEMSTATE %DAPATH% %DESTINATION%\*%BACKUPMODE%_%NOTE%* 60 >> %BATCHLOG% & GOTO CLLOG
::FOR /F "TOKENS=1-7 DELIMS=_-." %%A IN ('DIR %DESTINATION%\*%BACKUPMODE%_%NOTE%* /B') DO IF "%%F%%E%%D" LSS "%DTE%" SET BF=%%A_%%B_%%C_%%D-%%E-%%F.%%G & DEL %DESTINATION%\%BF% & IF ERROLEVEL NEQ 0 ECHO PREVIOUS BACKUPFILE %DESTINATION%\%BF% DELETED >> %BATCHLOG% & GOTO CLLOG
ECHO NO (PREVIOUS) BACKUPFILE(S) DELETED >> %BATCHLOG%

:CLLOG
::CLEANUP ARCHIVED LOGFILES
::IF NO PREVIOUS LOGFILE EXISTS, SKIP TO NEXT
IF NOT EXIST %DESTINATION%\*%BACKUPMODE%_%NOTE%* ECHO NO PREVIOUS LOGFILE(S) FOR %BACKUPMODE%_%NOTE% EXIST >> %BATCHLOG% & GOTO NEXT
::IF PREVIOUS LOGFILE(S) EXIST, DELETE ALL PREVIOUS LOGFILES. FILTERED BY VARIABLES %BACKUPMODE% & %NOTE%
::THE FOLLOWING STATEMENT IS DEPENDEND ON THE MAKEUP OF THE VARIABLE %BATCHLOG% & %NTBARCLOG%
IF %NOTE% == note %DAPATH% %ARCHIVEPATH%\*%BACKUPMODE%_%NOTE%* 60 >> %BATCHLOG% & GOTO NEXT
::FOR /F "TOKENS=1-5 DELIMS=_-." %%A IN ('DIR %ARCHIVEPATH%\*%BACKUPMODE%_%NOTE%* /B') DO IF "%%F%%E%%D" LSS "%DTE%" SET LF=%%A_%%B_%%C-%%D-%%E_*.* & DEL %ARCHIVEPATH%\%LF% & IF ERROLEVEL NEQ 0 ECHO PREVIOUS LOGFILE(S) %ARCHIVEPATH%\%LF% DELETED >> %BATCHLOG% & GOTO NEXT
ECHO NO (PREVIOUS) LOGFILE(S) DELETED >> %BATCHLOG% & GOTO NEXT

:FAILBACKUP
IF %SUCCESS% == 0 %MAILPATH% /smtpserver %SMTPSERVER% /to %TO% /from %FROM% /subject %BACKUPMODE% Backup %NOTE% Failed /body %BACKUPMODE% Backup %NOTE% Failed /attachment %ARCHIVEPATH%\%NTBARCLOG% >> "%BATCHLOG%"

:NEXT
IF EXIST "%ARCHIVEPATH%\%NTBARCLOG%" ECHO NTBACKUP.EXE .LOG FILE ARCHIVED AT "%ARCHIVEPATH%\%NTBARCLOG%" >> %BATCHLOG%
ECHO BATCHLOG .TXT FILE ARCHIVED AT "%BATCHLOG%" >> %BATCHLOG%

GOTO END

:NOPARAMS
ECHO ********************************************************
ECHO THIS FILE MUST BE CONFIGURED BY OPENING IT IN NOTEPAD.
ECHO SETTING THE VARIABLES IN THE VARIABLES SECTION.
ECHO TO RUN THE FILE AFTER YOU CONFIGURE IT,
ECHO YOU MUST SPECIFY A BACKUP TYPE AS THE FIRST PARAMETER.
ECHO THE FIRST PARAMETER SHOULD BE IN lowercase.
ECHO SOURCE AND NOTE CAN BE PASSED FROM THE COMMAND LINE,
ECHO READ THE VARIABLES SECTION OF THIS FILE
ECHO.
ECHO EXAMPLE 1:
ECHO MYBACKUP systemstate [source] [note]
ECHO.
ECHO EXAMPLE 2:
ECHO MYBACKUP normal [source] [note]
ECHO.
ECHO EXAMPLE 3:
ECHO MYBACKUP differential [source] [note]
ECHO.
ECHO ********************************************************
PAUSE

:END
CLS
EXIT
Source and support files can be downloaded here

GameGuard working on Windows 7 European Aion Beta Access (Fix for Error: 0)

Did some digging because my stupid do nothing game guard client for Aion would just crash with error message 0 a few seconds after starting. Deleting the GameGuard folder does not seem to matter.

Apparently you can use the .des files from the Chinese version over top of yours and it works fine.

Found the solution at the bottom of this thread.

But, because you don't need to go chasing links... here's the .des file in the form of a .rar as linked on the forums (no registration required to download for this site!).
CLICK HERE

Open up this folder for 64bit:
C:\Program Files (x86)\NCSoft\Aion\bin32
For 32bit, AION will just BSOD in Win7 right now, I'm assuming this is going to be fixed by Aion later.

Then rename GameGuard.des to GameGuard.bullshit and copy in the one from the rar supplied here. Presto, it works! You're Welcome!

If you appreciate, please drop a comment thanks!
Oh.. and if you can't extract it, download winrar from www.rarlabs.com

(written from windows live writer)

</p>

Fetch the local weather and optionally put it on the footer of your motd

I wanted to put my local weather on the bottom of my /etc/motd - because i'm dorky like that.

I found a bunch of random ways to fetch the weather forecast in plain text... and a lot of complicated grep/sed/curl commands to get there. I wanted something as simple as possible.

So first, the shell script to get the weather in plain text...

echo " Weather for Palmyra, Virginia:"
lynx --dump "http://mobile.weather.gov/port_mp_ns.php?CityName=Palmyra&site=AKQ&State=VA&warnzone=VAZ048/" | head -n 7 | tail -n 1 | sed -e 's/\ //g' | sed -e 's/°/°/'

The echo is just for looks, obviously. You'll need to change it to whatever you want it to be headed up by... or remove it completely.

The first lynx command fetches the website how a browser sees it.
The second head takes only the top portion of that resulting website.
The third command, tail, only takes the bottom line of the resulting head display.
The fourth sed command takes all the spaces out.
The fifth command gets rid of some weird character interpritation from lynx and makes it nicely have a Fahrenheit symbol.

You'll need to get your own weather url and replace mine with it by entering your zip code here: http://mobile.weather.gov/.

Okay, so now we have a way to get the local weather. I actually setup an alias in .bashrc for this, too.

Next, we have the script that puts this at the bottom of your /etc/motd file. That script, in my case, assumes that you have sudo without entering your password:


cat ~/sh/backup/motd.backup > ~/motd.new
sh ~/sh/weather.sh >> ~/motd.new
echo "" >> ~/motd.new
sudo mv ~/motd.new /etc/motd

The first line copies my standard motd message into place within what will be the new motd file. This makes sure we maintain the same static message at the top. You'll have to update that line and point it to a basic motd message file that serves as a template.
The second line, runs my weather script, which you'll need to update and point to where yours is.
The third adds a blank line to the bottom of the new motd file, for better formatting.
The last of the 3 lines moves the new motd into place on top of your current.

Then, I scheduled that last script to run each hour in my user's crontab with the command 'crontab -e':
0 * * * * ~/sh/generateMOTD.sh

Adobe's AC_FL_RunContent and Lightbox 2 can't work at the same time in IE 7

I don't know much javascript, but from what I can see... using both the AC_FL_RunContent.js script from Adobe for embedding flash objects and Lightbox 2 from http://www.huddletogether.com/projects/lightbox2/ won't work at the same time in IE7.

It looks like IE7's fault - like normal - because Chrome and FireFox do it right. Taking out either lightbox's javascript or Adobe's makes it work... but no combination of the two I see does.

Anyway, If i have both of these javascript files in the header of any web page, I get the following:

Line: 3975
Char: 9
Error: Object doesn't support this property or method
Clode: 0

The image then comes up as if it were linked to directly.

Total crap!!! Anyone else getting this out there?

Quickly see how much space each object in your current dir is using via command line in linux

EDIT: Apparently this is done with du -sh * for all files, du -sh */ for all directories. As a side note, you can use c in there to calculate the total size of all listed items. Thanks Yan Morin!

Sounds kinda stupid, but I was unable to find a trigger with du or dh to show the size of the directories in my current directory. If i did du -h it showed me the size of the file, but also showed me the size of every file within. I just wanted a high level overview of the total size of each folder I had in front of me.

I noticed du -h DirName showed the size of the dir or file on the last line....

A quick for loop solved the problem! Maybe it will be useful for someone else, too!

for d in *; do du -h "$d" | tail -n 1; done

Find an IP address and tons more in Apache logs with Regular Expression and egrep or grep

I wanted to look through my apache logs and pull out a list of IPs. I wanted to use SSH. I am kinda new to regular expression so it took me 3 tries, but I ended up with this big string of junk to do just that. The -o tag makes it only spit out the IP out of each line while the rest will grab any grouping of numbers separated by 4 periods.

egrep -o ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+

the ^ means beginning of line
[0-9] means any number, 1-10
the + means one or more times in a row
the \ means literally interpret the next character

Then i went a step further and wanted to figure out how many UNIQUE users I had visiting for a particular access log.

cat access.log.0 | egrep -o ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+ | sort | uniq -c | sort
note: uniq -u -c seemed to leave duplicates... this worked a little slower, but reliably. I'm probably doing something wrong there.

Okay so now the NEXT step! Lets do a reverse lookup on all those IPs to see some hostnames... why not?!?!?! We're going crazy here.

for ip in `cat access.log.0 | egrep -o ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+ | sort | uniq`; do dig -x $ip | grep PTR | egrep -v ^\; >> hosts.txt; done

The for loop above lists all the reverse PTR records for every unique IP in the entire log file. The output is dumped to hosts.txt. I had to do a little hackery and exclude lines starting with ; because i was getting duplicates, but it worked. Phew, that command took awhile. I hope my shared web host doesn't mind about 4,000 reverse DNS lookups.

So what will he do next?! Well, to avoid running that big command ever again, we dumped the results to hosts.txt. Try doing a grep on that for .gov, or something like that. That would look like this:

grep \.gov\.$ hosts.txt

That should roughly be all the government addresses that hit my web server.
Then you could count how many that is, too:

grep \.gov\.$ hosts.txt | uniq | wc -l

Nice!