If httpd fullstatus isn’t available, you can use this to enumerate the current working directories of httpd processes:

for i in `ps -elf | grep http | awk '{print $4}' | sort | uniq`; do ls -la /proc/$i/cwd ; done | awk '{print $11}'| grep -Ev ^$ | sort | uniq -c | sort -n

If you do not have mytop, mysqladmin, or access to the mysql console, you can use this to enumerate the file descriptors of the mysqld process:

ls -al /proc/`pidof mysqld`/fd/ | awk -F\> '{print $2}' | awk -F/ '{print $5}' | sort | uniq -c | sort -n

If you do not catch them quickly enough after spawning, malicious processes can reparent themselves under init so their cwd is reset. You can look in /tmp, /var/tmp, and /dev/shm for any clues, as well as reference the /proc/$PID timestamps in logs. You can always strace the suspicious process and see what files it has open with lsof and netstat.

To identify malicious files in a suspicious directory:

1. Scan their directory using maldet, which should finds most newer PHP and Perl shells. It likely won’t find older or custom payloads.

2. Scan their directory with clamav if there is time to spare. For our purpose there is fingerprint overlap with maldet but clamav has a larger database and may catch something.

3. Use the following for an audit against common shells and malicious files:

If they have gigs of files they may be a file dump. If they have only a few kilobytes it may be a new fraudulent account.

du -sh

Custom 404 redirects are not very common and are usually only found in the webroot’s .htaccess. A common attack is creating .htaccess files which 404 redirect to malicious files (usually a PHP shell with all-numeric filename) which connect to a remote server and open a shell.

find . -name .htaccess -print0 | xargs -0 fgrep ErrorDocument\ 404

These are search strings for common PHP shells. Note that intruders may use incorrect extensions to confuse you and prepend image file headers so that the malicious file is recognized as an image by file.

find . -iname \*php -print0 | xargs -0 grep -E 'c99_|c99shell|999sh|fx29_|fx29shell|r57_|r57shell|aHR0cDov'

Base64-encoded content should be scrutinized as a potential payload. Intruders will copy system file headers/templates from your software’s (eg: WordPress) installation and encode (gzinflate, base64, rot13) them to confuse you on whether they’re malicious code or legitimate footers which sometimes contain images.

find . -iname \*php -print0 | xargs -0 fgrep base64_

Files with 777 permissions are a security concern and should be investigated.

find . -perm 777 -type f

find . -perm 777 -type d

Files which are 000′d may have been disabled by the user or an administrator due to suspicious behavior.

find . -perm 000

Files owned by root hopefully are from an administrator forgetting to assign them proper ownership and not a privilege escalation.

find . -user root

Files owned by the http server may indicate access was gained through the customer’s software.

find . -user nobody -o -user apache

Files modified recently are of primary concern. For a thorough check run a second pass for ctime, as intruders are able to set their mtimes to past datetimes.

find . -mtime -1

find . -mtime +1 -mtime -10

find . -ctime -1

find . -ctime +1 -ctime -10

You can supplement your investigation by parsing the access-log of the suspected target for suspicious POST’s:

Referrers: cut -d " " -f11 access_log | sort | uniq -c | sort -n

Visitors: cut -d " " -f1 access_log | sort | uniq -c | sort -n

POSTs: fgrep POST access_log | fgrep -v " 404 " | cut -d " " -f7 | sort | uniq -c | sort -n

The error_log will have stderr output for injections and subsequent attempts to retrieve payloads from other servers, often using wget or curl.

grep -E 'wget|curl' error_log

Once you have filenames and their checksums and IP addresses, you can search other servers and their logs.

0. Build screen from source to fix the slowdown when scrolling in a vertical split.

1. This is my .screenrc with keyboard shortcuts and a nifty status bar.  –help?

Control + Up:  previous split screen

Control + Down: next split screen

Control + Left:  previous screen window

Control + Right: next screen window

F1: kill current tab

F2: write paste buffer to file

F3: split horizontally

F4: split vertically

F5: remove single split

F6: remove all splits

F7: urlview pulls URL's from your current output, very useful

F8: new tab

F9: resize split +1 line

F10: resize split -1 line

2. Install screen_ssh.sh so your window title automatically renames to your ssh destination.  To have the title revert after closing your shell follow this.

3. color = readability.  Install color wrapper to colorize your terminal output.  Alias man to use most as its pager.  Use the following for a colorful prompt with $? in $2:

00:43:18 0 user@hostname:~/$

\[\033[1;36m\]\t\[\033[0m\] `LastExitValue=$?; if [ $LastExitValue = 0 ]; then echo \[\033[1\;32m\]$LastExitValue\[\033[0m\]; else echo \[\033[1\;31m\]$LastExitValue\[\033[0m\]; fi` \[\033[1;34m\]\u@\h\[\033[0m\]:\[\033[1;33m\]\w\[\033[0m\]\[\033[1;32m\]\$\[\033[0m\]

To remove color use:  s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g

4. Yakuake is the best terminal emulator. Configure these shortcuts:

Shift + Left: previous shell

Shift + Right: next shell

F11: full screen

F12: drop down