To locate a suspicious process’s (httpd, sendmail, perl) current working directory look in /proc/$PID/cwd

If httpd fullstatus isn’t available, you can use this to enumerate the current working directories of httpd processes:

for i in `ps -elf | grep http | awk '{print $4}' | sort | uniq`; do ls -la /proc/$i/cwd ; done | awk '{print $11}'| grep -Ev ^$ | sort | uniq -c | sort -n

If you do not have mytop, mysqladmin, or access to the mysql console, you can use this to enumerate the file descriptors of the mysqld process:

ls -al /proc/`pidof mysqld`/fd/ | awk -F\> '{print $2}' | awk -F/ '{print $5}' | sort | uniq -c | sort -n

If you do not catch them quickly enough after spawning, malicious processes can reparent themselves under init so their cwd is reset. You can look in /tmp, /var/tmp, and /dev/shm for any clues, as well as reference the /proc/$PID timestamps in logs. You can always strace the suspicious process and see what files it has open with lsof and netstat.

To identify malicious files in a suspicious directory:

1. Scan their directory using maldet, which should finds most newer PHP and Perl shells. It likely won’t find older or custom payloads.

2. Scan their directory with clamav if there is time to spare. For our purpose there is fingerprint overlap with maldet but clamav has a larger database and may catch something.

3. Use the following for an audit against common shells and malicious files:

If they have gigs of files they may be a file dump. If they have only a few kilobytes it may be a new fraudulent account.

du -sh

Custom 404 redirects are not very common and are usually only found in the webroot’s .htaccess. A common attack is creating .htaccess files which 404 redirect to malicious files (usually a PHP shell with all-numeric filename) which connect to a remote server and open a shell.

find . -name .htaccess -print0 | xargs -0 fgrep ErrorDocument\ 404

These are search strings for common PHP shells. Note that intruders may use incorrect extensions to confuse you and prepend image file headers so that the malicious file is recognized as an image by file.

find . -iname \*php -print0 | xargs -0 grep -E 'c99_|c99shell|999sh|fx29_|fx29shell|r57_|r57shell|aHR0cDov'

Base64-encoded content should be scrutinized as a potential payload. Intruders will copy system file headers/templates from your software’s (eg: WordPress) installation and encode (gzinflate, base64, rot13) them to confuse you on whether they’re malicious code or legitimate footers which sometimes contain images.

find . -iname \*php -print0 | xargs -0 fgrep base64_

Files with 777 permissions are a security concern and should be investigated.

find . -perm 777 -type f

find . -perm 777 -type d

Files which are 000′d may have been disabled by the user or an administrator due to suspicious behavior.

find . -perm 000

Files owned by root hopefully are from an administrator forgetting to assign them proper ownership and not a privilege escalation.

find . -user root

Files owned by the http server may indicate access was gained through the customer’s software.

find . -user nobody -o -user apache

Files modified recently are of primary concern. For a thorough check run a second pass for ctime, as intruders are able to set their mtimes to past datetimes.

find . -mtime -1

find . -mtime +1 -mtime -10

find . -ctime -1

find . -ctime +1 -ctime -10

You can supplement your investigation by parsing the access-log of the suspected target for suspicious POST’s:

Referrers: cut -d " " -f11 access_log | sort | uniq -c | sort -n

Visitors: cut -d " " -f1 access_log | sort | uniq -c | sort -n

POSTs: fgrep POST access_log | fgrep -v " 404 " | cut -d " " -f7 | sort | uniq -c | sort -n

The error_log will have stderr output for injections and subsequent attempts to retrieve payloads from other servers, often using wget or curl.

grep -E 'wget|curl' error_log

Once you have filenames and their checksums and IP addresses, you can search other servers and their logs.