Post on 11-Feb-2017
transcript
API DESIGNBEST PRACTICES
FROM A HACKER’S VIEW
• Overview
• Stories
• CrawlallprojectsandbidsfromFreelancer.com
• Crawl6billionflightticketpricefromatravelwebsite
• Summary
MONOLITHICAPP
• Hidesysteminformationinsideapp• Nointernalsyscallisexposedtooutside
MICROSERVICEAPP
• Hackersknow your systembetter
• Servicecallsareexposedtouser
• RESTful APIasstandard, easy to guess
• Needtoconsidersecurity between every
service
TRIDITIONAL MODERN
XPATH
WEB PAGE API
Pure Data
STORY 1CRAWL FREELANCER.COM
FREELANCER.COM
8MProject Information
Bid Information
• Reputation and price, which is the most important factor for a success bid?• How can I get most chance to be awarded when bidding for Australia employer?
• Should Iputalowestpriceorshould Idomoreprojecttoearnreputation
HOW CAN I GET THE INFORMATION AS FAST AS POSSIBLE?
https://www.freelancer.com/projects/Javascript/Web-Page-Scraper/
• NeedaHTMLparserandjavascript executor
• Heavyworkforbothcpu andbandwidth
• Noteasytoiteratethroughalltheprojects
TIP: MOST OF THE TIMEMOBILE SITE IS MUCH EASIER
TOGETINFORMATION
https://m.freelancer.com/projects/Javascript/Web-Page-Scraper/#info
RESTFUL APIS
https://www.freelancer.com/api/projects/0.1/projects/Javascript%2FWeb-Page-Scraper%2F/?compact=true&full_description=true&job_details=true&location_details=true&selected_bids=true&upgrade_details=true&user_avatar=true&user_details=true&user_employer_reputation=true&user_reputation=true&user_status=true
https://www.freelancer.com/api/projects/0.1/projects/9844976/bids/?compact=true&limit=20&offset=0&reputation=true&user_avatar=true&user_details=true
https://www.freelancer.com/api/projects/0.1/projects/9844976/?compact=true&full_description=true&job_details=true&location_details=true&selected_bids=true&upgrade_details=true&user_avatar=true&user_details=true&user_employer_reputation=true&user_reputation=true&user_status=true
https://www.freelancer.com/api/projects/0.1/projects/${id}/?compact=true&full_description=true&job_details=true&location_details=true&selected_bids=true&upgrade_details=true&user_avatar=true&user_details=true&user_employer_reputation=true&user_reputation=true&user_status=true
API Rate Limit1000 / HOUR
8M/1k=8kHOUR=333DAYS
172.246.149.100216.219.130.17195.227.99.19717.124.253.149137.238.189.20734.155.214.35
• Number of threads depends on how many proxies you have• Https proxy is hard to find• Proxiesareunstable• Proxies will beusedoutquickly• High costifyoubuyproxies
WORKAROUND : USE HIGH ANONYMOUS PROXY
160.124.89.7113.193.36.236182.3.152.4485.72.136.122……
• Loads of IPs, canbe changed every 10s• Highquality socks proxiesacrosstheworld• Able to use docker to start 10 tor clients in 1 minute
WORKAROUND: USE TOR NETWORK
USINGTHESEHACKSIMANAGEDTOGETALLTHEPROJECTSANDBIDSIN10DAYS
USEASINGLEDIGITALOCEAN5$SERVER
WHAT DO I LEARN?
APIRateLimitation MobileAPI
Easytoguess filters PredicableURL
https://www.freelancer.com/api/projects/0.1/projects/Javascript%2FWeb-Page-Scraper%2F/?compact=true&full_description=true&job_details=true&location_details=true&selected_bids=true&upgrade_details=true&user_avatar=true&user_details=true&user_employer_reputation=true&user_reputation=true&user_status=true
Information leak
HOW CANWE FIX THEM?
ONLYSUPPLYINFORMATIONCLIENTNEEDS
MAKESUREURLISNOTPREDICTABLE
https://www.freelancer.com/api/projects/0.1/projects/UUID/?compact=true&full_description=true&job_details=true&location_details=true&selected_bids=true&upgrade_details=true&user_avatar=true&user_details=true&user_employer_reputation=true&user_reputation=true&user_status=true
REDUCEANONYMOUSNETWORKATTACK
• IfyourcustomersareinAUonly,restrictaccesswhenIPaddressisoutsideAU
• Setdifferentlimitationbasedonlocation
• 1k/hAPIusage • 100/hAPIusage• Captchatoverifyhuman
STORY 2LEARN FROM CAWLING FLIGHT TICKET PRICE
Howmany days ahead do I need to get a cheapest price?Ineedtocrawlasmanyflightticketpriceandanalysis.
FIND API FROMMOBILE PAGE
data=%7B%22searchType%22%3A%……
useNative=true&ttid=201300@travel_h5_3.1.0&appKey=12574478
t=1426062775998&sign=3feb52aed67967a2c47aa7a2b9f2a417Ifyouaccessthesameurl toreproduceAPIcalls,itwillafter10seconds
ANALYSEAPI
• Parameters inside data parameter:
• Fixed parameter:
• Sign
HOWCANWEGENRATEAVALIDAPICALL?
FIND TRIGGER POINT
Search source code to find API endpoint
REFORMATSOURCECODE
• Reformat code to get readablesource code
• Help to set breakpoint
FINDAPIURLGENERATOR
• Trace down the code to find out howto generate the url
FIND OUT TOKEN GENERATION ALGORITHM
• Set breakpoint andwatch variables to find out the secret
WHAT DO I LEARN?
• Use time token to generate dynamic urls
• Use parameter sign token to verify parameter
• PreventrepeatAPIcalls
• JS obfuscated code is easy to hack
SUMMARY
• Makesureurl isnotpredictable
• Onlysupplyinformationclientneeds
• Reduceanonymousnetworkattack• By different strategy to different location
• Use time token to generate dynamic url
• Use sign to verify request is valid
THANK YOU
Github: derekhe