ABSTRACTContent delivery networks (CDNs) commonly use DNS to mapend-users to the best edge servers. A recently proposed EDNS0-Client-Subnet (ECS) extension allows recursive resolvers to includeend-user subnet information in DNS queries, so that authoritativeDNS servers, especially those belonging to CDNs, could use thisinformation to improve user mapping. In this paper, we study theECS behavior of ECS-enabled recursive resolvers from the perspec-tives of the opposite sides of a DNS interaction, the authoritativeDNS servers of a major CDN and a busy DNS resolution service.We find a range of erroneous (i.e., deviating from the protocol spec-ification) and detrimental (even if compliant) behaviors that mayunnecessarily erode client privacy, reduce the effectiveness of DNScaching, diminish ECS benefits, and in some cases turn ECS fromfacilitator into an obstacle to authoritative DNS servers’ ability tooptimize user-to-edge-server mappings.
ACM Reference Format:Rami Al-Dalky, Michael Rabinovich, and Kyle Schomp. 2019. A Look atthe ECS Behavior of DNS Resolvers. In Internet Measurement Conference(IMC ’19), October 21–23, 2019, Amsterdam, Netherlands. ACM, New York,NY, USA, 14 pages. https://doi.org/10.1145/3355369.3355586
1 INTRODUCTIONAside from resolving hostnames to IP addresses, the domain namesystem (DNS) has been widely used by major content deliverynetworks (CDNs) [2, 7, 17] for assigning end-users to the nearestedge servers. Since the only topological information available toauthoritative nameservers in a basic DNS query is the source IPaddress (which belongs to the recursive DNS resolver rather thanthe end-user), many CDNs utilize the resolver IP address to selectan edge server for a given query, using the resolver as a proxy forthe end-user location. However, the increasing number of publicDNS services in the last few years that are less likely to be a goodapproximation for end-user location compared to ISP-providedresolvers results in an increase in suboptimal user-to-edge servermapping [1, 6, 19].
To combat this issue, an extension to the DNS has been pro-posed called EDNS-Client-Subnet (ECS) [9] which allows recursiveresolvers to convey to authoritative nameservers a prefix of theIP address of the client (loosely referred to as the client’s subnet)requesting resolution service from the recursive resolver. Thus,instead of using the recursive resolver’s IP address, the authorita-tive DNS server can use the client subnet information to provideedge server selection tailored to the ultimate client rather than therecursive resolver.
Recursive resolvers add the ECS option to DNS queries fillingin a prefix of the client’s address (the RFC recommends 24-bitsfor IPv4) and setting the source prefix length field to the numberof bits added. When the authoritative nameserver that supportsECS receives a query with an ECS option, it can optionally usethe client subnet information to tailor a response. Since the sourceprefix length may not be at the appropriate granularity level, theauthoritative nameserver returns the scope prefix length in an ECSoption included in the DNS response, indicating the range of clientIP addresses for which this answer is appropriate. For example, if arecursive resolver sends queries with source prefix length 24 butthe authoritative answer is appropriate for all clients within theencompassing /16 prefix, the authoritative nameserver will set thescope prefix length to 16. The recursive resolver upon receiving aresponse with the ECS option, caches the records and should returnthem for the duration of the TTL to any clients covered by theprefix at the scope prefix length number of bits.
ECS was proposed in 2012 and standardized in 2016 [9], yet littleis known about its adoption by recursive resolvers. At the sametime, while ECS adoption may appear low in absolute numbers (asprojected in the RFC and seen from the numbers in this paper), thisis an important technology enabling third-party DNS resolutionservices to interact efficiently with CDNs and other distributedcontent delivery platforms, already in use by many prominentDNS and CDN providers. In this work, we investigate the ECS-related behavior of recursive resolvers and make the followingcontributions:• We analyze ECS deployment by recursive resolvers via passiveobservations from a large CDN perspective and “in the wild”,i.e., among resolvers found through active scans. Passive ob-servation discovers many more ECS resolvers while activelydiscovered resolvers allow a closer study of their behavior.
• We study probing strategies used by recursive resolvers to de-cide whether to include the ECS option in their queries. Weobserve some probing behaviors that lead to unnecessary pri-vacy leakage and some causing suboptimal CDN edge serverselections. We offer a recommendation that would fulfill theprobing purpose without these drawbacks.
• We consider the caching behavior of recursive resolvers thatsupport ECS. We find various deviations from the expected
IMC ’19, October 21–23, 2019, Amsterdam, Netherlands Rami Al-Dalky, Michael Rabinovich, and Kyle Schomp
caching behavior, ranging from not following the RFC recom-mendation on exposing no more that 24 bits of the client subnetinformation to completely ignoring the scope restrictions whenreusing cached DNS responses.
• We study the impact of ECS on DNS caching. Namely, theamount of cached state at the resolver and the cache hit rate.Since ECS limits reuse of cached records across client subnets,one can expect some negative impact on cache size and hit rate.However, we find these effects can be quite significant. We showthat the cache size needed by the resolvers to store results ofqueries to a CDN can increase by an order of magnitude, and abusy resolver may need around 4 times the cache size to storeresponses in all interactions that involve ECS, while its cachehit rates for these interactions declines to less than half of whatit would be without ECS.
• We offer several cautionary tales of real-life setups that maynot just diminish or even negate the ECS benefits but turn ECSinto an obstacle to effective user-to-edge-server mappings.
• Tangentially, we are, to our knowledge, the first to discoverand provide an initial glimpse into hidden resolvers betweenforwarders and recursive resolvers [20, 22, 27], so called be-cause they were previously believed to be unobservable. Wefind that some hidden resolvers are far away from the clients,and in fact not infrequently (in 8% of the cases we observe)are farther away from the clients than the recursive resolvers.Since many resolvers derive ECS prefixes from the IP address ofthe immediate source of the queries received, hidden resolversare one example of a setup where ECS may be detrimental foruser-to-edge-server mapping by CDNs.The rest of the paper is organized as follows. We briefly discuss
related work in Section 2, the terminology we use in Section 3, andour datasets that we use in this work in Section 4. Then, we expandon each aforementioned contribution in Sections 5 - 8. Finally, weoffer concluding thoughts in Section 10.
2 RELATEDWORKSeveral studies have investigated ECS from different perspectives.In [4, 30], the authors show how ECS can be used from a singlevantage point as a measurement tool to study the deployment ofECS adopters (such as Google). In our work, we look at the ECS-related behavior of recursive resolvers and some ECS implicationsfor DNS caching. Chen et al. [6] study the impact of enabling ECSon mapping end-users to edge-server from Akamai’s perspective.The authors show that ECS deployment has improved the laten-cies between end-users that use public resolvers and their edgeservers by 50%, at the cost of 8 times increase in the number of DNSqueries their authoritative DNS servers receive from those publicresolvers. In our work, we consider another ECS overhead, namely,the recursive resolvers’ cache size, and identify situations whereECS can lead to suboptimal mapping. Calder et al.[5] study ECSadoption by recursive resolvers from a cloud provider perspective,while our study focuses on the ECS behavior by those recursive re-solvers that did adopt ECS. Kintis et al. [21] discuss the privacy andsecurity of ECS including the erosion of privacy for Internet usersand facilitation of running selective cache-poisoning attacks to
target specific subnets/regions. Vries et al. [12] use 2.5 years of pas-sive ECS-enabled queries to study Google Public DNS. The authorsshow that DNS traffic to Google Public DNS is frequently routed todatacenters outside the country even though a local datacenter isavailable in country. Moreover, they uncover a new privacy risk ofECS when mail servers are configured to use Google Public DNS,involving mail servers revealing the domains of their email sendersthrough Google to authoritative DNS servers. While some privacyleakage is an inherent cost of ECS, we show that some resolverserode user privacy unnecessarily via probing strategies they use todetect ECS-enabled authoritative nameservers.
3 TERMINOLOGYIn this paper, we refer to DNS queries that include an ECS optionas “ECS queries”, and to DNS responses with an ECS option as“ECS responses”. We call resolvers that directly interact with theauthoritative nameservers “egress resolvers” or simply “recursiveresolvers”. The resolvers that receive queries directly from user de-vices and end hosts are called “ingress resolvers”. Ingress resolvesoften act as forwarders: they take a query from an end host andforward it to an egress resolver, then forward a response from theegress resolver back to the end host. There are also many ingressresolvers that act as egress resolvers as well, i.e., they both takequeries from the end hosts and interact with authoritative name-servers directly. Further, in some deployments an ingress resolverroutes its queries to an egress resolver through (a chain of) inter-mediaries called “hidden resolvers”.
4 DATASETSIn this study, we consider four datasets: (i) one day of DNS trafficlogs from a major CDN’s authoritative DNS servers (CDN dataset),(ii) queries arriving at our own experimental authoritative name-server as the result of a full DNS scan of the IPv4 address space withDNS queries for names within our own DNS zone (Scan dataset),(iii) traces of traffic from a major public DNS service to a majorCDN’s authoritative nameservers (Public Resolver/CDN dataset),and (iv) logs of all DNS traffic to/from a single busy recursiveresolver (All-Names Resolver dataset).CDN Dataset: The CDN dataset is derived from aggregated DNSquery logs from all authoritative nameservers of a major CDN. TheCDN uses whitelisting in handling ECS queries, which means thatthe CDN only considers the ECS option in queries from, and in-cludes the ECS option in its responses to, pre-approved (“whitelisted”)resolvers. Queries from non-whitelisted resolvers are handled asif the CDN did not adopt ECS: ECS options in these queries aresilently ignored and responses provide no indication that ECS issupported. Whitelisting a resolver involves human negotiation andbusiness agreement between the resolver’s operator and the CDN.We collect one day of logs, November 6 2018, which contain DNSqueries from 3,741,983 resolvers. Of these over 3.7M resolvers, only7737 resolvers may be ECS-enabled (i.e., send at least one ECSquery during that day), including 3590 whitelisted and 4147 non-whitelisted resolvers. For the purpose of this study, we extract theDNS interactions involving ECS-enabled non-whitelisted resolvers.The resulting dataset contains 1.5B queries, including 847M withan ECS option. The 4147 different IP addresses (4002 IPv4 and 145
A Look at the ECS Behavior of DNS Resolvers IMC ’19, October 21–23, 2019, Amsterdam, Netherlands
IPv6) belong to 83 different ASes. 3067 IP addresses belong to asingle AS, which we refer to as the “dominant AS” below.Scan Dataset:We collect this dataset by scanning the IPv4 addressspace with DNS queries for hostnames from our own domain andrecord the queries that arrive at our experimental authoritativenameserver. Following a technique from [10], we use hostnamesthat encode the IPv4 address being probed, which allows our au-thoritative nameserver to associate the discovered ingress resolverswith the egress resolvers they employ. We use queries withoutan ECS option because most of the open DNS resolvers are homewifi-routers [27] with simplified DNS functionality and may be un-able to handle, or blindly forward, an unknown option. Either casewould lead to inaccurate detection of ECS support. We configureour authoritative nameserver to respond to queries carrying an ECSoption with scope length L = S −4, where S is the ECS source prefixlength from the query. Responses to non-ECS queries carry no ECSoption (per the RFC). We reiterate that, while this scan leveragesopen ingress resolvers, most of these resolvers are actually simpleforwarders that send incoming queries to their recursive resolversfor processing, often the default ISP-provided resolvers, which aretypically closed to external queries. These closed resolvers are in-cluded in our analysis.
The scan was conducted from a machine on our campus networkusing a Python script that issued 25K DNS queries per second (therate restriction imposed by the university’s network administrators).We also ran the PF_RING variant of tcpdump on both the scanningmachine and the authoritative nameserver to capture the DNSqueries and responses. The scan ran from Feb 22 to Feb 23, 2019and took 42 hours to complete. We identified 2.743M open ingressresolvers in our scan, which is roughly in line with the numberreported in [28]1. Of this number, the queries from 1.53M openingress resolvers arrived at our authoritative nameserver with anECS option added, indicating they use ECS-enabled egress resolvers.These 1.53M ingress resolvers are spread across 7.9K autonomoussystems (ASes) and 195 countries, and employ 1534 egress resolverIP addresses that support ECS. Google Public DNS is the largestcontributor of recursive resolvers accounting for 1256 IP addresses.The remaining 278 recursive resolver IP addresses belong to 45 ASes,with Chinese ISPs being the largest contributor responsible for 19ASes. The dataset is available on request. Although the apparentskew in ECS-enabled egress resolvers toward Google and Chinamight give an impression of a bias in the dataset, this is not the casehere. We note that the dominant AS from the CDN dataset is alsoChinese. Because of the global reach of the major CDN that deliversa large portion of all Web traffic, we are confident that what weobserve in both the CDN and Scan datasets is that two operators– Google and one Chinese operator – dominate the ECS-adoptingrecursive resolvers. In other words, this is the state of ECS adoption,not a skew in either dataset. The apparent skew in ECS-enabledegress resolvers toward China confirms the similar finding in [5].Public Resolver/CDN Dataset: This dataset contains DNS trafficlogs from a major CDN’s authoritative nameservers for ECS queriesarriving from a major public DNS service that is whitelisted for ECS
1The number of open resolvers on the Internet is decreasing. The openresolverproject.org used to show a longitudinal graph but the site is no longer up. Still, we note that[27] found over 30M open resolvers in 2012, vs. around 10M observed by [3] in 2016,vs. only 2.74M we observe now.
support by the CDN. The dataset covers 3 hours during the busytime of the day in the Americas, between 00:00:00 - 03:00:00 UTCon March 1, 2019, and includes 3.8B A/AAAA queries from 2370different resolver IP addresses. All queries carry the ECS optionand all responses include a non-zero scope prefix length.All-NamesResolver Dataset: This dataset is DNS traffic collectedfrom a busy recursive resolver instance of an anycast DNS resolu-tion service. The service accepts client queries at anycasted front-ends, which forward these queries to the egress resolvers whileadding an ECS option carrying clients’ source IP addresses. Anegress resolver issues queries to authoritative DNS servers and re-turns the responses, along with the authoritative ECS scope, to thefront-ends. The All-Names Resolver dataset contains all queries andresponses exchanged between front-ends and one egress resolverwhere the responses include an ECS option with non-zero scopeprefix length. The unique feature of the dataset is that it containsboth the client IP address and the authoritative ECS scope. Thedataset, collected for 24 hours starting from 09:00 UTC on March27, 2019, contains 11.1M A/AAAA queries and responses comingfrom 76.2K different client IP addresses (37.4K IPv4 and 38.8K IPv6addresses) which belong to 15.1K different client subnets (12.3K /24IPv4 client subnets and 2.8K /48 IPv6 client subnets). The queriesare for 134925 unique hostnames from 19014 unique second-leveldomains2.
5 DISCOVERING ECS-ENABLED RESOLVERSOur CDN and Scan datasets represent two orthogonal methodolo-gies to discover ECS-enabled resolvers - using passive observationsfrom a busy authoritative nameserver perspective and using activemeasurements. Both can miss resolvers but an argument can bemade that both may catch a large number of them. Indeed, thepassive method may miss a resolver that never needs to resolve adomain from the authoritative nameserver’s zone during the obser-vation period, but for a busy authoritative nameserver (such as amajor CDN), one can assume that, given a sufficient observationtime, many resolvers will have at least some of its clients accessat least some URLs accelerated by the CDN. Similarly, the activemethod will miss resolvers that are not accessible through any openingress resolvers, but it is conceivable that – given millions of openingress resolvers – many large resolvers would be used by at leastone open ingress resolver.
The number of non-Google ECS-supporting egress resolvers thatwe found through our scan is lower than the number in the CDNdataset (278 vs 4147). Moreover, of the resolvers discovered by thescan, most (234 out of 278) are also present in the passive logs.Clearly, between the above reasons for missing ECS resolvers, theactive method is impacted more. In addition, there can be severalfurther reasons for this difference. First, some recursive resolvers(like OpenDNS [14]) maintain a whitelist of domains/nameserversto which it sends ECS-enabled queries. The CDN domains are morelikely to be whitelisted by such resolvers while our experimentaldomain is likely not as we did not submit any whitelisting requests.Second, our authoritative nameserver is IPv4-only and would missIPv6 recursive resolvers (there are 145 IPv6 ECS-enabled resolvers
2Second-level domains, or SLDs, in DNS denote the two most senior labels in a host-name, e.g., cnn.com, or ac.uk.
IMC ’19, October 21–23, 2019, Amsterdam, Netherlands Rami Al-Dalky, Michael Rabinovich, and Kyle Schomp
in the CDN dataset). Third, one can imagine that a recursive re-solver might only send ECS queries on behalf of whitelisted ingressresolvers (e.g., from those users who explicitly opt-in, since usingECS entails privacy implications). Our scan may miss such a re-solver. Overall, between a large-scale passive observation and alarge-scale active measurement, the former is more effective at dis-covering resolvers of interest. With the decreasing number of openresolvers, the utility of the active measurements will decline further.Still, although incomplete, the active dataset allows us to to studymore closely ECS-related aspects of resolver behavior, using therecursive resolvers accessible externally (either directly or throughopen ingress resolvers) as an example.
6 ECS BEHAVIOR OF RESOLVERSThis section considers strategies the ECS-enabled resolvers usefor deciding whether to include an ECS option in their queriesto authoritative nameservers, the ECS source prefix length theyuse when they do send the option, and how they use the scopeprefix length returned by the authoritative nameservers in control-ling caching. The first two aspects affect the degree to which theresolvers reveal clients address information and the third aspectassesses if the resolvers implement ECS cache control prescribedby the authoritative nameservers correctly.
6.1 ECS Probing StrategiesRFC 7871 recommends that recursive resolvers not send ECS op-tions blindly with every DNS query. Indeed, sending an ECS optionto an authoritative nameserver that does not support ECS need-lessly reduces privacy. Additionally, some authoritative nameserverimplementations that do not support the ECS option have var-ious bugs that result in dropped queries, and nameservers thatdo not support the EDNS0 mechanism in general [11] will returnFORMERR responses. RFC 7871 specifies two strategies that a re-cursive resolver can employ to decide whether to include ECS datain queries to a given authoritative nameserver. The first strategyis probing for ECS support: a recursive resolver can periodicallysend an ECS query (e.g., hourly or daily) and omit the ECS data forsubsequent queries if the response returned by the authoritativenameserver doesn’t have a valid ECS option. The second strategyis to maintain a whitelist of authoritative nameservers or zones towhich the resolver will send the option. The second strategy canreduce the complexity associated with probing and improve privacyas recursive resolvers would send client subnet information only toauthoritative nameservers that are known to use this informationin generating a response. However, maintaining a whitelist is notscalable as it requires out-of-band interaction between the resolversand authoritative nameserver, and the whitelist can quickly becomestale as the authoritative nameservers supporting ECS may changewith time.
Since the major CDN’s authoritative nameservers only respondto ECS-enabled queries from whitelisted resolvers, it appears as anon-ECS supporting site to non-whitelisted resolvers included inthe CDN dataset. Thus, the CDN dataset reflects the strategies ofthese resolvers in probing authoritative nameservers whose ECSsupport is either unknown or which previously were found to
not support ECS. We identify four distinct behavior patterns forincluding the ECS option.
First, 3382 out of 4147 resolvers in the CDN dataset send 100%of their A and AAAA queries with an ECS option, including allthe resolvers from the dominant AS and 287 out of 1,080 resolversfrom non-dominant ASes. These resolvers either maintain a per-authoritative nameserver whitelist and have whitelisted the CDN’sauthoritative nameservers, or send the ECS option indiscriminatelyfor all A/AAAA queries to all authoritative nameservers. We areunable to distinguish between these possibilities.
Second, 258 resolvers send ECS queries consistently but only forspecific hostnames. Furthermore, they send repeated queries forthese hostnames within the TTL periods, evenwhen the TTLs of theresponses are very short, e.g., 20 seconds. Given past findings thatresolvers are unlikely to evict records quickly or further shortensuch short TTLs [27], it appears that these resolvers disable orlimit caching for these hostnames. We have no speculation for thereasons behind this behavior except that perhaps these resolversselect certain hostnames for ECS probing and conduct the probingregardless of the cache hits.
Third, we find 32 resolvers that send ECS queries at the intervalof a multiple of 30 minutes, and non-ECS queries otherwise. More-over, we observe that all of the probes are for a single query stringand carry the loopback IP address as client subnet information.The use of the loopback address for ECS probing is an interestingapproach as it avoids revealing any real client information unneces-sarily, before the ECS support by a given authoritative nameserveris determined. However, as discussed in Section 8.1, this may causemajor confusion, and very poor edge server selection, at the author-itative nameservers (assuming the probing queries are triggeredby real client queries). A better approach to accomplish the samegoal is to use the resolver’s own address in the ECS option. TheRFC already suggests this as an option for queries that arrive atthe resolver with source prefix length 0. We recommend to usethis approach for probing and make it mandatory. Further, sincea loopback could be technically viewed as one the resolver’s ownaddresses, the RFC should clarify that by “own address” it meansthe public IP address the resolver uses to send the query.
Fourth, 88 recursive resolvers consistently send ECS-enabledqueries for specific hostnames as in the second category above, butnot within a short (i.e., at most one minute) time window from aprevious query for the same name. We speculate these resolversinclude ECS in their queries for specific hostnames on a cache miss.
The remaining 387 recursive resolvers send ECS-enabled queriesfor a subset of hostnames and on a subset of queries for thosehostnames. From our dataset, we are unable to discern a pattern totheir probing behavior.
Finally, we have observed that some resolvers send client subnetinformation unnecessarily, for queries that are unlikely to be an-swered based on ECS information, such as NS queries, which theRFC recommends to be answered with zero scope. We were curiousif any resolvers violate the RFC outright by sending ECS queries toroot DNS servers. We analyze 24-hours of logs from one instanceof the A-root server using DITL data from April 2018 (the latestDITL data available [13]) and do find 15 resolvers exhibiting thiserroneous behavior.
A Look at the ECS Behavior of DNS Resolvers IMC ’19, October 21–23, 2019, Amsterdam, Netherlands
Source Prefix # of Resolvers # of ResolversLength (Scan dataset) (CDN dataset)
18 321 6022 8 1924 1384 757
24,25,32/jammed last byte 124,32/jammed last byte 3
25 1 125,32/jammed last byte 7832/jammed last byte 130 3002
Table 1: ECS source prefix lengths. The rows sum up tomorethan the total number of recursive resolvers because someresolvers convey both IPv4 and IPv6 source prefixes.
6.2 Prefix Source LengthsA resolver needs to make a policy decision on the length of the ECSprefix to be conveyed to authoritative nameserver. The longer theprefix the higher its utility for user mapping but the greater theprivacy erosion. RFC 7871 [9] recommends that recursive resolverstruncate IPv4 addresses to at most 24 bits and IPv6 addresses to atmost 56 bits in the source prefix to maintain client privacy.
We observe that the ECS source prefix length sent by recursiveresolvers varies. Table 1 shows the number of recursive resolversthat sent specific source prefix lengths in the Scan and CDN datasets.While all recursive resolvers in the Scan dataset sent only a sin-gle source prefix length per IP version, we observe that in theCDN dataset 82 recursive resolvers sent multiple IPv4 source prefixlengths, and 3 recursive resolvers sent multiple IPv6 source prefixlengths in different queries. We include in the table the combina-tions of source prefix lengths observed from these resolvers. A rowfor a given combination lists the number of resolvers that sent everyprefix length in the combination. In the Scan dataset, we find thevast majority of the resolvers follow the RFC recommendation andsend source prefix length 24. However, a possible sense of content-ment can be deceptive as these are mostly Google resolvers. Almosthalf of non-Google ECS resolvers do not indicate any truncation ofuser IP addresses at all. The vast majority of these (118 out of the130) are in Chinese ASes. The CDN dataset shows similar resultsas it does not include Google’s queries to inflate the number of /24prefixes. Moreover, the resolvers sending /32 prefixes include all3067 resolvers from the dominant AS, which also happens to befrom China. It appears that this aspect of ECS behavior is especiallycommon among Chinese ISPs.
At a first glance, the resolvers sending /32 prefixes appear to ig-nore the above RFC recommendation on client privacy preservation.However, we find that all 130 such resolvers in the Scan dataset, and3084 out of 3323 such resolvers in the CDN dataset (including 2912from the dominant AS), convey the client IP addresses with thelower byte of the address set to a fixed value, mostly 0x01 and some0x00. Thus, these resolvers effectively reveal only 24 senior bits ofthe client address, even though the source prefix length is 32. Still,this is an incorrect implementation of the RFC recommendationand provides misleading information to authoritative DNS.
In addition, there is a sizable number of resolvers in the CDNdataset that submit 25-bit prefixes, violating the RFC-recommended24-bit maximum. Because BGP routers typically limit advertisedprefix lengths to at most 24 bits [29], such specific ECS prefixesadd little benefit and unnecessarily erode client privacy. We con-sider implications of using fewer than 24 bits in ECS prefixes inSection 8.3.
Finally, we note that some recursive resolvers sent IPv6 prefixesin the ECS option and in many cases those prefixes included morethan 48 bits of the client IP address. Research [25] shows that thismay not be sufficient to anonymize many IPv6 clients dependingupon the address assignment practices in use. Thus, these recur-sive resolvers may be eroding client privacy as well even if theyfollow the RFC guidance (which recommends 56-bit prefixes or,presumably, less).
6.3 Caching BehaviorIn this section, we investigate how ECS-enabled recursive resolvershandle caching of DNS records in the presence of the ECS optionand, specifically, whether these resolvers honor cache restrictionsimposed by ECS scope in authoritative responses. Not followingthese cache restrictions can interfere with the authoritative name-servers’ traffic engineering policies that ECS is supposed to facili-tate.
6.3.1 Methodology. Our general approach involves delivering pairsof queries for our own domain, with different client subnet infor-mation, to the ECS-enabled recursive resolvers, and returning re-sponses with specially selected scope prefix length values. The firstquery populates the resolver’s cache, and the second tests whetherthe resolver treats it as a hit or a miss, thus assessing if resolverrespects the caching restrictions imposed by the ECS scope fromthe response. The method used to deliver the pair of queries to therecursive resolver depends upon its accessibility. We are able tostudy the caching behavior of various recursive resolvers using thefollowing techniques:
For recursive resolvers egressing resolution paths that accept ar-bitrary ECS prefixes we submit with our queries, we deliver querieswith our chosen ECS source prefixes either directly (if the recur-sive resolver is open) or through the forwarders that use them. Wefind 32 recursive resolvers to be amenable to this measurement,including 24 open resolvers with which we could interact directlyand 8 closed resolvers accessible through a forwarder that passedalong our arbitrary ECS prefixes, which were then accepted by theresolvers and used in the resolvers’ own queries to our authoritativenameserver.
Our next technique leverages two open forwarders that use thesame recursive resolver to deliver our queries. Specifically, for thepurpose of our experiment, we need the two forwarders to havedifferent /24 prefixes but share the same /16 prefix. We find 164resolvers with appropriate forwarders and thus amenable to thistechnique.
Finally, when the resolution path includes hidden resolvers (seeSection 8.2 for details on our hidden resolver detection), we attemptto use two open forwarders that are behind two hidden resolversthat are in different /24 prefixes and share a /16 prefix to deliver
IMC ’19, October 21–23, 2019, Amsterdam, Netherlands Rami Al-Dalky, Michael Rabinovich, and Kyle Schomp
our queries. There are 7 resolvers with suitable hidden resolversfor this technique.
In summary, across the above cases, we are able to deliver twosuccessive queries to 203 resolvers such that the resolver treats thetwo queries as if they arrived from clients in different /24 but thesame /16 address blocks. In addition, for the 32 resolvers amenableto our first measurement technique, we can explore their handlingof ECS prefixes of arbitrary lengths. One of these 32 resolvers turnedout to be in Google’s IP address space, although it is not part ofGoogle Public DNS according to their published list of IP blocksused for this purpose or observed in the CDN dataset – and was nolonger accessible by the time we wanted to notify Google3.
Overall, of 278 non-Google egress resolvers in the Scan dataset,there were 76 recursive resolvers that we could not study, including64 that did not have appropriate forwarders or hidden resolversand 12 for which forwarders became unavailable in the time lag(two weeks) between the scan and the experiment in this section.
6.3.2 Results. Using the above techniques, we conduct two experi-ments. First, we deliver pairs of successive queries with different /24but the same /16 ECS prefixes for our own hostname to each recur-sive resolver, and when they arrive at our authoritative nameserver,return the scopes /24, /16, and /0. To avoid cached records from onetrial affecting other trials, we use a unique hostname for each trial.If the recursive resolver honors scope restrictions as prescribed inthe RFC, it will not use the cached record from the first query toanswer the second query for scope /24 but will reuse the cachedrecord for scope /16 and /0. Thus, with a compliant resolver, we ex-pect our authoritative nameserver to see both queries for scope/24and only the first query for scopes /16 and /0. Second, for the 32resolvers which accept an arbitrary ECS prefix from our queries,we explore how these resolvers handle ECS prefixes and scopesthat are longer and shorter than 24.
Based on these experiments, we classify the resolvers into oneof the following categories:• We find 76 resolvers with correct behavior: they honor thescope from the authoritative answers and never submit ECSprefixes longer than 24 to authoritative nameserver, even whenthe resolvers accept arbitrary ECS prefixes from the clients, andeven when these prefixes are longer than /24 (in which casethey truncate the excessive bits).This is proper behavior as, according to the RFC recommenda-tion, recursive resolvers should not convey more than 24 bitsin ECS prefixes to preserve the client privacy. The resolvers inthis category include 9 recursive resolvers that accept arbitraryECS prefixes. For these resolvers, we are able to also test thatthey enforce an RFC stipulation that the scope length in theresponses cannot exceed the source prefix length in the query.
3At the time of the study, Google’s anycast front-end did not accept ECS optionsfrom incoming external queries and instead derived ECS prefix from the IP addressof the sender of the query. However, during the preparation of the camera version,we noticed that Google has returned to its previous behavior reported in [30] in itsinteractions with non-whitelisted authoritative nameservers, namely that it passes theECS submitted by the external queries along to our authoritative nameserver. Thus,we could use our first technique to directly measure the behavior of at least someof Google’s egress resolvers and confirm that they exhibits the same ECS cachingbehavior as the resolver reported in the paper, i.e., that it shows the correct ECSbehavior.
All 9 resolvers correctly apply scope length 24 to control thereuse of their cached records, even when we return a greaterscope length. Finally, these 9 resolvers include the one Googleresolver that we can study.
• On the other hand, we find 103 recursive resolvers, or overhalf of all recursive resolvers we could study, that don’t controlcaching based on scope at all: they reuse cached responses irre-spective of the clients’ addresses, as if they did not understandECS. This is particularly interesting because either (i) recursiveresolver adds ECS to queries but then ignores it or (ii) somehidden resolver adds ECS and the resolver does not understandit at all.
• Among the 32 resolvers willing to accept arbitrary ECS prefixesfrom the queries sent to them, we find 15 open resolvers accept-ing ECS prefixes longer than /24, and caching the responsesbased on correspondingly longer scopes. This behavior runscounter to the RFC recommendation on client privacy.
• Conversely, another 8 resolvers among the 32 resolvers accept-ing arbitrary ECS prefixes cache responses based on subnetgranularity coarser than /24. Specifically, these resolvers imposethe maximum cacheable prefix length of 22. When receivingqueries with source prefix length longer than 22, the recursiveresolver only conveys the first 22 bits to our authoritative name-server. In addition, they impose scope length 22 to control thereuse of their cached records even whenwe return greater scopelength. Such behavior can be lead to highly suboptimal user-to-edge-server mapping with some CDNs (as we will discuss inSection 8.3).
• Finally, we find one misconfigured resolver that sends an ECSprefix from a private address block (10.0.0.0/8) even after receiv-ing answers from our authoritative nameserver indicating itsECS support, and even when relaying queries from forwardersthat share with the resolver the /24 address prefix (so there areno privacy issues). Moreover, from coordinated queries to twoforwarders using this resolver, we observe that this resolverdoes not handle the ECS scope properly, as it does not cache(or does not reuse) responses with scope prefix length zero.
7 ECS IMPACT ON CACHINGECS facilitates fine-grained server selection by DNS, with responsestailored to different clients. In doing so, it limits the resolver’s abilityto reuse a cached DNS record to serve multiple clients and thus canincrease the cache size needed to avoid evictions (since multiplerecords for the same question can co-exist in cache if the questioncame from different client IP prefixes) and reduce the cache hitrate. This section quantifies these tendencies. Using the PublicResolver/CDN dataset, we examine the ECS impact on cache size ofa large number of resolvers4 but only considering accesses of theCDN-accelerated content. Using the All-Names Resolver dataset,we measure the caching impact of ECS (on both the cache size andhit rate) due to all DNS queries/responses that carry ECS but onlyconsidering a single resolver.
4As a reminder, while the dataset represents queries from a single major public resolu-tion service, there are a large number of individual egress resolvers in the dataset. Weconsider cache blow up per each such resolver.
A Look at the ECS Behavior of DNS Resolvers IMC ’19, October 21–23, 2019, Amsterdam, Netherlands
0 2 4 6 8 10 12 14 16 18 20 22 24 26 280
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Cache Blow Factor
CD
F
20 Sec. TTL40 Sec. TTL60 Sec. TTL
Figure 1: Distribution of the blow-up factor in the cache sizefor various TTL values.
7.1 Resolver Cache SizeAs mentioned, we expect that when a recursive resolver enablessupport for ECS, the size of its cache state will grow. We assessby how much the cache grows using two datasets that offer com-plementary perspectives. We use the Public Resolver/CDN dataset,which records interactions between a major resolution service anda major CDN, to assess the increase in resolver cache sizes due toaccesses of the CDN-accelerated Internet content. We complementthese results with the analysis of the All-Names Resolver dataset,which allows us to quantify the increase of a resolver cache sizeneeded to store responses in all interactions that involve ECS, in-cluding all authoritative DNS servers that support ECS. We usetrace-driven simulations to conduct these analyses. In our simu-lations, we assume that the recursive resolvers adhere to the TTLvalue returned by the authoritative nameserver (i.e., retain therecords for no longer than the TTL) and do not evict records fromthe cache before they expire.
We begin with the Public Resolver/CDN dataset. The public DNSservice in question employs a large number of individual egressresolvers. Our logs contain 2370 different recursive resolver IPaddresses with varying traffic volume per IP address. We assumeeach resolver maintains its own isolated cache (no cache sharingamong the resolvers). The CDN and the public DNS service involvedboth support ECS in their DNS interactions, and the dataset includesthe ECS options (both source prefixes of the queries and scope prefixlengths of the responses) exchanged in the DNS interactions.
To simulate the cache of the resolvers without ECS, we replay thelogs disregarding the ECS information. In other words, we assumethat once a given resolver records the answer for a given query, anysubsequent queries would be answered from the cache, irrespectiveof the client, for the duration of the TTL, with only the initialanswer occupying cache space. To simulate the cache with ECS, wereplay the logs while obeying the ECS source and scope prefixeslisted. The authoritative nameservers always return a 20 secondTTL in responses and our simulated recursive resolvers removerecords from cache at that TTL, i.e., 20 seconds after insertion.
For each recursive resolver, we calculate the cache blow-up fac-tor as the maximum cache size with ECS at any time during the
10 20 30 40 50 60 70 80 90 1001.5
2
2.5
3
3.5
4
4.5
Percentage of Client IP Addresses
Cac
he B
low
-up
Fact
or
Figure 2: Cache size blow-up factor for random fractions ofthe client IP addresses.
simulation divided by the maximum cache size without ECS. Fig-ure 1 shows the CDF of the blow-up factor for all the recursiveresolvers in the “20 Sec. TTL” line. We find that the maximumcache size blow-up factor is 15.95 and 50% of the resolvers have ablow-up factor of more than 4, meaning that at peak cache usagethe resolvers held 4x more records from the CDN with ECS thanthey would have held without ECS.
Next, we note that TTL also has an impact on cache size. TheCDN records in our simulation have a TTL of 20 seconds but manyDNS records in the wild have longer TTLs. We repeat the simula-tions using TTLs of 40 and 60 seconds in the remaining lines ofFigure 1. The maximum cache size blow-up factor grows to 23.68with 40 second TTLs and 29.85 with 60 second TTLs. We anticipatethat the cache size blow-up factor will continue to increase withTTL values greater than 60. Supporting ECS increases query vol-ume for both recursive resolvers and authoritative nameservers [6].Increasing TTL values to reduce the load on authoritative name-servers, however, will further exacerbate the impact of ECS on therecursive resolvers’ cache size.
The above results demonstrate the ECS impact on recursiveresolver cache size due to accesses to content delivered by a singleCDN. However, recursive resolvers resolve hostnames for manydomains, supported by different CDNs with varying ECS and TTLbehavior. Next, we study the impact on the cache size of a singlerecursive resolver considering its interactions with all authoritativenameservers with which the resolver exchanges ECS information.We use the logs from the All-Names Resolver dataset to run trace-driven simulations of the resolver cache similar to above, usingreal-life authoritative ECS and TTL information as they appear inthe log. We find that the cache size blow-up factor for this resolveris 4.3. Note that the cache size blow-up factor is calculated onlyon records that carry ECS. The recursive resolver can send querieswithout ECS and/or receive responses without ECS, thus the blow-up factor on the overall resolver cache may be smaller than whatwe report here.
As part of an anycast DNS resolution service, the resolver inthe All-Names Resolver dataset receives DNS queries from a large
IMC ’19, October 21–23, 2019, Amsterdam, Netherlands Rami Al-Dalky, Michael Rabinovich, and Kyle Schomp
10 20 30 40 50 60 70 80 90 10020
30
40
50
60
70
80
Percentage of Client IP Addresses
Cac
he H
it R
ate
(%)
No ECSWith ECS
Figure 3: Cache hit rate for various random fraction of theclient IP addresses.
distributed set of clients. As noted in Section 4, it has 76.2K clientsin 12.3K /24 IPv4 and 2.8K /48 IPv6 client subnets. We would liketo assess the blow-up factor for recursive resolvers with smallerclient populations. To this end, we simulate a public resolver with asmaller client population by randomly sampling client IP addressesin the dataset. Using a random subset of clients reflects publicresolver usage, where clients individually decide to use this resolverregardless of their subnet.
We re-run our simulations only using queries from the randomsamples of client IP addresses in the dataset, where we vary thefraction of all clients in the sample. For each fraction of clients, werun simulations with three different random samples of client IPaddresses and report the average values.
Figure 2 shows the cache size blow-up factor for different frac-tions of the client IP addresses. As we can see from the figure, thereis a clear relation between the blow-up factor in cache size andthe increase in the client population because having a diverse setof clients results in caching several copies of the response for thesame question if the clients’ IP addresses are not covered by thesame ECS scope. In fact, the curve does not appear to flatten as thefraction of clients reaches 100%. Thus, busier resolvers, with moreclients, than the resolver from the All-Names Resolver dataset arelikely to experience even larger blow-up factors.
In summary, large TTL values and a diverse client populationwould result in a large increase of the cache size recursive resolverswould need if they were to preserve low rates of premature cacheevictions observed recently [27]. Thus, supporting ECS entails re-source cost for the operators who need to weigh these costs againstthe benefits to clients’ quality of experience in deciding on ECSadoption.
7.2 Cache Hit RateWe now use the All-Names Resolver dataset to study the impact ofECS on DNS cache hit rate. While this dataset only produces a pointassessment, it still reflects a concrete observation of an operationalbusy resolver. Figure 3 shows the hit rate that a resolver wouldexperience when serving a random fraction of clients of a givensize without ECS (e.g., ignoring any ECS scope from authoritative
answers) and when obeying the ECS scope restrictions from the log.The figure was obtained by trace-driven simulation of the loggedqueries while obeying all authoritative TTLs from the log. Eachdata point reflects the average values of three runs, using differentseeds for random client selection.
The results show a drop in hit rate due to ECS by more thanhalf, for all client populations. For the full client population, thehit rate declines from around 76% to around 30%. Moreover, the hitrate in the presence of ECS increases much slower than withoutECS as client population grows. The latter fact can be explained bycontradictory effects of the client population growth on the hit rateunder ECS. On one hand, as the number of clients grows, popularhostnames are more widely shared, leading to higher hit rate (thisis the effect captured by the growing hit rate without ECS). On theother hand, the larger client population is likely to be fragmentedamong more /24 blocks, which would depress the hit rate in thepresence of ECS. It appears that the two tendencies largely canceleach other.
8 ECS PITFALLSECS was proposed to improve the performance of end-users byenabling authoritative nameservers to tailor a response based onthe topological location of the client’s subnet. However, we findseveral types of real-life resolver setups that interact with ECS inways that diminish or negate its benefits and in fact can turn ECSfrom a facilitator to an obstacle to proximity-based server selection.
8.1 Using Non-Routable ECS PrefixesFrom our Scan dataset, we observe that a fraction of ECS-enabledqueries arrive at our authoritative nameserver with non-routableIP addresses in client subnet information. Specifically, we observequeries with the loopback and self-assigned address prefixes (mostcommonly 127.0.0.1/32, 127.0.0.0/24, and 169.254.252.0/24). Therewere 33 resolvers from 6 ASes with this behavior, including 27resolvers from a single AS5. We suspected this may not be part ofthe probing behavior we observed in the CDN dataset (Section 6.1)because we observe these ECS prefixes repeatedly from the sameresolvers despite our authoritative nameserver responding to eachquery with the ECS option. After investigating ECS functionalityof several resolvers, we found (and confirmed with the PowerDNScommunity) that the PowerDNS recursor software [24] can exhibitthis behavior under some scenarios. Specifically, a private IP ad-dress or loopback address is sent in an ECS query if (i) the resolverreceives a query from a client with source prefix length of 0 (inwhich case, the RFC stipulates the resolver must either not includeany ECS option at all or include its own address information, al-though in view of our findings the RFC should be more specific andsay explicitly this must be the public address used by the resolver tosend the query) or (ii) the resolver receives (ii) the resolver receivesa query from a client whose IP address is not whitelisted for ECSusage.
We investigate whether sending such information to authorita-tive nameservers can cause a confusion at the server side. We send
5Since, as discussed later, we found this behavior to be problematic, we notified theadministrators of the AS containing the majority of these resolvers of the issue.
A Look at the ECS Behavior of DNS Resolvers IMC ’19, October 21–23, 2019, Amsterdam, Netherlands
ECS Prefix First answer RTT LocationNone 172.217.9.46 35 ms Chicago
/24 of src addr 172.217.8.206 35 ms Chicago127.0.0.1/32 172.217.168.46 155 ms Switzerland127.0.0.0/24 173.194.196.136 47 ms Mountain View, CA
169.254.252.0/24 216.58.223.142 285 ms South Africa
Table 2: Authoritative responses to queries forwww.youtube.com sent from Cleveland with unroutableECS prefix (RTT is the average of 8 pings).
five queries for www.youtube.com directly to Google’s authorita-tive DNS server from a lab machine using dig, one query each usingno ECS, an ECS prefix matching the lab machine’s IP address, andthe three most common unroutable ECS prefixes mentioned above.From the responses, we harvest the returned IP addresses of edgeservers that Google maps our lab machine to with the varying ECSoption. Note that sending a single query with each variation of ECSoption is sufficient because our lab machine interacts directly withthe authoritative name server and thus the responses representthe answers that a real resolver would get when sending a similarquery, regardless of caching or any other implementation details atthe authoritative nameserver.
We receive the same set of 16 IP addresses for the first twoqueries (in different permutations) but different sets of IP addressesfor the queries with unroutable ECS prefixes, and these sets do notoverlap with the 16-address set or with each other. Thus, sendingan unroutable ECS prefix results in different answers for Google’sauthoritative DNS server than the answers one would receiveswithout ECS or with ECS matching the source IP address.
We then test if the quality of user-to-edge-server mapping isaffected. Table 2 shows the ping RTT from the lab machine usedto issue the queries (located in Cleveland, OH) to the first IP ad-dress returned by the authoritative nameserver and the locationof this IP address that we determine from the hostnames of thenearby hops from traceroutes (the traceroute was unresponsivenear the address returned for the query with the 127.0.0.0/24 prefix,and we geolocated this address using EdgeScape, a commerciallyavailable geolocation service [15], instead). The RTT values reflectthe average of 8 pings. The table shows that sending no ECS orECS matching the lab machine’s IP address results in mapping toa nearby edge server in Chicago. On the other hand, sending un-routable ECS prefixes results in mapping across the globe. Clearly,submitting unroutable ECS prefixes can result in a very substantialpenalty in mapping quality and must be viewed as a performancebug6. To avoid negative impact on their clients, the resolvers shouldeither convey its own IP address in the ECS option or omit the ECSoption instead. We also believe that the RFC should explicitly directresolvers to use their own public IP addresses, the same that is usedto send the query, when they wish to insert an ECS option withoutdisclosing client information. This would eliminate grey areas andavoid divergence in ECS implementations between resolvers andauthoritative nameservers.
6While the RFC recommends that authoritative nameservers treat unroutable prefixesin queries’ ECS options as the resolvers’ own identity, it uses the word “SHOULD”,and – as our findings show – not all servers follow this recommendation.
8.2 Using Hidden ResolversWe noticed that around half of the ECS-enabled queries in our Scandataset carried an ECS prefix that covers neither the open ingressIP address we probed nor the recursive (egress) IP address thatcommunicated with our authoritative nameserver. This suggeststhat those IP addresses in the client subnet belong to intermediaryresolvers between ingress resolvers and recursive resolvers, oftenreferred to as hidden resolvers [20, 22, 27] because they were previ-ously believed to be unobservable7. Thus, ECS can help to uncoveran additional component in the DNS ecosystem that was previouslyhidden from observation. We find 32170 different potential hiddenresolver prefixes in the Scan dataset, representing 198 countries and7.2K ASes8. Out of the 32170 prefixes, we find that 31011 prefixesare conveyed by the resolvers of the major public DNS service.
We attempt to validate that the discovered ECS prefixes cor-respond to hidden resolvers as follows. First, we exclude sourceIP spoofing as a possible confounding factor because our scannerreceived responses to most of the queries that discovered thosehidden ECS prefixes. Next, using the Public Resolver/CDN dataset,we verify that the prefixes we discover from the Scan dataset belongto actual hidden resolvers. We leverage the observation that themajor Public DNS service discards any ECS option in the incomingqueries and replaces it with the actual prefix of the query senderbefore sending the query to the major CDN authoritative name-servers9. Thus, the ECS prefixes in the Public Resolver/CDN datasetrepresent actual query senders. It turned out that we could finda vast majority of those presumed hidden ECS prefixes from thescan dataset in the Public Resolver/CDN dataset. Specifically, wefind that out of the 31011 potential hidden prefixes conveyed bythe resolvers of the major public DNS service, 28892 prefixes arepresent in the Public Resolver/CDN dataset. Similarly, 815 prefixesout of the 1159 potential hidden prefixes conveyed by the otherresolvers (i.g., not belonging to the major public DNS service) arepresent in the Public Resolver/CDN dataset. Overall, 29707 out of32170 hidden prefixes in the Scan dataset are found in the PublicResolver/CDN dataset and thus belong to actual resolvers.
Regardless, even if some of these ECS prefixes do not belongto physical hidden resolvers but represent pre-configured ECS set-ting in the egress resolver, an authoritative nameserver will usethis information when mapping the user to the nearest contentserver. Therefore, we investigate whether these prefixes provide agood approximation of the location of the open forwarders whosequeries contained those prefixes10. Since a forwarder may use sev-eral hidden resolvers and a hidden resolver may use several re-cursive resolvers, we consider unique combinations of (open for-warder, hidden resolver, recursive resolver) in the Scan dataset.
7We caution against inferring the general prevalence of hidden resolvers from theirprevalence in the Scan dataset as the dataset can be biased from being limited to openingress resolvers.8Since recursive resolvers report the hidden resolvers information at the /24 prefixlevel, the actual number of hidden resolvers can be greater than 321709We also note that the major public resolver exhibited the same behavior with ourexperimental authoritative nameserver at the time of the Scan experiment, i.e., it didn’taccept ECS prefixes we submitted in our queries and replaced them with the source IPaddress of the requester before sending them to our authoritative nameserver.10An orthogonal question is whether the forwarders provide good approximation forthe end-device. While outside the scope of our work, we note that [27] found evidencethat most forwarders are residential networking devices and thus typically colocatedwith end-devices.
IMC ’19, October 21–23, 2019, Amsterdam, Netherlands Rami Al-Dalky, Michael Rabinovich, and Kyle Schomp
Figure 4: Distance between forwarders and hidden and re-cursive resolvers (MP resolvers).
Using EdgeScape [15], we measure the distance between the for-warder and the hidden resolvers as well as between the forwarderand the recursive resolver. Given the prominence of the majorpublic resolver represented in the Public Resolver/CDN dataset (re-ferred to in this section as “MP resolver” for brevity), we divide theresults into two cases, studying the major public resolver (referredto in this section as “MP resolver” for brevity) separately from otherrecursive resolvers.Case 1 (MP resolvers):We find 725K unique (forwarder, hiddenresolver, recursive resolver) combinations where the recursive re-solvers belongs to MP. We observe that in 57.7K (8%) combinations,the hidden resolvers are farther away from the forwarders thanthe recursive resolvers are. This means that a CDN’s edge serverselection without ECS, using the egress resolver location, would bebased on better understanding of the client location than with ECS,using the hidden resolver location. In other words, ECS handicaps,rather than facilitates, server selection in these cases by deliveringa worse client location approximation to the authoritative DNS.
To convey the extent of this problem, Figure 4 shows a hexbinscatter plot of the geographic distances from the forwarder to thehidden (F-H) and recursive resolvers (F-R). Points below the diag-onal (8% of the combinations) indicate that the hidden resolver isfarther away from the forwarder than the recursive resolver. Onecan see that the difference between these distances can be great, onthe order of thousands of kilometers. For instance, in one combina-tion, we find that the distance between the forwarder and recursiveresolver is 0 km (as both are in Santiago, Chile). However, the hid-den resolver is in Italy, 12000 km away from the forwarder. Weverified the correctness of geolocation by using traceroutes to the IPaddresses of the forwarder, hidden resolver (padding the ECS prefixwith random bits), and recursive resolver. Further, we observe thissame combination of the hidden and recursive resolvers in a queryin the Public Resolver/CDN dataset. Therefore, this example repre-sents a configuration, verified to be in use, where ECS prevents theCDN from providing any meaningful user-to-edge-server proximitymapping.
Figure 5: Distance between forwarders and hidden and re-cursive resolvers (non-MP resolvers).
Further, there are 9.7K (1.3%) combinations lying on the diagonal,meaning that the forwarder is equidistant from the hidden and re-cursive resolver. In these combinations, ECS – while not hamperingauthoritative nameserver’s understanding of client location – doesnot improve its understanding. Even when ECS is still helpful in thepresence of hidden resolvers, a hidden resolver can greatly diminishECS utility. Indeed, as the figure shows, among the 90.7% of thecombinations above the diagonal (i.e., where the hidden resolver iscloser than the recursive resolver to the forwarder and thereforeimproves the approximation of the forwarder’s location), the hid-den resolver can still be thousands of kilometers away from theforwarder. ECS benefits would greatly improve if it would carrythe forwarder’s prefix rather than the hidden resolver’s. It mayappear that a way to achieve this is to adopt ECS along the entireresolution path and make all resolvers along the path copy the ECSprefix as they forward the query towards the authoritative name-server. Unfortunately, to prevent spoofing, many recursive resolvers(including the MP resolver at the time of the study) override anyECS information in received queries with the ECS prefix derivedfrom the source IP address of the immediate query sender, beforesubmitting the query to the authoritative server. We conclude thatto obtain the most benefits from ECS, the users should connectto ECS-enabled recursive resolvers directly, rather than throughintermediaries.Case 2 (Non-MP resolvers): The non-MP resolvers exhibit thesame trends. We find 217K unique combinations of (forwarder,hidden resolver, recursive resolver) with recursive resolvers that donot belong to MP Public DNS. Of this number, 17K (7.8%) have theegress resolver closer to the forwarder compared to hidden resolver,entailing a detrimental effect of ECS on user mapping, 42.3K (19.5%have the same distance from the forwarder to both hidden andrecursive resolvers (including 18.45K combinations where all threeparties are placed by EdgeScape in the same location), and theremaining 157.7K (72.7)% combinations contain a hidden resolverthat is closer to, and hence provides better approximation for thelocation of, the forwarder than the recursive resolver is. In other
A Look at the ECS Behavior of DNS Resolvers IMC ’19, October 21–23, 2019, Amsterdam, Netherlands
words, ECS improves the authoritative nameserver’s understandingof the client location in only 72.7% of the combinations, has no effecton this understanding in 19.5%, and worsens it in 7.8% of the cases.
Figures 5 shows the hexbin scatter plot for the distances betweenthe forwarders and their hidden and egress recursive resolvers. Asannotated on the figure, we observe many combinations where theforwarder and recursive are nearby while the hidden resolver is∼1000 km away. Similarly, we observe many combinations wherethe forwarder and hidden resolver are nearby while the recursiveresolver is ∼1000 km away. We find that this combinations corre-spond to recursive resolvers in Shanghai while the hidden resolverand forwarder can be in either Beijing or Shanghai (the distancebetween Beijing and Shanghai is ∼1000 km). There are also manycombinations where the distance from forwarder to recursive re-solver is ∼2000 km which we find to be forwarders in Beijing andrecursive resolvers in Guangzhou. We believe that the prominenceof these distances is mostly a product of the skew in ECS supporttowards China and the fact that Beijing, Shanghai and Guangzhouare the three largest cities in the country. The overall distances be-tween forwarders and resolvers, as well as location approximationpenalties due to hidden resolvers tend to be somewhat lower thanin the case of the MP resolver but are still substantial.
The overall take-away point from these results is that hidden re-solvers can interact with ECS to negate ECS benefits and sometimeseven turn ECS from a facilitator into a handicap for the authoritativenameservers’ ability to conduct effective traffic engineering. Oneway to avoid potential performance degradation is for the partiesinvolved (ISPs, DNS resolution service provides, and client sites) toconsider carefully the relative location of clients, hidden resolvers,and egress resolvers before adopting ECS, and either avoid usinghidden resolvers with misleading locations or avoid including ECSprefixes for queries coming from those hidden resolvers. Anotherway is to develop trust between hidden and egress resolvers (whichare sometimes operated by different organizations) so that hiddenresolvers would include ECS prefixes based on end-client subnets,and egress resolvers would pass this information (provided it comesfrom the trusted senders) to the authoritative nameservers, ratherthan replacing it with prefixes based on the sender IP addresses.
8.3 Using Improper Source Prefix LengthAs we observe in Section 6.2, different resolvers submit differentECS prefix lengths with their queries. We already discussed thatusingmore than 24 bits in the ECS prefixes counters RFC recommen-dation. We now investigate whether sending ECS source prefixeswith fewer than 24 bits would have an impact on the quality ofuser-to-server mappings for the answers returned by authoritativeDNS servers. This question was previously considered by Otto etal. through measurements embedded in an application on end-userdevices [23]. Based on the measurements of mappings by Googleauthoritative DNS, the authors concluded that increasing the prefixlength from /16 to /24 provided only small extra benefits but alsonoted that other CDNs may show different results. We considertwo major CDNs that support ECS from non-whitelisted hosts, de-noted CDN-1 and CDN-2, and observe the impact of prefix lengthis different from that reported in [23].
1 10 100 10000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Time to connect in msec (log scale)
CD
F
161718192021222324
Figure 6: Distribution of quality of mapping of a hostnameaccelerated by CDN-1.
1 10 100 10000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Time to connect in msec (log scale)
CD
F
2021222324
Figure 7: Distribution of quality of mapping of a hostnameaccelerated by CDN-2.
We use RIPE Atlas measurement platform [26], which providesprogrammatic access to over ten thousand measurement pointsthroughout the world, to conduct our study. We randomly select800 IP addresses of RIPE Atlas probes, resulting in a sample cov-ering 174 countries and 599 autonomous systems. Because Atlasdoes not support ECS queries, we use our lab machine to submitrepeated queries for two hostnames, one accelerated by each CDN,directly to their respective authoritative nameservers. However,to make the authoritative nameservers use probes’ location ratherthan ours, we include into our queries ECS options with client sub-net prefixes derived from the 800 IP addresses of our Atlas probesample. For each DNS response received, we leverage RIPE AtlasSSL measurements to perform three certificate downloads from thecorresponding Atlas probe using the first IP address in the DNSresponse, and use the median of the TCP handshake latencies asour metric for the resulting user-to-edge-server proximity.
For CDN-1, with 24-bit prefixes of our 800 probe addresses, theauthoritative nameserver returns 400 unique first IP addresses intheir responses. However, with any shorter source prefixes (westudy prefix lengths between 16-24), the total number of unique firstIP addresses returned by the authoritative server falls drastically,to between 5 and 14.
IMC ’19, October 21–23, 2019, Amsterdam, Netherlands Rami Al-Dalky, Michael Rabinovich, and Kyle Schomp
Major Public DNS Service
Client [C1]
1. ex.com
DNS Provider Authoritative Nameserver
2. ex.com
Major CDN Authoritative Nameserver
3. ex.cdn.net 4. ex.cdn.net
[E1]
5. ex.com [E1]
6. ex.com [E1]
E1 7. HTTP Request [ex.com]
8. HTTP Redirection [www. ex.com]
9. www. ex.com
14. www. ex.com [E2]
12. ex.cdn.net (C1)
13. ex.cdn.net [E2]
E2
10. www. ex.com
11. www. ex.com
[ex.cdn.net]
Figure 8: CNAME flattening when accessing customer.com.
Figure 6 shows the cumulative distribution function (CDF) of themedian TCP handshake latencies for the hostname accelerated byCDN-1, using the edge server obtained with an ECS prefix of a givenlength. The figure shows a huge degradation in CDN-1 mappinglatency when reducing source prefix length from 24 to 23, whilefurther shortening of the prefix has no visible effect. Together withthe observation on the number of edge servers produced by differentprefix lengths, it appears that CDN-1 does not use proximity-basedserver selection for prefixes shorter than /24.
Turning to CDN-2, we find that when sending source prefixlength values between 16-20 a single IP address is returned fromCDN-2 authoritative DNS server for all queries with scope prefixlength of zero. Using traceroute, we find that the IP address is inToronto, near our lab machine (located in Cleveland). Thus, CDN-2appears to ignore the ECS information when source prefix length is20 or less and use the IP address of the resolver that sends the queryas a proxy for the end-device location. However, as soon as theprefix length reaches 21 or above, CDN-2 returns 41-42 different IPaddresses. Figure 7 shows the CDF of the median TCP handshakelatencies for the hostname accelerated by CDN-2, excluding theresults for source prefix length 16-19 because they are identical tolength 2011 We can see that using source prefix length of 21 andlonger have the same quality of mapping but dropping to /20 leadsto a dramatic penalty. It appears that CDN-2 leverages ECS prefixesin its edge server selection for prefixes of at least 21 bits not forshorter prefixes.
The results in this section bring interesting questions abouthow resolvers should choose the prefix lengths when sending ECSqueries to CDNs. They could just blindly use the most specific pre-fixes recommended by the RFC (/24), but in the case of CDN-2 thiswould expose more client information than needed for proximitymapping by the CDN – sending 21 bits would suffice. However,sending any fewer than 24 bits to CDN-1 would negate any benefitsfrom ECS and then whatever client information is submitted infewer bits would still be exposed unnecessarily.
11We note a similarity of the CDFs in both CDNs.We explain it by the disproportionallyhigh representation of Europe in RIPE Atlas probes, where both CDNs have densefootprint.
Furthermore, while the RFC suggests that the resolvers utilizeknowledge of their clients to use shorter source prefixes whenall addresses covered by these prefixes are known to be in thesame location, this recommendation assumes that the authoritativenameservers would use whatever number of client subnet bits theyreceive for server selection. The results from both CDNs aboveshow this assumption is not always accurate. Both CDNs appearto stop using ECS once the source prefix length drops to a certainlimit.
On the balance, it would appear that using /24 for all ECS queriesis the most practical approach. An alternative would be to track thesource prefix lengths needed per CDN, or even per subdomain andper client address block, since a CDN can in principle use differentprefix lengths for different subdomains and clients. This can getcomplicated very quickly.
8.4 CNAME FlatteningThe DNS standard [16] does not allow CNAME records to co-existwith other record types at the same name. This poses a problem asCNAME records are a common method to onboard traffic to a CDNand content providers often desire for their content to be reachablefrom the apex of their zones, e.g., example.com, which at a mini-mum must have NS and SOA records. CNAME flattening [8, 18]has emerged to circumvent this obstacle. With CNAME flattening,when an authoritative nameserver receives a query for N1 thatit would normally redirect to a CDN by returning a CNAME N2from the CDN’s domain, the authoritative server instead resolvesN2 itself by interacting with the CDN’s authoritative DNS on thebackend, and then returns the final A/AAAA record(s) of the edgeserver(s) to the original querier. Thus, recursive resolvers query-ing the authoritative DNS server receive A/AAAA record(s) forN1, while N2 is invisible externally, outside the authoritative DNSservers of the website and the CDN involved. In this section, wedemonstrate how careless implementations of CNAME flatteningmay eliminate the ECS benefits even when ECS is supported byboth the recursive resolver and the CDN.
We use an imaginary domain “customer.com” in this discussion,as our goal is to draw attention to this pitfall and not to single
A Look at the ECS Behavior of DNS Resolvers IMC ’19, October 21–23, 2019, Amsterdam, Netherlands
out one of the websites that fell into it. However, this discussionrepresents a real website of a prominent content provider thatwe tested. The DNS zone is hosted with a major DNS providerwhile its web acceleration is provided by a major CDN. The website can be accessed either via the apex of the zone, i.e., throughURL http://customer.com (using CNAME flattening) or with wwwprepended, with URL http://www.customer.com.
We access customer.com from the Chrome browser using a majorPublic DNS service as our recursive resolver as this Public DNSand the major CDN are known to support ECS with one another.We use Wireshark to collect a packet trace while loading the page.The order of actions is depicted in Figure 8 and is as follows: (i)(Steps 1-6) The client resolves customer.com to the IP address E1of an edge server of the major CDN via CNAME flattening; (ii)(Steps 7-8) The client performs HTTP interaction with E1, receivingHTTP redirect to www.customer.com. This incurred 125 ms tocomplete TCP handshake and 650ms in total elapsed time from step1. (iv) (Steps 9-14) The client resolves www.customer.com, usingthe regular CNAME-based DNS redirection, to the IP address of adifferent edge server of the major CDN, and finally (v) (Not shownin the figure) the HTTP download of the page from the second IPaddress, taking 45ms for the TCP handshake. Note that when theclient accesses the same page using www.customer.com directly, itwould only execute phases (iv) and (v).
From the results, we can infer that the mapping of customer.comto edge server E1 is poor (likely due to the absence of ECS in theDNS transaction in steps 3-4 between the DNS provider and themajor CDN for the flattened CNAME, forcing the CDN to mapthe query based on the IP address of customer.com’s authoritativenameserver, which has no bearing on the client’s location), andHTTP redirection is used to correct the mapping. The overall Webaccess incurs 650ms penalty due to the lack of ECS on part of theresolution path.
It would appear authoritative DNS servers that implement CNAMEflattening could mitigate this issue by using ECS and passing ECSsource prefixes to the CDN when they resolve the flattened name.This would indeed help if customer.com’s authoritative nameserverreceived queries directly from clients or from clients’ nearby ISPresolvers. However, with public DNS resolvers, the queries can stillarrive from senders that are distant from the end-devices. Further-more, even if the public DNS and the CDN mutually whitelist eachother to support ECS, the problem remains unless the Public DNSalso whitelists customer.com for ECS support. In summary, fullelimination of the performance penalty due to CNAME flatteningrequires careful planning to enact pair-wise coordination of multi-ple parties: customer.com’s authoritative DNS service provider, theCDN used to accelerate customer.com’s content delivery, and anypublic DNS resolution services.
9 LIMITATIONS & FUTUREWORKIn this section, we describe topics that we did not study yet wouldcomplement our work nicely. This includes extensions of our anal-ysis as well as entirely new research questions.
Our study of source prefix lengths in Section 6.2 uses the datafrom our scan that probes each forwarder only once. Thus, the num-ber of times egress resolvers are engaged is variable and depends
on how many open forwarders share a given resolver. Further, ourauthoritative nameserver in the scan always answers ECS querieswith a deterministic scope (4 shorter than the source prefix length).It would be interesting to engage the same resolver repeatedly ina more systematic manner and explore if changing the scope inauthoritative nameserver’s responses would affect the source prefixlength of subsequent queries.
In Section 7, we study the impact ECS has on cache size for theportion of DNS responses that carry the ECS option only. To un-derstand the impact on overall cache size including DNS responsesthat do not carry the ECS option, future work should focus onthe fraction of DNS responses that carry ECS options today andattempt to predict what that fraction will be as ECS support grows.From such a study, it would be possible to predict that overall cacheblow up factor for recursive resolvers at both present levels of ECSdeployment by authoritative nameservers and future increases indeployment.
Another direction for future work is to conduct a compara-tive analysis of different whitelisted recursive resolvers as wellas whitelisted vs. non-whitelisted resolvers in terms of their com-pliance with RFC recommendations and consequences of ECS oncaching.
Our study of the ECS caching behavior of recursive resolvers inSection 6.3 includes only recursive resolvers that are discoverablethrough open forwarders in the Scan dataset and is not exhaustiveof all recursive resolvers. Other techniques for probing recursiveresolvers including Ripe Atlas [26] would complement our study,increasing overall coverage.
In Section 8.1, we report a behavior of PowerDNS that has im-plications on authoritative nameserver handling of ECS. Similarnuanced behaviors may exist in other recursive resolver software.A lab based analysis of ECS behavior in popular recursive resolversoftware could detect the PowerDNS behavior and many other sim-ilar behaviors, and would be beneficial to the developer community.
10 CONCLUSIONThis paper studies the behavior of recursive resolvers that haveadopted EDNS0-Client-Subnet (ECS) extension to the DNS protocol.ECS has been proposed to facilitate proximity-based server selec-tion by content delivery networks (CDNs), especially in the faceof increasing use of public DNS resolvers that can be far removedfrom the end-devices. Using diverse sources of data, we examineimportant aspects of ECS-related behavior and find a wide rangeof detrimental behaviors that negatively affect client privacy, ECSbenefits in improving server selection, and effectiveness of DNScaching. This shows that despite its apparent simplicity, ECS adop-tion requires careful engineering of a proper setup to get the mostbenefits from ECS and avoid harm.Acknowledgements.We thank the anonymous reviewers for ex-tensive and useful comments. We are especially grateful to ourshepherd, Tobias Fiebig, for his insightful comments and discus-sions during the final revision, which significantly improved thepaper. The work of Michael Rabinovich was supported in part byNSF through grant CNS-1647145.
IMC ’19, October 21–23, 2019, Amsterdam, Netherlands Rami Al-Dalky, Michael Rabinovich, and Kyle Schomp
REFERENCES[1] Bernhard Ager, Wolfgang Mühlbauer, Georgios Smaragdakis, and Steve Uhlig.
2010. Comparing DNS resolvers in the wild. In Proceedings of the Internet Mea-surement Conference. ACM, 15–21.
[2] Akamai 2019. Akamai Technologies, Inc. Retrieved 2019-09-07 from https://www.akamai.com/
[3] Rami Al-Dalky, Michael Rabinovich, and Mark Allman. 2018. Practical challenge-response for DNS. ACM SIGCOMM Computer Communication Review 48, 3 (2018),20–28.
[4] Matt Calder, Xun Fan, Zi Hu, Ethan Katz-Bassett, John Heidemann, and RameshGovindan. 2013. Mapping the expansion of Google’s serving infrastructure. InProceedings of the Internet Measurement Conference. ACM, 313–326.
[5] Matt Calder, Xun Fan, and Liang Zhu. 2019. A Cloud Provider’s View of EDNSClient-Subnet Adoption. In Network Traffic Measurement and Analysis Conference(TMA). IEEE, 129–136.
[6] Fangfei Chen, Ramesh K Sitaraman, and Marcelo Torres. 2015. End-User Map-ping: Next Generation Request Routing for Content Delivery. ACM SIGCOMMComputer Communication Review 45, 4 (2015), 167–181.
[7] CloudFront 2019. Amazon CloudFront. Retrieved 2019-09-07 from https://aws.amazon.com/cloudfront/
[9] C. Contavalli, W. van der Gaast, D. Lawrence, and W. Kumari. 2016. Client Subnetin DNS Queries. RFC 7871. RFC Editor. https://tools.ietf.org/html/rfc7871
[10] D. Dagon, N. Provos, C.P. Lee, and W. Lee. 2008. Corrupted DNS ResolutionPaths: The Rise of a Malicious Resolution Authority. In Network and DistributedSystem Security Symposium.
[11] J. Damas, M. Graff, and P. Vixie. 2013. Extension Mechanisms for DNS (EDNS(0)).RFC 6891. RFC Editor. https://tools.ietf.org/html/rfc6891
[12] Wouter B De Vries, Roland van Rijswijk-Deij, Pieter-Tjerk de Boer, and Aiko Pras.2018. Passive observations of a large DNS service: 2.5 years in the life of Google.In Network Traffic Measurement and Analysis Conference (TMA). IEEE, 1–8.
[13] DITL 2018. A-Root DITL Data, submitted to DNS-OARC by Verisign. https://www.dns-oarc.net/oarc/data/ditl/2018.
[15] EdgeScape 2019. Akamai EdgeScape. Retrieved 2019-09-07 from https://developer.akamai.com/edgescape
[16] R. Elz and R. Bush. 1997. Clarifications To the DNS Specification. RFC 2181.https://tools.ietf.org/html/rfc2181
[17] Fastly 2019. Fastly, Inc. Retrieved 2019-09-07 from https://www.fastly.com/[18] T. Finch, E. Hunt, P. van Dijk, and A. Eden. 2018. Address-specific DNS aliases
[19] Cheng Huang, David A Maltz, Jin Li, and Albert Greenberg. 2011. Public DNSsystem and global traffic management. In IEEE INFOCOM - The 30th Conferenceon Computer Communications. 2615–2623.
[20] Ben Jones, Nick Feamster, Vern Paxson, Nicholas Weaver, and Mark Allman. 2016.Detecting DNS root manipulation. In International Conference on Passive andActive Network Measurement. Springer, 276–288.
[21] Panagiotis Kintis, Yacin Nadji, David Dagon, Michael Farrell, and Manos Anton-akakis. 2016. Understanding the Privacy Implications of ECS. In InternationalConference on Detection of Intrusions and Malware, and Vulnerability Assessment.343–353.
[22] D. Leonard and D. Loguinov. 2008. Turbo King: Framework for Large-Scale Inter-net Delay Measurements. In IEEE INFOCOM - The 27th Conference on ComputerCommunications. 31–35.
[23] J Ott, M Sanchez, J Rula, and F Bustamante. 2012. Content delivery and thenatural evolution of DNS. In Proceedings of the Internet Measurement Conference.ACM, 523–536.
[24] PDNS 2019. PowerDNS Recursor. Retrieved 2019-09-07 from https://www.powerdns.com/recursor.html
[25] David Plonka and Arthur Berger. 2017. kIP: a Measured Approach to IPv6 AddressAnonymization. arXiv preprint arXiv:1707.03900 (2017).
[26] RIPE Atlas 2019. Welcome to RIPE Atlas. Retrieved 2019-09-07 from https://atlas.ripe.net/
[27] Kyle Schomp, Tom Callahan, Michael Rabinovich, and Mark Allman. 2013. OnMeasuring the Client-Side DNS Infrastructure. In Proceedings of the InternetMeasurement Conference. ACM, 77–90.
[28] Shadow 2019. Open Resolver Scanning Project. Retrieved 2019-09-07 from https://dnsscan.shadowserver.org/
[29] Philip Smith, Rob Evans, and Mike Hughes. 2006. RIPE routing working grouprecommendations on route aggregation. Document ripe-399, RIPE (2006).
[30] Florian Streibelt, Jan Böttger, Nikolaos Chatzis, Georgios Smaragdakis, and AnjaFeldmann. 2013. Exploring EDNS-Client-Subnet Adopters in your Free Time. InProceedings of the Internet Measurement Conference. ACM, 305–312.
