The Registration Data Access Protocol (RDAP) is a successor to the WHOIS protocol and enables programmatic access to the registration data of internet resources. Using RDAP, we investigated phishing attacks observed over four days, focusing on DNS domain names. In this paper, we present the opportunities and problems identified in the process. We find that low RDAP adoption among ccTLD registry operators, strict rate limiting, diferences in data representation, and the (un-)availability of data due to privacy regulations continue to be hindrances to the widespread use of RDAP in cybercrime investigations. While these issues are currently preventing security researchers from solely relying on RDAP for accessing domain name registration data, we recognize its potential as a valuable data enrichment source for investigating phishing attacks at scale.
in the context of phishing attack investigations.
The expected response, assuming that the RDAP service has information about the queried object, is given in JSON following a schema defined by RFC 9083 [ 8]. Thus, the response should be given in a machine-readable format under the mime type application/rdap+json.
With RFC 9224, the IETF standardized a way to identify the authoritative RDAP service for the internet resource that should hold valid registration information on a domain name, IP address, or ASN. The so-called "bootstrap services" associate TLDs, IP ranges, and ASN ranges with their respective authoritative RDAP services
Some registry operators do not hold all of the required Finally, the domain name registration data should
condomain name registration data for all their registered do- tain the abuse contacts of the sponsoring registrar. If a
main names ("thin" registries). Instead, they direct RDDS domain name is deemed to be registered with malicious
queries to the sponsoring registrar through which the intent, security researchers can report it to the registrar,
domain name was registered [
With its properties of machine-readability,
transportencryption, and internationalization, RDAP is a signifi- To understand whether RDAP can be used to gather
docant improvement over its RDDS predecessor, the WHOIS main name registration data to be subsequently employed
protocol. WHOIS does not define structured responses in the fight against phishing as described in 3, we built a
and instead returns unstandardized plain text, is not software system that analyzes the availability and data
transport-encrypted, and does not standardize handling quality of domain name registration data and tested it
of encodings other than ASCII, which is a problem for against a realistic workload of domain names suspected
languages with non-ASCII character sets [
Domain name registration data, accessed either via WHOIS or RDAP, can be applied in the fight against phishing for multiple purposes: The age of a domain name, as given by its registration date and subsequent renewal dates, is often used as an indicator of its trustworthiness; with older domains being considered more trustworthy than more recently registered ones. This practice has led to some threat actors waiting for a while between registering a domain name and starting to use it for malicious purposes, to "age" their domain names and thereby giving them a higher chance of evading detection [12] [2].
If a domain name is confirmed to be involved in a phishing attack, its age can be used to diferentiate between domain names that were registered solely for this malicious purpose and benign domain names merely pointing to a host that was compromised by the threat actors [13]. A third option to be considered is the misuse of legitimate file- or web-hosting services for phishing purposes. This classification is important, as the counter-actions taken by security researchers difer based on the type of phishing at hand.
While the registrant data provided by threat actors
is often falsified, it can be used to cluster domains that
were registered in bulk, and thereby help to detect new
phishing domains [
The purpose of the "rdapper" software system was to correctly process requests for registration data of domain names following the RDAP standard, temporarily store results in a database for caching purposes, and schedule the requests to RDAP services in order to comply with the providers’ terms of service.
Key operations of the system were configured to send telemetry data to a monitoring and visualization system.
This allowed us to oversee the operation of the system and extract metrics after the experiment had concluded.
The RDAP standard, which is heavily based on well established web-standards and technologies, makes it straightforward to implement an RDAP client that identifies the authoritative RDAP service for a given internet resource [9], sends an HTTP request to the RESTful API [7] [5], and parses the JSON response [8]. But in this scenario, we would be ignoring the rate limits that are defined in the "Terms of Service" or "Acceptable Use" policies of the many RDAP services the client might connect to. To adhere to these rate limits, we built a scheduling system for our RDAP queries. It took into account the time of the last lookup to this RDAP host and possible back-of requests in the form of HTTP responses with status code 429 ("Too many requests") and their Retry-After response header values if they were defined by the server. This delay was individually conifgurable for each RDAP host. When building such a scheduling system, it is necessary to track queries per RDAP host, not per RDAP service, as some RDAP services for diferent TLDs are hosted on the same server and track RDAP queries across all services.
We chose a default delay of five minutes between queries to the same host based on the longest observed required delay when we began this project. After each run of the experiment, we identified the RDAP services
RDAPSupport generic
com top xyz or g nt e that had accumulated the longest delay between the request of domain registration data and the execution of the RDAP query. We then tried to find stated rate limits in the Terms of Service documents of these registry operators or registrars. If we could not find any public rate limit information, we would reach out to the RDAP service provider and ask them for safe rate limits for their RDAP API. Then, we would adjust the delays for those RDAP hosts accordingly.
The system identified itself to the RDAP service providers by sending request headers with a unique User-Agent and an email address in the From header, in order to give RDAP service providers the ability to contact us should our queries cause issues or breach any terms of service. 1 We did not rotate our clients’ IP addresses or implemented any other attempts at evading any potential terms of service enforcement by the RDAP service providers. 4.2. Analyzed domain names de tk cn ru ga biczcza fr br c it auus ws in pl chtwsezaesphjpbeatme e o g u n q cf ml l country-code N
oRDAP gen-restricted
country-code uk z zek buliwvostrolroeanvipshopclubsite online icu info 1We did not receive any emails. 2The data collection days were May 10th 2023, May 22rd 2023, May 31st 2023, and June 5th 2023, with the processing days being the following day respectively. on May 12th 2023 resulted in a calculated coverage of services from this list. 67.45% of registered domain name having an oficial au- One registrar-run RDAP service was served behind thoritative RDAP service assigned to them [16]. This a bot detection service of a content delivery network coverage can be expected when the distribution of TLDs provider, making programmatic access impossible. in a workload is similar to the distribution of all registered One registrar had restricted access to the individual domain names across TLDs. domain name lookup endpoint, an example of which is
For our workload of domain names suspected to be shown in 1. When notified about this, they pointed us to involved in phishing, the distribution across TLDs difers their RDAP endpoint to search for domains instead. This slightly from the general population. In practice, and is not a viable solution, as the thin registry RDAP serincluding the seven unoficial RDAP services, we reached vice continues to re-redirect queries to the RDAP query a coverage of 78.86% of our domain name test dataset, endpoint for individual domain lookups. the TLD distribution of which can be seen in Table 3.
As described in Section 4.1, scheduling RDAP queries to Depending on the setup as a thin or thick registry, the comply with the terms of service of the RDAP services registration data lookup for one domain name might in- and, in particular, the rate limits is a crucial aspect when volve RDAP queries to one or two RDAP services run by developing an RDAP client to operate at a certain scale. a registry operator or a registrar. Generic and generic- We found that our default delay of 300 seconds between restricted TLD registry operators and registrars are re- requests generally seemed to work, and for the most part, quired by ICANN to operate RDAP services [17]. But did not result in back-of responses or even permanent these services are no profit centers and are not "mission- IP blocks by RDAP service providers. critical" for most paying customers of these organizations. We identified rate limits either in public terms of serDuring our experiment, we saw 399 authoritative RDAP vice documents or in other parts of the the web presence services run by registry operators and 229 RDAP services of eight RDAP service providers. One of those has since run by registrars, which we were pointed to by RDAP removed the document, but the rate limits of the remainresponses of registry operators. ing seven RDAP service providers are listed in Table 4.
We observed two RDAP services run by registry op- We contacted eleven RDAP service providers, priorierators that continuously returned HTTP 500 ("Inter- tizing those who had accumulated the longest processing nal server error") responses. One of those services also delays during our experiment. Two of those, both gTLD started to serve an expired TLS certificate for around registry operators, refused to share the rate limits they ifve days, and returned to serving HTTP 500 responses enforce. Four shared their rate limits after being conafterwards. Both RDAP services have since been fixed. tacted, with the highest rate limit being one query per
Of the 229 RDAP services run by registrars, we ob- second and the lowest five queries per minute. In four served eight that served an invalid or expired TLS certifi- cases, we never got an answer to our contact attempts via cate or that were only available via HTTP without TLS contact forms on the organization’s websites or support on port 80. email addresses. In one case, we did not manage to reach 25 registrar-run RDAP services were consistently un- people with knowledge about the RDAP service of that available, meaning that they never served valid RDAP organization. responses. If we only contacted these services on two The lowest rate limit we observed was determined distinct days or less, we manually confirmed the unavail- through experimentation, as the registrar did not react to ability of the services after the experiment phase had con- our contact attempts: Their RDAP service was configured cluded, before categorizing them as consistently unavail- to allow just one query per hour per IP, before responding able. Through this method, we excluded six registrar-run with an HTTP 429 "Too many requests" response.
In total, we sent 67,607 queries to RDAP services. This is because 25,873 (48.57%) of all domain names in the test dataset caused us to make two or more queries, the first to the TLD registry operator and the following ones to the sponsoring registrars that we were pointed to in the RDAP response of the registry operator. We had wrongly assumed that only thin WHOIS registries would include links to the RDAP services of sponsoring registrars, but this was incorrect: Some thick WHOIS registries, like CentralNic for the .xyz TLD, include links to the RDAP service of the sponsoring registrars, too.
In 13 cases (six .org and seven .info domain names) we found "related" links to a third RDAP service in the response of the second RDAP service. None of the RDAP services behind these links responded to our queries.
Of the 42,010 domain names with RDAP support, 13,206 (31.44%) were not registered, according to the received RDAP responses. This might be an artifact of the test dataset, which includes domain names that were parsed from spam emails and SMS. As the goal of this process is to extract any potential link from the messages, some of the extracted domain names, while being technically valid domain names, might not actually be registered. It might also be caused by the fact that we were querying the registration data of these domains 24 hours after they were reported, and by that time, they might have already been suspended by the registry operator or sponsoring registrar. However, in these cases, the status of the domain name is usually changed to "server hold" if the action was taken by the registry operator, and "client hold" if the action was taken by the registrar [19]. We observed 1,364 domain names with a "server hold" status, 2,447 domain names with a "client hold" status, and 437 domain names with both statuses, in total 7.97%. 3 Another explanation could be that some RDAP services might have a significant delay between the registration of a domain name and the update of the registration data database, on which the Registration Data Directory Services rely. We investigate this option in the following section. 5.4.1. Data Freshness 46,034 of the recorded RDAP responses include an event called "last update of RDAP database", which, according to the ICANN RDAP Response Profile specification, must contain "a value equal to the timestamp when the RDAP database was last updated" [20].
We compared this self-reported timestamp to the time of our system storing the RDAP result. This timestamp is recorded just after the HTTP response of the RDAP service has been received. Because of that, and to account for minor time-keeping ofsets between our database server and the RDAP servers, we allowed for a 10-second bufer time.
866 RDAP database update timestamps were excluded because they lay significantly in the future without specifying a time zone. 1,220 timestamps were excluded because the RDAP service was likely implemented incorrectly, as the supposed timestamp of the RDAP database update always matched the "last changed" event, stating when the information about the object was last changed, or the "registration" event of the domain name [8].
36,260 RDAP database update timestamps (82.5% of the available and correct timestamps) were within 30 seconds of our request, which indicates that they configured their RDAP service with a "live" registration data database setup. In this case, the time of the request is taken as the time of the last RDAP database update. 3This includes variations like "client_hold" and "ServerHold" 5.4.2. Data Redaction 4Required in the specification is a remark of type "object truncated due to authorization", but we also included variations like "object redacted due to authorization" and "object redacted due to privacy laws". structures of their RDDS backend. An anonymized example of such an entity object representation is shown in Appendix Listing 2.
Another two registrar-run RDAP services returned HTTP responses in which JSON arrays, as used in the RDAP standard to list the events, were instead represented by JSON objects with the array indexes as names and the array elements as the respective values [8]. These two registrars were only responsible for eight domain names in our dataset.
The General Data Protection Regulation of the European Union (GDPR), which took efect in 2018, required ICANN to update its policies for gTLD registry operators, in order to enable them to stay compliant both with the law in these jurisdictions and their responsibilities as registry operators [21]. As the GDPR also applies to data controllers not based in the EU who are storing data of EU citizens, this also afected non-EU registry operators and registrars [22]. ICANN agreed on a "Temporary Specification for gTLD Registration Data" specifying which registration data must be redacted by gTLD registry operators and registrars [21].
As it is not always possible to programmatically distinguish between redacted, removed, and un-available registration data, we check for the existence of markers required by ICANNs RDAP Response Profile specification [20] to indicate truncation or redaction of entity objects.4 17,716 RDAP responses for 16,546 distinct domain names contained at least one entity that was truncated or redacted. This constitutes 61.66% of the domain names for which we got successful RDAP responses. Not all remaining RDAP responses necessarily contain unobfuscated registrant information, as some RDAP service providers do not declare the information as redacted, even if it is obfuscated.
Apart from attempts to cluster domain names registered with malicious intent via the registrant information, another use case for RDAP in the fight against phishing is the identification of abuse contacts of the sponsoring registrar. For 19,250 unique domains, we got a thick RDAP response containing a contact entity with an "abuse" role. This constitutes 71.72% of all domains for which we got successful thick RDAP responses.
Listing 2: Shortened and anonymized example a nonstandard RDAP entity object representation "objectClassName": "entity", "vcardArray": { "properties": [ { "name": "FN", "value": { "stringValue": "Domain Administrator", "typeName": "text" "name": "ADR", "value": { "components": [ { "name": "street", "value": { "values": [ { "name": "TEL", "parameters": {}, "value": { "stringValue": "tel:+0.13371337", "typeName": "uri" "name": "EMAIL", "value": { "stringValue": "example@example.com", "typeName": "text" "stringValue": "1337 Lowland Ave.", "typeName": "text" }, { "stringValue": "PMB# 333", "typeName": "text" } ], "typeName": "text" "stringValue": "Example City", "typeName": "text" "name": "locality", "value": { "values": [ { } ], "typeName": "text" } }, ], "typeName": "text" 300/day and 10/min 1/min
Normalized query delay
Source https://registrar-console.centralnic.com/pub/whois_guidance https://img1.wsimg.com//Sitecore/3/B/ GDR-RDAP-Access-Policy-0.2.pdf https://www.isnic.is/en/rdap https://domain.tatar/users/docs/WhoisTermsOfUse_en.php https://media.nominet.uk/wp-content/uploads/2019/06/ gTLD-Acceptable-Use-Policies-version-1.pdf https://teknisk.norid.no/en/integrere-mot-norid/rdap-tjenesten/ https://tucowsdomains.com/rdap/help/