Supported TLDs include br.com, cn.com, de.com, eu.com, gb.com, gb.net, gr.com, hu.com, in.net, no.com, qc.com, ru.com, sa.com, se.com, se.net, uk.com, uk.net, us.com, uy.com, za.com, jpn.com, web.com, com, za.net, net, eu.org, za.org, org, llyw.cymru, gov.scot, gov.wales, edu, gov, int, e164.arpa, arpa, aero, asia, biz, cat, coop, info, jobs, mobi, museum, name, post, pro, tel, travel, xxx, ac, ae, af, ag, ai, am, ar, as, priv.at, at, au, aw, ax, be, bf, bg, bh, bi, bj, bm, bn, bo, br, by, bw, bz, co.ca, ca, cc, cd, ch, ci, cl, cm, edu.cn, cn, uk.co, co, cr, cx, cz, de, dk, dm, do, dz, ec, ee, eu, fi, fj, fm, fo, fr, gd, ge, gf, gg, gh, gi, gl, gp, gq, gs, gy, hk, hm, hn, hr, ht, hu, id, ie, il, im, in, io, iq, ir, is, it, je, jp, ke, kg, ki, kn, kr, kw, ky, kz, la, lb, lc, li, lk, ls, lt, lu, lv, ly, ma, md, me, mg, mk, ml, mm, mn, mq, mr, ms, mt, mu, mw, mx, my, mz, na, nc, nf, ng, nl, no, nu, nz, om, pe, pf, pk, co.pl, pl, pm, pr, ps, pt, pw, qa, re, ro, rs, ac.ru, edu.ru, com.ru, msk.ru, net.ru, nov.ru, org.ru, pp.ru, spb.ru, ru, rw, sa, sb, sc, sd, se, sg, sh, si, sk, sl, sm, sn, so, ss, st, msk.su, nov.su, spb.su, su, sx, sy, tc, td, tf, tg, th, tk, tl, tm, tn, to, tr, tv, tw, tz, biz.ua, co.ua, pp.ua, ua, ug, ac.uk, gov.uk, uk, fed.us, us, uy, uz, vc, ve, vg, vu, wf, ws, yt, ac.za, co.za, gov.za, net.za, org.za, web.za, zm among many others.
Opensource WHOIS Library by Xeptagon
When it comes to getting information regarding domain name registrations, the most trusted and often the most accurate source is the WHOIS service. The information provided by WHOIS is publicly available and includes valuable information such as the date of registration, the registrar, status and expiry date among others.
This domain information is critical for some stakeholders in various domains when it comes to making business decisions as well as for domain drop catching. The WHOIS services always have the most updated (public) information regarding domain name registrations and because of this, there also exists a separate market for the historical WHOIS information as well.
When it comes to collecting data from WHOIS servers, one can use either text-based services (using TCP) or web-based services. Since these services are publicly available, they usually have rate limitations depending on the host regarding how many queries can be performed for a defined time limit. In addition, these services have limitations on the number of TLDs supported and don’t support batch processing, parallelization, proxies, or parsing as well as paid services.
Considering these issues and limitations, Xeptagon developed a highly customizable opensource TypeScript WHOIS library which collects and parses given WHOIS queries from Text-based WHOIS servers. We used our experience in developing state-of-the-art domain drop-catching systems when developing the WHOIS library.
The library contains the WHOIS servers for the specific TLDs which are used to collect data for the submitted domains. In scenarios where the list of WHOIS servers does not contain the WHOIS server for the submitted domain’s TLD, then the IANA WHOIS server is used to query the TLD and get the WHOIS server for the submitted domain.
The application then creates a TCP connection to the <WHOIS-SERVER:PORT> (default port 43) and sends the WHOIS query for the submitted domain and collects the response. The connection can also be established through a provided proxy server (Socks4 or Socks5) as well (with or without authentication), where the proxy IP will be used to collect the WHOIS data for the domain. Apart from that, the user can decide which data they want to extract from the WHOIS response of the domain by submitting a key-value paired Object alongside other options as well.
The Xeptagon WHOIS library is also capable of processing multiple domains as batches (parallelly) based on user-defined configurations. The batch processing also includes proxy support which is useful for collecting large amounts of data from WHOIS servers, reducing the delay caused by the rate limitations as well.
Overall, the Xeptagon WHOIS library addresses all the limitations the other applications currently on the market have and is publicly available to be used. The usage is simplified a lot, even the less tech-savvy users can use it to collect data with a few lines of code.
The complete documentation of the Xeptagon WHOIS library of different configurations can be found here with multiple examples.
Need more information or support in WHOIS data collection? Contact us at info@xeptagon.com.