How does DHT in torrents work?

With trackerless/DHT torrents, peer IP addresses are stored in the DHT using the BitTorrent infohash as the key. Since all a tracker does, basically, is respond to put/get requests, this functionality corresponds exactly to the interface that a DHT (distributed hash table) provides: it allows you to look up and store IP addresses in the DHT by infohash.

So a “get” request would look up a BT infohash and return a set of IP addresses. A “put” stores an IP address for a given infohash. This corresponds to the “announce” request you would otherwise make to the tracker to receive a dictionary of peer IP addresses.

In a DHT, peers are randomly assigned to store values belonging to a small fraction of the key space; the hashing ensures that keys are distributed randomly across participating peers. The DHT protocol (Kademlia for BitTorrent) ensures that put/get requests are routed efficiently to the peers responsible for maintaining a given key’s IP address lists.