robots.txt lives at the root of your domain and lists rules per crawler, using User-agent lines with Allow and Disallow paths. Well-behaved bots read it before crawling and follow it. You can also list your XML sitemap in it.
Two important limits. First, it controls crawling, not indexing: a blocked URL can still appear in search results if other sites link to it, and a page you want kept out should use a noindex signal instead (which only works if the page can be crawled to see it). Second, it is advisory. It does nothing to stop a bot that ignores it and is not a way to protect private content.
Increasingly it is also where you set rules for each AI crawler, since training, search and user-triggered bots use different names. A single wrong line, such as a stray Disallow: /, can block a whole site, so review changes carefully.
Related: Crawling, Technical SEO, llms.txt.