Modern antivirus combines three methods: signature scanning (comparing file hashes against a database of known threats), heuristic analysis (looking for suspicious behavior patterns that resemble known malware even in new files), and behavior monitoring (watching running processes for actions associated with malware, such as encrypting files en masse).
Antivirus has significant limitations. It cannot detect zero-day exploits — attacks using vulnerabilities that have not yet been patched or catalogued. It cannot stop social engineering attacks where you willingly install something or hand over credentials. File-less malware that runs entirely in memory leaves no file for antivirus to scan.
The built-in security tools in modern operating systems are more capable than most people assume. Windows Defender in Windows 10 and 11 consistently scores well in independent tests. macOS Gatekeeper prevents running unsigned software. Using the built-in option and keeping it updated is a reasonable baseline for most users.
If you choose a third-party security suite, look for one with independent test results from organizations like AV-TEST or AV-Comparatives. Avoid suites with a history of false positives, high performance impact, or aggressive upselling. More expensive does not automatically mean more effective.