An algorithm is by definition "a finite sequence of well-defined instructions, typically used to solve a class of specific problems", in the case of SEO, that problem is website ranking.
When most SEO professionals refer to "the algorithm" or "Googles algorithm", we're typically talking about the math behind Google's (and others) ranking factors.
These algorithms take the data gathered by millions of web crawls, look at all the data points and calculate page ranking based on those and several other factors, like backlinks and domain authority.
Google has a whole page dedicated to talking about how their algorithm works, should you wish to learn more.