GTA Online is Rockstar's official multiplayer mode. It has set content, set rules, a fixed number of players per session and a focus on missions, heists and free-for-all play.
FiveM
FiveM is a modification framework that connects you to community-run servers. Each server chooses its own rules, maps, vehicles and systems, and many can host far more players than a GTA Online session. Roleplay servers use that freedom to build persistent worlds with jobs, economies and storylines.
Which should you choose?
Choose GTA Online if you want Rockstar's official missions and heists.
Choose FiveM if you want to play a character, work a job or build a story with other people.