This works by using each line in b.txt as a pattern (-f b.txt) and treating it as a plain string to match (not a regular regex) (-F). You force the match to happen on the whole line (-x) and print out only the lines that don't match (-v). Therefore you are printing out the lines in a.txt that don't contain the same data as any line in b.txt.
So remember the order of file a and b matters if you really understand what the command is doing.