Discussions

Ok, so perhaps i have some confusion about this issue, but with a yeild sign, are you not expected to slow down, and if another car is approaching on the street, yeild the right of way to them?  
I'm asking because my husband and I were driving in Brandon a couple of days ago, and were pulled over for a "Failure to yeild". Now I know we slowed down and gave a slight pause at the sign, and when we saw that there were absolutely NO cars driving on the street we continued. (It was after midnight, not too many people out, it was easy to see there was no one driving that road at all). a few seconds later bright flashing lights pulled us over. (We were given just a warning because of our clean driving record which i appreciate, but i'm still confused. lol.)  
Am I wrong in my understanding of what to do at a Yeild sign?