It does not matter how long they have stayed. Every individual you see in your life, all influences your thoughts, your experiences and your life. You may not know them. You may not talk to them. You may not be in love. True to my experience, the feelings you might have, experiences you might have, thoughts you might think, all those things on which your life is depending on, get affected by all those people.
You cannot do anything about it. Because that is how your universe keeps the balance in the reality of your life.
So it is okay if you do not get to know someone or befriend with everyone or may be in love with the one. At least, unknowingly, that person got to be a part of your world and may be at least once you might have got to be the part of their thought.
Now, before you get overwhelmed about it, take a deep breath, inhale all of those good influences and exhale all those unnecessary over-thoughts. So that the universe can take care of everything else.