If God knows that we are going to end up in heaven or hell in the future, is that a stamp on us regardless of our actions? For eg, when a baby is born, God knows the baby will go to heaven or hell in the next life?

It’s not a stamp on us in the sense that we have no free will. Allah’s knowledge doesn’t compel us to out destiny/fate. Allah is above time and space, so He can see our entire life all at once. This is like seeing the film of a movie with one look (imagine you can see all the scenes at once). Even though He knows what we will do in the future, we still have free will.