Might be I think too much
That even for a milliseconds my mind say that you must give this xxx money as sadqa

Some of my brothers says that if you intend to give this ____ as sadqa you have give and don’t use for your own(it will haram for You)

This I think is for milliseconds
Do I need to give that(xxx money) as sadqa?
Can I prevent that and can I use that for my own use?

It all depends on your niyyat (intention). If you have done your niyyat confirmed then you can not use it again for yourself despite that the amount or thing has not been separated out from you and still in your possession. But if your Niyyat is not confirmed yet your are still in dilemma then you can use it.