I agree with the above posters. The best way I have found to glue down sand or ballast to a base is to 'sandwich' it between layers of glue. I put down the PVA and then add the sand and let it sit overnight. I then take my finger or a stiff brush and try to rub off as much as I can to get rid of the loose stuff. I then take and put down a 1:1 water/glue mixture and let that dry for at least an hour.
I have a full writeup over at my blog that has a few pictures if you are interested -
link