Tap water in Jordan is generally safe to drink, but for a short trip it's better to stick to bottled water. This is readily available but check the seal has not been broken. Alternatively, you can boil tap water for 10 minutes, use water purification tablets or a filter.
The tap water in southern Jordan, particularly Wadi Rum, comes from natural springs at Diseh and so is extremely pure. Avoid drinking water from wadis in the wild as pools may have been used as waterholes for livestock. In the Jordan Valley, amoebic dysentery can be a problem.
If you get stuck in the desert without water, remember that you are more likely to be seriously ill (and even die) from dehydration than you are from an upset stomach, however unpleasant it may be. In summary, if water is offered and you need it, worry about its provenance later!