If water gets for a skimmer and freezes, damage is almost inevitable.
Another thing better about fencing is that it doesn't cover the beautiful landscaping your pool water provides. If it is due to the budget, look to obtain one with mildew inhibitors, reinforced seams and even UV stabilizers.