A man who subjected a young mother to sectarian intimidation following a Rangers vs Celtic football match has abandoned a legal challenge to his prison sentence.