No problem, its happened to me before. ^.^
Also, its really helpful to the refs if you put a (C) next to the person's name who originally made the challenge, and put a (A) next to the person's name who accepted to challenge. Its not a requirement, but it does help us out when we take a battle.