Did I mess up my first ritual badly, or can I just learn from it?

A first attempt being imperfect does not mean you caused anything bad. You noticed what went wrong, and you can use that to make the next attempt simpler. There is no need to repeat it right away just to make it feel correct.

Since you already felt uneasy at night, try practicing the spoken or visual parts in daylight without candles first. You can also pause or stop if your anxiety starts climbing. A ritual should not require pushing through panic.

I'd separate the practical stuff from the ritual stuff. Moving flames and dark smoke are good reasons to extinguish the candles and check the wick, airflow, and room ventilation. Keep candles in proper holders, away from anything that can catch, and don't leave them burning unattended.

For the directions, pick one set of instructions you trust and write out the sequence beforehand. If a step gets muddled, stop and regroup rather than improvising while stressed. A mistake in the sequence is not, by itself, evidence of harm.

The OCD part sounds like the piece I'd be gentlest with. Trying to redo a gesture until it feels exactly right can turn practice into a checking loop. You could set a simple rule ahead of time, such as continuing after a small mistake instead of restarting, or skipping the ritual for now if that feels safer.

Feeling keyed up afterward can happen when you've been concentrating and anxious, but you don't have to read it as a sign or act on it. Ground yourself, eat or drink something, and give yourself time to settle. If the intrusive thoughts or paranoia are getting harder to manage, consider talking with a mental health professional familiar with OCD.