Hi Isabella,

Is it possible that what is falling is the fruit and not the buds? Maybe it's too warm where you have it located? Take a look here.

http://www.thegardenhelper.com/passion.html

From this site:
http://www.floridata.com/ref/p/pass_cae.cfm

quote:

---

Blue passionflower does better overwintered in a cool greenhouse where it can go semi-dormant as opposed to in a hothouse where it will be tempted to put too much energy into weak off-season growth. In either case, it is important to keep the soil on the dry side in the winter.

---

Newt

* * * *
When weeding, the best way to make sure you are removing a weed and not a valuable plant is to pull on it. If it comes out of the ground easily, it is a valuable plant.