In such diseases, the fungi are actively growing on and invading the body of their hosts.  There is another means by which fungi can cause harm without invading our bodies.  When fungi grow on a living organism or on stored food material that we consume, they may produce harmful metabolites that diffuses into their food.  It is believed that fungi evolved these metabolites as a means of protecting their food supply by preventing other organisms from eating it.  These metabolites are referred to as mycotoxins, which literally means "fungus poisons."  Great lecture by George Wong, Department of Botany, University of Hawaii.