How can I get my deposit back and/ or take legal action?

How much was your deposit? He shouldn't be charging more than about $25 to remove a bag of trash from the unit. Do you have a copy of your lease. If not, request a copy from him and read through it. You may find some answers there. Also, I would ask him for proof (pictures) of the what you were charged for. Sounds like a small claims court situation if he did in fact unfairly keep your deposit. If you went to court, I would think the burden would be on him to prove why he kept all of your deposit.