hello Archy
Its not so. Infact your page rank get improved. Directories submissions are the part of seo works, infact most of the SEO recommend this.

This is how it goes....

Your are submitting your site to the directories which are the most usefull source of traffic to your site. So if any one searches in google for the keyword of your site then your site appears in the top depending on the many factors.