Will a large postmeta table slow a site down?

I have created a hit counter that stores all time hits and daily hits. This means that if a single post gets at least one view a day it will have 365 meta fields added to it by the end of the year. Multiply this by N posts and the number can grow considerably large. Will this ever be an issue? I know WordPress caches stuff and this is why i am worried about it. Will all of these be loaded each time the post is loaded.

This will better explain what i have implemented.

$dailyhits = get_post_meta( $post->ID, 'hits_'.date('Y-m-d'), true );
if($dailyhits == null || $dailyhits == "" || $dailyhits == false) $dailyhits = 1;
else $dailyhits = $dailyhits + 1;   
update_post_meta($post->ID, 'hits_'.date('Y-m-d'), $dailyhits);

“You should not write to the DB on front end requests” is the eleventh commandment. Your performance will deteriorate due to the writes locking down the table much more than the impact the size of the table will have.

You should probably write such data in files instead of DB, or at least use a different table.