So, from what I can tell, you want four columns of relatively equal length on a page, with a title letter every time you move on to a new letter of the alphabet.
<?php
$num_cols = 4; // set the number of columns here
$paged = (get_query_var('paged')) ? get_query_var('paged') : 1; // for pagination
$args = array(
'orderby' => 'title',
'order' => 'ASC',
'posts_per_page' => '199',
'paged' => $paged,
'post_type' => 'wiki'
);
$wikis = new WP_Query($args);
//end of query section
if ($wikis->have_posts()) :
// figure out where we need to break the columns
// ceil() rounds up to a whole number
$break_at = ceil( $wikis->post_count / $num_cols );
// start with the first column
$col_counter = 1;
$post_counter = 1;
// Set the title letter empty so that it's always output at the beginning of the cols
$initial="";
?>
<div id="col-<?php echo $col_counter ?>" class="col">
<?php while ($wikis->have_posts()) : $wikis->the_post();
// Start a new column (but not the first one)
if( $post_counter % $break_at == 0 && $post_counter > 1 ) :
$col_counter++;
?>
</ul></div>
<div id="col-<?php echo $col_counter ?>" class="col"><ul>
<?php endif;
$title = get_the_title();
$wiki_letter = strtoupper(substr($title,0,1));
if( $initial != $wiki_letter) : ?>
<?php if ( $post_counter > 1 ) : // close the previous ul ?>
</ul>
<?php endif; ?>
<h3><?php echo $wiki_letter ?></h3>
<ul>
<?php $initial = $wiki_letter;
endif; ?>
<li>
<a href="https://wordpress.stackexchange.com/questions/124053/<?php the_permalink(); ?>" rel="bookmark" title="<?php _e( 'Permanent Link to', 'buddypress' ); ?> <?php the_title_attribute(); ?>"><?php echo $title ?></a>
</li>
<?php $post_counter++; ?>
<?php endwhile; ?>
</ul>
</div>
<?php wp_reset_postdata();
endif; ?>
This code does have a small bug where if a new letter of the alphabet starts at the same time as a new column, there’s an empty <ul> tag at the beginning of the column.